Check this study
You do not need a credential to find what is wrong with a survey. You need the questions, the rules and the history in front of you. Here they are, with the places we think the study is weakest, and a form that puts your objection on a public count whether or not we agree.
No study data is collected from anyone yet (the pilot’s answers and objections sent here are the only things kept), and the plan is not registered. This page is how the study is made checkable before it opens, not after. No one who has read the plan and did not write it has signed off on it, and we say so below.
01
The questions, word for word
What a person in the pilot is asked, from the same copy their own page reads. The study asks the same four things (how strongly, where it went, when, what was going on), but by text or voice note and with only the four places to choose from, so this form is not the study’s instrument. The pilot adds three choices the study will not have, to find out whether the four are enough.
How frustrated were you?
0 = barely registered, 10 = as worked up as you get. Eleven choices, 0 to 10.
Where did most of that frustration end up?
Pick the one that is closest. The first three are shown in one of six orders, the same each time for a given person; the rest are always last.
- On a tool, app or system
- On someone else
- On myself
- It faded without landing anywhere
- I kept it in
- Split about evenly between two or more of these
- Somewhere else
The study’s own names for the four places are the tool, other people, myself, nowhere; “nowhere” is two choices here. Underneath: “If you picked “split” or “somewhere else”: between what, or where? A few words.” and a box, “None of these fit well, even the closest one”.
When did it happen?
Just now · Within the last hour · Earlier today · Before today. Moments from before today are counted but left out of every test.
What was going on?
A sentence or two helps: what you were doing, who or what was involved, what got in the way. Skip anything you’d rather keep private.
Whether a system was involved is never asked. It is read from the story by two coders, so no question or example can put the idea in anyone’s head.
02
The rules that decide what we may claim
Each is written into the plan before any data, to be registered before the study opens, with what happens if it is missed. If a result does not clear its bar, the bar does not move.
The four places frustration can go are kept only if the pilot shows they fit.
≥ 90% clean · lower bound ≥ 80% · ≥ 40 answers from ≥ 10 people
If missed: The choices change before the study opens, for instance by splitting “nowhere” in two.
The main finding counts only if the odds are at least one and a half times higher and the 95% interval stays above 1.
odds ratio ≥ 1.5 · interval above 1
If missed: Reported as no meaningful difference, or as unclear when the estimate clears 1.5 but the interval reaches 1.
The second finding counts at half a point higher on the 0 to 10 scale.
+ 0.5
If missed: Reported as no meaningful difference.
Two coders, working apart, must agree on whether a system was involved before that measure is used at all.
Cohen’s κ ≥ 0.70 on a fifth of entries
If missed: The main analysis is reported as failed, not patched.
Only people with at least one moment of each kind can inform the comparison.
≥ 30 such people
If missed: Reported as too small to test, and only described.
Recruitment stops at a number or a date and does not reopen to chase a result.
90 people or six weeks
If missed: Nothing changes: analysis runs once, after the last person’s week ends.
The full wording of each is in the plan. Where this page and the plan differ, the plan wins, and that is a mistake to tell us about.
03
Where we think it is weakest
From an audit on November 12, 2026 of the wording, scales and order against survey practice, and from what the audit could not reach. “Fixed” means the change is on the site or in the plan. “Open” means it is not done.
- 01
Fixed
The pilot’s example could have taught people the answer.
The one example paired an app with snapping at someone, the study’s own pattern. The examples now pair no app with a reaction.
- 02
Fixed
The public pages stated the hypothesis before anyone enrolled.
The recruiting page no longer states the hypotheses or the sizes they must reach; they are in the plan, with a link, and the effect of having read them is written into the plan as a limit. It cannot be removed, only compared within each person.
- 03
Fixed
The key predictor is read from a story that is optional, so thin stories look like “no system”.
Entries are marked by how much was written and the main analysis is repeated on the longer ones. Uncodable entries are reported, never guessed. A neutral structured item is still undecided.
- 04
Fixed
The first model blended differences between people with differences within a person.
The registered model now separates them, and a conditional model is run as a check.
- 05
Open
The pilot form and the study’s text-message flow are different instruments.
The pilot form now has the same “when” item, but the free-text parser is not yet checked against what people actually pick. A labeled test set for it does not exist yet, and a second short pilot by text is still to come.
- 06
Open
We have not simulated whether 60 to 90 people can answer the main question.
The plan says the simulation is run before filing, and says so if the honest answer is “underpowered”.
- 07
Open
The screener items that make the automation score are not published.
They block registration until they are published here, in public, so they can be checked.
- 08
Open
No one has read the questions aloud who works the jobs the study is for.
Five to eight drivers, nurses and support agents say what each question means to them, before the wording is frozen. The pilot is our own network, which leans toward design and software.
- 09
Open
The only outside reading so far was an AI model’s, and then ours.
The plan requires a survey methodologist who is a person before it is filed, and we are looking for one who did not write it, because an author cannot see their own priming. An objection below is welcome, but it does not replace that reader.
04
What has changed, and when
Every change to the plan or the questions, newest first. Every change since this page was made is listed; earlier ones are as complete as the plan’s own history.
This page
The questions, the deciding rules, the weak spots and the changes are public in one place, and anyone can object. The questions here and on the pilot’s page now come from one copy.
The plan amended after an audit of the survey science
Neutral examples and recruiting pages; “frustration” and “where did that end up” replace “it”; the registered anchors on every prompt; choices shown in a per-person order; “nowhere” split into “it faded” and “I kept it in”; a “split” choice and a “none fit” flag; a “when” item; the pilot rule restated with its interval; other people and myself reported separately; a within-between model; thin and uncodable entries handled; a power simulation and interviews required before filing.
Two rules confirmed
The pilot rule (at least 90% clean, lower bound at least 80%, at least 40 answers from 10 people) and the main-finding rule (odds ratio at least 1.5, interval above 1).
The pilot moved from texting to a private page
A texting number needs carrier registration, which takes weeks. The pilot tests the wording, not texting; a second pilot by text comes before the study opens.
First draft of the plan
Question, hypotheses, what would count as no effect, the analysis and its limits, written before any data.
05
Object
So far: 0 received, 0 answered (0 changed the plan), 0 open. The count includes the ones not shown.
- Every objection that gets through is counted, including any we do not publish. Submissions our spam filter or daily limit turns away are not.
- We publish each one that is not abusive and does not identify a person, with our reply, whether or not we agree.
- We never publish a name or an address. You may leave one only if you want an answer by email.
- An objection that changes the plan is marked so, and the change is added to the list above, dated.