Everyone’s a critic

In 2019 I started an article about a method called PURE, and I never finished it. I wanted what everyone in my job wanted: one number that tells the people with the budget how good the product is, and whether it’s getting better.

The pull of that number is real. A score fits on a slide. A score can go up. And a score is cheaper than the alternative, which is putting the product in front of real people, watching where they get stuck, and doing it again after every change. Studies like that take weeks and money, so most teams run one a year, if that.

PURE promised a shortcut. I liked it then. Seven years later, I’ve read what it rests on, and I’d like to finish the article properly.

PURE stands for Practical Usability Rating by Experts. A team agrees on the five to ten tasks that matter most and the person who’d do them. Then a small panel of trained evaluators walks the happy path through each task and rates every step: 1 if it’s easy, 2 if it takes some thought, 3 if it’s hard. The steps add up to a task score, the tasks add up to a product score, and each gets a color, green through red. Lower is better.

When I first wrote about it, I compared it to the Tomatometer on Rotten Tomatoes: a panel of critics, a single score, a color you can read from across the room. I still think that’s the right comparison, and it’s worth following all the way through. On Rotten Tomatoes, the critics’ score and the audience score sit side by side, and the interesting movies are the ones where they disagree.

PURE was introduced at CHI 2016, the main academic conference for human-computer interaction, as a case study in the extended abstracts, not as a full peer-reviewed paper. Four of its five authors worked at Intel Security, where the method was developed. The fifth runs MeasuringU, the firm that ran the study meant to validate it.

That study covered three software products. New evaluators, with about an hour of training, agreed with each other at a correlation of about 0.5 on their first try; Intel’s own team did better, between 0.6 and 0.88. The scores lined up with what users said about ease of use at about 0.5, and with the System Usability Scale at about 0.4 to 0.5. The paper calls that fair. It’s an honest word: a correlation of 0.5 means the expert score explains about a quarter of what users report. None of the write-ups says how many users took part.

Since then, the method has been taught by Nielsen Norman Group, where one of its authors went on to work, and written up by MeasuringU. I looked for an independent replication, by anyone who didn’t build or sell the method, and didn’t find one.

To their credit, its authors say much of this themselves. NN/g’s article calls PURE’s numbers less precise than measures based on user data, says it covers only ease of use, and warns that not everyone is qualified to do it. That last line is where the trouble starts.

PURE depends on experts agreeing about what’s hard. Usability research has been measuring how often they do for more than twenty-five years, and the answer is: not often.

In 2001, Morten Hertzum and Niels Ebbe Jacobsen reviewed eleven studies in which several evaluators looked at the same system with the same method. On average, any two of them agreed on between 5% and 65% of the problems they found. No method fixed it, experience didn’t fix it, and it held for severe problems as much as trivial ones. They wrote that relying on a single evaluator is highly questionable.

It holds even when experts watch the same people. In an early study, four evaluators watched the same four sessions, and no problem made all four of their top-ten lists. In a 2014 study, nineteen experienced professionals analyzed recordings of the same five users. The average professional reported about a third of all the problems the group found, and when two of them reported the same problem, up to 30% of the time one called it critical and the other called it minor.

PURE asks a panel to settle on one number per step, and three people in a room can do that. What the research says is that a different three people would very likely settle on a different number.

So who’s qualified? In our industry, the most common answer is a certificate from Nielsen Norman Group.

Here’s what it is. To be UX Certified, you take five of NN/g’s live online courses, about thirty hours in all, and pass an exam after each one. The exams are online and unproctored: thirty questions, an hour, 80% to pass. Only people who attended the whole live course can sit its exam. NN/g puts the total at about $6,500, and its UX Master level, fifteen courses, at about $19,500.

NN/g states plainly that its certification is not accredited. It isn’t a degree, and nobody outside the company reviews it. It’s continuing education, sold and examined by the same company, about that company’s methods. That’s a perfectly good thing to buy if you want to learn. It isn’t evidence that the person holding it will rate a step the way the next certified person would, and the research says they mostly won’t.

None of this is a secret, and none of it is a scam: the prices, the requirements and the lack of accreditation are all published. The problem is the loop. A company teaches a method, certifies the people who learn it, and those people become the experts whose agreement the method depends on. Each piece points back at the others, and none of them points at a user.