How to Test AI UX With Real Users
Usability testing usually assumes that every participant sees the same design. AI features break that assumption: two participants who make the same request can get different answers, and one of those answers might be wrong. You can still test AI experiences with real people. You just have to design the study around the variation instead of hoping it averages out.
What changes when the output varies
In a conventional test, a problem that several participants hit is a design problem. In an AI test, a participant who struggles might be reacting to the interface, or to one bad output that nobody else saw. To tell those apart, you need to know exactly what each person was shown.
That leads to three changes in how you run sessions:
- Record the output. Save the exact response for every task in every session, not just what the participant did with it.
- Plan the inputs. Choose inputs deliberately, so the range of outputs participants encounter is something you designed rather than something that happened.
- Split the analysis. Look at how people respond to good outputs and bad outputs separately, because the goal is not only task success but appropriate reliance.
Write scripts with varied inputs
A standard task script gives every participant the same scenario. For AI features, prepare a small set of inputs for each task and rotate them across participants:
- Typical inputs that represent the most common real use.
- Ambiguous inputs where a well-designed system should ask a question or offer options.
- Hard inputs where you already know the current model often struggles.
- Out-of-scope inputs that the feature should decline or redirect.
Where it is safe, also let participants bring their own material: their documents, their questions, their data. Real inputs can expose problems that invented ones hide, because they carry the messiness of actual work. Get consent first, and decide before the session how you will handle anything sensitive that appears on screen.
Focus your observation notes on the moments that matter most for AI features. Did the participant notice an error? Did they open a source to check a claim? Did they edit the output or accept it as-is? When something went wrong, how long did it take them to recover, and what did they try first?
Measure whether trust is calibrated
Satisfaction ratings cannot tell you whether people trust a feature appropriately. A participant can love a feature that is misleading them. The concept you need comes from human factors research on automation. John Lee and Katrina See's 2004 review, Trust in Automation: Designing for Appropriate Reliance, opens with the core problem: "Automation is often problematic because people fail to rely upon it appropriately." Trust is well calibrated when how much people rely on a system matches how reliable it actually is.
To measure that, you need to know which outputs were correct. Either review every output after the session, or control some of them in advance with a prototype where you decide what the AI returns. Then sort each decision a participant made into a simple grid:
| Output quality | Participant relied on it | Participant checked, edited, or rejected it |
|---|---|---|
| Correct | Appropriate reliance | Under-reliance |
| Wrong | Over-reliance | Appropriate skepticism |
Over-reliance is often the more dangerous cell. When participants accept wrong outputs, look at what the interface showed at that moment. Was there a confidence signal? A source they could have checked? An easy way to edit? Under-reliance matters too: a feature people feel they must double-check every time saves them little effort.
A short self-report after each task adds a second lens. Ask "How sure are you that this result is correct?" before revealing whether it was. Comparing stated confidence with actual correctness shows whether the interface is giving people accurate signals.
If you plant deliberately wrong outputs, tell participants at the end of the session which ones they were.
Prototype with a Wizard of Oz
You do not need a working model to test an AI experience. In a Wizard of Oz study, a person behind the scenes plays the system. The name traces to J. F. Kelley, whose 1984 paper on designing natural language office applications called the technique the "OZ paradigm." In the early stages of that project the experimenter simulated the whole system, a calendar program that people operated by typing ordinary English.
For AI features, the wizard approach has one especially useful property: you control the output. You can make the system right, wrong, slow, or uncertain on purpose, then watch how the interface holds up in each case. That makes it a good way to test confidence displays, error recovery, and review steps before a model behaves consistently enough to test them for real.
The Usability Body of Knowledge entry on the method lists the main risks. Wizards need training to respond credibly, it is hard to keep responses consistent across sessions, and fatigue can change a wizard's responses over time. Reduce those risks with a bank of prepared responses for expected inputs and written rules for improvising the rest. Remember that results tell you about the design, not about how well a future model will perform. And because participants may believe they are using a real system, correct that impression by the end of the session at the latest.
Decide how many sessions to run
Jakob Nielsen's well-known guidance is that testing with 5 users uncovers most of the usability problems in a design, and that several small rounds beat one large study. That still holds for the interface questions in an AI feature. It does not cover output variation: five participants see only a small sample of what the model can produce. Keep sessions small and frequent for interface questions, and cover the variation with the two tools below.
Log what users saw, not just what they did
Outside the lab, logs show the full range of outputs real people receive. For each AI interaction, capture:
- The input, plus any context the system added to it.
- The model and prompt version that produced the output.
- The output itself, as it was displayed.
- What the user did next: accepted, edited, regenerated, copied, dismissed, or abandoned.
- How much they changed the output, if they edited it.
Edits and regenerations are among the most useful signals you have, because they mark outputs that were close but not right. Handle these logs with care: tell users what you collect, redact personal details you do not need, and set a retention limit.
Build a small evaluation set from what you learn
Every test session and every revealing log entry is a candidate test case. Collect them into a small evaluation set: real inputs, each paired with a short description of what a good response must do and, where relevant, the failure you observed. A few dozen well-chosen cases, drawn from the same four categories as your task scripts, is a reasonable place to start.
Run the set whenever the prompt, the model, or the retrieval setup changes, and have a person review the results against the descriptions. Automated scoring can come later, but early on, human review keeps the team honest about what "good" means. When a user session reveals a new failure, add it to the set so the fix stays fixed.
Connect the methods into one loop
Each method covers another's blind spot. Sessions show how people react to outputs. Wizard of Oz studies test the design before the model is ready. Logs show the real range of outputs at scale. The evaluation set checks that changes do not bring old failures back. Run them as a cycle: test, log, add failures to the set, fix, and test again. The single most useful habit is also the simplest: never record a participant's reaction without saving the exact output that caused it.