Tree test metrics: success, directness and the pietree

Every metric uses the task's own number of attempts, so a participant who skipped task 3 still counts in task 3's skip rate. With fewer than five attempts, the results page warns that the numbers are too few to lean on.

The procedure

1. Record each attempt as the nodes the participant opened or chose, in order. Closing a branch isn't a step.
2. Mark a step as a backtrack when its node isn't a child of the previous step's node. The first step is a backtrack unless it's a top-level node.
3. Give the attempt one of six outcomes: success, fail or skip, each direct (no backtracks) or indirect.
4. Success is direct plus indirect successes over every attempt at the task, with a 95% adjusted Wald interval.
5. Directness is every direct outcome, right, wrong or skipped, over every attempt, with the same interval.
6. First click right is the share of attempts whose first step was the top-level node on a correct path.
7. Time is reported as the median and as a geometric mean with its interval, leaving out times more than three log SDs from the rest.

The six outcomes follow Optimal Workshop's published definitions [1], so the numbers line up with a Treejack study. The intervals are adjusted Wald intervals [2], which hold up at the sample sizes tree tests use, and the geometric mean for time follows Sauro and Lewis [3].

A worked example

The sample Larkspur Bank tree test on the surveys and IA page asked 50 people: “You want to remove a card you saved with an online shop. Where would you go?” The correct answer is Cards, then Card settings.

• Success: (20 plus 11) out of 50 = 62%, 95% interval 48% to 74%.
• Directness: (20 plus 4 plus 0) out of 50 = 48%.
• First click right: 40% went to Cards first.

A success rate in the low sixties with directness under half tells you the answer is findable but the first place people look isn't Cards. The pietree shows where they look instead.

Reading the pietree

Every node visited on the task is drawn as a pie, sized by how many participants passed through it. Each time a node appears in someone's path, the step after it decides the slice:

• Went the right way: the next step was a child on a correct path.
• Went the wrong way: the next step was any other child.
• Went back: the next step was anywhere that isn't a child.
• Chose as answer: it was the last step, and the participant confirmed it.
• Ended here: it was the last step of a skipped attempt.

At the root, 20 first steps went into Cards and 30 went elsewhere. Inside Cards, 31 went on to Card settings and 9 opened another card page. The label inside the branch works; getting people into the branch is the problem.

Task score

Some teams want one number per task. Stroop can show (3 times success plus directness) divided by 4, on a scale of 0 to 10. It's off by default, because it hides the difference between a findable label and an obvious one.

Used on Tree testing. Stroop is the all-in-one research and testing platform. Run usability tests, prototypes, surveys, tree tests, diaries and IATs in one study, and get results back transcribed, scored and charted, with every finding linked to the session it came from.