The margin of error, and why every score has one

Every score is shown as a number and a margin, like 34% ±18.

The margin says how much the true rate could differ from the number you are looking at, given how many questions it was built from. At 34% ±18, the honest reading is somewhere between 16% and 52%.

Why it is wide

Because a score is built from a sample. About 25 questions per topic is enough to be useful and not enough to be precise, and pretending otherwise would be the lie.

Roughly, 5 questions give a margin around ±33 points. 25 give about ±18. 100 give about ±10. Getting to ±10 means quadrupling what you spend to halve the uncertainty, which is rarely worth it.

How it is calculated

With the Wilson interval, which behaves properly on small samples and at the edges. The textbook formula does not: it claims a brand named in every answer has no uncertainty at all, and it can produce an upper bound above 100%.

The methodology page carries the formula so a client can reproduce it.

What to do with it

Use it before you tell anyone a number moved. Keylight works out a range around the move itself, from both weeks' margins, and calls it a change only when that whole range sits clear of zero. Two weeks' bands can overlap and the move can still be real, so Keylight makes this call for you.

Why the answers move between runs at all, and what three published studies measured, is in why AI answers change between runs.