The margin of error, and why every score has one

Every score is shown as a number and a margin, like 34% ±18.

The margin says how much the true rate could differ from the number you are looking at, given how many questions it was built from. At 34% ±18, the honest reading is somewhere between 16% and 52%.

Why it is wide

Because a score is built from a sample. About 25 questions per topic is enough to be useful and not enough to be precise, and pretending otherwise would be the lie.

Roughly, 5 questions give a margin around ±33 points. 25 give about ±18. 100 give about ±10. Getting to ±10 means quadrupling what you spend to halve the uncertainty, which is rarely worth it.

How it is calculated

With the Wilson interval, which behaves properly on small samples and at the edges. The textbook formula does not: it claims a brand named in every answer has no uncertainty at all, and it can produce an upper bound above 100%.

The methodology page carries the formula so a client can reproduce it.

What to do with it

Use it before you tell anyone a number moved. If this week’s band and last week’s band overlap, you do not have a change, you have noise. Keylight does this comparison for you and says which it is.