Quick Pick / rank a list / how many comparisons
How many pairwise comparisons does it take to rank a list?
Asking every pair once grows with the square of the list: 190 questions for 20 options, 1225 for 50. An adaptive ranker that picks the most informative pair each time needed far fewer. We ran the free ranker's own code on simulated people to see how many.
Picks before the ranker stopped
Mean over 24 simulated people per cell (fewest to most in brackets). It stops when the order is 90% settled, or at its cap of about 2 n log2 n.
| Options | Every pair once | Perfect-sort minimum | Clear-cut | Ordinary | Close calls | Cap |
|---|---|---|---|---|---|---|
| 4 | 6 | 5 | 11 (11–12) | 13 (10–16) | 16 (13–16) | 16 |
| 5 | 10 | 7 | 15 (14–17) | 21 (15–24) | 24 (20–24) | 24 |
| 6 | 15 | 10 | 18 (17–21) | 25 (18–32) | 30 (20–32) | 32 |
| 8 | 28 | 16 | 26 (24–27) | 40 (29–48) | 44 (32–48) | 48 |
| 10 | 45 | 22 | 33 (31–34) | 48 (34–67) | 59 (37–67) | 67 |
| 12 | 66 | 29 | 40 (38–42) | 58 (37–87) | 78 (53–87) | 87 |
| 15 | 105 | 41 | 51 (50–53) | 73 (52–118) | 104 (61–118) | 118 |
| 20 | 190 | 62 | 72 (69–75) | 93 (76–152) | 141 (74–173) | 173 |
| 25 | 300 | 84 | 92 (88–95) | 118 (95–202) | 189 (96–233) | 233 |
| 30 | 435 | 108 | 112 (106–118) | 146 (117–222) | 212 (116–295) | 295 |
| 40 | 780 | 160 | 150 (145–155) | 204 (154–339) | 344 (158–426) | 426 |
| 50 | 1225 | 215 | 189 (181–195) | 263 (194–420) | 398 (193–565) | 565 |
Clear-cut: someone who never picks against their own order; Ordinary: someone who picks their favourite over their least favourite 998 times in 1,000, but finds neighbours on a 15-option list a 61/39 call; Close calls: someone who picks their favourite over their least favourite 95 times in 100, and finds neighbours on a 15-option list a 55/45 call. "Perfect-sort minimum" is log2(n!) rounded up, the fewest questions any method could use on someone who never wavers.
What the numbers say
- Past a handful of options, adaptive asks far fewer questions for nearly the same order. At 20 options, an ordinary person took 93 picks against 190 for every pair, and the order shown scored 0.94 Spearman against 0.95 for asking every pair once; at 50, 263 picks against 1225, 0.92 against 0.98. Every pair once gives the more accurate order when someone will sit through it; almost nobody answers 1225 questions.
- On short lists, every pair once is fewer questions. The ranker asked an ordinary person fewer questions than every-pair-once only from 12 options up. Below that it re-asks close pairs to be sure of them. Asking each pair once instead gave 0.90 at 4, 0.88 at 5, 0.88 at 6, 0.91 at 8, 0.92 at 10 Spearman for an ordinary person, against the ranker's 0.97, 0.98, 0.96, 0.96, 0.95 (table below): the ranker's extra questions buy a better order (0.07 Spearman on average), so they are worth answering if you have the minute.
- Close calls cost the most, and some never settle. When the options are close calls for someone, 78% of short-list runs (4 to 6 options) reached the cap without the order reaching 90%. That is the honest answer: their answers don't contain a firm order. The result then shows wide rank ranges rather than a false one.
- For a consistent person it lands near the sorting minimum, and on long lists below it. At 30 options the minimum is 108 and a never-wrong person was asked 112: the ranker allows for the odd slip, so it won't call an order settled on one answer per pair. From 40 options it stopped below the minimum (150 against 160; 189 against 215 at 50), because it stops when the order is 90% settled, not exact: those orders scored 0.99 Spearman, nearly but not always perfect.
How good was the order at the stop?
Rank correlation (Spearman, 1 = exactly right) between the order shown and the simulated person's real order, and how often the top option shown was really their favourite. "Every pair once" is the same person asked each pair exactly once and refit the same way (n(n-1)/2 questions). The last column is the share of close-call runs that stopped at the cap.
| Options | Clear-cut: Spearman | top right | Ordinary: Spearman | top right | Close calls: Spearman | top right | Every pair once: ordinary | Every pair once: close calls | Close calls at cap |
|---|---|---|---|---|---|---|---|---|---|
| 4 | 1.00 | 100% | 0.97 | 96% | 0.96 | 100% | 0.90 | 0.79 | 79% |
| 5 | 1.00 | 100% | 0.98 | 92% | 0.90 | 88% | 0.88 | 0.76 | 88% |
| 6 | 1.00 | 100% | 0.96 | 100% | 0.90 | 67% | 0.88 | 0.75 | 67% |
| 8 | 1.00 | 100% | 0.96 | 83% | 0.88 | 67% | 0.91 | 0.76 | 58% |
| 10 | 1.00 | 100% | 0.95 | 67% | 0.88 | 58% | 0.92 | 0.81 | 46% |
| 12 | 1.00 | 100% | 0.94 | 63% | 0.90 | 50% | 0.92 | 0.84 | 46% |
| 15 | 1.00 | 100% | 0.94 | 58% | 0.87 | 58% | 0.94 | 0.85 | 58% |
| 20 | 1.00 | 100% | 0.94 | 42% | 0.86 | 38% | 0.95 | 0.88 | 50% |
| 25 | 0.99 | 100% | 0.93 | 42% | 0.85 | 25% | 0.96 | 0.89 | 50% |
| 30 | 0.99 | 100% | 0.93 | 33% | 0.83 | 33% | 0.96 | 0.91 | 29% |
| 40 | 0.99 | 92% | 0.92 | 38% | 0.88 | 21% | 0.97 | 0.94 | 50% |
| 50 | 0.99 | 79% | 0.92 | 42% | 0.84 | 4% | 0.98 | 0.94 | 38% |
The top option is often wrong for close-call and long lists even when the whole order is good: the ranker is built to settle the order, not the winner. If you only need the winner, its "be sure of the favourite" mode spends questions on the front-runners instead.
Only need a shortlist? How many picks for a top 3 or 5
Often the job is "which 3 go on the shortlist", not the whole order. The ranker's "Find my top k" mode asks about the options that could still fall either side of the cut and stops when the shortlist is 90% settled. Below, the same simulated people (16 per cell) ranked the full order with its first k read off, and then ran the shortlist mode. "Right" counts the runs whose shown top k was exactly the person's true top k, as a set.
| Options | Person | Full order: picks | right | Shortlist mode: picks | right | at cap |
|---|---|---|---|---|---|---|
| 10, top 3 | Clear-cut | 33 | 16/16 | 41 | 16/16 | 0/16 |
| Ordinary | 51 | 14/16 | 52 | 13/16 | 7/16 | |
| Close calls | 58 | 4/16 | 61 | 10/16 | 12/16 | |
| 15, top 3 | Clear-cut | 51 | 16/16 | 63 | 16/16 | 0/16 |
| Ordinary | 73 | 4/16 | 100 | 9/16 | 8/16 | |
| Close calls | 101 | 0/16 | 118 | 6/16 | 15/16 | |
| 20, top 3 | Clear-cut | 71 | 16/16 | 82 | 16/16 | 0/16 |
| Ordinary | 86 | 4/16 | 161 | 12/16 | 11/16 | |
| Close calls | 140 | 5/16 | 173 | 5/16 | 16/16 | |
| 30, top 3 | Clear-cut | 111 | 16/16 | 116 | 16/16 | 0/16 |
| Ordinary | 169 | 6/16 | 268 | 7/16 | 10/16 | |
| Close calls | 212 | 0/16 | 295 | 1/16 | 16/16 | |
| 10, top 5 | Clear-cut | 33 | 16/16 | 37 | 16/16 | 0/16 |
| Ordinary | 51 | 12/16 | 47 | 14/16 | 2/16 | |
| Close calls | 58 | 4/16 | 56 | 2/16 | 9/16 | |
| 15, top 5 | Clear-cut | 51 | 16/16 | 68 | 16/16 | 0/16 |
| Ordinary | 73 | 7/16 | 102 | 11/16 | 7/16 | |
| Close calls | 101 | 4/16 | 115 | 6/16 | 13/16 | |
| 20, top 5 | Clear-cut | 71 | 16/16 | 93 | 16/16 | 0/16 |
| Ordinary | 86 | 0/16 | 146 | 7/16 | 6/16 | |
| Close calls | 140 | 3/16 | 169 | 4/16 | 15/16 | |
| 30, top 5 | Clear-cut | 111 | 11/16 | 136 | 16/16 | 0/16 |
| Ordinary | 169 | 4/16 | 262 | 8/16 | 9/16 | |
| Close calls | 212 | 0/16 | 295 | 1/16 | 16/16 |
- On 15 to 30 options, an ordinary person's shortlist came out right 56% of the time against 26% (54 against 25 of 96 runs), for 1.6 times the picks. E.g. 20 options, top 3: 12/16 right after 161 picks against 4/16 after 86.
- Someone sure of every answer pays 20% more picks (558 against 466 summed over the 15-30 option rows) for a shortlist the full order mostly had right anyway (91 of 96).
- On short lists it buys little: at 10 options an ordinary person's shortlist was right 27 of 32 times against 26, because a short list is asked nearly every pair either way.
- Close calls on long lists mostly hit the cap (91 of 96 runs) and the shortlist is still mostly wrong (23 of 96 right, against 12 for the full order): the answers don't hold a firm cut, and the result says so with each option's chance of being in.
Script: scripts/top-k-sweep.mjs; rows: docs/top-k-2026-11-14.jsonl (arms order and boundary); people as above. A shortlist link carries its k, so everyone you send it to answers for the same cut.
Questions people ask
- How many pairwise comparisons do you need to rank n items?
- Asking every pair once takes n(n-1)/2: 190 for 20 items, 1225 for 50. An adaptive picker that asks only the informative pairs needed about 93 picks for 20 items and 263 for 50 in our simulations of a person with ordinary consistency (an order nearly as good as every pair once gave: 0.94 against 0.95 Spearman at 20), and 72 and 189 for a person who never contradicts themselves.
- Is asking every pair ever the better choice?
- On short lists it is fewer questions. Below 12 items, a person with ordinary consistency was asked more questions by the adaptive ranker than there are pairs, because it re-asks close pairs to be sure of them. Compared on the same simulated people, the ranker's extra questions buy a better order (0.07 Spearman on average), so they are worth answering if you have the minute. Above that, every pair once quickly becomes too many questions to ask anyone.
- How many comparisons does it take to find the top 3 of 20?
- If you only need a shortlist, rank for the cut, not the whole order. For 20 options and a top 3, a simulated person with ordinary consistency got the exact top 3 right 12 times in 16 after about 161 picks with a picker that asks about the options near the cut, against 4 in 16 after 86 picks when the full order was ranked and its first 3 read off. Someone who never contradicts themselves needed about 82, and every pair once would be 190.
- What is the fewest comparisons any method could use?
- A perfect sort needs at least log2(n!) comparisons, about n log2 n: 62 for 20 items. That bound is for getting the order exactly right from someone who never answers inconsistently. At 30 options the minimum is 108 and a never-wrong person was asked 112: the ranker allows for the odd slip, so it won't call an order settled on one answer per pair. From 40 options it stopped below the minimum (150 against 160; 189 against 215 at 50), because it stops when the order is 90% settled, not exact: those orders scored 0.99 Spearman, nearly but not always perfect.
How this was measured
Each simulated person has true scores evenly spaced over a range (100 logits for clear-cut, 6 for ordinary, 3 for close calls), shuffled, and picks A over B with probability 1/(1+e−(a−b)), never "about equal". The questions come from the ranker's own picker (the pair with the most expected information, sides by coin flip) and it stops by the ranker's own rule: a Bradley-Terry refit after every answer, the order's posterior confidence at 0.9, rechecked at a prior fitted to the person's own answers. Real people may differ: they tire, warm up and have position habits, which these simulations leave out. Script: scripts/picks-needed.mjs; rows: docs/picks-needed-2026-11-14.jsonl; the method and its other measurements.
Ready lists to start from: a feature backlog, your personal values, a name shortlist, a group trip, team goals, customer pain points, a to-do list, or all ready lists.