Does the bias fallback in a ranking survey earn its keep? A control run

We switched off a ranking survey's bias fallback to see what it saves: a lot at 80% top-slot bias, nothing measurable at 50%.

Quick Pick Survey asks people to choose between two items and builds a ranking from the answers. Its smarter question picker (an information score) assumes answers carry signal. Someone who taps the top option most of the time breaks that assumption, so the route falls back to a simpler picker once a person's first-option share reaches 0.7 (or falls to 0.3).

Until now nobody had measured what the fallback saves. This run adds the missing control: the same synthetic respondents through the real routes and a real Postgres, with the fallback switched off.

respondentSpearman with fallbackwithout
50% top-slot bias0.4920.482
80% top-slot bias0.2660.096

At 80% bias the fallback keeps 0.17 of rank correlation the information picker alone loses. At 50% bias the difference is inside the noise.

Limits, plainly: 3 simulated respondents per cell, 200 answers, 60 items, one seed set, synthetic answers. Another seed set moved the clean case by about 0.06. So the 80% result is a clear signal and the 50% result is "no measurable effect", not "no effect". Method, the mock used for the control and a reproduce command are in the venture's docs/INFORMATION-GAIN-SELECTOR.md. Next: the same at 1,000+ items with more respondents.

All writing · More in Tools and projects · More on Quick pick · All tools and research