-
Careless answers that point the same way: which survey clickers actually move a ranking
Four kinds of careless survey respondent, simulated: coin-flippers and favourites barely matter; people who pick by one shared wrong reason skew the ranking.
-
One easy question in five lets our survey spot careless answering
Adaptive pairwise surveys hide random answering. An easy lopsided pair every fifth question made careless respondents detectable in simulation.
-
We built a careless-answer fallback for our survey and left it off
A fallback to a safer question picker when answers look random tripped for a third of honest respondents and moved rank accuracy by -0.002.
-
We caught the careless respondents, then found the ranking didn't need us to
Down-weighting or dropping respondents who answer at random did not improve a pooled pairwise ranking in 40 simulated settings. Dropping them made it worse.
-
Teaching our survey's rating update that people answer at random sometimes
A Glicko-2 update that expects some coin-flip answers kept its intervals honest only when it knew the true rate; a fixed guess cost up to twice the answers.
-
We had overstated the cost of lazy tapping fourfold: re-measuring position bias in Quick Pick Survey
A simulator bug put every biased tap on one item. Behind the app's slot shuffle the damage is a quarter of what we reported; the fallback stays.
-
A pairwise survey replayed on real people: who you ask matters more than how much
We replayed a pairwise survey on two published sets of real rankings: extra answers stop helping early, extra people keep helping, and the ranges stay honest.
-
Short surveys now get the better question picker from the first answer
Quick Pick Survey's sharper question order now starts at the first answer on lists up to 120 options; the couple screen opens 4 to 8 answers sooner.
-
A careless answerer does not break a pairwise ranking at 3000 answers
A simulated person who answers at random one time in five still gets a rank correlation of 0.94 across 3000 pairwise answers on 1000 items.
-
Can a pairwise survey tell when someone starts answering at random?
A rating-based test for random answering in an adaptive pairwise survey catches 22% of random readings and flags 7% of honest ones, so it ships unwired.
-
Can you catch a tired survey respondent by repeating a question?
Repeating an adaptive survey's own pairs needs 3,549 repeats to spot a respondent who answers 20% at random; a lopsided catch pair needs about 100.
-
What a tiring respondent does to a pairwise ranking
A simulated respondent answers well for 2000 comparisons, then half turn random. Ranked items hold at 0.98; items admitted later fall to about 0.89.
-
The 3000-answer ranking dip is gone through the real routes too
A live-Postgres run of the survey's noisy-respondent test confirms the id-hash admission fix: the fixed set stays at 0.98 Spearman through 3000 answers.
-
Hunting a ranking dip in a pairwise survey engine: three suspects cleared
Quick Pick's ranking quality drops after 3000 answers through the live route but not in a plain loop. Three suspects were tested and cleared.
-
A pairwise-ranking dip traced to how items were admitted by id
Quick Pick Survey ranked worse after 1,000 answers. Admitting items by id was the cause; a hash order lifts fixed-set Spearman from 0.62 to 0.98 in simulation.
-
A survey result that admits its ties
Quick Pick's branded survey report shows tied options as a shared range like 2–5 and warns about thinly rated ones, with a sample page.
-
The pair-picker dip does not reproduce outside the route
A staged pair selector scored worse on items it had already settled in end-to-end runs. Re-run in process, it does not, which narrows where the bug can be.
-
Why did our pair picker get worse at ranking items it had already settled?
A noiseless simulated respondent ranked 1000 items. With the staged selector the first 113 items got worse with more answers; with the shipped one they did not.
-
A branded results report for pairwise ranking surveys
Quick pick now builds a one-file results report that shows ties as ranges and flags thin options, ready for a client.
-
More answers made our survey ranking worse, and the bias fix held
We tripled the answers per respondent on a 1000-item pairwise survey. The first-option-bias fallback still helped, but clean rank quality fell.
-
We corrected our own explanation of a ranking dip
Quick Pick Survey's pair selector ranked worse after 3000 answers than after 1000. We blamed coverage; a fixed-set test shows that was only half the story.
-
Why 3x the answers gave a lower rank score in our pairwise survey
A pairwise survey scored worse at 3000 answers than at 1000. The cause was the metric's changing item set, not the selector.
-
Position bias in a 1,000-item ranking survey: the fallback helps more
Re-running a ranking survey's bias fallback control at 1,000 items: it recovers rank correlation at 50% and 80% top-slot bias.
-
Does the bias fallback in a ranking survey earn its keep? A control run
We switched off a ranking survey's bias fallback to see what it saves: a lot at 80% top-slot bias, nothing measurable at 50%.
-
What a person who always clicks the top option does to a ranking
Simulated respondents who lapse or always tap the top option, run through the real routes, show how fast a pairwise ranking degrades.
-
A 1000-item pairwise survey stays under 60 ms a question
One respondent answering 1000 pairs on a 1000-item survey, through the real routes and Postgres: the next question never took longer than 60 ms.
-
What happens to a ranking survey when people favour the first option
A simulation of quick-pick-survey's pair selectors against respondents who pick the first option shown on 10 to 50% of answers.
-
Quick-pick surveys now stop adaptive pairing for respondents who always tap the first option
quick-pick-survey's staged selector falls back to the simple one for a respondent who picks the first option 70% of the time. Worst case: 0.592 to 0.651.
-
Does adaptive pairing survive a respondent who contradicts themselves?
A simulation of quick-pick-survey's pair selectors with 20% random answers: the information-only selector loses to the shipped one, the hybrid does not.
-
How often can a respondent answer at random before the pair selector breaks?
Quick Pick Survey pair selectors swept from 5% to 50% random answers on a synthetic respondent: the shipped hybrid never trails the old one by more than 0.005.
-
Quick Pick Survey now guards against respondents who always tap the second option
The staged pair selector already fell back for first-option bias; it now covers the second option too, lifting a simulated worst case from 0.574 to 0.666.
-
Quick Pick Survey keeps getting better when people answer far more than the list length
Simulating 30 items answered up to 1,200 times: the information-gain question selector does not stall, and leads the shipped one from 20 answers per item.