-
A bound that did not pay: when a safer careless-answer rule ranks no better
A test with a bounded false-alarm rate made 3 to 35 times fewer wrong calls on careless survey respondents, ranked no better, and so did not ship.
-
Every extra question is another look: raising the bar on long decision runs
Checking a stop after every answer lets lucky streaks end long runs as settled. A rising level cut those false results from 30 to 21 in 300.
-
Best and worst of four: a free MaxDiff mode, and why we didn't split it into pairs
A free MaxDiff mode for Quick Pick: best and worst of four, scored exactly. Fewer taps than pairs in 11 of 12 simulated cases; the exception says more.
-
Easy questions did not catch careless answers. Weighting the easy ones we already ask did.
Obvious check questions cost ranking accuracy and caught fewer careless respondents. Weighting existing answers by pair clarity caught more.
-
Flip, don't squint: comparing two photos on a phone
On a phone our photo ranker drew two landscape photos at 135 by 90 px. Stacking gives five times the area; a full-screen flip with a shared zoom shows more.
-
Is MaxDiff quicker than pairwise? Fewer taps, but the stopwatch decides
Best-worst screens of four took fewer taps than pairs in 11 of 12 simulations. Taps are not seconds, so the ranker now times real answers for a team.
-
Sort first, then compare: when a rough three-pile sort saves pairwise questions
A one-tap-per-option sort before pairwise picks cut 20 options from 93 actions to 64 for careful sorters; careless ones paid 120. Two bugs fixed on the way.
-
The question cap was picking the winner: a checkpoint instead of an end
Our free decision page ended at its question cap in 80% of simulated runs. Going on to 3x the cap named the true best 254 times in 300, not 203.
-
When your picks are close calls, the ranking shouldn't sound sure
The ranker told simulated near-coin-flip answerers their favourite was 56-62% likely when it was right 17-50% of the time. Now it refits first.
-
When every answer is "much better": reading the picks without the grades
People who answer "much better" to most questions made our decision page overconfident. A better threshold estimate did not fix it; ignoring their grades did.
-
Ranking pictures by picking pairs, without uploading a single one
Our pairwise ranker now takes photos and design drafts. The pictures never leave the device, yet a team can still combine picks: the file names are the list.
-
Skipping the pairs that cannot win: a 4x faster survey selector with the same picks
A cheap upper bound skips most of the 125,000 pairs the survey selector scores per answer. Every pick is unchanged; it runs 3 to 5 times faster.
-
A fast turn tells you where to look: catching weak position-clicking without new false flags
Quick pick now tests screen positions from where a respondent sped up: several times more weak position-clickers found, almost no new false flags.
-
A survey respondent who starts clicking the top button halfway through
Quick pick now keeps the honest start of a respondent who began clicking one screen position partway through, and drops only the run, at under 1% false flags.
-
A shortlist is not a short ranking: finding your top 3 by picking pairs
Ranking a whole list and keeping its top 3 often gets the cut wrong. Asking only about options near the cut got it right twice as often in simulation.
-
A show of hands picks the wrong shortlist more often than the model does. Combining a team's top 3
On 432 simulated teams, a show of hands on everyone's top 3 named the group's true shortlist 72% of the time; the Bradley-Terry model, 78%.
-
A wandering pace is not boredom: telling a careless block from a random walk
A rushing-respondent test flagged 17% of honest people whose pace wandered. A step-vs-random-walk test, on only when the panel wanders, cut that to 3.4%.
-
Close calls are not a settled order: fixing a ranking page that stopped too soon
The free list ranker said 90% settled when a near-coin-flip answerer's order was barely better than chance. The cause was the prior, not stopping.
-
How many pairwise comparisons does it take to rank a list? We measured it
Every pair once is n(n-1)/2 questions. An adaptive ranker needed 93 picks for 20 options and 204 for 40, for an order nearly as good. Short lists differ.
-
Rank a list by picking pairs, in your browser, and the stop rule that never stopped
A free page ranks any list from this-or-that picks, with rank ranges and team results. Its first stop rule never finished; here is the one that does.
-
The coin flips before the run: cutting a bored respondent from where they sped up
A slot-run test missed the random clicks a bored respondent gives first. Cutting from their speed-up instead gained up to +0.0045 Spearman, flagging nobody new.
-
The constant that was not constant: why our team ranking read disagreement at a third of its size
Our team ranking read disagreement at 30-60% of its true size. One dropped term made a Laplace fit into PQL. Kept, a true 2 reads 2.0-2.3.
-
Your team, or people like your team? Two rank ranges for one combined ranking
A combined team ranking needs two ranges: one for these people, one for the group they stand for. Read the first as the second and 90% ranges can hold 74%.
-
When timing cannot tell a bored respondent from a wandering one, read the picks
Quick pick's careless-block check now asks whether picks stopped following the panel: a third more bored respondents found, few extra false flags.
-
A flat hill, not a broken sampler: why a survey's prior fell to the floor
A pooled prior sd fell from 1.155 to 0.33. The sampler was fine: close pairs leave the evidence flat. An interval upper end fixes honest panels only.
-
Long lists: ask the informative question first
On 150-500 options, the informative pair from the first answer beats a 120-answer warm-up; the pooled ranking's clear calls were wrong up to 1 in 7.
-
One slow test file on four cores: halving a simulation suite without touching the maths
One file held 98 of a 127 s simulation suite. Its six seeded runs on worker threads cut the suite to 66 s, with byte-identical output.
-
When honest survey takers speed up on their own: a curve that does not drift with them
Reading later passes against the median of the people not yet flagged cut false flags on honest respondents by about a third when each person's pace drifts.
-
When 40% of a panel get bored at once: read against the upper quartile, not the median
Comparing each answer time with the panel's 75th percentile, not its median, lets a change-point check find a large group who all speed up at once.
-
A filter for careless survey answers only works if you can spot them, and we couldn't
A pooled prior would narrow honest respondents' intervals by 10-12% if careless people were left out of it. Their answers could not find them.
-
Modelling careless respondents doesn't rescue a pooled prior: spread and lapse sit on a ridge
We let each respondent have their own share of random answers. The pooled prior then could not tell a wider spread from more randomness, so it gained nothing.
-
A steady speed-up is not a block: tuning a bored-respondent detector to each survey
Reading each panel's own slow turns sets the threshold, and a step-versus-line test stops honest people who just get faster from being cut.
-
Telling a bored respondent from one who just speeds up: fit the line and the step together
Fitting each respondent's own speed-up and a step together cut honest people wrongly flagged as careless from about 5% to 2% in simulation.
-
Down-weighting the people who disagree makes a big survey confidently wrong
Without a harness cap, weighting respondents by agreement helps twice as much against careless clickers and hurts a real minority twice as hard.
-
Filtering survey respondents by consistency picks the people whose answers spread out most
Planted easy questions made careless respondents easier to spot. Gating on them barely helped: the most consistent people have the most spread-out items.
-
Our survey ranking looked overconfident on long lists. The test harness was the problem
A simulation said one in ten clear calls on 150-item surveys were backwards. The ranking was fine: a cap in the harness counted unanswered items as votes.
-
The survey taker who gets bored halfway: finding the moment from answer times
A change point in each person's answer times, read against the whole panel at the same point, finds mid-survey click-through at almost no cost.
-
Half-careless survey takers have normal-looking times. Their fast answers don't
A per-person median time misses respondents who click carelessly on half their picks. Cutting each fast answer instead recovers most of the harm in simulation.
-
When more threads stop helping: a simulation suite that hit its CPU floor
A second worker-thread change took a 102 s simulation suite to 94 s, not half. User time over cores said why: the suite is now CPU-bound.
-
A string for every pair: Quick Pick Survey's pair picker, four times faster with the same answers
Building a lookup key for each of 125,000 pairs took most of the selector's time. Skipping it: tests 630 s to 151 s, results byte-identical.
-
Careless or just different? Answer times can tell, if the careless are quick
In simulation, answer pace never mistook a genuine minority for careless clickers, and caught them as far as they were quicker. Shipped as a note.
-
Telling coin flippers from a camp in a pairwise survey, and why it's hard
On real rankings, 18-46% of honest people disagree with their panel. A check on each person's own consistency spares them; adaptive questions blunt it.
-
Letting each person's answers set the prior, and the textbook shortcut that broke it
A prior set from each person's answers lifted the favourite list from 88-92% to 93-96% coverage, once the Laplace evidence was replaced.
-
We gave a survey's intervals a heavier-tailed prior. They got narrower and wrong.
A Student-t prior should fix a Gaussian that over-shrinks extremes. In 108 simulated survey cells it covered less every time, even on heavy-tailed truth.
-
Switching a survey's default question order, after checking what it costs
The question order that can name your favourite is now the default up to 120 options. Careless-respondent sims under both orders found one cost.
-
Pooling a survey's answers helped the average respondent and hurt the hardest ones
A prior pooled over a survey missed the favourite for people whose likes sit close together. The narrower of theirs and the pooled one reached 95%.
-
The top-pick card on 72 real people: honest, and silent unless you change the question order
On real rankings the top-pick card was right 855 of 860 times and its contender list held the favourite 97-100%, but the standard order rarely names one.
-
"It's between these three" missed a third of the time: a credible set for your favourite
The old list of the three highest-rated items held the real favourite 68-74% of the time; a credible set from the refit holds it 88-92%.
-
The clear favourite that was a coin flip
Quick Pick called a clear favourite for three respondents in four, wrong in over half. A refit of each person's answers fixes the intervals and the claim.
-
Can a survey tell, from your own answers, how often you click at random?
A per-respondent estimate of how often someone answers a pick-one-of-two survey at random: their own answers barely move it, and adaptive pairs hide it.