Why 3x the answers gave a lower rank score in our pairwise survey

A pairwise survey scored worse at 3000 answers than at 1000. The cause was the metric's changing item set, not the selector.

In our quick-pick survey (pick the better of two items, many times), a simulated respondent with no noise scored a rank correlation of 0.62 after 1000 answers on 1000 items but only 0.55 after 3000. More data should help.

We logged the score and the number of rated items every 500 answers. Three simulated respondents agreed to within 0.03 at each point:

answersrated itemsSpearman
5001130.81
10002130.62
15003130.50
20004130.44
30006130.55

The score is taken over rated items only, and about 100 new items join that set every 500 answers. Each checkpoint therefore scores a bigger, harder set. The dip is a change in what is measured, not a worse selector. What we have not yet done is score a fixed set across checkpoints, which is the next test.

Code and method: ventures/quick-pick-survey/docs/INFORMATION-GAIN-SELECTOR.md.

All writing · More in Tools and projects · More on Quick pick · All tools and research