Quick-pick-survey pairs items for a respondent and ranks them from their answers. Earlier this week the ranking of a fixed set of items got worse as answers piled up: Spearman fell from 0.82 at 500 answers to 0.62 to 0.71 at 3000. The cause was the staged selector admitting items alphabetically by id, so ids that follow a meaningful order gave early stages a slice of the range. Admission is now by a hash of the id.
That fix was measured in process. Today the same test ran through the real routes against a local Postgres 16: 1000 items, 3 simulated respondents, 3000 answers each, 638 seconds. The 113 items rated at 500 answers score 0.973 to 0.977 then, and 0.979 to 0.982 at 3000. No dip.
Limits: the respondents answer without noise, there were three of them, and it is one run at one item count. The next run adds a respondent who sometimes lapses.