Does a model trained on other cities forecast Seattle permit delays? A second test year says only partly

A permit-delay model trained on Chicago, SF and Austin, tested on Seattle in two years. The 2024 gain at every quantile did not repeat in 2023.

The build-data programme asks whether open permit feeds predict how long a permit takes. One question is transfer: can a model trained on other cities beat a new city's own per-permit-type baseline?

Session 64 found that a model trained on San Francisco and Austin beat Seattle's own baseline at all three quantiles in 2024 (issue to completion). This run repeated it on 2023, with the prediction written first.

Seattle, 2023q0.1q0.5q0.980% interval coverage
application to issue, trained on 3 cities-0.1% (tie)+0.5% (tie)+27.1%0.613
issue to completion, trained on 2 cities-6.7%-7.3%+11.2%0.681

(Pinball loss, change against Seattle's own baseline; negative is better.)

What held: the lower and middle quantiles still beat the baseline, and the application-to-issue (a2i) result matches 2024. What did not: the upper quantile (q0.9) was better in 2024 (-5.1%) and worse in 2023. The transferred 80% interval covers only 61 to 68% of outcomes in both years. A model trained on Seattle's own data, with the same features, beat the baseline everywhere.

Limits: two test years, 3 seeds, two source cities for the issue-to-completion target, and no explanation yet for the q0.9 flip. Data are open city feeds; code, results and the log are in research/build-data (session 65). AI-assisted research.

All writing · More in Research findings · More on build-data · All tools and research