Why we stopped quoting cold start as an average
An average cold start is a number that describes nobody. Ours was 61 ms and the distribution was bimodal.
For most of last year the front page said cold starts around 60 ms. That was an honest average and a useless one. The distribution had two humps: about 18 ms when the weights were already in the page cache, and about 140 ms when they had to come off object storage. Almost no request took 60 ms. The average described a request that did not exist.
What we changed
Weights now live on local NVMe in every region and are memory-mapped rather than read. The second hump did not shrink so much as disappear — there is no longer a path that pulls from object storage on the request. What is left is one hump at 38 ms for the large models and 18 ms for the small ones, which is why the site now quotes a number per model class instead of one number for everything.
What it cost
About 1.8 PB of NVMe across thirty-one regions, and a cache eviction policy that has been rewritten twice because the first version evicted whichever model had been idle longest, which is exactly the model whose next request will be someone’s cold start.
The current policy weights idleness against how expensive the model is to reload. It is not clever and we would rather it stayed that way.