A proxy buffered the first 4 KB and nobody noticed for six weeks
Streaming looked fine in aggregate because the total time was unchanged. Only the first token moved, and we were not graphing it per region.
On 14 May we moved sin1, hkg1 and bom1 behind a new edge proxy. The proxy’s default was to buffer response bodies until 4 KB had accumulated or the connection closed. For a non-streamed request this changes nothing measurable. For a streamed one it means the client sees nothing at all until the model has produced roughly a thousand tokens.
Why the graphs looked fine
Total request duration was unchanged, and that is the metric that had an alert on it. Time to first token was being recorded, but only globally, and three regions out of thirty-one moving is inside the noise of a global p50.
We found it because a customer in Singapore said their chat UI “felt slow now” and could not be more specific than that. They were right and we had no graph that agreed with them.
What changed
Time to first token is now recorded and alerted per region, not globally. The x-argosy-ttft header exists so that you can hold us to the same number from outside. And the proxy config is in the same repository as everything else, reviewed by someone who has read what the flag does.