Operations
The outage that was actually a clock
M. Rautenberg · 8 days ago
The first hour went into storage, because that is what the dashboards said. Write latency was up, the queue depth graph had the shape everyone recognises, and the storage team had a maintenance window open on an unrelated cluster. Three people spent an hour disproving a coincidence.
What actually happened was that two application hosts had drifted 400 milliseconds apart after an NTP source was quietly removed from the pool in a configuration change eleven days earlier. The lease logic treated a lease issued by the faster host as already expired, renewed it, and the slower host did the same in reverse. Neither host logged anything unusual, because from each one's point of view nothing unusual had happened.
The change we made afterwards was not better time synchronisation, although we did that too. It was to make the lease code log both clock readings whenever it rejects a lease. The next occurrence will be diagnosed in a minute rather than six hours, and that is worth more than the prevention we cannot guarantee.
Observability
Reading a flame graph you did not generate
S. Vogelsang · 31 days ago
Profiles arrive in tickets without context more often than anyone would like. The sampling interval is unstated, the workload is described as 'normal load', and the person who captured it has gone home. There is still usually something to be had.
Start at the bottom and read the width, not the height. Depth tells you about call structure, which is rarely the problem; width tells you where time went, which usually is. A frame occupying a third of the width is worth understanding even if you have no idea what it does, and the name is often enough to find out.
The common trap is treating a wide frame near the bottom as the culprit. It is an ancestor. Follow it upward until the width splits, and the split is where the interesting decision happens. If it never splits, you are looking at a single hot path and the answer is in that function.
Also recent
Storage
A storage migration measured in deleted code
K. Pflueger · 44 days ago
Moving 40 terabytes between object stores was the easy part. The retry logic that had accumulated around the old one was not.
Build systems
Build caches lie by omission
A. Treichel · 58 days ago
A cache key that covered the inputs but not the toolchain produced a build that was reproducible and wrong.
Networking
Rate limits are a conversation, not a wall
N. Heidrich · 73 days ago
Returning 429 without telling the client when to come back converts one problem into two.
Full archive