Incident Debugging and Observability
Synthesis
Difficult incidents often cross abstraction layers. Pinterest's account of failing Ray-based ML workloads began with intermittent network failures, then linked ENA driver resets to CPU starvation, then used temporal per-core profiling to identify unexpectedly large memory-cgroup state. The eventual culprit was a repeatedly crashing ECS agent in a GPU-node base image; disabling it and rebooting nodes restored stable job success. [src]
The case argues for observations that preserve time and scope: per-core profiles, resource counts, image provenance, availability-zone differences, and a clear timeline of mitigation. A plausible first symptom is not necessarily the causal boundary, and a healthy-looking control plane can conceal accumulating state beneath it. [src]
Debugging model
Start from the failed behavior, generate hypotheses across the full dependency path, and prefer measurements capable of falsifying each layer. Once a change appears to work, preserve the evidence that distinguishes correlation from an actual fix.