Investigating an Intermittent API Failure in Production

The API failures were intermittent. That made them harder to understand and easier to dismiss as unrelated noise. They appeared in a system I supported at Softcraft, where the same workflow had behaved correctly in development and testing. There was no single line of code that explained everything. The problem appeared only when production data volume, concurrent requests, and slower dependency responses happened together. Each condition was manageable alone. The test environment didn’t reproduce their combination very well.

What Happened in Production

We intended requests to complete within a certain period. We assumed a database query would remain fast enough. We expected a connection pool to absorb normal variation and an external response to arrive before resources became constrained. Under heavier use, one slow step held resources longer. New requests continued arriving. Waiting increased in another part of the system, and retries added more work at exactly the wrong time. No component was completely broken. One slow step held resources for longer, new requests continued arriving and the system got worse over time. None of the assumptions looked dangerous alone, but production found the combination.

The first failures recovered without intervention, which made them look isolated. As the slower periods became more frequent, requests overlapped for longer and the same pool had less room for new work. A retry often arrived while the original dependency was still slow, so the recovery mechanism increased the pressure instead of reducing it. By the time the API returned an error, several earlier decisions had already contributed to it.

Following the First Symptom

Users saw errors at the API boundary, so the API was the first place we investigated. The useful evidence came from following the complete request. We compared timing across application stages, looked at database behavior, and traced how the external call affected resource usage. The API error was where the system finally gave up, but the problem had started earlier. We addressed the immediate bottleneck, adjusted how the workflow handled slow dependencies, and improved the signals around resource saturation. The technical changes mattered, but so did the new understanding of how the system degraded.

Some logs had timestamps and errors but no common operation identifier, so the first investigation involved matching requests by time and related data. We added the identifier across the main stages after the incident. We also exposed the time spent waiting for a connection alongside the database query time. That wait had been almost invisible in our normal application metrics.

For the immediate recovery, we reduced the pressure on the slow path and allowed the existing work to finish before returning traffic to the normal level. Restarting every instance would have cleared some local state and created another wave of requests against the same dependency. We waited for the backlog and resource usage to stabilize before considering the system recovered.

After the Incident

After restoring service, we reviewed the evidence we had been missing, the alert that arrived too late and the recovery command only one person remembered. We improved monitoring around the workflow and documented the relationship between latency, resource usage, and retries. We also changed the test scenarios to use more realistic data and concurrent behavior. We couldn’t guarantee that the same class of incident would never happen again, but we could detect and contain it earlier. We discussed the individual decisions involved without stopping at “someone should have been more careful,” because the change had also passed through review, testing, deployment and the team’s existing conventions.

We recreated the combination with a larger dataset, controlled concurrency and a slower dependency response. The goal wasn’t to copy production exactly, which would have been difficult, but to make the interaction visible enough that the team could test the change. That scenario later became part of the performance checks for the workflow instead of remaining a one-time incident script.

The review wasn’t about pretending nobody made a mistake. We wanted more than “someone should have been more careful,” because that wouldn’t help much during the next incident. We left with better signals around the workflow, a clearer recovery path and tests that looked a little more like production. I’d rather have found the problem earlier, but at least the details didn’t stay only in the heads of the people who were there that day.