Investigating an Intermittent API Failure in Production

The API failures were intermittent. That made them harder to understand and easier to dismiss as unrelated noise. They appeared in a system I supported at Softcraft, where the same workflow had behaved correctly in development and testing. There was no single line of code that explained everything. The problem appeared only when production data volume, concurrent requests, and slower dependency responses happened together. Each condition was manageable alone. The test environment didn’t reproduce their combination very well.

What Happened in Production

We intended requests to complete within a certain period. We assumed a database query would remain fast enough. We expected a connection pool to absorb normal variation and an external response to arrive before resources became constrained. Under heavier use, one slow step held resources longer. New requests continued arriving. Waiting increased in another part of the system, and retries added more work at exactly the wrong time. No component was completely broken. One slow step held resources for longer, new requests continued arriving and the system got worse over time. None of the assumptions looked dangerous alone, but production found the combination.

Following the First Symptom

Users saw errors at the API boundary, so the API was the first place we investigated. The useful evidence came from following the complete request. We compared timing across application stages, looked at database behavior, and traced how the external call affected resource usage. The API error was where the system finally gave up, but the problem had started earlier. We addressed the immediate bottleneck, adjusted how the workflow handled slow dependencies, and improved the signals around resource saturation. The technical changes mattered, but so did the new understanding of how the system degraded.

After the Incident

After restoring service, it’s easy to stop at the fix. For me, the useful part of the review was asking what we had been unable to see. Which evidence was missing? Which alert arrived too late? Which recovery step depended on one person remembering a command? We improved monitoring around the workflow and documented the relationship between latency, resource usage, and retries. We also reviewed the test scenarios so they represented more realistic data and concurrent behavior. We couldn’t guarantee that no incident would ever happen again. We could make the same class of problem easier to detect, understand, and contain. Incidents can create pressure to find the person or change responsible. Individual decisions matter, but production behavior usually passes through code review, testing, deployment, infrastructure, and team conventions. If the only conclusion is that someone should have been more careful, the system has learned very little. More useful questions examine the conditions around the decision. Why was the assumption difficult to see? What signal could have shown the risk earlier? Did the process encourage a local fix without an end-to-end view?

The review wasn’t about pretending nobody made a mistake. We wanted more than “someone should have been more careful,” because that wouldn’t help much during the next incident. We left with better signals around the workflow, a clearer recovery path and tests that looked a little more like production. I’d rather have found the problem earlier, but at least the details didn’t stay only in the heads of the people who were there that day.