Handling Dependency Failures in Distributed Systems

While working on a high-availability security platform, I dealt with workflows that crossed APIs, processing workers, event pipelines, databases, caches and AWS services. Each component could be healthy according to its own dashboard while the complete business operation was delayed or incomplete. We had cases where every individual health check was green and the user’s operation still wasn’t moving.

Different Dependency Failures

A dependency doesn’t simply alternate between available and unavailable. It can respond slowly, return incomplete data, accept work but lose the response, throttle traffic, or behave normally for most requests while failing a particular case. The caller still has to decide whether the operation can retry, continue with partial information or wait without becoming a business problem. A generic error handler can’t make these decisions because they depend on what the operation means for the business.

One enrichment service, for example, could be available but take much longer than normal for a subset of requests. From its own perspective it was still answering. From the pipeline’s perspective, workers stayed occupied, events accumulated behind that stage and the delay continued growing. We needed to monitor the age and progress of the work alongside the dependency errors.

Incomplete data created a different choice. Some events could continue with a missing optional enrichment and be updated later. Others would produce a misleading result if processed without it. We made that decision in the workflow instead of allowing every caller to treat a non-error response as complete information.

Timeouts

A timeout looks like a configuration value until someone has to decide whether thirty seconds is patient or irresponsible. It defines how long the workflow is willing to wait for a dependency. If the timeout is too short, the system creates unnecessary failures during normal variation. If it’s too long, threads, connections, or workers remain occupied while users wait for an answer that may never arrive. The correct value depends on the operation. A user-facing request and a background enrichment task don’t necessarily have the same tolerance. We needed to connect the timeout to the delay and recovery behavior the platform promised. Retries help distributed systems survive temporary problems, and they can also synchronize a large amount of new traffic against a dependency that’s already struggling. At scale, the recovery behavior needs as much attention as the normal behavior. We used controlled retries, delays, and processing states so work could resume without immediately repeating everything. Operations needed identifiers so a second attempt didn’t become duplicate business activity.

We also limited how much work could wait on the same dependency at once. Increasing the number of workers looked like a quick way to reduce a backlog, but it could send even more concurrent requests to the slowest component and make the recovery take longer. In some incidents, allowing the queue to absorb the delay was safer than trying to process everything immediately.

The retry delay included some variation so thousands of operations didn’t all return at the same moment. We kept the original operation identity and the completed stages between attempts. A retry that started the complete workflow again could duplicate work in earlier services even if the failing dependency had never received the first request.

We also separated cases the system could recover automatically from cases that required investigation. Automation is valuable when it makes a safe decision. Repeating an unknown operation indefinitely only makes the uncertainty harder to see.

Following the Complete Operation

When a workflow crosses several services, looking at each component separately is slow and often misleading. We needed to follow one event through the processing path and understand which stages completed, which dependency responded unexpectedly, and whether another attempt was in progress. That required consistent identifiers and logs that recorded decisions as well as exceptions. “Request timed out” described the technical event. “Enrichment delayed, retry scheduled, base event preserved” told the person investigating what the system had done about it. A service returning success could mean that the work was accepted, one stage finished, or a message was published for another component. Without a shared understanding of those boundaries, every component could report success while users waited for an operation nobody owned end to end.

We added metrics around the complete path, including how many operations entered and finished important stages and how old the oldest pending item was. These didn’t replace the service dashboards. They connected them to the work the platform was expected to complete. When the numbers diverged, we could start with the stage where progress had slowed instead of checking every component in an arbitrary order.

During incidents, the service dashboards were still useful, but I usually needed a different level of detail. I wanted to know what had already completed, whether we could repeat the operation and which component was expected to move it forward. A line between two services in the diagram couldn’t answer that, and a green health check often couldn’t either.

After the immediate problem was resolved, we also checked what happened to the operations that had been delayed. Recovering the dependency didn’t automatically mean the user-facing workflow was caught up. Sometimes there was still a queue to process or a smaller group of operations that had exhausted their attempts. The incident ended for us only after that work had a clear result, not when the health check returned to green.