Handling Dependency Failures in Distributed Systems

While working on a high-availability security platform, I dealt with workflows that crossed APIs, processing workers, event pipelines, databases, caches, and AWS services. Each component could be healthy according to its own dashboard while the complete business operation was delayed or incomplete. This means every dashboard can look normal while the user’s operation isn’t moving.

Different Dependency Failures

A dependency doesn’t simply alternate between available and unavailable. It can respond slowly, return incomplete data, accept work but lose the response, throttle traffic, or behave normally for most requests while failing a particular case. Each failure mode creates a different question for the caller. Should the operation retry? Can it continue with partial information? Is the result safe to cache? Does processing need to stop, or can another stage proceed? How long can the system wait before the delay becomes a business problem? A generic error handler can’t make these decisions because they depend on what the operation means for the business.

Timeouts

A timeout looks like a configuration value until someone has to decide whether thirty seconds is patient or irresponsible. It defines how long the workflow is willing to wait for a dependency. If the timeout is too short, the system creates unnecessary failures during normal variation. If it’s too long, threads, connections, or workers remain occupied while users wait for an answer that may never arrive. The correct value depends on the operation. A user-facing request and a background enrichment task don’t necessarily have the same tolerance. We needed to connect technical limits to the behavior the platform promised. The timeout wasn’t only a number in the configuration. It also affected the delay and recovery of the operation. Retries help distributed systems survive temporary problems. They can also synchronize a large amount of new traffic against a dependency that’s already struggling. At scale, the recovery behavior needs as much attention as the normal behavior. We used controlled retries, delays, and processing states so work could resume without immediately repeating everything. Operations needed identifiers so a second attempt didn’t become duplicate business activity.

We also separated cases the system could recover automatically from cases that required investigation. Automation is valuable when it makes a safe decision. Repeating an unknown operation indefinitely only makes the uncertainty harder to see.

Following the Complete Operation

When a workflow crosses several services, looking at each component separately is slow and often misleading. We needed to follow one event through the processing path and understand which stages completed, which dependency responded unexpectedly, and whether another attempt was in progress. That required consistent identifiers and logs that recorded decisions, not only exceptions. “Request timed out” describes a technical event. “Enrichment delayed, retry scheduled, base event preserved” describes what the system decided to do about it. The second message gives the person investigating a place to start. The first only says that something went wrong. A service returning success doesn’t always mean the business workflow is complete. It may mean the work was accepted, one stage finished, or a message was published for another component. Teams need a shared understanding of those boundaries. Otherwise, every component reports success while users wait for an operation nobody owns end to end. I still need to know if each service is healthy, but that doesn’t tell me if the complete operation is healthy.

During incidents, the service dashboards were still useful, but I usually needed a different level of detail. I wanted to know what had already completed, whether we could repeat the operation and which component was expected to move it forward. A line between two services in the diagram couldn’t answer that, and a green health check often couldn’t either.