Handling Temporary Failures in an External Integration

An external service could be unavailable for a few minutes. Recovering our work sometimes took much longer. That imbalance became clear on a government integration platform I worked on at Softcraft. One integration had a frustrating problem: a short interruption outside our system created a list of operations for us to repair later. The interruption might last only a few minutes. The operational consequences lasted much longer. Someone had to identify the affected operations, determine which ones were safe to repeat, and move them through the process again. A small dependency failure became a production incident for our team. We could improve our code, infrastructure, and deployment process. We couldn’t guarantee that every external system would always be available. This meant we had to treat availability as part of the integration contract. The initial flow assumed an immediate answer: receive the request, call the dependency, and return the result. It worked well when everything was healthy. When the dependency timed out, the application had too little information to know what should happen next.

The timeout was especially difficult because it didn’t always mean the external operation had failed. The other system might have completed the work while the response was lost. Blindly sending the request again could create duplicate effects.

The Unknown Result

We separated failures we understood from results we didn’t know yet. If the dependency rejected invalid data, there was nothing useful to retry until the input changed. If the connection failed before a request was sent, another attempt was usually safe. A timeout after sending the request was different because our system couldn’t tell whether the operation had completed on the other side.

Where the external service allowed us to query an operation or send our own identifier, we used that before creating a second request. Where it didn’t, the operation sometimes had to stop for review instead of automatically retrying. That meant recovery wasn’t one generic policy around the HTTP client. It depended on what the business action did and what information the dependency exposed.

Operation Identity and State

We redesigned the flow so each business operation could be recognized across attempts. Instead of treating every call as new work, the system recorded the operation and its current state. Some processing moved to asynchronous workers, temporary network failures could retry with limits and invalid business data stopped for review. We used the existing Node.js and NestJS stack; most of the discussion was about which operations were safe to repeat and what the system knew after each attempt. We also showed whether an operation was waiting, retrying, completed or needed attention, because background processing wasn’t useful if it only made the failure harder to see.

The support team needed the operation identifier, the number of attempts and the last confirmed stage. We kept the technical error as well, but it was usually less important than knowing if the external call had been sent and whether a later status check had found a result. This reduced the time spent searching across application logs before somebody could decide what to do.

Retries waited longer between attempts and stopped after a limit. During a longer outage, this avoided sending the same traffic continuously to a service that was already unavailable. It also kept the workers from spending all their time on older failed requests while new operations were still arriving.

Without that state, the background processing would only replace a visible error with uncertainty. After the changes, the dependency still had interruptions, but a normal interruption didn’t create the same amount of investigation and manual repair for our team. The system could wait, retry and expose the operations it couldn’t resolve alone. I couldn’t control whether another service had a bad afternoon, but at least every bad afternoon didn’t become an emergency for us too.