Handling Temporary Failures in an External Integration

An external service could be unavailable for a few minutes. Recovering our work sometimes took much longer. That imbalance became clear on a government integration platform I worked on at Softcraft. One integration had a frustrating problem: a short interruption outside our system created a list of operations for us to repair later. The interruption might last only a few minutes. The operational consequences lasted much longer. Someone had to identify the affected operations, determine which ones were safe to repeat, and move them through the process again. A small dependency failure became a production incident for our team. We could improve our code, infrastructure, and deployment process. We couldn’t guarantee that every external system would always be available. This meant we had to treat availability as part of the integration contract. The initial flow assumed an immediate answer: receive the request, call the dependency, and return the result. It worked well when everything was healthy. When the dependency timed out, the application had too little information to know what should happen next.

The timeout was especially difficult because it didn’t always mean the external operation had failed. The other system might have completed the work while the response was lost. Blindly sending the request again could create duplicate effects.

Operation Identity and State

We redesigned the flow so each business operation could be recognized across attempts. Instead of treating every call as new work, the system recorded the operation and its current state. Some processing moved to asynchronous workers, temporary network failures could retry with limits and invalid business data stopped for review. We used the existing Node.js and NestJS stack; most of the discussion was about which operations were safe to repeat and what the system knew after each attempt. We also showed whether an operation was waiting, retrying, completed or needed attention, because background processing wasn’t useful if it only made the failure harder to see.

Without that state, the background processing would only replace a visible error with uncertainty. After the changes, the dependency still had interruptions, but a normal interruption didn’t create the same amount of investigation and manual repair for our team. The system could wait, retry and expose the operations it couldn’t resolve alone. I couldn’t control whether another service had a bad afternoon, but at least every bad afternoon didn’t become an emergency for us too.