Building a Reliable API Integration

At Softcraft, I worked on a government platform that needed to communicate with services outside our control. One of the integrations looked simple when we first described it. A user started an operation, our API validated the request, called the external service and stored the result.

The implementation became more complicated when we considered what happened if the response didn’t come back. The external service could complete the operation and our connection could still time out before receiving the confirmation. For our API it looked like a failure, but trying again could execute the same business operation twice. We also had cases where our database update completed and the next stage didn’t, so returning a generic error wasn’t enough because the system still needed to understand what had already happened.

The Happy Path

The normal flow received valid data, called the service, recorded its response and returned the result to the user. We used Node.js, TypeScript, NestJS, Redis and PostgreSQL, and the controller for this part wasn’t especially difficult. The problems appeared because the other services had their own availability and response times, and a temporary issue on their side could keep our user waiting or leave an operation without a clear result.

We moved parts of the workflow to asynchronous processing. The API validated and accepted the request while workers handled the integration with controlled retries. We also recorded the state of the operation between the attempts. Some important workflows went from minutes to seconds after these changes, but most of the implementation discussions weren’t really about performance.

Changing the flow also changed the API contract. Before, a successful response meant the external work had completed. After the change, it could mean that our platform had accepted a valid request and would continue processing it. We had to make that difference clear to the systems using the API and provide a way to see the final status. Returning faster wasn’t useful if the caller assumed the operation was finished and continued with information that was still being processed.

We kept some validations synchronous because there was no reason to accept work we already knew was invalid. The slower integration steps moved to workers, while format, permission and basic business checks still returned feedback immediately. This kept obvious errors out of the queue and made the asynchronous part responsible only for work that could actually continue.

Trying Again

We spent more time deciding how to identify the same operation and when trying again was safe. If the external service completed a request but we lost the response, the next attempt had to be connected to the first one instead of being treated as unrelated work. We also needed limits because continuously retrying an unavailable dependency would only add more requests to a system that was already having problems.

This meant that the HTTP response couldn’t be the only record of what had happened. The API might return that the work was accepted while the operation continued in the background, and another part of the system needed to know if it was waiting, being retried or had reached a point where somebody should look at it.

We used an operation identifier across the request, worker execution and external call. Where the other service supported an idempotency key or a way to query an existing operation, we used it. Where it didn’t, we had to be more conservative because our database could tell us that a request had been sent but not always whether the other side had completed it. A timeout in that moment was an unknown result, not simply a failed result.

That distinction affected the retry policy. Network failures before a request was sent could usually retry automatically. A timeout after sending it sometimes needed a status check or manual review before another attempt. Invalid data shouldn’t retry at all. We ended up with different recovery paths even though all of these cases could originally appear as an error from the same integration client.

Partial States

An operation could be validated but not sent, sent without confirmation or confirmed externally while our local update was still pending. We made these states visible enough for the application and the support team to see where the flow had stopped. We also used operation identifiers and logs with the attempt and completed stage, because “the integration failed” didn’t give us much to work with during support.

We tried to keep the number of states understandable. It was tempting to create one status for every technical step, but the user and support team didn’t need to know every internal detail. They needed to distinguish work that was waiting normally from work that needed attention, while the engineering logs could contain the lower-level information. We changed some names after seeing how people interpreted them because a technically accurate status could still suggest the wrong action to the person reading it.

The workers also introduced operational work that the synchronous endpoint didn’t have. We needed to monitor pending jobs, see if the oldest operation was getting older and make sure a deployment didn’t abandon work that was already in progress. Redis helped us manage the background processing, but it didn’t decide what to do with a job that had used all its attempts. That remained part of the business workflow and needed a visible outcome.

I still remember this project because most of our time went into situations that didn’t appear when we tested the normal flow: a response that never arrived, a request sent twice or a process that stopped after only some steps. After the changes, temporary failures created less manual recovery and we could usually see whether an operation was waiting, completed or needed somebody to investigate it. Before that, a timed-out request often left us without any idea of what the other system had done.