Moving an Integration Workflow to Asynchronous Processing

We had an integration workflow that tried to finish every step before answering the user. It looked simpler that way. It also meant the user waited on every dependency in the chain. The request crossed multiple systems. When all dependencies responded quickly, the experience was simple: the user submitted the operation and received the final result. As the workflow grew, this became a problem. A slow external service made our API slow. A temporary outage made the whole operation unavailable. More traffic meant more requests waiting on resources we didn’t control. Moving the work to background processing made technical sense. It also meant we could no longer pretend that “request accepted” and “work completed” were the same thing.

Accepted and Completed

In a synchronous flow, a successful response usually suggests that the work is finished. In an asynchronous flow, the system may only be promising that the work has been accepted and will continue. We had to decide whether the user could continue with another task, how they would know when processing finished, what happened when it took too long and which status another system should treat as final. The queue didn’t answer any of this. We needed to agree on the lifecycle of the operation before the backend and the interface could describe the same behavior.

The Operation States

We represented the operation through clear states rather than a single success-or-failure result. The API validated the request and recorded the accepted work. Workers performed the slower integration steps. The system updated progress and made failures visible when automatic recovery wasn’t possible. Retries handled temporary problems, but they weren’t hidden forever. A user shouldn’t see “processing” indefinitely because the backend doesn’t know how to describe a failed operation. This also affected support and operations. A clear status allowed the team to determine where in the process the work had stopped without searching through unrelated logs.

We kept validations that could answer immediately in the API. Invalid input, missing permissions and operations that violated a known business rule didn’t need to enter a queue only to fail a few seconds later. The background work started after the request was valid and accepted. This gave the user useful feedback for problems they could correct while still separating them from the slower dependencies.

The status names changed more than once. Our first version described technical stages that made sense to the engineers but didn’t help the people using the platform. They needed to know if they could continue, if they should wait or if somebody had to correct the operation. We kept more detailed stages internally and exposed a smaller set that reflected what the user could do next.

What the Queue Added

Asynchronous architecture can make an API respond faster, but the work still exists. The team now owns a backlog. We need to know if messages are being processed, delayed, duplicated or abandoned. We also need to plan for traffic spikes and deployments while work is still running. A queue moves the waiting from the request to another part of the system. That can be exactly the right decision, but only if the new waiting is observable and manageable. I’ve seen diagrams where adding a message broker appears to solve scalability by creating an arrow between two services. In production, that arrow represents retention, retries, ordering, monitoring, and recovery. For the integration platform, asynchronous processing separated the user-facing request from the availability of slower dependencies. It allowed the system to absorb temporary instability and process work when the dependency recovered. It also made the workflow easier to scale because workers could scale separately from the API.

We monitored how many operations were waiting, how old the oldest one was and how often jobs needed another attempt. The total queue size alone wasn’t enough because a short traffic peak could create many jobs that disappeared quickly, while one operation waiting for hours could indicate a different problem. Support also needed to search for one business operation without knowing which worker had processed it.

Deployments became part of the design as well. A worker running a new version could receive work created by the previous one, and we couldn’t assume every pending message had the newest format. We kept message changes compatible during transitions and allowed active jobs to finish when shutting workers down. These details weren’t visible in the API contract, but missing them could turn a normal deployment into a set of abandoned operations.

There were flows we didn’t move to asynchronous processing. Some operations were fast, local and needed a final result before the user could continue. Adding a queue to them would have introduced more states and operational work without solving an actual problem. The decision was made per workflow, based on the waiting and failure behavior we already had.

The improvement only worked after we agreed on what “accepted” meant, which states the user could see and what happened to an operation that couldn’t finish automatically. The queue was only one part of that discussion. I remember spending more time on the meaning of each status than on the worker itself, because the backend could process everything correctly and still leave the user looking at “processing” without knowing if they should wait or try again.

After the change, the API no longer held a connection through every external step, and a temporary dependency problem didn’t block the complete user-facing request. We accepted more backend state and monitoring because this workflow had already shown us why we needed them.