Architecture Decisions on a High-Availability Platform

While working on a high-availability security platform, I participated in decisions around APIs, asynchronous workers, event processing, databases, and AWS infrastructure. We could keep a workflow inside one service or split processing into another component, calculate information when requested or maintain it as events arrived, and introduce another data store or accept limitations in the existing model. Several options worked technically. We had to choose which operational cost the team was prepared to carry after the design meeting.

Where the Complexity Went

Splitting a service can make the codebase and deployment boundary smaller. It also introduces network communication, another deployment, additional monitoring, and questions about data consistency. Processing work asynchronously can protect a user-facing API from slow dependencies. It also creates a backlog the team needs to monitor and a recovery path it needs to operate. Caching can reduce latency. It introduces freshness, invalidation, and memory behavior that somebody must understand during an incident. These patterns are useful when their benefits are bigger than their costs, but they don’t remove complexity for free. In one discussion, a question cut through a lot of design detail: who would know that this worker was stuck, and what could that person safely do about it? We had good answers for producing the work and weaker answers for operating it. This made the decision more practical. We needed to know where the complexity would be and if the team could control it.

For that workflow, we added a visible operation state, metrics for pending work and a recovery action before moving more processing to the worker. The queue already existed and the code change was small. Giving the team a way to distinguish normal waiting from work that had stopped took more thought. Without it, the asynchronous design would have improved the API response while making support less certain.

In another case, we considered storing a calculated view to make reads faster. The read path would become simple, but every incoming event would then need to update the view correctly and late events could change an earlier result. We kept the calculation at read time until the measured latency justified owning that update logic. The choice wasn’t that one model was more correct. We chose where the current volume allowed the complexity to stay smaller.

Data and Consistency

Data decisions stayed with the platform for a long time, so we spent more time on them than on some framework choices. A separate data store could improve one access pattern, but then the team needed to synchronize it, rebuild it and explain what happened when it was behind the source. Keeping the data in the existing store avoided that work but could make a query or scaling path more difficult.

When we introduced derived data, we tried to make it rebuildable from an owned source. That gave us a recovery path if processing missed an update or the representation changed. The design included the event history and replay process along with the fast query that motivated the new model. If we couldn’t explain how the derived state would be corrected, the read performance benefit wasn’t enough yet.

Consistency requirements varied between features. A dashboard could sometimes show information a few seconds behind without causing a problem. A security decision based on incomplete enrichment had a different risk. We didn’t use the same architecture only because both workflows consumed events. The acceptable delay and the consequence of a wrong result changed where we were willing to keep asynchronous state.

The Team Operating the System

It’s easy to discuss architecture as if the system will be operated by an unlimited platform organization. I’ve done it too, especially when a design looked clean and the operational details were still outside the diagram. The real team has a size, an experience profile, deadlines, and existing production responsibilities. A new component adds work to on-call support, deployment, security, observability, and upgrades. If only one person understands the new technology, that’s part of the risk. If the team can’t monitor the additional service well, that’s part of the design. If delivery requires three other initiatives to finish first, that’s part of the cost. This doesn’t mean we always choose the easiest implementation. It means we include the operational work in the decision. Not every feature needs the same level of resilience. A low-impact internal report and a security event-processing flow shouldn’t have the same architecture only because the organization has a standard pattern. For critical processing, additional state, retries, redundancy, and monitoring may be necessary. For a small reversible feature, the same design could create more maintenance work than business value.

We involved the people who would operate the component before the design was finished. They asked different questions from the feature team: how configuration would be rotated, what an alert would mean, whether work survived a restart and how a failed deployment could be reversed. A few design options became less attractive after those questions, while others only needed additional work to become supportable.

Team familiarity didn’t automatically decide the technology, but it changed the adoption plan. When a new tool solved a real problem, we included documentation, deployment templates and more than one engineer in the implementation. Otherwise, the system would have gained a technical capability and a human dependency at the same time.

I started asking what happens if the feature fails, how fast it needs to recover and if we can rebuild the data. These answers gave the discussion a clearer boundary.

The answers also prevented us from giving every feature the most expensive version of reliability. Some information could be recalculated and tolerate a delay, so a simpler recovery was enough. Other processing affected security results and needed clearer state, faster detection and a controlled replay path. The architecture followed the impact instead of a platform-wide preference.

Reversible and Irreversible Decisions

Some choices were easy to change later and others ended up inside data models, contracts and production routines. We spent more time on the second group because the cost would stay with the team for longer. In architecture discussions, I started paying attention whenever we could describe the benefit of a new component but not who would monitor it or recover its work. On that platform, the designs that moved forward were usually the ones where we could name both sides: another service and an independent deployment, another processing state and a safer recovery path. Once both were visible, the conversation became much less abstract.

We recorded the larger decisions with the constraints and the point where we expected to revisit them. This was useful because the platform changed and a trade that made sense at one volume could look different later. The record didn’t make the choice permanent. It kept the next discussion from depending on somebody remembering why the team had rejected an apparently cleaner option six months earlier.