Architecture Decisions on a High-Availability Platform

While working on a high-availability security platform, I participated in decisions around APIs, asynchronous workers, event processing, databases, and AWS infrastructure. There were usually several valid ways to build a capability. Finding another component to add to the diagram wasn’t difficult. We could keep a workflow inside one service or split processing into another component. We could calculate information when requested or maintain it as events arrived. We could introduce another data store optimized for the access pattern or accept limitations in the existing model. The difficult part was choosing which cost the team was prepared to carry after the design meeting.

Where the Complexity Went

Splitting a service can make the codebase and deployment boundary smaller. It also introduces network communication, another deployment, additional monitoring, and questions about data consistency. Processing work asynchronously can protect a user-facing API from slow dependencies. It also creates a backlog the team needs to monitor and a recovery path it needs to operate. Caching can reduce latency. It introduces freshness, invalidation, and memory behavior that somebody must understand during an incident. These patterns are useful when their benefits are bigger than their costs, but they don’t remove complexity for free. In one discussion, a question cut through a lot of design detail: who would know that this worker was stuck, and what could that person safely do about it? We had good answers for producing the work and weaker answers for operating it. This made the decision more practical. We needed to know where the complexity would be and if the team could control it.

The Team Operating the System

It’s easy to discuss architecture as if the system will be operated by an unlimited platform organization. I’ve done it too, especially when a design looked clean and the operational details were still outside the diagram. The real team has a size, an experience profile, deadlines, and existing production responsibilities. A new component adds work to on-call support, deployment, security, observability, and upgrades. If only one person understands the new technology, that’s part of the risk. If the team can’t monitor the additional service well, that’s part of the design. If delivery requires three other initiatives to finish first, that’s part of the cost. This doesn’t mean we always choose the easiest implementation. It means we include the operational work in the decision. Not every feature needs the same level of resilience. A low-impact internal report and a security event-processing flow shouldn’t have the same architecture only because the organization has a standard pattern. For critical processing, additional state, retries, redundancy, and monitoring may be necessary. For a small reversible feature, the same design could create more maintenance work than business value.

I started asking what happens if the feature fails, how fast it needs to recover and if we can rebuild the data. These answers gave the discussion a clearer boundary.

Reversible and Irreversible Decisions

Some choices were easy to change later and others ended up inside data models, contracts and production routines. We spent more time on the second group because the cost would stay with the team for longer. In architecture discussions, I started paying attention whenever we could describe the benefit of a new component but not who would monitor it or recover its work. On that platform, the designs that moved forward were usually the ones where we could name both sides: another service and an independent deployment, another processing state and a safer recovery path. Once both were visible, the conversation became much less abstract.