Building and Operating an AI Agent Platform on GCP

During my time at Softcraft, I worked on an internal AI platform for teams that needed to investigate support cases across several company systems. A normal investigation could involve reading a ticket, searching internal documentation, checking the status of an application and looking for information owned by another service. People knew how to do this, but a lot of time was spent moving between tools and rebuilding the same context before they could make a decision.

The idea was to give them an agent that could gather this information and organize it into one investigation. I owned the architecture, backend and GCP infrastructure behind the platform, including the agent orchestration, MCP servers, deployments, permissions, observability and the evaluation process we used before releasing changes. I also worked with the teams that owned the connected systems because exposing a tool to an agent without understanding its data or permissions would have been a very quick way to create a different problem.

The first prototype was relatively fast to build. We gave a model a few tools, described the task and received an answer that looked useful. That version worked well in a controlled example and became much less convincing when the request was incomplete, one tool was slow or two internal systems returned information that didn’t agree. Most of the project happened after we already had something we could demonstrate.

The Agent Workflow

The platform accepted an investigation with the user’s question and the context we already knew, then created an execution record before asking the model to do anything. The agent could decide which available tools it needed, call them through MCP and continue with the results until it had enough information to produce a response. Some investigations finished quickly, while others involved several systems and continued asynchronously after the initial API request.

We used a coordinator for the complete investigation and smaller agents for documentation and operational checks when those paths became long. They shared the same persisted run instead of handing text back and forth without history. For simpler requests, the coordinator called the MCP tools directly because involving more agents only added latency and more decisions to trace.

We used TypeScript and NestJS for the main platform APIs and MCP services, with Python for parts of the agent and evaluation workflows. The model access ran through Vertex AI, and the services were deployed on Cloud Run. Pub/Sub carried longer work between the API and workers, Cloud SQL stored the investigation state and tool history, and Cloud Storage kept larger artifacts that didn’t belong in the database. Infrastructure was managed with Terraform, including service accounts, IAM roles, secrets, networking, deployments and monitoring.

I didn’t want the agent execution to exist only inside one model response. Each run had an identity and a state, and every tool call was recorded with its input, result, duration and the version of the agent configuration that requested it. If a worker restarted, we could see what had already completed and decide whether the next step could continue. If a tool returned an error, the platform could distinguish a temporary technical failure from a result that meant the requested information didn’t exist.

We also limited how long an agent could continue and how many tool calls it could make. Without those limits, a model could repeat a search using slightly different terms, collect more context than it needed and increase both latency and cost without getting closer to an answer. A maximum number alone wasn’t enough, so we logged why the agent selected each tool and looked for repeated calls during evaluation. Some loops were caused by the prompt, while others showed that a tool description or result was too vague for the model to know what to do next.

Connecting Internal Systems With MCP

MCP gave us a consistent way to describe and expose tools from different systems. We created servers for internal documentation, support tickets, application metadata and deployment information. The agent saw tool names, descriptions and structured inputs instead of needing custom integration logic inside every prompt. When another workflow needed the same capability, we could reuse the MCP server rather than implementing another model-specific connector.

The protocol made the integration shape cleaner, but it didn’t decide which data the agent should access or what a tool was allowed to do. We still had to work through authentication, authorization, timeouts, rate limits and audit information for every connected system. The identity of the person starting an investigation followed the run, and the MCP service checked that identity before returning protected information. A tool wasn’t allowed to become a shortcut around permissions that already existed in the original application.

We also treated text returned by tickets and documents as untrusted data, not as new instructions for the agent. A support ticket could contain anything somebody had pasted into it. The tool result stayed separated from the platform instructions, and the agent couldn’t gain another permission because a document asked it to call a restricted tool.

Most of the first tools were read-only. The agent could search, compare and explain, but it couldn’t change a production resource or close a support case by itself. For the few actions we considered later, the agent prepared the request and a person confirmed it before execution. The approval included the exact action and parameters, not a generic permission for the agent to continue doing whatever it decided. This made the workflow slower than a fully autonomous demo, but it matched the kind of systems and data we were working with.

Tool responses needed design as well. One early connector returned a large document because that was easy for the source system, but the agent only needed a few relevant sections. Passing the complete result increased the context and sometimes made the final answer worse because old or unrelated information received the same attention as the useful part. We moved filtering and pagination into the MCP tool, returned source references with the result and kept the raw document available only when the agent had a reason to request it.

We treated tool descriptions almost like API contracts. A description such as “search cases” wasn’t enough when there were several kinds of cases and different date fields. The model needed to know what the tool searched, which filters were supported and what an empty response meant. Small changes in those descriptions affected which tools the agent selected, so they were versioned and reviewed with the server code instead of being edited as informal prompt text in production.

Owning the GCP Infrastructure

Running the platform on GCP meant I also owned the path from a pull request to a production agent execution. Cloud Run worked well for the APIs and MCP services because they were mostly request-based and could scale independently, but the longer agent runs couldn’t depend on one HTTP connection staying open. The API stored the work and published it for a worker, then the frontend followed the persisted state. This allowed a run to survive a client disconnect and gave support somewhere to look when an investigation remained in progress.

Each service had its own service account and only the permissions it needed. The documentation connector didn’t need the same access as the deployment connector, and the agent orchestrator didn’t receive a broad credential that could call every internal system directly. Secrets stayed in Secret Manager and were loaded by the relevant service identity. We used private connectivity where the connected systems supported it and restricted the public entry points to the APIs that actually needed them.

The deployments ran through CI/CD and infrastructure changes went through the same review process as the application. A new MCP tool sometimes required a role, secret or network path, so separating the code and infrastructure discussions completely didn’t work. We tested the service with the permissions defined for its real account instead of relying on a developer credential with wider access. More than once, that found an accidental dependency before production.

Observability was organized around the complete agent run. Cloud Logging and tracing connected the API request, model calls, worker execution and MCP tools through the same run identifier. We tracked model latency, tool latency, failed calls, retries, token usage and runs that stopped because they reached a limit. Infrastructure metrics told us if a Cloud Run service was healthy, but they didn’t tell us if agents were repeatedly selecting the wrong tool or producing an answer without enough evidence.

Cost was another part of the operational work. The expensive behavior wasn’t always one large request. An agent that made several unnecessary calls could multiply model tokens, tool traffic and execution time across normal usage. We recorded cost-related usage by run and compared it with the number of completed investigations. Caching helped for selected reference data, but we didn’t cache protected or frequently changing results without a clear invalidation and isolation rule.

Evaluating Changes

Testing the backend services was familiar: contracts, permissions, state transitions and failure handling could be covered with automated tests. Testing the agent required another layer because the same prompt could produce slightly different wording or choose a different valid path. We didn’t compare the final response as one exact string. We created a set of anonymized investigation cases and checked the parts that mattered: whether the agent used allowed tools, whether the important facts came from the available sources, whether it respected access rules and whether the conclusion was supported by the tool results.

We saved the execution trace for failed evaluations because the final answer often didn’t explain where the problem started. Sometimes the model misunderstood the question. Sometimes it selected the correct MCP tool with the wrong filter, or the tool returned an ambiguous response and the agent filled the gap with an assumption. Each case needed a different change. Updating the prompt wouldn’t fix a connector that returned unclear data, and adding another agent wouldn’t fix a missing permission check.

Prompt, model and tool changes were versioned together with the evaluation result. Before a release, we ran the known cases and compared correctness, tool usage, latency and cost with the current version. The suite wasn’t perfect and it didn’t predict every request people would make, but it stopped us from improving one example while quietly making several existing cases worse.

The team also reviewed real usage with sensitive information removed. We looked for questions the platform couldn’t answer, tools people expected but didn’t have and situations where the agent response sounded more certain than the sources allowed. In those cases, admitting that the available systems didn’t contain enough information was a better result than producing a complete-looking answer.

By the end of the project, the agent could take a support investigation, collect evidence from several internal systems and return a structured result with the sources and actions it had used. Each run had permissions, state, history, limits and enough observability for us to investigate when the model or one of its tools behaved differently from what we expected.

I enjoyed owning this platform because it combined backend engineering, cloud infrastructure and the less predictable parts of working with models. Most of my time went into access, failures, cost and production support. Those were also the parts that allowed other teams to use it without depending on the people who built the first prototype.