Why we put a gateway in front of every AIOps agent

The first version of the pilot didn't have a gateway. We wired an agent straight to the observability stack, gave it tools to query metrics and read runbooks, and let it loose on a stream of alerts. It worked in the demo. It fell apart the first week it saw production volume.

This is the story of what broke, and the one architectural decision — a gateway sitting between the agents and everything else — that made the rest of the system tractable.

The problem an AIOps pilot is actually solving

The pitch for AIOps is seductive: an on-call engineer wakes up to a triaged incident instead of a wall of alerts. The agent has already correlated the spike, checked the recent deploys, pulled the relevant logs, and written a two-line summary with a confidence score.

That's the output. The hard part is everything between the alert firing and that summary landing in Slack — and almost none of it is the model.

When we mapped the real flow, a single "triage this alert" task fanned out into a dozen tool calls: query the metrics backend, look up the service owner, read the last three runbook revisions, check the deploy log, search past incidents for a similar signature. Each of those is a credential, a rate limit, a timeout, and a thing that can page us at 3am if it misbehaves.

What broke without a gateway

Three things, in order of how much they hurt.

Credentials sprawled. Every agent needed direct access to every backend. That's a secret in every agent's environment, multiplied by every backend, and no single place to rotate or revoke. The security review stalled here, and it was right to.

Nothing was observable. When an agent did something dumb — and they do — we had no single tap to see what tool it called, with what arguments, and what came back. Debugging meant reading scattered logs across five services and guessing at the order.

Cost and rate limits were invisible until the bill. An agent in a retry loop can hammer a metrics API a thousand times before anyone notices. We had no backpressure, no per-agent budget, no kill switch short of pulling the deployment.

None of these are model problems. They're plumbing problems. And plumbing problems want a plumbing solution.

The topology we landed on

We put a single agent gateway between the agents and the world. Every tool call — every query, every runbook fetch, every write back to the incident channel — goes through it. The agents no longer hold credentials or know where the backends live. They know the gateway, and the gateway knows everything else.

That one move collapsed the three problems into one place you can actually reason about:

• Auth lives at the gateway. Backends trust the gateway, not the agents. Rotating a credential is one change, not twelve. Revoking an agent is flipping a flag.
• Every call is logged at the gateway. One tap, full request/response, in order. The first time we could replay an agent's exact decision path, mean time to "oh, that's why" dropped from an afternoon to minutes.
• Policy lives at the gateway. Rate limits, per-agent budgets, and an allow-list of which tools each agent can even reach — all enforced before a request ever touches a backend. The retry-loop incident becomes a 429 the agent has to handle, not a surprise on the invoice.

The shape is boring on purpose. It's the same reason you put an API gateway in front of microservices, or a service mesh between them: when you have N things talking to M things, you do not want N×M relationships. You want N+M, with one controlled seam in the middle.

The agents got simpler once the gateway existed. They stopped being distributed systems components and went back to being what they're good at: reasoning over the data the gateway hands them.

What the gateway is not

It's tempting to let the seam grow into a god object. We held a line on two things.

The gateway does no reasoning. It routes, authenticates, logs, and enforces policy. The moment it starts deciding which runbook is relevant, you've moved the interesting logic into the one component you can't easily test in isolation.

The gateway is not a cache pretending to be a brain. We do cache idempotent reads — the same metrics query inside one triage shouldn't hit the backend twice — but caching is a transparent optimization, not a place to encode judgment.

Was the pilot worth it?

The honest answer: the value of the pilot wasn't proving that an LLM can triage an alert. We knew it could. The value was discovering that the model was maybe 20% of the work, and the other 80% was the boring, critical infrastructure that decides whether you can run this safely at all.

If you're scoping an AIOps pilot, budget accordingly. Spend the first week on the seam — the gateway, the auth model, the observability — and the agents will have a stable floor to stand on. Skip it, and you'll spend that week anyway, except in production, at 3am, debugging an agent that paged the whole team.

---

We design and build platform and AI infrastructure like this end-to-end. If you're standing up something similar and want a second pair of eyes on the topology, get in touch.