AI agent architecture review
For engineering leads, CTOs and founders whose AI agent works in the demo and now needs an honest answer on whether it's ready for real users. I review one agent and give you a written report, a fix list ranked by risk and a call to walk through it.
What an AI agent architecture review is
An AI agent architecture review is an independent check of one AI agent and the release you plan to ship. A reviewer reads how it is built, looks at what it can do and who approves it, tests the ways it can fail, and gives you a written list of fixes ranked by risk. It is not a pen test, a compliance certificate or a rebuild.
It's about the agent you actually have, running on your tools and your data. It isn't a workshop about diagrams.
What I check
An AI agent architecture review checks what the agent is allowed to touch, where a person must approve before anything risky happens, how steps that depend on earlier results are ordered, what it remembers, how it behaves when a service is slow or fails, and whether you can see and stop what it did. It also looks at how you know it works, meaning tests and evaluations you run before each change.
Who the agent acts as, and what it can touch
I start with permissions: which accounts and keys the agent uses, and which tools can read, write or run commands. Most of the risk sits in the write and execute tools, so I check each one is only available where it's needed. In my own agent platform, Goose, dangerous tools are only switched on for the phase that carries out actions (how that works).
Where a person must approve first
I look at which actions need a human yes before they run, what the approver actually sees, and what happens if nobody answers. Goose gates file writes and shell commands behind approval so they don't happen silently (Goose).
What happens when step 5 needs the result of step 2
I check that steps which depend on earlier results really wait for them. Some models write every tool call in one go, so a later call goes out with placeholder or guessed arguments. I hit this in Goose with 14B models and fixed it by running each mission phase as its own model call, with the previous phase's output passed in (the write-up).
What it remembers, and for how long
I review what the agent stores, how long it keeps it and how it gets deleted. Memory needs hard limits. My Discord bot keeps conversation history but prunes it by both message count and token budget (case study).
What happens when things go wrong
I test how the agent behaves when a model is slow, an API fails or requests arrive in a burst. That covers queues, timeouts, rate limits, retries and shutdown. The Discord bot serialises requests with queue limits and timeouts, and applies rate limits before any expensive work starts. The Wee Table falls back across several data providers and caches in layers, so people get stale but usable data instead of an error (The Wee Table, how the fallback works).
Whether you can see what it did, and stop it
I check that actions are logged, that you can trace a bad result back to the step that caused it, and that someone can stop the agent quickly. Rollback and a kill switch matter far more in production than they do in a demo.
How you know it works
I look at the tests and evaluations you run before each change, and whether they would catch the failures you've already seen. Without them, an agent can get worse after a prompt or model change and nobody notices until a user does.
What a review looks like on a real agent
Most pages on this topic give you a checklist. This section shows the kind of findings a review turns up, taken from Goose, the agent platform I build and run on my own hardware. These are real problems I found and changed, and one that is still open.
| Finding | Risk | Fix | Status |
|---|---|---|---|
| Scheduled missions with dependent steps sent tool calls with placeholder or guessed arguments, because smaller models wrote every call at once. | High | Split each mission into phases, each its own model call, with the previous phase's output passed in. | Fixed in v1.2.0 |
| Dangerous tools were available for a whole mission, including the steps that only gather data or write text. | High | Dangerous tools allowed only in the execute phase. Tools turned off for writing phases. | Fixed in v1.2.0 |
| Earlier workarounds for the ordering problem split missions in two and tied plugins to each other. | Medium | Replaced by phases, and the workarounds removed. | Fixed in v1.2.0 |
| Simple jobs like backups still go through the model, which adds delay and the chance of a made-up step. | Low | Run single-tool jobs directly, with no model involved. | Open, planned |
The open item stays in the table on purpose. A review should tell you what hasn't been fixed yet as plainly as what has.
What you get
- One agent reviewed, along with the release you plan to ship.
- A written report with every finding ranked by risk.
- A prioritised fix list, so your team knows what to do first.
- A walkthrough call to go through the findings and answer questions.
How it works
The review takes about one week from the day you share access.
- Intro call. We talk about your agent, what it does and what's worrying you. You bring whoever owns the agent.
- Access and evidence. You share the code, config, prompts, the list of tools it can call, and any logs or test results you have.
- Review. I read how it's built and test the ways it can fail. I may ask the engineer who built it a few short questions.
- Report. You get the written report and the fix list.
- Walkthrough call. We go through the findings together, with whoever will make the fixes.
What the review covers and what happens next
The review covers one agent: its permissions, approvals, step ordering, memory, failure handling, logging and tests. It doesn't cover fixing what it finds, penetration testing or compliance sign-off.
To start, get in touch through the contact page. We'll have a short scoping chat about your agent, then I'll send you a quote tailored to it.
What this is not
An AI agent architecture review is not a code review, a security test or a compliance audit, though it touches on all of them.
- Not a code review. I read the code, but the question is how the system behaves, not how tidy it is.
- Not a pen test. I look at what the agent is allowed to do. I don't try to break into it.
- Not a compliance certificate. You won't get a sign-off for an auditor or regulator.
- Not a rebuild. You get a fix list, not new code.
Sometimes the right answer is a simpler workflow or no agent at all. If a job is a fixed sequence of steps with no judgement in it, a plain script is cheaper to run and easier to trust. In Goose I'm moving single-tool jobs like backups off the model entirely for that reason (the write-up). If that's what I see in your setup, I'll tell you.
When your agent is ready for a review
Your agent is ready for a review once it runs end to end on real tools, even if only in a demo. That's usually the point where someone senior asks whether it's safe to launch. It also makes sense if the agent is already live and has done something odd, like calling a tool with the wrong arguments or taking an action nobody approved.
If nothing is built yet, a review is too early. If you need a certificate for a regulator, you need a different kind of audit.
A good fit:
- One agent heading for production
- An agent already live and causing worry
Not a fit:
- Nothing built yet
- You need formal certification or a security sign-off
About the reviewer
I'm Rory Standley, an AI architect based in England. I build and run my own software, including Goose, a local-first agent platform with approval-gated tools, a Discord bot running on local models, Rune and The Wee Table. I work with UK teams remotely.
Questions
What is an AI agent architecture review?
An AI agent architecture review is an independent check of one AI agent and the release you plan to ship. A reviewer reads how it is built, looks at what it can do and who approves it, tests the ways it can fail, and gives you a written list of fixes ranked by risk. It is not a pen test, a compliance certificate or a rebuild.
What does an AI agent architecture review check?
It checks what the agent is allowed to touch, where a person must approve before anything risky happens, how steps that depend on earlier results are ordered, what it remembers, how it behaves when a service is slow or fails, and whether you can see and stop what it did. It also looks at how you know it works, meaning tests and evaluations you run before each change.
What do I get from a review, and how long does it take?
You get a written report with findings ranked by risk, a prioritised fix list and a walkthrough call. For one agent, the review takes about one week from the day you share access. It covers permissions, approvals, step ordering, memory, failure handling, logging and tests; it does not cover fixes, penetration testing or compliance sign-off. To start, get in touch for a short scoping call and I'll send a quote tailored to your agent.
How does pricing work?
Get in touch for a short scoping call, and afterwards I'll send you a quote tailored to your agent.
What do I need to give you, and who do you need to talk to?
You give me access to the agent's code, config, prompts and tool list, plus any logs or test results you have. I talk to whoever owns the agent on the intro call, and to the engineer who built it for a few short questions during the review.
Do you fix what you find?
No. The review ends with the report, the fix list and the walkthrough call, so the findings aren't shaped by who gets the follow-on work. If you'd like help with the fixes afterwards, we can talk about that separately.
Does it work on agents that run locally, for example on Ollama?
Yes. Goose and my Discord bot both run on Ollama, so I review local agents the same way as ones built on a hosted API.
Book a review
Tell me about your agent and we'll set up a short scoping call. After it, I'll send you a quote for the review.