development
Agents Are Just Distributed Systems With Delusions of Grandeur
The Honest Version
I’ve been building agentic AI systems and — if I’m being honest — winging significant portions of it.
Not the prompts. Not the tool integrations. Those are fun, and there’s no shortage of tutorials. I mean the architecture. The decisions about what happens when a tool call fails mid-chain. What happens when two agents write to the same state. What happens when your system scales past the point where you can hold it all in your head and reason about failure modes on a whiteboard.
I kept hitting problems that felt novel. An agent retrying a failed API call in a tight loop, burning through tokens and accomplishing nothing. A multi-step workflow where step four failed and I had no way to undo steps one through three. State management that worked fine with one agent and fell apart with three.
Then I started reading. And I discovered something humbling: none of these problems are new.
Everything Old Is New Again
CAP Theorem — published in 2000 — explains why your agent’s shared memory might contain stale data mid-task. Netflix figured out circuit breakers a decade ago because slow dependencies are more dangerous than dead ones; the same applies to every tool call your agent makes. The Saga pattern, documented by Martin Fowler, is literally the solution to “step four failed, now what?” And Google’s SRE team wrote the definitive guide to cascading failures — the exact thing that happens when your multi-agent system accidentally DDoS-es your own infrastructure.
Every problem I thought was unique to agents has a chapter in someone else’s book.
This shouldn’t be surprising. Agents are distributed systems. They hold state that might be stale. They call services that might fail. They run in environments that might be hostile. They make decisions that might need to be reversed. Strip away the LLM and you’re looking at a microservices architecture with natural-language RPC calls and a non-deterministic orchestrator.
The difference is that most of us building agents came up through machine learning, data science, or product engineering — not distributed systems. We learned to fine-tune models, not to design for partial failure. We learned prompt engineering, not blast radius containment. And so we keep reinventing wheels that were forged in production fires at Netflix, Google, and Amazon years ago.
What the Guide Covers
So I sat down and worked through the reading list I should have done six months ago. Fourteen sources, from foundational theory to modern frameworks to regulatory obligations. I’ve compiled them into a structured guide with plain-English summaries, key extracts, and — critically — what each source actually means for agent architecture.
The guide is organised in five parts, and the reading order matters:
Part I: Distributed Systems Theory. CAP Theorem teaches you to accept that no distributed system is perfect — your agent’s memory will contain stale data, and you need to design for that. The Saga pattern gives you a way to live with that imperfection — compensating transactions for multi-step agent workflows. Netflix Hystrix shows you how to stop a failing tool call from taking down your entire reasoning loop.
Part II: Design Principles. The Twelve-Factor App teaches you not to hold state you don’t need to — agents keeping conversation history in local memory become unreproducible snowflakes. The AWS Well-Architected Framework introduces blast radius thinking: a sub-agent doing research should not have access to production databases. Donella Meadows’ Thinking in Systems gives you the vocabulary to reason about reinforcing loops and leverage points — why changing an agent’s goal has more impact than tweaking its temperature.
Part III: Data and Failure at Scale. Kleppmann’s Designing Data-Intensive Applications makes the case for logging every agent action as an event — reproducibility, debugging, and fine-tuning datasets all follow from that one decision. Google SRE’s chapter on cascading failures explains exactly how your multi-agent system can accidentally coordinate a denial-of-service attack against your own infrastructure.
Part IV: Agent Frameworks. Anthropic’s “Building Effective Agents” provides the clearest articulation of when to use workflows versus agents, and the five workflow patterns that cover most use cases. LangGraph makes agent state explicit and typed — which turns out to be how you implement the Twelve-Factor principle of externalised state. AutoGen formalises multi-agent coordination as conversation, which is the one thing LLMs already understand.
Part V: Security and Regulation. OWASP’s LLM Top 10 maps out the attack surface — indirect prompt injection hidden in documents your agents read is the primary vector, and excessive agency amplifies everything. The EU AI Act’s Article 14 on human oversight is directly actionable: implement checkpoints, surface reasoning not just conclusions, maintain audit logs. And Gartner’s Hype Cycle provides the market context: your differentiator is not the agent itself but the reliability, observability, and governance scaffolding around it.
By the time you reach LangGraph and AutoGen, the earlier foundations make their design decisions obvious rather than arbitrary.
Who This Is For
If you’re reading this blog, there’s a decent chance you’re already building with agents — or about to start. And there’s a decent chance you came to it from a background that didn’t include designing for partial failure at scale.
That’s not a criticism. Six months ago I was in the same position. I could get an agent to call tools, maintain conversation state, and produce useful outputs. What I couldn’t do was explain what would happen when the third tool call in a five-step chain timed out. Or articulate why my state management approach wouldn’t survive the addition of a second agent. Or reason about the security implications of letting an agent read arbitrary documents.
This guide is for people who aspire to be professional practitioners in the agent space — people looking to move beyond being “vibe coders” or dabblers, and into building systems that are robust enough for real-world deployment. Whether that’s research, an ambitious hobby project, or enterprise-grade production.
You don’t need to become an expert in distributed systems theory. The guide skips across the lake of knowledge — giving you the essential context, the key concepts, and the core sources across fourteen areas. Enough to recognise the patterns, enough to know what questions to ask, enough to avoid the mistakes that distributed systems engineers learned to avoid a decade ago.
When to Use It
Here’s the thing I wish someone had told me: use it while you’re building, not before.
Don’t use it to procrastinate. Don’t use it to pontificate at architecture meetings. Read the CAP Theorem summary, then go look at your agent’s state management and ask whether you’ve thought about stale reads. Read the Saga pattern section, then go look at your multi-step workflow and ask what happens when step three fails. Read the circuit breaker section, then go look at your tool calls and ask what happens when one of them is slow.
The guide is designed to be practical at any scale. Starting a weekend project? The foundations will prevent you from painting yourself into a corner. Building for production? The security and regulation sections will save you from the kind of surprises that arrive as legal emails. Somewhere in between? Pick the sections that match your current pain point and come back for the rest when you need them.
The Core Insight
If I had to compress the entire guide into one sentence, it would be this:
Agents are distributed systems trading latency and cost for flexibility. They hold untrustworthy state, call potentially-failing tools, operate in possibly-adversarial environments, and make potentially-reversible decisions.
Build them as Netflix builds services: fail fast, recover quickly, instrument comprehensively, and grant no more permissions than the task requires.
Everything else is details. Important details — fourteen sources worth of details — but details that hang from that one insight. Once you internalise it, agent architecture stops feeling like a new discipline and starts feeling like an old one with a new interface.
The full guide is available here: Agent Architecture: A Reading Guide for Practitioners.
Steve Chambers is the founder of Viewyonder, a GenAI consultancy that helps teams build reliable, well-architected AI systems. He compiled this reading guide while building Injectionator, a security tool for AI applications — and learned most of these lessons the hard way. If your team is building with agents and wants to get the foundations right, get in touch.