Multi-agent system architecture: practical patterns for LLM agents that work

A multi-agent system sounds impressive on a slide. In practice it's a set of trade-offs: more agents can mean clearer jobs and better results, or more cost, more failure points and more ways to go in circles. I build these systems for a living, including one that produces finished YouTube videos with 14 agents, so here is what I've learned about structuring them.

This post is about architecture: how agents are arranged, how they share information, and where people fit in. If you're still deciding what to automate, start with what to automate first.

What a multi-agent system is (and isn't)

An LLM agent is a model running in a loop: it reads a goal, decides on an action, calls a tool, looks at the result and decides again. A multi-agent system is several of these, each with its own instructions, tools and context, working towards one outcome.

The point isn't to have many agents. The point is that each agent gets a narrow job and a small, clean context. A model asked to research, write, fact-check and edit at once does all four worse than four focused agents, mostly because its context fills up with material that is irrelevant to the step it's on.

Four multi-agent system architecture patterns

Most systems I've seen or built are combinations of four shapes.

1. Orchestrator and specialists

One agent (the orchestrator, or director) holds the plan. It breaks the goal into tasks, hands each to a specialist, checks what comes back and decides what happens next. Specialists don't talk to each other; they talk to the orchestrator.

This is the most flexible pattern. It suits work where the next step depends on the last result. Its weak point is the orchestrator itself: if it plans badly, every specialist does good work on the wrong thing.

2. Pipeline

Agents run in a fixed order, each taking the previous output as input: research, then draft, then edit, then format. No one decides the order at runtime because the order is part of the design.

Pipelines are predictable, cheap to debug and easy to measure stage by stage. Use one whenever the workflow really is the same every time. Many "multi-agent" projects would be better off as a plain pipeline.

3. Parallel workers

Several agents do independent pieces of work at the same time, and something combines the results. Scouts researching different angles, or implementers working on separate tickets.

The rule here is isolation. Parallel agents must not write to the same place. In my coding workflow, each implementer agent works in its own git worktree so two tickets can move at once without collisions.

4. Reviewer or critic

One agent produces, another tries to find what's wrong with it. The reviewer should have different instructions, ideally a different framing ("find the defect"), and its findings need evidence. Without that, reviewers produce confident noise. I wrote about this in why every AI code reviewer needs a failing repro.

A worked example: an autonomous video pipeline

VidForge is my own system that turns a subject into a finished 10 to 15 minute YouTube video. It combines all four patterns:

Parallel workers at the start. Two scout agents research in parallel and come back with about 20 video ideas, each with a hook and a recommendation.
One human decision. I pick the idea. That's the only choice it needs from me.
Orchestrator and specialists. A Director agent then runs production through 13 specialists: script, fact-checking, narration, footage, images, the on-screen host, motion graphics, editing and thumbnails.
Optional checkpoints. Review points at the script, rough cut and final cut. In overnight mode it skips them and the video is ready by morning.

One constraint shaped the architecture more than any agent design: most of the heavy work runs on local models on a single 16 GB consumer GPU. So generation jobs go through a queue and run one model at a time. Good multi-agent design is often about resources and scheduling, not just prompts.

Shared state and handoffs

How agents pass work to each other decides whether the system is debuggable.

Hand off artefacts, not conversations. Each agent should produce a file or structured record (a script, a shot list, a JSON spec), not a chat transcript the next agent has to interpret.
Keep one source of truth. A shared project state (a folder, a database row, a structured document) that every agent reads from and writes to in defined places.
Validate at every boundary. If the next agent expects a list of scenes with durations, check that before passing it on. Catching a malformed output at the handoff is far cheaper than discovering it three agents later.
Log everything. Inputs, outputs, tool calls and costs per agent. When something goes wrong, you need to know which agent made the first mistake.

When a single agent is better

More agents are not automatically better. Choose one agent, or a plain script with one LLM call, when:

the task fits comfortably in one context and doesn't need different tools at different stages,
latency matters (every handoff adds time),
the budget is tight (every agent re-reads context, and that costs tokens),
you can't yet measure quality. Splitting an unmeasured system into more parts just makes it harder to see what's wrong.

Start with the simplest thing that works, measure it, and split only where one agent is clearly doing too much.

Failure modes to design for

Error compounding. If each step is right most of the time, a long chain is right much less often. Short chains, validation at handoffs and reviewers at the risky steps help.
Loops. Two agents can bounce work back and forth forever. Cap retries and iterations, and escalate to a person when the cap is hit.
Cost creep. An orchestrator that re-plans often, or reviewers that run on every small change, multiply token usage. Track cost per run from day one. See how to reduce LLM API costs.
Vague roles. If two agents could both reasonably do a task, sooner or later both will, or neither. Each agent needs a job description you could give to a new hire.

Where people fit

The most reliable systems I build keep a human at the points where a wrong decision is expensive: choosing what to make, approving before anything is sent or published, and signing off the final result. In my coding pipeline, approval is per commit and the final sign-off is mine, as described in how I ship production code with AI agents. Checkpoints can be made optional once the data shows they rarely change anything.

Working together

I design and build multi-agent systems end to end, from the agent roles and handoffs to the production software, logging and local models around them. See VidForge and my other case studies.

If you have a workflow you think needs several agents, send me a short brief. I'll tell you honestly whether it does, or whether a simpler design will do the job.