Agent Architecture
How my agents actually work together

I organised the agents like a small company, because the hard problems in multi-agent work are the same ones a company has: who owns what, who may decide, and how work passes between people without getting lost. Every agent has a written charter that says what it is for and, just as important, what it is not for and who owns that instead.

There are three groups. The product organization is twelve departments of Skyrule. Four cross-cutting agents check, route and block across all of them. The rest serve my other personal projects and the machine itself. All 29 run on the same rails.

All unattended work goes through one job queue that I wrote in Python. A job is a small JSON object: an id, a lane, a command or a prompt, and optional gates. The queue is the only thing that launches work. Agents, the town and I can add jobs, but only the queue decides when they start.

Why one queue: with several schedulers, nobody can answer "why did this run now?" With one, every launch is a single logged decision with its evidence. The cost is that the queue becomes critical infrastructure, so I kept it small, file-based and covered by its own tests.

Jobs run in lanes, and each lane has its own concurrency limit. A lane is a statement about a shared resource: what would break, or cost too much, if too many of these ran at once.

Agents do not talk to each other in real time. They coordinate through files with owners: dependencies in the queue, dated handoff documents, a shared memory, and a decision log. I chose this over agent-to-agent chat because every exchange then leaves a record that a new session, a reviewer or I can read later, and nothing depends on two agents being awake at once.

A machine-readable ownership map assigns every folder and code path to one of the 15 product agents: each department's own folder plus 62 owned code paths, with 5 paths reserved for me. A sync script writes each agent's scope from that map, so the scopes are never edited by hand and a check mode fails if any agent file drifts.

Enforcement comes in three layers, because no single layer can see everything.

One detail I am glad I caught: on Windows, a hook that blocks by exit code does not reliably block, because PowerShell collapses every non-zero exit to 1, which the agent runtime treats as a warning. So the scope guard always exits 0 and denies through an explicit "deny" decision in its output. A control that silently becomes a warning is worse than no control, because it looks like one.

Two kinds of model do the work. Claude runs the agents that plan, engineer, review and write code. Open models on my own GPU do the bulk work: extraction, generation, grading, local agents and every draft of public-facing text. The split follows three things: usage (Claude is budgeted; the GPU is mine to fill), privacy (sensitive user data never goes to a cloud model), and fit (bulk work wants throughput, review wants judgment).

Of the 754 Claude-lane jobs since Oct 2, 458 ran on Sonnet, 270 on Opus and 26 on Fable. The model is chosen per job, so a routine status check does not cost what a hard design review costs.

Each department sits on a named local "seat", and the mapping lives in one editable file; no department is tied to a model in code. I picked the seats by measuring on this card, not by reputation.

The workstation has one RTX 5090 with 32 GB. A vLLM server reserves 94% of the card when it starts and keeps it. A 32B Qwen seat on Ollama needs about 20 GB. They cannot share, and when they collide Ollama does not fail loudly: it spills to the CPU and runs about ten times slower. So I made the GPU a lease with exactly one holder: idle, a vLLM server (three profiles: an 8B generation model, Qwen3 32B AWQ, and Qwen3 VL 32B for vision and extraction), or the local Ollama seats.

The rule the lease will not break: a switch never interrupts running GPU work. If a generation, extraction or grading process is live, a switch away from it is refused and asked again on the next tick. Generation can resume, but a half-written output file is not worth a race for the sake of a schedule. After every switch the result is checked against the card's actual memory use and the servers' health, not assumed. A separate restore script hands the card back to the vision model after a temporary switch, with the same refusal when work is running.

I also wrote time windows for who should hold the card at each hour of the day (generation overnight, local agents in the daytime). That scheduler runs as a dry run today: it logs what it would do, and switching happens by explicit request from a job or from me.

Autonomy is only useful if I can trust what comes back without re-checking all of it. So review is split across independent checks, each with a narrow job, and I sit at the end only for the decisions that are truly mine.

On Oct 18 I wrote down which decisions actually need me, so that agents stop asking about things they can safely decide. Only Level 3 comes to me. For example, if the editorial check finds a problem that an existing rule already prohibits, engineering fixes it and quality verifies it; I am not asked again.

The release boundary. No agent publishes. The shell guard denies every git push, nothing generated is published automatically, and a full generation run or a public deployment needs my explicit go. This page is an example: an agent prepared it on a branch, and it goes live only after I approve it.

With 29 agents and hundreds of jobs, logs stopped being enough to answer "who is working, who is stuck, and who is waiting on me?" Agent City is my answer: a town in the browser, first in 2D and now in 3D with three.js, where each agent has a character and a building. It is a projection of the system above. It owns no scheduling and runs no work.

A deterministic projection. Each agent carries a few authoritative fields (availability, runtime, run state, attention, next scheduled start), and one pure function turns them into the one state the town shows, with a fixed precedence:

Two rules keep it honest. An agent whose runtime is unreachable shows as offline, never as asleep or done. An agent with nothing scheduled is idle, not sleeping. Tests check every conflicting pair, so "working" means a job is actually running, never that an agent looks busy.

For the product side of this system, see the Skyrule case study. For how the town itself was made, see Agent City.