2026.11.16 Benjamin Powell

How I Build Software With a Team of Agents

A plain-English tour of the system I build software with. One agent writes the code, others review and test it, automatic checks guard secrets and documentation, and three decisions always stay with me.

I build software by directing coding agents. An agent is a program that reads files, runs commands and edits code on its own, with an AI model doing the thinking. One agent working alone is fast, and it has three problems. It forgets everything when the session ends. It marks its own homework. And it drifts away from what the project was meant to be.

So I built a system around the agent that fixes those three things. It gives the agent a memory, a second opinion, checks it cannot talk its way past, and a scoreboard. This page explains that system without the jargon. The full technical version, with every part named, is here: Inside the harness: how A://IDE, Pi and Herdr work together.

The four pieces

Two of the pieces are other people’s tools. Two are mine.

  • Pi runs one agent. It connects the AI model to the tools the agent needs and keeps a full transcript of everything said and done. I did not build Pi.
  • Herdr keeps many agents running side by side, each in its own pane, and lets them send each other messages. If I close the app window the agents carry on. I did not build Herdr either.
  • The harness is mine. It is a set of rules, tools and checks that loads into every agent I start. It turns one agent into something closer to a small development team.
  • A://IDE is mine too. It is the desktop app I look at all day: every project in one window, a chat with each agent, the files, a to-do list, and a view of what the checks are finding.
The layers: you, A://IDE, Herdr, Pi agents and the harnessA stack. You sit at the top and keep three decisions. Below you is A://IDE, the desktop app. It talks to Herdr, which hosts one pane per agent. Each pane runs a Pi agent. Every Pi agent loads the harness. To the side: project memory, the telemetry store and outside services.YouSay what to build. Approve the plan. Merge. Deploy to production. Everything elseruns by itself.A://IDE: the desktop appThe cockpit. Projects, chat, terminal, editor, to-dos, code graph, schema,wireframes, review and settings. It starts agents and shows everything they do.local socketHerdr: the agent multiplexerOne workspace per project, one pane per agent. Agents keep running when the appwindow closes, and any agent can message any other by name.Pi agent: builderThe agent you chatwith. Its own copy ofthe code, its owntranscript.Pi agent: fixerAny number of agentsper project, each withits own model.Background agentsPlan reviewer, critic,tester. They reportunder the chat thatstarted them.every agent loadsThe harness: one set of rules for every agentContext: what the agent knows before it acts.Workflow: onboard, plan, offboard.Quality: gates, reviewers, tests, secret and dependency scans.Tools: code graph, library docs, browser, shell.Measure and learn: telemetry, daily review, autopilot.Project memorydocs: the what and the whycode graph: how code connectsschema: the shape of the datagit: history and hand-offswireframes: screen designssprint record: tasks and verdictsTelemetry storeEvery session measured: tool use,timings, token counts. Never thecontent.Outside servicesModel providers, error tracking,task server, library docs, websearch, CI.you decidean agent at worksoftware and stored memory
Figure 1. The layers. I sit at the top with three decisions. Everything under that line runs by itself.

The team

The agent that writes the code is never the one that judges it. That one rule shapes the whole team.

Does the work
The builder
The agent I chat with. It plans, writes code, writes documentation and saves its work on its own private copy of the project.
Second opinion
The plan reviewer
A different AI model reads the plan before any code is written. Is each piece needed? Does it already exist? Could it destroy data?
Second opinion
The tester
Another model writes the tests first. They fail until the work is really done. The tester is only allowed to touch test files.
Second opinion
The critic
Reviews each finished task, then the whole job at the end. It can read everything and change nothing.
Cannot be argued with
The gatekeeper
Plain code, no AI. It scans for leaked passwords, unsafe code, vulnerable libraries and tests that were quietly switched off.
Final say
Me
I decide what gets built, what gets merged and what goes live.

How a piece of work goes

  1. I describe what I want.
  2. The builder writes a plan: what done looks like, how each part will be tested, and a short numbered list of tasks.
  3. The plan reviewer checks the plan. The builder then stops and waits for me.
  4. I say yes. Nothing is written before that.
  5. For each task, the tester writes failing tests, the builder makes them pass, and the critic reviews that one task.
  6. At the end the gatekeeper runs every scan. If anything is red, the work is blocked and no AI gets a vote.
  7. I get one summary card: what was asked, which tests prove it, what the reviewers found, how risky the change is, and what the checks would not have caught. Then I decide.
The workflow from start to finish, with who decides at each stepEighteen steps in four phases. Start: open the project, create an agent, onboard. Plan: the agent writes a plan, a second model reviews it, you accept it. Build, repeated per task: tests first, read gate, implement, task review, commit, write gate. Finish: offboard, merge gate, optional review of the diff, deploy, sentinel, cleanup. Each step is coloured by who decides: you, code, a model, or the agent doing the work.you decidecode decidesa model judgesthe agent worksthe appSTART1 Open the projectThe app gives it a workspaceand remembers the folder.2 Create an agentPick a name and a model. Itgets its own pane andcredentials.3 OnboardSync with the remote, read theproject docs, take a privatecopy of the code.PLAN4 Write the planAcceptance criteria, a test foreach one, small numbered tasks.5 Plan reviewA second model asks: needed?already built? destructive?tested?6 You acceptThe agent stops and waits. Yourreply is the go-ahead.BUILDper task7 Tests firstA separate model writes teststhat fail until the work isdone.8 Read gateNo edit until the docs, codegraph and schema for that areaare read.9 ImplementThe agent writes the code,checking callers and currentlibrary docs.10 Task reviewA critic reviews that one task.Fix now or log it as debt.11 CommitEvery commit is scanned forsecrets and for tampered tests.12 Write gateDocs updated and committed, orthe run cannot end.FINISH13 OffboardFull scan, then a final review,then one evidence card.14 Merge gatePark it, send it to dev, ormerge. Always your call.15 Review the diffOptional. Read the changes andsend line notes back.16 DeployOnly when you say so: preview,smoke test, confirm,production.17 SentinelNext session: compare errorsbefore and after. Offer arollback.18 CleanupMerged, clean work copies areremoved.next task: the build loop repeats until every task is reviewedSide flows: a production error becomes a failing test, then the same loop. A to-do card can start its own agent.
Figure 2. The whole workflow. Red steps are mine, blue steps are decided by code, yellow steps are judged by a second model, black steps are the agent working.

The checks and balances

These are the rules I trust most, because none of them depend on an agent behaving well.

  • The writer never reviews its own work. Reviews are done by a different model, usually from a different company, so the two are less likely to share the same blind spots.
  • Reviewers cannot write. They can read and report. They cannot fix things quietly.
  • The tester only writes tests. If it changes anything else, the change is undone automatically.
  • Tests have to bite. The system removes the new code and runs the tests again. If they still pass, they prove nothing, and the task is marked not done.
  • Secrets are scanned on every save. Each commit is checked for passwords and keys before it is recorded. A second, deeper scan runs before anything can be merged.
  • Read before you edit. An agent cannot change code in an area until it has read the documentation for that area.
  • Document before you finish. An agent cannot end its run with code changed and the documentation untouched.
  • Destructive changes wait for me. Deleting a table or wiping data needs my confirmation.

How the project remembers

Agents start every session with no memory, so the project carries the memory for them. Each layer has one job.

  • Documentation holds intent: what each part is for and why it was built that way.
  • The code graph holds mechanics: which piece of code calls which, and what breaks if one changes.
  • The schema file holds the shape of the data.
  • Wireframes hold what the screens should look like. I sketch them in A://IDE and the agent edits the same file.
  • Git holds history, plus a hand-off note at the end of each session saying what was done and what comes next.

At the start of a session the agent reads all of this before it touches anything. At the end it has to update whatever it changed. That is how a project stays understandable after hundreds of sessions.

What stays with me

Three decisions never leave my hands:

  1. What to build. I approve every plan.
  2. What gets merged. Nothing reaches the main branch of a project without my go-ahead.
  3. What goes live. Production deploys happen only when I say so, and the system keeps the rollback command ready.

It measures itself

Every session is recorded as numbers: which tools were used, how long things took, how much the model read. The content of the work is never stored in that record. Once a day a script turns those numbers into a scorecard, compares it with targets, and files a to-do list of what to improve. Small tuning changes are tried automatically and reversed a week later if the numbers got worse.

Where it falls short

The scorecard is honest, and so I will be too.

  • It costs more than I want. A session costs about three and a half times my target, mostly because sessions run long.
  • Reviews miss things. About four in ten of the bug fixes I have traced were for bugs that slipped past a review.
  • The rituals get skipped. The start-of-session routine runs in just over half of sessions and the end-of-session routine in under four in ten. My target for both is eight in ten.
  • I still drive each step. The system tracks the plan, the tests and the reviews, but nothing yet runs a whole job from start to finish without me nudging it along.

The technical version lists all ten weaknesses with the numbers behind them.

The models

The builder currently runs on Claude Opus 5.5 from Anthropic. The reviewers run on GLM-5.3 from Z.ai, with Kimi from Moonshot as the stand-in when GLM is unavailable. I still use GLM-5.3 as a builder as well, and I am about to run the two side by side on the same work to see what difference the model makes once the checks are the same.

If you want the story of how I got here, from pasting ChatGPT answers into an editor to this, that is in Coding with AI: from copy-paste to custom harness.

This case study was written with the assistance of AI.