Inside the Harness: How A://IDE, Pi and Herdr Work Together
The technical breakdown of the system I build software with: the agent runtime, the multiplexer, the desktop app and the harness of gates, reviewers, tests and telemetry that sits around every coding agent. Shipped behaviour only, weaknesses included.
This is the technical version. It describes shipped behaviour only; planned work is listed at the end. The numbers are a snapshot from mid November 2026, and I give them as ratios against my own targets.
In plain English
A coding agent on its own is one model in a loop. It reads files, runs commands and edits code. It is fast, but it forgets everything between sessions, grades its own homework and drifts from what the project intended. The system described here adds memory, a second opinion, deterministic checks and measurement around that loop.
The rule that holds it together: code decides whatever code can decide reliably (gates, routing, retries, counting, test runs). Models are used only for judgement (writing code, reviewing plans and diffs, classifying, summarising). I keep three decisions: what to build, what gets merged or pushed to a primary branch, and what gets deployed to production.
If you want that story without the part names, read How I build software with a team of agents. The rest of this page is the reference.
The four parts
| Part | What it is | Its job |
|---|---|---|
| Pi | An open agent runtime (CLI and SDK). Not mine. | Runs one agent: model, tools, session transcript, extensions. |
| Herdr | A terminal multiplexer built for coding agents. Not mine. | Hosts many agents in panes, keeps them alive, lets them message each other. |
| The harness | My own repository of Pi extensions, skills, rules and telemetry. | Turns every Pi agent into a small dev team: plan review, critic, tester, docs gates, quality gates, learning loop. |
| A://IDE | My macOS desktop app (Electron and React). | The cockpit: projects, chats, terminals, editor, knowledge pages, review, settings, and windows onto the harness. |
Who owns what:
- Herdr owns agent lifecycle, agent terminals, persistence (agents outlive the app window) and agent-to-agent messaging.
- Pi owns the agent loop, the model call, the tools and the session transcript: a structured log of every turn, which is the source of truth for the chat.
- The harness owns behaviour: what the agent must read, what it may edit, who reviews it and what gets measured.
- A://IDE owns composition and visibility. It starts agents and shows everything. It never re-implements what Herdr or Pi already do, and its interface never spawns a process directly: every action goes from the UI through a narrow bridge to the main process.
Pi: the agent runtime
- One process is one agent, with a model, a thinking level, built-in tools (read, bash, edit, write, grep, find, ls) and whatever extensions add.
- Every message, tool call, tool result, thinking block and token count is appended to a session file. The chat UI, the telemetry and the critic all read that file.
- Extensions are TypeScript modules that hook the agent’s lifecycle: before the agent starts, on each tool call, on each tool result, before the run settles. They can add tools, add sections to the system prompt, block a tool call or add a message. Everything in the harness is built from them.
- Skills are Markdown playbooks the model loads when a task matches, for example wireframing, UI work or a repository audit.
- Library documentation and web search arrive through two MCP servers (Context7 and Tavily) that are loaded on demand.
- The main agent defaults to Claude Opus 5.5 at a high thinking level. A pinned list of other models can be chosen per agent, and I still run GLM-5.3 as a main agent too. I am about to compare the two on the same work.
Herdr: the agent multiplexer
- Herdr is like tmux, except it understands agents. It runs a server with a local socket and a JSON-lines API.
- Its concepts nest: session, workspaces, tabs, panes. An agent is a recognised program inside a pane.
- A://IDE uses one dedicated session, so the app never touches my own Herdr session. There is one workspace per project and one pane per agent.
- Herdr’s Pi integration reports each agent as working, blocked, idle or done, and reports the path of its live transcript. The app uses that for status dots, toasts and sounds.
- Every pane can prompt any other agent by name. A://IDE detects agent-to-agent traffic and shows it in the chat with the sender.
A://IDE: what the app does
Projects and agents. Open a folder and it becomes a project space that stays open. Each project holds its own agents, each with a name, a model and a thinking level. Creating an agent never shows a blank wait: a six-stage boot sequence plays in the chat. Background jobs (critic, tester, plan review) appear as sub-rows under the chat that caused them and end with it.
Lanes. Every session works in its own git worktree on its own branch, so two agents never share a working tree. Lanes that are merged, clean and unoccupied are swept automatically. Quitting an agent runs the end-of-session routine first.
Chat. Chat bubbles are parsed from the Pi transcript. Terminal text is never scraped for meaning. Thinking and tool progress stream live from a side file. Each element is colour-coded (reply, thinking, read, write, shell, docs, critic, web), failures are always red, and documentation work is classified by path so I can watch the docs discipline happen. The reviewer’s conversation opens in place under the chat header. Every Pi slash command works from the chat, and each turn shows tokens and context share.
Workbench. A terminal that runs local shells at the project root, a file tree, a Monaco editor with tabs scoped per project, and a to-do panel. A to-do card can start an agent that onboards first and then takes the task. To-dos sync both ways with Barrydo, my task server.
Knowledge pages. Four views per project: a graph of the code index, an entity diagram drawn from the schema file, an embedded Excalidraw board over the wireframe files (the agent and I edit the same file, and the page reloads when the agent writes), and a review page that shows a lane’s diff and lets me send line notes back to its agent.
Settings. One status page that answers “is everything working”, a services page for every key or login agents depend on, model and provider pages that edit Pi’s own stores, per-project environment and credentials, switches for harness behaviour, and a read-only window onto the quality loop: live reviews, verdict trend, open findings, hot spots, debt.
Plumbing. Every call from the interface to the main process returns the same result envelope, so error codes survive the bridge. Logging is structured. Releases are signed and notarised, and the pipeline fails closed.
The harness: what loads into every agent
All of this loads into every Pi session on my machine, inside A://IDE or not. There are 22 extensions. They fall into five groups.
Context: what the agent knows before it acts
| Piece | What it does | When |
|---|---|---|
| Operating principles | Smallest complete change, deep modules, docs over comments, structured logs, fail loud, hard constraints. | every prompt |
| Docs briefing | Injects the project’s structural map into the system prompt and adds a registry lookup tool. | session start |
| Context router | A cheap calibrated classifier (Jev) decides per turn which skills load, which tool groups surface and which knowledge cards to inject. Code owns the mechanics, the classifier owns relevance. If it is down, nothing breaks. | every user turn |
| Quality brief | Hot spots, open findings, debt, last verdict, fired rules, memory status. | every prompt |
| Cache probe | Sets the long prompt-cache lifetime and logs cache hits and breaks. Its contract stops any extension from rewriting the system prompt, so caching survives. | every request |
| Headroom | Compresses large structured tool output before it enters context and keeps the original, which can be fetched back by hash. | every tool result |
Workflow: the rituals
- Onboard. Syncs with the remote (fast-forward only, and it stops and asks me if histories diverge), reads the project context within about three minutes, creates or resumes the worktree lane, builds the code index, and installs the commit-time checks.
- Plan mode. A read-only planning mode with a required plan shape. Task markers in the plan are the review points.
- Ask mode. Read, search and list only.
- Offboard. Final review, findings acknowledged, knowledge layer synced, docs and schema updated, a commit with hand-off notes, then the merge gate.
Tools the agent gets
- A persistent shell: one long-lived shell per workspace, so directory, environment and virtualenv carry across commands.
- Code graph tools: callers, call paths, blast radius and which tests a change affects, in one call.
- A script tool that batches many tool calls into one step.
- Context7 for current library documentation. The rule is that integration code is never written from memory. Tavily is the web search fallback.
- Deep reasoning through sub-agents, for hard problems. Only a bounded summary comes back.
- A headed browser with a hand-over to a human for logins and CAPTCHAs.
- Image description for models that cannot see.
- Sentry, with one login per project, and the Barrydo task tools.
- The quality tools themselves: plan review, tester, critic, memory, floor, status, deploy.
Observability: how the system sees itself
- Telemetry writes one event file per session: tool names, argument hashes, sizes, timings, usage. Never payloads.
- Health writes one line per provider state change (ok, warning, quota, login, down) and adds a line to the failing tool result so the agent stops retrying and tells me.
- Two side files per session carry the live thinking stream and the slash-command interface to the app.
- The quality loop keeps run, finding and plan ledgers per project, plus a KPI summary the app reads.
All the agents
“Agent” here means anything that runs a model call or a model loop. The last group is deterministic code, listed so the picture is complete.
Separation of powers: the writer never reviews itself, the reviewers cannot write, the tester can only write tests, the gate is code, and I own merge, push and deploy. Agents cannot fake a gate receipt, because receipts are keyed by the exact git tree they were run against.
Memory: how agents remember across sessions
Agents start every session with no memory. The project carries memory in four layers, each with one job.
| Layer | Holds | Read through | Kept current by |
|---|---|---|---|
| docs | Intent: what, why, contracts. An index, a code map, canonical pages, a context file per capability, decision records. | The injected map, the registry lookup, the read gate | The write gate, offboard |
| code graph | Mechanics: symbols, callers, call paths, blast radius. Never committed. | The code graph tools, the Graph page, the critic’s callers section | Automatic sync after edits |
| schema file | The shape of the data, as DBML. | The read gate for data code, the Schema page | The write gate on data changes |
| git | History and hand-offs: commits carry notes on what the session did and what the next one should do. | The onboard audit | The offboard commit |
Wireframes carry UI intent on top of that, and each accepted plan leaves a sprint record with its criteria, tests, review verdicts and commit per task. A missing layer is treated as a gap to fill. Onboard builds what code can build, and the plan review demands the rest.
The workflow, start to finish
Start. Opening a project creates or reuses its workspace. Creating an agent picks the name, model and thinking level, opens a pane with the project’s environment and credentials, and starts Pi. Onboard then syncs, reads, takes a lane and builds the memory layer. By the time I type, the system prompt already holds the project map, the quality brief and the routed context.
Plan. I describe the task. The agent writes a plan: description, files, what is reused and what is new, data changes, acceptance criteria in given, when, then form, a test for each criterion, and numbered tasks. The plan reviewer answers proceed, revise (two rounds at most) or ask the user. The agent presents the plan and stops. My reply accepts it.
Build, per task. The tester writes failing tests. The read gate blocks the first edit in an area until its docs, code graph and schema have been read. The agent implements. Marking a task done starts a background review of exactly that task’s diff; its card arrives with a later tool result. The agent fixes each finding or defers it, and a deferral adds a row to the debt ledger. The teeth check then proves the tests fail with the change reverted. Each commit runs the floor. The write gate will not let the run finish until the docs are updated and committed.
Finish. Offboard commits the sprint record and runs the full gate. Green means the final critic runs and I get one evidence card. Then the merge gate: park the lane, send it to a dev branch, or merge. I can read the diff on the review page first. Deploys happen only when I say the word, in order: preview, smoke test, a confirmation dialog, production, smoke test again, with the rollback command recorded. In the first session after a production deploy, a sentinel compares errors before and after and offers the rollback if a new class of error appeared.
Every day. In the first session of the day, with no background service: all sessions load into a database, KPIs are rolled up against a baseline, the rules run, the autopilot judges experiments and applies bounded tuning steps (reverted after a week if the KPI got worse), fix commits are traced back to the commits that introduced the bugs, and the resulting backlog is filed for agents to work.
Checks and balances
Every check, who owns it, who decides, and whether it can stop work.
| Check | Owner | Decides | Can it stop work? |
|---|---|---|---|
| Read gate: docs, context, code graph and schema read before the first edit | docs extension | code | Blocks, with a breaker after two blocks |
| Write gate: docs updated and committed before the run ends | docs extension | code | Blocks |
| Plan gate: no code before a reviewed, accepted plan | quality loop | code | Blocks |
| Plan review: needed, already exists, destructive, against decisions, tested | second model | model | Revise, or ask me |
| Tester: failing tests first, test files only | separate model | model | Its tests become protected |
| Teeth: tests must fail with the change reverted | quality loop | code | Task goes back to not done |
| Task review on each finished task | second model | model | A card: fix or defer |
| Data guard: drop, truncate, remove a table | quality loop | code | Blocks until I confirm |
| Hot spots: files with a history of fixes need a declared design intent | quality loop | code | Per project |
| Floor: secret scan, test-tamper check and fast checks on every commit | project kit | code | Blocks the commit |
| Full gate: gitleaks, TruffleHog, Semgrep, Trivy, tamper check, project tests | quality loop | code | Red means no model review and no merge |
| Risk tier: auth, billing, migrations, CI and deploy paths are high | quality loop | code | Sets review depth |
| Final review and evidence card | second model | model | Strong findings, per project |
| Review team (opt-in): lenses and voters | several models | model | Security finding blocks on high risk |
| CI: the shared harness check on every push | GitHub | code | Blocks |
| Dedup guard and ritual reminder | guards | code | Nudges only |
| Health: credits, logins, outages | health extension | code | Agent stops retrying and tells me |
| Merge, push, production deploy | me | human | Only me |
A few details behind the table. The full gate reports new Semgrep findings only, and Trivy is ratcheted against a baseline, so an old problem does not block new work but a new one does. The test-tamper check looks for tests that were deleted, skipped or narrowed. A nudge that points an agent at the code graph when it reaches for a text search exists but is switched off while I measure it.
Telemetry and self-assessment
Pi already writes complete transcripts. The gap was never collection. It was a store, KPIs and experiments. Rules without measurement are wishes.
| KPI | Result | Target | Verdict |
|---|---|---|---|
| Cost per session | about 3.4 times target | 1x | Fail |
| Input tokens, 90th percentile | about 2.4 times target | 1x | Fail |
| Prompt-cache hit ratio | 97.8% | 95% | Pass |
| Duplicate tool calls, all sessions to date | 4.3% of calls, in 54% of sessions | 2%, in 10% | Fail |
| Code graph used first | 7% | 25% | Fail |
| Failed command rate | 4.9% | under 15% | Pass |
| Onboard rate | 55% | 80% | Fail |
| Offboard rate | 38% | 80% | Fail |
Critic precision, meaning findings later judged real, is about 70% across my Node, Python and Elixir projects.
Experiments run with KPI targets registered before they start, and code gives the verdict: keep, revert, no effect or unmeasurable. One example: during its trial window the dedup guard cut the share of sessions with duplicate calls from 53% to 13%. Its target was under 10%, so the recorded verdict was “no effect”, and the change stays.
Harness self-improvement is autonomous. Improvement plans are written by agents, reviewed by a second model and approved by the harness itself. I am told what changed. Only reverting shipped code or crossing one of my hard constraints comes to me.
Weaknesses
Measured, not guessed.
How it was built
A://IDE started on 3 September 2026. The harness grew beside it, first as extensions the app deployed and later as its own repository. In about ten weeks that came to more than 300 commits and 86 recorded decisions for the app.
- 3 September. First day: the Electron shell, the Herdr connection, the project rail, chat parsed from transcripts, terminal, editor and file tree, and a first signed build. The interface was ported from an earlier editor of mine.
- 4 to 5 September. The persistent shell, written to work around a macOS kernel memory leak triggered by constant process spawning. Then a reliability pass and the first notarised release.
- 7 to 8 September. The knowledge layer: the code graph, the schema diagram, and docs re-scoped to carry intent only. To-dos that start agents, then two-way sync with the task server.
- 15 to 18 September. Lanes: every session in its own worktree, with git state on each project card. Then the wireframe page and the uniform result envelope across the bridge.
- 16 to 23 September. Telemetry went live in the harness and the first experiment (the dedup guard) was registered. The context router landed.
- 5 October. Living docs became gates: read before the first edit, update before the run ends.
- 15 to 21 October. The second-model critic and its page in the app, then plan-first working with one bounded review per task.
- 22 October. A prompt-cache fix. Measurement showed about half the main model’s cost was cache writes, most of them re-writing context that was already cached.
- 25 to 27 October. The harness became the product: the gate (secret scanners, static analysis, dependency scan, test-tamper check), CI on every push, the tester, the sprint record and the evidence card. Slash commands in chat and the review page followed.
- 2 November. The autopilot: the harness judges its own experiments and tunes itself.
The app and the harness were built by agents working inside earlier versions of the same system. Several of the checks exist because an earlier version let something through.
Glossary
- Space: a project, which is a Herdr workspace.
- Lane: a git worktree on its own branch, one per session.
- Critic: the second model that reviews.
- Gate: deterministic checks that leave receipts.
- Floor: the commit-time hook.
- Kit: what the harness installs into a project (floor, CI caller). It is versioned.
- Hot spot: a file with a history of fixes.
- Evidence card: the one-screen summary at the merge gate.
- Teeth: proof that the tests fail without the change.
- Headroom: tool-output compression.
- Side file: a file the app tails for live data.
Planned, not shipped
- A sprint orchestrator: one agent runs plan, tester, coder, gate, review and evidence card end to end.
- The review team as the default, once the data says it earns it, and a merge queue that rebases and re-runs the gate.
- Evidence, sentinel and weekly reports rendered as cards in A://IDE.
- A replay gate over the collected escape cases, so a change to the critic’s prompt or model is tested against bugs it previously missed.
- Write scopes per agent role, enforced by code.
- Lint packs beyond Node.
- The plan gate extended to files written through the shell.
This case study was written with the assistance of AI.