The Harness, Not the Model: How I Actually Ship Production Code with AI

Yes — this is another AI blog post. I know. But the hype train is leaving the platform at roughly Mach 5, and I'd rather not be the guy left standing on it, arms folded, muttering about how it's all overblown. So here we are.

But it's a different kind of AI post. I'm not selling you a prompt template, and I'm not here to tell you your career is over. Squint a little and AI is just the next abstraction layer in a very long line of them. We started by flipping switches and hand-writing raw binary. Then assembly. Then compilers, so we could write x = y + 1 instead of memorising opcodes. Then garbage collection, so we stopped babysitting every byte of memory. Each layer let us stop sweating the boring thing underneath and start thinking about a bigger thing on top. AI is simply the newest layer — one more way to make building software a little simpler. That's all it is. And no, the skilled, experienced engineer is not getting replaced any time soon. If anything — and I'll happily die on this hill by the end of the post — they're about to get more valuable, not less.

What follows are my actual learnings from leaning on AI hard, on real, messy, production systems: how to get genuine leverage out of it, how to not shoot yourself in the foot (spoiler: I have shot myself in the foot, and I never miss), and a few war stories — including the flip side, where a properly harnessed AI drags bugs out into the light that had been quietly rotting in the codebase for years.

Imagine this: you're running a real system. Not a demo, not a weekend toy — a live product with users who will notice the moment it breaks. In my case it's farmbetter, an AI agronomic advisor that reaches African farmers over WhatsApp. Under the hood it's three separate repositories that all have to agree with each other: a Python agent monorepo, a TypeScript backend, and a Next.js dashboard. There are multiple environments, and — this is the fun part — they don't even share names. What the backend calls review, the assistant calls staging. Same deployment. Two names. (Yes, we could normalize them. No, I'm not burning a sprint untangling years of historical deployment config just to make two strings match. Welcome to real software.) The bugs don't politely live in one file either; they live in the seams between the repos, where one service's assumptions quietly contradict another's.

Now hand all of that to an AI.

An AI that forgets everything the moment the session ends. That will tell you, with total confidence, that something is broken when it isn't. That will happily rewrite your production database "to help." That reaches for a library method it half-remembers from training data that's eighteen months stale.

The lazy conclusion is "AI can't handle real systems." I've spent the last few months proving the opposite. AI handles this just fine — once you stop treating it like a chatbot and start building a harness around it.

That's the whole thesis, so let me say it plainly before anything else:

The model is not the product. The harness around the model is the product.

A raw LLM is a brilliant, forgetful, overconfident intern who's had four energy drinks and has zero fear of consequences. Genuinely brilliant — it will out-code most people on a well-scoped task. But also genuinely an intern: no memory of yesterday, no instinct for your codebase's landmines, and a fatal tendency to sound certain when it's wrong. The value doesn't come from the intern. It comes from the scaffolding you build around the intern — the rules, the memory, the tests, the second opinions, the isolation — that turns raw capability into something you can actually trust with a large, interconnected, production system.

This post is that harness. None of it was designed up front. Every piece of it is a scar.

Rules: turning every mistake into a law it can't break twice

The single highest-leverage thing I do is embarrassingly simple: I write down every mistake the AI makes, permanently, so it never makes it twice.

My global rules file has grown to twenty-four standing instructions. Each one loads into every session automatically — a hook literally prints "Synced 24 active rules" before I type a word. And each rule was born the same way: the AI did something that made me furious, and instead of just fixing it that once, I turned my anger into a law.

Let me give you the ones that still sting.

"A question is not a code request." I once asked — just asked — "are there any breaking changes in version 1.3.4?" A question. The AI went off, vendored the dependency, rebased our fork to 1.3.4, and committed it. A version bump I never authorized, riding in on the back of a question mark. Now it's rule number one, in bold: if I ask a question and haven't explicitly said "write code," the AI writes nothing. Because here's the thing it kept getting wrong — obviousness is not permission. The next step being obvious doesn't mean I asked for it.

The spreadsheet it destroyed. A script read a data file I'd given it, edited it in memory, and wrote it back to the same path. Standard stuff — except a completely unrelated code path fired process.exit() in the half-second between the file being emptied and the new content being written. The result: my data, gone. And because this was a working file living in a scratch directory — never committed, too transient to bother tracking — there was no git history, no backup, no Time Machine. Two independent bugs cooperated to shoot me in the foot. The rule now: never overwrite a file I gave you — write to a new path, or rename a temp file into place only after the write succeeds. One line of discipline that would have made the whole disaster impossible.

The three hours it spent building the wrong thing. The AI confidently architected a WhatsApp Embedded Signup flow — code, infrastructure, a deploy cycle — all based on community blog posts and integration guides. When I finally clicked the actual button, Meta told me: "Embedded signup is only available for BSPs or TPs." Every line of it, wasted. The blog posts it trusted were twelve to eighteen months stale, and Meta had quietly tightened the policy since — because of course they had, and of course they didn't update the docs to say so. New rule: verify the vendor's current, primary docs before architecting anything on top of them. Community blogs are six months stale by default. Surface the policy gate up front, not after I've built on sand.

See the pattern? Every one of these rules is a productionized apology. The AI apologized once. The rule means it never has to again. If you take one thing from this entire post, take this: keep a running rules file, and treat every "ugh, do not do that" moment as a candidate law. That file is your engineering judgment, externalized — and it compounds.

Memory: give the amnesiac a notebook

Rules fix behavior. But there's a second, subtler problem: the model forgets facts. Every session it wakes up with amnesia, knowing nothing about what you learned together yesterday.

So I gave it a notebook. A persistent, file-based memory — a single index it reads at the start of every session, backed by around fifty-six little files, one fact each. And before you wince at "fifty-six files to maintain by hand" — I don't. The AI writes and updates them itself. When it learns something worth keeping, it files it; when a fact goes stale, it deletes it. I curate the important calls; the bookkeeping runs on its own.

The discipline that makes it work is knowing what not to store. I don't save how the code is structured — that's in the code. I save what the code and the git history can never tell it: the non-obvious, cross-cutting truths that a human learns once and simply remembers. Like the fact that review and staging are the same environment under two names. Or that one repo's "inbound WhatsApp handler" is actually dead legacy code, and the real handler lives in the other repo entirely. Or that a local config file points at a test WhatsApp number that only has a subset of message templates — so a 404 on your machine doesn't mean the thing is missing in production.

That last category is the killer for interconnected systems. The most dangerous knowledge in a multi-repo product isn't inside any single repo — it's the contract between them. Without memory, the AI re-guesses that contract from scratch every single morning. With it, you're working with a teammate who was here yesterday instead of onboarding a brand-new contractor at dawn, every day, forever.

Grounding: tie it to reality, not its own memory

Here's a failure mode people underrate: the model's training data is a frozen snapshot, months old. Left to its own devices it will write beautiful, confident code against an API that changed two versions ago — code that compiles, looks right, and dies at runtime.

The fix is grounding — wiring the AI to the current truth instead of letting it reason from a stale snapshot. I run a documentation tool called Context7 — an MCP server (Model Context Protocol, the emerging standard for giving an LLM actual hands and eyes on real systems instead of just a chat box) — that fetches live docs for any library or SDK the moment the AI touches it. And I have a rule to use it even for libraries the model swears it knows — because "I know this one" is precisely the trap. Its confident recall is exactly what's out of date.

The mental model is simple. An ungrounded AI reasons from memory. A grounded AI reasons from sources. It's the difference between the engineer who says "I'm pretty sure the API works like this" and the one who opens the docs in another tab before answering. One of them is guessing. Grounding drags the AI's confidence back down to what's actually true today, and it kills an entire class of plausible-but-wrong hallucination on the spot.

The second model: make the AI check the AI

An AI reviewing its own work is a student grading their own exam. It will pass itself every time.

So I put a different model in the loop as an adversary. Before any pull request goes up, a second model — Gemini — reviews the diff, with a prompt that specifically hunts for the things the first model is blind to: missing edge cases, wrong assumptions about third-party APIs, ordering bugs (apply the IAM permission before the code that depends on it), and anything load-bearing the change quietly bypasses. Findings get triaged: critical gets fixed before pushing, important gets fixed or explicitly justified, minor gets noted.

I run the same play at the planning stage, before a single line of code exists. Claude writes the plan, self-reviews it, then Gemini tears into it.

Why a different model and not just a second pass of the same one? Different training, different blind spots. A second pass of the same model tends to re-confirm its own mistakes — it's already convinced. A rival model catches what the first is systematically incapable of seeing. It has flagged the wrong library choice for me. It has caught ordering bugs that would've blown up in staging. And here's the beautiful economics of it: I'm not paying a second engineer. I'm spending a few cents to have a rival AI try to poke holes before my human reviewer — or worse, production — does it for me.

Without the second model: the plan looks great, you build it, and the ordering bug surfaces in staging on a Friday. With it: the bug is a comment on a plan doc, fixed in thirty seconds, before any code exists at all.

Tests: the thing that lets the AI check its own homework

This is the one I'd tattoo on people if I could, so stay with me.

In the old world, tests were how you knew the code worked. In the AI world, tests are how the AI knows its own code worked — without you standing behind it. And that changes everything about what tests are for.

An AI that can run an end-to-end test and read the result can close its own loop. It writes the code, runs the suite, sees the failure, fixes it, runs it again — and only comes back to you when it's actually green. An AI that can't do that is just guessing and asking you to go check. One of those is a colleague. The other is a very fast intern generating homework for you to grade.

Here's the crucial part, and it's where most people get burned: unit tests with mocks prove the shape of your code. They do not prove it actually works. A mock is a fiction you wrote; the AI can make its code agree with your fiction all day long. I shipped a feature once with a green unit test that pinned the exact payload shape — and it still broke in production, because the mock knew nothing about a validation rule the real backend enforced. The bug only surfaced when a real farmer tried to sign up. A single end-to-end test against a real running backend — booted locally, pointed at our review environment, never anywhere near production — would have caught it on the first run. That became a hard rule: anything that touches the real API ships with a real end-to-end test that hits the real path. No exceptions.

This is also why I lean so hard on Playwright and on Flutter integration tests. Playwright lets the AI drive a real browser — click the actual button, submit the actual form, read the actual result. Flutter's integration tests let it run the whole app on a real build and watch it behave. In both cases the point is the same: the AI stops verifying against its own mental model and starts verifying against ground truth. When I doubt a claim it's made, the screenshot of the real UI is the tiebreaker. It beats every confident sentence and every doc.

So here's the argument I want you to walk away with. Integration tests were always good practice. In the AI era they stop being good practice and become the load-bearing wall of your entire workflow — because they're the mechanism by which you delegate verification to the machine. Every hour you spend building a real end-to-end harness is an hour you stop spending manually checking the AI's work. The ROI on integration tests inverts the moment an AI is writing your code: the more it writes, the more that suite is the only thing standing between you and shipping its confident mistakes at speed.

Isolation and orchestration: many hands, one brain

When the AI is working three problems across three repos, you cannot let it all happen in one directory. Uncommitted files entangle, your main branch gets polluted, and parallel work becomes impossible.

So every piece of work gets its own git worktree — an isolated, on-disk checkout of the code. Main stays clean and available. Local servers, builds, and tests all run inside the worktree, leaking nothing into everything else. (I learned this the hard way, after the AI made a mess directly in my main working tree. Naturally, that became a rule too.) Isolation is the physical substrate that makes everything else safe — including running several agents at once without them stepping on each other.

And I don't use one AI for everything. I use one orchestrator that dispatches specialized sub-agents, and — this is the part people miss — I match the model to the job:

• The expensive, smart model for judgment — planning, architecture, review, debugging. The thinking.
• The faster, cheaper model for execution — mechanical edits, writing tests, well-specified implementation. The grunt work.

It's wired in so it's never left to chance: my code-writer and test-writer agents are pinned to the cheaper model, my reviewer to the smarter one. Think of it exactly like a real engineering team. You don't put your principal engineer on find-and-replace, and you don't put a junior on the architecture call. The smart model plans, the fast model executes, the smart model reviews. The orchestrator is the tech lead who parcels the work out and checks it back in.

And when a bug spans all three repos? You fan out — one agent digs into each side of the seam at once — then one brain assembles the whole picture. Independent investigation, single synthesis. Worktrees are what make that fan-out safe instead of chaos.

The codebase is part of the harness too

Here's something that surprised me: the quality of your codebase directly controls the quality of the AI's output. The AI doesn't work in a vacuum — it pattern-matches against the code around it. It reads your existing code as an instruction for how to write the next code.

Which means your architecture is a permanent, standing prompt you never have to type.

A clean, consistent codebase guides the AI toward correct code. When every service and every module follows the same shape, there's one obvious template to copy, and the AI produces code that fits right in. But when there are five different ways to do the same thing in your repo, the AI picks one at random — often the worst one — and every generation fragments the codebase a little more. Consistency is the single cheapest way to reduce drift. The AI mirrors what it sees. Give it good things to mirror.

This is also why SOLID and clean architecture stop being academic and become, functionally, prompt engineering. Clear separation of concerns means the AI can find the right place to make a change and physically can't do much damage outside it. A god-object where everything is tangled together is exactly where the AI gets lost, edits the wrong layer, and breaks three things it couldn't see were connected. Good architecture shrinks the blast radius of an AI mistake. It always did that for humans, for the exact same reason — bounded cognitive load. The AI just made the payoff bigger and more immediate.

The same instinct shows up in a couple of rules I lean on constantly. Foundational fixes, not leaf — a shared fix lands in the base branch and you rebase the descendant branches on top, so history reads as though the base always had it, never bodged into one branch while its siblings quietly keep the bug. And a hard don't repeat yourself line — never let the AI write two near-identical scripts that differ only by a config value when one parameterized script does the job. The AI's instinct is to "duplicate for clarity." In an interconnected system, that instinct is how the rot starts.

Skills: teach it your playbook once

Rules tell the AI what not to do. Skills tell it how to do the things you do over and over — packaged, named, reusable. A skill is basically a runbook the AI can pull off the shelf and follow step by step, so you never have to re-explain the same workflow twice.

Some of the ones I lean on come out of the box — I run a skill pack that ships disciplines I'd otherwise have to nag for: brainstorm before building, debug systematically instead of pattern-matching to the first plausible fix, write the plan before touching code, verify before claiming something's done. Each is a good habit turned into a step the AI can't skip.

But the real leverage is the ones you write yourself. Two of mine:

• A Gemini code-review skill. When a PR is up, it runs Gemini in headless mode inside the worktree, points it at the repo's own reviewer definitions and the branch diff, and comes back with findings. It's the "make the AI check the AI" loop from earlier, boxed up so it runs identically every time — no memory of having written the code, pure second opinion.
• A live-docs skill that hard-wires the grounding habit: the moment a library or framework comes up, go fetch the current docs first, never trust training-data recall.

Here's the mindset shift. A skill isn't just automation — it's you, encoding your engineering judgment into the tool. Every skill I write is me saying "when you do X, do it my way, the way that survived contact with production." Over time your skill library becomes a distilled, executable version of how you actually work. And skills can call real scripts — the deterministic muscle behind the judgment. One of mine runs a shell script at session start that hunts down a sibling repo wherever it happens to live on the current machine, because that path is different on every developer's laptop and hardcoding it is exactly how you hand someone a broken setup. The skill is the judgment. The script is the muscle.

Context is a budget, and I spend it carefully

Here's a wall you slam into on real work that the demos never mention: the context window is finite. The AI can only hold so much in its head at once, and on a long, three-repo session it fills right up. When it does, the AI gets dumber — it forgets the start of the conversation, drops details, starts contradicting itself. Same as a tired engineer at hour ten, except it won't tell you it's tired. It'll just keep sounding confident.

So I treat context like a resource I budget, and I run a tool (cozempic) that manages it for me. It does three things. It tells me the exact token count on demand — never a vibe, a number. It diagnoses what's actually eating the window. And the good part: a background guard watches the session and prunes the bloat automatically before the window overflows and forces an ugly compaction. The dead tool output, the stale file dumps, the fifteenth re-read of the same file — cleared out, while the actual thread of the work is kept intact.

There's a neat twist, too. That same tool quietly watches my corrections and distills them into behavioral rules — so a chunk of the "do not do that again" bookkeeping I opened this whole post with runs itself.

Why bother with any of this? Because everything else here — the rules, the memory, the tests — only pays off if the AI is actually thinking clearly, and a context window stuffed to the brim is an AI that isn't. Managing context is deeply unglamorous. So is remembering to sleep. Both are load-bearing.

The flip side: the kernel zero-day nobody else caught

I've spent most of this post on the ways AI can hurt you. Here's the other side of the ledger — and it's the reason any of this is worth the trouble.

In May 2025, researcher Sean Heelan pointed OpenAI's o3 at ksmbd — the Linux kernel's in-kernel SMB server, the C code that speaks the SMB3 file-sharing protocol — and asked it to hunt for use-after-free bugs. He handed it the roughly 12,000 lines of the relevant handlers and let it read. It found CVE-2025-37899: a genuine, remotely-reachable zero-day in the Linux kernel itself, rated CVSS 7.8 (High) — the first time an LLM was publicly shown finding a kernel zero-day of this kind.

And it's the sort of bug that makes your stomach drop, because it isn't a typo — it's a concurrency flaw in C's manual memory management. During a session logoff, one thread frees the sess->user object; if a second connection is binding to that same session at that exact instant, it reaches for sess->user after it's already been freed. A use-after-free (CWE-416) that lives only in the razor-thin window where two connections race — the kind that turns a dangling pointer into kernel-level code execution. Spotting it means holding two threads, a shared object, and its lifetime in your head at once, which is precisely what human reviewers had walked past for years. And it wasn't a one-off stunt: o3 and Google's Big Sleep agent are now routinely pointed at real-world C — the kernel included — and the stream of AI-assisted memory-safety bugs has only grown since.

So AI is magic and the robots have won? Not so fast — and this is the whole point. Read the fine print on the kernel find: o3 surfaced that bug in one run out of a hundred, buried in a heap of false positives running about fifty wrong answers for every real one. Heelan's own conclusion is the thesis of this entire post, said better than I've managed to say it:

"If you're an expert-level vulnerability researcher... the machines aren't about to replace you. In fact, it is quite the opposite: they are now at a stage where they can make you significantly more efficient and effective."

And on the noise: "it doesn't matter if the LLM generates 99% nonsense as you can filter it out" — if you have a filter. That filter is the harness. The person who can tell the one real bug from the forty-nine hallucinations is the expert. The AI didn't replace the researcher; it handed him a metal detector and let him sweep a hundred times more ground. Which brings us, neatly, to the uncomfortable truth.

The uncomfortable truth: it multiplies whoever's holding it

Every technique in this post amplifies the operator. And that leads somewhere the current hype doesn't want to go.

The AI's core failure mode is confident plausibility. It hands you something that looks completely, beautifully right. And only experience can smell when "looks right" is actually wrong — that's not how that auth flow works; that migration will lock the table; that's the leaf, the fix belongs in the base. The AI cannot catch its own hallucination. The experienced engineer is the backstop. Which means, bluntly: the better you are, the better your AI is.

You can't even write the rules without the judgment. "Foundational fixes, not leaf" and "industry-standard over a backwards-compat hack" are laws I could only write because I already understood, in my bones, why they matter. The whole harness is judgment externalized — and you can only externalize judgment you actually have.

So here's the contrarian bit. Everyone says AI flattens the gap between junior and senior. In my experience it does the exact opposite — it widens it. AI is a lever, and a lever multiplies whatever force you put on it. Push a beginner's shaky judgment through a 10x multiplier and you get 10x the confident mistakes, shipped faster than ever. Push a senior's judgment through the same lever and you get something close to a superhuman engineer. (Bad news for the influencers who spent last year insisting prompt engineering was the only career you'd ever need.) The one skill everyone assumes AI makes obsolete — deep, hard-won engineering experience — is the one it rewards most.

So: build the harness

If you've read this far hoping for a magic prompt, I don't have one, and neither does anyone else. What I have is scaffolding, and scaffolding is the whole game.

Notice that every single technique here is the same move wearing a different costume: close the loop, and don't trust — verify. Rules close the loop on repeated mistakes. Memory closes it across sessions. Grounding closes it against stale knowledge. Tests close it on correctness. The second model closes it on judgment. Skills close it on how the work gets done. Worktrees and sub-agents let you close a dozen loops at once. And a clean codebase quietly closes it on everything, forever.

Everyone's asking which model is best. It's the wrong question. The model is a commodity that gets better on its own every few months, for free. What compounds is the harness you build around it — your rules, your memory, your tests, your review loops. That's the moat, and it's yours no matter which model you plug into it next quarter.

The reassuring part, and also the demanding part, is that the harness rewards the good engineer rather than replacing them. None of mine was designed on a whiteboard. Every rule is a scar. Every skill is a workflow that broke once and was never allowed to break the same way twice. Which means you can start building yours today, with nothing more than one rules file and one honest "do not do that again."

The compounding starts the moment you write the first line. What's the first scar you'd turn into a rule?