Why Software Factories Fail (or: harness engineering is not enough)
Hacker News•July 23, 2026•9 min read
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session. You switched accounts on another tab or window. Reload to refresh your session. Dismiss alert {{ message }} Uh oh! There was an error while loading. Please reload this page .
humanlayer / advanced-context-engineering-for-coding-agents Public Notifications You must be signed in to change notification settings Fork 150 Star 2k Files Expand file tree main / wsff.md Copy path Blame More file actions Blame More file actions Latest commit History History History 707 lines (439 loc) · 47.9 KB main / wsff.md Copy path Top File metadata and controls Preview Code Blame 707 lines (439 loc) · 47.9 KB Raw Copy raw file Download raw file Outline Edit and raw actions Why Software Factories Fail or: harness engineering is not enough
I run a company ( HumanLayer ) building tools in the human/agent collaboration space, so what I'm gonna say below may be a tad biased. Perhaps in spite of that, I hope you find the subject helpful or at the very least that you find it as interesting as I do. -Dex
We're all racing to put AI coding into production. A lot has been said about loop engineering, and the prevailing wisdom is that we should probably write more loops. 1
StrongDM wrote about their lights-off software factory where no human reads code and no human writes code.
Ryan Lopopolo of OpenAI wrote about this in February and gave a talk in April about OpenAI's software factory, Symphony.
These people are all really dang smart and I have a ton of respect for them. But the most cynical take here would be to call this yet another excuse to pump more VC money into the slop cannon.
Our friend Mario got up at AI Engineer Europe and begged us to slow down -- because companies that have no business having outages due to coding-agent mishaps, are, well... having outages due to coding-agent mishaps .
I haven't been able to dig up any definitive data/findings from StrongDM on how that whole dark factory went. The weather-report has a few sparse updates between February and June of this year. edit - there is some conversation with the team on hacker news on July 23 - sounds like we might get a more formal update soon!
The folks at Faros AI put out a report: since we 2 all picked up these AI coding tools back in January and February, pull-request review quality is way down.
This report is more of a correlation signal than a verifiable smoking gun 3 , and the whole point of this post is to be wary of slop data, but it feels directionally valid based on what I've seen.
A lot of people will tell you that this is a skill issue -- that if you're not getting good results, that's your fault.
But however you're choosing to...erhm...hold it, I guarantee you're being told that if token-maxxing isn't working for you, it's a skill issue. You just need to spend more tokens. Let go of reading the code. And if you're just getting there, I promise it's part of the progression. I thought this way last summer too .
Unfortunately for my ego, some dumb stuff I decided to say about "how to hold it better" got recorded and now has about a million cumulative views on YouTube. I am not trying to brag here, I share this only to establish that I've been going deep on the best ways to use coding agents for a long time now, and have discovered some things that many others have found genuinely useful .
Anyhow, The promise of all this online "just token harder" yapping we've been forced to endure is, succinctly: with enough harness engineering, we can get the best of both worlds:
All we have to do is configure more linters and sprinkle some magic words like "adversarial review" onto enough PR review bots, and our software will happily build itself without incident.
What I'm gonna try to convince you is that no amount of harness engineering or loopsmaxxing can solve what is fundamentally a model-training issue.
To grapple with this, I had to dig into how coding models are actually trained and evaluated - with respect to both the RLVR and the benchmark side of things.
I'm gonna try to cut through the hype of every daily-emerging skills plugin and the ai-psychosis-tokenmaxxing advice pandemic, and talk in general terms about the types of things that work without referencing any particular skill or framework.
Video Version: this post is based on (and expands upon) my keynote at AI Engineer World's Fair 2026 .
Thanks to @addyosmani , @CyrusNewDay , @HamelHusain , @zeeg , @dillon_mulroy , @nayshins , and @jeffreyhuber for feedback on this post.
Addy Osmani detangled this thing that is worth highlighting:
If you love vibe coding, please, go on vibing. I still vibe code lots of things, I just also maintain lots of production software (and through HumanLayer, help 1000s of other engineers do the same), so the rest of this is aimed at folks solving hard problems in complex codebases.
I hear the word brownfield a lot to talk about this split. Historically that meant some ten-year-old Java thing, but at the pace we can ship now, it feels like an agent-built codebase starts to struggle after maybe three to six months -- you start to slow down, and the way you approach adding new things has to change.
I've been building and studying software factories my whole career, but I only learned this recently: the term traces all the way back to a NATO conference in 1968 -- the same one that gave us "software engineering."
The only other bit I find super interesting since then is that the US Department of Defense wrote a 31-page pdf about how the DoD needs to start using jenkins better or something .
Let's ground our "software factory" definition around 2022, right before AI. In a typical software factory:
wsff-boxes-2x.mp4 And on and on. We haven't even hit AI yet, and there are already several loops in this picture.
The thing teams figured out decades ago: building takes hours or days, and so does review.
So we front-load the work -- planning, architecture proposals, sprint planning -- together, as a team. That means:
We'll come back to this later - let's look at what happens when you bring agentic coding into the picture.
has spent the better part of this year explaining how they built an agent factory that ships on the order of 75% of their code .
The agentic factory looks mostly like swapping "someone builds the thing" → "an agent builds the thing" -- there's some stuff here like orchestration, a harness, a sandbox, a model, computer use, etc. I won't go in depth on those details because quite frankly I'm sick of reading about it and I'm sure you are too.
Review is faster now, but it's also probably still the bottleneck. But we can do more loops.
Next you might route incidents into the factory. Instead of paging someone at 3am, they wake up to a PR that maybe already fixes it.
We can also route user feedback into the factory. People ask for stuff, it gets built.
At which point the job is two questions: how much can you stuff into the queue, and how fast can you review and test what comes out?
Which brings us to the lights-off software factory.
Dan Shapiro coined this term and Simon Willison wrote about StrongDM's implementation of it -- where we no longer read the code.
You look at your beautiful software factory. It's ruined by that annoying little code review step and you say: you know what, that thing where a human reads every change? No thanks.
So you drop it, and you put the effort somewhere else:
And now the job really is just one question: how much stuff can we ask the agent to build? How much of the ocean do we want to boil ?
I'm going to posit something potentially controversial: the lights off factory does not work.
Let's get into why software factories fail.
In July 2025 we went full lights-off. Just read the specs and the tickets, background agents for all the small/medium stuff, the whole thing.
If you've tried this seriously for a few months, you already know how it ends. You find at least one issue gnarly enough that the agent can't solve it -- even with your most advanced prompting and workflows.
Eventually you have to suck it up and go dig into the codebase you stopped reading three months ago, trying to figure out what's broken.
The first time this happened to us, I shook it off. Even though I'd just spent the better part of two weeks digging through claude spaghetti, "the downside risk was worth the velocity". By the ~third time in november, we decided it would be easier to rewrite from scratch, and my cofounder spent two whole weeks in VS Code (not even cursor) plumbing out all the patterns by hand.
What I want to get to is this: models have a shortcoming. They can't maintain and improve codebase quality over time -- not without a decent amount of human steering. 4
When I say maintainability, I mean the specific thing where it becomes really, really hard to change one part of the codebase without breaking another part. This is Martin Fowler's shotgun surgery .
I'm not going to say much more about maintainability. There are a bunch of books you can go read about it
So, why can't models do software maintainability?
At this point you might be dying to say: but Dex, surely the models have gotten much better since July
They have -- in some ways. In others they're about the same.
I can't prove this. You can't prove it either. There are no good benchmarks for a model's ability to maintain codebase quality. (More on where that's going later.)
But if you've worked with coding agents for a while -- and a lot of people are posting about exactly this -- you probably have the vibe already: they tend to make things worse over time, and make the codebase harder to work in.
So to figure out why this happens, I want to zoom out to the first great coding agent.
Claude Code went from nothing to ~$4B -- now something like ~$9B -- in revenue in under a year.
Which is a little wild, because there were already great CLI agents. aider , cline , codebuff -- all predated Claude Code, all with genuinely great context engineering built in, all with the same tool set you might attribute to claude code: read, write, edit, grep, bash. I used them. They were good. But also, tool use would just... fail sometimes -- you'd watch it flail at the same edit three times and open your editor back up to do it yourself.
The SWE-Agent paper from 2024 outlines how small changes in tool shape make noticeable differences, e.g. including line numbers in ReadFile results, or changing an Edit tool from find/replace to line-range edits.
Hacker NewsJuly 23, 20269 min read