Quality and predictability are the two most popular topics in every LLM dev shop right now. Everyone wants autonomous agents, loops, and recursive PR processes, but the results are not matching the marketing hype.
My answer is turn guards: small automations that insert rules, force interruptions, and audit output at every agent turn. Before I explain my implementation, here's the background.
We really have two core issues.
First, model behavior is just not reliable. This can be due to core model limitations, network fragility, server infrastructure stress, or black box logic in the harness or LLM itself. I can certainly say that in my own work the unpredictability from the models is unnerving. It doesn't matter if it's Opus, Fable, GPTx, Kimi 3, they all speculate a scary percentage of the time, at times up to 15% of responses. Often the agent has not read a file that it references in its response. It will make a definitive statement about a file it hasn't read. It will also speculate about technical details for a language, package, or API when it has not read the specific documentation prior to its response.
The second issue is context scope. Front-loading agent rules in AGENTS/CLAUDE.md or growing a "living" memory system both completely fail as context grows. Memory and recall degrade terribly. As our task duration goes longer, or subagents are involved, there is a decent chance our core rules will be completely ignored at critical times. This causes thrashing and endless rule additions to our rules/memory system. We try to find 5 different ways to tell the agent to follow a rule it cannot follow when memory and recall become almost impossible in a huge context window.
If I can't get my agents to perform without grossly speculating or ignoring rules across 5-8 response turns, or during longer running subagent tasks, how could it be a great idea to unleash these models on an autonomous quest across a huge monolithic feature or project?
As I work more on my LLM processes, I keep coming back to my Lean roots. Each misdirection or speculation from my agent plants a seed of potential product or code debt. I want that misdirection to be remediated as close to its source as possible.
We could argue that we'll just clean things up in subsequent audit passes, PR reviews, or other recursive audits. Should a robust LLM process rely on recursive pruning of errors and misdirection by subagents until we have something that's ultimately shippable and matches our real specs? This sounds horribly wasteful, both in terms of token consumption and product direction.
Code and context scope are cheap, to be sure. But attention, focus, and product direction are very expensive. Figuring out how the agent became so misdirected is very difficult after 50 subagent turns and 5k lines of code are already written.
"Just wait until Fable/Sol and that won't be necessary anymore." None of the new models have solved the core reliability problems yet. Investment in automation is the practical solution, and one that will continue to bear fruit even as models improve. Our core reliability problems may be permanent. Obviously core LLM abilities keep improving, but variability has been a constant so far.
I have the same questions every day:
- How can I get my agent to communicate more concisely with me?
- How can I reliably enforce my coding and response rules?
- How can I catch my agent speculating?
- How can my agent continue to operate predictably as the context window grows?
I'm using the Pi agent via the oh-my-pi package. I use Pi because it is the minimal harness. I don't want to wonder what's being injected with all of my prompts or subagent work. Verbosity only confuses LLMs. If our specs and our process are tight, our harness shouldn't need to do much funny business.
Pi supports plugins and event hooks similar to Claude Code, Codex, or OpenCode harnesses. These let you automate activity at many different parts of the LLM agent lifecycle like startup, stop, message, file read/write/edit, and more.
I wrote my own plugin for Pi/oh-my-pi https://github.com/wbelk/pi-behavior-control. It does the following:
- Response reminder before every message that I send. This 13-line payload (225 words) has "Communication Style" and "Operating Rules."
- After each agent response, audit the response for speculation using a subagent with a desired model.
- Check that the agent has read files before edit/write. This is enforced with a read log that uses a sliding window covering the last 4 turns.
- After each file edit/write, include a reminder to review the changes according to the coding rules.
- Slash command for adversarial subagent review. This will loop up to 6 rounds until resolution, and the session agent may ask 1 clarifying question per finding. I've found auditing code via the main session agent to be insufficient. It seems that the main session agent will always require recursive passes of audits — no matter how many times I ask for the main session agent to audit changes, it will always find new issues. This slash command saves time and headaches when auditing delivered work to increase quality before my review.
Something similar can be replicated on most harnesses. It has dramatically improved my confidence as a daily driver for AI coding.
Turn guards can help any developer or team. Gathering context inline at each turn, or conditionally at specified turns, is the way to keep things Lean and improve predictability. As the saying goes, an ounce of prevention is worth a pound of cure.