Human-in-the-loop AI coding: the five moments that actually need you
Most agent notifications are noise. These are the five events worth an interruption, and how to design your workflow around them.
Every tool that watches an AI agent faces the same temptation: report everything. Every file touched, every test started, every token of progress. It feels thorough. It is actually how you teach a developer to swipe away your notifications forever.
The value of a supervision layer is inversely proportional to how often it interrupts you. After watching how agent sessions actually go wrong, we landed on five events that deserve your attention. Everything else stays silent.
The five moments
- The agent has a question. The single highest-value interruption. An unanswered clarifying question stalls the entire run; a one-tap answer keeps it moving. This is the core of what researchers and practitioners call human-in-the-loop design, and it is where Anthropic's agent-building guidance puts the checkpoint too: at decisions, not at activity.
- Tests failed. Not every red test needs you, agents retry on their own. But a run that ends with failures, or loops on the same failure, is a decision point: intervene, redirect, or stop.
- A deploy or build failed. Anything that touches the outside world gets a notification, no exceptions.
- An approval is required. Destructive or irreversible operations, deleting files, force-pushing, overwriting config, should pause and wait for an explicit yes from a human.
- The run finished. The happy path. You get a summary, files changed, tests passed, and you decide whether to review now or later.
Why silence is a feature
Interruption research has been consistent for decades: a context switch costs far more than the seconds it takes, and knowledge workers reassemble their focus slowly. Birgitta Böckeler's exploring generative AI series on martinfowler.com makes a related point about working with coding assistants: the human role shifts from typing to judging, and judgment needs uninterrupted room.
A supervision tool that pings you eleven times per run does not give you that room. One that pings you only when your judgment is genuinely needed turns the agent into what it should have been all along: a teammate who taps you on the shoulder rarely, and only for good reasons.
Designing your own workflow around this
- Route the five events to a device you actually carry. A notification on a desktop you left is a notification that does not exist.
- Give every notification an action. A question should carry its answer buttons; a failure should carry pause and retry. A notification you can only read, not act on, just relocates your anxiety.
- Compress everything else into a timeline you can skim afterwards. The quiet activity should be reviewable, never pushed.
This is exactly the model Clarsky implements: five notifications, a glanceable live state, and one-tap answers from your phone. The interactive demo shows the whole loop, including the one interruption that matters.