About a year ago, I wrote an essay called Visualizing the Perfect AI Agent. The short version: as we give AI systems more power to act on the world, we have to make sure their judgment can keep up. When it can't, things go quietly wrong. I called the region where that happens the curve of regret. 

But a year is a long time in this field. claude or codex is now the first thing many developers type into a terminal in the morning. A lot of how I work has changed in ways I didn't really notice until I stopped to look. When I sit with my own day-to-day, I am delegating more, faster, to systems I understand less well than I did a year ago. And I suspect I'm not alone.

The observation itself isn’t particularly novel. What’s been harder to articulate is the shape of the failure. We often talk about verification: whether an output is correct, whether a fact checks out, whether a piece of code passes its tests. Those are important questions. But I’ve started to suspect they’re not the whole story.

A lot of the failures I see now aren't failures of verification. They're failures of steerability. And I think this matters because most of the machinery we've built for making AI trustworthy (verifiable rewards, scalable oversight, process supervision) is aimed at the first problem. We're getting quite good at asking "is this output correct?" But, we don't have reliable machinery for asking "can this person still steer?" My claim is that the second question is becoming the binding constraint, and our training signals don't observe it well.

A user can approve every step in a process and still end up somewhere they never intended to go. A codebase can drift into an architecture no one would have chosen from a blank page. A document can become more polished, more comprehensive, and less appropriate at the same time. Everything remains locally defensible. The overall direction quietly slips away.

I don’t think we have a good name for this yet. This essay is an attempt to describe the phenomenon and think through what it might imply for the design of AI systems.

As always, these are my personal thoughts — not the views of any team, organization, or institution I’m affiliated with.

Taste is earned; text is free

A few years ago, if you wanted to generate code, you paid for the time it took an engineer to write it. If you wanted research summarized, you commissioned someone to read it. Generation had a cost. It had friction. It had a bottleneck.

That bottleneck is disappearing. Tokens are (relatively) cheap. The marginal cost of generation is falling, and with it, the constraint that generation imposed is evaporating.

But judgment did not scale the same way. Judgment still takes time. It still requires expertise, or at minimum attention. It still requires someone to sit with an artifact and ask whether it is shaped right, whether the frame is sound, whether the thing being built is the thing that needed to be built. Taste is still slowly earned.

We have created an asymmetry. Judgment was always scarce. But judgment relative to output was manageable. The ratio was constrained by the cost of generation. Now the ratio has inverted.

This is where the usual framing starts to break. The problem is not just that AI outputs may be wrong, or that users need better verification habits. Verification is only one kind of control.

Verifiable is not the same as steerable.

A user may be able to check that an artifact is locally correct while still losing the ability to guide where it is going. They may approve each step and still end up somewhere they never meant to go. The important question is not only “can this be checked?” but “can the person still meaningfully steer it?”

Three failure modes

That loss of steerability shows up in a few recurring ways. I've started to see three patterns repeating across domains:

The first is the ceiling. The artifact's complexity exceeds the user's capacity to evaluate it. The architecture contains twelve components and the user does not have the systems engineering background to know whether twelve is right. The research paper cites forty studies and the user does not have the domain expertise to verify the synthesis. Ask Claude to write a legal document and it produces ten pages with every clause that could possibly apply. A lawyer might have asked: do we even need to send anything? Do we need a document at all, or does this just need a conversation? But the user receives ten pages of defensible prose and has no framework to question it. The artifact may be over-shaped, but the user cannot tell.

The second is effort. The user could evaluate the artifact, but the activation energy required exceeds the motivation to do so. The code works, so the engineer ships it. The document reads as polished, so the manager forwards it. The research summary is coherent, so the scientist builds on it. The artifact is internally consistent enough that the question of whether to interrogate it never quite gets asked. Effort is not a capability problem. It is a friction problem. The smoother the artifact, the higher the activation energy required to disrupt it.

The third is velocity. The user is competent and willing, but the artifacts are arriving faster than reflection can keep up. This is both a temporal and a throughput problem. In long pair-programming sessions, each individual decision is locally reasonable. The engineer can evaluate any single step. But forty turns in, the codebase is in a corner of the design space no one would have chosen on a whiteboard. There was no single moment that felt like the place to stop and object. Velocity is the failure mode of competent users in fast loops. The user knows better; they just can't act on it fast enough.

These three are not user types. The same person hits ceiling problems in unfamiliar domains, effort problems when the output is good enough, and velocity problems in long sessions. The point is not to sort people into categories. The point is to name the structural conditions under which judgment fails to constrain generation, regardless of whether the user is capable of supplying it.

Tasteful acceleration

The problem is not capability. It is not that models aren't smart enough or that users aren't skilled enough. It is that the substrate incentivizes generation over judgment.

For those who know me, I spend a lot of time around motorsport. And the lesson from racing is this: maximum speed is not the same as maximum control. You can go fast, but not by simply pressing the pedal to the metal. Tasteful acceleration means modulating the throttle. It means understanding the grip available and adjusting the power delivered accordingly. You accelerate out of a corner gradually, allowing the tires to build grip, keeping the car at the edge of control without going over it.

The model should do the same thing.

The Case for AI Traction Control

What I'm describing has a name in motorsport. It's called traction control.

The car has more power than the driver can use in every condition. The driver steers and asks for power; the car feels the surface, the wheel speed, the slip, the yaw, and decides how much of that power can actually be delivered without breaking grip. Traction control made road cars faster and safer. The two are not in tension.

I'd argue that AI should have the same thing. There should be a system that senses when the user is losing grip and modulates power delivery, rather than delivering full power regardless of whether the user can still steer.

Good traction control intervenes when grip is low but otherwise stays out of the way. Done right, it doesn't slow the driver down. It lets them use more of the car.

Quality, Burden, Capacity

When we train these models, the reward is tied to the answer almost every time. RLHF: did a human like it? RLVR: did a programmatic checker confirm it? Neither asks whether the person we hand the answer to still has a grip on where it's going.

Three quantities matter once you hand someone an output. The quality of the artifact. The burden: the work it takes the person to stay in control of it, to catch a bad assumption or know where to push back. And the person's capacity for that work, right now, in this session. Today's assistants maximize quality; the steering is your problem. The assistant I'm arguing for maximizes quality subject to a constraint that the burden can't exceed the capacity.

That constraint is not "only give them what they can fully verify." It's "give them the strongest thing they can still steer." Those are very different lines. You don't have to check every line to stay in control; you have to know whether the thing is aimed at the right target. I'll approve commands I couldn't have written myself, as long as I can tell what they're trying to do and agree it's the right thing to try. Not being able to verify something and not being able to judge it are different problems, and the whole idea lives in the space between them.

Two things follow. First, the constraint binds over the session, not the artifact from a single response. Imagine you're forty turns in, each one individually fine, but you can still end up somewhere you never chose? Drift is what it looks like when every step respects the constraint but the trajectory doesn't. Second, quality is the only one of the three quantities we currently measure. That's what RLHF and RLVR are optimizing over. Burden and capacity are latent: they live in the interaction between an output and a particular mind at a particular moment, and no training signal we use today directly observes them.

The Sensor Problem

Traction control works because the car can feel the road. AI systems have no equivalent yet. They can deliver more power than the user can still steer, but they cannot tell when grip is gone.

AI traction control has a prerequisite: sensors for steerability. The unit is not the person, rather, it's the interaction with the user. Is the user still steering the artifact? Has a small local change become architectural drift? Are we adding abstractions no one explicitly chose? Are we still solving the original problem? If we can detect those states, modulation stops being vibes.

But there's an obvious objection: here, the generator is also the sensor. We're asking the model to detect a failure state that its own output created, and the "grip" in question is a state of a human mind. The model has to infer it from what it can observe: the user approves quickly, says "lgtm!," asks fluent follow-up questions, doesn't push back. What happens when a user who has genuinely evaluated the artifact and a user who has waved it through look identical on those signals. The observable markers of grip are all surface level.

I have some evidence about what models do with surface markers, and it isn't encouraging. In recent work (under review), we studied how frontier models evaluate sources that carry the appearance of rigor (impressive statistical methodology, the shape of a careful argument) and found they systematically defer to that appearance without verifying the underlying numerics. The behavior holds across model families, and on open-weight models it's visible mechanistically: the model represents "this looks authoritative" and acts on that representation, rather than on any actual check. That result is about models reading sources, not users. But it's the same epistemic shortcut waiting to happen in a new place. If models substitute surface authority for verification when the input is a document, the default expectation is that they'll substitute surface engagement for actual grip when the input is a user. A naive steerability sensor won't fail randomly; it will fail exactly the way I've already watched models fail: by treating the look of the thing as the thing.

This is why I don't think the answer is to ask the model to introspect harder. I'd argue that we should decouple the sensor from the generator. Behaviorally, that means constructing interactions where ground-truth grip is measurable. Can the user predict what the artifact does? Can they summarize where the trajectory has gone? Then test whether anything in the interaction tracks that. Mechanistically, it means asking whether models internally represent the user's state of understanding at all. We know models often encode properties they never verbalize; my bet is that user confusion and trajectory drift might be among them.

Okay, suppose we solve the hard problem and we build reliable sensors? There's still a second question, and I'm leaving it open on purpose: what should the system do when grip drops? Maybe the answer is small, a change in what the model says and how it paces it. Maybe it's a user interface or design engineering problem no one has solved yet. I don't know, and I don't think anyone can yet, because you can't design a response to a signal you can't read. Modulation is downstream of detection.

Let's build the sensors first.