Automation vs Co-Driving

The eleventh debate I keep having with myself: when an agent can do the work, what am I still there to do?

An agent is filling out a form for me. It reaches the obvious button, pauses, takes another screenshot, and thinks about it. I can feel my hand going for the mouse.

The click is not risky. It is just so obvious that I can do it faster than I can explain it, wait for an interpretation, and check the result.

Two fields later, I want the opposite. There is a box asking for a careful explanation. I do not want to type it. The agent has the context. It can turn a messy situation into a clearer answer than I will write in the moment.

Same form. Different actions. That is the bit most agent demos miss.

We keep calling this automation versus manual work, or computer use versus browser automation, or a human in the loop. Each phrase assumes that one side owns the task. Real work does not split that cleanly.

The useful unit of delegation is the action.

Full automation is great when the work is bounded and repeatable. Let the agent run the process, check the state, retry the boring failures, and leave evidence behind. Having a person watch every click is not careful engineering. It wastes a person's attention for little return.

Live supervision has the opposite appeal. A visible cursor makes the work feel legible. You can see where the agent is going. You can interrupt it when something looks wrong.

But watching a cursor is a poor audit trail. You might miss the one important step. You will not remember a twenty-step flow later. And you are still left asking what actually changed in the system.

Then there is the human side. The person sees the agent pause on something trivial and takes over. The agent sees an unfamiliar screen, loses its place, and the handoff turns into recovery work.

Headless automation fails in its own way. It is quick and reliable until the human needs to step in. At that point, the person becomes an emergency brake. The workflow was built for the agent to run alone, not for two parties to take turns.

Neither approach knows how to share work.

The screenshots versus selectors debate matters because it gives us two different ways to act.

A computer-use agent works through the same interface I use. It sees pixels, moves a cursor, scrolls, and types. That is useful for native apps, remote desktops, canvas-heavy tools, old vendor portals, and anything with no usable API. A screenshot loop is not a toy when the screen is the only way in.

A browser often gives an agent another option. It exposes roles, accessible names, page state, and a DOM. Playwright can find the button named "Pay", wait until it is clickable, act, and check the state afterward. It can keep a trace with the step, timing, page state, and network activity.

When that channel exists, use it. It is faster, cheaper to repeat, and easier to diagnose. Do not make a model infer from pixels what the page can already state directly.

But choosing the right automation channel does not answer the harder question. What happens when I want to take one click and hand the rest back?

I used to treat watchability and control as the same thing. They are not.

Watchability means I can understand what happened. A trace, a log, a screenshot at the important moment, and the resulting state can give me that. Playwright's Trace Viewer is useful because it lets me inspect the failed step instead of making me stare at every harmless one in real time.

Interruptibility means I can change what happens next. I can take a click. I can stop before a consequential action. I can rewrite the agent's draft, then hand the work back. The agent can read the new state and continue without acting confused or starting over.

A moving cursor can make a run easier to watch. It does not make that handoff work.

That is why a shared cursor is not automatically collaboration. It can leave both parties guessing whose turn it is.

Co-driving means the workflow treats the human and the agent as different capabilities. It gives each action to the party better placed to do it.

The agent should search, compare documents, draft, fill repetitive fields, and check the result. The person should take actions that are cheaper to do than to explain, depend on private judgment, change authority, or carry a consequence they have not chosen to hand over.

Sometimes I grab the mouse because I do not trust the agent. Sometimes I grab it because narrating one obvious click would take longer than doing it. Those are different problems. We should not treat every human takeover as a failure of trust.

The reverse is also true. Asking a person to type a routine but delicate answer when the agent already has the relevant context is not responsibility. It is just making the person touch the keyboard so they feel involved.

The question is not "Who controls the browser?"

It is "Who should do this action, given the current state, and how does the other side catch up afterward?"

For repeatable browser work, build the fast path on the machine interface. Use the DOM, accessibility tree, APIs, assertions, and traces. Use computer use when the machine channel is missing.

Then make the handoffs part of the product:

I am less interested in an agent that can imitate every mouse movement than in one that can share a task without making me choose between trust and control.

I still want to grab the mouse when a cursor hovers over an obvious button. I probably always will. The answer is not to slow the agent down enough for me to watch it think.

I want to take that click, hand back the form, and let the agent write the part that deserves its attention.

The future is not full automation or permanent supervision. It is a workflow where human and agent can trade actions without either one losing the thread.

Next time: agents increasingly act from rules scattered across repository files, skills, user preferences, and tool policies. When those instructions disagree, which one is actually in charge?