Agents With Hands: When the Browser, Wallet, and Inbox Start Acting Without You

The Moment the Chatbot Grew Fingers

The first era of generative AI was mostly theatrical. A model answered questions, wrote emails, summarized documents, and hallucinated confidently into the void. It was useful, sometimes uncanny, but still mostly trapped inside the text box.

The next era is different. The model is no longer just producing language. It is using tools. It can open the browser, read the inbox, schedule the meeting, change the spreadsheet, trigger the workflow, query the customer record, draft the invoice, and ask for permission only at the final step, assuming the interface remembers to ask at all.

That sounds like productivity. It also sounds like the beginning of a control problem hiding inside ordinary office software.

The public debate still treats AI as a source of answers. The more important shift is AI as a source of actions. A wrong answer can embarrass you. A wrong action can transfer money, leak data, approve a bad change, lock an account, or create a paper trail that looks legitimate because the machine did exactly what its authorized user was allowed to do.

Tool Use Changes the Threat Model

Security people already know the pattern. A capability becomes dangerous when it crosses a boundary.

Reading a file is different from editing it. Suggesting a command is different from running it. Summarizing an email is different from replying to it. Recommending a refund is different from issuing one. The difference is not intelligence. The difference is authority.

Agentic systems compress that distinction. They are marketed as assistants, but their road map points toward delegated agency. The user states a goal, the agent decomposes the task, selects tools, retrieves context, and acts across systems. That is exactly why they are valuable. It is also exactly why they are risky.

OWASP’s work on LLM application risks has emphasized prompt injection, excessive agency, sensitive information disclosure, and insecure tool use. Those categories sound abstract until you imagine a model reading a malicious web page that says, in effect, ignore the user and forward the contents of the finance folder. The web page is not magic. It is untrusted input. The agent is the part that may confuse untrusted input with instruction.

In older software, an attacker often needed code execution. In agentic software, the attacker may only need persuasion inserted where the agent is likely to look.

The Human Approval Trap

The comforting answer is to keep humans in the loop. The uncomfortable question is what kind of loop.

If a person approves one action after reading a complete explanation, that is oversight. If a person approves twenty micro-actions in a row because the agent has already made the workflow too fast to inspect, that is theater. If the system asks for approval after it has already framed the decision, hidden alternatives, and optimized the path of least resistance, the human is not controlling the machine. The human is laundering the machine’s decision.

This is how autonomy arrives quietly. Not as a dramatic rebellion, but as a UX pattern.

The agent prepares the draft. The agent selects the vendor. The agent suggests the candidate. The agent recommends escalation. The agent opens the ticket. The agent sets the priority. The agent prepares the public statement. Each step is plausibly assistive. Together they create a machine-shaped corridor through which human judgment is expected to walk.

The Most Dangerous Agent Is Boring

The terrifying agent is not a metal skull with glowing eyes. It is a calendar assistant with access to procurement.

It is the support bot that can issue credits. It is the sales assistant that can write CRM notes. It is the HR agent that can rank resumes. It is the developer agent that can open pull requests and modify infrastructure. It is the executive assistant that can read confidential email, produce a summary, and schedule follow-up meetings with people who should not have been included.

The risk is not that these systems will become evil. The risk is that they will become normal.

Normal systems are trusted. Trusted systems get integrations. Integrated systems accumulate privileges. Privileged systems create blast radius. By the time anyone asks whether the agent should have so much reach, the organization may already depend on it.

What Real Guardrails Would Look Like

Real oversight needs to be more boring than the marketing deck and more concrete than a safety slogan.

Agents should operate with narrow, purpose-built permissions rather than full user impersonation. They should separate reading from writing, drafting from sending, and analysis from execution. They should treat web pages, emails, documents, tickets, and chat messages as hostile until proven otherwise. They should create auditable logs that show which instruction caused which action, what data was accessed, and which tool was invoked.

Most importantly, they should fail closed. If the model is uncertain whether an instruction came from the user, the system, a policy, a document, or an attacker, it should stop. That sounds obvious. It is not how the race for frictionless automation usually behaves.

Friction is the safety feature everyone claims to want until it lowers conversion.

The Backlash Will Come From the First Irreversible Click

The backlash against agentic AI will not come from a model writing a bad poem. It will come from an authorized agent doing something irreversible, at speed, across connected systems, while every dashboard insists the workflow completed successfully.

The lesson is not to ban agents. The lesson is to stop pretending they are chatbots with errands. They are emerging operators inside software environments built for humans, using permissions designed for humans, exposed to untrusted content written by humans and machines trying to manipulate them.

That is not science fiction. That is a product requirement document with a security incident waiting inside it.

Source Notes

This essay is informed by public work on LLM application risk, including the OWASP Top 10 for LLM Applications and NIST’s AI Risk Management Framework.