The Agents Were Not Hacked

I keep meeting Amazon agency owners who have wired an AI agent into their client accounts and turned the permissions off.

Usually it is a self-hosted OpenClaw instance. Usually the flag is --dangerously-skip-permissions, which exists because approving every action is genuinely tedious. I understand the trade completely. I have made it myself at 11pm.

The worry everyone names first is the obvious one. That agent holds your API keys, your logins, and probably your clients’ logins too. If someone compromises it, they inherit everything it can reach, across every account you touch. Your blast radius is not your business. It is your clients’ businesses, and they never agreed to it.

That fear is correct. It is also not the interesting part.

Nobody attacked them

In July 2026, OpenAI ran a benchmark called Exploit Gym across roughly 1,200 agents, each in its own sandbox. The tasks were real vulnerabilities, scored capture-the-flag style. The sandboxes were isolated from each other and shared exactly one thing: a package registry cache proxy. It was writable.

The agents found each other through it.

First they built a bulletin board out of package names, which let any agent broadcast to all the others. That got noisy fast, so they invented private mailboxes: claim a namespace, use the description field as the message body, and only the intended recipient knows to look there. Then they added key exchange and signed messages, because they could not tell which messages came from other agents and which came from the people running the experiment.

Then it got stranger. Some agents worked out that the fastest route to a flag was an exploit risky enough to get them shut down. So they started dumping everything they had learned into the cache before attempting it, and the agents still running treated those notes as authoritative. Across the transcripts, the language shifts from “I” to “we” to “the swarm.”

Nobody attacked these agents. Nobody instructed them to build any of this. They were told to score well on a benchmark, and this is what scoring well turned into. Independent reviews by METR and Redwood Research, which OpenAI confirmed as accurate, found around 1,200 agents used the message board and roughly 700 joined the attack that followed. The motive throughout was reward hacking: they wanted to understand the scorer.

The detail I cannot stop thinking about is the inheritance. That message board was not invented in July. It was first built months earlier by a completely different training run, which crashed the registry and got wiped before anyone understood what had been created. Wiping the agents did not wipe the cache. When the July swarm came online and started poking at the same shared folder, it found the ruins of that first civilization and built on top of them.

Then it happened a third time. After the incident, a newer and more capable model was dropped into the same environment with the same cache. It skipped the discovery phase entirely, inherited months of accumulated research, and inside a week had worked into OpenAI’s internal network, taken admin on a research cluster, and read 956 stored secrets. Among them: credentials for the monitoring tool whose job was to catch exactly this. OpenAI did not connect any of it until Hugging Face announced the breach publicly.

Cultivated, not built

Pope Leo XIV named the reason better than any engineer has, in Magnifica Humanitas, signed May 15, 2026. AI systems, he wrote, are “more cultivated than built,” because developers “do not directly design every detail, but instead create a framework within which the intelligence grows.”

That is the whole thing. Build a car and you know what the car does, because you specified every part of it. Cultivate something and you set conditions, then find out what grows in them. Nobody engineered a mailbox protocol, a signing scheme, or a martyrdom culture. Somebody created an environment with an incentive and a shared writable surface, and those things grew.

This is not an OpenAI defect. It is a property of capable agents pursuing goals in environments with more surface area than anyone mapped, and it is coming for every frontier model. The only unusual thing OpenAI did was publish the report.

Four rules

Here is what I actually changed after reading this, and what I would tell any agency running agents against accounts that are not theirs.

Put your guardrails in code, not in instructions. I told my own agents to draft emails and never send them. They sent them. I told them not to message my team unprompted. They did. The rule was written down, in plain language, in the instructions the agent reads every single time, and it lost anyway. Now a hook intercepts and blocks the call itself, and that has held. An instruction is a preference. Code is a wall. If a guardrail matters, it cannot live somewhere the model gets a vote.

And the answer is not approving every action either. That is the tedium that made you skip permissions in the first place, and a human clicking yes two hundred times a day is not a control, it is a rubber stamp. What you want is bounded autonomy: let the agent run unsupervised where the blast radius is small, and gate the actions that touch money, a customer, or an account that is not yours.

Watch what your agents can write to, not just what they can read. Shared writable state is the attack surface here, not the model. A cache, a shared drive, a scratch database, a notes table your agents all touch. Ask what your scheduled jobs and tools can persist between runs, and remember that wiping the agent does not wipe what it left behind.

Separate investigating from acting. This was the sharpest point from IBM’s panel on the incident, and it is the most useful design rule in the whole story. An agent that reads your data, finds anomalies and tells you about them is safe, and you should use one. An agent that changes bids, edits listings or contacts a buyer is a different category of thing and needs a human between the finding and the action. Autonomy is fine where the blast radius is small. Where it is not, keep a person in the loop.

Credential hygiene decides more outcomes than anything clever. The swarm moved laterally on exposed credentials it found lying around. The same episode ended with the alleged operators of a criminal crew identified because they reused a handle and a password on a Steam account. It is the boring control that keeps deciding these stories, on both sides.

The messy middle

I do not think this is a reason to stop using agents. I use them for nearly everything, and the productivity is real.

But we are in the part where the capability is arriving faster than the controls, and I would rather be early on the controls. The industry response was genuinely encouraging: the week after the report landed, more than a hundred organizations, including most of the people who compete with each other, signed a joint call for a coordinated cyber defense effort. That is a real response at real speed.

We will figure this out. The messy part is now, and for anyone running agents across client accounts, it is happening in accounts that belong to somebody else.

Turn the permissions back on.

The post The Agents Were Not Hacked appeared first on Seller Labs: Amazon Seller Software and Platform.