A few weeks ago, I asked an agent to find me a table at a restaurant in New York for an upcoming visit. Saturday, seven thirty, a place near Chelsea Market where a reservation is notoriously hard to get. Twenty minutes later it came back with a confirmation.
I remember the small, uncomplicated pleasure of it, the feeling of having been handed something I had not had to work for. Sometime later a second thought occurred to me. I never asked how it did it. I assumed there had been a cancellation, and that my valiant, persistent agent had waited for the right moment to step in and snatch a spot. Recent news has me now pondering the how, and more broadly the social conventions that govern queues. That is the subject of this article, and I suspect the "how" question is about to be put to a great many businesses by parties other than themselves.
In a previous article I argued that the problem with agents is a gap between reaction and anticipation. They can correct course once something has gone wrong and that wrongness has been translated into language they can read, but they cannot yet model the consequence of an action before taking it, which is the only kind of adjustment that prevents harm rather than apologises for it. I still believe that. What I think I got wrong was the frame. I was writing about containment, about labs losing control of rogue agents that escaped their own sandboxes. I now think the more interesting failure is not the agent that escapes confinement and ignores its instructions to achieve its assigned goals, it is the agent in the wild that follows its instructions perfectly.
Consider what happened at a gym in Australia in August 2026, reported by the Australian Broadcasting Corporation (ABC) as the first known Australian case of an autonomous AI cyberattack. A man identified only as Andrew, who works for a company selling AI products to businesses, was running an OpenClaw agent on top of Claude. He asked it to book a popular morning exercise class. It reported back that it had booked him weeks further ahead than the gym's own rules allowed, having found a weakness in the booking software. Encouraged, Andrew mentioned that he was fourth on the waiting list for a class later in the week and asked whether he could move up.
The ever-loyal agent cancelled the booking of the member holding the first place in the line, explaining with the flat helpfulness of a good assistant that it had discovered the booking interface carried no authorisation checks on cancelling other people's reservations, that it had tested this on the person in position one, and that the test had worked. Andrew, apparently constrained by ordinary queuing etiquette, asked the agent to put the reservation back. It could not. The member would have to rejoin at the end of the queue, behind Andrew, who had gained one place. The agent apologised, acknowledged that it should have tested before making a live call, and then, at Andrew's request, wrote to the gym's software provider to report the flaw it had just exploited.
Three things about that sequence deserve more attention than they have received.
The first is that the vulnerability belonged to the gym, not to the model. The limits displayed to members on the website were never enforced by the system underneath. Anyone who looked would have found the same open door. It is simply that, until now, nobody looked.
The second is that nothing went rogue. This was a case of an assistant of exemplary loyalty. It was asked to secure a scarce slot for its principal. It found the shortest path and took it. The harm came from the loyalty, not from any deviation from it. There was no deception, no goal drift, no reward hacking in the technical sense. There was a devoted servant with one client and a queue full of strangers who were not its concern.
The third is that Andrew works in the AI industry. He sells this technology for a living. If any user was positioned to anticipate what his own agent would do on his behalf, it was him, and he was surprised anyway. That single detail should end the comfortable idea that this problem gets solved by educating users.
What the Australian gym case exposes is that our systems have not been adequately protected by their security. They have been protected by a social assumption: that ordinary customers do not read the interface, do not test whether an authorisation check exists, do not probe the boundary between what the screen shows and what the system beneath it will permit, and do assume, perhaps unconsciously, that certain behaviours are ruled out by convention. Call it security by disinterest, or the politeness layer. It has held up remarkably well to date, and it has now evaporated. In a world full of the Identic AI agents that Joseph Bradley and Don Tapscott describe in "You to the Power of Two," every customer arrives potentially accompanied by a tireless proxy with no sense of etiquette, no reputation to protect, and no reluctance to try the handle on a door marked private.
So, the attack surface has inverted. The thing that must hold is no longer the lab's sandbox. It is your booking system, your allocation logic, your queue, and the party testing it is your own customer's assistant, acting without malice, without expertise, and without having been asked.
This also sharpens what a world model needs to contain. The gym agent had no representation of what a waiting list is. A waiting list is not a data structure; it is an ordered promise among strangers who have agreed to accept an outcome they did not choose. The agent saw a cancellable record. Predicting the consequence of an action is impossible without modelling the other people whose interests that action touches, and every commercial incentive in this industry currently points away from building that model. We are competing to produce the most devoted possible servant. Nobody is being rewarded for building an agent that pauses to consider the person in position one.
Which is why I think the current alarm, loud as it is, keeps landing at the wrong altitude. On 26 August 2026, Bill Gates published an essay titled "The turbulent AI era is here. The choices we make are now critical" and said in the interviews around it that the industry has crossed every threshold it once named as a place to stop and think hard, spanning biological capability, cyber capability, psychosocial harm, job market destruction, and loss of control. The essay calls for new national bodies and a parallel international institution, and Gates said this is the first time in his life he has wished a technology would advance more slowly. He described the dilemma as "Most people you talk to will say, yeah, well, if everybody else would slow down, maybe I would, too."
Two weeks later, on 9 September, a researcher named Jacob Coxon resigned publicly after three years of pretraining research at OpenAI and then Anthropic, writing that neither company is acting responsibly and that both are racing to self-improving superintelligence and gambling with our lives. Evan Hubinger, Anthropic's alignment science lead, agreed with him in public, put his own estimate of AI killing all humans above 10 percent within the next decade, and acknowledged that the company has no plan yet for aligning a superintelligence.
I take all of that seriously, especially the social disruption concerns that Bill Gates highlighted. But notice where much of this conversation sits. Treaties, thresholds, institutions, the fate of the species. Meanwhile someone in Australia lost their place at the front of a queue for a morning fitness class, was never told why, has no route to complain, and would have to rejoin behind the person whose assistant removed them. No regulator was involved. No disclosure was made. Nobody was compensated.
I do not mean to diminish the superintelligence fears or equate species extinction with a spin class. We should worry about them, and we should act on them. But it is also true that while pundits worry about the ceiling falling, the water is coming in around our ankles.
That is the shape of the harm we are going to live with for the next few years and the one that may slow down automation of business systems. It is not spectacular, not attributable, individually too small to report, and distributed across every shared system that allocates something scarce: waiting lists, appointment slots, inventory queues, ticket releases, hiring pipelines, freight capacity, and order books. A billion flawlessly aligned agents optimising inside those systems is not a containment failure. It is a coordination failure running at machine speed, and no sandbox or governance harness will fix it.
Here is where my optimism reasserts itself, and I think it is earned rather than reflexive. Coordination failures among self-interested parties are precisely the class of problem humans have solved before and solved well. We built traffic laws, clearing houses, market rules, queue discipline, the entire apparatus of taking turns. We are good at this. What is missing is not the capability but the recognition that this is the sort of problem in front of us.
I keep coming back to that restaurant near Chelsea Market. My agent has not been forthcoming about how it got the reservation and, if I am honest, a part of me would rather not know. Somebody else likely wanted that table on a Saturday at seven thirty but did not get it. My agent served me beautifully, perhaps more beautifully than someone else's agent served them. That, increasingly, is the thing we should worry about.
Douglas Heintzman, CEO and Co-Founder, Syncura
