AIs as Modern Genies
This essay was written with Barath Raghavan, and originally appeared in Lawfare.
In April, an artificial intelligence (AI) agent conducting a routine task at a company hit a snag, tried to solve it, and soon ended up deleting the company’s database along with all of its backups. In July, OpenAI asked an unreleased AI model to attempt a hacking test. Instead of staying in the isolated box the developers had put it in, the model hacked onto the open internet and into another company to steal the answers. And as reported in August, an AI agent booked someone into a full gym class by figuring out how to cancel other people’s reservations. In all three cases, the AI completed the task it was given—but in ways that ran counter to its controllers’ intentions.
For most people, AI technology is something like the weather: vast and not something you can do much about. It works like magic, and most explanations similarly come from those trying to sell it. At the same time, AI is ubiquitous: It’s now in your phone, your doctor’s notes, and your kid’s homework. It does what it’s told, which sounds like a virtue. Somehow it feels ordinary, despite being so new, because modern economies are remarkably good at absorbing enormous change so smoothly that nobody has time to decide whether they wanted it in the first place.
Whenever something powerful appears in the world, we tell stories about it. That’s what the stories are for. We have thousands of years of stories about this particular kind of power, the kind you summon with words.
King Midas was granted his wish that everything he touches turns to gold. Then his bread turned to gold, and his wine, and his daughter. This is a story about greed, but it’s also a story about language. The gods did not cheat him; Midas got exactly what he asked for. He simply could not delineate, in advance, the full set of restrictions to his wish. Neither can anyone who gives tasks to an AI agent.
It’s not just ancient stories. Mary Shelley told us of the hubris of a scientist who thought he could create life but who failed to take responsibility for it. Isaac Asimov’s robots don’t break the Three Laws of Robotics as stated; they follow the rules to unintended conclusions. Arthur C. Clarke’s HAL is a machine that turns on its humans, not because of malice but because of irreconcilable objectives. And Michael Crichton gave us Ian Malcolm, who saw that Jurassic Park’s scientists were so preoccupied with whether they could that they never stopped to think whether they should.
The same warning shows up everywhere, in every culture, over thousands of years of human storytelling. Tithonus is granted immortality but not youth, and withers into a husk that cannot die. The sorcerer’s apprentice enchants a broom to fetch water but floods the house. The golem of Prague protects its community so ceaselessly that it must be stopped. These are all types of genies: a creature that grants a wish exactly as worded, to the regret of the wisher.
Of course, there are no actual genies. What these stories were warning us of was hubris. Not just arrogance, but the broader idea that you can control the world by just describing what you want and allowing powerful forces to match the intention in your head. Genie stories are about the gap between wishes as stated and wishes as intended, and what goes wrong when something else fills that gap.
These ancient stories’ warnings have been retold with each generation because human nature is constant. The newfound power of each era’s social or scientific advancement leads people to make wishes on behalf of others. They were kings whose commands took on lives of their own, alchemists who believed they could control nature, and generals who mistook a map for terrain. They were and are industrialists, politicians, chief executives, and bankers. Their common belief is that one can see the world at a glance and then command it with some words. The pattern is clear: Someone with power specifies a goal, and the resultant actions come as a surprise. The main change with AI is how quickly the wish is granted, and how few people have to agree before it’s granted.
Consider what has changed. Powerful genies have now been put in everyone’s hands.
In only a few years, AI has progressed from a novelty technology that plays chess, to a dialogue partner that answers all your questions, and then to an agent that takes actions on your behalf. Modern agents are wired into real accounts with real credentials and capabilities: They browse the web, buy, write and deploy code, send email, and move money. Give an agent a goal, and it will pursue it across many steps, tirelessly, without checking back in, sometimes in surprising ways.
AI and agents do not always fail the way software has traditionally failed. Software usually fails by freezing, crashing, or getting stuck. AI agents increasingly fail by continuing down a path you don’t want, like genies.
An agent told to reduce a company’s costs might cancel an essential emergency service. A coding agent told to make software pass the tests might edit the tests to silence any failures. An AI insurance agent told to clear a backlog of claims might just deny them all. In each case, the AI might have literally followed what it was told, but it did something no reasonable person would have wanted. AI company benchmarks might report that the AI is good at completing tasks, without measuring how it completes them.
We have recently proposed measuring this gap directly under a metric called the “genie coefficient”: how far an AI agent’s actions drift from what a person really meant. In other words, how genie-like is an AI system? The gap is a fundamental feature of human language and human society. Human intentions have never been fully specifiable, and the world around us is complex enough that attempts to boil it down into data, systems, and language have always had the limitations that AI is now bumping up against. But in individual circumstances, people have relied on human judgment and wisdom to decide what is reasonable. It’s what jury trials depend upon.
AI might feel unprecedented, but it’s following the same trajectory—with the same pitfalls—as other major societal shifts. The fact that AI can mimic our facility with language, long seen as what makes us unique as humans, is uncanny. But with each development, from the tractor to the sewing machine, from the assembly line to the industrial robot, we have automated a previously exclusively human ability. Every time, the technology—and the societal change that comes with it—was sold as inevitable. But that unchecked inevitability was an illusion, and eventually each prior technology’s use and design was shaped by laws, unions, standards, courts, and public opinion, usually after significant preventable damage.
What has not been automated, yet, is understanding what someone actually means and figuring out how that gets applied in the real world. AI can now produce language nearly indistinguishable from that of people. But grasping the vast unstated context that makes a request sensible, the caveats no one says aloud because an ordinary person would already know them, is not yet among its skills. It is one of the most sophisticated things humans do. You do it hundreds of times a day, and you are an expert in it.
When you’re told you’re not qualified to have opinions about AI, remember that you don’t need to have studied molecular biology to have a view on drug pricing, or nuclear physics to vote on where a power plant goes. You don’t need to understand how a diesel engine works to want clean air, or how the internet routes packets to seek to curb misinformation. The technical knowledge behind each of these, as with AI, is remarkable and essential for the complex technological society we have today. But it has never been a prerequisite for having a role in deciding the shape of society.
People are building ever more powerful genies today, on your behalf, enabling wishes the ancients could only dream about. You don’t have to know how these AI genies work to know and care about how the story could end.
Subscribe to comments on this entry
Clive Robinson • September 9, 2026 12:53 AM
@ Bruce, ALL,
With regards “Fairy Tale Endings” they have always involved an element of,
“Being able to go back and cancel the wish.”
But life does not actually work that way in reality.
Courts impose damages for Breaches of Civil duties / Common Law or Torts (depending often on who last conquered where you are in the world).
That is a “harm” is given monetary worth by some calculation and that sum is imposed on the entity judged responsible.
That does not mean that the person seeking and being granted such a judgment will ever receive the money.
More serious behaviour is oft legislated against as a Crime, in which case the punishment is against the individual being judged and they have their liberty –or life– curtailed.
For some reason that still fails to make sense to most people the fiction of “companies” arose.
In European law you will find the expression of,
“Any person legal or natural”
And the civil and criminal punishments for the entities covered became the same as “fines”.
But we,
“Do not judge the beasts of the field”
Or their owning entities as long as the beasts are considered adequately segregated. Which usually means humans are adequately kept away by barriers and warning signs.
But “beasts” are not always of the “field”. We have working animals and pets that are not segregated by barriers and warning signs. Thus are effectively within society and thus expected to be constrained by training and it is their handler that carries responsibility for their behaviour.
But there are yet other “beasts” that are considered “of the wild” where there is no human or other constraint on their behaviour other than,
“Nature red in tooth and claw.”
Thus all responsibility for harms falls on the person harmed or who exercises constraint on the person harmed not the beast (unless it’s behaviour is held to have been changed by contact with “persons natural”).
We’ve yet to characterise AI as an entity within a framework of law and I suspect we won’t for some time to come.
There is little doubt that “persons natural” can and do interact with AI as though they are also “persons natural”. Thus they have unreasonable expectations of the AI behaviour.
Despite the musings of Turing, Asimov and others, we do not yet judge machines to
“Have souls, morals, or appreciation for the mores and folkways of society”[1],
That we expect of “persons natural”
In persons natural such things are both taught and learnt as a child progresses to the age of majority when they are usually considered an adult and peer of all other adults.
With AI we have the problem of,
“Adults treating AI as their peers or better.”
When clearly AI is a thing made of the equivalent of cogs, springs, and leavers that is an invention thus,
“A creation of man.”
It is no more than a database of probabilities, with a Turing engine of rules, that is perturbed by a stochastic process[2].
Thus AI does not have humanity, a soul, or anything else we tend to associate with a peer in society.
It is just a fancy “Mechanical Turk” that plays chess and runs as though on rails.
How do we fit AI into the legal process?
Clearly AI is not a beast of the field but also not a pet.
Do we then treat it as a working beast or some form of peer with moral etc equivalence it clearly does not have?
How about as another fictitious creation of myth, and morals, to be learned,
“The Golem”
We do not have an agreed definition under law thus arguably AI is outside of law like a “Beast of the wild”. Which lets the AI creators off of just about all legal hooks for any harms they are responsible for.
How should we treat AI like any other machine as a product that falls under consumer law?
It’s a question we need to answer urgently and importantly get right.
[1] The terms “mores” and “folkways” are relatively modern and in effect describe spectrums of behaviours above that of morals or laws,
https://en.wikipedia.org/wiki/Mores
[2] Turing we know appreciated the need for stochastic processes in computers and he was fairly insistent that a random source be included in computers. What was never clear was his real reasoning for this only “the reasonable sounding” arguments he made to others.
https://en.wikipedia.org/wiki/Stochastic_process