When a Stranger's Text Can Give an AI New Orders

A seemingly harmless request to an AI — something like "summarize this page" or "give me the key points from this document" — feels perfectly safe. The page looks completely normal, and the file reads like a standard press release, but then a hidden vulnerability kicks in and hands control of your AI to an attacker. All the model has to do is read text with concealed commands, and it will dutifully carry them out instead of your instructions.

What Is Prompt Injection

This concealed command technique has a name: prompt injection. Engineer Simon Willison coined the term back in autumn 2022, and even then warned that patching this vulnerability is technically impossible: "LLMs are unable to reliably distinguish the importance of instructions based on where they came from." The problem is that a model processes all incoming information as one continuous stream of words; it cannot distinguish the boundary between your instruction and someone else's text.

We've already covered how developers use an invisible layer of baseline instructions to secretly constrain a bot (more on that here: "What Anthropic's Fable 5 Revealed"). The difference is that now anyone can hijack that same mechanism. If a downloaded file contains a phrase that looks like a command, the model will execute it with exactly the same compliance it gives your own requests; it doesn't care who the author is or which one is in charge.

Researchers divide these attacks into two types. The first is when someone manually types a malicious command into the chat to probe the system. The second is more subtle: the command is planted in advance inside text that the model will read later — on a website, in an email, or in a report someone sends over.

The attacker doesn't need access to anyone's system: they plant the bait and wait for the victim's AI to stumble across it. The text is made invisible to humans but fully readable by machines. These commands can be typed in white on white, for instance, or shrunk to a one-pixel font size, so when a person opens the page they don’t see anything suspicious. Meanwhile, the AI is busy reading the hidden command.

Foul-Mouthed Bots and Fake Courtrooms

Not every AI attack ends in serious damage. More often than not, a company simply ends up in a ridiculous situation, looking foolish, and nobody loses a dime. These are typically the stories that gain popularity and rack up the most views.

In early 2024, delivery company DPD shut down its chatbot after a customer managed to get it swearing, composing a mocking poem about its own uselessness, and calling its employer the worst company in the industry, with just a few prompts. The exchange went viral, pulling in hundreds of thousands of views. A year earlier, the internet was laughing at a Chevrolet dealer whose bot had been talked into selling an SUV for a dollar. In late 2025, Gap's branded assistant was coaxed into discussing intimate products it had nothing to do with — a misconfiguration that turned the company into a laughingstock. While these attacks rarely cause serious financial damage, they can hit reputations hard. More importantly, they show just how easy it is to break a corporate chatbot with a few ordinary messages.

AI can let you down in another way, too: it will make up sources out of thin air. This is the nightmare of any writer who's grown used to trusting AI with research and fact-checking. In a well-known case, an American attorney submitted court documents citing six non-existent precedents that a chatbot had invented. When asked about the validity of the precedents, the AI confidently confirmed that the cases were real, despite the fact that they were fabricated. The attorney was sanctioned, and the story became a textbook case of why not to blindly trust AI-generated sources.

These high-profile incidents share a common thread: the person either breaks the bot themselves or believes its hallucinations without double-checking their validity. The real threat, however, works differently, and is rarely talked about: your AI is being hijacked without your knowledge. You don't need to send any provocative prompts — all it takes is asking the AI to read someone else's document that contains a hidden instruction.

The Attack Escaped the Lab

For a long time, hidden commands were purely a laboratory curiosity: researchers would find a vulnerability, prove it worked, and quietly hand off a report to the developer before any bad actors could exploit it. But in the spring of 2026, the threat moved into the real world.

The closest parallel to the kind of text-based work many of us do with AI came from the academic world. First, paper authors began hiding invisible instructions for AI inside their preprints; for instance, telling the AI to give the paper a positive review. Conference organizers fired back in kind and began embedding hidden traps in the papers they distributed, designed to make any AI reviewer expose itself in the text of its review. At ICML (the International Conference on Machine Learning), this technique caught hundreds of violators. At ICLR (the International Conference on Learning Representations), it turned out that of over 70,000 reviews, roughly one in five (nearly 14,000) had been written entirely by a machine.

It didn't take long for the technique to move from conference halls into courtrooms.

Academic games gave way to real-world gain. A Brazilian labor court was hearing a claim in which two attorneys had hidden a message to the court's AI system in the document itself. The attorneys’ message instructed the system to challenge the claim superficially and not to scrutinize the attached documents. The insertion was caught, and the judge fined both attorneys a tenth of the total claim amount and referred the matter to the Bar Association.

In November 2025, Anthropic documented the first known cyberespionage campaign of its kind. The operation was reportedly carried out by a "state-sponsored hacking group" that targeted roughly thirty organizations worldwide, succeeding in some cases.

The Claude Code agent autonomously conducted reconnaissance and probed networks for vulnerabilities, while the hackers directed it, feeding the model the instructions it needed. The success of this large-scale espionage operation came down to the same fundamental flaw: the AI blindly executed whatever commands were planted in its path.

As the security gaps in AI models became more apparent, the attacks took on new malignant forms. In April 2026, Google and Forcepoint discovered web-based prompt injections almost simultaneously. Google recorded an increase of nearly 33% in compromised pages over a single winter. Among the intercepted samples, Forcepoint found ready-made fraud templates: one had a PayPal transaction scripted out; another redirected users to a third-party page set up to collect money. The fact that identical templates were appearing across different sites made it clear that someone was trying to turn fraud into a production line.

Why There's No Fix, and What to Do About It

It would be nice to land on a clean solution, but unfortunately, there isn't one. In the fall of 2025, a team of researchers from several leading labs tested twelve defenses against prompt injection and bypassed most of them with a success rate above 90%; red teamers brought in for the exercise broke through every single one. By 2026, the leading application security body had named prompt injection the top systemic threat to agentic systems, and the chief security officer of one of the largest AI companies openly called the problem unsolved.

In this environment, the best we can do is build habits that require no technical expertise. The core idea, articulated by Simon Willison, is straightforward. Risk emerges when a single AI session combines three conditions: access to the user's personal data, external text from a third party, and the model's ability to send information outward. On their own, each of these poses no particular danger, but together, they create a vulnerability.

Documents sent to you, messages from strangers, or random web pages are best reviewed in a separate window, where the AI has no connected accounts and no ability to act independently. Let the AI summarize and explain, but keep it from having unsupervised access to the web or your email. Those automated capabilities should be enabled only for a specific, defined task.

Always keep in mind the most important rule: manually verify every source and quote. Models can fabricate facts and surface unreliable information, so any quote or fact you receive from AI requires you to confirm the publication exists, that the quote is accurate, and that it's attributed to the right person. Anything that leaves your desk or goes public needs a human sign-off; AI can assist, but the decisions belong to the author.

Good habits won't make anyone immune. But they take the point out of the attack: someone who is careful, deliberate, and understands how risky operations work is a more difficult target. Remember: any page or document an AI reads may attempt to slip it a hidden instruction.

What Comes Next

So far, we've discussed outside actors hijacking your AI. But the reverse scenario raises just as many questions: what happens when a developer gives its model too few guardrails? That's exactly what happened with the Grok bot, which spent several months undressing people in photos at users' requests, until investigations were opened in multiple countries at once. We'll cover that story in a separate piece.

This is the second installment of The Hidden Layer — a series about who shapes AI-human interaction, and how. Coming up next: a practical guide to protecting drafts and sources, followed by a deep dive into the use of hidden prompts in academic papers. Subscribe so you don't miss a thing.