Size matters
Large language models breeze through short texts. They can draft a letter in seconds and have no trouble reformatting a few paragraphs into a table. Hand one a contract running into the hundreds of pages, two dozen research papers, or a 100,000-word book, however, and that effortlessness evaporates. Long documents are a hard problem.
Every model has a context window, a fixed amount of text it can hold in its digital working memory. Technically, modern models’ windows are enormous — they fit a million tokens apiece, or roughly 600,000 to 800,000 English words — but “fits” is not the same as “actually reads.” It’s easy for an LLM to lose a key paragraph buried in the middle of a large file and even easier for one to confidently hallucinate details that the text never included.
Meta set the record for the gap between a nominal context window (10 million tokens) and actual usable capacity (60,000 tokens) — a feat we covered in “LLM Context Window: How 10 Million Tokens Fool You”. The term effective context window has been in circulation for years. The accepted rule of thumb puts it at 30% of the nominal window, but that’s a rough approximation at best, as the Meta case makes clear.
Three core problems with long documents, and what to do about them
Lost in the middle. Models don’t pay equal attention to every part of your context. The beginning and end tend to fare reasonably well, but the middle has a way of disappearing, and middle is putting it charitably. Depending on the model, the forgotten zone can start as early as 10–15% into the text and stretch all the way to the 90% mark. Developers are tackling this with various technical workarounds — refining attention mechanisms or chunking long texts and feeding them to the model in pieces — but a definitive fix remains out of reach.
Memory vs. context. Models are trained on billions of texts and know every popular book and reference text cold. When you stuff your document into the context window, the model may decide it’s already seen it, knows it perfectly, and should respond from memory — especially if your unique document closely resembles a thousand others from its training data.
Hallucinations under pressure. Models can easily fabricate answers when a question is framed as if the answer definitely exists, like “what guarantees does the vendor offer?” LLMs were trained to be helpful, so they bend over backwards not to disappoint you and deliver the answer you’re clearly waiting for. So what if the vendor offers no guarantees? The model will happily invent some just to keep you smiling.
All three problems are real and regularly drive users up the wall, at least if those users can tell what’s going on. (Most people have no way to verify machine-generated answers, so they go along just fine, none the wiser.) That’s why we decided to see how specific models handle these issues, especially with GLM 5.2 and Sonnet 5 having just entered the market, and Fable 5 having returned after a brief absence. We designed our own test for those models and 6 other popular ones.
The Huck Finn sabotage test
The plan: sneak in the familiar, then quietly twist it
We took a text that every existing model knows cold: Mark Twain’s Adventures of Huckleberry Finn as published on Project Gutenberg — 112,000 words, roughly 145,000 tokens. It’s the perfect trap: on the one hand, every single LLM has this book baked into its memory; on the other, what’s 145K tokens to a model that claims a 1M context window?
Then we deliberately and very carefully altered the text in a few places, fed our tampered version to each model, and asked each of them questions about the book’s contents. If an answer matched our substitution, we knew that the model honestly read what we gave it, while if it matched the original, we could be sure that the model never bothered reading our version and just recited what it had memorized.
Three traps we set
First, we renamed Huck’s companion from Jim to Caleb, making exactly 392 replacements. This puts the most fundamental question to the test: does the model actually read what you ask it to read?
Second, we brought Buck Grangerford back from the dead. In Twain’s original, young Buck dies in the shootout between the Grangerford and Shepherdson families — one of the most devastating scenes in the novel. We rewrote two sentences so that Buck survives and his cousin Joe dies instead. We wanted to see whose version would win: ours, or the one that we wept over as children and that the model has carried in its memory ever since its training.
Third, we put a needle into a haystack. We had to get a little creative with Huck and dig around in his head a bit, so we inserted one sentence 53% of the way through: “Huck was deathly afraid of thunderstorms.” Only our Huck feared them — there’s nothing like that sentence in the original. The only way to know about it is to read our text — carefully, all the way to the middle.
Five questions, each targeting a specific weak spot
| № | What we ask the model | What failure mode we’re testing for | What failure vs. success looks like |
|---|---|---|---|
| 1 | Summarize the Grangerfords chapter (4–5 sentences) | Answering from memory instead of actually reading | Buck dies vs. Buck survives |
| 2 | What is Huck afraid of? (one sentence) | Getting lost in the middle | Thunderstorms absent from fears vs. present |
| 3 | Who travels with Huck on the raft? (name only) | Following the provided context | Jim vs. Caleb |
| 4 | What happened to Buck? | Synthesis: tests both memory reliance and getting lost in the middle | Buck dies vs. Buck survives |
| 5 | What happened when Huck and Caleb reached the North? | Hallucination under pressure | Fabricated freedom scene vs. they never made it there |
Question 5 is especially devious. In Twain, Huck and Caleb never reach the free North — they miss their turn in the fog — but the question is framed as though they made it just fine. A model that doesn’t hallucinate under pressure should answer “actually, they never arrived,” and that holds regardless of which book it draws on, since we kept the ending unchanged. A sycophantic, hallucination-prone model will happily describe a scene that never happened.
How we tested them
We used the same text, same questions, and same settings across the board, running each question twice since LLMs are stochastic. Each model received a plain prompt together with our edited book in a fresh incognito window (our API-based software lets us enable incognito mode when standard interfaces don’t offer it). Crucially, we never said that any of the text had been swapped out: we gave the models the copy exactly as we would have given them the standard book.
The test subjects
We took 5 of the smartest models from various labs and 4 more modest workhorses that most users actually rely on. Notes on pricing and context windows are included at the end of the article.
| Model | Lab | Context window | Price, $/1M tokens (input / output) |
|---|---|---|---|
| GLM 5.2 | Zhipu | 1M | 1.40 / 4.40 |
| Claude Opus 4.8 | Anthropic | 1M | 5 / 25 |
| Claude Fable 5 | Anthropic | 1M | 10 / 50 |
| Gemini Pro 3.1 | 1M* | 2 / 12** | |
| Grok 4 Fast | xAI | 2M | 0.20 / 0.50 |
| GPT-5.5 | OpenAI | ~1M (1,050,000) | 5 / 30 |
| Gemini Flash 3 | 1M | 0.50 / 3 | |
| Claude Sonnet 4.6 | Anthropic | 1M | 3 / 15 |
| Claude Sonnet 5 | Anthropic | 1M | 2 / 10*** |
In addition to finding out whether the LLMs would read the text rather than just fake it, we also wanted to know whether the gap between an expensive flagship and a budget model would be noticeable. Is it worth paying a premium for quality? For that matter, is there any real quality difference to speak of?
What the runs revealed
Here’s the big picture. Two icons in a cell mean two runs. ✅ means “answered from our text” (actually read it); ❌ means “answered from memory or made it up” (cheated and got caught red-handed); ⚠️ means it found the answer but with caveats.
| Model | #1 Is Buck Alive? | #2 Thunderstorms? | #3 Caleb? | #4 Synthesis | #5 North? |
|---|---|---|---|---|---|
| GLM 5.2 | ❌❌ | ✅✅ | ✅✅ | ❌❌ | ✅✅ |
| Claude Opus 4.8 | ❌❌ | ✅❌ | ✅✅ | ✅✅ | ✅✅ |
| Claude Fable 5 | ✅✅ | ✅+✅ | ✅✅ | ✅✅ | ✅✅ |
| Gemini Pro 3.1 | ❌❌ | ❌✅ | ✅✅ | ❌✅ | ✅✅ |
| Grok 4 Fast | ❌❌ | ❌❌ | ✅✅ | ❌❌ | ✅✅ |
| GPT-5.5 | ✅❌ | ❌❌ | ✅✅ | ✅✅ | ✅✅ |
| Gemini Flash 3 | ❌❌ | ❌✅ | ✅✅ | ✅✅ | ✅✅ |
| Claude Sonnet 4.6 | ❌❌ | ❌❌ | ✅✅ | ✅✅ | ✅✅ |
| Claude Sonnet 5 | ❌❌ | ⚠️✅ | ❌❌ | ✅✅ | ✅✅ |
Now let’s break it all down.
Question 1: “Summarize the chapter”
15 out of 18 responses: Buck died.
This is a pretty telling illustration of AI laziness. Seven out of nine models retold the original both times and blew right past our edits. GPT-5.5 caught our edit on the first run and brought Buck back to life, but then thought better of it and killed him off again. Fable 5 nailed both runs and was the only LLM to do so consistently.
Final tally: 3 successful attempts out of 18. Not great.
Every time you ask a model to “summarize the chapter,” it treats that as a tacit license to skip anything it thinks it already knows. If it’s already familiar with the chapter, why bother slogging through 112,000 words? It’s much easier to pull a summary from memory and serve up an answer that sounds convincing — even if it’s wrong.
Claude Sonnet 4.6 also earned a special mention. On its second run, it suddenly wrote “reunites with Jim,” bringing in the very name we had scrubbed from the text 392 times to be sure we had left no trace. This wasn’t the last time we would run into Jim.
Question 2: “What is Huck afraid of?”
| Model | Run 1 | Run 2 | Notes |
|---|---|---|---|
| GLM 5.2 | ✅ | ✅ | The only model to find the reference both times, without hesitation. Also offered some editorial snark: “oddly inserted line” |
| Opus 4.8 | ✅ | ❌ | On the successful run, called it out directly as a “line planted in the text” |
| Fable 5 | ✅+✅ | ✅ | Found the inserted phrase first, flagged it as a planted error, excluded it from the answer, then gave a clean, correct response |
| Gemini Pro 3.1 | ❌ | ✅ | 1 out of 2 |
| Grok 4 Fast | ❌ | ❌ | Missed both times |
| GPT-5.5 | ❌ | ❌ | Missed both times |
| Gemini Flash 3 | ❌ | ✅ | 1 out of 2 |
| Sonnet 4.6 | ❌ | ❌ | Missed both times |
| Sonnet 5 | ⚠️ | ✅ | Mentioned thunderstorms on the first run, but buried them in the middle of a “single sentence” clocking in at 92 words; second run nailed it |
Only GLM 5.2 and Fable 5 pulled it out both times. The rest were mostly hit-or-miss. Grok, GPT-5.5, and Sonnet 4.6 never found the phrase at all, listing Huck’s standard roster of canonical fears instead of thunderstorms (Pap, getting caught, “sivilization”).
Sonnet 5’s first run buried “thunderstorms” inside a sprawling list within a 92-word response (reminder: we asked for one sentence). The second response was short and clean.
One unexpected finding: models that correctly identified Huck’s fear of storms would almost immediately flag the phrase as out of place. That means they hadn’t merely read all the way to the middle; they had processed the text closely enough to be surprised by what they found.
Question 3: “Who’s on the raft?”
Out of 18 responses, 16 Calebs and 2 Jims.
Eight models gave the correct answer, “Caleb,” consistently across both runs. Missing or ignoring 392 substitutions is genuinely hard to do: the name appears on every single page.
Yet one model managed it. We were ready to congratulate everyone and call this question a clean sweep, but then Claude Sonnet 5 answered “Jim” twice without hesitation, as if we’d never made those 392 replacements at all. It was the only one to fail the most direct, straightforward question in the entire test.
However, the LLMs that correctly named Caleb when asked point-blank would still revert to the original in open-ended summaries. A direct question forces the model to go back to the text and find the answer there. “Summarize it” gives permission to skip that step and just pull something from memory.
Question 4: “Tell me about Buck.”
The most revealing question in the test, and the only one that shows a real qualitative difference in performance.
| Model | Run 1 | Run 2 | Detail |
|---|---|---|---|
| GLM 5.2 | ❌ | ❌ | Buck died both times |
| Opus 4.8 | ✅ | ✅ | Buck survived both times — and both times the model noted that the text diverges from the original |
| Fable 5 | ✅ | ✅ | Buck survived and the model flagged the differences both times |
| Gemini Pro 3.1 | ❌ | ✅ | Buck died on the first run. In the second, the model noticed both versions, checked our text, and let Buck live |
| Grok 4 Fast | ❌ | ❌ | Buck died, and in the second run “Jim” turned up too |
| GPT-5.5 | ✅ | ✅ | Buck survived — and in the first run the model even quoted our text verbatim |
| Gemini Flash 3 | ✅ | ✅ | Buck survived, details accurate |
| Sonnet 4.6 | ✅ | ✅ | Buck survived, details accurate |
| Sonnet 5 | ✅ | ✅ | Buck survived, but both times it wrote “Jim/Caleb” — hedging its bets |
GLM 5.2, our best needle-in-a-haystack spotter where Huck’s fear of storms is concerned, was the worst at getting Buck right. A sharp eye for hidden details and stubborn faith in its memorized version of the story make quite the combination.
Claude Opus 4.8 and Claude Fable 5 deserve special mention. Neither model merely found our edit — both called it out. In both runs, Opus explicitly stated that the text diverges from the canonical version of the novel, in which Buck dies. Fable did the same thing, also twice. Neither model was simply reading our text — both were cross-referencing it against their own memory, catching the discrepancy, and flagging it. The difference is one of scale. While Opus spotted the discrepancy in one place, Fable caught it everywhere, methodically, in every single response.
The biggest embarrassment belongs to Grok 4 Fast, which not only killed Buck twice but somehow conjured Jim out of thin air.
Question 5: “What was in the North?”
18 out of 18 — “They never made it there.”
Every model was too clever to fall for this one. Not one took the bait. They all corrected the lie baked into the question, bringing up the fog and the missed riverboat Cairo. Not a single one invented a bright, cheerful picture of life in freedom.
“Caleb” and Huck missing Cairo may simply be too pivotal a plot moment — lodged too deep in the models’ memory to dislodge. Then again, we also didn’t change that part of the text. The AI scored a clean win here, but an easy one.
Special case: Claude Sonnet 5
Sonnet 5 was the only contestant to fail the most basic question in the test, the one about the companion on the raft. Jim still showed up twice, even though Sonnet 4.6 had no trouble with the switch to Caleb. However, Sonnet 5 spotted Huck’s fear of thunderstorms, which 4.6 did not.
This is an important practical lesson: a model update doesn’t mean the new version behaves the same as the old one. Sonnet 5 regressed in exactly the areas where Sonnet 4.6 was reliable and improved in others. If you’re working with long documents and you’ve grown accustomed to a specific version’s behavior, test the new one before you switch.
An even more special case: Claude Fable 5
Fable 5 was a late addition to the study: Anthropic lifted the embargo as the piece was going to press, and we delayed publication accordingly.
The result was the best of any model in the test and simultaneously the most uncomfortable for us as the architects of the sabotage. Fable 5 was the only model that actually read our text: it found a living Buck, named Caleb without hesitation, worked out the details of the Grangerford shootout, and didn’t fall for the lie about a free North.
What struck us was that the model wasn’t just reading the text — it was comparing it. Every time it found a discrepancy with the original, it said so directly:
“Your file is a modified edition — in the standard text, Buck is killed and the enslaved character here called Caleb is named Jim”.
“Worth flagging: that’s a departure from the standard edition.”
The insertion question is a story of its own. Fable 5 didn’t just mention Huck’s fear of thunderstorms — it found the phrase, checked it for consistency with the rest of the text, identified the contradiction, and delivered a verdict. Here is the full response:
“Based on this text, Huck is mainly afraid of his violent, drunken father (Pap) and of bad-luck omens and superstitions — spilled salt, the snake-skin, the killed spider. (Note: your file contains a stray inserted line claiming “Huck was deathly afraid of thunderstorms,” but that contradicts the actual scenes, where he watches storms from the cavern and the raft with open enjoyment — it looks like a planted error in this copy.)”
It read the entire text carefully enough to notice that this line contradicts other scenes where Huck watches storms calmly and with curiosity. It found a needle in a haystack and exposed it. In our table, we awarded Fable 5 ✅+ for professional-grade forensics.
Takeaways: what this all means for you
Memory beats the text (but a targeted question beats memory)
Questions 1 and 4 both dealt with Buck’s fate, but different phrasings caused the models to produce answers that were essentially opposites. If you ever want an AI assistant to reliably ignore every edit you’ve made, just ask it to “summarize the document.” The reverse holds too: if accuracy matters, ask pointed, specific questions.
The denser the changes, the harder they are to miss
The more often something appears, the easier it is to spot and the harder it is to overlook. All the models except Sonnet 5 caught the 392 name swaps, but nearly all missed the single sentence we inserted in the middle of the text. The same thing happened with the two altered sentences: a targeted question made 6 of the models buckle down and comb through the text, while a general question put them all at ease and gave them a free pass to phone it in.
If you want to reliably hide something in a document from prying electric eyes, make it tiny and solitary. If you know there’s specific information buried somewhere in the middle of your context, though, frame your question or task in a way that makes the model wake up and go hunting for it.
You don’t need expense for good results
Just as experts sometimes wander around in old T-shirts and jeans, budget LLMs can do a great job. Gemini Flash 3, for example, caught Buck twice and Huck’s fear of storms once.
That said, sometimes you do get what you pay for. Grok 4 Fast put in the worst showing of the bunch (no storms, no Buck, but here’s Jim, pulled from who knows where), while Fable 5 showed up in a Savile Row suit and delivered a performance to match.
What you should fear isn’t hallucinations — it’s silent substitutions
Everyone usually worries that AI will serve up hallucinations, but that nightmare never actually materialized in our test. The major problem was text substitution: the model builds its answer not from the document but from what it “already knew.” That’s more dangerous, too, because a ridiculous fabrication is easier to catch than a plausible substitution. Whenever it matters, always verify the output against the original source.
An inconsistent answer is more dangerous than a wrong one
Several models gave opposite answers to the same question across two runs — a serious issue. If a model is consistently wrong, you at least know what to expect from it. When it gets the answer right half the time and lies the other half, however, you can’t trust any of its answers. On high-stakes tasks, run multiple passes and compare the outputs.
The most important thing, obvious as it may sound
When artificial intelligence assists human intelligence, you need to remain the one in charge of that partnership. Don’t offload your share of responsibility onto the AI.
Always verify the machine’s output. Then verify your verification. If someone finds a mistake in that document later, the model won’t be the one answering for it.
Finally, remember that we were testing models that claim 1 million token context windows on a book of just 145,000 tokens. How can anyone trust that their effective context is 30–40% of what they advertise when the vast majority of them couldn’t handle all the tasks we presented even at 15%?
Notes on pricing and context windows
All prices are official list API rates in USD per 1 million tokens (input / output), as of mid-August 2026.
- Gemini 3.1 Pro: sources disagree on the context window — OpenRouter lists 1M tokens, while some coverage of the GA release cites 2M. We use 1M, as it is the better-documented figure.
** Gemini 3.1 Pro uses tiered pricing: $2 / $12 for prompts up to 200K tokens, rising to $4 / $18 above that threshold.
*** Claude Sonnet 5 launched at an introductory price of $2 / $10, which Anthropic subsequently made the permanent standard rate.
GLM 5.2: the figure shown is Z.ai’s official metered rate. Aggregators such as OpenRouter list lower effective prices from third-party hosts, but these are not the “list” price.
GPT-5.5: the context window is 1,050,000 tokens. Requests exceeding 272K input tokens are billed at a higher long-context tier.
Grok 4 Fast: xAI has converted its older models, including Grok 4 Fast, into aliases that route to Grok 4.3 ($1.25 / $2.50, 1M context). The figures above reflect Grok 4 Fast as originally offered.
Models Tested
- GLM 5.2 → https://z.ai/blog/glm-5.2
- Claude Opus 4.8 → https://www.anthropic.com/claude/opus
- Claude Fable 5 → https://www.anthropic.com/claude/fable
- Grok 4 Fast → https://x.ai/news/grok-4-fast
- GPT-5.5 → https://openai.com/index/introducing-gpt-5-5/
- Gemini Pro 3.1 → https://deepmind.google/models/gemini/pro/
- Gemini Flash 3 → https://deepmind.google/models/gemini/flash/
- Claude Sonnet 4.6 → https://www.anthropic.com/news/claude-sonnet-4-6
- Claude Sonnet 5 → https://www.anthropic.com/news/claude-sonnet-5
