Context rot is a drop in an AI model's reliability as the text it reads grows longer, even when the answer is present and the input fits inside its context window. In survey and interview analysis, that creates a risk of missed minority views, wrong theme counts and inconsistent coding. Reading smaller batches can reduce the risk, but cannot establish that an analysis is correct by itself.
The useful question is not just whether 40 transcripts fit. It is whether you can trace each code to a respondent and find what was missed.
Disclosure: Meaningful designs analysis around this problem. The cited experiments do not benchmark Meaningful or establish an interview failure rate. Sources checked 6 November 2026.
What is context rot?
Context rot describes worsening performance as relevant material is surrounded by more input. A context window is how much text a model accepts; tokens are pieces of words or punctuation. Fitting within the limit does not ensure reliable use. Hong, Troynikov and Huber's Chroma report (2025) tested 18 models in controlled long-input tasks.
| Problem | What happens | A practical clue |
|---|---|---|
| Context rot | Relevant evidence is inside the accepted input, but the answer uses it less reliably as the input grows. | A minority account disappears when you add more transcripts. |
| Context overflow | The input exceeds the available window or another product limit; material is rejected, truncated or handled another way. | The system reports a limit or cannot account for a file. |
| Hallucination | The answer invents a claim or quotation unsupported by the supplied sources. | You cannot find the quoted words in the cited transcript. |
They can coexist: an answer may overlook an interview and invent a quotation. Check the input manifest and final prose.
Why doesn't a bigger context window fix it?
A larger window lets a model receive more text; it does not promise equal attention or accuracy throughout that text. You can be well inside a model's advertised limit and still lose performance on a particular task.
In Chroma's LongMemEval comparison, all tested models did better with focused prompts of roughly 300 tokens than full prompts averaging roughly 113,000. This also changes the retrieval burden; it does not isolate length as the cause.
For that narrower question, Du and colleagues, Findings of EMNLP (2025) tested five models on mathematics, question answering and coding. Performance fell 13.9%–85% within claimed limits, even with perfect evidence retrieval, evidence just before the question or irrelevant material reduced to whitespace. Those are not predicted research-project error rates.
Fitting 400 open ends in one prompt solves capacity, not consistent coding.
What causes context rot?
Position, length and competing information can each make a task harder. Effects vary: neither “the middle always disappears” nor “the whole file fits, so it was read accurately” is a rule.
Position: lost in the middle
Lost in the middle means less reliable use of evidence between a long input's edges. Liu and colleagues, Transactions of the Association for Computational Linguistics (2024) often found better question-answering and retrieval performance near those edges. Check a contrary interview halfway through a pack. Chroma's needle-question test found no notable position effect, so the pattern is not universal.
Length even when the answer can be found
In Du and colleagues' tests, performance fell despite successful evidence retrieval. A model could repeat a respondent's words yet apply the wrong code; putting the strongest quote at the end is no cure.
Distractors that look relevant
Chroma found distractors hurt in its tests, increasingly with length; coherent surrounding text was harder than shuffled text in one setup. Similar answers across waves may be competing evidence. These experiments do not measure error rates on agency interviews.
Does context rot affect AI analysis of survey and interview data?
It can: coding classifies passages, and models can miss items in long inputs that they detect in shorter ones. Evidence from real agency studies remains thinner than benchmark and safety-monitoring evidence, so test the risk; no fixed interview block is ignored.
The closest long-transcript analogue is Martin and Roger's Classifier Context Rot (2026 preprint), from the Anthropic Fellows Program and Anthropic. They detected dangerous actions in coding-agent transcripts, not human interviews. With Opus 4.6 using thinking, recall for obvious actions fell from 99.7% at 100,000 tokens to 69% at 800,000. In a separate subtle-action test, recall fell from 98.6% alone to 88% after 800,000 tokens of benign activity. GPT 5.4 Thinking and Gemini 3.1 Pro also showed weaknesses. Middle placement was difficult; reminders, more reasoning and incremental checks partly helped.
These safety-detection figures do not mean 31% of respondents disappear. The analogy: a model scanning a long transcript may miss something it finds with less surrounding material.
| Research task | Possible failure to test | Evidence and its limit |
|---|---|---|
| Code one theme across 40 interviews | One respondent's qualifying passage is not flagged. | Martin and Roger (2026 preprint) test safety-action detection, not interview codes. |
| Count respondents mentioning price | An uncoded case changes the numerator, or a respondent with several mentions is counted twice. | Long-context misses are plausible from classification work; distinct-person counting also needs a separate deduplication step. |
| Select quotations | An easy-to-find quote crowds out a quieter negative case, or the quote is paraphrased as verbatim. | Liu et al. (2024) study position in retrieval; quote integrity must be checked at source. |
| Compare waves or segments | A similar response from another wave is used to justify the wrong comparison. | Chroma (2025) tests distractors, not survey-wave attribution. |
| Apply a codebook consistently | The model finds a passage but applies a category differently after more material is added. | Du et al. (2025) show that finding evidence and using it correctly differ on other tasks. |
Not every error is context rot. Ashwin, Chhabra and Rao (2023 preprint) examined 2,407 open-ended interviews, 789 expert-annotated, and found coding errors associated with respondent characteristics. That bias differs from length effects; batching alone cannot fix it. Check misses by group and position.
Does retrieval-augmented generation solve context rot for research data?
Retrieval-augmented generation, or RAG, selects passages from a collection for a question. Short inputs help lookup, but the answer depends on what retrieval selected. Lewis and colleagues (2020) describe the method, not exhaustive research coverage.
“What did respondent 14 say about price?” suits retrieval. “How many distinct people objected?” also needs a source set, codebook and coverage check. A keyword search may miss “too expensive”; semantic search may admit near-misses. Retrieval can support exhaustive work if its recall and uncaught records are checked. See our chatbots versus research platforms guide.
How do you reduce context rot in research analysis?
Shorten each coding job and keep a record-level trail. This workflow makes errors easier to find, not impossible.
- Freeze and split the source set. Record file and respondent IDs. Batch by respondent or question, keeping enough surrounding context to code accurately. Note failed imports and exclusions.
- Use one versioned codebook. Give batches identical inclusion, exclusion and uncertainty rules. If definitions change, reconcile earlier records.
- Request evidence before each code. Ask for the passage, respondent ID, source location, label and reason. Evidence-first prompting helped in Du et al. and Martin and Roger; still verify quotations.
- Keep respondent-level rows. Store each respondent–code decision with evidence, batch, codebook version and present/absent/uncertain status. Preserve negative cases and a processing manifest.
- Calculate counts from rows. Use a spreadsheet or code to count distinct eligible people by code and segment. State the denominator and missing-data rule.
- Reconcile in a shorter second pass. Merge labels, resolve disagreements and test cross-batch contradictions against the underlying rows, not just summaries.
- Audit before delivery. Check a random sample, pivotal minority and contrary cases, and all client-facing quotations. Inspect differences by position, subgroup and wave.
A log proves input reached a step, not comprehension; a citation cannot prove uncited records were assessed. See our transcript-tool scorecard.
How can you test your own workflow in an hour?
Test the same coding question on a permission-cleared, completed project at three input sizes. Look for changing answers and missed cases, not a universal threshold.
- Freeze a previously human-coded project's codebook, eligible IDs and one minority or contrary view. Record its exact passages before running the tool.
- Make packs with one batch, roughly a quarter and all allowed data. Keep the case in each; record its position. Hold the question and output format constant.
- Request codes, verbatim passages with IDs and distinct-person counts. Compare shared cases with the human reference; check each pack's own denominator.
- Repeat the full pack. To isolate position, also move the case to an edge within the same-size pack. Log errors and checking time.
If a view vanishes or a code flips, inspect the trail and try smaller batches. A stable answer is encouraging, not proof of universal safety. Upload client data only through an approved route.
How does Meaningful handle this risk?
Meaningful says its agents read each included response and transcript in focused batches, then reconcile findings across sources. Its product explanation shows a processed-record count separate from readout examples; inspect both on your project.
It avoids one sprawling prompt but cannot prove perfect coding or interpretation. Imports, codebooks and synthesis can still fail. Apply the same manifest, quotation, minority-case and human-review checks to Meaningful. The research director owns the client conclusion.
What do researchers ask about context rot?
The answer depends on the task and input, not a magic document count.
Is context rot still a thing with newer models?
Yes. Chroma's 2025 tests found degradation across the 18 models it examined; Martin and Roger's 2026 preprint found long-transcript detection weaknesses in Opus 4.6, GPT 5.4 Thinking and Gemini 3.1 Pro. Results depend on the task, prompt and setup. Re-test your own workflow when a model or product changes.
Is context rot the same as hallucination?
No. Context rot concerns failing to use relevant information already supplied; hallucination concerns an unsupported invented claim. A model can do both: overlook a respondent's answer, then supply a plausible quotation that nobody said. Check source coverage and quotation fidelity separately.
How many interviews can I give ChatGPT before context rot starts?
There is no universal safe interview count. Length, model, task and similarity between answers all matter; an accepted upload is not a quality certificate. OpenAI's file-upload guidance describes document analysis and notes that upload limits vary by plan and account settings. Test short, medium and full inputs on approved material and audit respondent-level output.
Does putting important text at the start or end fix it?
No. Liu and colleagues found a position effect in their tasks, so placement can matter. But Du and colleagues found degradation even with relevant evidence right before the question. Check position and shorten the job; do not rely on a placement trick.
Does RAG fix context rot for survey data?
Not on its own. Retrieval can shorten a lookup prompt, but exhaustive counts and minority-view searches depend on whether relevant responses were selected. Ask what proportion of the eligible records was assessed, how retrieval recall was tested and how absent or uncertain codes were recorded before calling the resulting answer comprehensive.
What is the difference between context rot and lost in the middle?
Lost in the middle is a position-related pattern: material between the start and end may be harder to use. Context rot is broader. It includes worse performance as input grows and the effects of competing material; not every experiment shows the same middle dip. Liu et al. and Chroma tested different setups.
Can context rot make survey counts wrong?
Yes, if a model misses a qualifying response, changes a code or counts mentions instead of distinct people. Calculate counts from checked respondent-level decisions with a stated denominator. Then audit edge cases, negative cases and subgroup differences against the original answers; a spreadsheet calculation cannot repair a missed or biased code.
What should you do next?
Run the one-hour check on a finished project before changing your agency's method. If you want to compare an exhaustive, batched workflow with a long-chat approach, start Meaningful's current three-day trial on a permission-cleared study and ask to inspect processed-record counts, minority cases and source-linked quotations. Keep the final interpretation and client sign-off with your research team.