Why attention attribution transfers across LLMs

TL;DR: TokenPath attributes any model's answers with Qwen3.5-9B's attention. Reading answers from GPT, Claude, Gemini and Grok, Qwen disagrees with a quarter to a third of tokens — but those are the reasoning and voice, not the parts that restate the document. On grounded tokens it agrees with every generator alike (perplexity 1.3–1.4 across all six, against 1.9–5.3 elsewhere), and attribution only reads those. 53 of 53 answers.

Setup

An answer is fixed text. Run it through a model and you get, at each position, the probability that model would have assigned to the token that is actually there — agreement with the observed token, not a window into the generator, whose distribution we never see. We report it as perplexity: 1.0 means Qwen would have written exactly this, higher means it would have chosen otherwise more often. An attributed span is a phrase the production API decided to cite, chosen by its own probe with no input from us.

Two kinds of token matter. Hard tokens are the reasoning, structure and voice that make a frontier answer frontier. Grounded tokens restate the document.

Five models — GPT-6 Astra, GPT-5.6 Luna, Claude Opus 5, Gemini 3.8 Flash, Grok 4.6 — plus Qwen3.5-9B itself as the baseline, answering nine questions across three multi-document corpora. 53 answers, 21,285 answer tokens, all of it in summary.json.

The hard tokens are hard

Across a whole answer, Qwen agrees with itself at perplexity 1.66 and with the other five at 2.90 (Claude Opus 5) to 4.79 (GPT-6 Astra) — a quarter to a third of their tokens are not what it would have picked. That is what a 9B model reading frontier output should look like. It only sharpens the question: if a third of the answer is text Qwen would not have written, what is attribution reading?

Attribution never touches them

Not that third. Pooled across all 53 answers, tokens inside attributed spans sit at perplexity 2.03 against 4.86 for everything else.

53 of 53 answers have lower mean surprisal inside their attributed spans than outside, and spans cover 51.2% of answer tokens on average, so this is not a narrow slice. Nor is it punctuation inflating the unattributed side: dropping every token without an alphanumeric character widens the gap to 2.03 against 6.00. Claude Opus 5 shows the split most cleanly — its cited spans sit at 1.67, which is where Qwen's own unaided prose sits (1.66), against 5.33 for the rest of the same answers.

Here is one of those answers, every token coloured by the log-probability Qwen gave it, dotted boxes around what the API chose to cite.

The red is Opus working. It opens with a markdown header Qwen ranks 326th, and when it names the amendment that supersedes the clause it has just quoted, that " Amendment" is rank 1,012 — Qwen expected the sentence to go somewhere else entirely. " Since", " expressly", " deal" and " lands" are all choices it would have made differently.

Inside the boxes, the worst token in the whole answer is rank 21. The reconciliation — which clause survived, which amendment replaced it — is Opus's work, and Qwen would not have produced it. The words being reconciled are the document's, and those are the ones in boxes.

One objection could deflate all of this: the probe choosing those spans reads Qwen's layer-15 residual stream, the same tensor the logits come from, so of course it picks tokens the model finds easy. So we relabelled every token without consulting the model at all — is it inside a run of five or more consecutive answer words appearing verbatim in the document set? That is string matching, and the two labels agree on only 60.3% of tokens.

The green column is the answer. Across six models of wildly different capability, verbatim quotation runs 1.28 to 1.42 — a range of 0.14. Everything else runs 1.88 to 5.30, a range twenty-four times wider. Whatever separates a 9B model from a frontier one does not show up in how hard it is to predict a quotation. And grounded is broader than verbatim: among tokens that are not literal quotes, the ones the API attributed sit at 2.35 against 5.06 for the ones it passed over, so the probe is finding restatement too — easy for the same reason quotation is.

Why this works

Attribution reads attention from the answer's query rows — the vector each answer position uses to look back over the document — and takes the phrase-level answer from a retrieval head, an attention head that empirically tends to point at the source of whatever is being restated.

A query row at position i is built from the same prefix that determines Qwen's distribution over token i. When Qwen finds the observed token easy, that row is close to one the model would have built for itself, so the retrieval head gets a well-formed query and points where it normally points. When the token is one Qwen would never have produced, the row is off-distribution and the head has no particular reason to behave. Predictability is the precondition for a well-formed row — not a promise the span it resolves to is right.

Which is why the asymmetry matters. Grounded tokens are easy for any competent reader holding the document, and attribution only reads those. The generator's fingerprint lives almost entirely in the hard tokens, and those are precisely the ones that should never carry a citation.

What this doesn't show

• Easy to predict is not correctly attributed. Nothing here says the spans are right — that is measured against gold citations, in the LongBench-Cite post and the multilingual benchmark. Span confidence correlates with mean log-probability at only r = 0.32, so we are not shipping predictability as a confidence signal either.
• Correlation, not intervention. We show attributed tokens are the easy ones and the easy ones are the grounded ones. We did not intervene on the forward pass to show that surprisal causes the attention row to degrade, which needs a counterfactual pass we left out.

Scope: 53 answers over three English corpora of 19k–64k characters, on extraction-heavy questions. Every answer was written after a reasoning pass the attribution prompt never contains, so the result holds under one real prompt mismatch — but we did not vary document order or paraphrase the question, and that is the next experiment.

Reproducing it

Five scripts in experiments/cross-model-logits/: generate.py (answers, via OpenRouter), attribute.py (spans, via POST /v1/attributions with automatic span selection), logits.py (one Qwen forward pass per answer, on the exact prompt gpu_backend/prompting.py builds, reading attention at L15 H13 and L19 H12), verbatim.py (the model-free control) and analyze.py. The logits pass needs one GPU and about 90 seconds for all 53 answers; everything else is API calls.