Self-attention beats asking a model for citations

TL;DR: We gave one model both jobs. Qwen3.5-9B writes an answer, then either cites its own answer when asked or has its attention read for the same answer. Reading attention scored 0.744 F1 against 0.644 for asking — and returned in 1.2s instead of 53s.

The setup: one model, one answer, one variable

Qwen3.5-9B — the model our production attribution API serves — plays every role:

1. It writes the answer. Document plus question in, answer out, with citations explicitly forbidden. That answer is frozen.
2. Condition A — ask it. A second call hands the model the document, the question, and its own frozen answer, and asks for the supporting quotes.
3. Condition B — read it. One /v1/attributions/heatmap call over that same frozen answer; attribution mass becomes sentence citations.

Both conditions cite the same statements of the same answer text from the same weights, scored by the same judge (gemini-2.5-flash, running LongCite's own scoring prompts). We use LongBench-Cite's English subset, 480 test examples, with the attention mass threshold tuned on a separate 120-example validation split.

Condition A is only interesting if it isn't handicapped, so we ran it twice: once plainly, and once with Qwen's thought channel enabled and a 40k-token completion budget. Thinking is worth about +0.10 F1 to the prompted side, so every headline number below quotes the thinking variant. The plain variant scored 0.501 — we report the better one.

Result 1: attention cites better than asking

Attention reaches 0.739 against 0.603 for the model asked to cite itself. Excluding the ~6% of prompted responses that broke format, the fair paired comparison is 0.744 vs 0.644 — a +0.100 gap. Both framings favour attention; the honest range is +0.10 to +0.14.

The gap is precision as much as recall: attention scores 0.843 precision against 0.707. When this model points at a source sentence in prose, it is more often pointing somewhere irrelevant than its attention is.

The gap grows with synthesis

The advantage is largest on the harder shapes. On longbench-chat — open-ended chat over long documents — asking collapses to 0.290 while attention holds 0.639. On gov_report summarisation it's 0.773 against 0.889, and on multifieldqa_en 0.773 against 0.915. The pattern is consistent: the more the answer paraphrases and synthesises rather than quotes, the worse the model is at reconstructing where its own content came from — and the more the attention map is worth reading.

What the two citations look like

The judge scored 4,272 statement pairs where both conditions cited the same statement of the same answer. On 759 of them attention's citation was fully supported and asking's was not; on 305 it went the other way. Three of the 759 — selected by script as the clearest case in each dataset where asking did produce a citation, not picked by eye:

The first two are the same failure: asking returned the clause carrying the keyword and stopped, leaving the rest of its own statement uncited. The third is a different one — a fluent, on-topic citation of the wrong sentence.

These three flatter asking, though. They were chosen as legible illustrations, and two of them earned partial credit; across the full 759, asking's citation was judged not supported at all in 590 and partly supported in 169. The raw records have every case in both directions.

Result 2: if you want a small model doing citations

Many teams reach for a small open model precisely to keep citation cheap. For them, the two rows above are the realistic options, and the third beats both:

44× faster than the good option, 3× faster than the cheap one, and better than both. The mechanism is why this holds rather than being a tuning artifact: asking a model to cite means generating tokens — thousands of them, when the model reasons first. Reading attention means one prefill and no decode at all. No prompt makes a decode pass cheaper than not decoding.

The latency comparison also cuts against us: the prompted rows ran on a local, dedicated H200 with no network hop, while the attention numbers are real calls to api.tokenpath.ai — production hardware, network included.

Don't ask a small model to cite inline

A tempting shortcut is to have the model emit citations while it writes, in LongCite's <statement>…<cite>[[3]]</cite> format. We tried it. It scored 0.265.

That number is easy to misread, so we ran the control: taking those same inline-generated answers and asking the same model to cite them in a separate call scored 0.561 — more than double, on identical answers and identical statements. Attention on those same answers scored 0.803.

So most of the inline deficit is not an inability to find sources; it is the difficulty of interleaving citation tags correctly while generating prose. The model held the format 97% of the time and still cited badly. If you want citations out of a 9B model, use a second pass — and if you want them cheaper and better than that, don't ask the model at all.

Limitations

The threshold is load-bearing. Attention scores 0.767 at a 0.10 mass threshold and 0.552 at 0.30. We tuned it on a 120-example validation split and report on a disjoint 480-example test split — but this is a tuned system compared against an untuned prompt.

One model, one benchmark. Qwen3.5-9B on LongBench-Cite, whose documents are 2023-era. Attention sharpness varies a lot between model families, and this experiment can't tell you the gap holds elsewhere.

Reproducibility

The harness, prompts, and aggregation live in the benchmarking harness in our cookbook repo.

• Model under test: Qwen/Qwen3.5-9B, served locally with vLLM for the generation and prompted conditions; attention via api.tokenpath.ai
• Judge: google/gemini-2.5-flash, running LongCite's auto_scorer.py prompts verbatim
• Aggregation: {threshold: 0.10, row_norm: true, merge_adjacent: true, max_spans: 4}, tuned on val (n=120), reported on test (n=480)
• Split: LongBench-Cite English subset, deterministic seed 0

Questions, or a workload where you'd like to see this reproduced? support@tokenpath.ai.