Multilingual attribution with TokenPath

TL;DR: We kept the source contracts in English, changed the questions and answers into 23 other languages, and asked each method to find the supporting English clause. TokenPath found the correct clause for 720 of 724 valid claims (99.4%). The embedding-and-rerank setup from our previous benchmark found it for 636 claims (87.8%) when returning its top result. TokenPath had a statistically clear paired advantage in 15 of 23 languages.

In a separate pilot across all 22 scheduled Indian languages, TokenPath found the correct clause for all 371 translated claims that passed validation.

The use case: answer locally, cite the English contract

Many multilingual legal products have an asymmetry. The contract, policy, or case file is stored as English text, while a customer asks a question and gets an answer in their preferred language. The attribution layer then has to map a claim such as “the tenant must notify the landlord within seven days” — in Arabic, Japanese, Hindi, or Bengali — back to the exact English clause that supports it.

That is the task we test here. The input is extracted text, and the answer is held fixed so we can isolate attribution.

How we built the benchmark

We wrote four synthetic English contracts and annotated eight atomic claims per contract: 32 claims per language. For each target language, a configurable generation model translated the question and the already-grounded answer. A separate model checked meaning preservation, language, and script. Translations that failed this gate were excluded before attribution scoring.

This produced 724 valid target-language claims out of 736 across 23 languages. The English source and gold clause never changed.

For every valid claim, we compared:

1. TokenPath, attributing the target-language answer to the English source.
2. Embedding + rerank top-1, using the models from our LongBench-Cite benchmark: text-embedding-3-large retrieves five candidates and BAAI/bge-reranker-base returns the highest-ranked one.

The headline metric

The main metric is gold-clause hit rate: does the returned span overlap the annotated supporting clause? This is objective and treats a narrow TokenPath span and a whole retrieved sentence consistently. We report character-level intersection-over-union separately as a boundary diagnostic; it should not be used to compare methods that return evidence at different granularities.

International results

TokenPath's paired advantage was statistically clear in 15 languages: Arabic, Bengali, Dutch, French, German, Hebrew, Indonesian, Italian, Persian, Polish, Portuguese, Russian, Swahili, Ukrainian, and Vietnamese.

Spanish, Turkish, Hindi, Japanese, Korean, Thai, and both Chinese variants were closer. They still achieved 96.7–100% with TokenPath, but the paired difference from retrieval top-1 was not clear on this sample. In Hindi, retrieval hit 100% while TokenPath hit 96.7% on 30 valid claims.

So which languages are “supported”?

For this specific workflow — target-language answer, extracted English legal text, short synthetic contracts — TokenPath achieved at least 96.7% correct- clause hits in every tested international language. That is direct evidence that the attribution mechanism can cross these language boundaries.

For customer decisions, the strongest relative evidence is in the 15 languages where the advantage over retrieval top-1 was clear. The other eight also had high TokenPath accuracy, but were closer to the retrieval baseline. We prefer showing those measurements to applying one universal “supported” label.

India: all 22 scheduled languages

India deserves a separate view rather than being folded into a global average. We ran a pilot on three additional synthetic English legal documents, with six claims per document. It covered every language in the Eighth Schedule of the Constitution of India, plus English as a control.

Across the 22 target languages, 371 of 396 translated claims passed the meaning, language, and script gate. TokenPath returned evidence overlapping the correct English clause for all 371 valid claims.

The chart below shows a stricter boundary diagnostic: mean character IoU between TokenPath's span and the full annotated clause. A lower score can mean the attribution is wrong, but it can also mean TokenPath returned a shorter, still-sufficient phrase. The dashed line is the English control at 71.1%.

Sixteen Indian languages retained all 18 translated claims. Punjabi had 15, Urdu 17, Maithili 17, Manipuri/Meitei 6, Nepali 17, and Santali 11. Tamil, Telugu, Gujarati, and Nepali had lower boundary overlap than English, despite still finding the correct clause. Because the samples are small and this pilot does not yet include the retrieval baseline, we treat these as preliminary results rather than a relative-performance ranking.

How to read these results

• This is an attribution test on short, synthetic English contracts using extracted text. PDF and OCR are outside scope.
• Translation was model-generated and independently validated. The attribution scores are conditional on a question and answer passing that gate.
• The sample sizes — 30–32 claims internationally and 6–18 in India — show large differences, but do not certify rare failure rates or production use.

Before a customer rollout, we would rerun the evaluation on longer, customer-approved documents with human-reviewed claims and unsupported answers.

Inspect and reproduce

The compact, machine-readable result summary and per-language counts includes the model routes, metric definitions, exclusions, and every number plotted here. The embedding baseline implementation is the same cookbook harness used in our earlier benchmark.

We generated the multilingual material through OpenRouter with a pinned model route, temperature zero, and a fixed seed; we cached every response and stored no credentials. We retained the independent runner and full run artifacts — translated questions and answers, validation decisions, TokenPath spans, retrieval scores, and gold clauses — separately from the website application. If you are evaluating TokenPath for a multilingual workload, talk to us and we will help design the customer-specific follow-up.