Hundreds of millions of Indians type to AI in romanized Latin script — aap kaise hain, meeru ela unnaru, neevu hegiddira. These are not English translations; the vocabulary, grammar and social register all belong to the native language. And none of it shows up in mainstream multilingual evaluation. A new paper measures exactly that blind spot, and the result is uncomfortable: the automatic metrics award their highest scores to the worst failures.

The setup

The study uses 142 English query-response pairs drawn from OpenAssistant, stratified across technical, emotional, narrative and factual prompts. The same inputs go to three target languages — Hindi, Telugu, Kannada — and four models: Claude Opus 4.8, GPT-5.5, Gemini 2.5-flash and Qwen3.5-35B, all at temperature 0, with Qwen served locally via Ollama on an NVIDIA RTX PRO 6000 Blackwell GPU.

Human evaluation covers 72 responses (24 per language). Three native speakers, one per language, score each response on a ten-dimension rubric worth 0-12 points: completeness, script purity, English-romanization balance, romanization consistency, semantic faithfulness, naturalness and fluency, grammar, word choice, register, overall usability.

On the automatic side, BGE-M3 cosine similarity is the primary cross-lingual measure and multilingual BERTScore F1 the secondary one. The two strongest models come out nearly tied: GPT-5.5 averages 0.790 on BGE-M3, Claude Opus 4.8 averages 0.786 — a gap of 0.004. Native speakers disagree sharply: 10.73 out of 12 for GPT-5.5 against 9.30 for Claude, a gap of 1.43 points. The paper does the division itself: 1.43 divided by 0.004 is 357.

One answer, two scores

The individual cases are worse than the aggregate suggests.

One Qwen Telugu response simply returned the English prompt verbatim, word-overlap 1.00. BGE-M3 gave it 1.000. BERTScore F1 gave it 1.000. Both metrics called it a perfect answer. The native speaker gave it 0.

Another response had collapsed into a repetition loop, the same word or phrase repeated until it hit the token limit. BGE-M3 still awarded 0.72. Human usability was still 0.

The paper’s appendix lines up the contrast: Telugu pure-English scores BGE 1.00 / BERTScore 1.00 / usability 0.00; Kannada degeneration loop scores BGE 0.38 / BERTScore 0.44 / usability 0.00; Hindi degeneration loop scores BGE 0.69 / BERTScore 0.54 / usability 0.50.

The paper supplies a correlational explanation. Across Qwen’s outputs, raw BERTScore-F1 correlates with English word overlap at r=0.955, and BGE-M3 at r=0.805. The metrics are largely rewarding how much English remains, not how well anything was romanized. There is also a plainer detail: baseline rescaling was left disabled, so empty responses still score about 0.42.

Two unrelated ways to fail

Gemini and Qwen break in opposite directions, and the failure modes never overlap.

Gemini leaks native script into romanized text. Twenty-four of 142 Telugu responses (16.9%), 19 Kannada responses (13.4%) and 8 Hindi responses (5.6%) — 2,635 native Telugu characters in total. The contamination crosses languages too: a Russian word appears mid-sentence in Telugu row 53, Kannada row 33 emits Telugu script instead, and Telugu row 19 mixes Tamil and Telugu in one line.

Qwen never leaks script at all. It fails the other way, refusing to romanize and returning English unchanged. Under the strict word-overlap = 1.0 threshold that is 3 Hindi rows, 16 Telugu rows and 19 Kannada rows. Under the broader threshold (word-overlap at or above 0.90) the paper reports 70 English-only returns across all models, of which Qwen accounts for 51, and 41 are genuine failures once English-task prompts are excluded. GPT-5.5 produces zero.

Qwen has a third failure mode: degeneration into repetition. The paper identifies 13 such instances, every one from Qwen, none from the others. Kannada suffers most, with 49 responses hitting the 4,096-token limit and mean output tokens rising from 541 to 1,768.

The counting rule is not cosmetic

The paper’s most transferable move is to fix the counting rule: invalid rows score zero and stay in the denominator rather than being dropped.

Four conditions mark a row invalid: empty or whitespace-only output; an English-only copy at word-overlap 1.0; native-script characters present, excluding legitimate romanization diacritics such as ā, ē, ṭ; and degenerate output, a single token repeated at least 20 times or a four-token phrase repeated at least 50 times.

This directly changes the ranking. If invalid rows were quietly excluded, Gemini’s Telugu BGE-M3 would climb from 0.630 to roughly 0.82, indistinguishable from GPT-5.5’s 0.794. Kept in at zero, the gap stays visible.

The paper also checks the metrics against each other: BGE-M3 and BERTScore disagree by more than 0.15 on 44% of rows, with a maximum divergence of 0.28. When two standard metrics disagree on nearly half the outputs, their agreement is not validation.

I recomputed it from the authors’ own repository

The official repository publishes the per-row scoring tables. Taking the authors’ per-language means and reweighting by sample count reproduces all four model totals exactly: 10.73 for GPT-5.5, 9.30 for Claude Opus 4.8, 8.91 for Gemini, 4.48 for Qwen. The three language-level cross-model means 9.42, 7.11 and 6.89 reproduce, Qwen’s per-language 4.07, 2.86 and 8.28 reproduce, and the degeneration count 1 + 10 + 2 = 13 reproduces.

One calibration drift is worth flagging. The repository’s failure-rate table uses the broader threshold (word-overlap at or above 0.90) and reports 30.40% English-only returns for Qwen on Kannada. The abstract says 16.2%, while the paper’s own Table 2 counts 19 of 142 under the strict = 1.0 threshold, roughly 13.4%. Three numbers, three definitions. The direction holds either way — Kannada is Qwen’s worst language — but the specific percentage is meaningless without naming the threshold.

The boundaries

The authors list their limitations without hedging: one native speaker per language, so no formal inter-rater agreement was computed; temperature 0, so there is no sampling variance and the figures are point estimates; the 72-response diagnostic set was stratified on failure conditions out of 1,704 total responses and does not reflect population failure rates; three languages only, with no models specifically trained on Indic data such as AI4Bharat’s; and only the English-to-romanized direction, with no explicit code-switching tested.

One limitation deserves emphasis. The paper’s own conclusion says the core finding is not that romanized Indic is hard, but that automatic metrics cannot be trusted as sole arbiters of quality for this task. That is a claim about evaluation methodology, not a verdict on any model.

Why it matters

This is not an academic edge case. A real corpus of patient WhatsApp conversations, cited by the paper, puts romanization at 83% of Hindi messages and 73% of Telugu messages. Another cited finding: tokenizers trained on English-dominant corpora need up to 15 times more tokens to represent the same non-English content. And prior work measured that romanized input degrades performance on structured classification tasks by up to 24 points relative to native script.

In other words, the place where the evaluation method fails is precisely the daily interface of hundreds of millions of people. If you run only BGE-M3 and BERTScore, produce a handsome table, and pick your model from it, the model you picked may be the one that hands the English text straight back.

So the practical takeaway is plain: a score has to be bound to whether the response is valid at all. Without that, a high number just means nobody was looking.

Paper: arXiv:2611.09085; code and per-row scoring tables: MadhuBeesetty/llm-romanization.