Late last year a claim spread widely: after ChatGPT, posts in one conservative American forum became visibly more homogeneous, while a progressive forum showed no comparable change. It read as hard evidence that AI is flattening how people express themselves.
The researcher who made that claim has now re-run it himself. His answer: the signal was never measuring convergence of ideas at all. It was measuring text length. The right-flaired users in that conservative subreddit saw their average comment shrink from 109 words in 2019 to 61 in 2022, then stabilize. Comments got shorter, and pairwise cosine similarity rises as text gets shorter.
A Re-measurement That Undoes Its Own Predecessor
The original study, arXiv:2608.13674, used 6.06 million Reddit comments from r/AskConservatives and r/AskALiberal spanning 2019 to 2025. The measure was the mean pairwise cosine between mean-pooled DeBERTa-v3-base embeddings, and note that this encoder was never trained for sentence similarity.
The headline claim: after December 2022, within-group similarity among right-flaired users in r/AskConservatives broke upward against its prior downward trend, with gamma = +0.0081 at p < 0.001, while left-flaired users in r/AskALiberal moved far less.
The re-measurement draws a stratified sample of 915,225 comments across 336 subreddit-by-ideology-by-month cells, swaps in two sentence-similarity encoders, and runs six diagnostics.
Every Diagnostic Points at Length
- Truncation: truncating the same comments from 10 to 150 words raises mean cosine by eight times the reported break, 0.064 against 0.0081.
- Length-only placebo: swapping every comment for a random one of equal length yields 0.0072 against an observed 0.0089, so length alone reproduces 81% of the asymmetry, and 58% of placebo draws reach p < 0.05.
- Fixed-content simulation: 78% of the effect is manufactured by length variation alone, and under BGE the share is 96%, 0.0144 against 0.0151. Even the centered point estimates that were not significant in the real data are 66-85% length.
- Leading direction: 99.9% of raw DeBERTa homogeneity sits on the first principal direction, which is exactly where the length signal lives.
Once length is handled, the asymmetry disappears under every encoder. The crudest control, adding mean word count and its interaction with group to the difference-in-differences, cuts the estimate from 0.0089 (p = 0.021) to 0.0024 (p = 0.073). Recomputed on the full 6.06 million comments, the corrected estimate is 0.00 SD (p = 1.00).
The Same Trap On Another Platform
The core diagnostics repeat on Stack Exchange: 8 sites, January 2018 to March 2024, 575,000 answers. The measure reports post-ChatGPT homogenization of technical sites, and the placebo fully reproduces it. Note that answers here grew longer, the opposite direction from Reddit. A metric built to detect converging discourse was picking up comments shrinking on Reddit and answers growing on Stack Exchange.
What Survives The Correction
With the measure repaired, the two camps converged between 2019 and 2022 and stopped converging before ChatGPT arrived. The Left-Right centroid distance fell from 14.5 x 10^-3 in 2019 to 1.8 x 10^-3 in 2022. The author is blunt about the limit: he has no test of why the camps converged and treats the decline as a description of these two forums.
Replies are the interesting part. Sampling 192,975 direct replies, cross-camp replies accommodate their targets more than same-camp replies do: right-to-left by +0.017, left-to-right by +0.006, and right-to-right by -0.006 (p < 0.001). Cross-camp replies quote their parents 19.5% of the time against 13.6-14.8% within camps. And that accommodation did not rise after ChatGPT.
Even The Standard Fix Can Flip The Sign
The paper also runs a reverse control against the Doshi and Hauser 2024 experiment. The usual remedy is centering plus renormalization. Applied here, the effect reverses sign under every encoder: -0.39 SD under BGE, -0.23 under mpnet, -0.30 under DeBERTa. AI-assisted stories sit closer to the pooled mean, so after centering their residual vectors are only 0.31-0.61 SD long, and renormalizing them to unit length turns closeness into noise. A measure that does not renormalize recovers the effect at 0.29-0.52 SD.
And a harder point: even with no length change at all, running the time-series design on fixed-content synthetic series rejects the null in 10-15% of runs, two to three times the nominal rate, because 84 monthly observations with six Newey-West lags give anti-conservative intervals for a level shift.
Why This Is Not Just One Paper
The mechanism was flagged earlier, just never connected to this ruler. Xiao et al. (EMNLP 2023) showed that contrastive-learning encoders suffer length-induced semantic drift. Zhou et al. (ACL 2025) named the phenomenon Length Collapse and explained it in the Fourier domain: self-attention acts as a low-pass filter, so longer text keeps more low-frequency components and long-text embeddings cluster together.
The paper also censuses 31 published and preprinted studies, mapping where anisotropic encoders meet time-series or length-unequal designs.
The Cold Water
Three caveats. First, this is the original author re-measuring their own earlier work, not an independent replication; the original was careful across interrupted time series, regression discontinuity in time, and trend-corrected difference-in-differences, but every design consumed the same series, and that series measured length. Second, the paper covers two platforms, and the magnitude and even the sign of the artifact will vary with other data and encoders. Third, the agenda-voice split uses a single principal direction.
The author's own conclusion is the one I find right: a study that finds AI-era homogenization with a length-sensitive measure may be detecting that people write shorter or longer, which is a finding about volume of expression and says nothing yet about the range of ideas.
So What
The takeaway is not that AI did not homogenize anything. It is that when a metric becomes a time series, ask what it actually measures first. The diagnostic order here is cheap: report the random-pair baseline, run the length-only placebo, rerun under several encoders, then calibrate the test itself.
So the next time a headline says AI made some group sound alike, go to the methods section and check whether anyone reported text length.