Token embedding matrices in language models scale linearly with vocabulary size — the "vocabulary tax" has plagued the industry for years.
arXiv 2606.28057 proposes MultiHashFormer, which drops the exclusive embedding vector and instead uses multiple independent hash functions to generate a "hash signature" (a short sequence of discrete hash IDs) for each token. A Hash Encoder compresses the signature into a single latent vector fed to the Transformer decoder, and a Hash Decoder generates the next token's signature before mapping it back to text.
The trick: as long as there are enough hash functions, the signature combination space vastly exceeds vocabulary capacity, and the autoregressive LM breaks free from the "many-to-one collision" curse. The paper verifies across 100M / 1B / 3B scales: MultiHashFormer consistently beats the standard Transformer LM.
The standout is multilingual vocabulary extension: when you add a new language to an existing vocabulary, no retraining of the embedding layer is needed — the parameter footprint stays constant. That matters a lot for LLM and localization vendors pushing globalization.
The paper is currently under review, and whether it scales to 70B+ and how much inference-time hash-lookup overhead will be are still open questions.