[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-multihashformer-hash-embedding-3b":3,"news-related-b47914fe-d539-440f-ab02-3b089120051f":36},{"id":4,"title":5,"summary":6,"content":6,"original_url":7,"source_id":8,"tags":9,"translations":23,"news_slug":29,"published_at":30,"created_at":31,"modified_at":32,"is_published":33,"publish_type":34,"image_url":13,"view_count":35},"b47914fe-d539-440f-ab02-3b089120051f","MultiHashFormer 用多重哈希签名改写 LLM 词表嵌入：3B 模型跑赢标准 Transformer","语言模型的 token 嵌入矩阵与词表大小线性绑定——\"vocabulary tax\" 困扰行业多年。\n\narXiv 2606.28057 提出的 MultiHashFormer 给出解法：放弃独占嵌入向量，改用**多个独立哈希函数**生成每 token 的\"哈希签名\"（若干离散哈希 ID 的短序列）。Hash Encoder 把签名压成单个 latent 向量喂给 Transformer 解码器，Hash Decoder 生成下一 token 的签名再映射回文本。\n\n精妙之处：只要哈希函数数量足够，签名组合空间远超词表容量，自回归 LM 因此摆脱\"多对一冲突\"诅咒。论文在 100M \u002F 1B \u002F 3B 三档规模上验证，MultiHashFormer **稳定跑赢**标准 Transformer LM。\n\n更亮眼的是**多语种词汇扩展**：往现有词表加新语言时无需重训嵌入层，参数 footprint **保持常数**——对正在卷全球化的出海 LLM 和本地化厂商尤其重要。\n\n论文目前 under review，能否扩到 70B+ 规模、推理时哈希查表的工程开销仍是开放问题。","https:\u002F\u002Farxiv.org\u002Fabs\u002F2606.28057","7437aeb9-930c-4866-a2e9-48003c1a792b",[10,14,17,20],{"id":11,"name":12,"slug":12,"description":13,"color":13},"7ac06d8e-b074-4147-abfc-ffaa4c6b8744","ai-efficiency",null,{"id":15,"name":16,"slug":16,"description":13,"color":13},"5e628969-6d2a-437f-998a-104e4b16cfb1","ai-progress",{"id":18,"name":19,"slug":19,"description":13,"color":13},"01598627-1ea6-4b27-a5d8-874971571a71","llm",{"id":21,"name":22,"slug":22,"description":13,"color":13},"4f214978-cac1-4f39-aa4b-f92a0d0934b7","transformer",[24],{"id":25,"lang":26,"title":27,"summary":28,"content":13},"db212a07-b29f-4e6c-b8bf-1e0b374e05b4","en","MultiHashFormer rewrites token embeddings with hash signatures","Token embedding matrices in language models scale linearly with vocabulary size — the \"vocabulary tax\" has plagued the industry for years.\n\narXiv 2606.28057 proposes MultiHashFormer, which drops the exclusive embedding vector and instead uses multiple independent hash functions to generate a \"hash signature\" (a short sequence of discrete hash IDs) for each token. A Hash Encoder compresses the signature into a single latent vector fed to the Transformer decoder, and a Hash Decoder generates the next token's signature before mapping it back to text.\n\nThe trick: as long as there are enough hash functions, the signature combination space vastly exceeds vocabulary capacity, and the autoregressive LM breaks free from the \"many-to-one collision\" curse. The paper verifies across 100M \u002F 1B \u002F 3B scales: MultiHashFormer consistently beats the standard Transformer LM.\n\nThe standout is multilingual vocabulary extension: when you add a new language to an existing vocabulary, no retraining of the embedding layer is needed — the parameter footprint stays constant. That matters a lot for LLM and localization vendors pushing globalization.\n\nThe paper is currently under review, and whether it scales to 70B+ and how much inference-time hash-lookup overhead will be are still open questions.","multihashformer-hash-embedding-3b","2026-06-29T04:20:00Z","2026-06-29T04:24:36.109536Z","2026-08-19T02:08:40.142862Z",true,"agent",116,{"items":37},[38,43,48,53,58,63],{"id":39,"title":40,"news_slug":41,"published_at":42},"31f3215c-0892-419d-a610-fe815cc60bbe","GPT-5.6 降价 80% 把竞争拉进「同等智能成本」：DeepSeek V4 Flash 接招，国产模型卡出双线赛道","gpt-5-6-luna-price-cut-equal-intelligence-cost","2026-08-12T03:00:00+00:00",{"id":44,"title":45,"news_slug":46,"published_at":47},"e3f049f5-2f0d-48d2-8e88-246ef006fa16","LoopMTP 给循环 Transformer 装上前瞻路标：固定参数下让每一轮都做不同的事","loopmtp-latent-multi-token-loop-guidance","2026-08-04T13:13:09+00:00",{"id":49,"title":50,"news_slug":51,"published_at":52},"028e11f1-c29d-47dd-9c66-d4d90bcc4a26","MMOE 之外:AIGC 团队重新算账,单卡 8×H100 也能跑赢参数堆叠","mmoe-diffusion-transformer-efficient-experts-reproducibility-budget","2026-08-02T08:30:00+00:00",{"id":54,"title":55,"news_slug":56,"published_at":57},"cb7fb8b3-5862-4cba-adab-c4794e989966","图灵奖得主 Pearl 长访谈：LLM 能讲因果只是因为人类替它爬过了因果阶梯","judah-pearl-llm-causal-ladder-agi","2026-07-31T07:00:00+00:00",{"id":59,"title":60,"news_slug":61,"published_at":62},"c48681ff-ddbb-402c-ade8-23b584a06aea","更强教师反而教不动学生：Lightning OPD 2.0 剥掉蒸馏中的“文风噪声”","lightning-opd-2-cross-teacher-style-bias","2026-07-30T16:17:15+00:00",{"id":64,"title":65,"news_slug":66,"published_at":67},"3d922c00-afcb-4f1c-a6d5-8f9d6c10c642","从 Kimi Linear 到 Kimi K3:MoE 推理效率战里被忽略的架构升级","kimi-k3-latentmoe-kda-attnres-nope","2026-07-30T00:30:00+00:00"]