Low-Bit N-Gram Embeddings for Language Models

Paper recorded by Signals 4 on 2026-10-22 in cs.AI. Abstract reproduced from arXiv; link to the original below.

Published 2026-10-22 on arXiv · recorded by Signals 4 on 2026-10-23

Category: cs.AI · 人工智能 · first seen 2026-10-23

Abstract

Many recent open-weight language models use n-gram embedding tables to improve performance. These tables store learned representations of short token sequences, such as pairs or triples of consecutive tokens, which the model retrieves during processing. However, they can require substantial memory: DeepSeek-V4.1-Flash, for example, allocates 196 billion parameters to its Engram n-gram memory. We t

Read on arXiv →