Attention, Dynamic Wave and Hybrid Sequence Mixers in Language Models

Paper recorded by Signals 4 on 2026-10-29 in cs.LG. Abstract reproduced from arXiv; link to the original below.

Published 2026-10-29 on arXiv · recorded by Signals 4 on 2026-10-30

Category: cs.LG · 机器学习 · first seen 2026-10-30

Abstract

Large Language Models (LLMs) rely on Multi-Head Attention (MHA) for sequence modeling, but scaling to long contexts is constrained by the quadratic cost of attention, motivating sub-quadratic recurrent alternatives. This paper formalizes a Dynamic Wave recurrent layer, based on input-dependent exponential damping and cosine phase modulation, and evaluates three 10.94M-parameter configurations (Pur

Read on arXiv →