Persistent Recurrent Memory Between Transformer Layers - Improves Language Model Generalization

Paper recorded by Signals 4 on 2026-09-15 in cs.CL. Abstract reproduced from arXiv; link to the original below.

Published 2026-09-15 on arXiv · recorded by Signals 4 on 2026-09-16

Category: cs.CL · 自然语言处理 · first seen 2026-09-16

Abstract

We introduce a simple architectural modification to decoder-only transformers: a persistent recurrent state that observes hidden representations via cross-attention, updates itself through a GRU, and modulates subsequent processing via gated addition. Inserted between the lower and upper halves of a 6-layer transformer, this module adds only 3.7\% additional parameters while reducing evaluation lo

Read on arXiv →