Is Weight Tying Still Beneficial for Decoder-Only LLMs in Private Settings Under DP-SGD?

Paper recorded by Signals 4 on 2026-09-30 in cs.LG. Abstract reproduced from arXiv; link to the original below.

Published 2026-09-30 on arXiv · recorded by Signals 4 on 2026-10-01

Category: cs.LG · 机器学习 · first seen 2026-10-01

Abstract

Differentially Private Stochastic Gradient Descent (DP-SGD) is a leading approach for privacy-preserving fine-tuning of large language models (LLMs). Many decoder-only LLMs employ weight tying between input and output embeddings, a design choice originally introduced for parameter efficiency and improved language modeling performance in the non-private setting. However, the impact of weight tying

Read on arXiv →