Trust the Direction, Search the Step: Zero-and-First-Order Methods for LLM Fine-Tuning

Paper recorded by Signals 4 on 2026-10-01 in cs.LG. Abstract reproduced from arXiv; link to the original below.

Published 2026-10-01 on arXiv · recorded by Signals 4 on 2026-10-02

Category: cs.LG · 机器学习 · first seen 2026-10-02

Abstract

Step-size selection remains a central challenge in large-scale neural network optimization; conservative steps slow convergence, while aggressive steps can destabilize it. We combine \textbf{Z}ero-and-\textbf{F}irst-\textbf{O}rder optimization~(ZFO) and propose a lightweight framework that decouples direction selection from step-size. ZFO uses a trusted first-order optimizer to determine the direc

Read on arXiv →