Bridging the Gap Between Homogeneous and Heterogeneous Asynchronous Optimization Is Surprisingly Difficult

Paper recorded by Signals 4 on 2026-09-15 in cs.LG. Abstract reproduced from arXiv; link to the original below.

Published 2026-09-15 on arXiv · recorded by Signals 4 on 2026-09-16

Category: cs.LG · 机器学习 · first seen 2026-09-16

Abstract

Modern large-scale machine learning tasks often require multiple workers, devices, CPUs, or GPUs to compute stochastic gradients in parallel and asynchronously to train model weights. Theoretical results typically distinguish between two settings: (i) the homogeneous setting, where all workers have access to the same data distribution, and (ii) the heterogeneous setting, where each worker operates

Read on arXiv →