Knowingly Wrong: Training LLMs Against Their Own Beliefs can Induce Misalignment

Paper recorded by Signals 4 on 2026-11-03 in cs.AI. Abstract reproduced from arXiv; link to the original below.

Published 2026-11-03 on arXiv · recorded by Signals 4 on 2026-11-04

Category: cs.AI · 人工智能 · first seen 2026-11-04

Abstract

Fine-tuning a large language model (LLM) on harmful data in a narrow domain, such as bad medical advice, can make it broadly misaligned. How realistic this risk is remains unclear, since models are very rarely fine-tuned on explicitly harmful data in practice. In this work, we show that fine-tuning on data the model believes to be false but that has no harmful content, can also induce broad misali

Read on arXiv →