FuTransformer: FP8 prefill and INT4 decode for Llama-style inference on Hopper

Paper recorded by Signals 4 on 2026-10-16 in cs.AI. Abstract reproduced from arXiv; link to the original below.

Published 2026-10-16 on arXiv · recorded by Signals 4 on 2026-10-19

Category: cs.AI · 人工智能 · first seen 2026-10-19

Abstract

Transformer prefill reuses weights across tokens and is dominated by matrix multiplication and activation data movement; single-token decode has limited reuse and is dominated by weight and KV-cache reads. We present FUTRANSFORMER, a CUDA implementation of a Llama-style Transformer block for NVIDIA Hopper GPUs. It applies CODA's GEMM-plus-epilogue approach to inference, adding direct KV-cache plac

Read on arXiv →