Delta-Matching aims to unlock 2× FP8 training throughput for large language models
AI Summary- MIT and Carnegie Mellon researchers introduced Delta-Matching for native FP8 training of large language models.
- NVIDIA H100 FP8 tensor cores deliver 3,958 teraflops versus 1,979 teraflops for BF16.
- Tests covered models with 569 million, 1.67 billion, and 5.29 billion parameters.