Skip to content
Back

Radtrain

AI Research

Transformer training under simulated space radiation. With no defense, training crashes at 10x the low-Earth-orbit bit-flip rate. With fault-aware training it holds from 1x to 10,000x at a 3.2% quality cost.

Python
PyTorch
Transformers
Fault Tolerance
Space

Key features

  • About 6 bit flips are enough to crash training of a 94M-parameter model
  • Fault-aware training holds validation loss between 1.011 and 1.012 from 1x to 10,000x the low-Earth-orbit rate, against 0.980 clean
  • Only exponent bits matter. Mantissa and sign flips caused no degradation
  • Sparse bit flips work as a training-time defense. Dense Gaussian noise does not
  • A portable bit-flip injector for any PyTorch model, with rates calibrated to measured low-Earth-orbit radiation
  • 45 experiments on one H100, with every result in a single table

Radiation in orbit flips bits in GPU memory. Earlier work measured what that does to a model that is already trained. I wanted to know what it does while a model is still training, where one bad weight feeds into every later step.

What I found

With no defense, training crashed at 10x the low-Earth-orbit rate. A single flipped exponent bit can do it.

The defense that worked was fault-aware training: flip one random bit on purpose at every training step. The optimizer learns weights that survive a flip. With that and a repair step that zeroes any NaN or infinite weight, training held from 1x to 10,000x the orbital rate. The price was 3.2% in quality, and it did not grow with the radiation level.

What’s in the repo

The training script, the bit-flip injector, the results table, and a paper that walks through all 45 experiments. I also wrote it up as a post, Training in orbit.

Resume (PDF)