Training in Orbit

There’s an idea floating around that we should put GPUs in space. Google published a paper about a TPU constellation, Starcloud launched the first H100 into orbit last November, and honestly the pitch is hard to argue with, free solar power, free cooling by radiating heat into the void, no land use, no neighbors to annoy with your megawatt power draw.

The part nobody talks about is that space is trying to kill your hardware.

Cosmic rays and solar particles slam into silicon and flip bits. These are called Single Event Upsets and they happen at a measurable rate, on the Chaohu-1 satellite researchers measured $4.76\times 10^{-7}$ errors/bit/day in orbit. That number looks harmless until you multiply it out. A 94 million parameter model in bfloat16 is about $2.5\times 10^9$ total bits, so you’re looking at roughly 1,200 random bit flips per day.

Now you might be thinking, so what, computers deal with errors all the time. For inference you’d be right, a flipped bit produces one wrong answer, you move on. Training is different. A flipped bit corrupts a gradient, that gradient gets baked into every weight, those weights produce the next gradient, so the error doesn’t just sit there, it compounds.

I wanted to know exactly how fragile training actually is, so I built a simulation.

The setup

I took a 94M parameter GPT-style transformer from the autoresearch project and added a bit-flip injector. At each training step, with probability calibrated to the measured LEO radiation rate, I flip a random bit in a random weight, uniform across all 16 bits of the bfloat16 representation. Then I swept the radiation rate from 1x realistic LEO up to 10,000x and watched where training breaks.

Six flips is all it takes

Going in I figured the model would degrade gracefully. Neural nets are famously overparameterized, one weight out of 94 million shouldn’t matter, that was my whole intuition. I was wrong. At 10x the realistic LEO rate, about 42 bit flips over a 5 minute training run, the model doesn’t degrade, it crashes. Loss goes to NaN and training is dead.

The reason is bfloat16’s format. The 16 bits break down as:

$\text{[sign: 1 bit]}\text{[exponent: 8 bits]}\text{[mantissa: 7 bits]}$

A single flip in the exponent field can multiply a weight by $2^{128}$, that’s a value around $10^{38}$ sitting in the middle of your network, and it propagates through the next forward pass, blows up the attention scores, and produces NaN gradients that instantly destroy every parameter in the model.

At $1\times$ realistic LEO the model barely notices, about 4 flips in 5 minutes and the degradation is less than $0.03\%$. But extrapolate to a 24 hour training run and you’d accumulate $\sim 1,200$ flips, enough to crash undefended training many times over.

The obvious defense doesn’t work well enough

You might say just check for NaNs and zero them out, and that’s exactly what I tried first. After each bit flip, check if the result is NaN or infinity, zero it out, keep going. It helps, it extends the survivable radiation rate by roughly 100x, and for a while I thought that was the answer. Then I looked at the quality numbers. Most exponent flips don’t produce NaN, they produce large-but-finite values that pass the check, accumulate, and gradually rot the model from the inside.

At $100\times$ LEO with NaN repair the model finishes training, but with a val_bpb of $2.11$ compared to $0.98$ clean. That’s barely functional. Technically alive, practically useless.

The fix that actually works

Here’s where it gets interesting, and I’ll be honest, this felt wrong the first time I tried it. I added one line of logic to the training loop: before each forward pass, flip one random bit in one random weight. That’s it. Train with bit flips on purpose. Deliberately injecting the exact fault you’re afraid of, into every single step.

I call this Fault-Aware Training, and the results surprised me.

The FAT-trained model achieves $1.011$ val_bpb with no radiation, a $3.2\%$ quality cost from clean. Then I tested it at $10\times$ LEO. Still $1.011$. At $100\times$, still $1.012$. At $1,000\times$, the rate where NaN repair alone crashes instantly, still $1.012$. At $10,000\times$, $1.012$. At some point I stopped expecting the number to move.

The model becomes radiation-invariant across four orders of magnitude:

$\text{val\_bpb}(r)\approx 1.011\forall r\in [1\times ,10,000\times \text{ LEO}]$

I should be precise about what’s happening here, because “the model learned to handle bit flips” is not it. FAT pushes the optimizer into a flat region of the loss landscape where individual weight perturbations don’t matter much. The mechanism is related to Sharpness-Aware Minimization and other flat-minima methods, but with a difference that turns out to matter a lot: the perturbation is sparse and heavy-tailed, matching the actual physics of radiation, rather than smooth and Gaussian.

You might ask, if it’s just flat minima, wouldn’t any noise during training do the same thing? I asked the same thing, so I tested it directly. Replacing bit-flip noise with Gaussian noise during training does not produce radiation resilience, the model crashes at 100x LEO just like an undefended one. The perturbation has to be sparse and occasionally extreme, a few random weights getting large shocks, not every weight getting a small nudge.

Only half the bits matter

I ran a separate set of experiments where I restricted flips to specific bit positions, and the results came out cleaner than I expected.

Mantissa flips (bits 0-6, 44% of all bits): 434 flips at 100x LEO caused zero measurable degradation. Literally zero.

Sign flips (bit 15, 6% of all bits): also zero. Flipping a weight’s sign is a large perturbation, you’d think it would hurt, but the optimizer corrects for it within a few steps.

Exponent flips (bits 7-14, 50% of all bits): these account for 100% of the damage.

This has a practical implication. If you could protect just the exponent byte of each bfloat16 weight with ECC, you’d eliminate essentially all radiation vulnerability while protecting only half the bits, which is a much cheaper hardware solution than protecting everything.

The variance result

This might be the most practically significant finding, and it’s not about the average, it’s about the spread. Without FAT, the outcome of training under radiation is a gamble. I ran three seeds at the same radiation rate:

Same rate, same code, and one seed comes out fine while another comes out barely usable, because whether your model works depends on whether a flip happens to land on a critical exponent bit in a critical weight. You can’t predict it in advance.

With FAT, all seeds converge to $\approx 1.010$, the standard deviation drops from $\sigma \approx 0.05$ to $\sigma \approx 0.001$.

For a satellite this is the difference between “training might work” and “training will work”. You can’t retry a training run from orbit, the system has to be reliable on the first attempt, and FAT turns radiation from a high-variance gamble into a predictable, fixed tax.

The other constraints

Radiation isn’t the only problem with training in orbit, so I simulated two more.

Eclipse power cycling. LEO satellites lose power for about 30 minutes every 90 minute orbit. I simulated this by resetting all optimizer state every 200 training steps, expecting it to hurt, and it basically didn’t, about 0.002-0.004 val_bpb per cycle. The optimizer momentum rebuilds in a few steps.

Progressive degradation. Cumulative radiation damage increases error rates over a satellite’s lifetime, so I linearly ramped the flip rate from 1x to 10x over training. FAT handled it without any additional degradation.

The full combined simulation, FAT plus eclipse cycling plus progressive degradation plus $100\times$ radiation, produced a model with val_bpb of $1.016$. That’s $3.7\%$ worse than clean, trained entirely in software on a standard GPU, no hardware modifications.

What I learned

The big takeaway is that training in space is harder than inference in space, but not as hard as I expected going in. The vulnerability is real but narrow, it’s entirely about exponent bits in floating-point weights, and the defense is simple, train with one bit flip per step, repair NaN values immediately, and the model becomes effectively immune to radiation across a wide range.

There are things I can’t test in simulation. Vacuum cooling limits how many GPUs you can run, inter-satellite bandwidth limits distributed training, micrometeorite impacts are low-probability but catastrophic. Those are hardware problems and they need hardware answers, I have nothing useful to say about them from a software simulation.

But the software side, making training robust to the bit flips that will inevitably happen, seems surprisingly tractable. A few lines of code, a $3.2\%$ quality cost, and the model just works.

Will we actually train large models in space anytime soon? I don’t know, and I’m not going to pretend to, the economics are still challenging and there are easier ways to get more compute on the ground. What I am confident about is narrower: if the power and cooling economics ever work out, the bit-flip problem won’t be what stops us.

The code is at github.com/tylergibbs1/radtrain. Forty-five experiments, three saved model checkpoints, and a portable bit-flip injection library you can use with any PyTorch model.

---

References

Space-based AI infrastructure:

• Exploring a Space-Based, Scalable AI Infrastructure System Design. Google’s TPU constellation proposal. First published radiation test results for a cloud AI accelerator. Survived 15 krad TID with no hard failures.
• Starcloud Trains First AI Model in Space. First H100 in orbit, November 2025. Successfully ran nanoGPT training on a 60 kg satellite.

Radiation effects on hardware:

• RedNet: A Case for Application-Aware Space Radiation Tolerance. Measured LEO SEU rate of $4.76\times 10^{-7}$ errors/bit/day on Chaohu-1 satellite. The radiation rate used in all experiments in this post.
• Understanding Silent Data Corruption in LLM Training. Meta’s study showing 6 SDC events in a 54-day training run on terrestrial hardware. Training diverges silently.
• A Single Bit-Flip Destroys Half of an LLM’s Accuracy. Demonstrates that one bit flip in a quantized LLM can drop accuracy from 73.5% to 0%.
• GPU Resilience at Scale: H100 vs A100. H100s show 3.2x lower MTBE for memory errors vs A100. 5% overprovisioning needed.

Radiation defenses (inference-focused, pre-dating this work):

• Fault-Aware Training for SEU Mitigation. Injects bit flips during training to improve inference resilience. Only validated in simulation, only for inference. Our work extends FAT to pretraining.
• Rotated Robustness: Training-Free Defense Against Bit-Flip Attacks on LLMs. Uses orthogonal transforms to smooth activation outliers. Inference-only.
• Enhancing DNN Robustness Through Saturated Activation Functions. Bounds weights via tanh during training. Related to our finding that logit softcap acts as a radiation defense.

Flat minima and heavy-tailed noise (theoretical grounding):

• A Tail-Index Analysis of Stochastic Gradient Noise in Deep Networks. Simsekli et al. showed SGD gradient noise is heavy-tailed, not Gaussian. Heavier tails find flatter minima.
• Hausdorff Dimension, Heavy Tails, and Generalization. Connects tail index of optimization noise to fractal dimension of minima. Foundational for understanding why bit-flip noise (heavy-tailed) produces different minima than Gaussian noise.

Orbital edge computing:

• Bringing Federated Learning to Space. First systematic FL study adapted for orbital dynamics. 768 constellation configurations, 9x speedup.
• A Comprehensive Survey on Orbital Edge Computing. Full stack survey of compute in orbit.
• FPGA-Based Neural Network Accelerators for Space: A Survey. FPGAs dominate actual space deployments due to radiation tolerance through scrubbing.

Training infrastructure (this work builds on):

• autoresearch. Karpathy’s autonomous research framework. The training script and model architecture used in all experiments.