Lyapunov Exponents for Attention Composition

AI Research January 9, 2026

First Lyapunov exponent framework for eigenvalue dynamics in composed attention layers. A bridge from transformer theory to dynamical systems.

Python

Deep Learning

Transformers

Dynamical Systems

Research Paper

Key Features

First computation of full Lyapunov spectrum for attention products
Proof that Lambda_1 = 0 exactly and Lambda_k < 0 for k > 1
Quantified relationship between temperature and spectral gap
Refined closed-form formula for rank collapse prediction
Analysis of the residual connection mechanism (2.4x contraction reduction)
All theoretical results experimentally verified

Abstract

I develop the first Lyapunov exponent framework for the analysis of eigenvalue dynamics in composed attention layers. The framework builds on foundational rank collapse results. It provides novel tools that connect transformer theory to dynamical systems.

Theoretical Framework

For a sequence of attention matrices $A_{1}, A_{2}, \dots, A_{L}$ , the $k$ -th Lyapunov exponent is:

$Λ_{k} = lim_{L \to \infty} \frac{1}{L} lo g ∣ σ_{k} (P_{L}) ∣$

Here, $σ_{k}$ is the $k$ -th singular value of the product $P_{L} = A_{1} A_{2} \dots A_{L}$ .

Key Results

Theorem 1: Dominant Lyapunov Exponent

For any sequence of row-stochastic attention matrices:

$Λ_{1} = 0$

Proof: For any stochastic $A$ , the all-ones vector $1$ satisfies $A 1 = 1$ . As a result, $Λ_{1} = lim_{L \to \infty} \frac{1}{L} lo g (1) = 0$ .

Theorem 2: Contraction Exponents

For i.i.d. random attention matrices with spectral gap $γ < 1$ :

$Λ_{k} < 0 for all k > 1$

Theorem 3: Collapse Prediction Formula

The number of layers until effective rank drops below threshold $r$ is:

$L_{collapse} = \frac{l o g ( \frac{r - 1}{d - 1} )}{l o g γ}$

Here, $d$ is the dimension. $γ = ∣ λ_{2} ∣$ is the magnitude of the second eigenvalue.

Experimental Results

Lyapunov Spectrum (d=50, T=1.0, L=100 layers):

$Λ_{1} = 0.000000$ (std dev $< 1 0^{- 15}$ ), verified to machine precision
$Λ_{2} = - 1.790$ (std dev $0.013$ )
$Λ_{3} = - 1.805$ (std dev $0.011$ )

Temperature Effect on Spectral Collapse

A lower temperature causes sharper attention and faster collapse:

T = 0.5: $∣ λ_{2} ∣ = 0.417$ , $Λ_{2} = - 0.875$ (Slowest)
T = 1.0: $∣ λ_{2} ∣ = 0.195$ , $Λ_{2} = - 1.636$ (Moderate)
T = 2.0: $∣ λ_{2} ∣ = 0.080$ , $Λ_{2} = - 2.524$ (Fast)

Residual Connection Analysis

Residual connections reduce $∣ Λ_{2} ∣$ by a factor of $\approx 2.4 \times$ . This reduction slows information loss through the layers.

With residual connections, the gradients at layer 1 improve from $5.5 \times 1 0^{- 7}$ to $2.7 \times 1 0^{- 3}$ . This is a $\sim 5000 \times$ improvement.

Citation

@misc{gibbs2026lyapunov,
  title={Lyapunov Exponents for Attention Composition: A Dynamical Systems Perspective on Deep Transformers},
  author={Gibbs, Tyler},
  year={2026},
  publisher={Zenodo},
  doi={10.5281/zenodo.18202128}
}