Why I'm Learning AI as a Physicist: Optimization, Energy, and Continuous Math

1. The Illusion of Disconnection

When software engineers and computer scientists approach deep learning, the field is often framed through an imperative, algorithmic mindset: layers of matrix multiplications, backwards autodiff passes, and learning rate schedulers.

To a physicist, however, a modern deep neural network does not feel like an algorithm at all. It feels like a high-dimensional dynamical system governed by statistical mechanics and potential field theory.

The foundational equations that power modern AI are not native to computer science — they were discovered centuries ago by Hamilton, Lagrange, Boltzmann, and Gauss.

2. Loss Surfaces as Potential Energy Wells

In classical Newtonian mechanics, a physical particle in a conservative force field experiences a potential energy function $V(\mathbf{x})$. The force $\mathbf{F}$ acting on that particle is strictly the negative gradient of potential energy:

$\mathbf{F} = -\nabla V(\mathbf{x})$

Now look at standard gradient descent in machine learning. We define a scalar loss function $\mathcal{L}(\mathbf{w})$ over a weight vector $\mathbf{w} \in \mathbb{R}^d$. The parameter update rule is:

$\mathbf{w}_{t+1} = \mathbf{w}_t - \eta \nabla \mathcal{L}(\mathbf{w}_t)$

In the continuous limit as step size $\eta \to 0$, this is simply an overdamped particle settling into a gravitational potential well:

$\frac{d\mathbf{w}}{dt} = -\nabla \mathcal{L}(\mathbf{w})$

The parameter space is identical to a physical configuration manifold. When an optimizer escapes saddle points or navigates narrow ravines, it is obeying the exact laws of dissipative fluid dynamics.

3. Momentum as Newtonian Inertia

Standard gradient descent frequently oscillates across steep canyon walls rather than making progress down the central valley. In 1986, Polyak momentum was introduced to deep learning:

$\mathbf{v}_{t+1} = \beta \mathbf{v}_t + (1 - \beta) \nabla \mathcal{L}(\mathbf{w}_t)$ $\mathbf{w}_{t+1} = \mathbf{w}_t - \alpha \mathbf{v}_{t+1}$

To a physicist, this is nothing other than Newton's Second Law with a viscous friction drag coefficient:

$m \frac{d^2 \mathbf{x}}{dt^2} + \gamma \frac{d\mathbf{x}}{dt} = -\nabla V(\mathbf{x})$

Where:

• $m$ is the virtual particle mass.
• $\gamma$ represents viscous damping (analogous to $1 - \beta$).
• $\nabla V(\mathbf{x})$ is the conservative restoring force.

By integrating this second-order ordinary differential equation (ODE), the optimizer accumulates kinetic momentum in consistent directions while averaging out orthogonal, high-frequency oscillations.

4. Softmax is the Maxwell-Boltzmann Distribution

One of the most profound realizations occurs when deriving the standard attention softmax function. Consider the probability $P(i)$ of selecting a discrete state $i$ given logit energies $z_i$:

$P(i) = \frac{\exp(z_i / \tau)}{\sum_j \exp(z_j / \tau)}$

Compare this directly to the Maxwell-Boltzmann distribution of statistical physics, which defines the probability of an ideal gas particle occupying energy state $E_i$ at thermodynamic temperature $T$:

$P(E_i) = \frac{\exp(-E_i / k_B T)}{\mathcal{Z}}, \quad \text{where } \mathcal{Z} = \sum_j \exp(-E_j / k_B T)$

The mathematical equivalence is exact:

1. Negative energy $-E_i$ corresponds to the dot-product similarity logit $z_i = \mathbf{q}^T \mathbf{k}_i$.
2. The thermodynamic temperature $k_B T$ corresponds exactly to the attention temperature $\tau = \sqrt{d_k}$.
3. The partition function $\mathcal{Z}$ is the denominator normalizer that guarantees conservation of probability.

When temperature $\tau \to 0$, the system collapses into a zero-entropy ground state (argmax / Dirac delta). When $\tau \to \infty$, the thermal fluctuations dominate and the distribution converges to maximum entropy (uniform random noise).

5. The First-Principles Advantage

Viewing artificial intelligence through first principles changes how you debug and design models:

• Diffusion models are not random denoising tricks; they are time-reversed Langevin dynamics integrating non-equilibrium thermodynamics.
• Normalizing flows are invertible Hamiltonian phase-space transformations preserving symplectic volume (Liouville's theorem).
• Residual connections (ResNets) are Euler discretizations of continuous neural ODEs: $\mathbf{x}_{t+1} = \mathbf{x}_t + f(\mathbf{x}_t)$.

When you stop treating machine learning as black-box code and start treating it as applied geometry and physics, the entire field becomes intuitive, predictable, and beautiful.