Everything is Math · Athar Qadri

It’s strange how little attention (pun intended) gets paid to the math that is actually powering the rise of LLMs. To give you an idea, the advent of LLMs was around 2017 when researchers at Google published a ground-breaking paper¹, but the one algorithm (out of many) that made it possible had already been lying there for around 170 years.² While everyone’s chasing the latest AI breakthrough, I dug into the math that actually powers LLMs, and it’s amazing how simple a function such as ReLU can help power a system that mimics a human brain and does it pretty well.

This write-up is an appreciation to all the mathematians that have made it possible.

ReLU (Rectified Linear Unit)

In an LLM, this tiny rule is applied to activate a neuron through a simple rule: ‘If the number is negative, make it zero. If it’s positive, do nothing.’

Without something like ReLU (or its cousins), the entire LLM would collapse into one big linear equation and couldn’t learn complex language patterns.

ReLU activation function: flat at zero for negative inputs, rising linearly for positive inputs

Euler’s Constant (e)

2.7182817...

The constant itself had already been there for decades but it was Leonhard Euler who gave it a name. In 1727, when he was just 21 years old, Euler used the letter e for it in a paper recording his own experiments on the firing of a cannon.³ The paper wasn’t published until 1862, nearly 80 years after his death.

From being used in financial calculations to radioactive decay, it has found its way into the world of AI and machine learning and that brings us to Sigmoid Functions.

Sigmoid Function

The first sigmoid function was introduced by Pierre François Verhulst, a Belgian mathematician, in year 1838 in his paper ⁴ on population growth. LLMs use the standard logistic function, a type of sigmoid function, that maps any real input onto a value between zero and one:

\sigma(x) = \frac{1}{1 + e^{-x}}

Where ReLU is sharp and one-sided, sigmoid is smooth and symmetric.

Sigmoid function: S-shaped curve

Because its output is always bounded in $(0, 1)$ , sigmoid is useful when you need a probability: the neuron’s decision making ability that a token belongs to a given class, for example.

Softmax

\text{softmax}(z_i) = \frac{e^{z_i}}{\sum_{j=1}^{n} e^{z_j}}

Sigmoid handles a single binary decision: it gives you one probability of how likely the answer is between two options. For an LLM to choose its next token from its vocabulary, it needs to give it a probability distribution. That’s what softmax is for: it takes a whole list of raw scores (called logits) and turns them into a proper probability distribution, where every value sits between 0 and 1 and the whole set adds up to 1.

Let’s say the model is finishing the sentence “The cat sat on the ___.” Before softmax, each word just has a raw score. After softmax, those scores become a probability distribution the model can actually sample from:

Token	Logit (before softmax)	Probability (after softmax)
mat	2.5	0.710
floor	1.0	0.158
chair	0.5	0.096
roof	-0.5	0.035

The term “softmax” wasn’t coined until 1989, by researcher John S. Bridle, who described it as a smooth, differentiable stand-in for simply picking the largest value.⁵ But the shape of the function is much older and the physicists were already using in the 1860s to describe how particles spread across energy states.

To be continued…

ReLU (Rectified Linear Unit)

Euler’s Constant (e)

Sigmoid Function

Softmax

Footnotes