semi-sentient code

The Gate in the Middle of the Block

· 36 min read

tagged llm · deep-learning · algorithms

contents (8 sections)
  1. What a nonlinearity is for
  2. Saturation
  3. ReLU
  4. GELU as an expectation
  5. Swish
  6. The gate
  7. The signal through depth
  8. Where it is going

I was reviewing Llama’s PyTorch code and came across silu in the feed-forward layer:

return self.w2(F.silu(self.w1(x)) * self.w3(x))

It has been a while since I programmed in PyTorch, and admittedly I have not kept up with activation functions. That line is what made me want to figure out what this function is and why Llama uses it.

Historically I would have written w2(relu(w1(x))): two matrices with ReLU between them, which is what the 2017 transformer used.1 Llama’s line has a third matrix, and it multiplies two projections of the input where ReLU had a threshold on one. The layer is called SwiGLU. In between, GPT-2 used GELU and PaLM introduced SwiGLU to large language models. Llama 7B has 6.7 billion parameters, and 4.3 billion of them are in these layers.2

I took this as an opportunity to write about how and why the world moved from ReLU to SwiGLU. The post covers the sigmoid’s shrinking gradient, He initialization for ReLU, GELU as the expected value of a random gate, and the product of two input features xixjx_i x_j that one gated unit computes exactly. It ends with the published comparisons since 2020, in which the multiplication accounts for most of the measured gain and the choice of Swish inside the gate for about 0.010.01 nat. Every demo below runs in the page.

What a nonlinearity is for

Before getting into activation functions, it helps to be clear on what the feed-forward layer is doing in the first place. A transformer block has two parts. Attention lets each token look at the other tokens and pull in information from them, but what it pulls in is a weighted average of their value vectors. The feed-forward layer comes next and works on each token by itself, with no view of its neighbors. It is where the model transforms what attention gathered, and it holds most of the parameters. Geva and colleagues looked inside trained models and found these layers behave like key-value memories: the first matrix detects patterns in the input, and the second writes out what tends to follow them.3

We can define the feed-forward layer as

FFN(x)=f(xW1) W2,\mathrm{FFN}(x) = f(x W_1)\thinspace W_2,

where x∈Rdx \in \mathbb{R}^d is one token’s vector, W1∈Rd×hW_1 \in \mathbb{R}^{d \times h} projects it up to a hidden width hh, W2∈Rh×dW_2 \in \mathbb{R}^{h \times d} projects it back down, and ff is the activation function, applied to each of the hh hidden values on its own. I leave out biases because the models in this post leave them out.4

So why do we need ff at all? Take it away and the layer is xW1W2x W_1 W_2. The product W1W2W_1 W_2 is a single d×dd \times d matrix MM, so the layer computes xMxM, and we could have stored MM with d2d^2 parameters instead of 2dh2dh. Stacking more of these layers does not help either: two of them compute xM1M2x M_1 M_2, which is again one matrix. A single matrix cannot compute even something as simple as ∣x1∣|x_1|, because a linear map that sends x1=1x_1 = 1 to 11 has to send x1=−1x_1 = -1 to −1-1. Put a ReLU in the middle and two hidden units are enough: ∣x1∣=ReLU(x1)+ReLU(−x1)|x_1| = \mathrm{ReLU}(x_1) + \mathrm{ReLU}(-x_1).

Sigmoid, tanh and ReLU are all nonlinear, and for representing functions any of them will do. Leshno and colleagues proved in 1993 that one hidden layer can approximate any continuous function on a bounded region, given enough units, as long as ff is not a polynomial.5 Where they differ is in what they do to the gradient during training. Backpropagation multiplies the gradient by f′(z)f'(z) at every layer it passes through, so through LL layers the gradient picks up LL such factors. If each factor is 1/41/4, twelve layers scale the gradient by 4−124^{-12}, about 6×10−86 \times 10^{-8}. If each is 11, the gradient arrives at the size it left.

Saturation

The first activation most of us learned was the sigmoid. It squashes any input into the range (0,1)(0, 1), which made it a natural fit for a unit that is either on or off. We can write it and its derivative as

σ(x)=11+e−x,σ′(x)=σ(x) (1−σ(x))≤14.\sigma(x) = \frac{1}{1 + e^{-x}}, \qquad \sigma'(x) = \sigma(x)\thinspace (1 - \sigma(x)) \le \tfrac{1}{4}.

To see where the derivative comes from, write σ(x)=(1+e−x)−1\sigma(x) = (1 + e^{-x})^{-1} and apply the chain rule:

σ′(x)=e−x(1+e−x)2=11+e−x⋅e−x1+e−x.\sigma'(x) = \frac{e^{-x}}{(1 + e^{-x})^2} = \frac{1}{1 + e^{-x}} \cdot \frac{e^{-x}}{1 + e^{-x}}.

The first factor is σ(x)\sigma(x). The second is 1−σ(x)1 - \sigma(x), because e−x1+e−x=(1+e−x)−11+e−x=1−11+e−x\frac{e^{-x}}{1 + e^{-x}} = \frac{(1 + e^{-x}) - 1}{1 + e^{-x}} = 1 - \frac{1}{1 + e^{-x}}. So σ′(x)=σ(x)(1−σ(x))\sigma'(x) = \sigma(x)(1 - \sigma(x)).

Now call s=σ(x)s = \sigma(x), which is always between 00 and 11. The derivative is s(1−s)=s−s2s(1 - s) = s - s^2, a downward parabola in ss. Setting its derivative to zero gives 1−2s=01 - 2s = 0, so the peak is at s=12s = \tfrac{1}{2}, and the value there is

12(1−12)=14.\tfrac{1}{2}\left(1 - \tfrac{1}{2}\right) = \tfrac{1}{4}.

The sigmoid equals 12\tfrac{1}{2} only at x=0x = 0, so that is where the derivative peaks. Anywhere else ss is further from 12\tfrac{1}{2} and s(1−s)s(1 - s) is smaller.

The derivative is the problem. It peaks at a quarter when x=0x = 0 and falls off exponentially on both sides. During backpropagation the gradient gets multiplied by σ′(z)\sigma'(z) at every unit it passes through, so each sigmoid layer shrinks it by at least a factor of four, and by closer to ten for a unit sitting at z=2z = 2, where σ′\sigma' is 0.1050.105. Stack LL layers and the gradient is down by at least 4L4^{L} before we even count the weights.

You might think larger weights could make up for it. They cannot, because larger weights push the inputs zz out into the flat tails, where σ′\sigma' is close to zero and the gradient shrinks even faster. This is the vanishing gradient problem, and it is why deep sigmoid networks had to be pretrained one layer at a time, which the next section gets into.

tanh⁡\tanh helps a little. It is the same S-shaped curve, stretched and shifted so that tanh⁡(x)=2σ(2x)−1\tanh(x) = 2\sigma(2x) - 1. Its derivative, 1−tanh⁡2(x)1 - \tanh^2(x), reaches 11 at zero instead of a quarter, and its output is centred on zero, which the next layer handles better. The tails are still flat, though: at x=3x = 3 the derivative of tanh⁡\tanh is 0.010.01 and the sigmoid’s is 0.0450.045. Move the crosshair below past ∣x∣=3|x| = 3 and watch both derivatives go to nearly zero.

-101234-4-3-2-101234x
f   f′
move the pointer over the chart to read values
Sigmoid and tanh with their derivatives (dashed). The derivative of the sigmoid never exceeds one quarter. Both go flat past |x| of about 3.

ReLU

ReLU is the one I would have reached for, and it is about as simple as an activation gets: ReLU(x)=max⁡(0,x)\mathrm{ReLU}(x) = \max(0, x). It gets around the sigmoid’s problem by not having a flat tail on the right at all. For x>0x > 0 it passes the input straight through, so its derivative is 11. For x<0x < 0 it outputs zero, so its derivative is 00.

That matters because backpropagation multiplies the gradient at each unit by the derivative. With ReLU, a unit whose input was positive passes the gradient back at full size, and a unit whose input was negative passes back nothing. Nothing in between gets shrunk. Compare that with the sigmoid, which cuts the gradient by at least a factor of four at every unit.

In practice this changed how deep networks were trained. Before 2011, deep sigmoid networks were usually trained in two stages: first each layer on its own without labels, and only then the whole network on the actual task. Glorot, Bordes and Bengio showed that a deep ReLU network can skip the first stage. In their words, rectifier networks “can reach their best performance without requiring any unsupervised pre-training” on supervised tasks with large labeled datasets. They also liked the zeros. A ReLU layer produces “sparse representations with true zeros”, which they found suited naturally sparse data.6

ReLU has one more property that turns out to be useful. It is positively homogeneous: ReLU(ax)=a ReLU(x)\mathrm{ReLU}(ax) = a\thinspace \mathrm{ReLU}(x) for any a>0a > 0, so if we scale the input, the output scales by the same amount. GELU and Swish, which come later, do not have this. It means we can say exactly what a ReLU does to the size of a signal, regardless of how big the signal is.

Here is how. Say a pre-activation zz is Gaussian with mean zero and variance σ2\sigma^2. When z>0z > 0, ReLU(z)2=z2\mathrm{ReLU}(z)^2 = z^2, and when z<0z < 0 it is 00. The Gaussian is symmetric, so half of E[z2]\mathbb{E}[z^2] comes from the positive side and half from the negative side. ReLU keeps the positive half and drops the rest:

E ⁣[ReLU(z)2]=12 E[z2]=σ22.\mathbb{E}\negthinspace \left[\mathrm{ReLU}(z)^2\right] = \tfrac{1}{2}\thinspace \mathbb{E}[z^2] = \tfrac{\sigma^2}{2}.

So every ReLU layer halves the mean square of its input. That is true even when the weights are drawn the usual way that keeps a plain linear layer’s output the same size as its input. The halving compounds. After 20 layers the mean square is 2−202^{-20} of what went in, about a millionth, so values that started around 11 are now around 0.0010.001.

That is a problem for two reasons. First, the network’s output barely depends on its input. Every input has been scaled down by the same factor of a thousand, so the differences between inputs that the last layer needs to see have been scaled down too, and at initialization the model produces nearly the same output for everything. Second, training slows to a crawl. The gradient for a weight is the activation coming into it times the gradient coming back to it. When the activations feeding the later layers are a thousand times too small, so are the updates to those layers’ weights, and the backward pass shrinks the gradient on its way to the early layers by the same kind of factor. The deeper the network, the worse both get. If the layers doubled the mean square instead of halving it, the same compounding would run the other way, and after 20 layers values around 11 would be around 10001000 and still climbing.

He, Zhang, Ren and Sun fixed this in 2015, and the fix happens entirely at initialization: draw the starting weights a little larger, by exactly enough to cancel the halving. Their method is now called He initialization, and it is what PyTorch’s kaiming_normal_ implements.7

To see how much larger, we can follow the signal through one layer. A hidden unit with nn inputs computes

z=∑i=1nwixi.z = \sum_{i=1}^{n} w_i x_i.

Draw each weight wiw_i independently, with mean zero and variance vv, and independently of the inputs. Squaring zz gives a sum over every pair wixi wjxjw_i x_i \thinspace w_j x_j. For i≠ji \neq j the two weights are independent and each has mean zero, so those terms average to zero. Only the nn terms with i=ji = j survive, and each contributes E[wi2] E[xi2]=v E[x2]\mathbb{E}[w_i^2]\thinspace \mathbb{E}[x_i^2] = v\thinspace \mathbb{E}[x^2]:

E[z2]=n v E[x2].\mathbb{E}[z^2] = n\thinspace v\thinspace \mathbb{E}[x^2].

So the weights multiply the mean square by nvn v. Then the ReLU halves it, as we just showed, and the output of the layer has

E[ReLU(z)2]=12 n v E[x2].\mathbb{E}[\mathrm{ReLU}(z)^2] = \tfrac{1}{2}\thinspace n\thinspace v\thinspace \mathbb{E}[x^2].

We want that to equal E[x2]\mathbb{E}[x^2], the mean square that came in. That means 12 n v=1\tfrac{1}{2}\thinspace n\thinspace v = 1, so v=2/nv = 2/n. In practice we draw each weight from a Gaussian with mean zero and standard deviation 2/n\sqrt{2/n}. For a layer with 40964096 inputs that is 0.02210.0221. With that choice each layer hands the next a signal of the same size, so after 20 layers, or 100, values that started around 11 are still around 11.

The earlier standard, from Glorot and Bengio in 2010, used a variance of about 1/n1/n instead. Its derivation assumed the activation was linear, which is the right size for a layer with no ReLU and exactly a factor of two too small for a layer with one. For a layer with 40964096 inputs the difference is a standard deviation of 0.01560.0156 against 0.02210.0221. That looks minor, but it compounds the same way the halving does. He and colleagues tested both. A 22-layer network trained with either one, though theirs started reducing the error sooner. At 30 layers theirs still trained, and the older one stalled completely, with gradients they confirmed were all shrinking toward zero.

The paper also works through the backward pass. The gradient flowing back through the layer gets the same treatment, with nn counting the outputs of the layer instead of its inputs, and they show that matching either direction is enough to keep both under control. The variance section later checks this numerically and shows what happens with the smooth functions, which have no constant like this.

ReLU does have a cost, and it comes from the same zero that makes it work: the dead unit. If a hidden unit’s input is negative for every example in the data, its output is zero and its gradient is zero on every example. Gradient descent can never move it again, because the only way a gradient reaches that unit’s weights is through its own derivative, and that derivative is zero everywhere it gets evaluated. Lu and colleagues showed that with symmetric initialization, a deep enough ReLU network is dead with probability approaching one before training even starts, and they proposed asymmetric initializations to avoid it.8 Leaky ReLU and PReLU take a different route and give the negative side a small slope, so a unit that drifts negative can still recover.9

GELU as an expectation

GELU is what GPT-2 used. The name stands for Gaussian error linear unit, and the idea behind it comes from asking what ReLU and dropout have in common.10

Both of them multiply an input by a mask that is either 00 or 11. ReLU’s mask is 1[x>0]\mathbf{1}[x > 0]: it is 11 when the input is positive and 00 otherwise, so it is fixed by the input’s sign. Dropout’s mask is random. During training it zeroes each unit with some fixed probability, regardless of what the input is. Hendrycks and Gimpel asked for something in between, a mask that is random but where the probability depends on the input. Their choice was to multiply xx by

m∼Bernoulli(Φ(x)),m \sim \mathrm{Bernoulli}(\Phi(x)),

where Φ\Phi is the standard normal cumulative distribution function, the probability that a standard Gaussian lands below xx. A large positive input almost always passes, a large negative one almost never does, and an input near zero passes about half the time. They picked the Gaussian because the inputs to a unit tend to look roughly Gaussian, especially in a network with normalization layers.

A random mask is fine during training, but at test time we want a fixed function. So we take the average. The mask is 11 with probability Φ(x)\Phi(x) and 00 otherwise, so the expected output is xx times that probability:

E[x m]=x⋅Φ(x)+0⋅(1−Φ(x))=x Φ(x).\mathbb{E}[x\thinspace m] = x \cdot \Phi(x) + 0 \cdot (1 - \Phi(x)) = x\thinspace \Phi(x).

That average is GELU. Writing Φ\Phi with the error function gives the form you usually see:

GELU(x)=x Φ(x)=x2(1+erf⁡ ⁣(x2)).\mathrm{GELU}(x) = x\thinspace \Phi(x) = \tfrac{x}{2}\left(1 + \operatorname{erf}\negthinspace \left(\tfrac{x}{\sqrt{2}}\right)\right).

The demo below does the averaging by sampling. Set the sample count to one and each dot is either xx or 00, a single draw of the mask. Raise it and the average settles onto x Φ(x)x\thinspace \Phi(x), with the scatter shrinking like 1/N1/\sqrt{N} as with any Monte Carlo average. You can also swap the mask probability. Switch it to the logistic σ(x)\sigma(x) and the same construction gives x σ(x)x\thinspace \sigma(x), which is the function in the next section. Switch it to a hard step and the randomness disappears and the average is ReLU.

keep probability p(x) =
max |mean − exact| = 0.138
0123-3-2-10123mean of x·m, m ~ Bernoulli(p(x))01p(x), the probability the input is kept
x·Φ(x) = GELU x·σ(x) = SiLU x·1[x > 0] = ReLU Monte Carlo mean
GELU as the mean of a random gate. Each dot is the average of N samples of x times a Bernoulli mask with probability p(x). The solid line is x·p(x). The strip below shows p(x) itself: a soft version of ReLU's step.

The shape differs from ReLU in a few ways worth knowing. Taking the derivative with the product rule gives

GELU′(x)=Φ(x)+x φ(x),\mathrm{GELU}'(x) = \Phi(x) + x\thinspace \varphi(x),

where φ\varphi is the standard normal density. Unlike ReLU’s, this derivative is not capped at one. It peaks at about 1.131.13 near x=1.1x = 1.1. The function also is not monotone. It dips to −0.170-0.170 at x=−0.752x = -0.752 before coming back up to zero, so an input between about −3-3 and 00 produces a small negative output and a small but nonzero gradient. A ReLU unit with the same input would have a gradient of exactly zero, which is how ReLU units die. A GELU unit in that range can still be nudged back.

There is also a square hiding near zero. Close to zero, Φ(x)\Phi(x) is roughly a straight line, Φ(x)≈12+φ(0) x=12+x2π\Phi(x) \approx \tfrac{1}{2} + \varphi(0)\thinspace x = \tfrac{1}{2} + \tfrac{x}{\sqrt{2\pi}}. Multiplying by xx gives

GELU(x)≈x2+x22π+O(x3).\mathrm{GELU}(x) \approx \tfrac{x}{2} + \tfrac{x^2}{\sqrt{2\pi}} + O(x^3).

ReLU is a straight line on each side of zero. GELU is curved everywhere, and near zero it has an x2x^2 term with coefficient 1/2π=0.3991/\sqrt{2\pi} = 0.399. That square shows up again when we get to the gate.

The exact function needs the error function, so in practice people use one of two cheaper approximations. GPT-2 shipped the tanh form,

GELU(x)≈x2(1+tanh⁡ ⁣[2/π (x+0.044715 x3)]),\mathrm{GELU}(x) \approx \tfrac{x}{2}\left(1 + \tanh\negthinspace \left[\sqrt{2/\pi}\thinspace (x + 0.044715\thinspace x^3)\right]\right),

which is within 0.00050.0005 of exact on [−4,4][-4, 4].11 The other is x σ(1.702x)x\thinspace \sigma(1.702x), which is within 0.020.02. Both are in the plot below, and the second one comes back in the Swish section.

GELU as
-101234-4-3-2-101234xmin −0.170
f   f′
move the pointer over the chart to read values
ReLU and GELU with derivatives. The toggle switches GELU between the exact erf form and its two approximations and reports the largest error on the visible range.

Swish

This is the function from the Llama line I started with. PyTorch calls it silu, and in general form it is

Swishβ(x)=x σ(βx).\mathrm{Swish}_\beta(x) = x\thinspace \sigma(\beta x).

It is the same construction as GELU from the last section, xx times a probability, with the sigmoid in place of the Gaussian Φ\Phi. The name and the history are a little tangled. Ramachandran, Zoph and Le reported it in 2017 as Swish, after an automated search over combinations of simple functions trained and evaluated on CIFAR-10.12 The version with β=1\beta = 1 had already been published earlier that year by Elfwing, Uchibe and Doya as the sigmoid-weighted linear unit, SiLU, for reinforcement learning, and the 2016 GELU paper had named it as an alternative.13 Every model in this post that uses it sets β=1\beta = 1, so in practice Swish, SiLU and silu are the same thing: x σ(x)x\thinspace \sigma(x).

The parameter β\beta controls how sharp the function is, and it moves between two functions we already know. As β→0\beta \to 0, σ(βx)→12\sigma(\beta x) \to \tfrac{1}{2} for every xx, so Swish becomes the straight line x/2x/2. As β→∞\beta \to \infty, σ(βx)\sigma(\beta x) becomes a hard step that is 00 for negative xx and 11 for positive xx, so Swish becomes ReLU. Drag the slider to see it move between the two.

-101234-4-3-2-101234xmin −0.278
f   f′
move the pointer over the chart to read values
Swish with β from 0 to 4. At 0 it is the line x/2; as β grows it sharpens into ReLU. The dot marks the minimum, which deepens and moves toward zero as β grows.

The derivative comes from the product rule and the sigmoid derivative we worked out in the saturation section, σ′=σ(1−σ)\sigma' = \sigma(1 - \sigma):

Swishβ′(x)=σ(βx)+βx σ(βx) (1−σ(βx)).\mathrm{Swish}_\beta'(x) = \sigma(\beta x) + \beta x\thinspace \sigma(\beta x)\thinspace (1 - \sigma(\beta x)).

Like GELU’s, it is not capped at one, and like GELU, Swish dips below zero before coming back up. At β=1\beta = 1 the minimum is −0.278-0.278 at x=−1.278x = -1.278, deeper and further out than GELU’s −0.170-0.170. So a negative input still gets a small gradient, and the unit is not stuck the way a dead ReLU unit is.

Near zero the same square shows up. The sigmoid at zero is 12\tfrac{1}{2} and its slope there is the 14\tfrac{1}{4} we derived earlier, so σ(βx)≈12+βx4\sigma(\beta x) \approx \tfrac{1}{2} + \tfrac{\beta x}{4}. Multiplying by xx:

Swishβ(x)≈x2+β4 x2+O(x3).\mathrm{Swish}_\beta(x) \approx \tfrac{x}{2} + \tfrac{\beta}{4}\thinspace x^2 + O(x^3).

That is the same shape as GELU, half the identity plus a square, and the two functions are close. If we pick β\beta so the x2x^2 coefficients match, β/4=1/2π\beta/4 = 1/\sqrt{2\pi}, we get β=1.596\beta = 1.596. If we instead pick the β\beta that best matches the whole curve on [−4,4][-4, 4], we get β=1.702\beta = 1.702, and the largest gap between the two functions is 0.020.02. That is where the second GELU approximation from the last section comes from: x σ(1.702x)x\thinspace \sigma(1.702x) is Swish with β=1.702\beta = 1.702. Llama’s SiLU is the same function with β=1\beta = 1.

So how much does any of this matter? Far from zero, not much. For ∣x∣>3|x| > 3, GELU is within 0.0050.005 of ReLU and SiLU is within 0.150.15. Closer to zero the smooth functions differ from ReLU in three ways: the gradient never reaches exactly zero, there is a shallow negative dip, and there is curvature near the origin. Shazeer measured what those are worth in a transformer with the parameter count held fixed, which the next section covers in detail. After 524,288 steps, GELU and Swish finished within 0.0060.006 of ReLU, and slightly behind it.14 SwiGLU, which puts Swish inside a gate, finished 0.0410.041 ahead. On its own Swish did not move the loss, and inside a gate it did.

The gate

Every activation so far takes one number in and gives one number out. The gate is a different idea. Dauphin, Fan, Auli and Grangier introduced the gated linear unit, GLU, in 2016 for convolutional language models.15 Instead of one projection followed by an activation, they computed two projections of the same input, passed one of them through a sigmoid, and multiplied the two together:

GLU(x)=(xW+b)⊗σ(xV+c).\mathrm{GLU}(x) = (xW + b) \otimes \sigma(xV + c).

Here ⊗\otimes means elementwise multiplication. The sigmoid branch acts as a gate. For each hidden unit it produces a number between 00 and 11 that decides how much of the other branch, the linear one, gets through. A gate near 11 lets the linear value pass almost unchanged, and a gate near 00 shuts it off.

Their argument for this was about the gradient, which ties back to the saturation section. Take the product rule on X⊗σ(X)X \otimes \sigma(X):

∇[X⊗σ(X)]=∇X⊗σ(X)+X⊗σ′(X) ∇X.\nabla\big[X \otimes \sigma(X)\big] = \nabla X \otimes \sigma(X) + X \otimes \sigma'(X)\thinspace \nabla X.

The second term goes through σ′\sigma', which we know is at most 14\tfrac{1}{4}. The first term does not. It goes through the value of the gate, σ(X)\sigma(X), and wherever the gate is open that value is close to 11, so the gradient passes through at nearly full size. A plain sigmoid layer only has the second kind of term. They compared GLU against the gated tanh unit from the PixelCNN and WaveNet work, tanh⁡(xW)⊗σ(xV)\tanh(xW) \otimes \sigma(xV), where both branches go through a squashing function and every term in the gradient picks up a tanh⁡′\tanh' or a σ′\sigma'. The version with the plain linear branch trained faster and reached lower perplexity.

In 2020 Shazeer took the GLU and put it in the transformer’s feed-forward layer.14 He then asked what happens if we swap out the sigmoid in the gate, and tried four replacements: nothing at all, ReLU, GELU and Swish. With biases dropped, the family is

FFNGLU(x)=(σ(xW)⊗xV) W2\mathrm{FFN}_{\mathrm{GLU}}(x) = \big(\sigma(xW) \otimes xV\big)\thinspace W_2 FFNBilinear(x)=(xW⊗xV) W2\mathrm{FFN}_{\mathrm{Bilinear}}(x) = \big(xW \otimes xV\big)\thinspace W_2 FFNReGLU(x)=(max⁡(0,xW)⊗xV) W2\mathrm{FFN}_{\mathrm{ReGLU}}(x) = \big(\max(0, xW) \otimes xV\big)\thinspace W_2 FFNGEGLU(x)=(GELU(xW)⊗xV) W2\mathrm{FFN}_{\mathrm{GEGLU}}(x) = \big(\mathrm{GELU}(xW) \otimes xV\big)\thinspace W_2 FFNSwiGLU(x)=(Swish1(xW)⊗xV) W2.\mathrm{FFN}_{\mathrm{SwiGLU}}(x) = \big(\mathrm{Swish}_1(xW) \otimes xV\big)\thinspace W_2.

The last one is the Llama line from the start of the post. In Llama’s code, w1 is WW, the projection that goes through the gate, w3 is VV, the linear branch, and w2 is W2W_2, the projection back down:

return self.w2(F.silu(self.w1(x)) * self.w3(x))

Every variant in the family has three matrices where the plain layer has two, so a fair comparison has to account for the extra one. Shazeer kept the parameter count and the arithmetic the same by shrinking the hidden width to two thirds. With hidden width hh, a plain layer has 2dh2dh parameters and a gated layer has 3dh3dh, so matching them means a gated width of 23h\tfrac{2}{3}h. In his T5-base encoder-decoder, with d=768d = 768, that took the hidden width from 30723072 to 20482048. He pre-trained each variant on C4 for 524,288 steps with the span-corruption objective, and also ran four short 65,536-step runs of each to measure how much results vary between seeds. Here are the held-out log-perplexities, where lower is better:

Variant log-perplexity, 65k steps (std) log-perplexity, 524k steps
ReLU 1.997 (0.005) 1.677
GELU 1.983 (0.005) 1.679
Swish 1.994 (0.003) 1.683
GLU 1.982 (0.006) 1.663
Bilinear 1.960 (0.005) 1.648
ReGLU 1.953 (0.003) 1.645
GEGLU 1.942 (0.004) 1.633
SwiGLU 1.944 (0.010) 1.636

Start with the top three rows, the plain two-matrix layers. In the long runs GELU and Swish finish 0.0020.002 and 0.0060.006 behind ReLU. GELU is 0.0140.014 ahead of ReLU in the short runs, but that lead is gone by 524k steps. So swapping the activation in a plain layer did very little.

Now look at Bilinear. It has no activation function at all, only the product of two linear projections, and it beats all three plain layers by at least 0.0290.029 in the long runs and 0.0230.023 in the short ones. The standard deviation between seeds is about 0.0050.005, so that gap is well outside the noise. GEGLU and SwiGLU are another 0.0120.012 to 0.0150.015 ahead of Bilinear. Put together: of the 0.0440.044 between ReLU and GEGLU, 0.0290.029 is already there in the Bilinear row, before any activation function goes into the gate. Most of the gain comes from the multiplication.

The fine-tuning results are less tidy. On SuperGLUE every gated variant averages above every plain one, 73.6673.66 to 74.5674.56 against 72.4072.40 to 72.9872.98. On GLUE four of the five do, and Bilinear at 83.7983.79 is level with ReLU at 83.8083.80. On SQuAD the F1 scores of all eight fall between 90.6990.69 and 91.1891.18, and GLU comes last. The best variant also changes with the task: GEGLU on perplexity, ReGLU on GLUE and SQuAD F1, SwiGLU on SuperGLUE. Shazeer did not offer a reason for any of it. His conclusion reads: “We offer no explanation as to why these architectures seem to work; we attribute their success, as all else, to divine benevolence.”

PaLM adopted SwiGLU in 2022, citing these results, but dropped the two-thirds correction. Its paper notes that SwiGLU “does require three matrix multiplications in the MLP rather than two” and then sets the hidden width to 4d4d anyway, so each PaLM feed-forward layer has 12d212 d^2 parameters where a GELU layer of the same width has 8d28 d^2.16 Some of what PaLM attributed to SwiGLU was half again as many feed-forward parameters. Llama put the correction back in 2023: “We use a dimension of 234d\tfrac{2}{3} 4d instead of 4d4d as in PaLM.”17 That is where Llama 7B’s hidden width of 1100811008 comes from: two thirds of 4⋅40964 \cdot 4096 is 1092210922, rounded up to a multiple of 256256.

Products of input features

So why would multiplying two projections help this much? It comes down to what a single hidden unit can compute.

A plain hidden unit computes f(w⋅x)f(w \cdot x). Everything it sees about the input goes through one number, the projection w⋅xw \cdot x. Move xx in any direction perpendicular to ww and that number does not change, so neither does the output. A function like that is called a ridge function, and a plain feed-forward layer is a sum of hh of them.

A gated unit computes f(w⋅x) (v⋅x)f(w \cdot x)\thinspace (v \cdot x), which depends on two projections. Take the simplest case, the bilinear unit, where ff does nothing:

(w⋅x)(v⋅x)=x⊤(wv⊤) x.(w \cdot x)(v \cdot x) = x^\top (w v^\top)\thinspace x.

That is a quadratic form in xx, with the rank-one matrix wv⊤w v^\top. A layer of hh of these can represent any quadratic form of rank up to hh, and W2W_2 mixes them. In particular, one bilinear unit can output the product of two input features exactly. Set ww to pick out feature ii and vv to pick out feature jj, and the unit computes xixjx_i x_j.

A plain layer has to build that same product out of ridge functions, and how well it can depends on ff. ReLU has no curvature, so a ReLU layer approximates the saddle shape of xixjx_i x_j with flat pieces joined at kinks, and the error only goes down as we add more pieces. If ff were an exact square, two plain units would be enough, because xy=14[(x+y)2−(x−y)2]xy = \tfrac{1}{4}\big[(x+y)^2 - (x-y)^2\big]. GELU does have a square, but only in its expansion near zero, x2+0.399 x2\tfrac{x}{2} + 0.399\thinspace x^2, from the GELU section. By ∣x∣=3|x| = 3 it is within 0.0050.005 of ReLU, so that identity does not hold for inputs of that size.

The demo below trains both kinds of layer on f(x,y)=xyf(x, y) = xy with the same parameter budget. Choose the budget, pick the plain activation and the gate, and compare the two surfaces and their errors. The exact-solution button sets one bilinear unit to w=(1,0)w = (1, 0) and v=(0,1)v = (0, 1), which computes the product with zero error using five parameters.

plain
gate
target x·y
 
plain relu
gated bilinear
Fitting f(x, y) = xy. Left: the target. Middle: a one-hidden-layer network of plain units. Right: a network of gated units with the same number of parameters. Test error on a 48 by 48 grid under each.

The gate as a quadratic spline

There is another way to see what the gate adds. Queiruga’s 2026 paper, titled “Divine Benevolence is an x2x^2” in answer to Shazeer’s conclusion, describes a ReGLU unit as a piece of a spline.18

A spline is a function built by gluing simple pieces together end to end. The points where one piece ends and the next begins are called knots. A linear spline uses straight-line pieces, so it looks like a connect-the-dots drawing. A quadratic spline uses pieces of parabolas, so each piece can bend.

A layer of ReLU units on a single input xx is a linear spline. Each unit a max⁡(0,gx+g0)a\thinspace \max(0, g x + g_0) is zero on one side of the point x=−g0/gx = -g_0/g and a straight line on the other. That point is a knot. Adding the units together gives a function that is a straight line between knots and changes slope at each one, with one knot per unit.

A layer of ReGLU units is a quadratic spline. Put the biases back for a moment and expand one unit, with gate weights gg and g0g_0, linear-branch weights uu and u0u_0, and output weight aa, on the side where its gate is open:

a max⁡(0, gx+g0) (ux+u0)=a (gu x2+(g0u+gu0) x+g0u0)a\thinspace \max(0,\thinspace g x + g_0)\thinspace (u x + u_0) = a\thinspace \big(g u\thinspace x^2 + (g_0 u + g u_0)\thinspace x + g_0 u_0\big)

wherever gx+g0>0g x + g_0 > 0. On that side the unit is a parabola in xx, and on the other side it is zero. So each ReGLU unit also adds one knot, but what it adds after the knot is a curve instead of a line. Adding the units together gives a function made of parabola pieces.

Why does that matter? Because how well a spline can fit a smooth function depends on how its error shrinks as the knots get closer together. Say the knots are spaced hh apart and we fit a smooth target between each pair.

A straight line can match the target’s value and slope, but not its curvature. Over a piece of width hh, the curvature bends the target away from the line by an amount proportional to h2h^2. For linear interpolation the exact bound is

∣f(x)−line(x)∣≤h28 max⁡∣f′′∣.|f(x) - \text{line}(x)| \le \tfrac{h^2}{8}\thinspace \max |f''|.

A parabola can match the curvature too, so the leftover error comes from the next term in the Taylor series, the third derivative, and it shrinks like h3h^3. Halve the spacing and the linear spline’s error drops by a factor of four, while the quadratic spline’s drops by a factor of eight.

Now connect that to the network. Each unit adds one knot, so a layer with nn units on an input range of fixed width has knots about 1/n1/n apart. The parameter count is proportional to nn as well. With one input and one output, a ReLU layer has 3n+13n + 1 parameters and a ReGLU layer has 5n+15n + 1. So the spacing shrinks like 1/P1/P, and plugging h∼1/Ph \sim 1/P into the two error rates gives errors ε\varepsilon that fall as

εReLU∼P−2,εReGLU∼P−3.\varepsilon_{\mathrm{ReLU}} \sim P^{-2}, \qquad \varepsilon_{\mathrm{ReGLU}} \sim P^{-3}.

The gated layer gets one more power of PP for the same parameter budget. Doubling the parameters cuts the ReLU layer’s error by four and the gated layer’s by eight.

Queiruga tested this on the one-dimensional target f(x)=1/(1+cos⁡2πx)f(x) = 1/(1 + \cos^2 \pi x) on [−1,1][-1, 1], with widths from 1 to 50 units, and measured the root-mean-square error at each width. The fitted exponents were −2.18-2.18 for the ReLU network and −3.12-3.12 for the gated one, close to the −2-2 and −3-3 the argument predicts. This is a result for a shallow network with a single input. A transformer’s feed-forward layer has thousands of inputs and is trained by gradient descent rather than fit exactly, so the spline argument is about what a gated layer can represent. It does not say what gradient descent will find.

The signal through depth

Back in the ReLU section we saw that a ReLU layer halves the mean square of its input, and He initialization picks the weights to cancel that. Two things made that work. ReLU halves the mean square by the same factor no matter how big the input is, which is the homogeneity property, and so one fixed choice of weights fixes every layer. Now we can run the same calculation for every function in this post and see which ones have a fix like that.

Here is the setup. Take a deep stack of layers, xl+1=f(Wlxl)x_{l+1} = f(W_l x_l), where each weight is drawn from a Gaussian with mean zero and variance g2/ng^2/n, and the width nn is large. The number gg is the gain. It is the knob we turn to make the weights bigger or smaller, and He initialization is g=2g = \sqrt{2}. Call rlr_l the root-mean-square of the signal coming into layer ll. By the same argument as in the ReLU section, each pre-activation zz is then Gaussian with variance g2rl2g^2 r_l^2, and the signal coming out has mean square

rl+12=E ⁣[f(z)2],z∼N(0, g2rl2).r_{l+1}^2 = \mathbb{E}\negthinspace \left[f(z)^2\right], \qquad z \sim \mathcal{N}(0,\thinspace g^2 r_l^2).

For ReLU that is g2rl2/2g^2 r_l^2 / 2. The ratio of mean squares from one layer to the next is g2/2g^2/2 whatever rlr_l is, so g=2g = \sqrt{2} makes it exactly one and the signal stays the same size at any depth.

GELU and Swish do not work out this neatly, because they are not homogeneous. How much they shrink the signal depends on how big it already is. At unit scale and g=1g = 1, E[GELU(z)2]=0.425\mathbb{E}[\mathrm{GELU}(z)^2] = 0.425 and E[SiLU(z)2]=0.356\mathbb{E}[\mathrm{SiLU}(z)^2] = 0.356, against ReLU’s 0.50.5. For a small signal both functions look like x/2x/2, which multiplies the mean square by 14\tfrac{1}{4}. For a large signal they look like ReLU, which multiplies it by 12\tfrac{1}{2}. So there is no single gain that holds a deep GELU stack at a constant scale. The best we get is a scale the signal drifts toward and settles at.

The gated functions are harder still, because the multiplication squares the scale. Take the bilinear unit, where the output is the product of two projections z1z_1 and z2z_2. If the two are independent,

E ⁣[(z1z2)2]=E[z12] E[z22]=g4rl4,rl+1=g2rl2.\mathbb{E}\negthinspace \left[(z_1 z_2)^2\right] = \mathbb{E}[z_1^2]\thinspace \mathbb{E}[z_2^2] = g^4 r_l^4, \qquad r_{l+1} = g^2 r_l^2.

The output scale is the square of the input scale. Taking logs makes the problem clear: log⁡rl+1=2log⁡rl+2log⁡g\log r_{l+1} = 2 \log r_l + 2 \log g, so the logarithm of the scale doubles at every layer. There is one balance point, r=1/g2r = 1/g^2, where the signal stays put. Start a little above it and the signal grows faster than exponentially. Start a little below and it collapses just as fast. Squared ReLU behaves the same way, since E[ReLU(z)4]=32σ4\mathbb{E}[\mathrm{ReLU}(z)^4] = \tfrac{3}{2}\sigma^4 also goes as the fourth power of the scale. SwiGLU and GEGLU behave like a square for large inputs and somewhere between linear and square near zero. No choice of gain fixes any of them.

The demo below runs this through a stack of random layers. You can change the gain and the input scale, and turn on the normalization that transformers use between layers.

predicted from E[f(z)²]
1e-61e-41e-211e21e41e604812162024signal RMS at layer llayerclippedclipped1e-61e-41e-211e21e41e604812162024gradient RMS at layer l, unit RMS at the outputlayer
activationE[x²] ratio per layer, measuredpredictedgradient E[u²] ratio per layer
ReLU0.9881.000 g² / 20.999
GELU0.6360.671 c(σ) · g² (c(σ=1.40) = 0.460)0.680
Bilinear6.06e+6, out of range at layer 71.59e+6 g⁴ · r²n/a
SwiGLU4.48e+6, out of range at layer 98.94e+6 c(σ) · g⁴ · r² (c(σ=1.40) = 0.398)n/a
n = 256, 4 input vectors, fresh W and V per layer. r is the RMS of a layer's input (1 with RMSNorm). c(σ) = E[f(z)²]/σ² for z ~ N(0, σ²), σ = g·r, which depends on the scale for GELU and Swish because neither is homogeneous.
Signal scale (left) and gradient scale (right) through a stack of random layers of width 256. Change the gain and the input scale; turn on the per-layer RMSNorm that pre-norm transformers have.

So if the gated layers are this unstable, how do transformers train at all? They do not rely on the gain to hold the scale. Every block normalizes its input before the feed-forward layer, which pins rlr_l to one by construction, and the layer’s output is added to a residual stream that the next block normalizes again. The signal cannot run away, because it is reset at every block.

What the gate still changes is the shape of the output, and in particular its tails. A product of two Gaussians has heavier tails than either one, and a function that grows like x2x^2 turns an outlier of size 1010 into one of size 100100. Two 2026 papers trace instabilities in low-precision SwiGLU training to that tail. PowLU replaces the positive side with a bounded power so that it grows slower than a square. MemGLU closes the gate entirely for large inputs, and at 9M and 30M parameters it finds the loss changes by about 0.10.1 percent.1920

The right panel of the demo tracks the gradient on the way back, and it shows Dauphin’s argument from the gate section in numbers. With normalization on and g=2g = \sqrt{2}, a plain ReLU or GELU layer passes the gradient back at about the size it arrived. A gated layer roughly doubles its mean square. That comes from the product rule again. The gradient has one term through the gate branch, W⊤ ⁣(f′(z)⊗(Vx)⊗u)W^\top\negthinspace \big(f'(z) \otimes (Vx) \otimes u\big), and one through the linear branch, V⊤ ⁣(f(z)⊗u)V^\top\negthinspace \big(f(z) \otimes u\big), where uu is the gradient arriving from above. At this gain each term is about the size of uu, and the two are independent, so their mean squares add.

Over 48 layers that doubling compounds to 2482^{48} in mean square, which is 2242^{24}, about 10710^7, in root-mean-square, which is why the demo has no residual connections. In a transformer the residual stream carries the gradient around every block, and each block’s own contribution only adjusts it. The doubling is still a useful picture of what Dauphin meant. The two terms are two routes for the gradient to get back through the layer, and the linear-branch route never passes through the gate’s derivative.

Where it is going

Once Llama shipped SwiGLU, the models that followed it mostly took its feed-forward layer as it was. The table below lists sixteen models with the activation and hidden width from each one’s config file. Mistral, Qwen, DeepSeek, OLMo and the later Llamas all use SwiGLU, at hidden widths between 83d\tfrac{8}{3}d and 3.5d3.5d. The Gemma models are the main exception, and they use a GELU gate instead.21

modelFFNdhiddenhidden/dFFN sharewidth rule
GPT-2 (2019)GELU (tanh)7683,072467%4d
GPT-3 (2020)GELU12,28849,152467%4d
PaLM 540B (2022)SwiGLU18,43273,728475%4d, no 2/3
LLaMA 7B (2023)SwiGLU4,09611,0082.6967%2/3·4d → 256
Llama 2 70B (2023)SwiGLU8,19228,6723.5072%2/3·4d·1.3 → 4096
Mistral 7B (2023)SwiGLU (config)4,09614,3363.5072%config only
Qwen 7B (2023)SwiGLU4,09611,0082.6967%8/3 d
DeepSeek 7B (2024)SwiGLU4,09611,0082.6967%8/3 d
OLMo 7B (2024)SwiGLU4,09611,0082.6967%8/3 d → 128
Gemma 2 9B (2024)GEGLU (tanh)3,58414,336475%report: 28672
Gemma 2 27B (2024)GEGLU (tanh)4,60836,864886%report: 73728
Llama 3 8B (2024)SwiGLU4,09614,3363.5072%2/3·4d·1.3 → 1024
Llama 3 405B (2024)SwiGLU16,38453,2483.2571%paper
Gemma 3 27B (2025)GEGLU (tanh)5,37621,504475%config
gpt-oss-120b expert (2025)clamped GEGLU2,8802,880143%α = 1.702
Gemma 4 31B (2026)GEGLU (tanh)5,37621,504475%config
Which activation, which width. The ratio column is the config file's intermediate size over the model width. Sources in the footnotes.

Two of these are worth a closer look. Mistral’s paper lists hidden_dim 14336 in its parameter table but never says which activation it uses. You have to go to the config file to find silu, and the rest of the architecture is Llama’s.22 gpt-oss, the open-weight model OpenAI released in 2025, calls its activation SwiGLU, with a footnote that the implementation is “unconventional, including clamping and a residual connection”. The code computes

xg σ(1.702 xg)⋅(xl+1),x_g\thinspace \sigma(1.702\thinspace x_g) \cdot (x_l + 1),

with both branches clamped at 77.23 That 1.7021.702 is the constant from the GELU section. The gate is the sigmoid approximation to GELU, so what OpenAI calls SwiGLU is a clamped GEGLU with a bias of one on the linear branch.

SwiGLU is not the only way to get the square that the gate provides. In 2021 David So and colleagues ran a search over changes to the transformer and found that the single most effective one was replacing ReLU in the feed-forward layer with its square, max⁡(0,x)2\max(0, x)^2.24 Squared ReLU is piecewise quadratic like ReGLU, so the spline argument applies to it too, and it needs no third matrix. In their runs it matched or beat the GLU variants. The paper points to its “drastically different asymptotics” from ReLU, GELU and Swish, and to the usefulness of higher-order terms. PaLM in 2022 and Llama in 2023 went with SwiGLU anyway.

Squared ReLU is getting attention again, and the reason is sparsity. A ReLU layer outputs exact zeros, and in a trained model most of its hidden units are zero for any given token. Li and colleagues measured that in a trained T5-base only about 3 percent of the hidden units are nonzero per token, and the fraction falls as models get bigger.25 That matters at inference. If most of the hidden vector is zero, the down-projection, which is a third of the layer’s arithmetic, can skip most of its rows.

Apple’s “ReLU Strikes Back” measured the same thing on open models. Over 90 percent of the hidden units are zero in a ReLU OPT model, compared with 0 percent in Llama’s SwiGLU layers and 1 percent in Falcon’s GELU layers. A smooth function or a product almost never lands on exactly zero.26 The same paper trained 1.3B-parameter models from scratch with ReLU, GELU, SiLU and a sharpened Swish, and found no significant difference in loss between them. That is Shazeer’s first three rows again, at a larger scale.

“ReLU² Wins” compared ReLU, SwiGLU, ReGLU and squared ReLU on the trade-off between sparsity and loss, and picked squared ReLU.27 A 2024 result at 1.1B parameters trained on 126B tokens put squared ReLU ahead of SwiGLU with the parameter count held fixed, at 10.3510.35 perplexity against 10.5210.52. A trainable variant called xIELU came out slightly ahead of both.28

So back to the line I started with, self.w2(F.silu(self.w1(x)) * self.w3(x)). The F.silu is Swish with β=1\beta = 1. On its own it does little: in Shazeer’s table and in Apple’s runs, ReLU, GELU and Swish reach the same loss in a plain layer, and they differ mainly in whether the hidden layer has exact zeros. Most of the gain is in the *, the multiplication of two projections. SwiGLU finishes 0.0410.041 nats ahead of ReLU at T5 scale, and 0.0290.029 of that is already in the Bilinear row, which multiplies the two projections with no activation at all. Putting Swish or GELU on the gate branch adds the other 0.0120.012 to 0.0150.015. Putting a sigmoid there, as the original GLU did, comes out 0.0150.015 behind Bilinear. Between Swish and GELU the difference is 0.0030.003, and gpt-oss ships a GELU gate under the SwiGLU name. Squared ReLU gets a similar square without the third matrix, and it keeps the zeros.

As for why Llama uses SwiGLU in particular: PaLM adopted it citing Shazeer’s paper, Llama took it from PaLM with the width corrected, and Mistral, Qwen and the other models in the table took it from Llama.

Footnotes

  1. Ashish Vaswani et al., “Attention Is All You Need”, NeurIPS 2017, section 3.3, for ReLU; Hugo Touvron et al., “LLaMA”, 2023, section 2.2, for SwiGLU. GPT-2 and PaLM are cited where their layers come up below. ↩

  2. Llama 7B has d=4096d = 4096, 32 layers and a feed-forward hidden width of 11008. Each feed-forward layer has three matrices of 4096×110084096 \times 11008, 135 million parameters, and 32 of them make 4.33 billion. Each attention layer has four 4096×40964096 \times 4096 projections, 67 million, for 2.15 billion in total. The embedding and output matrices add 0.26 billion. ↩

  3. Mor Geva, Roei Schuster, Jonathan Berant, Omer Levy, “Transformer Feed-Forward Layers Are Key-Value Memories”, EMNLP 2021. ↩

  4. Shazeer dropped the biases “following the T5 codebase,” with a footnote: “Also in the interest of ML fairness.” PaLM, Llama, and most models since have none in the feed-forward layer. ↩

  5. Moshe Leshno, Vladimir Lin, Allan Pinkus, Shimon Schocken, “Multilayer feedforward networks with a nonpolynomial activation function can approximate any function”, Neural Networks 6(6), 1993. ↩

  6. Xavier Glorot, Antoine Bordes, Yoshua Bengio, “Deep Sparse Rectifier Neural Networks”, AISTATS 2011. ↩

  7. Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun, “Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification”, ICCV 2015. Section 2.2 has the derivation; figure 3 has the 22- and 30-layer comparison against the initialization of Glorot and Bengio, “Understanding the difficulty of training deep feedforward neural networks”, AISTATS 2010. The same paper introduces PReLU. ↩

  8. Lu Lu, Yeonjong Shin, Yanhui Su, George Em Karniadakis, “Dying ReLU and Initialization: Theory and Numerical Examples”, 2019. ↩

  9. Andrew Maas, Awni Hannun, Andrew Ng, “Rectifier Nonlinearities Improve Neural Network Acoustic Models”, ICML workshop 2013. ↩

  10. Dan Hendrycks, Kevin Gimpel, “Gaussian Error Linear Units (GELUs)”, 2016. The first versions were titled “Bridging Nonlinearities and Stochastic Regularizers with Gaussian Error Linear Units,” which is the name Shazeer cites. ↩

  11. Alec Radford et al., “Language Models are Unsupervised Multitask Learners”, 2019. The activation is in the code, src/model.py: the tanh approximation of GELU, with nx*4 for the hidden width. GPT-3’s paper says it uses “the same model and architecture as GPT-2” with dff=4dmodeld_{ff} = 4 d_{\text{model}}. ↩

  12. Prajit Ramachandran, Barret Zoph, Quoc Le, “Searching for Activation Functions”, 2017. The first version was titled “Swish: a Self-Gated Activation Function.” ↩

  13. Stefan Elfwing, Eiji Uchibe, Kenji Doya, “Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning”, 2017. Hendrycks and Gimpel’s appendix B states that the 2016 GELU paper introduced xσ(x)x\sigma(x) under the name SiLU, that Elfwing and colleagues published it as “SIL” and later adopted the SiLU name, that the first Swish paper cited neither, and that the frameworks renamed swish to silu afterwards. The Swish paper’s second version added the learnable β\beta and the citation to Elfwing. ↩

  14. Noam Shazeer, “GLU Variants Improve Transformer”, 2020. Tables 1 through 4. The setup is T5-base, 12 encoder and 12 decoder layers, d=768d = 768, trained without dropout, which is why its baseline beats the T5 paper’s. ↩ ↩2

  15. Yann Dauphin, Angela Fan, Michael Auli, David Grangier, “Language Modeling with Gated Convolutional Networks”, ICML 2017. Section 3 has the gradient argument; figure 3 has the ablation against ReLU, tanh and the gated tanh unit. The bilinear variant without the sigmoid is credited to Andriy Mnih and Geoffrey Hinton, “Three New Graphical Models for Statistical Language Modelling,” ICML 2007. ↩

  16. Aakanksha Chowdhery et al., “PaLM: Scaling Language Modeling with Pathways”, 2022. Section 2: “We use SwiGLU activations … because they have been shown to significantly increase quality compared to standard ReLU, GeLU, or Swish activations,” and “The feed-forward size dffd_{ff} is always 4×dmodel4 \times d_{\text{model}}.” ↩

  17. Hugo Touvron et al., “LLaMA: Open and Efficient Foundation Language Models”, 2023, section 2.2. The rounding code is in llama/model.py. ↩

  18. Alejandro Queiruga, “Divine Benevolence is an x2x^2: GLUs scale asymptotically faster than MLPs”, 2026. Related: Michael Pearce, Thomas Dooms, Alice Rigg, Jose Oramas, Lee Sharkey, “Bilinear MLPs enable weight-based mechanistic interpretability”, ICLR 2025, which uses the same quadratic-form view of the bilinear layer to read features off the weights. ↩

  19. Jiang et al., “PowLU: An Activation Function for Stable Pre-Training of LLMs”, 2026. ↩

  20. Ge, Yang, Nie, “Is SwiGLU’s Open Positive Tail Necessary? Evidence from Closed-Tail Gating with MemGLU”, 2026. ↩

  21. Gemma Team, “Gemma: Open Models Based on Gemini Research and Technology”, 2024: “The standard ReLU non-linearity is replaced by the approximated version of the GeGLU activation function.” Gemma 2’s table lists “Non-linearity: GeGLU” and a feed-forward dimension of 28672 for the 9B model, whose config file has intermediate_size: 14336 and hidden_activation: gelu_pytorch_tanh. Gemma 3 and Gemma 4 configs set the same activation at 4d4d. ↩

  22. Albert Jiang et al., “Mistral 7B”, 2023, table 1. Mixtral’s paper, arXiv:2401.04088, does name it: “we use the same SwiGLU architecture as the expert function.” ↩

  23. OpenAI, “gpt-oss-120b & gpt-oss-20b Model Card”, 2025, and gpt_oss/torch/model.py, where swiglu is defined with alpha=1.702, limit=7.0 and returns out_glu * (x_linear + 1). ↩

  24. David So, Wojciech Mańke, Hanxiao Liu, Zihang Dai, Noam Shazeer, Quoc Le, “Primer: Searching for Efficient Transformers for Language Modeling”, NeurIPS 2021. ↩

  25. Zonglin Li et al., “The Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformers”, ICLR 2023. ↩

  26. Iman Mirzadeh et al., “ReLU Strikes Back: Exploiting Activation Sparsity in Large Language Models”, ICLR 2024. Table 1 for the sparsity figures; section 3 for the from-scratch comparison. ↩

  27. Zhengyan Zhang et al., “ReLU² Wins: Discovering Efficient Activation Functions for Sparse LLMs”, 2024. ↩

  28. Allen Hao Huang, “Deriving Activation Functions via Integration”, 2024. Table with Llama 1.1B on 126B FineWeb-Edu tokens, parameter-matched: SwiGLU 10.517, ReLU² 10.352, xIELU 10.207. ↩