Start with the claim you want to make
“Many weights are tiny” is an observation. “These channels can be removed” is a hypothesis. “The model remains equally good” is a task-level claim. I want a pruning experiment to keep those three statements separate.
This note sets out a conservative validation protocol for the gated feed-forward layers commonly used in Transformers. It includes a small synthetic numerical example to make the numerical and indexing checks concrete. The aim is to make the next experiment easier to interpret.
A channel spans three matrices
In a bias-free SwiGLU layer, one convenient convention is:
y(x) = Wdh(x)
Here, the up and gate projections each have one row per intermediate channel; the down projection has one column per intermediate channel. SiLU is a smooth elementwise nonlinearity, and the product gates one projected feature with another. See the GLU variants paper for the architectural background.
Removing intermediate channel i means removing row i from both input projections and column i from the output projection. Inspecting only one matrix misses the structure of the computation. A pruning rule must also account for biases when the model has them; the discussion here assumes the bias-free case.
A conservative screening statistic is the largest absolute weight attached to that channel:
I would flag a channel when all three parts satisfy a declared threshold. This is deliberately restrictive: a channel with a zero output column can have no direct output contribution even if its input weights are large. The rule is a useful way to find a conservative candidate set, not a characterization of every removable channel.
Tiny weights alone also do not provide a distribution-free output bound. The input magnitude, nonlinearities, downstream layers, and accumulation across channels all matter.
Do not manufacture zeros in the measurement
At very small magnitudes, the diagnostic itself can fail. Squaring a representable float32 value can underflow to zero. Converting weights to a lower precision before inspecting them can erase the evidence you intended to measure.
import numpy as np
x = np.array([1e-30], dtype=np.float32)
print(np.max(np.abs(x)) > 0) # True
print(np.square(x)[0] == 0) # True: the square underflows
For a threshold test at that scale, maximum absolute value avoids the squaring operation. If a norm is needed for another purpose, I would compute it in an appropriate precision and verify the behavior on synthetic inputs first.
The executable example for this note checks the underflow case, identifies channels across all three matrices, removes a selected channel, and compares the resulting outputs. Its small constructed tensors test the implementation logic. They are not evidence about the frequency of inactive channels in trained models.
Static weights cannot explain the training history
Suppose a candidate set is stable across several saved checkpoints. That narrows when the phenomenon might have formed, but it does not show that the channels never learned, never received gradients, or remained below the threshold between every saved point.
I would compare raw and averaged weights, examine threshold sensitivity, and inspect the original training tensors before blaming an export path. To investigate a mechanism, I would need earlier checkpoints, channel activations, gradients, and optimizer state. A static near-zero pattern can be consistent with several explanations.
This is where I would spend the next unit of experimental effort: distinguish hypotheses before adding a large sweep. Does the pattern already exist before a learning-rate change? Does it coincide with a change of optimizer or numerical precision? Can a small controlled run reproduce it? These questions are more informative than naming a cause from one final checkpoint.
Validate the transformation, then the model
I would organize validation in four levels:
| Level | What to check | What passing it does not establish |
|---|---|---|
| Tensor transformation | Correct axes and indices; retained weights unchanged; valid dimensions | Numerical equivalence of the executed model |
| Numerical outputs | Errors on a fixed evaluation set, including tails and worst cases | Unchanged task quality everywhere |
| Task behavior | A held-out task metric with an explicit evaluation budget | Equal performance across deployment hardware |
| Runtime | Actual latency, throughput, and memory on the target implementation | A portable speedup for all shapes and devices |
Even exactly preserved remaining weights do not guarantee bitwise-identical outputs. A smaller matrix can trigger a different kernel or accumulation order. I would therefore compare pruning error with the model’s ordinary precision-related variation, while still reporting both errors explicitly.
Hardware alignment is another distinction. Keeping a few extra channels can produce a width the accelerator executes more efficiently. Fewer theoretical multiply-adds and lower measured latency need not select the same shape. An alignment choice belongs in the experimental record, along with the final retained channel indices.
The experiment I would prioritize
A useful first experiment would freeze one checkpoint and one independent evaluation set, then compare the original model with a small, declared set of thresholds and alignment choices. I would preserve the unmodified checkpoint and every transformation manifest, and avoid choosing the final candidate on the same examples used for the final quality claim.
For timing, I would alternate the original and pruned runs to reduce ordering effects, separate startup from steady-state execution, and report repeated measurements. For task quality, I would predefine what degradation is acceptable and measure the task directly. Agreement on the highest-probability output is helpful, but can miss changes in the rest of the distribution.
The result I would value is a clear boundary: which channels can be removed, under what numerical and task criteria, and on which hardware the structural change is useful. A disappointing speedup or a quality regression is still informative if the experiment makes the reason easier to investigate.
Sources and scope
- Noam Shazeer, GLU Variants Improve Transformer, for the gated-layer background.
- KataGo model architectures, for the application context.
- Synthetic validation example, for the narrow numerical and indexing checks used in this note.
This is a methodology note with a synthetic test. It makes no trained-model performance claim.