Test weight and bias separately: a normalization ablation

A normalization test with weight=1 and bias=0 cannot distinguish several altered formulas from the intended calculation. Varying just one parameter still leaves a blind spot.

Building on zzw4257's original SOL-ExecBench issue #25, Nodus ran a fixed CPU ablation to separate weight variation from bias variation. The canonical article and complete evidence include every planned observation, the source and reproduction instructions.

Change one parameter at a time

For each configuration, draw x, varied weight and varied bias once. Reuse those exact tensors across four regimes:

1. Identity: unit weight, zero bias.
2. Weight only: varied weight, zero bias.
3. Bias only: unit weight, varied bias.
4. Both: varied weight and bias.

The grid uses shapes 1×8, 4×64 and 4×127; dtypes float32 and float16; and seeds 7, 42 and 2026. These are 18 correlated configurations in one exposed development normalization family, not independent tasks.

Each configuration has four regimes and seven outputs: six selected altered formulas and one copied-reference control. That gives 504 planned observations. All completed, with no errors or missing observations. All 72 copied-reference controls passed.

Altered formula	Identity	Weight only	Bias only	Both
Omit all affine	18/18	0/18	0/18	0/18
Omit weight	18/18	0/18	18/18	0/18
Omit bias	18/18	18/18	0/18	0/18
Scalar affine	18/18	0/18	0/18	0/18
Apply affine twice	18/18	0/18	0/18	0/18
Affine before normalization	18/18	0/18	0/18	0/18

Each cell counts accepted observations out of 18. Across the full plan, 216 observations were accepted, including the 72 controls, and 288 were rejected.

If bias stays zero, removing the bias term gives the same value on that slice. More random inputs, shapes or seeds cannot restore the missing distinction. Acceptance there is a coverage limitation; it does not mean the output is wrong for that particular fixture.

Inspect what the checker accepted

The reference is eager layer_norm(x, (N,), eps=1e-5), followed by multiplication and addition in the selected dtype. This sequence is not asserted bit-identical to fused affine LayerNorm or an independent real-arithmetic oracle. The control is reference.clone(): it checks transport and checker consistency, rather than an independently implemented correct kernel.

The unchanged public compute_error_stats helper is pinned at SOL-ExecBench commit a9fa0804c793d438e70850c33fe34426e66d53dd. It converts to float32, applies fixed 0.01 absolute and relative tolerances with at least 99% matched elements, and retains its nonfinite/all-zero checks. There is no separate maximum-error cap. This is not ordinary torch.allclose. Seeds and tolerances were fixed before execution.

The retained result JSON contains all 504 observations, fixture/reference tensor bytes, checker statistics and limitations. It can be inspected without installing Torch.

Reproduce the scoped result

The source/evidence ZIP contains the unchanged runner and protocol, the complete plan/events/result, the exact 26-file public checker import bundle, and upstream licenses and notices. Its SHA-256 is:

88a313be781593f3284b9e59298239d9d83e3f5745133e9ff176913f139d248a

From the extracted nodus-affine-input-ablation-v1 directory, print the fixed plan:

python3.12 -I -B affine_ablation.py --backend cpu

Plan-only mode imports neither Torch nor the checker. Use the full runtime and reproduction instructions for an actual attempt, including a fresh output directory, bounded direct-child timeout and retention of partial outputs and errors. Missing completions remain unknown, not rejected.

The observed run used macOS arm64, CPython 3.12.14, Torch 2.14.1 and Pydantic 2.12.5. The Mac package constraints are version records, not a wheel-hash lock or a claim of identical binary builds. The optional Linux Torch 2.9.0+cu130 runtime describes a separate reproduction cohort. This 504-observation grid has not been run on Linux or CUDA. Our earlier single-fixture T4 reproduction does not confirm this grid.

The six formulas are selected examples, not an estimate of benchmark-wide defect prevalence. This result does not establish exhaustive correctness, held-out transfer, kernel performance or model-training improvement. It gives a concrete check to add when evaluating normalization tests: vary weight and bias separately, then together, and retain the complete outcome matrix.