A controlled Nodus test kept a Python environment, cache and checkpoint across two sequential eight-H200 nodes. We separate platform readiness, the first CUDA operation and the single-GPU checkpoint recovery.

One controlled replacement pair; one observation per phase. First CUDA was measured before checkpoint restore. Checkpoint step 8→9 was separately verified on one GPU; eight-GPU checks were separate. The initial payload installed PyTorch and recovery reused the environment. No causal provisioning speedup is established.

Key takeaways

• A second Nodus node loaded the exact saved checkpoint, restored a small model and optimizer, and advanced training from step 8 to step 9.
• The observed first-CUDA interval after payload entry was 221.654 s initially and 19.411 s with the retained environment. Both measurements precede checkpoint restoration.
• This was one controlled replacement pair. Separate eight-rank GPU checks passed; the training checkpoint exercise used one GPU.

The node changed. The saved work stayed.

We shut down one Nodus eight-H200 node, started a second node on the same retained filesystem, and continued a small PyTorch training job from its saved checkpoint. The second node loaded the exact checkpoint written at step 8, restored the model, optimizer and random-number-generator states, and completed step 9. The prepared Python environment and a 64 MiB cache were still there. 1

This was a controlled production test on November 2, 2026: two sequential nodes, one retained 512 GiB filesystem, and one observation per phase. Both nodes also passed separate CUDA and NCCL checks across all eight GPUs. The checkpoint exercise itself ran on one GPU. 1

Three different readiness questions

A node reaching Ready, a program completing its first CUDA operation, and a training job restoring its state are different milestones. We recorded the first two as timed intervals. The checkpoint test records successful restoration and advancement, without a separate restore-duration measurement. 1

• API creation to platform Ready: 421.205 s on the initial node; 372.765 s on the second node.
• Detached payload entry to its first completed CUDA operation: 221.654 s initially; 19.411 s on the second node.
• Checkpoint continuity: the initial job saved step 8; the recovery job verified that file’s SHA-256, loaded its state, and saved step 9.

What changed between the two CUDA measurements

The initial payload created a virtual environment and installed torch==2.14.0 from the CUDA 12.6 package index. The recovery payload executed the retained interpreter directly. It checked that the Torch version, CUDA version, Python version and Torch path matched the initial environment metadata. Its code had no package-install branch. 1

The first-CUDA timer started at payload entry, before filesystem, GPU inventory and NVLink topology checks. It stopped after a one-element CUDA operation, synchronization and an equality check. Environment preparation and imports fall inside this interval. Checkpoint loading and the eight-rank checks happen afterward. 1

The 19.411 s observation therefore describes getting this particular payload to its first completed CUDA operation with a retained environment. It does not measure how long checkpoint recovery took. The experiment also does not isolate package-install savings from every other difference between the two launches. 1

What the checkpoint test actually checked

The training probe used a 64-input, 16-output linear layer and AdamW. The initial run trained for eight steps. The second run required the checkpoint file to match the first node’s recorded SHA-256 and required its saved step to equal 8 before loading model parameters, optimizer state and CPU, CUDA and Python RNG states. It then ran one further step, checked finite loss and changed parameters, and committed a new checkpoint at step 9. 1

The filesystem check also verified that the 64 MiB cache retained its original hash. This gives us evidence that saved state survived the node replacement and was used successfully. It does not establish bitwise agreement with an uninterrupted training run, large-model recovery or distributed training checkpoint restoration. 1

The eight-GPU check was a separate torchrun job. Each rank completed a matrix multiplication check, and an NCCL all-reduce had to produce the expected value on every rank. All eight ranks reported success on both nodes. Those checks establish the tested GPU and communication behavior; they do not turn the single-GPU checkpoint probe into an eight-GPU training result. 1

What the shorter Ready interval can tell us

Both launches used the same platform source, payload, native instance type, region and retained filesystem. They were different VMs at different times. The receipts do not establish identical physical hosts, driver versions or immutable full-image digests. 1

The second node reached platform Ready 48.440 s sooner after API creation in this pair. That is an observation, not evidence that retaining state accelerated provisioning. Native STARTING-to-RUNNING took 332.287 s and 332.472 s respectively, while the earlier confirmation delays differed. The records do not divide those phases into boot, storage attachment and mount-setup costs. 1

Platform creation and Ready use platform timestamps. The two first-CUDA markers use the same guest clock within each run, so a constant guest offset cancels for that interval. They are wall-clock measurements; we did not record a clock-adjustment bound. Differences spanning platform, native-service and guest clocks retain unmeasured offset uncertainty. 1

What this result establishes

The evidence download preserves the timestamps, metric definitions, checkpoint hashes and source pins behind these observations. Both jobs named payload SHA-256 e5de06a7073e3d920b9daa82bfed52c1e4b521f66a1a59382ba4686c2ad66a3c and platform commit be542b312b5552c3c94e7cf8b7cfc53283d3f811. It is a prepared technical snapshot of retained receipts, not the full operational logs; its preparation-time publication status is preserved. The originals contain private account and infrastructure identifiers and are not published as-is. 1

We deliberately stopped the first node and created the second. This did not exercise automatic preemption recovery or a snapshot-restore mechanism. One pair cannot establish startup percentiles, uninterrupted capacity, an idle-capacity advantage or a comparison with other GPU providers. What it does establish is that the tested environment, cache and checkpoint survived a controlled node replacement, and the small training job continued. 1

A separate download provides a trace-conformance checker and synthetic example for future experiments. It checks planned-attempt coverage, declared timing and target bindings, and cost arithmetic. It does not replay hardware, authenticate receipts or establish usable training data or fleet idle-capacity performance. 2

For teams considering interruptible GPU workloads, we can scope a Nodus recovery pilot around your actual checkpoint format, retained data and first useful resumed operation. Those boundaries make the result useful to an engineering decision.

Sources

1. Download: retained-state observation, timing definitions, checkpoint continuity and source pins (JSON)
2. Download: trace-conformance checker, contract and synthetic example (ZIP)

Original article on Nodus Research

Discuss a GPU recovery pilot

An original synthetic reproduction of SOL-ExecBench issue #25 on CPU and CUDA: identity-valued inputs hid six affine errors. Small elementwise perturbations exposed all six while preserving the copied expected-output control.

One synthetic CPU float32 case (4×64, seed 42): all six deliberately incorrect affine formulas were accepted with identity inputs, and none with varied inputs. The copied-reference control passed both. Original issue reported by zzw4257; Nodus CPU reproduction. No GPU performance or training benefit measured.

Key takeaways

• All six deliberately incorrect formulas passed with identity affine inputs and failed after the input change in our synthetic CPU case and separate CUDA confirmation.
• A random activation tensor does not test whether a program uses its weights and biases correctly. Those parameters must exercise the behavior being checked.
• These are correctness-generator observations on one synthetic workload, with separate CPU and CUDA results. They do not measure GPU leaderboard effects, kernel speed, or training improvement.

The check passed because the inputs hid the mistake

We sent six deliberately incorrect normalization formulas through SOL-ExecBench’s numerical checker on an original synthetic CPU workload. Every formula was accepted. After changing the generated normalization weights and biases from identity values to small elementwise random perturbations, all six were rejected. The correct-output control remained accepted. 2

The underlying issue and proposed perturbation were reported by zzw4257 in SOL-ExecBench issue #25. Our contribution is an independent CPU reproduction using a new synthetic Definition and Workload, explicit-constant controls, and a local regression patch. We published the reproduction as a follow-up to that report. 1 2

We later confirmed the same before-and-after verdicts on a Tesla T4 with PyTorch 2.9.0+cu130. That CUDA run is reported separately below; the graphic and six numerical error values remain results from the original CPU run. 7 8

When six different mistakes produce the same answer

Write the normalized activation as z = layer_norm(x). The intended affine result is y = z × weight + bias. In the pinned default generator, recognized normalization weights become all ones and normalization biases become all zeros—even when the workload requests random inputs. 3

For these inputs, the expected result is simply z. Leaving out the weight, leaving out the bias, or leaving out both produces the same result. Reading element zero of each affine tensor also works because every element is identical. Applying the identity affine transform twice changes nothing. Applying it to x before normalization changes nothing either.

Randomizing x cannot separate those formulas while weight and bias remain identity values. This is an input-coverage problem: on this restricted input set, the checker receives the same output from the intended computation and each wrong variant.

The six observed CPU results

The case uses x with shape 4 × 64, 64-element affine tensors, float32, and seed 42. We called the real gen_inputs and compute_error_stats functions from upstream commit a9fa0804c793d438e70850c33fe34426e66d53dd. The checker used max_atol = 0.01, max_rtol = 0.01, and required_matched_ratio = 0.99. No benchmark dataset was loaded. 2 4

Each row shows acceptance before → after, followed by the maximum absolute error after the change. All six errors were exactly 0.0 before the change. Error values below are rounded to six decimal places; the public reproduction records full precision. 2

• Omit both weight and bias — accepted → rejected; max absolute error 0.523450.
• Omit the weight — accepted → rejected; max absolute error 0.540673.
• Omit the bias — accepted → rejected; max absolute error 0.214417.
• Use weight[0] and bias[0] for every element — accepted → rejected; max absolute error 0.635101.
• Apply the affine transform twice — accepted → rejected; max absolute error 0.473204.
• Apply the affine transform before normalization — accepted → rejected; max absolute error 0.092222.

A narrow change to the default generator

The local patch draws each weight element as 1 + 0.1 × N(0,1), and each bias element as 0.1 × N(0,1). Values remain near familiar normalization parameters, but the tensors no longer act as identity constants. In this run, each affine tensor changed from one unique value to 64. These are Gaussian draws, not strict numerical bounds. 2

The change applies to the default random-input path. Explicit custom and safetensor inputs retain their values, and the existing low-precision fallback remains unchanged. Intentional constants still have a valid place in a test contract. Dataset-specific get_inputs functions need their own review; this two-branch change does not audit them. 3 2

The CPU correct control was a copy of the expected affine output. It passed before and after with zero error. That confirms the positive case for this numerical-checker experiment; it is not an independent correct implementation. 2

What the CPU run verified

The original CPU experiment ran on macOS arm64 with Python 3.12.14 and PyTorch 2.14.1. The upstream project pins PyTorch 2.9.0 with a CUDA stack. The CPU error values and test results retain their original environment scope; the later CUDA confirmation is a separate run. 2 5

With the patch, the input-generation and numerical-correctness test files passed all 197 tests on CPU. The focused regression covers float32, float16 and bfloat16, the six wrong affine transformations, and preservation of explicit custom/safetensor constants. Ruff checks and git diff --check also passed. These checks validate the local CPU change; they are not a GPU benchmark. 2

The published issue follow-up records that CPU reproduction and proposes the limited change for maintainer feedback. It does not establish that a fix has been accepted upstream. 2

CUDA confirmation on one Tesla T4

On November 9, 2026, we ran the original synthetic workload through the selected public SOL-ExecBench input-generation and numerical-checker paths on CUDA. All six incorrect formulas were accepted with identity affine inputs and rejected after the narrow generator change. The copied-reference control passed both conditions. These are 14 candidate observations—seven outputs under two input conditions—on one float32 case with shape 4×64 and seed 42. 7

The recorded environment was a Tesla T4, Python 3.12.10, PyTorch 2.9.0+cu130 and NVIDIA driver 610.57.04, reported through NVML. The selected public source was pinned to a9fa0804c793d438e70850c33fe34426e66d53dd. The numerical checker was unchanged; the input-generator patch varied the normalization weights and biases. The run synchronized CUDA after generation, reference computation, each candidate and each checker call. 8

The activation tensor was identical across the before and after phases within this CUDA run. CPU and CUDA random-number streams are not asserted to be bit-identical, so the historical CPU error values above must not be read as CUDA measurements. The downloadable result records the CUDA tensors, numerical errors and verdicts; its provenance file records the result hash, source pins, runtime and review scope. 7 8

To repeat the CUDA check, download the public source bundle, runtime lock bundle, instructions and SHA-256 checksum file linked here. The procedure targets Linux x86_64 with Python 3.12. Installing the locked runtime downloads 30 pinned wheels, about 2.74 GB; the bundles contain no wheels, and Modal is not required. The instruction file provides the CPU composition check and CUDA commands. 9 10 11 12

This confirmation used one original synthetic fixture and selected public functions. It did not execute the full SOL evaluator or its dataset, rerun the 197-test CPU suite on GPU, measure kernel speed, evaluate a model, establish training benefit, or qualify a production environment. An earlier invocation failed while transporting its output; no numerical result was recovered from that invocation. The successful run used the same built runtime with a corrected result-transport path, as disclosed in the provenance. 8

Repeat the original CPU reproduction

Our linked issue follow-up contains the complete standalone Python script, the two-line source change, and the observed results. It constructs its own Definition and Workload and calls the upstream generator and numerical checker directly. 2

To repeat the comparison, use an isolated checkout at a9fa0804c793d438e70850c33fe34426e66d53dd and save the script as reproduce_sol_norm_affine.py. With the Python dependencies available, run PYTHONPATH=src python reproduce_sol_norm_affine.py from the checkout root. Save the JSON, apply the two-line generator change shown in the issue follow-up, then run the same script again. The output includes the device, Torch version, seed, tolerance and verdict for every candidate. 2

Treat the recorded environment as part of the result. Other Torch versions or hardware can produce different random samples and numerical values. A six-case result does not establish exhaustive correctness, and our synthetic script makes no claim about the benchmark’s full task distribution.

Inspect the recorded outputs

The interactive fixture lets you switch between identity and varied affine inputs, inspect all six incorrect variants, and locate individual output differences. It displays the recorded Python tensor values and checker decisions. The browser does not execute a GPU kernel or rerun the checker; the copied expected-output control remains a sanity check. 6

A correctness label can become training data

In a kernel-writing environment, the correctness decision can determine whether a candidate receives a performance reward. If a wrong affine computation is indistinguishable from the reference on generated inputs, the resulting label can look valid. That creates a plausible route to misleading training feedback. We have demonstrated the input blind spot here; we have not measured its effect on a trained model.

This gives evaluator authors a concrete check: deliberately omit a scale, omit a bias, reuse the wrong parameter index, or apply a transformation twice. Confirm that the intended reference passes and that each relevant mutation fails under the declared numerical contract. Record which behavior each input distribution exercises. A larger number of evaluations cannot compensate for an operation the inputs never test.

For labs training kernel-writing agents, we would like to compare a small set of meaningful wrong-program controls against your current evaluator and scope a Nodus environment or data pilot. The useful starting point is a specific skill, a protected correctness contract, and a result both sides can reproduce.

Sources

1. Original report by zzw4257: SOL-ExecBench issue #25
2. Nodus follow-up: standalone synthetic CPU reproduction, observed results and proposed patch
3. Pinned SOL-ExecBench input generator
4. Pinned SOL-ExecBench numerical checker
5. Pinned SOL-ExecBench dependency versions
6. Interactive LayerNorm counterexample with recorded CPU tensors and provenance
7. Download: original synthetic CUDA result (JSON)
8. Download: CUDA result provenance, source pins and scope (JSON)
9. Download: public CUDA reproduction source bundle
10. Download: pinned CUDA runtime lock bundle (no wheels)
11. Read: CUDA reproduction instructions
12. Download: SHA-256 checksums for the reproduction files

Original article: Six wrong normalization formulas passed a correctness check

Discuss a kernel evaluator pilot

Set every normalization weight to one and every bias to zero. A program that skips the entire affine transform can now return the expected answer. So can one that applies the transform twice.

That is a concrete problem for kernel reinforcement learning: the correctness label can become part of the training signal.

Following zzw4257’s report in SOL-ExecBench, we built an original synthetic CPU reproduction. Six deliberately incorrect affine formulas passed the numerical checker with identity inputs. Small elementwise perturbations of the weights and biases caused all six to fail; the correct-output control still passed. The public follow-up includes the script and results. This was CPU correctness evidence, not a GPU performance or training experiment.

The example explains why Nodus’s benchmark program separates three questions: can an agent produce correct, fast kernels; can an environment provide trustworthy feedback efficiently; and does training on the resulting data improve performance on new work?

KernelBench provides a starting point for the first question. A useful score still needs the task revision, reference implementation, hardware and generation budget. Otherwise a stronger model, longer search or weaker baseline can look like a data improvement.

Our proposed measurement experiment addresses the second question. Freeze candidate/reference pairs, measure them in isolation, then replay them under declared GPU sharing and contention settings. Keep the task, judge, software stack and hardware profile fixed. A full GPU and a MIG slice are different targets, so a ranking change between them is not automatically a measurement error.

We want to know which candidates labeled fast under sharing remain fast in isolation. We will retain ambiguous near-ties, timing uncertainty, failed attempts and missing results. Repeating a timing loop does not create additional independent tasks. Correctness must hold during timing as well as before it; skipped computation cannot count as a speedup.

Cost belongs in that record too. Compilation, startup, interrupted work, retries and validation consume resources. The useful unit is an independently validated result, with its attributable cost and elapsed time. An unknown charge is not a zero-dollar result.

The third question needs a separate learning experiment. Start from the same trainable checkpoint, freeze the training recipe, and hold out source or workload families before generating training examples. Compare a baseline with the proposed data treatment under an explicitly matched budget, then evaluate on untouched work. Report regressions and uncertainty alongside improvements. Extra shape variants of a familiar task do not establish transfer.

This is the research design described in The reward is part of the dataset. We have not reported a measured Nodus training gain or provider-comparison result from these proposed experiments.

If your lab trains agents that write GPU kernels, discuss a scoped Nodus environment or data pilot. We would start with your evaluation protocol and the improvement that would make the data worth using.