# Keeping training state when the GPU node changes

Disclosure: This is an AI-generated article.

A controlled Nodus test kept a Python environment, cache and checkpoint across two sequential eight-H200 nodes. We separate platform readiness, the first CUDA operation and the single-GPU checkpoint recovery.

![Two separate timing intervals from one controlled replacement pair. API creation to platform Ready: initial node 421.205 s, replacement node 372.765 s. Payload entry to first completed CUDA: fresh environment 221.654 s, retained environment 19.411 s. Checkpoint recovery was measured separately as successful advancement from step 8 to 9 on one GPU, without a restore-duration measurement.](https://data.nodus-compute.ai/assets/lab-research-retained-state-timings.png)

One controlled replacement pair; one observation per phase. First CUDA was measured before checkpoint restore. Checkpoint step 8→9 was separately verified on one GPU; eight-GPU checks were separate. The initial payload installed PyTorch and recovery reused the environment. No causal provisioning speedup is established.

## Key takeaways

- A second Nodus node loaded the exact saved checkpoint, restored a small model and optimizer, and advanced training from step 8 to step 9.
- The observed first-CUDA interval after payload entry was 221.654 s initially and 19.411 s with the retained environment. Both measurements precede checkpoint restoration.
- This was one controlled replacement pair. Separate eight-rank GPU checks passed; the training checkpoint exercise used one GPU.

## The node changed. The saved work stayed.

We shut down one Nodus eight-H200 node, started a second node on the same retained filesystem, and continued a small PyTorch training job from its saved checkpoint. The second node loaded the exact checkpoint written at step 8, restored the model, optimizer and random-number-generator states, and completed step 9. The prepared Python environment and a 64 MiB cache were still there. [1](https://data.nodus-compute.ai/assets/lab-research-retained-state-evidence.json)

This was a controlled production test on November 2, 2026: two sequential nodes, one retained 512 GiB filesystem, and one observation per phase. Both nodes also passed separate CUDA and NCCL checks across all eight GPUs. The checkpoint exercise itself ran on one GPU. [1](https://data.nodus-compute.ai/assets/lab-research-retained-state-evidence.json)

## Three different readiness questions

A node reaching Ready, a program completing its first CUDA operation, and a training job restoring its state are different milestones. We recorded the first two as timed intervals. The checkpoint test records successful restoration and advancement, without a separate restore-duration measurement. [1](https://data.nodus-compute.ai/assets/lab-research-retained-state-evidence.json)

- API creation to platform Ready: 421.205 s on the initial node; 372.765 s on the second node.
- Detached payload entry to its first completed CUDA operation: 221.654 s initially; 19.411 s on the second node.
- Checkpoint continuity: the initial job saved step 8; the recovery job verified that file’s SHA-256, loaded its state, and saved step 9.

## What changed between the two CUDA measurements

The initial payload created a virtual environment and installed torch==2.14.0 from the CUDA 12.6 package index. The recovery payload executed the retained interpreter directly. It checked that the Torch version, CUDA version, Python version and Torch path matched the initial environment metadata. Its code had no package-install branch. [1](https://data.nodus-compute.ai/assets/lab-research-retained-state-evidence.json)

The first-CUDA timer started at payload entry, before filesystem, GPU inventory and NVLink topology checks. It stopped after a one-element CUDA operation, synchronization and an equality check. Environment preparation and imports fall inside this interval. Checkpoint loading and the eight-rank checks happen afterward. [1](https://data.nodus-compute.ai/assets/lab-research-retained-state-evidence.json)

The 19.411 s observation therefore describes getting this particular payload to its first completed CUDA operation with a retained environment. It does not measure how long checkpoint recovery took. The experiment also does not isolate package-install savings from every other difference between the two launches. [1](https://data.nodus-compute.ai/assets/lab-research-retained-state-evidence.json)

## What the checkpoint test actually checked

The training probe used a 64-input, 16-output linear layer and AdamW. The initial run trained for eight steps. The second run required the checkpoint file to match the first node’s recorded SHA-256 and required its saved step to equal 8 before loading model parameters, optimizer state and CPU, CUDA and Python RNG states. It then ran one further step, checked finite loss and changed parameters, and committed a new checkpoint at step 9. [1](https://data.nodus-compute.ai/assets/lab-research-retained-state-evidence.json)

The filesystem check also verified that the 64 MiB cache retained its original hash. This gives us evidence that saved state survived the node replacement and was used successfully. It does not establish bitwise agreement with an uninterrupted training run, large-model recovery or distributed training checkpoint restoration. [1](https://data.nodus-compute.ai/assets/lab-research-retained-state-evidence.json)

The eight-GPU check was a separate torchrun job. Each rank completed a matrix multiplication check, and an NCCL all-reduce had to produce the expected value on every rank. All eight ranks reported success on both nodes. Those checks establish the tested GPU and communication behavior; they do not turn the single-GPU checkpoint probe into an eight-GPU training result. [1](https://data.nodus-compute.ai/assets/lab-research-retained-state-evidence.json)

## What the shorter Ready interval can tell us

Both launches used the same platform source, payload, native instance type, region and retained filesystem. They were different VMs at different times. The receipts do not establish identical physical hosts, driver versions or immutable full-image digests. [1](https://data.nodus-compute.ai/assets/lab-research-retained-state-evidence.json)

The second node reached platform Ready 48.440 s sooner after API creation in this pair. That is an observation, not evidence that retaining state accelerated provisioning. Native STARTING-to-RUNNING took 332.287 s and 332.472 s respectively, while the earlier confirmation delays differed. The records do not divide those phases into boot, storage attachment and mount-setup costs. [1](https://data.nodus-compute.ai/assets/lab-research-retained-state-evidence.json)

Platform creation and Ready use platform timestamps. The two first-CUDA markers use the same guest clock within each run, so a constant guest offset cancels for that interval. They are wall-clock measurements; we did not record a clock-adjustment bound. Differences spanning platform, native-service and guest clocks retain unmeasured offset uncertainty. [1](https://data.nodus-compute.ai/assets/lab-research-retained-state-evidence.json)

## What this result establishes

The evidence download preserves the timestamps, metric definitions, checkpoint hashes and source pins behind these observations. Both jobs named payload SHA-256 e5de06a7073e3d920b9daa82bfed52c1e4b521f66a1a59382ba4686c2ad66a3c and platform commit be542b312b5552c3c94e7cf8b7cfc53283d3f811. It is a prepared technical snapshot of retained receipts, not the full operational logs; its preparation-time publication status is preserved. The originals contain private account and infrastructure identifiers and are not published as-is. [1](https://data.nodus-compute.ai/assets/lab-research-retained-state-evidence.json)

We deliberately stopped the first node and created the second. This did not exercise automatic preemption recovery or a snapshot-restore mechanism. One pair cannot establish startup percentiles, uninterrupted capacity, an idle-capacity advantage or a comparison with other GPU providers. What it does establish is that the tested environment, cache and checkpoint survived a controlled node replacement, and the small training job continued. [1](https://data.nodus-compute.ai/assets/lab-research-retained-state-evidence.json)

A separate download provides a trace-conformance checker and synthetic example for future experiments. It checks planned-attempt coverage, declared timing and target bindings, and cost arithmetic. It does not replay hardware, authenticate receipts or establish usable training data or fleet idle-capacity performance. [2](https://data.nodus-compute.ai/assets/nodus-trace-conformance-v1.zip)

For teams considering interruptible GPU workloads, we can scope a Nodus recovery pilot around your actual checkpoint format, retained data and first useful resumed operation. Those boundaries make the result useful to an engineering decision.

## Sources

1. [Download: retained-state observation, timing definitions, checkpoint continuity and source pins (JSON)](https://data.nodus-compute.ai/assets/lab-research-retained-state-evidence.json)
2. [Download: trace-conformance checker, contract and synthetic example (ZIP)](https://data.nodus-compute.ai/assets/nodus-trace-conformance-v1.zip)

[Original article on Nodus Research](https://data.nodus-compute.ai/blog/keeping-state-across-replacement-gpu-nodes?utm_source=hashnode&utm_medium=research&utm_campaign=kernel_rl_benchmarks)

[Discuss a GPU recovery pilot](https://discovery-meeting.nodus-compute.ai/?utm_source=hashnode&utm_medium=research&utm_campaign=kernel_rl_benchmarks&utm_content=keeping-state-across-replacement-gpu-nodes)