Repository navigation
Fix transplant, predict, NaN and checkpoint edge cases (3.14.1) - #29
Merged
Merged
Conversation
transplant_weights initialises W with micro_quiet_warm before copying the overlap, and that strategy fills the diagonal of the new region too. After a 3 -> 6 transplant the diagonal sat at ~1e-3 until the first optimizer step, so the first forward pass used self-connections that belong to memory_feedback. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Mgvqnofqp2HekNAmhM7Kov
train_batch injects an integer tensor by neuron index, but predict and evaluate sent it through prepare_input, which treats it as feature values and fails on the dtype mismatch. Both paths now share one input helper. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Mgvqnofqp2HekNAmhM7Kov
A non-pulse forward stored its scaled input on the model, which kept the
whole input graph alive until the next call and made copy.deepcopy(model)
fail ("only graph leaves support the deepcopy protocol"). Every call
refreshes the cache at t=0, so nothing needs it between calls; as an
attribute it could only re-inject a stale batch into an empty sequence.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mgvqnofqp2HekNAmhM7Kov
A NaN target or an overflowing loss on CPU (or bf16) sent NaN through ChaosGrad's moments and into W on the first step, and the model never recovered: the next good batch still produced NaN weights. GradScaler guards this only under CUDA AMP. The trainer now checks the norm that clip_grad_norm_ already computes, skips the step with a RuntimeWarning, and still calls scaler.update() so loss-scale backoff is unchanged. Not exercised on CUDA; the GradScaler path is covered by reading, not by a run. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Mgvqnofqp2HekNAmhM7Kov
Rebuilding ChaosGrad after an expansion recomputed rms0 from the trained, now larger weights, so the traction cap stopped being anchored to the initial scale (chaos_core 0.27 -> 0.21, projections 0.019 -> 0.042 on a 6 -> 8 neuron probe) and re-anchored again on every growth. The old value is now carried with the other per-group estimates. The "optimizer state transferred" line printed regardless of verbose. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Mgvqnofqp2HekNAmhM7Kov
save_checkpoint wrote straight onto the target, so a crash or a full disk mid-write destroyed the previous checkpoint, which for the _latest/_best pairs the long examples keep is the only copy. It now writes beside the target and renames over it, removing the partial file on failure. get_checkpoint_info summed every state_dict entry, buffers included: the index tables and, with hebb_type set, the N x N carries (280 against get_num_params()'s 132 at 8 neurons with hebb_type='both'). Buffers are now left out. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Mgvqnofqp2HekNAmhM7Kov
pytorch-lightning and tqdm were runtime dependencies that nothing in the package or the examples imports. The README quick start hard-coded device='cuda' and failed on a machine without a GPU. 398 tests pass on Python 3.13.16 with torch 2.14.1, so CI gains that version. Documents the skipped non-finite step and the atomic save, and bumps to 3.14.1. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Mgvqnofqp2HekNAmhM7Kov
weight_init='xavier_unifrom' silently became uniform(-0.1, 0.1), a mistyped vocab_mode built neither embed nor proj so the input was ignored, and neuron ids outside [0, num_neurons) failed as an IndexError deep in forward (negative ids wrapped to the last neuron without any error). All three now raise ValueError when the model is built; uniform becomes an explicit strategy so nobody relying on the old fallthrough by name breaks. One trainer test built a 4-neuron model with output id 4; it now uses valid ids. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Mgvqnofqp2HekNAmhM7Kov
The docstring and LIBRARY.md claimed strict=False ignores size mismatches and loads what fits. PyTorch skips only missing and unexpected keys; a 4 -> 6 neuron load still raises RuntimeError. The text now says that and points to transplant_weights for loading across sizes. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Mgvqnofqp2HekNAmhM7Kov
LIBRARY.md said a new neuron's incoming weights start at zero. expand() fills both directions with micro_quiet_warm noise (max 1.2e-3 into the new neurons and 2.5e-3 out of them on a 5 -> 7 probe), which the note directly below already stated. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Mgvqnofqp2HekNAmhM7Kov
OdyssNetTrainer defaulted to device='cpu' and its constructor calls model.to(device), so a model built with device='cuda' was moved back to the CPU unless the device was repeated on the trainer. Passing device= explicitly behaves as before. This is read from the code, not run: the machine this was written on has no GPU. The test pins that the default follows the model. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Mgvqnofqp2HekNAmhM7Kov
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A sweep over 192 model configurations plus a read of the core modules found a handful of real edge-case bugs; each is fixed with a test that fails on the old code. Seven CPU examples (
gates,identity,adder,latch,stopwatch,sine_wave,detective_thinking) produce line-for-line identical output onmainand on this branch.Changes
transplant_weightsleft the core diagonal at ~1e-3 after a scale-up; it is zeroed now.predict/evaluaterejected the integer index tensorstrain_batchaccepts; both share one input helper.forwardkept its input graph on the model and brokecopy.deepcopy; the cache is local to the call.Wfor good on CPU; the trainer skips the step with aRuntimeWarning.Neurogenesis.expandre-measured the traction anchor (rms0) on trained weights; it is carried over.verbose=Falseis silent.save_checkpointis atomic;get_checkpoint_infono longer counts buffers as parameters.weight_init,vocab_modeand out-of-range neuron ids raiseValueErrorat construction.OdyssNetTrainerdefaults its device to the model's own instead of'cpu'.pytorch-lightningandtqdmdependencies, added Python 3.13 to CI, fixed the CPU quick start, corrected three doc claims. Version 3.14.1.Not run: the CUDA paths (GradScaler with the skip, the trainer device default); there was no GPU. The 3.13 CI job runs for the first time here. Left alone on purpose: silent clipping of extra features and of tokens past
steps, both pinned by existing tests.Checklist
All contributions
sys.path.appendhacks — imports usefrom odyssnet import ...directlyLibrary changes (
odyssnet/)tests/New/modified example scripts (
examples/)🤖 Generated with Claude Code
https://claude.ai/code/session_01Mgvqnofqp2HekNAmhM7Kov
Generated by Claude Code