Skip to content

Support nnsight >= 0.6 - #10

Merged
Butanium merged 2 commits into
science-of-finetuning:mainfrom
Sitavi:nnsight-0.6-compat
Jul 2, 2026
Merged

Butanium merged 2 commits into
science-of-finetuning:mainfrom
Sitavi:nnsight-0.6-compat

Conversation

@Sitavi

@Sitavi Sitavi commented Jun 29, 2026

Copy link
Copy Markdown

What

Make dictionary_learning compatible with nnsight >= 0.6 (currently capped at
<0.6).

Why

nnsight 0.5.x runs .save() / tracing on process-global state (Globals.stack,
Globals.saves), mounting/unmounting the .save/.stop proxies on every trace
enter/exit guarded by a non-atomic counter. Under overlapping traces (e.g. multi-GPU
device_map hooks, or concurrent traces) this desyncs and surfaces as
AttributeError: 'Tensor' object has no attribute 'save' / SystemError: unmount ....
nnsight 0.6 fixes this by redesigning the mechanism to mount-once / never-unmount.

Upgrading is blocked by two 0.5-only idioms in ActivationCache.collect():

  • last_submodule.output.stop() — the proxy .stop() early-stop was removed in 0.6
    → AttributeError: 'Tensor' object has no attribute 'stop'.
  • the scan= / validate= trace kwargs — obsolete in 0.6 (accepted as no-ops);
    dropped for cleanliness.

Changes

  • cache.py: capture the tracer (with model.trace(...) as tracer:) and call
    tracer.stop() instead of last_submodule.output.stop().
  • cache.py, buffer.py: drop the scan/validate trace kwargs (the DEBUG-gated
    dict → {}; nnsight 0.6 no longer has these kwargs).
  • pyproject.toml: nnsight>=0.4,<0.6 → nnsight>=0.6.

No public API changes; no behavior change to activation collection.

Validation

Tested in isolated venvs that differ only in the nnsight/nnterp version (same torch,
transformers, model — so any difference is attributable to the upgrade):

  • ActivationCache.collect() is bit-identical on nnsight 0.6.3 vs 0.5.15
    (max_abs_diff = 0.0, np.array_equal == True) — verified on a tiny model and on a
    real 3B model
    (layer 27, 3072-dim).
  • Without the patch, collect() raises AttributeError: 'Tensor' object has no attribute 'stop' on 0.6.3 (the blocker, reproduced).
  • Test suite: pytest tests/ → 12 passed / 4 skipped (CUDA-gated), identical on
    nnsight 0.5.15, 0.6.3 and 0.7.0.
  • Whole-package check (not just collect()): every nnsight idiom in
    evaluation.py / buffer.py / interp.py (invoker_args, model.output.save() /
    model.input.save(), submodule.input[0].save(), and the assignment-writes
    submodule.output = (x_hat,) / submodule.output[0][:] = 0) behaves identically
    across 0.5.15 / 0.6.3 / 0.7.0 (intervention-effective, same logit deltas). The change
    is behaviour-preserving for the whole package, not only the cache path.
  • Version range: the pin is left open (>=0.6) — both nnsight 0.6.3 and 0.7.0
    (with nnterp 1.3.0) pass the full check above.

nnsight 0.6 reworked the tracing/.save() mechanism (mount-once, never
unmount) and removed the proxy `.output.stop()` early-stop plus the
scan/validate trace kwargs.

- cache.py: capture the tracer (`with model.trace(...) as tracer:`) and call
  `tracer.stop()` instead of the removed `last_submodule.output.stop()`.
- cache.py, buffer.py: drop the scan/validate trace kwargs (no-ops in 0.6).
- pyproject: require nnsight>=0.6.

ActivationCache.collect() output is unchanged: activations are bit-identical
under nnsight 0.6.3 vs 0.5.15 (verified on a 3B model).
@Sitavi
Sitavi force-pushed the nnsight-0.6-compat branch from f25003e to 0bb3175 Compare June 29, 2026 18:48
@Butanium

Butanium commented Jul 1, 2026

Copy link
Copy Markdown
Member

@Sitavi tracer_kwargs is dead code now no? I think we cna remove it directlyt instead of just having it empty

The scan/validate kwargs were dropped for nnsight 0.6, leaving tracer_kwargs
as an empty dict splatted into .trace(). Remove it and the ** splats
(cache.py, buffer.py). interp.py keeps its own (still uses invoker_args).
@Sitavi

Sitavi commented Jul 2, 2026 •

Copy link
Copy Markdown
Author

Yeah good point, removed it! @Butanium

@Butanium
Butanium merged commit d1fd6f3 into science-of-finetuning:main Jul 2, 2026
1 check passed
@Butanium

Butanium commented Jul 2, 2026

Copy link
Copy Markdown
Member

Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants