| title | Extending nnspec | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| one_liner | Adding an op, a featurizer kind, a component or a step kind: the decorator, the signature as the document's contract, where the tables come from, and the tests to add. | ||||||||||
| tags |
|
||||||||||
| related |
|
||||||||||
| sources |
|
nnspec's vocabulary is read off the code: an op's signature is what a document may write for it, a featurizer kind's constructor is its fields, and FORMAT.md is generated from both. So extending nnspec means adding a function or a class in the right place, registering it, and regenerating FORMAT.md. A new op, featurizer kind, component or step kind is a format change: propose it with before-and-after JSON, and it lands when the owner ratifies it (contributing.md). Adding an engine is engines.md.
nnspec/format/registry.py check binds a document's arguments to the signature of the function they go to (an op, a featurizer kind, an engine's constructor) before anything runs:
- A parameter's name is the argument's spelling, with a trailing underscore dropped (
with_is"with"). An unknown key is refused naming the valid ones, and a parameter without a default is required. - A parameter's annotation is how a document fills it.
Value(a named value or a number),Tokens(a column resolved through the value's vocabulary),Text(a column read as text) andFeaturizer(a featurizer's name) are resolved by compile; any other annotation is a literal type, checked strictly. - Parameters the run supplies are never the document's. For an op, those are
nnspec/ops/__init__.pySUPPLIED(engine,original,dims,layers,eos,rows,positions). A document that gives an op one of them is refused by name (nnspec/ops/__init__.pysupplied). For a kind, they arewidth,bundle,header,labelsandkept, and to itsapplythedimsandlayersof the value it meets. Name one in the signature to receive it. - The docstring's text is what
nnspec describeand FORMAT.md print beside it.
- Write a plain function over tensors in its group's module (
nnspec/ops/metrics.py,reductions.py,edits.pyortransforms.py), its first parameterof, the value it acts on. The function's__name__is the op's name in a document.- A builtin's name shadows the builtin in the whole module.
nnspec/ops/reductions.pysum,minandmaxalready do, and so donnspec/ops/transforms.pyabsandpow. Call the builtin there asbuiltins.<name>, or pick another name.
- A builtin's name shadows the builtin in the whole module.
- Decorate it with
@op(group, ...)(nnspec/ops/__init__.pyop), declaring its axes and checks:needs, the axes the value must carry;removesandadds, the axes it takes away and puts on.layout=, a function of the value's axes, the arguments and the featurizer kind, where the axes depend on the arguments.- A reduction that takes
overdeclareslayout=reduced(nnspec/ops/reductions.pyreduced). It removes the axesovernames, by defaultrowsandpos, and refuses a reduction that removes none. Namedimsin the signature to receive the value's axes. bundle=<kind>, marking a value a featurizer of that kind loads.check=, raising on arguments the types accept but the op cannot mean.positionwise=True, where its value at each position comes from that position's vector alone, with the same bits however many other positions the value holds. A generation then runs it at each call it reads (usage/generate.md). That holds for a gather, a softmax over one axis, a rank, and arithmetic IEEE rounds exactly (+,*,/,sqrt, a comparison).- On a reduction,
positionwise=Truesays each position's result over the feature axes it removes has the same bits whatever the value's shape.max,min,medianandcountdeclare it;mean,sumandstddo not. A generation runs it at each call only while the result keepsrowsandpos, so a reduction overposruns after the generation. - Leave
positionwiseoff an op that multiplies matrices or sums over an axis. Their reduction order follows the value's shape, so on a call's[rows, 1, ...]they can differ in the last bits from the same op on the whole read. Without it, the op runs after the generation.
- Add the function to the ordered list at the end of
nnspec/ops/__init__.py.@opdeclares the op; the list registers it.import nnspecfails withevery declared op is listed once; not <name>until the list has it, once. Within a group, that list's order is the tables' order. The prose FORMAT.md prints names the edits group from that list (nnspec/ops/__init__.pylisted, written intoABOUTand theInterventiondocstring innnspec/plan.py), so a new edit joins it there with nothing written by hand.
A minimal edit, added to the edits module and listed after clamp. It is for illustration only: the format already spells negation as the transform neg, and a second spelling does not land (contributing.md).
@op("edits")
def negate(of: Tensor) -> Tensor:
"""`-of`."""
return -ofnnspec describe ops then lists negate() [keeps the layout] with its docstring, an entry's {"op": "negate"} writes the negated value back on both engines, and {"op": "negate", "factor": 2} is refused when the plan is built:
steps.flipped.interventions.neg.ops[0]: 'negate' takes no 'factor'; its arguments: none
To see such a refusal without a run, compile a scratch document: nnspec compile scratch.json prints it, and Plan("scratch.json").compile() raises it (running.md).
A metric over next-token logits declares the vocabulary axis it scores:
@op("metrics", needs=(VOCABULARY,), removes=VOCABULARY)
def top_prob(of: Tensor) -> Tensor:
"""The probability of the most likely next token."""
return of.float().softmax(-1).amax(-1)A tuple inside needs is any one of its axes. nnspec/ops/__init__.py VOCABULARY is vocab_size or a probe's classes, so this metric scores logits and probe scores alike, and removes=VOCABULARY takes away whichever the value has.
What the group gives it, with no code of its own:
- An edit runs in float32 and is rounded once to the value's dtype (
nnspec/ops/__init__.pycall). Only an edit is upcast. - A metric or a reduction receives the value in the model's dtype and casts it itself, as
of.float()does above, since the format promises float32 from both (usage/ops.md). - An op whose result has no
rowsaxis is refused inside an entry, which sees one batch. - An op with an
originalparameter runs only inside an entry (nnspec/compile/arguments.pylayout). - A missing value arrives as NaN and should leave as NaN: compute plainly, with no
nansumornan_to_num.
What the groups mean for a document is usage/ops.md.
- One module under
nnspec/featurizers/holding one class that subclassesKind(nnspec/featurizers/__init__.pyKind). The class's name in lower case is the kind, and subclassing registers it inKINDS. - Import the module in the line at the end of
nnspec/featurizers/__init__.py. Until then the class never loads, and a document naming it ("kind": "shift"for the kind below) is refused:unknown featurizer kind 'shift'; valid: .... That line's order is the tables' order. - Its constructor is the document's fields. A field annotated
Valueis a number or the name of a value a step makes, as an op's is. Compile holds the name to one vector of the featurizer's width, made before its first use (nnspec/compile/uses.pycheck_named_values). The run passes the constructor that value where it makes the instance (instance), as a constant'sinitis. - Its class attributes say what its bundle holds, what a fit trains, its coordinates' axis, its header fields, what a drawn one needs, its scalars and which fields count directions; their meanings are in FORMAT.md. Its methods
encode(h),replace(h, z, new)andapply(h, ...)serveencode,decodeandapply; define only those it has, and it serves exactly those. - Hold tensors as plain attributes and move them to the value where they meet it (
self.b.to(h)). The three methods run with autocast off, and a missing (NaN) vector goes in as zeros and comes out missing, without code of the kind's own (nnspec/featurizers/__init__.pyexact,Kind.call). Anapplywhose every element of the result is made from that element of the value alone declareselementwise = True, as a gate's does. Then a missing element, not its whole vector, goes in as zero and comes out missing. That matters at akeyaxis, NaN past a short row's own keys, where a vector-wise rule would write the whole row back missing.
A minimal kind, which a fit trains and saves as a bundle with nnspec's header like any other:
class Shift(Kind):
"""An offset `b` (starting at zero) added to every vector: its one op is `apply`."""
saves = trains = ("b",)
def __init__(self, width: int, file_path: str | None = None, bundle: dict[str, Tensor] | None = None) -> None:
self.b = bundle["b"] if bundle is not None else torch.zeros(width)
def apply(self, h: Tensor) -> Tensor:
return h + self.b.to(h)A component is a row of nnspec/format/address.py COMPONENTS: its meaning, whether it is per_layer, its dims, and whether it is a residual-stream value readout may project (stream).
- Add a component only under a name nnterp's
support()lists. The engines reach a component only through nnterp, and its name is nnterp's name for the value. Check that nnterp'sStandardizedTransformer.support()lists the name before adding the row. A name it does not answer for is refused for every document, whether it uses the component or not: the hooks engine refuses it when it loads, the nnterp engine before the run's first step (nnspec/engines/standard.pyunserved:nnterp's support() does not name <name>, so nothing says whether it serves <component>; ...). - Its
dimsare nnterp's layout of it:batchfirst, then the token axisseq, orqueryon an attention pattern. A site'sselectmay narrow every other axis. - On nnterp and ndif, the engine reads and sets it on the envoy that hosts it (
nnspec/engines/standard.pyenvoy). It asks nnterp'ssupport()whether it is served and nnterp'srankwhere it falls. - A value nnterp serves on the attention or the MLP is also listed in
nnspec/engines/standard.pyBELOW, under its host. A mixture of experts' values are the MLP's, nnterp'sMoebeing one. - On hooks, the engine resolves at load the module boundary nnterp's key for it names. It refuses by name one nnterp reads inside a module's forward or among a module's call arguments.
- The load's two-token forward must reach the boundary;
nnspec/engines/hooks.pyorderrefuses by name one it does not reach. - The boundary's tensor must have the component's rank, one less where nnterp holds the value flat over tokens, or three for a four-axis per-head layout. A boundary of any other rank is served as it holds the value, without a refusal.
- The load's two-token forward must reach the boundary;
- An axis that is a size of the model joins
nnspec/engines/standard.pySIZES, whichsizesreads off nnterp (num_expertsandtop_koff itsMoe, absent on a model without one).
A step kind is the widest change: the kind is dispatched in each stage.
nnspec/plan.py: aRecordsubclass withkind: Literal["<kind>"], its entry inKINDS, and a tagged member of theStepunion. Where the step has a value of its own, the subclass is aLabelledone, whose record carries that value's layout andaxes.servingsays which of its points run.nnspec/compile/__init__.pycompile_step: the function that compiles it, writing its records' layout and the sweep axes each value depends on.nnspec/compile/preflight.pyprepare: what it checks with the model. A kind without a branch of its own there, asop,fitandloadhave, is prepared as a forward, its serving points'schedulewritten bynnspec/compile/schedule.pywrite_schedule.nnspec/steps/__init__.pyKINDS: the function that runs it, called as(engine, plan, name, step, state, batch_size, findings, cost), adding the model calls it makes and the tokens they feed tocost; a fit's body runs through the same table.nnspec/plan.pycalls_model: whether a run loads the shell's weights. A new kind that calls the model and is not listed there runs on a meta shell.nnspec/steps/__init__.pyNO_MODEL: the kindsmeasure_costtimes without synchronizing the model's devices.- The rest of the code that branches on a step's kind:
grep -rnE "\.kind (==|!=|in|not in)|NO_MODEL" nnspec/.
nnspec describe steps and FORMAT.md list nnspec/plan.py KINDS with each record's docstring.
| change | tests |
|---|---|
| an op | tests/test_ops.py: the op on plain tensors. Its two listing tests, test_every_op_is_listed_once_in_order_under_its_own_name and test_each_op_declares_its_group_and_bundle, fail until the new op is in their lists. A metric or reduction joins tests/test_findings.py test_every_metric_and_reduction_is_float32_on_a_bf16_model_and_an_edit_stays_bf16 in two places: a step in its steps, and the step's name in its names |
| a featurizer kind | tests/test_featurizers.py every_kind, so the device and autocast tests cover it; tests/test_featurizer_ops.py, whose test of each kind's ops lists every kind and fails until updated; a kind with both encode and decode joins the parameters of its identity-to-the-bit test |
| a component | a corpus document that uses it, run on both engines; a row in tests/test_engines.py, since a component means one thing on every engine |
| a step kind | a corpus document that uses it, run on both engines |
| a corpus document | pins for its numbers and its compiled form, and the README's corpus count, which tests/test_plan.py checks. One that trains or loads a featurizer is named in tests/conftest.py FIT, one that generates in GENERATE, and one the hooks engine refuses in INSIDE; any other runs as a forward in tests/test_documents.py |
A new op also changes the docs (contributing.md). It gets a when-to-use bullet in its group's section of docs/usage/ops.md, and joins every list of its group's ops under docs/, for a reduction in docs/usage/ops.md and docs/reference/glossary.md. Grep a sibling's name to find them.
An op, a kind, a component or a step kind changes FORMAT.md, which nnspec/docs.py markdown renders from the registries: regenerate it with uv run --no-sync nnspec describe --markdown, or tests/test_format.py fails (contributing.md). Pinning a new document is testing.md; what lands with the change is contributing.md.