| title | Steering | ||||||
|---|---|---|---|---|---|---|---|
| one_liner | How do I push the model with a direction? A mean difference at a dose set from the activations, a matched random control, clamping, and steering every generated token. | ||||||
| tags |
|
||||||
| related |
|
||||||
| sources |
|
Steering adds a direction to the residual stream to push the model toward a behavior, and measures what moves. In nnspec the direction is a value earlier steps compute (a mean, a difference of means), the write is add with a scale (the dose), and the evidence is a sweep over that dose with the unsteered run beside it.
documents/steer_renormalize.json takes its direction as the mean residual stream over the counterfactual prompts, then adds four times it at the last position, alone and followed by a renormalize:
"interventions": {
"steer": {"steer": {"site": "target", "pos": -1, "ops": [{"op": "add", "with": "direction", "scale": 4.0}]}},
"restore": {"restore": {"site": "target", "pos": -1, "ops": [{"op": "renormalize"}]}}
},
"steps": {
"harvest": {"kind": "forward", "input": "pairs.counterfactual_inputs[0]", "interventions": {"resid": {"site": "target", "pos": -1}}},
"direction": {"kind": "op", "op": "mean", "of": "harvest.resid"},
"steered": {"kind": "forward", "input": "pairs.input", "interventions": ["steer", {"logits": {"site": "head", "pos": -1}}]},
"steered_renormed": {"kind": "forward", "input": "pairs.input",
"interventions": ["steer", "restore", {"logits": {"site": "head", "pos": -1}}]}
}Adding a vector also lengthens the activation; renormalize after the add restores the norm and keeps the direction. Here the base residual stream at the last token has norm about 0.075 and steered.steer about 0.375. steered_renormed.restore is back to the base norm to float rounding, at cosine 1 with the steered vector. Comparing steered_ld with renormed_ld separates the effect of the new direction from that of the longer vector. restore comes after steer in the step's list (interventions has why).
A direction for a behavior is usually the difference between mean activations on prompts that show it and prompts that do not. The difference is an edit run as a step, sub over two means, and normalize makes it unit length so that the dose is a length:
{
"model": {
"key": "hf-internal-testing/tiny-random-LlamaForCausalLM",
"revision": "9fb191250dd56d0ba7ec9785a025ed29c03d5998",
"dtype": "float32"
},
"data": {"days": {"path": "weekdays/train"}, "counts": {"path": "counting/train"}},
"sites": {
"target": {"component": "layer_output", "layers": 0},
"head": {"component": "lm_head_output"}
},
"steps": {
"base": {"kind": "forward", "input": "days.input", "interventions": {
"resid": {"site": "target", "pos": -1}, "logits": {"site": "head", "pos": -1}}},
"other": {"kind": "forward", "input": "counts.input", "interventions": {"resid": {"site": "target", "pos": -1}}},
"mean_days": {"kind": "op", "op": "mean", "of": "base.resid"},
"mean_counts": {"kind": "op", "op": "mean", "of": "other.resid"},
"difference": {"kind": "op", "op": "sub", "of": "mean_counts", "with": "mean_days"},
"direction": {"kind": "op", "op": "normalize", "of": "difference"},
"steered": {"kind": "forward", "input": "days.input", "interventions": {
"steer": {"site": "target", "pos": -1, "ops": [
{"op": "add", "with": "direction", "scale": {"sweep": [0.0, 0.02, 0.08, 0.32], "label": "dose"}}]},
"logits": {"site": "head", "pos": -1}}},
"kl": {"kind": "op", "op": "kl", "of": "steered.logits", "against": "base.logits"},
"drift": {"kind": "op", "op": "mean", "of": "kl"},
"ld": {"kind": "op", "op": "logit_diff", "of": "steered.logits", "a": "days.cf_answer", "b": "days.base_answer"},
"effect": {"kind": "op", "op": "mean", "of": "ld"}
},
"saves": ["difference", "base.resid", "drift", "effect"]
}The means come from two datasets, which is allowed because a mean has no rows. base runs once; steered and the steps after it run once per dose. nnspec table <out> drift effect:
at drift effect
{"dose":0.0} 0.0 0.0009160377085208893
{"dose":0.02} 0.00020074972417205572 0.0005409605801105499
{"dose":0.08} 0.001653727376833558 -0.00017407722771167755
{"dose":0.32} 0.004028182942420244 -0.00015350989997386932
How to read and choose the doses:
- Dose 0 is the null (interventions).
addat"scale": 0.0leaves every logit equal to the unsteered run's, so itsdriftis exactly 0 and itseffectis the baseline. - Doses are lengths against the activation's norm.
base.residis saved so the norm can be read back: about 0.075 here, so the doses run from about a quarter of the activation's length to four times it. A dose far below the norm does nothing; one far above it replaces the activation. To set the doses from the norm in the document, see below. effectis the target,driftthe cost.klagainst the unsteered logits measures how far the whole distribution moved. The useful band is where the target metric moves while the drift stays small; past it the model is being broken, not steered. On a real model, generate at each dose and read the text as well.- Check the difference before using it. Save
differenceand compare its norm with the site's typical norm (typical, below) before trusting a dose curve. A difference orders of magnitude below that norm is rounding, not a direction. - A difference over the same prompts is noise. Two prompt sets that are the same prompts in another order (the base and counterfactual columns of
data/weekdays/train.jsonare) give a difference of rounding noise, orders of magnitude belowtypical.normalizeturns it into a unit vector pointing nowhere in particular. normalizeleaves a zero vector as it is. A difference of exactly zero (two harvests that agree) steers nothing at any dose, and the sweep looks flat.- A dose by layer grid is two labelled sweeps, which cross (sweeps). Sweep the site's
layersunder its own label beside thedose. Every read oftargetthen follows the layer, so each layer steers along its own difference. The document above gives eight points, keyed{"layer":0,"dose":0.0}on; both dose-0 points equal the unsteered run:
"target": {"component": "layer_output", "layers": {"sweep": [0, 1], "label": "layer"}}A dose written as a fraction of the activation's own length needs no number copied from a run. Take a norm of a read at the site, its median, and scale the unit direction by it; the write adds that vector (ops). Steps added to the document above:
"lengths": {"kind": "op", "op": "norm", "of": "base.resid"},
"typical": {"kind": "op", "op": "median", "of": "lengths"},
"unit_dose": {"kind": "op", "op": "scale", "of": "direction", "factor": "typical"},
"steered": {"kind": "forward", "input": "days.input", "interventions": {
"steer": {"site": "target", "pos": -1, "ops": [
{"op": "add", "with": "unit_dose", "scale": {"sweep": [0.0, 0.25, 1.0, 4.0], "label": "dose"}}]},
"logits": {"site": "head", "pos": -1}}}lengths is each row's norm at the last token, and typical their median, one number. unit_dose is the unit direction at that length (its norm equals typical), so dose 1.0 adds a vector as long as the median activation. Read the site the write is made at, at the positions it writes.
"The direction does this and a random one does not" is a fair comparison only when the random direction is as long. Draw one from noise and give it the difference's norm, as steps after direction:
"norm": {"kind": "op", "op": "norm", "of": "difference"},
"zero": {"kind": "op", "op": "scale", "of": "difference", "factor": 0.0},
"draw": {"kind": "op", "op": "noise", "of": "zero", "scale": 1.0, "seed": 7},
"random_unit": {"kind": "op", "op": "normalize", "of": "draw"},
"random": {"kind": "op", "op": "scale", "of": "random_unit", "factor": "norm"}norm is the difference's length, its L2 norm over hidden_size (ops). noise on a value with no rows is one Gaussian draw from its seed, normalize makes it unit length, and scale gives it the difference's norm: on the corpus, random's norm equals difference's at seeds 7, 8 and 9. Add random at the same doses as difference in a forward of its own, and sweep seed for the spread a real direction has to clear. Steering with the unit direction, as the document above does, the matched control is random_unit.
Steering with unit_dose, a dose set from the activations, the matched control is random_unit scaled the same way, a random direction as long as typical: {"kind": "op", "op": "scale", "of": "random_unit", "factor": "typical"}.
documents/clamp_write.json bounds every element of layer 1's residual stream at every token, BOS included, and reads the result back after the write:
"clamp": {"site": "target", "pos": {"index": ":"}, "ops": [{"op": "clamp", "min": -0.01, "max": 0.01}]},
"resid": {"site": "target", "pos": -1}clamped.resid lies in [-0.01, 0.01] where clean.resid reaches -0.04. The bounds are set against this checkpoint's residual stream, whose entries there are of order 0.01; read the site first and set bounds from what it holds. Clamping a featurizer's coordinates instead (one feature held at a value) goes between an encode and its decode (featurizers).
In a generate step a write's pos names forward calls by its frame (generate). pos: -1 is the last prompt token in the prefill only, and the generated tokens see that write only through the cache. To steer every token, write at every call:
"steered": {"kind": "generate", "input": "pairs.input", "max_new_tokens": 6, "min_new_tokens": 6, "do_sample": false,
"interventions": {"steer": {"site": "target", "pos": {"frame": ":"},
"ops": [{"op": "add", "with": "direction", "scale": 8.0}]}}}{"frame": ":"} is every prompt token in the prefill and each generated token at its decode call; {"frame": "1:"} is the generated tokens alone. Score the text with detokenize and contains (generation experiments). Keep do_sample: false so the steered and unsteered runs differ by the write alone.
To steer the prompt's last token and every generated token, and no other prompt token, write two entries: one at pos: -1 and one at {"frame": "1:"} (generate). In a generate step pos: -1 is the prefill's last token, the same as {"frame": 0, "index": -1}. documents/generated_steer.json does this: at "scale": 4.0 every row's generation changes from its third token on, and at 0 every value is the unsteered one. Its two sweeps carry no label, so both take the default label scale, one axis: the two entries move together, two doses in two generations (sweeps):
"steer_prefill": {"site": "resid", "pos": -1,
"ops": [{"op": "add", "with": "direction", "scale": {"sweep": [0.0, 4.0]}}]},
"steer_decode": {"site": "resid", "pos": {"frame": "1:"},
"ops": [{"op": "add", "with": "direction", "scale": {"sweep": [0.0, 4.0]}}]}Give both sweeps one label of their own when another scale in the document must not move with them.
To count the generated tokens a write changed, generate once more without the write and compare the ids, which are [rows, new]:
"clean": {"kind": "generate", "input": "pairs.input", "max_new_tokens": 6, "do_sample": false},
"changed": {"kind": "op", "op": "ne", "of": "steered", "with": "clean"},
"changed_per_row": {"kind": "op", "op": "sum", "of": "changed", "over": ["new"]},
"changed_frac": {"kind": "op", "op": "mean", "of": "changed", "over": ["rows", "new"]}Name new in over: a reduction's default removes rows and pos, so a bare mean of changed is [new] (ops). On documents/generated_steer.json, changed_per_row is 0 on every row at the dose-0 point, the null, and 4 at "scale": 4.0.