| title | Data and Inputs | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| one_liner | How do I load rows and pick what a step runs over? File, inline and hub datasets, narrowing, renamed and derived columns, chats, teacher forcing and answer columns. | |||||||||
| tags |
|
|||||||||
| related |
|
|||||||||
| sources |
|
The data block names a document's datasets. Each dataset is a list of rows, each row a JSON object, read from a file, written inline, or loaded from the Hugging Face hub. A step's input names the column it runs over, one row per example. Other fields name columns as text to search for or as answers to score. Every field is in FORMAT.md; this page is how to use them.
{
"model": {"key": "hf-internal-testing/tiny-random-LlamaForCausalLM",
"revision": "9fb191250dd56d0ba7ec9785a025ed29c03d5998"},
"data": {
"qa": {"rows": [
{"example_id": "fri", "input": "If today is Friday, is tomorrow Saturday? Answer:", "answer": [" Yes", " True"]},
{"example_id": "mon", "input": "If today is Monday, is tomorrow Sunday? Answer:", "answer": [" No", " False"]}
]},
"days": {"path": "weekdays_full/data", "split": "train", "shuffle": true, "take": 4,
"columns": {"prompt": "input"}}
},
"steps": {
"ask": {"kind": "forward", "input": "qa.input",
"interventions": {"logits": {"site": {"component": "lm_head_output"}, "pos": -1}}},
"p_answer": {"kind": "op", "op": "token_prob", "of": "ask.logits", "token": "qa.answer"},
"forced": {"kind": "forward", "input": ["days.prompt", "days.base_answer"],
"interventions": {"logits": {"site": {"component": "lm_head_output"},
"pos": {"scope": {"column": "days.base_answer", "occurrence": -1,
"before": true}}}}},
"nll": {"kind": "op", "op": "cross_entropy", "of": "forced.logits", "target": "days.base_answer"}
},
"saves": ["p_answer", "nll"]
}qais two inline rows whoseansweris a set:p_answeris the summed probability of either member.dayskeeps 4 rows of thetrainsplit ofdata/weekdays_full/data.json, in a fixed shuffled order, with itsinputcolumn renamedprompt.forcedruns each prompt with its answer appended (teacher forcing) and reads the positions that predict the answer's tokens."occurrence": -1picks the appended answer. Without it the scope takes the answer's first occurrence, which is in the prompt wherever the prompt says the answer too (positions.md).
A dataset has exactly one source; giving two is refused.
-
pathreadsdata/<path>.jsonunder the corpus root, the nearest directory holdingdata/, from the document's own directory up (running.md). The file is a JSON list of row objects. -
rowsare written inline. They travel inside the plan and the manifest, so a run's table reads them back from anywhere. -
hfis a hub dataset,<owner>/<name>, with a requiredrevision(a commit, asmodelpins one), its subsetconfigwhere it has several, andsplit:"glue": {"hf": "nyu-mll/glue", "revision": "bcdcba79d07bc864c1c254ccfcedcce55bcc9a8c", "config": "wnli", "split": "validation", "take": 3}
Its rows never go into the plan or the manifest. Compile loads them to check the document and records their digest, the run loads them again where the model runs, and
nnspec tableloads them again to label its rows. Offline (HF_HUB_OFFLINE=1) they come from thedatasetslibrary's cache, which one online load of that commit fills (tests/test_hub.pygives the command). A hub dataset with several splits and nosplitis refused, and the message lists them.
The five fields apply in this order: split, shuffle, skip, take, columns.
| field | does | refused |
|---|---|---|
split |
a file: keeps the rows whose split column equals it; the hub: which split |
on inline rows; a split no row has (data.pairs: the dataset has no rows in split 'train') |
shuffle: true |
one fixed random order, the same on every machine (below) | — |
skip: N |
drops the first N rows; with take, a shard of the order ("skip": 20000, "take": 10000 is rows 20,000 to 29,999) |
N that leaves no row; a negative N |
take: N |
keeps the first N of the rows left; after shuffle, a random sample of N |
N beyond the rows there are (after skip) |
columns |
{"<new name>": "<column as the source has it>"} renames columns; {"<new name>": {"from": "<column>", "after"?, "before"?, "strip"?}} derives one (below) |
a source the rows lack; a new name that is a column the rows keep; a derived column's text absent on a row |
- A renamed column answers to its new name only. In the example,
days.inputis refused anddays.promptis the column. Renaming lets one file serve under another dataset's column names, which a fit'sevalsubstitution needs (fit.md). - A swap,
{"a": "b", "b": "a"}, is allowed, since neither old name is kept. - A rename into a name the steps read changes what they read, unrefused. On
weekdays/train,{"input": "entity", "prompt": "input"}runs everypairs.inputstep on the entity alone (<s> ▁Fridayon row 1). Checkrun.tokensafter renaming. takewithoutshufflekeeps the first N rows. On a file sorted by label or difficulty, that is not a sample.
The manifest records each dataset's sha256 and n_rows after these fields apply, and a row's index in nnspec table counts these rows.
shuffle's order ranks each row's index i by the 8-byte blake2b hash of repr((0, i)) (nnspec/format/data.py shuffled), so it depends on the row count alone. Outside nnspec, with n the row count after split:
order = sorted(range(n), key=lambda i: hashlib.blake2b(repr((0, i)).encode(), digest_size=8).digest())skip: S then drops the first S of that order and take: N keeps the next N. A harvest too large for one run is made in shards this way (cached-harvest.md).
A columns entry that is an object derives a column from another column's text, where a renamed one moves it. On weekdays/train the day each prompt names is
"pairs": {"path": "weekdays/train",
"columns": {"day": {"from": "input", "after": {"text": "today is "}, "before": {"text": ","}}}}which gives Thursday, Friday, Saturday and Sunday.
afterkeeps the text after the first occurrence of its text.beforekeeps the text before the first occurrence of its text, afterafter's when both are given.stripdrops the surrounding whitespace of what is left.fromnames the column as the source has it, which stays under its own name.- Each text is
{"text": ...}, one string compared exactly, with nostrip,loweror set. A bare string is refused:data.pairs.columns.day.after: 'today is ' is a string; …. - Compile cuts every row and refuses, naming the row, one whose
fromis not text or lacks one of its texts:data.pairs.columns.day.after: row 0: 'yesterday was ' is not in its 'input', …. - The derived column is part of the rows, so the dataset's
sha256covers it, and a hub dataset's rows are cut again where the run loads them. It is text cut from text before any forward, never a computation on values.
A forward or generate step's input takes three forms:
| form | example | each row's prompt |
|---|---|---|
<dataset>.<column> |
"pairs.input" |
the column's text, or its chat |
<dataset>.<column>[i] |
"pairs.counterfactual_inputs[0]" |
entry i of a list column; every row needs one |
a list of columns and {"text": ...} texts |
["days.prompt", "days.base_answer"] |
the parts joined with nothing between |
Text is tokenized as the tokenizer does by default, with BOS where it adds one; how positions count those tokens is positions.md. The run records every input's tokens under its name in run.tokens (output.md). A list input's name joins its parts with +, a text as JSON: days.prompt + " and then" + days.base_answer.
documents/forced_answer.json writes "input": ["days.input", "days.base_answer"] to append each row's answer to its prompt. Nothing is inserted between parts, so the answer's own leading space separates the words. A text, {"text": ...}, one string alone, adds fixed text: ["days.input", {"text": " and then"}, "days.base_answer"] gives If today is Thursday, tomorrow is and then Friday. All columns of a list come from one dataset, a list needs at least one column, and a chat column in a list is refused, since a list joins text. Reading the answer's positions is positions.md; scoring each token is ops.md.
A column without the answer's leading space is appended bare, since the parts join with nothing between. A scope searches the row's whole text, prompt and appended parts as one string, for its text on token and word boundaries. Occurrences are ordered by where they end, so "occurrence": -1 is the last one found (positions.md). Whether it is refused depends on what it finds:
- After a word, the bare answer joins it.
["p.input", "p.said"]ondata/weekdays/said.jsongivesissubstr, and the scope onp.saidis refused:column 'p.said' ('substr') is not in its input; it occurs only inside 'issubstr', …. - After punctuation it is a word of its own, in another spelling, and nothing is refused. On GPT-2 (
openai-community/gpt2at607a30d783dfa663caf39e06633721c8d4cfcd7e, CPU, float32),Q: What day comes after Thursday? A:withFridayappended tokenizes the answer asFriday, notFriday. The scope on the bare column finds it, andcross_entropyscoresFridayat 8.60 nats, where the spaced answer scores 3.20 at the same position. - Where the prompt also names the answer, a scope on the spaced text lands on the prompt's copy. A spaced text never finds a word without the space, so the last occurrence it finds is the prompt's. On GPT-2,
Thursday comes before Friday. If today is Thursday, tomorrow iswithFridayappended, read at{"scope": {"column": "p.spaced", "occurrence": -1, "before": true}}, lands at position 2 (before). It scores the prompt'sFriday, 5.92 nats, where the spaced append is read at 11 and scores 0.94. Nothing refuses it: the scope found an occurrence, and the input holds the target's tokens where it landed. - Check the positions in
nnspec show, and write the space into the rows. An appended answer's positions lie past the prompt's last token. With the space in the rows the appended answer is the spelling scored, and it ends last, so"occurrence": -1finds it. - A text space fixes the input, not the target.
"input": ["p.input", {"text": " "}, "p.said"]spaces the appended word. On the corpus Llama, wheresubstrandsubstrare the same tokens, that gives the same numbers as rows with the space written in. On GPT-2 the bare target is other tokens than the input holds, and a metric of it is refused:'substr' is 2 tokens; the read has 1 position on that row; …, or, where the counts agree,position 6 predicts ' Friday' in the input, which is not token 0 of 'Friday' as it tokenizes alone …. There the space goes into the rows, since a derived column only cuts (below).
A column whose value on each row is a list of {"role", "content"} messages is a chat. nnspec renders it through the checkpoint's own chat template and tokenizes the result without adding special tokens, since the template writes its own. documents/chat_turn.json reads chats.chat from data/weekdays/chat.json; on the corpus Llama the first row renders as:
<s>[INST]▁Name▁a▁weekday.▁[/INST]▁Thursday.▁</s><s>[INST]▁If▁today▁is▁Thursday,▁tomorrow▁is▁[/INST]
The dataset's template is passed to the checkpoint's apply_chat_template as arguments, wherever a chat of that dataset renders:
"chats": {"path": "weekdays/chat", "template": {"add_generation_prompt": true}}- Without a template the prompt ends with the template's close of the last message, and
pos: -1is that close's last token. - With
add_generation_promptit ends with the assistant's header, so ageneratecontinues as the assistant. What an argument does is the template's: on a Llama-2-style template, as above,add_generation_promptchanges nothing. Check the rendering inrun.tokens. - A
templateon a dataset that no step reads as a chat is refused, and so is atokenizekey in it. Positions inside a turn are positions.md. - A chat comes only from a column that already holds
{"role", "content"}messages; nothing in a document turns a text column into one. To run a text dataset as user turns, write the messages outside nnspec as inlinerows(or a file), and record the hub source and its revision indescription.
A template can read the clock. Llama-3.2's template adds a system header when the chat has none and writes Today Date: <today> in it (strftime_now). The same document would then render other tokens on another day, which a dataset's digest cannot say: it covers the rows, not how they render. A chat of such a template is refused before any forward unless the dataset's template pins the date, and the refusal gives the pin (tests/test_plan.py test_a_chat_template_that_reads_the_clock_renders_only_with_its_date_pinned):
nnspec: blanked: chats.chat: the checkpoint's chat template calls strftime_now, which writes today's date into the prompt, so these rows render to other tokens on another day and nothing in the run record says so; pin it in data.chats.template, {"date_string": "26 Jul 2024"}
"chats": {"path": "weekdays/chat", "template": {"add_generation_prompt": true, "date_string": "26 Jul 2024"}}With the pin, the prompt carries that date; check it in run.tokens.
Where a field takes a column, as an answer to score or as text to find (a metric's answer, contains's text, a scope's column), write "<dataset>.<column>", or the object with flags:
"token": {"column": "qa.answer", "strip": true, "lower": true}- Flags say how text compares; they never rewrite the data.
lowercompares lowercased;stripmakes surrounding whitespace optional. Scored against logits, the answer's token then stands for every vocabulary token that compares the same: indocuments/answer_spellings.json,Yesunder both flags sums the probability of▁Yes,▁yes,Yes,yes,YESand▁YES. What that does to each metric is ops.md. - A list of strings is a set, any member of which is the answer:
"answer": [" Yes", " True"]above.nnspec tableprints it as JSON. - A fixed answer is a literal, not a column. Where an op takes an answer or a text,
{"text": "</think>"}stands in place of the column. It is the same text on every row, tokenized as written and compared under the same flags ({"text": " yes", "strip": true, "lower": true}; a list is a set). It gives the same numbers as a column holding that text on every row (tests/test_answers.pytest_a_literal_scores_as_the_constant_column_of_its_text_does), so a constant column built only to be scored is not needed. - A bare string is always a column reference.
"token": "</think>"is refused with'</think>' is not <dataset>.<column> over a dataset; …; text written as it is, the same on every row, is {'text': '</think>'}. - A path reaches inside a column's entries:
pairs.counterfactual_inputs_variables[0].entityis theentityof each row's first variables object. - An answer is tokenized exactly as written, without special tokens. Write the space the prompt implies. On a byte-level BPE tokenizer such as GPT-2's,
" Friday"and"Friday"are different tokens; on the corpus Llama both are▁Friday. - Where both spellings are one token, the wrong one is scored without a refusal. On
documents/gpt2_reach.json's GPT-2,isandisare each one token, and atoken_probof either runs. Where the bare word is more tokens than the spaced one, it is refused (answer-does-not-line-up.md); the symptom is in symptom-index.md. - A hub column without the space is fixed outside the document. A derived column only cuts a column's text, and
striploosens the comparison, not the answer's tokens. Write the rows inline, or to a file, from a script that adds the space, and record the hub source and commit indescription. - Text searched in a generation is decoded text, in which the first generated token has lost its space: generate.md.
A row's example_id column, when the dataset has one, leads every row of nnspec table. Give rows one when you shuffle or take, so a table row still says which example it is.
The datasets under the repository's data/, by the path that reads them. Every file has a split column:
path |
rows | columns | split |
|---|---|---|---|
weekdays/train |
4 | input, counterfactual_inputs, counterfactual_inputs_variables, entity, answer, base_answer, cf_answer, label |
all |
weekdays/data |
4 | those of weekdays/train |
train 2, test 2 |
weekdays/said |
4 | example_id, input, said, base_answer, cf_answer |
all |
weekdays/chat |
4 | example_id, chat, entity, base_answer, cf_answer |
all |
weekdays_full/data |
42 | example_id, input, counterfactual_inputs, entity, cf_entity, base_answer, cf_answer |
train 30, test 12 |
weekdays_aligned/data |
6 | those of weekdays_full/data |
all |
weekdays_even/train |
6 | input, counterfactual_inputs, counterfactual_inputs_variables, entity, base_answer, cf_answer |
all |
counting/train |
4 | input, counterfactual_inputs, base_answer, cf_answer, label |
all |
pile/data/sample |
3 | example_id, input, resample_input, ml_token |
sample |
natural_domains_arithmetic/data/weekdays |
49 | input, counterfactual_inputs, counterfactual_inputs_variables, entity, number, result, label, base_answer, cf_answer, label_forms, base_answer_forms, cf_answer_forms, string_mode, scoring_digest |
train 30, test 19 |
weekdays/train is not a training split: every row's split is all, and it pairs each day with another counterfactual day than weekdays/data does. A fit's training and held-out rows are weekdays/data's two splits (fit.md).
- A file dataset is read again when you table a run.
nnspec tableloads it fromdata/under the working directory and refuses rows that changed since the run (output.md). Inline rows do not have this problem.