BC-Bench is category-based. A category is a distinct evaluation scenario: bug-fix asks an agent to patch buggy code, test-generation asks it to write reproduction tests, code-review asks it to flag issues in a diff, nl2al asks it to turn a natural-language spec into AL code, extensibility-request-advisor asks it to classify and assess a fully specified extensibility need and draft a GitHub issue, extensibility-request-implement asks it to implement an approved extensibility request (e.g. add an integration event) in an existing repo, and extensibility-request-triage asks it to triage an extensibility request (emit managed labels, an advisory comment, and an open/closed decision).
Categories also differ in how they're scored and run. bug-fix and test-generation are execution-based: they build and run AL code, so they need a BC container. code-review, nl2al, extensibility-request-advisor, extensibility-request-implement, and extensibility-request-triage all leverage LLM-as-a-judge: code-review scores precision/recall/F1 of flagged issues against expected findings (an LLM judge only matches comments), and the other judge-based categories have an LLM grade the agent output against an LMChecklist. Advisor entries use an offline, single-shot proxy for the production skill's interactive flow: the prompt supplies the complete scenario, GitHub submission is disabled, and the agent persists classification, feasibility analysis, alternatives, and the final issue draft to advisor_result.json. The EvaluationCategory properties (requires_container, runner, evaluators, core_score) capture these differences for the workflows.
Categories may share a dataset (bug-fix and test-generation do today), but a new category should generally have its own: dataset schema, entry type, result type, pipeline, etc.
This doc is a map; the source files and their comments are the source of truth. To experiment with agent setup on existing categories, see EXPERIMENT.md.
Start with EvaluationCategory in src/bcbench/types.py. It is the category registry. Each enum value maps to the pieces the rest of the CLI and workflows consume:
dataset_path— the dataset file for raw tasks.entry_class— the typed Python model for one dataset row (aka one task).result_class— the recorded outcome for one evaluated task.summary_class/aggregate_class— the aggregate views used by result summaries and leaderboards.pipeline— the category-specific setup, agent run, and evaluation behavior.evaluators/core_score— the bc-eval evaluator list and headline score, emitted to workflows by src/bcbench/commands/category.py.requires_container/requires_repo/runner— whether the category needs a BC container, whether it needs the dataset repository cloned, and which runner evaluates it.- Prompt template — the category-specific prompt in src/bcbench/agent/shared/config.yaml, loaded by src/bcbench/agent/shared/prompt.py.
Keep dataset entry classes and result classes focused on typed data. Put category-specific behavior in the pipeline.
Use the existing implementations as examples: bug-fix and test-generation for execution-based categories, code-review and nl2al for judge-based ones.
- Add the enum value and mappings in src/bcbench/types.py.
- Add the category dataset JSONL and entry class in src/bcbench/dataset/dataset_entry.py.
- Add a result class under src/bcbench/results/ and map it from
EvaluationCategory.result_class. - Add a pipeline under src/bcbench/evaluate/.
- Add the prompt template to src/bcbench/agent/shared/config.yaml.
- Add the category to workflow choice lists in .github/workflows/, especially evaluation workflows and CI category selection.
- Add docs, leaderboard data, notebooks, and tests for the category where relevant.
At minimum, run the exhaustiveness tests and one local smoke test:
uv run pytest tests/test_type_exhaustiveness.py
uv run bcbench run copilot <some-instance-id> --category <new-category> --repo-path /path/to/repoThen trigger a CI test run before running the full dataset.