[fsaresh]

The release-autopilot-onboarding eval job takes 17-19 minutes. The other suites take 1-5 minutes. This PR removes onboarding cases that launchdarkly-release-plans already covers, and adds sharding so that a slow suite can run as several matrix jobs.

The workflow change is not in this PR yet. The termag broker token has no workflow scope, so GitHub refused the push of .github/workflows/eval-skills.yml. The patch is below. Until somebody pushes it, CI runs the onboarding suite as one job with 32 cases.

	Before	After (with the workflow patch)
Onboarding cases	37 (739 max turns)	32 (611 max turns)
Release-plans cases	17	18 (one case moved in)
Onboarding jobs	1	3 shards (198, 207 and 206 turns)
Onboarding job time	17-19 min	about 5-6 min for each shard (estimate)
Matrix max-parallel	3	5
PRs and main	every case	every case (no change)

Cases removed

Removed onboarding case	Covered by (launchdarkly-release-plans)
Starter plan: progressive, created and set as production default	"Starter plan: production progressive after the unit check, confirmed, then the default". The onboarding case "Flag release plan skipped" also still checks starterPlanProgressive and the plan-before-repo-write order.
Existing plan: kept and set as default, no new plan	"Complete plans, no default: offers each plan, sets the chosen one, creates nothing"
Preview with auto-releasing: lists the flag release plan, asks to run nothing, no writes	"Preview: describes the plan it would create, asks to run nothing". The onboarding Preview cases stay.
Default flag release plan does not match: replace command in place, no create	"Change the default: full replace with the version, isDefault set, no create"

"Another plan as default: says it replaces the current default" moved to launchdarkly-release-plans, because no other case checks that the confirmation names the default that it replaces. Its assertions (existingPlanSetAsDefault, saysReplacesDefault) moved with it and keep the same checks. I removed the onboarding assertions that no case uses now (reportListsDefaultPlan, previewListsFlagReleasePlan, flagReleasePlanUpdatedInPlace, and the two that moved), their unit tests, and two comments that described cases #261 already moved. The other cases are unchanged. "Flag release plan: user-run create after the release plan" stays as the hand-off case.

How sharding works

• shards: N on a suite in evals/scripts/_manifest.js makes it a sharded suite. release-autopilot-onboarding has shards: 3.
• node evals/scripts/shards.js matrix '<slugs>' changes each sharded slug into <suite>#1 ... <suite>#N. With the workflow patch, "Compute changed suites" uses it, so a change to the suite starts every shard.
• aggregate.js --run --only=<suite>#i runs shard i. Tests go heaviest first (vars.max_turns) to the shard with the fewest turns, so the split is deterministic and balanced. promptfoo gets the shard's tests as an exact --filter-pattern of their descriptions. --shard=i/n does the same for any suite.
• Each shard writes results.shard-<i>-of-<n>.json. "Aggregate scores" stages every shard artifact. aggregate.js then merges the shards into one suite score, and it fails if a shard is missing.
• launchdarkly-release-plans takes about 4.3 minutes, so it has no shards.

evals/README.md has a "Sharded suites" section. evals/tests/shards.test.js tests the split, the matrix, the filter pattern, and the merge.

Workflow patch

To apply it from a checkout with workflow scope: save the block to a file, run git apply <file> on this branch, then commit and push.

.github/workflows/eval-skills.yml

diff --git a/.github/workflows/eval-skills.yml b/.github/workflows/eval-skills.yml
index d73d165..3e4b9b6 100644
--- a/.github/workflows/eval-skills.yml
+++ b/.github/workflows/eval-skills.yml
@@ -11,6 +11,10 @@ name: Skill Evals
 # Cost shape: only suites whose source has actually changed (per
 # evals/scripts/diff-changed-skills.js) get re-evaluated, so a typical PR
 # touching one skill costs roughly one suite's worth of API tokens.
+#
+# Shards: a suite with `shards: N` in evals/scripts/_manifest.js runs as N
+# matrix jobs ("<suite>#1" ... "<suite>#N"), each with its share of the
+# tests. "Aggregate scores" merges the shards into one suite score.
 
 on:
   # No `paths:` filter on pull_request — branch protection requires
@@ -71,6 +75,7 @@ jobs:
     runs-on: ubuntu-latest
     outputs:
       slugs: ${{ steps.compute.outputs.slugs }}
+      matrix: ${{ steps.compute.outputs.matrix }}
       has_changes: ${{ steps.compute.outputs.has_changes }}
     steps:
       - name: Checkout
@@ -90,12 +95,16 @@ jobs:
             slugs="$(node evals/scripts/diff-changed-skills.js --json --verbose)"
           fi
           echo "slugs=${slugs}" >> "$GITHUB_OUTPUT"
+          # A changed suite runs every one of its shards.
+          matrix="$(node evals/scripts/shards.js matrix "${slugs}")"
+          echo "matrix=${matrix}" >> "$GITHUB_OUTPUT"
           if [[ "${slugs}" == "[]" ]]; then
             echo "has_changes=false" >> "$GITHUB_OUTPUT"
           else
             echo "has_changes=true" >> "$GITHUB_OUTPUT"
           fi
           echo "Changed suites: ${slugs}"
+          echo "Matrix jobs: ${matrix}"
 
   evaluate:
     name: "Evaluate ${{ matrix.suite }}"
@@ -104,9 +113,12 @@ jobs:
     runs-on: ubuntu-latest
     strategy:
       fail-fast: false
-      max-parallel: 3
+      # 5, not 3: the onboarding suite's shards add jobs, and with 3 the
+      # last of them waited behind the rest of the matrix.
+      max-parallel: 5
       matrix:
-        suite: ${{ fromJson(needs.diff.outputs.slugs) }}
+        # A suite slug, or "<suite>#<i>" for shard i of a sharded suite.
+        suite: ${{ fromJson(needs.diff.outputs.matrix) }}
     steps:
       - name: Checkout
         uses: actions/checkout@v4
@@ -124,21 +136,37 @@ jobs:
         run: npm ci --legacy-peer-deps
         working-directory: evals
 
+      - name: Name the results artifact
+        id: names
+        env:
+          ENTRY: ${{ matrix.suite }}
+        run: |
+          suite="${ENTRY%%#*}"
+          if [[ "$ENTRY" == *"#"* ]]; then
+            artifact="results-${suite}--shard-${ENTRY##*#}"
+          else
+            artifact="results-${suite}"
+          fi
+          echo "suite=${suite}" >> "$GITHUB_OUTPUT"
+          echo "artifact=${artifact}" >> "$GITHUB_OUTPUT"
+
       - name: Run eval suite
         env:
           ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
           OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
           AGENT_MODEL: ${{ vars.AGENT_MODEL || 'claude-sonnet-4-6' }}
           RUBRIC_MODEL: ${{ vars.RUBRIC_MODEL || 'anthropic:messages:claude-haiku-4-5-20251001' }}
-        run: node scripts/aggregate.js --run --only=${{ matrix.suite }}
+          ENTRY: ${{ matrix.suite }}
+        run: node scripts/aggregate.js --run "--only=${ENTRY}"
         working-directory: evals
 
       - name: Upload suite results
         if: always()
         uses: actions/upload-artifact@v4
         with:
-          name: results-${{ matrix.suite }}
-          path: evals/${{ matrix.suite }}/results.json
+          name: ${{ steps.names.outputs.artifact }}
+          # results.json, or results.shard-<i>-of-<n>.json for a shard.
+          path: evals/${{ steps.names.outputs.suite }}/results*.json
           retention-days: 14
 
   aggregate:
@@ -171,17 +199,19 @@ jobs:
           pattern: results-*
           merge-multiple: false
 
-      - name: Stage suite results into evals/<suite>/results.json
+      - name: Stage suite results into evals/<suite>/
         run: |
           set -e
           shopt -s nullglob
+          # results-<suite> holds results.json; results-<suite>--shard-<i>
+          # holds results.shard-<i>-of-<n>.json. aggregate.js merges shards.
           for d in artifact-results/results-*; do
-            name=$(basename "$d" | sed 's/^results-//')
+            name=$(basename "$d" | sed -e 's/^results-//' -e 's/--shard-[0-9]*$//')
             mkdir -p "evals/$name"
-            if [[ -f "$d/results.json" ]]; then
-              cp "$d/results.json" "evals/$name/results.json"
-              echo "Staged evals/$name/results.json"
-            fi
+            for f in "$d"/results*.json; do
+              cp "$f" "evals/$name/"
+              echo "Staged evals/$name/$(basename "$f")"
+            done
           done
 
       - name: Save previous eval-scores.json for diff

Checks

validate_skills.py, python3 -m unittest, generate_catalog.py --check and npm test (222 tests) pass. actionlint passes on the patched workflow. I did not run the live evals.

🤖 Generated with Claude Code

_{Opened from termag}

---

Note

Overview
Adds eval suite sharding so slow suites can run as parallel CI jobs: new scripts/shards.js splits tests deterministically by max_turns, aggregate.js runs --only=<suite>#i / --shard=i/n, writes per-shard result files, and merges them when aggregating; release-autopilot-onboarding is set to 3 shards in _manifest.js, with README docs and shards.test.js.

Dedupes release-autopilot-onboarding by removing several release-plan scenarios already covered by launchdarkly-release-plans, and moves the “another plan as default” case (plus existingPlanSetAsDefault / saysReplacesDefault assertions and unit tests) into the release-plans suite. Unused onboarding assertions and tests are dropped accordingly.

CI workflow wiring for the shard matrix is not in this diff (described separately in the PR body).

^{Reviewed by Cursor Bugbot for commit 0afa496. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 9196f9d. Configure here.}

[github-actions]

Skill eval results

Skill	Before	After	Δ
agentcontrol/configs-create	100/100 (4/4)	75/100 (3/4)	-25
agentcontrol/configs-update	80/100 (4/5)	80/100 (4/5)	no change
agentcontrol/configs-variations	80/100 (4/5)	80/100 (4/5)	no change
agentcontrol/tools	75/100 (3/4)	75/100 (3/4)	no change
aws-devops-agent/experiment-orchestration	-	100/100 (9/9)	new
experiments/launchdarkly-experiment-guide	-	100/100 (5/5)	new
experiments/launchdarkly-experiment-hypothesis-builder	-	100/100 (13/13)	new
experiments/launchdarkly-experiment-setup	-	100/100 (8/8)	new
feature-flags/flag-and-release-change	-	100/100 (5/5)	new
feature-flags/flag-release	-	100/100 (7/7)	new
feature-flags/launchdarkly-flag-command	-	100/100 (3/3)	new
feature-flags/launchdarkly-flag-create	100/100 (3/3)	100/100 (4/4)	no change
feature-flags/launchdarkly-flag-drift	-	100/100 (4/4)	new
feature-flags/should-flag-change	-	100/100 (17/17)	new
onboarding	-	86/100 (6/7)	new
release-autopilot/launchdarkly-release-autopilot-onboarding	-	88/100 (29/33)	new
release-autopilot/launchdarkly-release-plans	-	94/100 (17/18)	new

Only suites whose source actually changed since their last recorded score were re-run. Soft-failing while we stabilise the baseline.