[SarahSoutoul]

What does this solve? What changed?

DOCS-12297 adds the receiver half of stable CLI release evaluation. This is paired with the CLI sender PR, which dispatches after successful stable npm publish without waiting for npm's latest tag.

This is the CLI-release follow-up to DOCS-12112 and clerk-evals#73, which added evals for clerk/skills updates but did not add the stable CLI-release trigger.

• Accept clerk_cli_released alongside skills dispatches and manual runs.
• For CLI release events, wait up to 10 minutes for the requested version to become clerk@latest, retrying npm lookup failures as well as an older latest result. Other events fail immediately on an npm lookup failure. Never evaluate against a different version.
• Resolve one full Skills SHA, run both baseline and Skills columns for a new release, and preserve duplicate skipping and manual force reruns.
• Queue all add-auth runs together so overlapping skills and CLI-release events check for completed results in order, avoiding duplicate evaluations of the same version pair.
• Show source versions and the plan decision in the run summary.

Deadline

No fixed deadline. Merge this receiver before clerk/cli#539. It can safely wait for release messages; a dispatch sent before the receiver is live would not be replayed.

Other resources

• DOCS-12297
• clerk/cli#539 — paired release sender

Pre-merge checklist

• Focused add-auth plan tests pass locally (9 tests).
• Confirm the paired CLI sender dispatches only stable versions after npm publish succeeds; this receiver checks clerk@latest before evaluating.
• Confirm clerk/cli has the dedicated CLERK_EVALS_DISPATCH_TOKEN Actions secret, scoped to send repository dispatches to only clerk/clerk-evals. This receiver does not need that secret.
• Review CI and workflow permissions.

Post-merge checklist

• Observe a stable release-to-result run with both columns and the exact CLI version and Skills SHA.
• Repeat the event and confirm no duplicate results; confirm manual force still reruns.
• Check that automation failures are visible without treating a completed score regression as an infrastructure failure.

[manovotny]

If a newer release lands before this run leaves the queue, this fails prepare and alerts #team-docs. Say 1.1.0 and 1.1.1 ship close together, and 1.1.0's run waits behind a 90-minute eval. When it starts, clerk@latest is 1.1.1, so the loop waits the full 10 minutes for a version that won't come back, then exits 1. The pre/post npm view clerk version checks in evaluate hit the same thing if a release lands mid-eval.

The 1.1.1 run covers it, so the alert is noise, not a real automation failure.

Could this skip cleanly when latest is already ahead of CLI_INPUT (sort -V would do it)? At minimum, breaking out of the wait loop early saves the 10 minutes.

[manovotny]

This flag only changes the outcome when this CLI version was already evaluated at another Skills SHA. For a version that hasn't run yet, the lastCli !== cli check below already runs both columns.

When it has run (say a queued skills run picked up the new latest first), this forces a second baseline and Skills pass that the existing logic would skip or cover with a reused baseline. The new test shows it: evaluated(old) uses the same 1.0.0, so it tests an already-evaluated CLI, not a new one.

I know #73's comment planned for CLI releases to always run both. Is that still the intent, or could the planner drop cliRelease and let the CLI-change check handle it?

[manovotny]

The README still says CLI releases don't trigger the workflow (README.md lines 197–202), which stops being true once this merges. Maybe update that paragraph alongside this one? Something like:

`.github/workflows/add-auth-automation.yml` accepts `clerk_skills_updated` dispatches with
`client_payload.sha`, `clerk_cli_released` dispatches with `client_payload.version`, and manual
runs. Manual runs can select a full Skills SHA and the current latest CLI version. The canonical
prompt runs `npx -y clerk@latest init`, so a requested CLI version that isn't `clerk@latest` fails
instead of mislabeling the result. A CLI release waits up to 10 minutes for its version to become
`clerk@latest` first.

[manovotny]

The Skills SHA gets a format check above, but CLI_INPUT goes straight into echo and the comparisons. The sender in clerk/cli#539 already enforces X.Y.Z, so this mostly catches bad manual input early. Worth mirroring here?

Suggested change

	if [[ "$CLI_RELEASE" == 'true' && -z "$CLI_INPUT" ]]; then
	echo 'CLI release dispatch must include a version' >&2
	exit 1
	fi
	if [[ "$CLI_RELEASE" == 'true' && -z "$CLI_INPUT" ]]; then
	echo 'CLI release dispatch must include a version' >&2
	exit 1
	fi
	if [[ -n "$CLI_INPUT" && ! "$CLI_INPUT" =~ ^[0-9]+\.[0-9]+\.[0-9]+$ ]]; then
	echo 'Expected a stable Clerk CLI version like 1.2.3' >&2
	exit 1
	fi