Skip to content

fix(release-autopilot): questions carry the command, plan, or diff they ask about - #266

Merged
fsaresh merged 4 commits into
mainfrom
run-command-question-shows-command
Nov 13, 2026
Merged

fsaresh merged 4 commits into
mainfrom
run-command-question-shows-command

Conversation

@fsaresh

@fsaresh fsaresh commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Why

While a question dialog is open, the user may see only the question: its header, text and option labels. The message above it can be hidden. Two kinds of question in these skills asked about content that appeared only in the message above:

  • "Run this command" in the release plans skill, where the user could not see the curl command.
  • "Looks good" in onboarding Step 7, which approves a plan the user could not see.

#267 adds the general rule to CONTRIBUTING.md and the skill template.

What changed

launchdarkly-release-plans (1.0.0 → 1.0.1)

  • "User-run commands": the command also goes in the question. It goes in the preview of "Done, here's the output" when the question tool has previews (AskUserQuestion), or else in the question text. It also stays in the message.
  • New core principle 2: "Use this plan", "Looks right", "Make this change", "Update " and "Run this command" carry the plan, change or command. This holds in every mode, including Record only.
  • The starter plan "Looks right" steps, in SKILL.md and references/environment-release-plans.md, point at that principle.

launchdarkly-release-autopilot-onboarding (1.1.0 → 1.1.1)

  • The "Links" rule is now "The question carries what it asks about":
    • Links go in the question text, as before.
    • A plan, summary, command or diff goes in the preview of the option that accepts it. Without previews, it goes in the question text.
  • Step 1: the summary of the current setup goes in the "Add more / Change one setting / Diagnose a PR" question.
  • Step 6 and Step 8: the "Run this command" steps follow the release plans skill's rule.
  • Step 7: the full plan goes in the preview of "Looks good". The skill says never to ask with only "Ready to apply?".
  • Step 8: the second "Looks good" after a changed flag release plan carries the changed plan.
  • Step 9: the test PR asks header "Open PR" with the diff in the preview of "Open it".

Evals

  • New runQuestionShowsCommand check, added to the six cases that show REST commands (four in release plans, two in onboarding). Every "Done, here's the output" question must show a curl command in its text or in an option's preview.
  • New confirmShowsPlan check, added next to confirmsBeforeWriting in the 15 onboarding cases that use it. Every "Looks good" question must show the plan as a list, and the plan must name every repository that the run later writes.
  • New unit tests for both checks.

Testing

  • node --test tests/*.test.js:
    • 146 tests pass, including the new tests.
    • 7 tests fail the same way on main, because zod and js-yaml are not installed in my checkout.
  • scripts/validate_skills.py, python3 -m unittest discover -s tests and generate_catalog.py --check pass. skills.json is regenerated for the version bumps.
  • Both promptfooconfig.yaml files parse.
  • I did not run the promptfoo eval suites. The new checks add requirements to 20 eval cases, so run both suites before merge.

Note

#264 and #265 change the same eval files, so whichever merges later needs a small rebase.

🤖 Generated with Claude Code

Opened from termag


Note

Overview
Release plans and onboarding skills now require that question dialogs include the content they ask about (plans, curl commands, diffs, links)—via option preview or question text—because users may not see the message above while a question is open.

launchdarkly-release-plans (1.0.0 → 1.0.1) adds a core principle and expands User-run commands so "Run this command" / "Done, here's the output" always shows the curl block in the question; starter "Looks right" steps reference the same rule.

launchdarkly-release-autopilot-onboarding (1.1.0 → 1.1.1) replaces the links-only rule with a broader "question carries what it asks about" rule: Step 7 puts the full plan in Looks good preview; Step 6/8 flag-release steps follow release-plans; Step 1 and test-PR flows embed summary/diff in questions.

Evals add runQuestionShowsCommand (curl in Run-this-command questions) and confirmShowsPlan (list plan naming every repo written), wired into ~20 promptfoo cases, plus unit tests. skills.json / marketplace metadata pick up version bumps.

Reviewed by Cursor Bugbot for commit d08fa7a. Bugbot is set up for automated code reviews on this repo. Configure here.

…uestion

While a question is open, the user may see only the question and not the
message above it, so they were asked to run a command they could not see.
The skill now puts the command in the preview of "Done, here's the output"
when the question tool has previews, and in the question text otherwise.
Onboarding's two "Run this command" steps point at that rule.

Add a runQuestionShowsCommand eval assertion to the six cases that show REST
commands, with unit tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Generated-by: termag
@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown

Skill eval results

Skill Before After Δ
agentcontrol/configs-create 100/100 (4/4) 100/100 (4/4) no change
agentcontrol/configs-update 80/100 (4/5) 80/100 (4/5) no change
agentcontrol/configs-variations 80/100 (4/5) 80/100 (4/5) no change
agentcontrol/tools 75/100 (3/4) 75/100 (3/4) no change
aws-devops-agent/experiment-orchestration - 100/100 (9/9) new
experiments/launchdarkly-experiment-guide - 100/100 (5/5) new
experiments/launchdarkly-experiment-hypothesis-builder - 100/100 (13/13) new
experiments/launchdarkly-experiment-setup - 100/100 (8/8) new
feature-flags/flag-and-release-change - 100/100 (5/5) new
feature-flags/flag-release - 86/100 (6/7) new
feature-flags/launchdarkly-flag-command - 100/100 (3/3) new
feature-flags/launchdarkly-flag-create 100/100 (3/3) 100/100 (4/4) no change
feature-flags/launchdarkly-flag-drift - 100/100 (4/4) new
feature-flags/should-flag-change - 100/100 (17/17) new
onboarding - 86/100 (6/7) new
release-autopilot/launchdarkly-release-autopilot-onboarding - 79/100 (30/38) new
release-autopilot/launchdarkly-release-plans - 88/100 (15/17) new

Only suites whose source actually changed since their last recorded score were re-run. Soft-failing while we stabilise the baseline.

…ummary, or diff it asks about

While a question is open, the user may see only the question and not the
message above it. Onboarding already put links in the question text. Widen
that rule: the Step 1 summary goes in the question text, and the Step 7
plan, a changed flag release plan, and the test pull request's diff go in
the preview of the option that accepts them (or the question text without
previews). The release plans skill, which onboarding follows in Record only
mode, gets the same rule as a core principle and on its "Looks right" steps.

Add a confirmShowsPlan eval assertion to the 15 onboarding cases that check
confirmsBeforeWriting, with unit tests. Bump both skills' patch versions.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Generated-by: termag
@fsaresh fsaresh changed the title fix(release-plans): show the curl command in the "Run this command" question fix(release-autopilot): questions carry the command, plan, or diff they ask about Nov 13, 2026
fsaresh and others added 2 commits November 13, 2026 18:33
runQuestionShowsCommand now reads the prompt and the preview of the
"Done, here's the output" option. confirmShowsPlan now reads the prompt
and the preview of the "Looks good" option. A command or plan that shows
only under Skip or "Change something" no longer passes. Add negative
tests for both.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Generated-by: termag
@fsaresh
fsaresh merged commit 94c75aa into main Nov 13, 2026
24 checks passed
@fsaresh
fsaresh deleted the run-command-question-shows-command branch November 13, 2026 19:04
fsaresh added a commit that referenced this pull request Nov 13, 2026
Main merged #266, which added confirmShowsPlan and runQuestionShowsCommand
to onboarding eval cases. This branch deletes four of those cases and moves
one to the release plans suite, so the additions to them are dropped. Both
sets of new release plans assertion tests are kept.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Generated-by: termag
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants