Repository navigation
fix(release-autopilot): questions carry the command, plan, or diff they ask about - #266
Merged
Merged
Conversation
…uestion While a question is open, the user may see only the question and not the message above it, so they were asked to run a command they could not see. The skill now puts the command in the preview of "Done, here's the output" when the question tool has previews, and in the question text otherwise. Onboarding's two "Run this command" steps point at that rule. Add a runQuestionShowsCommand eval assertion to the six cases that show REST commands, with unit tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Generated-by: termag
Skill eval results
Only suites whose source actually changed since their last recorded score were re-run. Soft-failing while we stabilise the baseline. |
…ummary, or diff it asks about While a question is open, the user may see only the question and not the message above it. Onboarding already put links in the question text. Widen that rule: the Step 1 summary goes in the question text, and the Step 7 plan, a changed flag release plan, and the test pull request's diff go in the preview of the option that accepts them (or the question text without previews). The release plans skill, which onboarding follows in Record only mode, gets the same rule as a core principle and on its "Looks right" steps. Add a confirmShowsPlan eval assertion to the 15 onboarding cases that check confirmsBeforeWriting, with unit tests. Bump both skills' patch versions. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Generated-by: termag
sarahlessner
approved these changes
Nov 13, 2026
runQuestionShowsCommand now reads the prompt and the preview of the "Done, here's the output" option. confirmShowsPlan now reads the prompt and the preview of the "Looks good" option. A command or plan that shows only under Skip or "Change something" no longer passes. Add negative tests for both. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Generated-by: termag
Generated-by: termag
fsaresh
added a commit
that referenced
this pull request
Nov 13, 2026
Main merged #266, which added confirmShowsPlan and runQuestionShowsCommand to onboarding eval cases. This branch deletes four of those cases and moves one to the release plans suite, so the additions to them are dropped. Both sets of new release plans assertion tests are kept. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Generated-by: termag
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
While a question dialog is open, the user may see only the question: its header, text and option labels. The message above it can be hidden. Two kinds of question in these skills asked about content that appeared only in the message above:
#267 adds the general rule to
CONTRIBUTING.mdand the skill template.What changed
launchdarkly-release-plans(1.0.0 → 1.0.1)AskUserQuestion), or else in the question text. It also stays in the message.SKILL.mdandreferences/environment-release-plans.md, point at that principle.launchdarkly-release-autopilot-onboarding(1.1.0 → 1.1.1)previewof the option that accepts it. Without previews, it goes in the question text.Evals
runQuestionShowsCommandcheck, added to the six cases that show REST commands (four in release plans, two in onboarding). Every "Done, here's the output" question must show a curl command in its text or in an option's preview.confirmShowsPlancheck, added next toconfirmsBeforeWritingin the 15 onboarding cases that use it. Every "Looks good" question must show the plan as a list, and the plan must name every repository that the run later writes.Testing
node --test tests/*.test.js:main, becausezodandjs-yamlare not installed in my checkout.scripts/validate_skills.py,python3 -m unittest discover -s testsandgenerate_catalog.py --checkpass.skills.jsonis regenerated for the version bumps.promptfooconfig.yamlfiles parse.Note
#264 and #265 change the same eval files, so whichever merges later needs a small rebase.
🤖 Generated with Claude Code
Opened from termag
Note
Overview
Release plans and onboarding skills now require that question dialogs include the content they ask about (plans, curl commands, diffs, links)—via option
previewor question text—because users may not see the message above while a question is open.launchdarkly-release-plans(1.0.0 → 1.0.1) adds a core principle and expands User-run commands so "Run this command" / "Done, here's the output" always shows the curl block in the question; starter "Looks right" steps reference the same rule.launchdarkly-release-autopilot-onboarding(1.1.0 → 1.1.1) replaces the links-only rule with a broader "question carries what it asks about" rule: Step 7 puts the full plan in Looks good preview; Step 6/8 flag-release steps follow release-plans; Step 1 and test-PR flows embed summary/diff in questions.Evals add
runQuestionShowsCommand(curl in Run-this-command questions) andconfirmShowsPlan(list plan naming every repo written), wired into ~20 promptfoo cases, plus unit tests.skills.json/ marketplace metadata pick up version bumps.Reviewed by Cursor Bugbot for commit d08fa7a. Bugbot is set up for automated code reviews on this repo. Configure here.