Repository navigation
feat(release-autopilot): launchdarkly-release-plans skill for environment and flag release plans; onboarding follows it - #261
Merged
Conversation
…lds the project's flag release plan - New skill launchdarkly-flag-release-plans 0.1.0-experimental: list, get, create, replace, make default, and delete a project's flag release plans, and start a flag's release from one, through the REST API with curl and LD-API-Version: beta on every request. The hosted MCP has no flag release plan tools, and ldcli 3.16.0 has no command for these endpoints and no raw request command that sends the beta header. The token comes from LD_ACCESS_TOKEN in the user's shell; the skill never asks for it in the chat or runs a command that can print it. Confirm before every write; read-only requests send only GET; 400, 401, 403, 404, and 409 each have a next step. - Onboarding 1.1.0: Step 6 lists the project's flag release plans and records one to create, reuse, keep, or replace; Step 7 lists it; Step 8 creates it after the environment release plans and their defaults and before the repo writes; Step 9 reads it back. Preview sends only the read. Without a shell or token, or on a failed request, the plan is a follow-up with the exact command and setup continues. - Evals: the provider's ld_api_mock option starts a local REST API mock per test, points the agent's shell at it with a fake token, and records each request as an "ld-api" trajectory entry. New suite for the skill; three onboarding cases (with a shell, without one, Preview); unit tests for the mock and the new assertions. - Claude Code plugin 1.0.8 (adds the skill), Cursor plugin 1.0.9. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Generated-by: termag
fsaresh
marked this pull request as ready for review
November 12, 2026 20:24
fsaresh
marked this pull request as draft
November 12, 2026 20:25
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
…ironment and flag release plans
- Rename launchdarkly-flag-release-plans to launchdarkly-release-plans
1.0.0. It covers environment release plans (read, choose, starter and
custom plans, defaults and edits through the MCP tools; delete and
clear the default through the REST API) and flag release plans (REST
API with curl, LD-API-Version: beta, and LD_ACCESS_TOKEN). It has
Apply, Read-only, and Record only modes. Starter definitions and the
custom questions move here from onboarding's references/release-plans.md.
- A flag release plan uses existing environment release plans first. An
environment with none gets a release plan first, or is left out when
the user declines. The skill never invents a key or points at an
incomplete plan. A plan stores each environment's release plan key.
- Onboarding 1.1.0 follows the skill in Record only mode. A default flag
release plan that does not match is updated in place ("Update <name>"
or "Keep <name>"). When the flag release plan is a follow-up,
auto-releasing stays on, and the summary says it proposes no release
until the plan exists.
- Settings 1.6.0-experimental points to the new skill.
- Evals: the starter, guarded, no-contexts, and custom plan cases move to
the new suite, plus key collision, reuse, create-first, and decline
cases. Onboarding adds the update-in-place and auto-releasing-on cases.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Generated-by: termag
fsaresh
changed the title
fsaresh
marked this pull request as ready for review
November 12, 2026 20:57
Before any create (a direct request, Create, Build, or Record only), the skill reads the plans and, when a default exists, names it with a short summary and asks "Default plan" (environment: use it, create and replace it, or create and keep it) or "Flag release plan" (update, keep, or create another). It also names incomplete plans. Onboarding keeps only Update and Keep for the flag release plan. Evals: a create request when the environment has a default and when the project has a default flag release plan; both name the default first and write nothing when kept. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Generated-by: termag
Skill eval results
Only suites whose source actually changed since their last recorded score were re-run. Soft-failing while we stabilise the baseline. |
… a token
For every REST operation (flag release plans, and environment release
plan changes without an MCP tool), the skill shows the exact curl
command with LD-API-Version: beta, the user runs it in their own
terminal with their own API access token and pastes the output, and the
skill reads that output to continue. The agent no longer checks for,
reads, or runs anything with LD_ACCESS_TOKEN or LD_BASE_URI. Onboarding
makes the flag release plan a user-run step after "Looks good", like the
browser steps; a skipped step is a follow-up with auto-releasing on.
Evals: remove the local REST API mock. A scripted { when, text } answer
stands for pasted output, and scrub_ld_credentials removes LaunchDarkly
credentials from the agent's shell. Cases check that each command shows
the beta header, that the agent never runs curl or reads a token, and
that it reads pasted lists, creates, 409s, 403s, and a pasted token.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Generated-by: termag
fsaresh
marked this pull request as draft
November 12, 2026 21:09
fsaresh
marked this pull request as ready for review
November 12, 2026 21:14
…ns nothing; pick a default among complete plans
- Evals: a pasted answer is now { command, text } and answers a question
only when no scripted option answer fits and the last REST command the
agent showed (METHOD /api/v2/...) matches its regex, so pastes no longer
take plan or confirm questions (Bugbot 4556463727). Unit test included.
- Read-only mode, and a caller's preview in Record only mode, use MCP
reads only and ask the user to run nothing; REST reads become optional
follow-ups. Onboarding Preview says the same (Bugbot 4556463741). New
assertion asksToRunNothing on the preview case.
- An environment with complete plans but no default: show each plan and
ask which one becomes the default (Bugbot 4556312137). New case and
assertion picksDefaultAmongComplete.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Generated-by: termag
…onciles, pastes only answer the run step
- Settings: the description says "set up environment release plans and
the project's flag release plan" instead of "review release policies",
and points to launchdarkly-release-plans (skills.json regenerated).
- Onboarding Step 8: when Step 6 had no list, compare the pasted list
with the recorded plan; if it shows a default flag release plan, ask
Update or Keep, show the changed plan, and ask for "Looks good" again
before any write command.
- Evals: a { command, text } paste answers only the "Run this command"
step, so a retry confirmation after a 409 cannot take the next paste.
Unit test included.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Generated-by: termag
…iled tests in CI - With allow_builtins and expose_ask_question, disallow the built-in AskUserQuestion, so structured questions go through the harness's ask-question tool and get the scripted answers. The release plans suite runs with builtins on (to check the agent never runs a REST command), and its first CI run scored 3/16. - aggregate.js prints one GitHub warning annotation per failed test with the failed assertions' reasons, so a failure can be read from the check run's annotations when the job log cannot be downloaded. main() runs only when the script is run directly. Unit test included. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Generated-by: termag
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit d58ee33. Configure here.
…'s pull request proposals Release Autopilot's release proposals on pull requests use the project's flag release plans, and it proposes none without one. A rollout recorded for a flag (the flag-release skill, create-automated-rollout-config with policy) uses each environment's default release plan and does not need a flag release plan. Matches #260. The onboarding note now says Release Autopilot proposes no release on pull requests until the flag release plan exists. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Generated-by: termag
…tions show what the agent did - The release plans suite now runs with builtins off, like the onboarding suite. With builtins on, CI showed most cases never reached a create. - Failure annotations add the agent's tool calls, a max-turns stop, and the start of its final reply. - Preview case: Read-only lists write commands as optional commands, so the case checks that the user is asked to run nothing instead of that no write command is shown. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Generated-by: termag
The title property now encodes ":" and "," as well as "%", CR, and LF; the message keeps escapeData (%, CR, LF). Unit test with a description that has a comma and a colon. Bugbot 4556926792. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Generated-by: termag
…te loads the skill inline
- CI annotations showed two causes for the release plans suite failures:
- with force_skill_invocation, every MCP call was interrupted, and the
agent reported the MCP server as unreachable. The suite now uses
inline_skill with the new inline_references option, which also puts
the skill's references into the instructions (builtins are off).
- the agent showed the command and ended its turn without asking. The
skill now says to ask "Run this command" with the question tool in
the same message, so the user can paste the output as the answer.
- Unit tests for skillInstructions with references and readReferences.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Generated-by: termag
vroske-ld
approved these changes
Nov 13, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
launchdarkly-release-plans1.0.0. It is one skill for both kinds of LaunchDarkly release plan. Onboarding 1.1.0 follows it to set up release plans for auto-releasing. When Release Autopilot proposes a release on a pull request, it uses the project's flag release plan, and it proposes none in a project without one. A rollout recorded for a flag (theflag-releaseskill,create-automated-rollout-configwithpolicy) uses each environment's default release plan and does not need a flag release plan.Operations
get-environment-release-planscreate-environment-release-planupdate-default-environment-release-plan(RESTPUT/DELETE .../default-release-planwhen the tool is missing)update-environment-release-plan(RESTPUT .../release-plans/{releasePlanKey}when the tool is missing)DELETE .../environments/{env}/release-plans/{releasePlanKey}(no MCP tool)GET /api/v2/projects/{projectKey}/flag-release-plans[/{planKey}]POST .../flag-release-plansPUT .../flag-release-plans/{planKey}(full replace, withversion)DELETE .../flag-release-plans/{planKey}POST /api/v2/projects/{projectKey}/flags/{flagKey}/flag-releasesHow REST commands run
LD-API-Version: beta.ldcli3.16.0 has no command for these endpoints and no raw request command.Rules
search-contexts.analysisUnitsinclude the unit.progressive-2.Onboarding 1.1.0
references/release-plans.mdis gone.The settings skill (1.6.0-experimental) points to the new skill for release plans.
Evals
{ command, text }answer stands for output the user pasted after running a command the agent showed. It answers only the "Run this command" step.ask-questiontool.aggregate.jswrites one warning annotation per failed test with its reasons.scrub_ld_credentialsoption removes LaunchDarkly credentials from the agent's shell. The cases can then check that the agent never runs curl or reads a token. The local REST API mock from the first version is gone.launchdarkly-release-planssuite:npm test(213 tests),validate_skills.py, unittest andgenerate_catalog.py --checkpass. I did not run the LLM suites, because this box has no Anthropic key.Questions for review
enable-list-flag-release-plans-api). An account can get 403 on the list but not on the create.update-environment-release-plansays that it does.Possible follow-up: require a flag release plan for every rollout, if Valerie wants that.
Versions:
launchdarkly-release-plans1.0.0, onboarding 1.1.0, settings 1.6.0-experimental, Claude Code plugin 1.0.8, Cursor plugin 1.0.9.🤖 Generated with Claude Code
Opened from termag
Note
Overview
Introduces
launchdarkly-release-plans1.0.0, a single skill for environment release plans (MCP) and flag release plans (beta REST). Flag-plan operations are user-run only: the agent shows curl withLD-API-Version: beta, never executes API calls or handles tokens, and continues from pasted output.Release Autopilot onboarding 1.1.0 now follows that skill in Record only mode for release setup, adds an explicit flag release plan step (same user-run REST pattern), and drops the old
references/release-plans.mdin favor of the new skill’s references. Settings 1.6.0 delegates release-plan work to the new skill and clarifies that PR release proposals use project flag release plans.Ships a promptfoo suite (
eval:release-plans), shared REST command assertions, and harness updates: scripted pasted-command answers,scrub_ld_credentials, inline skill references, and GitHub Actions failure annotations. Claude/Cursor plugin manifests and skills.json register the skill (plugins 1.0.8 / 1.0.9).Reviewed by Cursor Bugbot for commit b077911. Bugbot is set up for automated code reviews on this repo. Configure here.