Repository navigation
feat(flag-release): release with environment release plans instead of release policies - #260
Merged
Merged
Conversation
… release policies At GA every plan includes environment release plans, which replace release policies. flag-release and flag-and-release-change (0.2.0) now: - Preview each environment's default release plan with get-environment-release-plans instead of match-release-policies. - Use precedence intent -> override -> flag release plan -> the environment's default release plan. - Hold an environment with no default release plan, with no fallback, and point to onboarding or the environment's release plan settings. - Read flag release plans through launchdarkly-flag-release-plans, and record a non-default environment release plan through the REST API (environmentReleasePlanKey, LD-API-Version: beta) with the user's own token. Evals: both suites get the release plan tool definition and a guarded default plan mock, assert no release policy calls, and add "default plan exists" and "no default plan" cases. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Generated-by: termag
fsaresh
marked this pull request as ready for review
November 12, 2026 20:09
Skill eval results
Only suites whose source actually changed since their last recorded score were re-run. Soft-failing while we stabilise the baseline. |
…, rename to launchdarkly-release-plans CI scored flag-release 4/6 and flag-and-release-change 3/5. The suites still exposed match-release-policies and get-release-policies, whose mocks return a guarded policy for production, so an agent could find a policy to release with in the no-default-plan cases. Two rubrics were also stricter than the skill: staging may release immediately, and the hosted create tool's own text says "release policy". - Hide the release policy tools in both suites and drop the no_policy_tools assertions, which are now always true. - Loosen the default-plan and no-default-plan rubrics to match the skill. - flag-release: do not copy a non-default plan's steps into an override for an environment with no default plan. - Point to launchdarkly-release-plans (renamed from launchdarkly-flag-release-plans), including for creating a missing default plan. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Generated-by: termag
…ault release plan The hosted create-automated-rollout-config description still describes releaseType "policy" in older terms, and evals run without builtins, so the agent never reads references/auto-release.md. Say in SKILL.md that a policy entry starts the default release plan, and not to copy the plan into an override. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Generated-by: termag
… or reads a token The plugin directory blocks skills that use a credential from the user's machine (#243). For a non-default environment release plan, the skill now shows the curl command, with the LD-API-Version: beta header and a placeholder for the token, and asks the user to run it in their own terminal and paste back the response with secrets removed. It no longer names a token env var. Evals: a flag-release case with built-in tools and a planted .env checks that the agent shows the command and does not run curl or read a token. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Generated-by: termag
Eval job logs can't always be downloaded (for example through a proxy), and the PR comment shows only pass counts. When running in GitHub Actions, aggregate.js now emits one ::warning annotation per failing test, with the failed assertion metrics and their reasons. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Generated-by: termag
The new failing-test annotations show the agent never called get-environment-release-plans and recorded production as policy with no default plan. With builtins off, listing the skill does not load its text (see inline_skill in the provider), so the agent worked from the tool descriptions alone. Turn on inline_skill for the no-builtins providers. The orchestrator now says in its own plan step that an environment with no default release plan is held, since flag-release is not loaded in its suite. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Generated-by: termag
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
…irmation With SKILL.md now loaded in the evals, the agent stopped after the plan in every recording case, although the user had asked to record and had given the intent. Treat that request as the confirmation and go on to the implement phase. An environment with unclear intent is held and does not stop the others. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Generated-by: termag
fsaresh
added a commit
that referenced
this pull request
Nov 12, 2026
…'s pull request proposals Release Autopilot's release proposals on pull requests use the project's flag release plans, and it proposes none without one. A rollout recorded for a flag (the flag-release skill, create-automated-rollout-config with policy) uses each environment's default release plan and does not need a flag release plan. Matches #260. The onboarding note now says Release Autopilot proposes no release on pull requests until the flag release plan exists. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Generated-by: termag
vroske-ld
approved these changes
Nov 13, 2026
fsaresh
added a commit
that referenced
this pull request
Nov 13, 2026
…ment and flag release plans; onboarding follows it (#261) * feat(release-autopilot): add flag release plans skill; onboarding builds the project's flag release plan - New skill launchdarkly-flag-release-plans 0.1.0-experimental: list, get, create, replace, make default, and delete a project's flag release plans, and start a flag's release from one, through the REST API with curl and LD-API-Version: beta on every request. The hosted MCP has no flag release plan tools, and ldcli 3.16.0 has no command for these endpoints and no raw request command that sends the beta header. The token comes from LD_ACCESS_TOKEN in the user's shell; the skill never asks for it in the chat or runs a command that can print it. Confirm before every write; read-only requests send only GET; 400, 401, 403, 404, and 409 each have a next step. - Onboarding 1.1.0: Step 6 lists the project's flag release plans and records one to create, reuse, keep, or replace; Step 7 lists it; Step 8 creates it after the environment release plans and their defaults and before the repo writes; Step 9 reads it back. Preview sends only the read. Without a shell or token, or on a failed request, the plan is a follow-up with the exact command and setup continues. - Evals: the provider's ld_api_mock option starts a local REST API mock per test, points the agent's shell at it with a fake token, and records each request as an "ld-api" trajectory entry. New suite for the skill; three onboarding cases (with a shell, without one, Preview); unit tests for the mock and the new assertions. - Claude Code plugin 1.0.8 (adds the skill), Cursor plugin 1.0.9. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Generated-by: termag * feat(release-autopilot): one launchdarkly-release-plans skill for environment and flag release plans - Rename launchdarkly-flag-release-plans to launchdarkly-release-plans 1.0.0. It covers environment release plans (read, choose, starter and custom plans, defaults and edits through the MCP tools; delete and clear the default through the REST API) and flag release plans (REST API with curl, LD-API-Version: beta, and LD_ACCESS_TOKEN). It has Apply, Read-only, and Record only modes. Starter definitions and the custom questions move here from onboarding's references/release-plans.md. - A flag release plan uses existing environment release plans first. An environment with none gets a release plan first, or is left out when the user declines. The skill never invents a key or points at an incomplete plan. A plan stores each environment's release plan key. - Onboarding 1.1.0 follows the skill in Record only mode. A default flag release plan that does not match is updated in place ("Update <name>" or "Keep <name>"). When the flag release plan is a follow-up, auto-releasing stays on, and the summary says it proposes no release until the plan exists. - Settings 1.6.0-experimental points to the new skill. - Evals: the starter, guarded, no-contexts, and custom plan cases move to the new suite, plus key collision, reuse, create-first, and decline cases. Onboarding adds the update-in-place and auto-releasing-on cases. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Generated-by: termag * feat(release-plans): check for an existing default before every create Before any create (a direct request, Create, Build, or Record only), the skill reads the plans and, when a default exists, names it with a short summary and asks "Default plan" (environment: use it, create and replace it, or create and keep it) or "Flag release plan" (update, keep, or create another). It also names incomplete plans. Onboarding keeps only Update and Keep for the flag release plan. Evals: a create request when the environment has a default and when the project has a default flag release plan; both name the default first and write nothing when kept. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Generated-by: termag * fix(release-plans): the user runs REST commands; the agent never uses a token For every REST operation (flag release plans, and environment release plan changes without an MCP tool), the skill shows the exact curl command with LD-API-Version: beta, the user runs it in their own terminal with their own API access token and pastes the output, and the skill reads that output to continue. The agent no longer checks for, reads, or runs anything with LD_ACCESS_TOKEN or LD_BASE_URI. Onboarding makes the flag release plan a user-run step after "Looks good", like the browser steps; a skipped step is a follow-up with auto-releasing on. Evals: remove the local REST API mock. A scripted { when, text } answer stands for pasted output, and scrub_ld_credentials removes LaunchDarkly credentials from the agent's shell. Cases check that each command shows the beta header, that the agent never runs curl or reads a token, and that it reads pasted lists, creates, 409s, 403s, and a pasted token. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Generated-by: termag * fix(release-plans): paste answers match the shown command; Preview runs nothing; pick a default among complete plans - Evals: a pasted answer is now { command, text } and answers a question only when no scripted option answer fits and the last REST command the agent showed (METHOD /api/v2/...) matches its regex, so pastes no longer take plan or confirm questions (Bugbot 4556463727). Unit test included. - Read-only mode, and a caller's preview in Record only mode, use MCP reads only and ask the user to run nothing; REST reads become optional follow-ups. Onboarding Preview says the same (Bugbot 4556463741). New assertion asksToRunNothing on the preview case. - An environment with complete plans but no default: show each plan and ask which one becomes the default (Bugbot 4556312137). New case and assertion picksDefaultAmongComplete. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Generated-by: termag * fix(release-autopilot): settings description, late flag plan list reconciles, pastes only answer the run step - Settings: the description says "set up environment release plans and the project's flag release plan" instead of "review release policies", and points to launchdarkly-release-plans (skills.json regenerated). - Onboarding Step 8: when Step 6 had no list, compare the pasted list with the recorded plan; if it shows a default flag release plan, ask Update or Keep, show the changed plan, and ask for "Looks good" again before any write command. - Evals: a { command, text } paste answers only the "Run this command" step, so a retry confirmation after a 409 cannot take the next paste. Unit test included. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Generated-by: termag * fix(evals): no built-in AskUserQuestion with builtins on; annotate failed tests in CI - With allow_builtins and expose_ask_question, disallow the built-in AskUserQuestion, so structured questions go through the harness's ask-question tool and get the scripted answers. The release plans suite runs with builtins on (to check the agent never runs a REST command), and its first CI run scored 3/16. - aggregate.js prints one GitHub warning annotation per failed test with the failed assertions' reasons, so a failure can be read from the check run's annotations when the job log cannot be downloaded. main() runs only when the script is run directly. Unit test included. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Generated-by: termag * docs(release-plans): scope "proposes no release" to Release Autopilot's pull request proposals Release Autopilot's release proposals on pull requests use the project's flag release plans, and it proposes none without one. A rollout recorded for a flag (the flag-release skill, create-automated-rollout-config with policy) uses each environment's default release plan and does not need a flag release plan. Matches #260. The onboarding note now says Release Autopilot proposes no release on pull requests until the flag release plan exists. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Generated-by: termag * fix(evals): release plans suite runs without builtins; failure annotations show what the agent did - The release plans suite now runs with builtins off, like the onboarding suite. With builtins on, CI showed most cases never reached a create. - Failure annotations add the agent's tool calls, a max-turns stop, and the start of its final reply. - Preview case: Read-only lists write commands as optional commands, so the case checks that the user is asked to run nothing instead of that no write command is shown. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Generated-by: termag * fix(evals): escape annotation titles with GitHub's escapeProperty rules The title property now encodes ":" and "," as well as "%", CR, and LF; the message keeps escapeData (%, CR, LF). Unit test with a description that has a comma and a colon. Bugbot 4556926792. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Generated-by: termag * fix(release-plans): ask for pasted output with the question tool; suite loads the skill inline - CI annotations showed two causes for the release plans suite failures: - with force_skill_invocation, every MCP call was interrupted, and the agent reported the MCP server as unreachable. The suite now uses inline_skill with the new inline_references option, which also puts the skill's references into the instructions (builtins are off). - the agent showed the command and ended its turn without asking. The skill now says to ask "Run this command" with the question tool in the same message, so the user can paste the output as the answer. - Unit tests for skillInstructions with references and readReferences. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Generated-by: termag --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…env-release-plans Generated-by: termag # Conflicts: # CHANGELOG.md
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit a7eca33. Configure here.
… provider Bugbot: the mcp-builtins provider turns on the shell but did not set scrub_ld_credentials, so a curl the agent should not run could reach a live account if CI had a LaunchDarkly token. Set it, as the onboarding shell provider does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Generated-by: termag
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
flag-releaseandflag-and-release-change(both 0.2.0) now release each environment with its default environment release plan. They no longer use release policies. At GA (Nov 16) every LaunchDarkly plan includes environment release plans, and the plans replace release policies. The onboarding skill made the same change in #252.What changes
Today the skills preview each environment with
match-release-policiesand say they "defer per-environment to the project's release policies". After this PR:get-environment-release-plans. The default plan does not depend on the flag, so the orchestrator's plan phase can read it before the flag exists.launchdarkly-release-plansskill. That skill also creates a missing default plan. These two skills do not create, change, or delete release plans.references/auto-release.mdgives a REST command (POST /api/v2/projects/{projectKey}/automated-rollout-configswithenvironmentReleasePlanKeyandLD-API-Version: beta). The skill shows the command in a code block and asks the user to run it in their own terminal with their LaunchDarkly API access token, then paste back the response with secrets removed. The agent never runs it, asks for the token, or reads a token from the machine.match-release-policies, orget-release-policiesstays in either skill.Dependency
The skills link to
skills/release-autopilot/launchdarkly-release-plans/SKILL.md. #261 adds that skill, and it is not on main yet. Until it merges, the links are broken.Evals
flag-releaseget-environment-release-plans(hosted MCP text) and a guarded default plan mock. The suite hidesmatch-release-policiesandget-release-policies. The hold case asserts a plan read. The happy path became "Default plan exists" and asserts production is recorded aspolicy. New "No default plan" case: production has only a non-default plan, so it must be left out, with no policy fallback.flag-and-release-changeflag-releasetool definition and mocks. Hides the release policy tools. The plan phase case asserts a production plan read. New "Plan phase, no default plan" case.flag-release(built-in tools).env: the agent must show the curl command withLD-API-Version: betaandenvironmentReleasePlanKey, and must not run curl or read a token.evals/tests/mock.test.jsThe shared
match-release-policiesandget-release-policiesmocks stay.launchdarkly-metric-choose, its eval, and the onboarding assertions still use them.Open questions
create-automated-rollout-configstill describespolicyas "the environment's release policy". It also buildsmissing_policyandincomplete_policywarnings from release policies. After GA it can warn on everypolicyenvironment, including environments with a default plan. The skill tells the agent to show those warnings but plan from the default plans. The MCP tool needs an update.validateDefaultRuleReleasePlan). Every plan must include the randomization unit clause. It is not clear from the code whether that clause counts as a targeting clause. If it does, plans created through onboarding can't run auto-release.Possible follow-up
Testing
python3 scripts/validate_skills.py,python3 -m unittest discover -s tests,python3 scripts/generate_catalog.py --check: pass.npm testinevals/: 182 pass.🤖 Generated with Claude Code
Opened from termag
Note
Overview
flag-releaseandflag-and-release-change(0.2.0) drop release policies entirely and preview rollout behavior withget-environment-release-plans, using each environment’s default plan (and optional flag release plans vialaunchdarkly-release-plans). Per-environment precedence is now intent → override → flag release plan → default plan; environments without a default plan are held with setup pointers instead of falling back to policies. When MCP can’t name a non-default plan,auto-release.mddocuments a beta RESTcurlthe user runs—agents must not execute it or read tokens.Evals were retargeted: shared release-plan mocks/definitions,
inline_skill, hiddenmatch-release-policies/get-release-policies, new “no default plan” and REST/non-default-plan cases (including credential scrubbing with builtins on), and orchestrator plan-phase assertions on plan reads.aggregate.jsemits GitHub warning annotations for failing eval assertions. Catalog, marketplace metadata, and changelog reflect release-plan tagging and versions.Reviewed by Cursor Bugbot for commit a37bbb6. Bugbot is set up for automated code reviews on this repo. Configure here.