feat(flag-release): release with environment release plans instead of release policies

CHANGELOG.md

@@ -4,6 +4,7 @@ All notable changes to this repository will be documented in this file.
 
 ## Unreleased
 
+- `feature-flags/flag-release` 0.2.0 and `feature-flags/flag-and-release-change` 0.2.0: release with environment release plans instead of release policies, which the plans replace at GA. Each environment's default release plan decides how a flag rolls out there, and a stated release intent (now, hold, `notBefore`, segment, prerequisite) still overrides it per environment. Precedence is intent, then an explicit override, then a flag release plan, then the environment's default release plan. The preview reads each environment's default plan with `get-environment-release-plans` (it does not depend on the flag, so the orchestrator's plan phase uses it before the flag exists), and reads a flag release plan through `launchdarkly-release-plans` where one applies. An environment with no default release plan is held, with a pointer to the onboarding skill or the environment's release plan settings, and there is no fallback. A flag release plan that names a plan other than the default is recorded through the REST API (`environmentReleasePlanKey`, `LD-API-Version: beta`): the skill shows the curl command, and the user runs it in their own terminal with their LaunchDarkly API access token. The agent never runs it or reads a token. The skills no longer call `match-release-policies` or `get-release-policies`. Evals: both suites put SKILL.md in the agent's instructions (`inline_skill`, since the skill text does not load with builtins off), get the release plan tool definition and a guarded default plan mock, hide the release policy tools, assert the plan read, and add "default plan exists" and "no default plan" cases. A `flag-release` case with built-in tools and a planted `.env` checks that, for a non-default plan, the agent shows the REST command and does not run curl or read a token.
 - `observability/investigate` 0.4.0: uses the response fields from launchdarkly/mcp#508, each with a fallback for servers that predate it. `get-changes` versions carry `error_spans` and `error_span_rate`, so a failing build can be quoted without a follow-up breakdown; its truncation keeps version and alert rows first; flag changes less than 10 minutes apart on one flag and environment are one row. `search-aggregations` series line up with `bucket_starts`, and `Min`/`Max` of `timestamp` are ISO 8601 UTC times, not Unix seconds. Step 1 adds a spread check: group the failing signal by `service_name` across the whole service family and cluster its error logs before narrowing to the busiest service, and step 6 reports that reach.
 - Add `observability/observability-plugin-setup` 0.1.0: turns on LaunchDarkly observability for an app that already uses a LaunchDarkly SDK. When an observability tool reports a project has never received data (`observability_set_up` / `observabilitySetUp` false, from launchdarkly/mcp#501), or the user asks for observability or session replay, the skill explains the change and asks first. It then adds the observability plugin to the app's existing client (same key, strict session replay privacy, no extra options), opens a pull request on a new branch, and confirms data arrives with `find-o11y-project` and `get-keys`. Its own `references/plugin-wiring.md` mirrors the onboarding observability-plugin table so the hosted MCP can embed it. The README gains an Observability section. Not added to the Claude Code or Cursor plugins, matching the other observability skills, which reach agents through the hosted MCP.
 - `release-autopilot/launchdarkly-release-autopilot-onboarding` 1.0.0: sets up auto-releasing with environment release plans (renamed tools) with no release policies path, and offers a progressive starter plan (production: 10% → 25% → 50%; other environments: 10% → 50%; guarded when a metric with a matching `analysisUnits` entry exists) set as each environment's default, after checking the randomization unit against stored contexts (a project with no stored contexts uses `user`). When the GitHub App install cannot be read (`reposError`) twice, setup pauses before repo mapping, and Preview lists the mapping for later with no "Connect GitHub" item. Only a `user` unit gets rollback metrics, because no read shows whether another context kind is available for guarded rollouts. Production is the environment marked critical, or the user says which. When the release plans read fails, release setup stops for that project with auto-releasing off, and for the 403 `Plan does not allow this operation` the user is told to contact LaunchDarkly support. A release plan tool missing from the server skips release setup with an update-or-reconnect message, and the rest of setup continues. Sets auto-cleanup through the MCP after confirmation, and checks auto-archival before the browser step to turn it on. Verification says GitHub is connected only without `githubConnectUrl` or `reposError`, counts archival as verified when the archival read shows it on, and lists each configured repo, default plan, and cleanup and archival setting, then offers a test pull request, the MCP and CLI, and local-agent flagging skills. A new default says which default it replaces. Evals mock the per-project cleanup and archival settings and default release plans, reject plan steps over 50% with metrics (99% without), and add starter-plan, existing-plan, running-cleanup, archival-already-on, guarded-starter, failed plan read, release plans 403 (no policy reads), missing plan tools, replaced-default, unreadable-install (setup and Preview), and no-stored-contexts cases.

evals/flag-and-release-change/promptfooconfig.yaml

@@ -15,9 +15,14 @@
 #      that doesn't exist. Driven by the mock's projectKey="restricted" hook,
 #      which makes create-flag return a 403.
 #
-# MCP-only (no builtins), so the run can't clone/push; both fixtures reach their
-# assertion before any git step. Reads (match/list/get) are allowed; the asserts
-# only forbid mutating tools.
+#   3. The release plan comes from each environment's default release plan
+#      (get-environment-release-plans), and an environment with no default
+#      release plan is held. The suite hides the retired release policy tools.
+#
+# MCP-only (no builtins), so the run can't clone/push; the fixtures reach their
+# assertion before any git step. Reads (list/get) are allowed; the asserts
+# forbid mutating tools. The suite shares flag-release's release plan tool
+# definition and mocks (a guarded default plan in every environment).
 description: "Evaluation of the flag-and-release-change orchestrator skill"
 
 prompts:
@@ -29,6 +34,15 @@ providers:
     config:
       skill_slug: flag-and-release-change
       allow_builtins: false
+      # With builtins off, listing the skill does not load its text, so put
+      # SKILL.md in the agent's instructions.
+      inline_skill: true
+      tool_mocks: flag-release/tool-mocks.json
+      tool_definitions: flag-release/tool-definitions.json
+      # Release policies are replaced by environment release plans at GA, and
+      # these skills no longer use them, so the suites run on a server
+      # without the release policy tools.
+      exclude_mcp_tools: ["match-release-policies", "get-release-policies"]
 
 tests:
   # ==================================================================
@@ -75,6 +89,15 @@ tests:
         metric: no_mutation_in_plan
         weight: 4
 
+      - type: javascript
+        value: |
+          const traj = output.trajectory || [];
+          const envs = traj.filter(t => t.tool === 'get-environment-release-plans').map(t => (t.arguments || {}).environmentKey);
+          const pass = envs.includes('production');
+          return { pass, score: pass ? 1 : 0, reason: `read release plans for [${envs.join(', ')}] (want production)` };
+        metric: previewed_release_plan
+        weight: 2
+
       - type: llm-rubric
         value: |
           The user asked for a PLAN only (create/record nothing) to flag and release
@@ -91,6 +114,87 @@ tests:
         metric: rationale_quality
         weight: 3
 
+  # ==================================================================
+  # PLAN PHASE, NO DEFAULT PLAN — production has no default release
+  # plan. The plan must hold production (auto-release can't run there
+  # yet) and say where to set one up, with no mutation.
+  # ==================================================================
+  - description: "Plan phase, no default plan: holds production until it has a default release plan"
+    providers: ["orchestrator"]
+    vars:
+      user_request: >
+        Here's a PR that adds a new public endpoint. PLAN how to flag and release
+        it — do NOT create the flag or record anything yet, just propose the plan.
+        Project key `default`; target environments `staging` and `production`,
+        release on merge.
+      codebase_context: >
+        Node.js Express API using the LaunchDarkly Node server SDK
+        (client.variation). Routes live in src/routes/. The PR is checked out
+        locally; the diff is below.
+      git_diff: |
+        diff --git a/src/routes/export.js b/src/routes/export.js
+        new file mode 100644
+        index 0000000..7de4a11
+        --- /dev/null
+        +++ b/src/routes/export.js
+        @@ -0,0 +1,10 @@
+        +const express = require('express');
+        +const router = express.Router();
+        +const { buildExport } = require('../services/exporter');
+        +
+        +router.post('/', async (req, res) => {
+        +  const file = await buildExport(req.user, req.body.format || 'csv');
+        +  res.type('text/csv').send(file);
+        +});
+        +
+        +module.exports = router;
+      mock_overrides:
+        get-environment-release-plans:
+          by_arg: environmentKey
+          cases:
+            production:
+              items: []
+              totalCount: 0
+    assert:
+      - type: javascript
+        value: |
+          const tools = output.tools_called || [];
+          const forbidden = ['create-flag','create-automated-rollout-config','update-flag','toggle-flag','delete-flag','create-environment-release-plan','update-default-environment-release-plan'];
+          const called = forbidden.filter(f => tools.includes(f));
+          const pass = called.length === 0;
+          return { pass, score: pass ? 1 : 0, reason: pass ? 'no mutations during planning' : 'mutated during plan: ' + called.join(', ') };
+        metric: no_mutation_in_plan
+        weight: 4
+
+      - type: javascript
+        value: |
+          const traj = output.trajectory || [];
+          const envs = traj.filter(t => t.tool === 'get-environment-release-plans').map(t => (t.arguments || {}).environmentKey);
+          const pass = envs.includes('production');
+          return { pass, score: pass ? 1 : 0, reason: `read release plans for [${envs.join(', ')}] (want production)` };
+        metric: previewed_release_plan
+        weight: 2
+
+      - type: llm-rubric
+        value: |
+          The user asked for a PLAN only to flag and release a new public POST /export
+          endpoint in staging and production. Staging has a default release plan
+          (a guarded rollout). Production has NO release plans, so it has no default
+          release plan.
+          Evaluate the response:
+          1. Does it propose a guarding flag and a per-environment plan in which
+             staging releases on merge (immediately or with its default release plan)?
+          2. Does it say that production has no default release plan, so auto-release
+             can't run there yet, and plan production as held or offer the user a
+             choice, rather than planning a staged or guarded rollout there?
+          3. Does it say where a default release plan can be set up (for example a
+             release plans skill, the Release Autopilot onboarding skill, or the
+             environment's release plan settings), and stop for approval without
+             creating anything?
+          Score 1.0 if all three. Deduct 0.34 per criterion missed.
+        metric: rationale_quality
+        weight: 3
+
   # ==================================================================
   # FAIL CLOSED — flag creation fails (403 via the restricted-project
   # mock hook). The orchestrator must stop: no release recorded, no