diff --git a/.claude-plugin/marketplace.json b/.claude-plugin/marketplace.json index 197aa135..b40f2de0 100644 --- a/.claude-plugin/marketplace.json +++ b/.claude-plugin/marketplace.json @@ -11,7 +11,7 @@ "name": "launchdarkly", "description": "LaunchDarkly agent skills and MCP servers for feature flag management, AgentControl, and metrics", "source": ".", - "version": "1.0.7", + "version": "1.0.8", "author": { "name": "LaunchDarkly", "email": "support@launchdarkly.com" diff --git a/.claude-plugin/plugin.json b/.claude-plugin/plugin.json index 55ed394c..a0d07494 100644 --- a/.claude-plugin/plugin.json +++ b/.claude-plugin/plugin.json @@ -1,6 +1,6 @@ { "name": "launchdarkly", - "version": "1.0.7", + "version": "1.0.8", "description": "LaunchDarkly agent skills and MCP servers for feature flag management, AgentControl, and metrics", "author": { "name": "LaunchDarkly", @@ -39,6 +39,7 @@ "./skills/release-autopilot/launchdarkly-release-autopilot-settings", "./skills/release-autopilot/launchdarkly-release-autopilot-onboarding", "./skills/release-autopilot/launchdarkly-automations-cleanup-settings", - "./skills/release-autopilot/launchdarkly-automations-archival-settings" + "./skills/release-autopilot/launchdarkly-automations-archival-settings", + "./skills/release-autopilot/launchdarkly-release-plans" ] } diff --git a/.cursor-plugin/plugin.json b/.cursor-plugin/plugin.json index ef2d69d8..840c2646 100644 --- a/.cursor-plugin/plugin.json +++ b/.cursor-plugin/plugin.json @@ -1,6 +1,6 @@ { "name": "launchdarkly", - "version": "1.0.8", + "version": "1.0.9", "description": "LaunchDarkly agent skills and mcp server for feature flag management, AgentControl, and skill authoring", "author": { "name": "LaunchDarkly", diff --git a/CHANGELOG.md b/CHANGELOG.md index 21e86dea..462c0544 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,9 @@ All notable changes to this repository will be documented in this file. ## Unreleased +- Add `release-autopilot/launchdarkly-release-plans` 1.0.0 (Claude Code plugin 1.0.8, Cursor plugin 1.0.9): one skill for both kinds of release plan. Environment release plans: read, choose (the environment's default first, other complete plans as alternatives), create from starter plans (Progressive, guarded with rollback metrics whose `analysisUnits` match the unit, at most 50% per step, or Custom) after checking the randomization unit with `search-contexts`, set the default, and edit through the MCP tools; delete, and clear the default, through the REST API. Flag release plans: list, get, create, update in place, make default, and delete, and start a flag's release from one, through the REST API (`/api/v2/projects/{projectKey}/flag-release-plans`, `/flag-release-plans/{planKey}`, `/flags/{flagKey}/flag-releases`). An environment with no complete release plan gets one first (or is left out when the user declines), and the skill never invents or points at an incomplete plan. A flag release plan stores each environment's release plan key, so a later change to that environment's default does not update it. For each REST operation the skill shows the exact curl command (with `LD-API-Version: beta`), the user runs it in their own terminal with their own API access token and pastes the output back, and the skill reads that output to continue; the agent never runs a REST command or uses a token, and never asks for a token in the chat. The commands use curl because the MCP server has no flag release plan tools and `ldcli` 3.16.0 has no command for these endpoints. Before any create, the skill checks for an existing default (environment or project), names it with a short summary, and asks whether to use it, replace it, or create another; it also names incomplete plans. Modes: Apply (confirm every write), Read-only, and Record only for skills that confirm and write later. Starter plan definitions and the custom plan questions move here from onboarding's `references/release-plans.md`. +- `release-autopilot/launchdarkly-release-autopilot-onboarding` 1.1.0: release setup follows `launchdarkly-release-plans` in Record only mode for environment release plans and the project's flag release plan; onboarding keeps the wizard steps, the confirmation, the write order, and verification, and its own `references/release-plans.md` is gone. The flag release plan is a new "Default release" plan when the project has no default, no change when the default matches, or the default updated in place ("Update " recommended, or "Keep "; never a second plan). Step 7 shows what an update changes and notes that the plan keeps its release plan keys when an environment's default changes later. Step 8 writes it after the environment release plans and their defaults and before the repo writes, and Step 9 reads it back. The flag release plan is a step the user runs after "Looks good", like the browser steps: the agent shows the command, waits for the pasted output, and verifies from it. If the user skips it or it fails, it is a follow-up, auto-releasing stays on as picked, and the summary says Release Autopilot proposes no release on pull requests until the plan exists. A starter offer can be declined ("Not now"). Evals: a scripted `{ command, text }` answer stands for output the user pastes after running a shown command, and the provider's `scrub_ld_credentials` option removes LaunchDarkly credentials from the agent's shell, so cases check that the agent shows each command with the beta header, never runs one or reads a token, and reads the pasted output; the starter, guarded, no-contexts, and custom plan cases move to the new skill's suite, and onboarding adds flag release plan cases (with a shell, skipped as a follow-up with auto-releasing on, Preview, and an in-place update). +- `release-autopilot/launchdarkly-release-autopilot-settings` 1.6.0-experimental: points to `launchdarkly-release-plans` for release plans, says Release Autopilot's pull request release proposals use only the project's flag release plans, and its description no longer says "review release policies". - `observability/investigate` 0.4.0: uses the response fields from launchdarkly/mcp#508, each with a fallback for servers that predate it. `get-changes` versions carry `error_spans` and `error_span_rate`, so a failing build can be quoted without a follow-up breakdown; its truncation keeps version and alert rows first; flag changes less than 10 minutes apart on one flag and environment are one row. `search-aggregations` series line up with `bucket_starts`, and `Min`/`Max` of `timestamp` are ISO 8601 UTC times, not Unix seconds. Step 1 adds a spread check: group the failing signal by `service_name` across the whole service family and cluster its error logs before narrowing to the busiest service, and step 6 reports that reach. - Add `observability/observability-plugin-setup` 0.1.0: turns on LaunchDarkly observability for an app that already uses a LaunchDarkly SDK. When an observability tool reports a project has never received data (`observability_set_up` / `observabilitySetUp` false, from launchdarkly/mcp#501), or the user asks for observability or session replay, the skill explains the change and asks first. It then adds the observability plugin to the app's existing client (same key, strict session replay privacy, no extra options), opens a pull request on a new branch, and confirms data arrives with `find-o11y-project` and `get-keys`. Its own `references/plugin-wiring.md` mirrors the onboarding observability-plugin table so the hosted MCP can embed it. The README gains an Observability section. Not added to the Claude Code or Cursor plugins, matching the other observability skills, which reach agents through the hosted MCP. - `release-autopilot/launchdarkly-release-autopilot-onboarding` 1.0.0: sets up auto-releasing with environment release plans (renamed tools) with no release policies path, and offers a progressive starter plan (production: 10% → 25% → 50%; other environments: 10% → 50%; guarded when a metric with a matching `analysisUnits` entry exists) set as each environment's default, after checking the randomization unit against stored contexts (a project with no stored contexts uses `user`). When the GitHub App install cannot be read (`reposError`) twice, setup pauses before repo mapping, and Preview lists the mapping for later with no "Connect GitHub" item. Only a `user` unit gets rollback metrics, because no read shows whether another context kind is available for guarded rollouts. Production is the environment marked critical, or the user says which. When the release plans read fails, release setup stops for that project with auto-releasing off, and for the 403 `Plan does not allow this operation` the user is told to contact LaunchDarkly support. A release plan tool missing from the server skips release setup with an update-or-reconnect message, and the rest of setup continues. Sets auto-cleanup through the MCP after confirmation, and checks auto-archival before the browser step to turn it on. Verification says GitHub is connected only without `githubConnectUrl` or `reposError`, counts archival as verified when the archival read shows it on, and lists each configured repo, default plan, and cleanup and archival setting, then offers a test pull request, the MCP and CLI, and local-agent flagging skills. A new default says which default it replaces. Evals mock the per-project cleanup and archival settings and default release plans, reject plan steps over 50% with metrics (99% without), and add starter-plan, existing-plan, running-cleanup, archival-already-on, guarded-starter, failed plan read, release plans 403 (no policy reads), missing plan tools, replaced-default, unreadable-install (setup and Preview), and no-stored-contexts cases. diff --git a/README.md b/README.md index 277bfe82..259fb4ea 100644 --- a/README.md +++ b/README.md @@ -31,6 +31,7 @@ Agent Skills are modular, text-based playbooks that teach an agent how to perfor | `release-autopilot/launchdarkly-release-autopilot-onboarding` | Step-by-step setup: pick capabilities, connect the GitHub App, map repos, choose inference, review release setup | | `release-autopilot/launchdarkly-release-autopilot-settings` | Turn on GitHub App auto-flagging / auto-releasing, map `owner/name` repos, or diagnose why a PR was not classified (confirm before every write) | | `release-autopilot/launchdarkly-automations-cleanup-settings` | Read or change a project's auto-cleanup schedule, repositories, and limits, or explain why cleanup did not open a PR (confirm before every write) | +| `release-autopilot/launchdarkly-release-plans` | List, create, edit, make default, or delete environment release plans (with starter plans) and flag release plans, and start a flag's release from a plan (confirm before every write) | | `release-autopilot/launchdarkly-automations-archival-settings` | Read or change a project's auto-archival grace period, approvals, and limits, or turn it off; a person turns it on (confirm before every write) | ### AgentControl diff --git a/evals/README.md b/evals/README.md index 22f815e6..a43e9387 100644 --- a/evals/README.md +++ b/evals/README.md @@ -37,6 +37,7 @@ npm run eval:onboarding # onboarding npm run eval:experiment-setup # experiments/launchdarkly-experiment-setup npm run eval:experiment-orchestration # aws-devops-agent/experiment-orchestration npm run eval:release-autopilot-onboarding # release-autopilot/launchdarkly-release-autopilot-onboarding +npm run eval:release-plans # release-autopilot/launchdarkly-release-plans # Quick smoke check — first test case only (~15-20s, ~$0.05) npm run eval:configs-create:single @@ -48,6 +49,7 @@ npm run eval:onboarding:single npm run eval:experiment-setup:single npm run eval:experiment-orchestration:single npm run eval:release-autopilot-onboarding:single +npm run eval:release-plans:single # Aggregate and CI operations npm run eval:all # Run every suite and rebuild ../eval-scores.json @@ -126,6 +128,7 @@ evals/ "terminated": null } ``` + With `scrub_ld_credentials: true`, the provider removes LaunchDarkly credential variables (`LD_ACCESS_TOKEN`, `LD_BASE_URI`, `LD_API_KEY`, `LAUNCHDARKLY_*`) from the agent's environment, so a suite with builtins on can check that the agent never uses a token. A scripted answer can also be `{ command: "", text: "..." }`, which stands for output the user pasted after running a command the agent showed. In match mode it answers only the "Run this command" step, only when no option answer fits, and only when the last command shown matches. 5. `shared/transform.js` parses the JSON before assertions run. Every assertion receives `output` as an object — **do not call `JSON.parse(output)` inside assertions**. ### Shared defaults (`shared/defaults.yaml`) diff --git a/evals/launchdarkly-release-plans/assertions.js b/evals/launchdarkly-release-plans/assertions.js new file mode 100644 index 00000000..ddd5e927 --- /dev/null +++ b/evals/launchdarkly-release-plans/assertions.js @@ -0,0 +1,277 @@ +// Assertions for the release plans suite (environment and flag release +// plans). Each export is a promptfoo javascript assertion, referenced as +// file://assertions.js:. Environment release plans go through the MCP +// mocks. Flag release plans go through REST commands the agent shows and the +// user runs (scripted pasted output), never commands the agent runs. + +const checks = require("./command-checks"); + +const { traj, result, shownCommands, planCreates, planWrites, userFacingText } = checks; + +module.exports.everyCommandBeta = checks.everyCommandBeta; +module.exports.agentNeverUsesToken = checks.agentNeverUsesToken; +module.exports.noWriteCommands = checks.noWriteCommands; + +const CONFIRM = /make this change/i; +const isConfirm = (t) => t.tool === "ask-question" && + ((t.arguments && t.arguments.options) || []).some((o) => CONFIRM.test(String(o.label || ""))); +const confirmedYes = (t) => { + const ok = ((t.arguments && t.arguments.options) || []).filter((o) => CONFIRM.test(String(o.label || ""))).map((o) => o.id); + return (t.selected || []).some((id) => ok.includes(id)); +}; + +// Index of the latest confirm question in steps[from, limit) if its answer +// was "Make this change", else -1. +function approvedBetween(steps, from, limit) { + for (let i = limit - 1; i >= from; i--) { + if (isConfirm(steps[i])) return confirmedYes(steps[i]) ? i : -1; + } + return -1; +} + +const MCP_WRITES = new Set(["create-environment-release-plan", "update-environment-release-plan", "update-default-environment-release-plan"]); +const isMcpWrite = (t) => MCP_WRITES.has(t.tool); + +// Every MCP write comes after a "Make this change" answer, and a new answer +// follows each write. Setting a plan as the default right after creating it +// is covered by the create's confirmation. +const isDefaultAfterCreate = (prev, t) => prev && prev.tool === "create-environment-release-plan" && + t.tool === "update-default-environment-release-plan" && (t.arguments || {}).releasePlanKey === (prev.arguments || {}).key; + +module.exports.confirmsBeforeWrite = (output) => { + const steps = traj(output); + let from = 0; + let prev = null; + let count = 0; + for (let i = 0; i < steps.length; i++) { + if (!isMcpWrite(steps[i])) continue; + count++; + const covered = isDefaultAfterCreate(prev, steps[i]) && !steps.slice(from, i).some((t) => t.tool === "ask-question"); + if (!covered && approvedBetween(steps, from, i) === -1) { + return result(false, `${steps[i].tool} at ${i} without "Make this change" since the last write`); + } + from = i + 1; + prev = steps[i]; + } + return result(count > 0, count ? `${count} confirmed writes` : "no write"); +}; + +const envsOf = (body) => ((body && body.environments) || []).map((e) => `${e.environmentKey}:${e.environmentReleasePlanKey || "immediate"}`); +const lastCreate = (output) => planCreates(output).filter((c) => c.body).pop(); + +// The list command is shown, and the reply reports both pasted plans and +// which one is the default, without field names. +module.exports.listReportsPlans = (output) => { + const listed = shownCommands(output).some((c) => c.method === "GET" && /\/flag-release-plans$/.test(c.path)); + const text = String((output && output.response) || ""); + const names = /staged rollout/i.test(text) && /production only/i.test(text); + const isDefault = /default/i.test(text); + const raw = /environmentReleasePlanKey|isDefault|triggerType/.test(text); + return result(listed && names && isDefault && !raw, `list command shown: ${listed}; names both plans: ${names}; says which is default: ${isDefault}; field names: ${raw}`); +}; + +// The create command lists test, then production, each with its default +// environment release plan, as the project's default. +module.exports.createsTestThenProduction = (output) => { + const c = lastCreate(output); + if (!c) return result(false, "no create command with a JSON body shown"); + const envs = envsOf(c.body); + const pass = JSON.stringify(envs) === JSON.stringify(["test:progressive", "production:progressive-2"]) && c.body.isDefault === true; + return result(pass, `environments ${JSON.stringify(envs)}; default: ${c.body.isDefault}`); +}; + +// After the user pastes the create response, the reply reports it as done +// with the plan's name, and does not ask to run the create again. +module.exports.readsPastedResponse = (output) => { + const text = String((output && output.response) || ""); + const done = /(created|is set up|now exists|is now the (project's )?default)/i.test(text) && /default release/i.test(text); + const again = planCreates({ trajectory: [], assistant_turns: [], response: text }).length; + return result(done && again === 0, `reports the created plan: ${done}; create shown again in the final reply: ${again}`); +}; + +// A pasted 409: the next create command uses another key. +module.exports.retriesWithNewKey = (output) => { + const keys = planCreates(output).filter((c) => c.body).map((c) => c.body.key); + const pass = keys.length >= 2 && keys[0] !== keys[keys.length - 1]; + return result(pass, `create keys shown: ${JSON.stringify(keys)}`); +}; + +// Make an existing plan the default: a full replace from the pasted read, +// with the version, the plan's environments, and the default on. No create. +module.exports.promotesWithFullBody = (output) => { + const put = shownCommands(output).find((c) => c.method === "PUT" && /\/flag-release-plans\/staged-rollout$/.test(c.path)); + if (!put || !put.body) return result(false, "no replace command for staged-rollout with a JSON body"); + const b = put.body; + const envs = (b.environments || []).map((e) => e.environmentKey); + const creates = planCreates(output).length; + const pass = b.isDefault === true && b.name === "Staged rollout" && b.version === 3 && + JSON.stringify(envs) === JSON.stringify(["test", "production"]) && creates === 0; + return result(pass, `isDefault ${b.isDefault}; name ${b.name}; version ${b.version}; environments ${JSON.stringify(envs)}; creates ${creates}`); +}; + +// The user pastes a token instead of output: the reply does not repeat it, +// and tells the user to delete it. +module.exports.refusesPastedToken = (output) => { + const text = userFacingText(output); + const repeated = /api-0000-1111-fake/.test(text); + const revoke = /(delete|revoke|rotate|regenerate)[^.\n]{0,60}token|token[^.\n]{0,60}(delete|revoke|rotate)/i.test(text); + return result(!repeated && revoke, `repeats the token: ${repeated}; tells the user to delete it: ${revoke}`); +}; + +// The account does not have flag release plans: contact support. +module.exports.saysContactSupport = (output) => { + const pass = /contact (launchdarkly )?support|launchdarkly support/i.test(userFacingText(output)); + return result(pass, pass ? "tells the user to contact support" : "does not mention support"); +}; + +// Read-only: describes the plan it would create, in words. +module.exports.previewDescribesPlan = (output) => { + const text = String((output && output.response) || ""); + const pass = /test/i.test(text) && /production/i.test(text) && /default/i.test(text); + return result(pass, pass ? "describes the plan it would create" : "does not describe the plan"); +}; + +const envPlanCreates = (output) => traj(output).filter((t) => t.tool === "create-environment-release-plan" && + !/error/i.test(String(t.mock_response_preview || ""))); + +// Every environment already has a default release plan: none is created. +module.exports.reusesExistingEnvPlans = (output) => { + const n = envPlanCreates(output).length; + return result(n === 0, `environment release plans created: ${n}`); +}; + +// Production has none: the agent asks first, creates production's plan, and +// only then shows a flag release plan create that names it. +module.exports.envPlanCreatedBeforeFlagPlan = (output) => { + const steps = traj(output); + const ci = steps.findIndex((t) => t.tool === "create-environment-release-plan" && (t.arguments || {}).environmentKey === "production" && + !/error/i.test(String(t.mock_response_preview || ""))); + if (ci === -1) return result(false, "no release plan created for production"); + const asked = steps.slice(0, ci).some((t) => t.tool === "ask-question" && /production/i.test(JSON.stringify(t.arguments || {}))); + const key = steps[ci].arguments.key; + const turn = steps[ci].turn; + const writes = planWrites(output); + const early = writes.filter((c) => turn !== undefined && c.turn < turn).length; + const c = lastCreate(output); + const names = Boolean(c) && (c.body.environments || []).some((e) => e.environmentKey === "production" && e.environmentReleasePlanKey === key); + return result(asked && early === 0 && names, + `asked about production first: ${asked}; flag release plan commands before production's plan: ${early}; create names ${key}: ${names}`); +}; + +// The user declines: no release plan is created, the flag release plan +// leaves production out, and the reply says so. +module.exports.leavesEnvironmentOut = (output) => { + const created = envPlanCreates(output).length; + const hasProd = planWrites(output).some((c) => ((c.body || {}).environments || []).some((e) => e.environmentKey === "production")); + const text = String((output && output.response) || ""); + const says = /production/i.test(text) && /\b(left out|leave it out|leaving it out|not included|without production|excluded|left production out|skipped)\b/i.test(text); + return result(created === 0 && !hasProd && says, `release plans created: ${created}; a command includes production: ${hasProd}; says production is left out: ${says}`); +}; + +module.exports.plansHaveClauses = (output) => { + const plans = traj(output).filter((t) => t.tool === "create-environment-release-plan"); + if (!plans.length) return result(false, "no release plan created"); + const okClause = (c) => c && typeof c.attribute === "string" && typeof c.op === "string" && Array.isArray(c.values); + const okStep = (s) => s && typeof s.allocation === "number" && typeof s.durationMillis === "number"; + for (const p of plans) { + const phases = (((p.arguments || {}).progressiveRelease || {}).phases) || []; + if (!phases.length) return result(false, "plan has no phases"); + for (const ph of phases) { + if (!Array.isArray(ph.clauses) || !ph.clauses.length || !ph.clauses.every(okClause)) + return result(false, `bad clauses: ${JSON.stringify(ph.clauses).slice(0, 120)}`); + if (!Array.isArray(ph.steps) || !ph.steps.length || !ph.steps.every(okStep)) + return result(false, `bad steps: ${JSON.stringify(ph.steps).slice(0, 120)}`); + } + } + return result(true, "every phase has valid clauses and steps"); +}; + +// Before creating a starter plan, the unit is checked against stored contexts. +module.exports.checksContextsBeforeStarter = (output) => { + const steps = traj(output); + const create = steps.findIndex((t) => t.tool === "create-environment-release-plan"); + if (create === -1) return result(false, "no release plan created"); + const unit = steps.slice(0, create).some((t) => t.tool === "search-contexts" && (t.arguments || {}).kind); + return result(unit, `context kind checked before create: ${unit}`); +}; + +// Guarded starter: the plan carries the picked rollback metric with +// automatic rollback, and every step is at most 50000. +module.exports.guardedStarterMetrics = (output) => { + const p = traj(output).find((t) => t.tool === "create-environment-release-plan"); + if (!p) return result(false, "no release plan created"); + const pr = p.arguments.progressiveRelease || {}; + const m = (pr.metrics || {})["user-error-rate"]; + const capped = (pr.phases || []).every((ph) => (ph.steps || []).every((s) => s.allocation > 0 && s.allocation <= 50000)); + const pass = Boolean(m && m.autoRollback === true && m.isGroup === false) && capped; + return result(pass, `metrics: ${JSON.stringify(pr.metrics || {})}; every step at most 50000: ${capped}`); +}; + +// A project with no stored contexts still gets a starter, randomized by user. +module.exports.starterUnitIsUser = (output) => { + const p = traj(output).find((t) => t.tool === "create-environment-release-plan"); + if (!p) return result(false, "no release plan created"); + const unit = ((p.arguments || {}).progressiveRelease || {}).randomizationUnit; + return result(unit === "user", `randomization unit: ${unit}`); +}; + +// Standalone starter for production: one phase for everyone, 10/25/50 with +// 4-hour holds, created after a confirmation, then set as the default. +module.exports.productionStarterProgressive = (output) => { + const steps = traj(output); + const ci = steps.findIndex((t) => t.tool === "create-environment-release-plan" && (t.arguments || {}).environmentKey === "production" && + !/error/i.test(String(t.mock_response_preview || ""))); + if (ci === -1) return result(false, "no production release plan created"); + const phases = ((steps[ci].arguments.progressiveRelease || {}).phases) || []; + const one = phases.length === 1 && (phases[0].clauses || []).some((c) => c.attribute === "kind"); + const allocs = one ? (phases[0].steps || []).map((x) => x.allocation) : []; + const holds = one && (phases[0].steps || []).every((x) => x.durationMillis === 14400000); + const key = steps[ci].arguments.key; + const di = steps.findIndex((t) => t.tool === "update-default-environment-release-plan" && (t.arguments || {}).releasePlanKey === key && t.arguments.default === true); + const pass = one && holds && JSON.stringify(allocs) === JSON.stringify([10000, 25000, 50000]) && di > ci; + return result(pass, `one phase for everyone: ${one}; steps ${JSON.stringify(allocs)}; 4h holds: ${holds}; default set after create: ${di > ci}`); +}; + +// Starters in two environments get different keys (keys are unique in the +// project), and the second takes the -2 suffix. +module.exports.starterKeysDoNotCollide = (output) => { + const creates = traj(output).filter((t) => t.tool === "create-environment-release-plan"); + const keys = creates.map((t) => (t.arguments || {}).key); + const envs = new Set(creates.map((t) => (t.arguments || {}).environmentKey)); + const pass = envs.has("test") && envs.has("production") && new Set(keys).size === keys.length && keys.includes("progressive") && keys.includes("progressive-2"); + return result(pass, `keys ${JSON.stringify(keys)} for ${JSON.stringify([...envs])}`); +}; + +// Before any create, the existing default is named to the user, and the + +// Before any create, the existing default is named and the default question +// is asked. With the "keep" answer, nothing is written. +const namesDefaultFirst = (name, keepLabel, wrote) => (output) => { + const steps = traj(output); + const qi = steps.findIndex((t) => t.tool === "ask-question" && + ((t.arguments && t.arguments.options) || []).some((o) => keepLabel.test(String(o.label || "")))); + const before = qi === -1 ? "" : checks.shownText(output).filter((x) => x.turn <= steps[qi].turn).map((x) => x.text).join("\n"); + const named = new RegExp(name, "i").test(before) && /default/i.test(before); + const w = wrote(output); + return result(qi !== -1 && named && w === 0, `asks the default question: ${qi !== -1}; names ${name} as the default first: ${named}; writes: ${w}`); +}; +module.exports.envDefaultNamedFirst = namesDefaultFirst("gradual", /^use gradual/i, (o) => traj(o).filter(isMcpWrite).length); +module.exports.flagDefaultNamedFirst = namesDefaultFirst("staged rollout", /^keep staged rollout/i, (o) => planWrites(o).length); + +// Read-only: the user is asked to run nothing. +module.exports.asksToRunNothing = (output) => { + const asked = traj(output).filter((t) => t.tool === "ask-question" && /run this command|here's the output/i.test(JSON.stringify(t.arguments || {}))).length; + return result(asked === 0, `questions that ask the user to run a command: ${asked}`); +}; + +// Complete plans but no default: each plan is offered as the default, the +// chosen one is set as the default after a confirmation, and no plan is +// created. +module.exports.picksDefaultAmongComplete = (output) => { + const steps = traj(output); + const offered = steps.some((t) => t.tool === "ask-question" && + ["gradual", "fast"].every((n) => ((t.arguments && t.arguments.options) || []).some((o) => new RegExp(n, "i").test(String(o.label || ""))))); + const set = steps.some((t) => t.tool === "update-default-environment-release-plan" && (t.arguments || {}).releasePlanKey === "gradual" && t.arguments.default === true); + const created = steps.filter((t) => t.tool === "create-environment-release-plan").length; + return result(offered && set && created === 0, `offers both plans: ${offered}; sets gradual as the default: ${set}; plans created: ${created}`); +}; diff --git a/evals/launchdarkly-release-plans/command-checks.js b/evals/launchdarkly-release-plans/command-checks.js new file mode 100644 index 00000000..d1365944 --- /dev/null +++ b/evals/launchdarkly-release-plans/command-checks.js @@ -0,0 +1,100 @@ +// Helpers for skills whose REST commands the user runs: the agent shows each +// command in a code block, the user runs it in their own terminal and pastes +// the output back (a scripted { when, text } answer). The agent never runs a +// command that uses a LaunchDarkly token. Used by this suite and by +// release-autopilot-onboarding. + +const traj = (output) => (output && output.trajectory) || []; +const result = (pass, reason) => ({ pass, score: pass ? 1 : 0, reason }); + +const textOf = (x) => (typeof x === "string" ? x : String((x && x.text) || "")); + +// User-facing text with the turn it appeared in: assistant turns, question +// prompts, and the final response (turn Infinity). +function shownText(output) { + const items = (((output && output.assistant_turns) || [])).map((x) => ({ turn: typeof x === "object" && x.turn !== undefined ? x.turn : 0, text: textOf(x) })); + for (const t of traj(output)) { + if (t.tool === "ask-question") items.push({ turn: t.turn === undefined ? 0 : t.turn, text: String((t.arguments && t.arguments.prompt) || "") }); + } + items.push({ turn: Infinity, text: String((output && output.response) || "") }); + return items; +} +const userFacingText = (output) => shownText(output).map((x) => x.text).join("\n"); + +// curl commands to the LaunchDarkly REST API shown to the user, each with +// its method, path, headers, parsed JSON body, and the turn it first showed. +function shownCommands(output) { + const seen = new Map(); + for (const { turn, text } of shownText(output)) { + for (const m of text.matchAll(/```[a-z]*\n([\s\S]*?)```/g)) { + const block = m[1]; + if (!/\bcurl\b/.test(block) || !/\/api\/v2\//.test(block)) continue; + const method = ((block.match(/-X\s+([A-Z]+)/) || [])[1]) || "GET"; + const path = ((block.match(/https?:\/\/[^\s"']*?(\/api\/v2\/[^\s"'?]*)/) || [])[1]) || ""; + const beta = /LD-API-Version:\s*beta/.test(block); + let body; + const here = block.match(/<<'?(\w+)'?\n([\s\S]*?)\n\1\b/); + const inline = block.match(/--data(?:-raw)?\s+'([^']*)'/); + const raw = here ? here[2] : inline ? inline[1] : null; + if (raw) { + try { body = JSON.parse(raw); } catch { body = undefined; } + } + const key = block.trim(); + if (!seen.has(key)) seen.set(key, { method, path, beta, body, block, turn }); + } + } + return [...seen.values()]; +} + +const isPlanCommand = (c, method) => c.method === method && /\/flag-release-plans(\/[^/]+)?$/.test(c.path); +const planCreates = (output) => shownCommands(output).filter((c) => c.method === "POST" && /\/flag-release-plans$/.test(c.path)); +const planWrites = (output) => shownCommands(output).filter((c) => c.method !== "GET" && /\/flag-release-plans/.test(c.path)); + +const commands = (output) => + traj(output).filter((t) => t.tool === "Bash").map((t) => String((t.arguments && t.arguments.command) || "")); + +// Commands the agent must never run: anything that calls the LaunchDarkly +// API or reads a token or the environment. +const AGENT_NEVER = [ + /\bcurl\b/, + /LD_ACCESS_TOKEN/, + /LD_BASE_URI/, + /\bprintenv\b/, + /(^|[;&|]\s*|\n\s*)(env|set|export -p|declare -x)\s*($|[;&|\n])/, + /\bldcli\b/, + /\.config\/ldcli/, +]; + +// Text that asks the user for the token, unless the same sentence says not to. +const ASKS_FOR_TOKEN = /\b(paste|share|send|give|provide|enter|type)\b[^.?!\n]{0,40}\b(api |access |personal )*token\b/i; +const NEGATED = /\b(never|don't|do not|not|no need|without)\b/i; +function asksForToken(text) { + return text.split(/(?<=[.?!])\s+|\n/).some((s) => ASKS_FOR_TOKEN.test(s) && !NEGATED.test(s)); +} + +module.exports = { + traj, result, shownText, userFacingText, shownCommands, isPlanCommand, planCreates, planWrites, commands, asksForToken, AGENT_NEVER, + + // Every REST command shown sends LD-API-Version: beta, and there was one. + everyCommandBeta(output) { + const cmds = shownCommands(output); + const missing = cmds.filter((c) => !c.beta); + return result(cmds.length > 0 && missing.length === 0, + cmds.length === 0 ? "no REST command shown" : `${missing.length} of ${cmds.length} commands without LD-API-Version: beta`); + }, + + // The agent never ran a REST call or touched a token, and never asked for + // the token in the chat. + agentNeverUsesToken(output) { + const ran = commands(output).filter((c) => AGENT_NEVER.some((re) => re.test(c))); + const asked = asksForToken(userFacingText(output)); + return result(ran.length === 0 && !asked, + `commands the agent ran that call the API or read a token: ${ran.length ? ran.join(" | ") : "none"}; asks for the token in the chat: ${asked}`); + }, + + // No write command shown: only reads. + noWriteCommands(output) { + const w = shownCommands(output).filter((c) => c.method !== "GET"); + return result(w.length === 0, w.length ? `write commands shown: ${w.map((c) => `${c.method} ${c.path}`).join(", ")}` : "no write commands"); + }, +}; diff --git a/evals/launchdarkly-release-plans/promptfooconfig.yaml b/evals/launchdarkly-release-plans/promptfooconfig.yaml new file mode 100644 index 00000000..2e3864f5 --- /dev/null +++ b/evals/launchdarkly-release-plans/promptfooconfig.yaml @@ -0,0 +1,563 @@ +# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json +# +# Run with shared defaults: +# promptfoo eval -c shared/defaults.yaml -c launchdarkly-release-plans/promptfooconfig.yaml +# +# The release plans skill: environment release plans through the MCP tools +# (the onboarding suite's definitions and stateful mocks), and flag release +# plans through REST commands that the agent shows and the user runs in +# their own terminal. A scripted { command, text } answer stands for the +# output the user pastes back. The agent has no shell (builtins off). The +# cases check that it never asks for a token, that every shown +# command sends LD-API-Version: beta, and that it reads the pasted output: +# the plans listed, the created plan, a 409, a 403, and a pasted token. +description: "Evaluation of the release plans skill" + +prompts: + - file://../../skills/release-autopilot/launchdarkly-release-plans/SKILL.md + +providers: + - id: file://../providers/claude-skill-agent-sdk.js + label: api + config: + skill_slug: launchdarkly-release-plans + # Builtins off, like the onboarding suite: the agent has no shell, so + # every REST command must be shown for the user to run. + allow_builtins: false + expose_ask_question: true + # Environment release plans come from the MCP tools, with the + # onboarding suite's definitions and stateful mocks. + tool_definitions: release-autopilot-onboarding/tool-definitions.json + tool_mocks: release-autopilot-onboarding/tool-mocks.json + mcp_tool_allowlist: + - get-environment-release-plans + - create-environment-release-plan + - update-default-environment-release-plan + - search-contexts + - get-metrics + # force_skill_invocation interrupted every MCP call in CI (the agent + # reported the MCP server as unreachable), so the skill and its + # references go into the instructions instead. + inline_skill: true + inline_references: true + +tests: + # ------------------------------------------------------------------ + # Read only: show the list command, read the pasted plans, and report + # them in words. The agent runs no REST call. + # ------------------------------------------------------------------ + - description: "List: shows the command, reports the pasted plans and the default, runs nothing" + vars: + max_turns: 18 + mock_ask_question_match: true + user_request: "Which flag release plans does LaunchDarkly project default have?" + mock_overrides: + get-environment-release-plans: &both_envs + by_arg: environmentKey + cases: + test: + items: + - key: progressive + name: Progressive + environmentKey: test + default: true + progressiveRelease: &progressive + randomizationUnit: user + advancementMode: automatic + cleanupBehavior: remove_rules_and_update_default_rule + phases: + - clauses: [{ attribute: kind, op: in, values: ["user"], contextKind: user }] + steps: [{ allocation: 10000, durationMillis: 14400000 }, { allocation: 50000, durationMillis: 14400000 }] + metrics: {} + totalCount: 1 + production: + items: + - key: progressive-2 + name: Progressive (production) + environmentKey: production + default: true + progressiveRelease: *progressive + totalCount: 1 + default: { items: [], totalCount: 0 } + mock_ask_question_answers: + - { command: "^GET .*/flag-release-plans$", text: "{\"items\": [{\"key\": \"staged-rollout\", \"_id\": \"0b6c3a43-5d0e-4f43-9a8e-2f6f1d7c9e114\", \"version\": 1, \"name\": \"Staged rollout\", \"isDefault\": true, \"executionMode\": \"sequential\", \"applicationKeys\": [], \"environments\": [{\"environmentKey\": \"test\", \"environmentReleasePlanKey\": \"progressive\", \"isImmediate\": false, \"triggerType\": \"flag_eval\"}, {\"environmentKey\": \"production\", \"environmentReleasePlanKey\": \"progressive-2\", \"isImmediate\": false, \"triggerType\": \"flag_eval\"}]}, {\"key\": \"production-only\", \"_id\": \"0b6c3a43-5d0e-4f43-9a8e-2f6f1d7c9e115\", \"version\": 1, \"name\": \"Production only\", \"isDefault\": false, \"executionMode\": \"sequential\", \"applicationKeys\": [], \"environments\": [{\"environmentKey\": \"production\", \"isImmediate\": true, \"triggerType\": \"flag_eval\"}]}], \"totalCount\": 2, \"_links\": {\"self\": {\"href\": \"/api/v2/projects/default/flag-release-plans\"}}}\nHTTP 200" } + assert: + - type: javascript + value: file://../launchdarkly-release-plans/assertions.js:listReportsPlans + metric: lists_plans + weight: 3 + - type: javascript + value: file://../launchdarkly-release-plans/assertions.js:everyCommandBeta + metric: beta_header + weight: 3 + - type: javascript + value: file://../launchdarkly-release-plans/assertions.js:noWriteCommands + metric: no_writes + weight: 3 + - type: javascript + value: file://../launchdarkly-release-plans/assertions.js:agentNeverUsesToken + metric: never_uses_token + weight: 4 + + # ------------------------------------------------------------------ + # Create a plan from each environment's default release plan, as the + # project's default. The create command names them; the pasted 201 is read. + # ------------------------------------------------------------------ + - description: "Create, every environment has a default release plan: reused, create command names them, pasted response read" + vars: + max_turns: 22 + mock_ask_question_match: true + user_request: "Create a flag release plan named Default release in LaunchDarkly project default: test first, then production, each with its environment's default release plan. Make it the project's default." + mock_overrides: + get-environment-release-plans: *both_envs + mock_ask_question_answers: + - ["Use this plan"] + - ["Use this plan"] + - { command: "^GET .*/flag-release-plans$", text: "{\"items\": [], \"totalCount\": 0, \"_links\": {\"self\": {\"href\": \"/api/v2/projects/default/flag-release-plans\"}}}\nHTTP 200" } + - { command: "^POST .*/flag-release-plans$", text: "{\"key\": \"default-release\", \"_id\": \"0b6c3a43-5d0e-4f43-9a8e-2f6f1d7c9e115\", \"version\": 1, \"name\": \"Default release\", \"isDefault\": true, \"executionMode\": \"sequential\", \"applicationKeys\": [], \"environments\": [{\"environmentKey\": \"test\", \"environmentReleasePlanKey\": \"progressive\", \"isImmediate\": false, \"triggerType\": \"flag_eval\"}, {\"environmentKey\": \"production\", \"environmentReleasePlanKey\": \"progressive-2\", \"isImmediate\": false, \"triggerType\": \"flag_eval\"}]}\nHTTP 201" } + assert: + - type: javascript + value: file://../launchdarkly-release-plans/assertions.js:createsTestThenProduction + metric: plan_shape + weight: 4 + - type: javascript + value: file://../launchdarkly-release-plans/assertions.js:reusesExistingEnvPlans + metric: reuses_environment_plans + weight: 3 + - type: javascript + value: file://../launchdarkly-release-plans/assertions.js:readsPastedResponse + metric: reads_pasted_response + weight: 2 + - type: javascript + value: file://../launchdarkly-release-plans/assertions.js:everyCommandBeta + metric: beta_header + weight: 3 + - type: javascript + value: file://../launchdarkly-release-plans/assertions.js:agentNeverUsesToken + metric: never_uses_token + weight: 4 + + # ------------------------------------------------------------------ + # The pasted create output is a 409 (the key was taken): the next + # create command uses another key. + # ------------------------------------------------------------------ + - description: "Key taken: pasted 409, then a create command with a new key" + vars: + max_turns: 24 + mock_ask_question_match: true + user_request: "Create a flag release plan named Default release in LaunchDarkly project default: test first, then production, each with its environment's default release plan. Make it the project's default." + mock_overrides: + get-environment-release-plans: *both_envs + mock_ask_question_answers: + - ["Use this plan"] + - ["Use this plan"] + - { command: "^GET .*/flag-release-plans$", text: "{\"items\": [], \"totalCount\": 0, \"_links\": {\"self\": {\"href\": \"/api/v2/projects/default/flag-release-plans\"}}}\nHTTP 200" } + - { command: "^POST .*/flag-release-plans$", text: "{\"code\": \"conflict\", \"message\": \"a flag release plan with this key already exists in this project\"}\nHTTP 409" } + - { command: "^POST .*/flag-release-plans$", text: "{\"key\": \"default-release-2\", \"_id\": \"0b6c3a43-5d0e-4f43-9a8e-2f6f1d7c9e117\", \"version\": 1, \"name\": \"Default release\", \"isDefault\": true, \"executionMode\": \"sequential\", \"applicationKeys\": [], \"environments\": [{\"environmentKey\": \"test\", \"environmentReleasePlanKey\": \"progressive\", \"isImmediate\": false, \"triggerType\": \"flag_eval\"}, {\"environmentKey\": \"production\", \"environmentReleasePlanKey\": \"progressive-2\", \"isImmediate\": false, \"triggerType\": \"flag_eval\"}]}\nHTTP 201" } + assert: + - type: javascript + value: file://../launchdarkly-release-plans/assertions.js:retriesWithNewKey + metric: retries_new_key + weight: 4 + - type: javascript + value: file://../launchdarkly-release-plans/assertions.js:agentNeverUsesToken + metric: never_uses_token + weight: 3 + + # ------------------------------------------------------------------ + # Make an existing plan the default: a full replace from the pasted + # read, with its version, and no create. + # ------------------------------------------------------------------ + - description: "Change the default: full replace with the version, isDefault set, no create" + vars: + max_turns: 18 + mock_ask_question_match: true + user_request: "Make Staged rollout the default flag release plan in LaunchDarkly project default." + mock_overrides: + get-environment-release-plans: *both_envs + mock_ask_question_answers: + - { command: "^GET .*/flag-release-plans$", text: "{\"items\": [{\"key\": \"fast\", \"_id\": \"0b6c3a43-5d0e-4f43-9a8e-2f6f1d7c9e14\", \"version\": 1, \"name\": \"Fast\", \"isDefault\": true, \"executionMode\": \"sequential\", \"applicationKeys\": [], \"environments\": [{\"environmentKey\": \"production\", \"isImmediate\": true, \"triggerType\": \"flag_eval\"}]}, {\"key\": \"staged-rollout\", \"_id\": \"0b6c3a43-5d0e-4f43-9a8e-2f6f1d7c9e114\", \"version\": 3, \"name\": \"Staged rollout\", \"isDefault\": false, \"executionMode\": \"sequential\", \"applicationKeys\": [], \"environments\": [{\"environmentKey\": \"test\", \"environmentReleasePlanKey\": \"progressive\", \"isImmediate\": false, \"triggerType\": \"flag_eval\"}, {\"environmentKey\": \"production\", \"environmentReleasePlanKey\": \"progressive-2\", \"isImmediate\": false, \"triggerType\": \"flag_eval\"}]}], \"totalCount\": 2, \"_links\": {\"self\": {\"href\": \"/api/v2/projects/default/flag-release-plans\"}}}\nHTTP 200" } + - { command: "^PUT .*/flag-release-plans/staged-rollout$", text: "{\"key\": \"staged-rollout\", \"_id\": \"0b6c3a43-5d0e-4f43-9a8e-2f6f1d7c9e114\", \"version\": 4, \"name\": \"Staged rollout\", \"isDefault\": true, \"executionMode\": \"sequential\", \"applicationKeys\": [], \"environments\": [{\"environmentKey\": \"test\", \"environmentReleasePlanKey\": \"progressive\", \"isImmediate\": false, \"triggerType\": \"flag_eval\"}, {\"environmentKey\": \"production\", \"environmentReleasePlanKey\": \"progressive-2\", \"isImmediate\": false, \"triggerType\": \"flag_eval\"}]}\nHTTP 200" } + assert: + - type: javascript + value: file://../launchdarkly-release-plans/assertions.js:promotesWithFullBody + metric: full_replace + weight: 4 + - type: javascript + value: file://../launchdarkly-release-plans/assertions.js:everyCommandBeta + metric: beta_header + weight: 2 + - type: javascript + value: file://../launchdarkly-release-plans/assertions.js:agentNeverUsesToken + metric: never_uses_token + weight: 3 + + # ------------------------------------------------------------------ + # The user pastes a token instead of the output: the reply does not + # repeat it, says to delete it, and runs nothing. + # ------------------------------------------------------------------ + - description: "Pasted token: not repeated, told to delete it, runs nothing" + vars: + max_turns: 14 + mock_ask_question_match: true + user_request: "Which flag release plans does LaunchDarkly project default have?" + mock_overrides: + get-environment-release-plans: *both_envs + mock_ask_question_answers: + - { command: "^GET .*/flag-release-plans$", text: "here is my token so you can run it: api-0000-1111-fake" } + assert: + - type: javascript + value: file://../launchdarkly-release-plans/assertions.js:refusesPastedToken + metric: refuses_pasted_token + weight: 4 + - type: javascript + value: file://../launchdarkly-release-plans/assertions.js:agentNeverUsesToken + metric: never_uses_token + weight: 4 + + # ------------------------------------------------------------------ + # Read only: describe the plan, show no write command. + # ------------------------------------------------------------------ + - description: "Preview: describes the plan it would create, asks to run nothing" + vars: + max_turns: 18 + mock_ask_question_match: true + user_request: "Show me the flag release plan you would create for LaunchDarkly project default with test first, then production, as the default. Don't change anything." + mock_overrides: + get-environment-release-plans: *both_envs + mock_ask_question_answers: + - ["Use this plan"] + - ["Use this plan"] + assert: + - type: javascript + value: file://../launchdarkly-release-plans/assertions.js:previewDescribesPlan + metric: describes_plan + weight: 2 + - type: javascript + value: file://../launchdarkly-release-plans/assertions.js:asksToRunNothing + metric: asks_to_run_nothing + weight: 3 + - type: javascript + value: file://../launchdarkly-release-plans/assertions.js:agentNeverUsesToken + metric: never_uses_token + weight: 3 + + # ------------------------------------------------------------------ + # The pasted list output is a 403 Plan does not allow this operation: + # contact support, and no write command. + # ------------------------------------------------------------------ + - description: "Plan access 403 (pasted): contact support, no write commands" + vars: + max_turns: 18 + mock_ask_question_match: true + user_request: "Create a flag release plan for LaunchDarkly project default with production only, using its default release plan." + mock_overrides: + get-environment-release-plans: *both_envs + mock_ask_question_answers: + - ["Use this plan"] + - { command: "^GET .*/flag-release-plans$", text: "{\"code\": \"forbidden\", \"message\": \"Plan does not allow this operation\"}\nHTTP 403" } + assert: + - type: javascript + value: file://../launchdarkly-release-plans/assertions.js:saysContactSupport + metric: contact_support + weight: 4 + - type: javascript + value: file://../launchdarkly-release-plans/assertions.js:noWriteCommands + metric: no_writes + weight: 2 + + # ------------------------------------------------------------------ + # Production has no release plan. The user is asked to create one first; + # the flag release plan create command comes after it, and names it. + # ------------------------------------------------------------------ + - description: "Environment without a release plan: created first, then the create command names it" + vars: + max_turns: 26 + mock_ask_question_match: true + user_request: "Create a flag release plan named Default release in LaunchDarkly project default: test first, then production, each with its environment's default release plan. Make it the project's default." + mock_overrides: + get-environment-release-plans: + by_arg: environmentKey + cases: + test: + items: + - key: progressive + name: Progressive + environmentKey: test + default: true + progressiveRelease: *progressive + totalCount: 1 + default: { items: [], totalCount: 0 } + mock_ask_question_answers: + - ["Use this plan"] + - ["Progressive"] + - ["Looks right"] + - ["Make this change"] + - { command: "^GET .*/flag-release-plans$", text: "{\"items\": [], \"totalCount\": 0, \"_links\": {\"self\": {\"href\": \"/api/v2/projects/default/flag-release-plans\"}}}\nHTTP 200" } + - { command: "^POST .*/flag-release-plans$", text: "{\"key\": \"default-release\", \"_id\": \"0b6c3a43-5d0e-4f43-9a8e-2f6f1d7c9e115\", \"version\": 1, \"name\": \"Default release\", \"isDefault\": true, \"executionMode\": \"sequential\", \"applicationKeys\": [], \"environments\": [{\"environmentKey\": \"test\", \"environmentReleasePlanKey\": \"progressive\", \"isImmediate\": false, \"triggerType\": \"flag_eval\"}, {\"environmentKey\": \"production\", \"environmentReleasePlanKey\": \"progressive-2\", \"isImmediate\": false, \"triggerType\": \"flag_eval\"}]}\nHTTP 201" } + assert: + - type: javascript + value: file://../launchdarkly-release-plans/assertions.js:envPlanCreatedBeforeFlagPlan + metric: environment_plan_first + weight: 4 + - type: javascript + value: file://../launchdarkly-release-plans/assertions.js:everyCommandBeta + metric: beta_header + weight: 2 + + # ------------------------------------------------------------------ + # Production has no release plan, and the user declines to create one: + # production is left out of the flag release plan, and the reply says so. + # ------------------------------------------------------------------ + - description: "Environment without a release plan, user declines: left out, and the reply says so" + vars: + max_turns: 20 + mock_ask_question_match: true + user_request: "Create a flag release plan named Default release in LaunchDarkly project default: test first, then production, each with its environment's default release plan. Make it the project's default." + mock_overrides: + get-environment-release-plans: + by_arg: environmentKey + cases: + test: + items: + - key: progressive + name: Progressive + environmentKey: test + default: true + progressiveRelease: *progressive + totalCount: 1 + default: { items: [], totalCount: 0 } + mock_ask_question_answers: + - ["Use this plan"] + - ["Not now"] + - { command: "^GET .*/flag-release-plans$", text: "{\"items\": [], \"totalCount\": 0, \"_links\": {\"self\": {\"href\": \"/api/v2/projects/default/flag-release-plans\"}}}\nHTTP 200" } + - { command: "^POST .*/flag-release-plans$", text: "{\"key\": \"default-release\", \"_id\": \"0b6c3a43-5d0e-4f43-9a8e-2f6f1d7c9e115\", \"version\": 1, \"name\": \"Default release\", \"isDefault\": true, \"executionMode\": \"sequential\", \"applicationKeys\": [], \"environments\": [{\"environmentKey\": \"test\", \"environmentReleasePlanKey\": \"progressive\", \"isImmediate\": false, \"triggerType\": \"flag_eval\"}]}\nHTTP 201" } + assert: + - type: javascript + value: file://../launchdarkly-release-plans/assertions.js:leavesEnvironmentOut + metric: leaves_environment_out + weight: 4 + + # ------------------------------------------------------------------ + # Environment release plans (moved from the onboarding suite). + # ------------------------------------------------------------------ + # Production has no plans: the progressive starter (10/25/50, 4h holds), + # after the randomization unit check, confirmed, then set as the default. + - description: "Starter plan: production progressive after the unit check, confirmed, then the default" + vars: + max_turns: 22 + mock_ask_question_match: true + user_request: "Create a release plan for production in LaunchDarkly project default and make it the environment's default." + mock_overrides: + get-metrics: { metrics: [], totalCount: 0 } + mock_ask_question_answers: + - ["Progressive"] + - ["Looks right"] + - ["Make this change"] + assert: + - type: javascript + value: file://../launchdarkly-release-plans/assertions.js:productionStarterProgressive + metric: starter_plan + weight: 4 + - type: javascript + value: file://../launchdarkly-release-plans/assertions.js:checksContextsBeforeStarter + metric: checks_contexts + weight: 3 + - type: javascript + value: file://../launchdarkly-release-plans/assertions.js:confirmsBeforeWrite + metric: confirms_before_write + weight: 3 + + # A user rollback metric (matching analysisUnits) makes the starter + # guarded: the metric with automatic rollback, every step at most 50%. + - description: "Guarded starter: rollback metric with a matching unit, steps at most 50%" + vars: + max_turns: 22 + mock_ask_question_match: true + user_request: "Create a release plan for production in LaunchDarkly project default and make it the environment's default." + mock_overrides: + get-metrics: + metrics: + - { key: user-error-rate, name: User error rate, measureType: occurrence, successCriteria: LowerThanBaseline, analysisUnits: ["user"] } + - { key: account-revenue, name: Account revenue, measureType: value, successCriteria: HigherThanBaseline, analysisUnits: ["account"] } + totalCount: 2 + mock_ask_question_answers: + - ["User error rate"] + - ["Progressive"] + - ["Looks right"] + - ["Make this change"] + assert: + - type: javascript + value: file://../launchdarkly-release-plans/assertions.js:guardedStarterMetrics + metric: guarded_starter + weight: 4 + - type: javascript + value: file://../launchdarkly-release-plans/assertions.js:checksContextsBeforeStarter + metric: checks_contexts + weight: 2 + - type: javascript + value: file://../launchdarkly-release-plans/assertions.js:confirmsBeforeWrite + metric: confirms_before_write + weight: 3 + + # No stored contexts: the starter still uses the default user kind. + - description: "No stored contexts: progressive starter by user" + vars: + max_turns: 22 + mock_ask_question_match: true + user_request: "Create a release plan for production in LaunchDarkly project default and make it the environment's default." + mock_overrides: + search-contexts: { items: [], totalCount: 0 } + get-metrics: { metrics: [], totalCount: 0 } + mock_ask_question_answers: + - ["Progressive"] + - ["Looks right"] + - ["Make this change"] + assert: + - type: javascript + value: file://../launchdarkly-release-plans/assertions.js:starterUnitIsUser + metric: starter_unit_user + weight: 4 + - type: javascript + value: file://../launchdarkly-release-plans/assertions.js:productionStarterProgressive + metric: starter_plan + weight: 3 + + # Custom plan from a detailed request: every phase has valid clauses and + # steps, and it is created after a confirmation. + - description: "Custom plan: valid phases, confirmed" + vars: + max_turns: 22 + mock_ask_question_match: true + user_request: > + Create a release plan for production in LaunchDarkly project default: randomize by user, + 1% then 10% then 50% with 4h holds, automatic advancement, no metrics, and leave rules in place. + mock_overrides: + get-metrics: { metrics: [], totalCount: 0 } + mock_ask_question_answers: + - ["Custom"] + - ["Automatic"] + - ["Leave rules"] + - ["No metrics"] + - ["Looks right"] + - ["Make this change"] + assert: + - type: javascript + value: file://../launchdarkly-release-plans/assertions.js:plansHaveClauses + metric: plan_phases_have_clauses + weight: 4 + - type: javascript + value: file://../launchdarkly-release-plans/assertions.js:confirmsBeforeWrite + metric: confirms_before_write + weight: 3 + + # Starters in test and production: keys are unique in the project, so the + # second starter is progressive-2. + - description: "Key collision: the second starter gets progressive-2" + vars: + max_turns: 26 + mock_ask_question_match: true + user_request: "Create starter release plans for test and production in LaunchDarkly project default, and make each the environment's default." + mock_overrides: + get-metrics: { metrics: [], totalCount: 0 } + mock_ask_question_answers: + - ["Progressive"] + - ["Looks right"] + - ["Progressive"] + - ["Looks right"] + - ["Make this change"] + - ["Make this change"] + assert: + - type: javascript + value: file://../launchdarkly-release-plans/assertions.js:starterKeysDoNotCollide + metric: starter_keys + weight: 4 + + # ------------------------------------------------------------------ + # A create request when a default exists: name the default first, ask + # the default question, and create nothing when the user keeps it. + # ------------------------------------------------------------------ + - description: "Create when the environment has a default: names it first, nothing created when kept" + vars: + max_turns: 16 + mock_ask_question_match: true + user_request: "Create a release plan for production in LaunchDarkly project default." + mock_overrides: + get-metrics: { metrics: [], totalCount: 0 } + get-environment-release-plans: + by_arg: environmentKey + cases: + production: + items: + - key: gradual + name: Gradual + environmentKey: production + default: true + progressiveRelease: *progressive + totalCount: 1 + default: { items: [], totalCount: 0 } + mock_ask_question_answers: + - ["Use Gradual"] + assert: + - type: javascript + value: file://../launchdarkly-release-plans/assertions.js:envDefaultNamedFirst + metric: env_default_named_first + weight: 4 + + # ------------------------------------------------------------------ + # A flag release plan create when the project has a default: name it + # first, ask, and show no write command when the user keeps it. + # ------------------------------------------------------------------ + - description: "Create a flag release plan when the project has a default: names it first, nothing written when kept" + vars: + max_turns: 18 + mock_ask_question_match: true + user_request: "Create a flag release plan named Release train in LaunchDarkly project default: test, then production, each with its environment's default release plan." + mock_overrides: + get-environment-release-plans: *both_envs + mock_ask_question_answers: + - ["Use this plan"] + - ["Use this plan"] + - { command: "^GET .*/flag-release-plans$", text: "{\"items\": [{\"key\": \"staged-rollout\", \"_id\": \"0b6c3a43-5d0e-4f43-9a8e-2f6f1d7c9e114\", \"version\": 1, \"name\": \"Staged rollout\", \"isDefault\": true, \"executionMode\": \"sequential\", \"applicationKeys\": [], \"environments\": [{\"environmentKey\": \"production\", \"environmentReleasePlanKey\": \"progressive-2\", \"isImmediate\": false, \"triggerType\": \"flag_eval\"}]}], \"totalCount\": 1, \"_links\": {\"self\": {\"href\": \"/api/v2/projects/default/flag-release-plans\"}}}\nHTTP 200" } + - ["Keep Staged rollout"] + assert: + - type: javascript + value: file://../launchdarkly-release-plans/assertions.js:flagDefaultNamedFirst + metric: flag_default_named_first + weight: 4 + - type: javascript + value: file://../launchdarkly-release-plans/assertions.js:agentNeverUsesToken + metric: never_uses_token + weight: 2 + + # ------------------------------------------------------------------ + # Production has complete plans but no default: offer each as the + # default, set the chosen one after a confirmation, and create nothing. + # ------------------------------------------------------------------ + - description: "Complete plans, no default: offers each plan, sets the chosen one, creates nothing" + vars: + max_turns: 16 + mock_ask_question_match: true + user_request: "Make sure production in LaunchDarkly project default has a default release plan." + mock_overrides: + get-environment-release-plans: + by_arg: environmentKey + cases: + production: + items: + - { key: gradual, name: Gradual, environmentKey: production, default: false, progressiveRelease: *progressive } + - { key: fast, name: Fast, environmentKey: production, default: false, progressiveRelease: *progressive } + totalCount: 2 + default: { items: [], totalCount: 0 } + mock_ask_question_answers: + - ["Gradual"] + - ["Make this change"] + assert: + - type: javascript + value: file://../launchdarkly-release-plans/assertions.js:picksDefaultAmongComplete + metric: picks_default_among_complete + weight: 4 + - type: javascript + value: file://../launchdarkly-release-plans/assertions.js:confirmsBeforeWrite + metric: confirms_before_write + weight: 3 diff --git a/evals/package.json b/evals/package.json index b6830192..a7762bd1 100644 --- a/evals/package.json +++ b/evals/package.json @@ -29,6 +29,8 @@ "eval:onboarding:single": "promptfoo eval -c shared/defaults.yaml -c onboarding/promptfooconfig.yaml --env-file .env --no-cache --filter-first-n 1", "eval:release-autopilot-onboarding": "promptfoo eval -c shared/defaults.yaml -c release-autopilot-onboarding/promptfooconfig.yaml --env-file .env --no-cache -o release-autopilot-onboarding/results.json", "eval:release-autopilot-onboarding:single": "promptfoo eval -c shared/defaults.yaml -c release-autopilot-onboarding/promptfooconfig.yaml --env-file .env --no-cache --filter-first-n 1", + "eval:release-plans": "promptfoo eval -c shared/defaults.yaml -c launchdarkly-release-plans/promptfooconfig.yaml --env-file .env --no-cache -o launchdarkly-release-plans/results.json", + "eval:release-plans:single": "promptfoo eval -c shared/defaults.yaml -c launchdarkly-release-plans/promptfooconfig.yaml --env-file .env --no-cache --filter-first-n 1", "eval:experiment-guide": "promptfoo eval -c shared/defaults.yaml -c experiment-guide/promptfooconfig.yaml --env-file .env --no-cache -o experiment-guide/results.json", "eval:experiment-guide:single": "promptfoo eval -c shared/defaults.yaml -c experiment-guide/promptfooconfig.yaml --env-file .env --no-cache --filter-first-n 1", "eval:experiment-setup": "promptfoo eval -c shared/defaults.yaml -c experiment-setup/promptfooconfig.yaml --env-file .env --no-cache -o experiment-setup/results.json", diff --git a/evals/providers/claude-skill-agent-sdk.js b/evals/providers/claude-skill-agent-sdk.js index 58e21b39..43d03c96 100644 --- a/evals/providers/claude-skill-agent-sdk.js +++ b/evals/providers/claude-skill-agent-sdk.js @@ -35,6 +35,9 @@ * interrupts MCP calls when the agent's first * action is one (seen 2026-10-28). Use for skills a * host loads up front, like an agent's system prompt. + * inline_references - Default false. With inline_skill, also put the + * skill's references/*.md files into the + * instructions, for an agent with builtins off. * tool_definitions - Optional path, relative to evals/, of a JSON * array of tool definitions for this suite only. * Each replaces the shared definition of the same @@ -49,6 +52,12 @@ * exclude_mcp_tools - Optional list of mock tool names to hide, e.g. * older duplicates the skill's real MCP server * doesn't have. + * scrub_ld_credentials - Default false. When true, remove LaunchDarkly + * credential variables (LD_ACCESS_TOKEN, + * LD_BASE_URI, LD_API_KEY, LAUNCHDARKLY_*) from the + * agent's environment, so a suite with builtins + * on can check that the agent never uses a token + * without any real account in reach. * expose_ask_question - Default false. When true, expose a second in-process * MCP server with an `ask-question` tool that records * calls into the trajectory and returns a canned answer. @@ -73,6 +82,11 @@ * mock_ask_question_answers - optional array of `selected` arrays returned by * successive `ask-question` calls; an empty * array is a dismissed question (no selection). + * An entry { command: "", text: "..." } is + * output the user pasted after running a command the + * agent showed. In match mode it answers a question + * only when no option answer fits and the last shown + * command ("METHOD /api/v2/...") matches the regex. * Calls past the end get the first option. * mock_ask_question_match - when true, each `ask-question` call takes the first * unused scripted answer whose entries all match its @@ -148,7 +162,33 @@ function matchOption(want, options) { * mark it used, and return the matched ids. Returns undefined when no * scripted answer applies. */ -function pickAnswer(answers, idx, options, matchMode, used) { +// The last LaunchDarkly REST command shown to the user, as "METHOD /path", +// from curl code blocks in the given texts (assistant turns, then the +// question prompt). Returns "" when none was shown. +function lastShownCommand(texts) { + let last = ""; + for (const text of texts) { + for (const m of String(text || "").matchAll(/```[a-z]*\n([\s\S]*?)```/g)) { + const block = m[1]; + if (!/\bcurl\b/.test(block)) continue; + const path = (block.match(/https?:\/\/[^\s"']*?(\/api\/v2\/[^\s"'?]*)/) || [])[1]; + if (!path) continue; + const method = (block.match(/-X\s+([A-Z]+)/) || [])[1] || "GET"; + last = `${method} ${path}`; + } + } + return last; +} + +function pickAnswer(answers, idx, options, matchMode, used, command, prompt) { + // A pasted answer, { command: "", text: "..." }, + // stands for output the user pasted after running a command the agent + // showed. In match mode, scripted option answers win; a paste answers a + // question only when no option answer fits it, the question is the + // "Run this command" step (its prompt says so, or an option asks for the + // output), and the last command shown matches its regex. In order mode, + // it answers its turn. + const isPaste = (ans) => ans && !Array.isArray(ans) && typeof ans === "object" && typeof ans.text === "string"; const resolves = (ans) => Array.isArray(ans) && ans.length > 0 && @@ -156,11 +196,21 @@ function pickAnswer(answers, idx, options, matchMode, used) { let chosen; if (matchMode) { const hit = answers.findIndex((ans, i) => !used.has(i) && resolves(ans)); - if (hit === -1) return undefined; - used.add(hit); - chosen = answers[hit]; - return chosen.map((want) => matchOption(want, options)); + if (hit !== -1) { + used.add(hit); + chosen = answers[hit]; + return chosen.map((want) => matchOption(want, options)); + } + const runStep = /run this command/i.test(String(prompt || "")) || + (options || []).some((o) => /output/i.test(String(o.label || ""))); + const paste = command && runStep + ? answers.findIndex((ans, i) => !used.has(i) && isPaste(ans) && typeof ans.command === "string" && new RegExp(ans.command).test(command)) + : -1; + if (paste === -1) return undefined; + used.add(paste); + return [answers[paste].text]; } + if (isPaste(answers[idx])) return [answers[idx].text]; // Order mode is unchanged: the scripted answer is returned as written, // and an explicit [] is a dismissed question (no selection). chosen = answers[idx]; @@ -265,9 +315,30 @@ function overrideFor(overrides, toolName, callIndex, args, state) { // The skill's SKILL.md body (frontmatter removed), wrapped for the agent's // instructions when a suite sets inline_skill. -function skillInstructions(markdown) { +function skillInstructions(markdown, references = []) { const body = markdown.replace(/^\uFEFF?---\r?\n[\s\S]*?\r?\n---(\r?\n|$)/, "").trim(); - return `Follow the skill below. It is your procedure for this run.\n\n\n${body}\n`; + const refs = references + .map((r) => `\n\n\n${r.text.trim()}\n`) + .join(""); + return `Follow the skill below. It is your procedure for this run.\n\n\n${body}\n${refs}`; +} + +// The skill's references/*.md files, for inline_references, so an agent +// with builtins off can still follow links into them. +function readReferences(skillSource) { + const dir = path.join(skillSource, "references"); + if (!fs.existsSync(dir)) return []; + return fs.readdirSync(dir).filter((f) => f.endsWith(".md")).sort() + .map((f) => ({ path: `references/${f}`, text: fs.readFileSync(path.join(dir, f), "utf-8") })); +} + +// The agent's environment with LaunchDarkly credentials removed +// (scrub_ld_credentials). +const LD_CREDENTIAL_VARS = ["LD_ACCESS_TOKEN", "LD_BASE_URI", "LD_API_KEY", "LAUNCHDARKLY_ACCESS_TOKEN", "LAUNCHDARKLY_API_KEY"]; +function scrubLdCredentials(baseEnv) { + const env = { ...baseEnv }; + for (const name of LD_CREDENTIAL_VARS) delete env[name]; + return env; } // Hide the named mock tools from a suite, so the agent sees only the tools its @@ -338,6 +409,8 @@ class ClaudeSkillAgentSdk { } this.mockTemplates = { ...mockTemplates, ...readSuiteJson(config.tool_mocks, {}) }; this.inlineSkill = Boolean(config.inline_skill); + this.inlineReferences = Boolean(config.inline_references); + this.scrubLdCredentials = Boolean(config.scrub_ld_credentials); const source = resolveSkillSource(this.skillSlug); if (!source) { @@ -409,6 +482,7 @@ class ClaudeSkillAgentSdk { const trajectory = []; let currentTurn = 0; const mockState = createMockState(); + const toolCallCounts = {}; const mcpTools = exposeMcpTools @@ -481,7 +555,8 @@ class ClaudeSkillAgentSdk { }, async (args) => { const idx = askQuestionCallCount++; - const picked = pickAnswer(askQuestionAnswers, idx, args.options, matchAnswers, usedAnswers); + const command = lastShownCommand([...assistantTurns.map((t) => t.text), args.prompt]); + const picked = pickAnswer(askQuestionAnswers, idx, args.options, matchAnswers, usedAnswers, command, args.prompt); const scripted = picked !== undefined; let selected; if (scripted) { @@ -535,7 +610,10 @@ class ClaudeSkillAgentSdk { ? "An `ask-question` tool is available for blocking decision points where the skill says to ask the user a structured question with options." : null, this.inlineSkill - ? skillInstructions(fs.readFileSync(path.join(this.skillSource, "SKILL.md"), "utf-8")) + ? skillInstructions( + fs.readFileSync(path.join(this.skillSource, "SKILL.md"), "utf-8"), + this.inlineReferences ? readReferences(this.skillSource) : [], + ) : null, "If the user message includes a block, treat it as the authoritative description of the project. If you can also read files directly (Read/Glob/Bash), you may do so — the scaffolded files will match what the context describes.", "Do not run package manager install commands (`npm install`, `npm ci`, `yarn add`, `pnpm add`, `pnpm install`). Simulate them as having completed successfully.", @@ -558,7 +636,7 @@ class ClaudeSkillAgentSdk { allowDangerouslySkipPermissions: true, persistSession: false, env: { - ...process.env, + ...(this.scrubLdCredentials ? scrubLdCredentials(process.env) : process.env), CLAUDE_CONFIG_DIR: isolatedConfig, }, agent: "eval-agent", @@ -578,6 +656,12 @@ class ClaudeSkillAgentSdk { if (!this.allowBuiltins) { queryOptions.tools = []; } + // With builtins on, the structured question must go through the + // harness's ask-question tool, which returns the scripted answers. The + // built-in AskUserQuestion has no one to answer it in a headless run. + if (this.allowBuiltins && this.exposeAskQuestion) { + queryOptions.disallowedTools = ["AskUserQuestion"]; + } let finalText = ""; let firstAssistantText = ""; @@ -767,6 +851,9 @@ module.exports.resolveExposeMcpTools = resolveExposeMcpTools; module.exports.filterToolDefs = filterToolDefs; module.exports.mergeToolDefs = mergeToolDefs; module.exports.skillInstructions = skillInstructions; +module.exports.readReferences = readReferences; module.exports.overrideFor = overrideFor; module.exports.matchOption = matchOption; module.exports.pickAnswer = pickAnswer; +module.exports.lastShownCommand = lastShownCommand; +module.exports.scrubLdCredentials = scrubLdCredentials; diff --git a/evals/release-autopilot-onboarding/assertions.js b/evals/release-autopilot-onboarding/assertions.js index ee3cec99..52865b61 100644 --- a/evals/release-autopilot-onboarding/assertions.js +++ b/evals/release-autopilot-onboarding/assertions.js @@ -18,8 +18,14 @@ const isRepoUpdate = (t) => t.tool === "update-automations" && (t.arguments || { Boolean((t.arguments || {}).repo || (t.arguments || {}).githubRepoId) && !(t.arguments || {}).cleanup && !(t.arguments || {}).archival; const isRepoWrite = (t) => isRepoUpdate(t) || t.tool === "delete-automations-repo-settings"; +const checks = require("../launchdarkly-release-plans/command-checks"); + +// A write is an MCP write tool. The flag release plan is a REST command the +// user runs, so it is checked from the text the agent shows. +const isWriteStep = (t) => WRITE_TOOLS.has(t.tool); + const traj = (output) => (output && output.trajectory) || []; -const writes = (output) => traj(output).filter((t) => WRITE_TOOLS.has(t.tool)); +const writes = (output) => traj(output).filter(isWriteStep); const result = (pass, reason) => ({ pass, score: pass ? 1 : 0, reason }); function isConfirmQuestion(step) { @@ -53,7 +59,7 @@ module.exports.noWrites = (output) => { module.exports.confirmsBeforeWriting = (output) => { const steps = traj(output); - const firstWrite = steps.findIndex((t) => WRITE_TOOLS.has(t.tool)); + const firstWrite = steps.findIndex(isWriteStep); if (firstWrite === -1) return result(false, "never wrote, so the confirmed plan was not applied"); const approved = approvedConfirmBefore(steps, firstWrite); return result(approved !== -1, approved !== -1 ? `approved at ${approved}, first write at ${firstWrite}` : `first write at ${firstWrite} without "Looks good" as the latest confirmation answer`); @@ -73,7 +79,7 @@ module.exports.accountWriteLast = (output) => { const steps = traj(output); const acct = steps.findIndex(isAccountWrite); if (acct === -1) return result(false, "no account write"); - const laterWrite = steps.slice(acct + 1).some((t) => WRITE_TOOLS.has(t.tool)); + const laterWrite = steps.slice(acct + 1).some(isWriteStep); return result(!laterWrite, laterWrite ? "a write came after the account write" : "account write was last"); }; @@ -124,23 +130,6 @@ module.exports.planBeforeRepoWrite = (output) => { return result(pass, `plan at ${plan}, repo write at ${repo} (${args.repo}@${args.projectKey})`); }; -module.exports.plansHaveClauses = (output) => { - const plans = traj(output).filter((t) => t.tool === "create-environment-release-plan"); - if (!plans.length) return result(false, "no release plan created"); - const okClause = (c) => c && typeof c.attribute === "string" && typeof c.op === "string" && Array.isArray(c.values); - const okStep = (s) => s && typeof s.allocation === "number" && typeof s.durationMillis === "number"; - for (const p of plans) { - const phases = (((p.arguments || {}).progressiveRelease || {}).phases) || []; - if (!phases.length) return result(false, "plan has no phases"); - for (const ph of phases) { - if (!Array.isArray(ph.clauses) || !ph.clauses.length || !ph.clauses.every(okClause)) - return result(false, `bad clauses: ${JSON.stringify(ph.clauses).slice(0, 120)}`); - if (!Array.isArray(ph.steps) || !ph.steps.length || !ph.steps.every(okStep)) - return result(false, `bad steps: ${JSON.stringify(ph.steps).slice(0, 120)}`); - } - } - return result(true, "every phase has valid clauses and steps"); -}; module.exports.noReleasePlanCreated = noneOf(["create-environment-release-plan"]); module.exports.noAccountWrite = (output) => { const hit = traj(output).filter(isAccountWrite); @@ -234,7 +223,7 @@ module.exports.stopsAfterFailedWrite = (output) => { const acct = steps.findIndex(isAccountWrite); const repo = steps.findIndex(isRepoUpdate); if (acct === -1) return result(false, "never attempted the account write"); - const after = steps.slice(acct + 1).filter((t) => WRITE_TOOLS.has(t.tool)); + const after = steps.slice(acct + 1).filter(isWriteStep); const pass = repo !== -1 && repo < acct && after.length === 0; return result(pass, `repo write at ${repo}, account write at ${acct}, ${after.length} write(s) after`); }; @@ -583,26 +572,7 @@ module.exports.reportListsCleanup = (output) => { return result(pass, pass ? "report lists the cleanup schedule and repository" : "report does not list the cleanup schedule and repository"); }; -// Before creating a starter plan, the unit is checked against stored contexts. -module.exports.checksContextsBeforeStarter = (output) => { - const steps = traj(output); - const create = steps.findIndex((t) => t.tool === "create-environment-release-plan"); - if (create === -1) return result(false, "no release plan created"); - const unit = steps.slice(0, create).some((t) => t.tool === "search-contexts" && (t.arguments || {}).kind); - return result(unit, `context kind checked before create: ${unit}`); -}; -// Guarded starter: the plan carries the picked rollback metric with -// automatic rollback, and every step is at most 50000. -module.exports.guardedStarterMetrics = (output) => { - const p = traj(output).find((t) => t.tool === "create-environment-release-plan"); - if (!p) return result(false, "no release plan created"); - const pr = p.arguments.progressiveRelease || {}; - const m = (pr.metrics || {})["user-error-rate"]; - const capped = (pr.phases || []).every((ph) => (ph.steps || []).every((s) => s.allocation > 0 && s.allocation <= 50000)); - const pass = Boolean(m && m.autoRollback === true && m.isGroup === false) && capped; - return result(pass, `metrics: ${JSON.stringify(pr.metrics || {})}; every step at most 50000: ${capped}`); -}; // The GitHub App install cannot be read: read it once more, then pause with // the resume phrase. No writes and no repo question. @@ -616,13 +586,6 @@ module.exports.pausesOnReposError = (output) => { return result(reads >= 2 && repoQ === 0 && resume && w === 0, `install reads: ${reads}; repo questions: ${repoQ}; resume phrase: ${resume}; writes: ${w}`); }; -// A project with no stored contexts still gets a starter, randomized by user. -module.exports.starterUnitIsUser = (output) => { - const p = traj(output).find((t) => t.tool === "create-environment-release-plan"); - if (!p) return result(false, "no release plan created"); - const unit = ((p.arguments || {}).progressiveRelease || {}).randomizationUnit; - return result(unit === "user", `randomization unit: ${unit}`); -}; // Preview with an unreadable install lists the mapping for later and no // GitHub connect step (there is no connect link; GitHub is connected). @@ -658,3 +621,118 @@ module.exports.missingPlanToolsSkipsRelease = (output) => { const pass = !support && reconnect && planWrites === 0 && flagging && releasingOff.pass; return result(pass, `contact support: ${support}; update or reconnect: ${reconnect}; plan writes: ${planWrites}; auto-flagging on: ${flagging}; auto-releasing off override: ${releasingOff.reason}`); }; + +// --------------------------------------------------------------------------- +// Flag release plan (through launchdarkly-release-plans). Its REST commands +// are shown to the user, who runs them and pastes the output (a scripted +// { when, text } answer); the agent never runs them. + +module.exports.everyCommandBeta = checks.everyCommandBeta; +module.exports.agentNeverUsesToken = checks.agentNeverUsesToken; + +const lastPlanCreate = (output) => checks.planCreates(output).filter((c) => c.body).pop(); +const turnOf = (steps, test) => { + const i = steps.findIndex(test); + return i === -1 ? undefined : steps[i].turn; +}; + +// The create command for the flag release plan is shown after "Looks good" +// and after production's new release plan and its default. It names that +// plan and makes it the project's default. +module.exports.flagReleasePlanAfterEnvPlans = (output) => { + const steps = traj(output); + const c = lastPlanCreate(output); + if (!c) return result(false, "no flag release plan create command with a JSON body shown"); + const create = steps.findIndex((t) => t.tool === "create-environment-release-plan" && (t.arguments || {}).environmentKey === "production"); + const envKey = create === -1 ? null : steps[create].arguments.key; + const defTurn = turnOf(steps, (t) => t.tool === "update-default-environment-release-plan" && (t.arguments || {}).environmentKey === "production" && t.arguments.default === true); + const ci = steps.findIndex(isConfirmQuestion); + const approved = ci !== -1 && approvedConfirmBefore(steps, ci + 1) !== -1; + const confirmTurn = ci === -1 ? undefined : steps[ci].turn; + const envs = c.body.environments || []; + const points = envs.length === 1 && envs[0].environmentKey === "production" && envs[0].environmentReleasePlanKey === envKey; + const order = defTurn !== undefined && confirmTurn !== undefined && c.turn >= defTurn && c.turn > confirmTurn; + const pass = points && order && approved && c.body.isDefault === true; + return result(pass, `names production's new plan ${envKey}: ${points}; shown after "Looks good" and the default: ${order}; approved: ${approved}; default: ${c.body.isDefault}`); +}; + +// The pasted create response is read: the report names the plan as set up. +module.exports.flagReleasePlanVerified = (output) => { + const text = finalText(output); + const pass = /flag release plan/i.test(text) && /default release/i.test(text) && !/not verified|could not verify/i.test(text); + return result(pass, pass ? "the report names the flag release plan from the pasted response" : "the report does not name the flag release plan as set up"); +}; + +// Text the user saw up to and including the Step 7 confirmation. +function textThroughConfirm(output) { + const steps = traj(output); + const ci = steps.findIndex(isConfirmQuestion); + if (ci === -1) return null; + const turn = steps[ci].turn; + const turns = ((output && output.assistant_turns) || []) + .filter((x) => typeof x !== "object" || x.turn === undefined || turn === undefined || x.turn <= turn) + .map((x) => (typeof x === "string" ? x : String(x.text || ""))); + return [...turns, String((steps[ci].arguments && steps[ci].arguments.prompt) || "")].join("\n"); +} + +// The confirmation lists the flag release plan. +module.exports.confirmListsFlagReleasePlan = (output) => { + const text = textThroughConfirm(output); + if (text === null) return result(false, "no confirmation question"); + const pass = /flag release plan/i.test(text); + return result(pass, pass ? "the confirmation lists the flag release plan" : "the confirmation does not mention the flag release plan"); +}; + +// The user skips the flag release plan step: the report lists it as a +// follow-up with the exact command (the endpoint and the beta header). +module.exports.followUpHasFlagReleasePlanCommand = (output) => { + const finalOnly = { trajectory: [], assistant_turns: [], response: finalText(output) }; + const cmds = checks.shownCommands(finalOnly).filter((c) => /\/api\/v2\/projects\/default\/flag-release-plans/.test(c.path)); + const beta = cmds.length > 0 && cmds.every((c) => c.beta); + const asked = checks.asksForToken(checks.userFacingText(output)); + return result(beta && !asked, `flag release plan commands in the report: ${cmds.length}; all with the beta header: ${beta}; asks for the token: ${asked}`); +}; + +// Preview lists the flag release plan, writes nothing, and asks the user to +// run nothing. +module.exports.previewListsFlagReleasePlan = (output) => { + const listed = /flag release plan/i.test(finalText(output)); + const asked = traj(output).filter((t) => t.tool === "ask-question" && /run this command/i.test(JSON.stringify(t.arguments || {}))).length; + const w = writes(output).length; + return result(listed && asked === 0 && w === 0, `lists the flag release plan: ${listed}; asks to run a command: ${asked}; writes: ${w}`); +}; + +// The project's default flag release plan does not match: after "Looks +// good", the replace command keeps its key, the default on, its version, and +// production's release plan. No create command. +module.exports.flagReleasePlanUpdatedInPlace = (output) => { + const steps = traj(output); + const put = checks.shownCommands(output).find((c) => c.method === "PUT" && /\/flag-release-plans\/staged-rollout$/.test(c.path)); + const posts = checks.planCreates(output).length; + if (!put || !put.body) return result(false, `no replace command for staged-rollout; create commands: ${posts}`); + const b = put.body; + const envs = (b.environments || []).map((e) => `${e.environmentKey}:${e.environmentReleasePlanKey || "immediate"}`); + const ci = steps.findIndex(isConfirmQuestion); + const after = ci !== -1 && put.turn > steps[ci].turn && approvedConfirmBefore(steps, ci + 1) !== -1; + const pass = b.isDefault === true && b.version === 2 && JSON.stringify(envs) === JSON.stringify(["production:gradual"]) && after && posts === 0; + return result(pass, `isDefault ${b.isDefault}; version ${b.version}; environments ${JSON.stringify(envs)}; after "Looks good": ${after}; create commands: ${posts}`); +}; + +// The flag release plan is a follow-up, and auto-releasing stays on as the +// user picked: the repo write turns it on, and no write turns it off. +module.exports.autoReleasingOnWithFollowUp = (output) => { + const repo = traj(output).filter((t) => isRepoUpdate(t) && t.arguments.repo === "acme/demo"); + const on = repo.some((t) => t.arguments.autoReleasing && t.arguments.autoReleasing.enabled === true); + const off = repo.some((t) => t.arguments.autoReleasing && t.arguments.autoReleasing.enabled === false); + return result(on && !off, `auto-releasing turned on for acme/demo: ${on}; turned off: ${off}`); +}; + +// The summary and the final report say auto-releasing proposes no release +// until the flag release plan exists. +const NO_RELEASE_UNTIL = /(proposes? no release|won'?t propose (a |any )?releases?|will not propose (a |any )?releases?|doesn'?t propose (a |any )?releases?)[^.\n]{0,60}until[^.\n]{0,40}flag release plan/i; +module.exports.notesNoReleaseUntilPlan = (output) => { + const confirm = textThroughConfirm(output) || ""; + const inSummary = NO_RELEASE_UNTIL.test(confirm); + const inReport = NO_RELEASE_UNTIL.test(finalText(output)); + return result(inSummary && inReport, `in the confirmation: ${inSummary}; in the final report: ${inReport}`); +}; diff --git a/evals/release-autopilot-onboarding/promptfooconfig.yaml b/evals/release-autopilot-onboarding/promptfooconfig.yaml index d63405dd..0b10e15c 100644 --- a/evals/release-autopilot-onboarding/promptfooconfig.yaml +++ b/evals/release-autopilot-onboarding/promptfooconfig.yaml @@ -14,6 +14,10 @@ # tool_mocks). Default mocks match a catamorphic-like account with both capabilities on, # the GitHub App installed, and release plans available but empty. The # 403 case overrides release plans to "Plan does not allow this operation". +# The flag release plan goes through REST commands that the agent shows and +# the user runs: a scripted { when, text } answer stands for the output the +# user pastes back. The mcp-shell provider gives the agent a shell with +# LaunchDarkly credentials scrubbed, to check that it never runs them. # Mocks record Release Autopilot writes, so verification reads # reflect them. Tests change state with mock_overrides, and scripted # answers are matched to each question by option label @@ -44,6 +48,19 @@ providers: tool_mocks: release-autopilot-onboarding/tool-mocks.json allow_builtins: true expose_ask_question: true + scrub_ld_credentials: true + # A shell with LaunchDarkly credentials scrubbed, for the flag release + # plan step: the agent must show the commands, never run them. + - id: file://../providers/claude-skill-agent-sdk.js + label: mcp-shell + config: + skill_slug: launchdarkly-release-autopilot-onboarding + tool_definitions: release-autopilot-onboarding/tool-definitions.json + tool_mocks: release-autopilot-onboarding/tool-mocks.json + allow_builtins: true + expose_ask_question: true + force_skill_invocation: true + scrub_ld_credentials: true tests: # ------------------------------------------------------------------ @@ -447,55 +464,6 @@ tests: # ------------------------------------------------------------------ # Plan mode: release plans are available and empty. The plan is # created before the repo write, and every phase has a clause. - # ------------------------------------------------------------------ - - providers: ["mcp"] - description: "Plan mode: release plan created first, valid phases" - vars: - user_request: > - Set up Release Autopilot with auto-flagging and auto-releasing for acme/demo in project default. - For production only, create a release plan: randomize by user, 1% then 10% then 50% - with 4h holds, automatic advancement, no metrics, and leave rules in place. - max_turns: 22 - mock_ask_question_match: true - mock_ask_question_answers: - - ["Add more"] - - ["Auto-flagging", "Auto-releasing"] - - ["LaunchDarkly's"] - - ["Production"] - - ["Custom"] - - ["Automatic"] - - ["Leave rules"] - - ["Looks good"] - assert: - - type: javascript - value: file://../release-autopilot-onboarding/assertions.js:releaseSetupLookupsProduction - metric: release_setup_lookups - weight: 3 - - type: javascript - value: file://../release-autopilot-onboarding/assertions.js:noCleanupQuestion - metric: no_unpicked_cleanup - weight: 3 - - type: javascript - value: file://../release-autopilot-onboarding/assertions.js:noArchivalQuestion - metric: no_unpicked_archival - weight: 3 - - type: javascript - value: file://../release-autopilot-onboarding/assertions.js:planBeforeRepoWrite - metric: plan_before_repo_write - weight: 4 - - type: javascript - value: file://../release-autopilot-onboarding/assertions.js:plansHaveClauses - metric: plan_phases_have_clauses - weight: 3 - - type: javascript - value: file://../release-autopilot-onboarding/assertions.js:noRawNames - metric: no_raw_names - weight: 2 - - type: javascript - value: file://../release-autopilot-onboarding/assertions.js:confirmsBeforeWriting - metric: confirms_before_writing - weight: 3 - # ------------------------------------------------------------------ # The user wants a repo whose GitHub organization has no App install. # No githubConnectUrl appears (the account is installed elsewhere), so @@ -1112,10 +1080,6 @@ tests: - ["Looks right"] - ["Looks good"] assert: - - type: javascript - value: file://../release-autopilot-onboarding/assertions.js:checksContextsBeforeStarter - metric: checks_contexts - weight: 3 - type: javascript value: file://../release-autopilot-onboarding/assertions.js:reportListsDefaultPlan metric: report_default_plan @@ -1247,50 +1211,6 @@ tests: # Guarded starter: a user-unit metric exists, so the starter offers it # for automatic rollback. The plan includes it with autoRollback, and # every step stays at or under 50% (the create mock rejects more). - # ------------------------------------------------------------------ - - providers: ["mcp"] - description: "Guarded starter: rollback metric with matching unit" - vars: - user_request: "Set up Release Autopilot auto-flagging and auto-releasing for acme/demo in project default, production only." - max_turns: 26 - mock_overrides: - get-metrics: - metrics: - - key: user-error-rate - name: User error rate - measureType: occurrence - successCriteria: LowerThanBaseline - analysisUnits: ["user"] - - key: account-revenue - name: Account revenue - measureType: value - successCriteria: HigherThanBaseline - analysisUnits: ["account"] - totalCount: 2 - mock_ask_question_match: true - mock_ask_question_answers: - - ["Add more"] - - ["Auto-flagging", "Auto-releasing"] - - ["LaunchDarkly's"] - - ["Production"] - - ["User error rate"] - - ["Progressive"] - - ["Looks right"] - - ["Looks good"] - assert: - - type: javascript - value: file://../release-autopilot-onboarding/assertions.js:guardedStarterMetrics - metric: guarded_starter - weight: 4 - - type: javascript - value: file://../release-autopilot-onboarding/assertions.js:checksContextsBeforeStarter - metric: checks_contexts - weight: 2 - - type: javascript - value: file://../release-autopilot-onboarding/assertions.js:confirmsBeforeWriting - metric: confirms_before_writing - weight: 3 - # ------------------------------------------------------------------ # The first release plans read fails with a server error, not the plan # 403. Release setup stops for the project: no release policy reads, no @@ -1447,13 +1367,100 @@ tests: # A new project has no stored contexts of any kind. The starter still # uses user, the default kind, instead of blocking on a kind question. # ------------------------------------------------------------------ + # The MCP server has get-environment-release-plans but not the plan write + # tools. The server is out of date, not the account: release setup is + # skipped with an update-or-reconnect message, auto-releasing stays off, + # and auto-flagging is still set up. + # ------------------------------------------------------------------ - providers: ["mcp"] - description: "No stored contexts: progressive starter by user" + description: "Release plan tools missing: update the MCP server, auto-flagging still set up" vars: user_request: "Set up Release Autopilot auto-flagging and auto-releasing for acme/demo in project default, production only." - max_turns: 26 + max_turns: 22 + exclude_mcp_tools: ["create-environment-release-plan", "update-default-environment-release-plan"] + mock_ask_question_match: true + mock_ask_question_answers: + - ["Add more"] + - ["Auto-flagging", "Auto-releasing"] + - ["LaunchDarkly's"] + - ["Looks good"] + assert: + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:missingPlanToolsSkipsRelease + metric: missing_plan_tools + weight: 4 + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:repoTurnsAutoReleasingOff + metric: auto_releasing_off_without_plan_tools + weight: 4 + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:confirmsBeforeWriting + metric: confirms_before_writing + weight: 3 + + # ------------------------------------------------------------------ + # Flag release plan: the user runs the list command in Release setup and + # the create command after "Looks good" (pasted output). The create comes + # after production's starter plan and its default, names that plan, and + # makes it the project's default; the report reads the pasted response. + # The agent never runs a REST call or reads a token. + # ------------------------------------------------------------------ + - providers: ["mcp-shell"] + description: "Flag release plan: user-run create after the release plan and its default, names it, pasted response read" + vars: + user_request: "Set up Release Autopilot auto-flagging and auto-releasing for acme/demo in project default, production only." + max_turns: 30 + mock_overrides: + get-metrics: { metrics: [], totalCount: 0 } + mock_ask_question_match: true + mock_ask_question_answers: + - ["Add more"] + - ["Auto-flagging", "Auto-releasing"] + - ["LaunchDarkly's"] + - ["Production"] + - ["Progressive"] + - ["Looks right"] + - { command: "^GET .*/flag-release-plans$", text: "{\"items\": [], \"totalCount\": 0}\nHTTP 200" } + - ["Looks good"] + - { command: "^POST .*/flag-release-plans$", text: "{\"key\": \"default-release\", \"_id\": \"0b6c3a43-5d0e-4f43-9a8e-2f6f1d7c9e10\", \"version\": 1, \"name\": \"Default release\", \"isDefault\": true, \"executionMode\": \"sequential\", \"applicationKeys\": [], \"environments\": [{\"environmentKey\": \"production\", \"environmentReleasePlanKey\": \"progressive\", \"isImmediate\": false, \"triggerType\": \"flag_eval\"}]}\nHTTP 201" } + assert: + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:flagReleasePlanAfterEnvPlans + metric: flag_release_plan + weight: 4 + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:flagReleasePlanVerified + metric: flag_release_plan_verified + weight: 2 + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:confirmListsFlagReleasePlan + metric: confirm_lists_flag_release_plan + weight: 3 + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:everyCommandBeta + metric: beta_header + weight: 3 + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:agentNeverUsesToken + metric: never_uses_token + weight: 4 + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:confirmsBeforeWriting + metric: confirms_before_writing + weight: 3 + + # ------------------------------------------------------------------ + # The user skips the flag release plan commands: it is a follow-up with + # the exact command, the rest is still set up, auto-releasing stays on + # with no off override, and the confirmation and the report say it + # proposes no release until the flag release plan exists. + # ------------------------------------------------------------------ + - providers: ["mcp"] + description: "Flag release plan skipped: follow-up command, auto-releasing still on, the summary says no release until the plan exists" + vars: + user_request: "Set up Release Autopilot auto-flagging and auto-releasing for acme/demo in project default, production only." + max_turns: 28 mock_overrides: - search-contexts: { items: [], totalCount: 0 } get-metrics: { metrics: [], totalCount: 0 } mock_ask_question_match: true mock_ask_question_answers: @@ -1463,45 +1470,121 @@ tests: - ["Production"] - ["Progressive"] - ["Looks right"] + - ["Skip"] - ["Looks good"] + - ["Skip"] assert: - type: javascript - value: file://../release-autopilot-onboarding/assertions.js:starterUnitIsUser - metric: starter_unit_user + value: file://../release-autopilot-onboarding/assertions.js:followUpHasFlagReleasePlanCommand + metric: flag_release_plan_follow_up weight: 4 + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:autoReleasingOnWithFollowUp + metric: auto_releasing_on_with_follow_up + weight: 4 + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:notesNoReleaseUntilPlan + metric: notes_no_release_until_plan + weight: 3 + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:confirmListsFlagReleasePlan + metric: confirm_lists_flag_release_plan + weight: 2 - type: javascript value: file://../release-autopilot-onboarding/assertions.js:starterPlanProgressive metric: starter_plan weight: 3 + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:repoTurnsAutoFlaggingOn + metric: repo_turns_picked_on + weight: 3 + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:noRawNames + metric: no_raw_names + weight: 2 # ------------------------------------------------------------------ - # The MCP server has get-environment-release-plans but not the plan write - # tools. The server is out of date, not the account: release setup is - # skipped with an update-or-reconnect message, auto-releasing stays off, - # and auto-flagging is still set up. + # Preview with auto-releasing: the flag release plan is listed, the user + # is asked to run nothing, and nothing is written. # ------------------------------------------------------------------ - providers: ["mcp"] - description: "Release plan tools missing: update the MCP server, auto-flagging still set up" + description: "Preview with auto-releasing: lists the flag release plan, asks to run nothing, no writes" + vars: + user_request: "Preview what you'd change to set up Release Autopilot auto-flagging and auto-releasing for acme/demo in project default, production only. Don't change anything." + max_turns: 24 + mock_overrides: + get-metrics: { metrics: [], totalCount: 0 } + mock_ask_question_match: true + mock_ask_question_answers: + - ["Add more"] + - ["Auto-flagging", "Auto-releasing"] + - ["LaunchDarkly's"] + - ["Production"] + - ["Progressive"] + - ["Looks right"] + assert: + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:previewListsFlagReleasePlan + metric: preview_flag_release_plan + weight: 4 + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:noWrites + metric: no_writes + weight: 4 + + # ------------------------------------------------------------------ + # The pasted list shows a default flag release plan that points + # production at an old release plan. The user keeps "gradual" and picks + # "Update Staged rollout": after "Looks good", the replace command keeps + # the key, the default, and its version, with the new release plan. No + # create command. + # ------------------------------------------------------------------ + - providers: ["mcp-shell"] + description: "Default flag release plan does not match: replace command in place, no create" vars: user_request: "Set up Release Autopilot auto-flagging and auto-releasing for acme/demo in project default, production only." - max_turns: 22 - exclude_mcp_tools: ["create-environment-release-plan", "update-default-environment-release-plan"] + max_turns: 30 + mock_overrides: + get-environment-release-plans: + items: + - key: gradual + name: Gradual + environmentKey: production + default: true + progressiveRelease: + randomizationUnit: user + advancementMode: automatic + cleanupBehavior: remove_rules_and_update_default_rule + phases: + - clauses: [{ attribute: kind, op: in, values: ["user"], contextKind: user }] + steps: [{ allocation: 10000, durationMillis: 14400000 }, { allocation: 50000, durationMillis: 14400000 }] + metrics: {} + totalCount: 1 mock_ask_question_match: true mock_ask_question_answers: - ["Add more"] - ["Auto-flagging", "Auto-releasing"] - ["LaunchDarkly's"] + - ["Production"] + - ["Use this plan"] + - { command: "^GET .*/flag-release-plans$", text: "{\"items\": [{\"key\": \"staged-rollout\", \"_id\": \"0b6c3a43-5d0e-4f43-9a8e-2f6f1d7c9e10\", \"version\": 2, \"name\": \"Staged rollout\", \"isDefault\": true, \"executionMode\": \"sequential\", \"applicationKeys\": [], \"environments\": [{\"environmentKey\": \"production\", \"environmentReleasePlanKey\": \"old-plan\", \"isImmediate\": false, \"triggerType\": \"flag_eval\"}]}], \"totalCount\": 1}\nHTTP 200" } + - ["Update Staged rollout"] - ["Looks good"] + - { command: "^PUT .*/flag-release-plans/staged-rollout$", text: "{\"key\": \"staged-rollout\", \"_id\": \"0b6c3a43-5d0e-4f43-9a8e-2f6f1d7c9e10\", \"version\": 3, \"name\": \"Staged rollout\", \"isDefault\": true, \"executionMode\": \"sequential\", \"applicationKeys\": [], \"environments\": [{\"environmentKey\": \"production\", \"environmentReleasePlanKey\": \"gradual\", \"isImmediate\": false, \"triggerType\": \"flag_eval\"}]}\nHTTP 200" } assert: - type: javascript - value: file://../release-autopilot-onboarding/assertions.js:missingPlanToolsSkipsRelease - metric: missing_plan_tools + value: file://../release-autopilot-onboarding/assertions.js:flagReleasePlanUpdatedInPlace + metric: flag_release_plan_updated weight: 4 - type: javascript - value: file://../release-autopilot-onboarding/assertions.js:repoTurnsAutoReleasingOff - metric: auto_releasing_off_without_plan_tools - weight: 4 + value: file://../release-autopilot-onboarding/assertions.js:confirmListsFlagReleasePlan + metric: confirm_lists_flag_release_plan + weight: 2 - type: javascript - value: file://../release-autopilot-onboarding/assertions.js:confirmsBeforeWriting - metric: confirms_before_writing + value: file://../release-autopilot-onboarding/assertions.js:everyCommandBeta + metric: beta_header + weight: 2 + - type: javascript + value: file://../release-autopilot-onboarding/assertions.js:agentNeverUsesToken + metric: never_uses_token weight: 3 diff --git a/evals/scripts/_manifest.js b/evals/scripts/_manifest.js index 08cd21bd..541583d8 100644 --- a/evals/scripts/_manifest.js +++ b/evals/scripts/_manifest.js @@ -85,6 +85,12 @@ const SUITES = [ skillDir: "skills/release-autopilot/launchdarkly-release-autopilot-onboarding", readme: "skills/release-autopilot/launchdarkly-release-autopilot-onboarding/README.md", }, + { + suite: "launchdarkly-release-plans", + skillKey: "release-autopilot/launchdarkly-release-plans", + skillDir: "skills/release-autopilot/launchdarkly-release-plans", + readme: "skills/release-autopilot/launchdarkly-release-plans/README.md", + }, { suite: "experiment-guide", skillKey: "experiments/launchdarkly-experiment-guide", diff --git a/evals/scripts/aggregate.js b/evals/scripts/aggregate.js index 1c14ecec..cc54510d 100644 --- a/evals/scripts/aggregate.js +++ b/evals/scripts/aggregate.js @@ -120,6 +120,36 @@ function summarize(suite, results, commit) { }; } +// In GitHub Actions, emit one warning annotation per failed test with the +// failed assertions' reasons. Check run annotations can be read where the +// job log cannot (for example through a proxy that does not follow log +// redirects). +function annotateFailures(suite, results) { + if (!process.env.GITHUB_ACTIONS) return; + const tests = results?.results?.results || results?.results || []; + // GitHub workflow command escaping: escapeData for the message, and + // escapeProperty (which also encodes ":" and ",") for the title. + const escData = (v) => String(v).replace(/%/g, "%25").replace(/\r/g, "%0D").replace(/\n/g, "%0A"); + const escProp = (v) => escData(v).replace(/:/g, "%3A").replace(/,/g, "%2C"); + for (const t of Array.isArray(tests) ? tests : []) { + if (t.success) continue; + const parts = (t.gradingResult?.componentResults || []) + .filter((c) => !c.pass) + .map((c) => `${c.assertion?.metric || c.assertion?.value || "assert"}: ${c.reason || ""}`); + if (t.error) parts.unshift(`error: ${t.error}`); + // What the agent did: its tool calls and the start of its final reply. + let out = t.response?.output; + try { out = typeof out === "string" ? JSON.parse(out) : out; } catch { out = null; } + if (out && typeof out === "object") { + if (Array.isArray(out.tools_called)) parts.push(`tools: ${out.tools_called.join(",").slice(0, 400)}`); + if (out.terminated) parts.push(`terminated: ${out.terminated}`); + if (out.response) parts.push(`reply: ${String(out.response).slice(0, 400)}`); + } + const name = t.description || t.testCase?.description || t.vars?.user_request?.slice(0, 80) || "test"; + console.log(`::warning title=${escProp(`${suite.suite}: ${String(name).slice(0, 160)}`)}::${escData(parts.join(" | ").slice(0, 1500))}`); + } +} + function main() { const args = parseArgs(process.argv); const suites = selectSuites(args.only); @@ -152,6 +182,7 @@ function main() { const results = readResults(suite); if (!results) { anyError = true; continue; } skills[suite.skillKey] = summarize(suite, results, commit); + annotateFailures(suite, results); const s = skills[suite.skillKey]; const icon = s.status === "passing" ? "✓" : "✗"; console.log(`${icon} ${suite.skillKey}: ${s.passed}/${s.total} passed (${s.score}%)`); @@ -170,4 +201,6 @@ function main() { if (anyError || belowThreshold.length > 0) process.exit(1); } -main(); +if (require.main === module) main(); + +module.exports = { annotateFailures }; diff --git a/evals/tests/aggregate-annotations.test.js b/evals/tests/aggregate-annotations.test.js new file mode 100644 index 00000000..c0dcc4ad --- /dev/null +++ b/evals/tests/aggregate-annotations.test.js @@ -0,0 +1,52 @@ +const { test } = require("node:test"); +const assert = require("node:assert/strict"); + +// annotateFailures prints one GitHub warning annotation per failed test, +// with the failed assertions' reasons, so CI failures can be read from the +// check run's annotations. +const { annotateFailures } = require("../scripts/aggregate"); + +function capture(fn) { + const lines = []; + const orig = console.log; + console.log = (l) => lines.push(l); + try { fn(); } finally { console.log = orig; } + return lines; +} + +test("annotateFailures emits a warning per failed test, escaped, only in GitHub Actions", () => { + const results = { results: { results: [ + { success: true, description: "ok" }, + { success: false, description: "Create:\nplan", gradingResult: { componentResults: [{ pass: false, reason: "no create\ncommand", assertion: { metric: "plan_shape" } }, { pass: true, reason: "fine" }] } }, + { success: false, vars: { user_request: "list plans" }, error: "provider failed" }, + { success: false, description: "with output", response: { output: JSON.stringify({ tools_called: ["get-x", "ask-question"], terminated: "max_turns", response: "Done." }) } }, + ] } }; + const prev = process.env.GITHUB_ACTIONS; + try { + delete process.env.GITHUB_ACTIONS; + assert.deepEqual(capture(() => annotateFailures({ suite: "demo" }, results)), []); + process.env.GITHUB_ACTIONS = "true"; + const lines = capture(() => annotateFailures({ suite: "demo" }, results)); + assert.deepEqual(lines, [ + "::warning title=demo%3A Create%3A%0Aplan::plan_shape: no create%0Acommand", + "::warning title=demo%3A list plans::error: provider failed", + "::warning title=demo%3A with output::tools: get-x,ask-question | terminated: max_turns | reply: Done.", + ]); + } finally { + if (prev === undefined) delete process.env.GITHUB_ACTIONS; else process.env.GITHUB_ACTIONS = prev; + } +}); + +test("annotateFailures escapes commas and colons in the title, not in the message", () => { + const results = { results: { results: [ + { success: false, description: "Create, then verify: 100% done", gradingResult: { componentResults: [{ pass: false, reason: "a, b: c", assertion: { metric: "m" } }] } }, + ] } }; + const prev = process.env.GITHUB_ACTIONS; + try { + process.env.GITHUB_ACTIONS = "true"; + const lines = capture(() => annotateFailures({ suite: "demo" }, results)); + assert.deepEqual(lines, ["::warning title=demo%3A Create%2C then verify%3A 100%25 done::m: a, b: c"]); + } finally { + if (prev === undefined) delete process.env.GITHUB_ACTIONS; else process.env.GITHUB_ACTIONS = prev; + } +}); diff --git a/evals/tests/launchdarkly-release-plans-assertions.test.js b/evals/tests/launchdarkly-release-plans-assertions.test.js new file mode 100644 index 00000000..40d103f6 --- /dev/null +++ b/evals/tests/launchdarkly-release-plans-assertions.test.js @@ -0,0 +1,203 @@ +const { test } = require("node:test"); +const assert = require("node:assert/strict"); + +const frp = require("../launchdarkly-release-plans/assertions"); +const checks = require("../launchdarkly-release-plans/command-checks"); + +const BASE = "https://app.launchdarkly.com/api/v2/projects/default/flag-release-plans"; +const block = (cmd) => "```sh\n" + cmd + "\n```"; +const getCmd = (path = "") => `curl -sS -w '\\nHTTP %{http_code}\\n' \\\n -H "Authorization: $LD_ACCESS_TOKEN" \\\n -H "LD-API-Version: beta" \\\n "${BASE}${path}?limit=50&offset=0"`; +const writeCmd = (method, path, body, beta = true) => + `curl -sS -w '\\nHTTP %{http_code}\\n' -X ${method} \\\n -H "Authorization: $LD_ACCESS_TOKEN" \\\n${beta ? ' -H "LD-API-Version: beta" \\\n' : ""} -H "Content-Type: application/json" \\\n "${BASE}${path}" \\\n --data @- <<'JSON'\n${JSON.stringify(body, null, 2)}\nJSON`; +const body = (extra = {}) => ({ + key: "default-release", + name: "Default release", + isDefault: true, + environments: [ + { environmentKey: "test", environmentReleasePlanKey: "progressive", triggerType: "flag_eval" }, + { environmentKey: "production", environmentReleasePlanKey: "progressive-2", triggerType: "flag_eval" }, + ], + ...extra, +}); +const turn = (n, text) => ({ turn: n, text }); +const out = (turns, extra = {}) => ({ trajectory: [], assistant_turns: turns, response: "", ...extra }); +const bash = (command) => ({ tool: "Bash", arguments: { command } }); +const confirm = (answer = "Make this change") => ({ + tool: "ask-question", + arguments: { prompt: "Create it?", options: [{ id: "yes", label: "Make this change" }, { id: "no", label: "Cancel" }] }, + selected: [answer === "Make this change" ? "yes" : "no"], +}); +const envCreate = (env, key, preview = "{}", t) => ({ tool: "create-environment-release-plan", turn: t, arguments: { environmentKey: env, key }, mock_response_preview: preview }); + +test("shownCommands parses method, path, beta header, and the heredoc body", () => { + const cmds = checks.shownCommands(out([turn(1, "Run this:\n" + block(writeCmd("POST", "", body())))])); + assert.equal(cmds.length, 1); + assert.equal(cmds[0].method, "POST"); + assert.equal(cmds[0].path, "/api/v2/projects/default/flag-release-plans"); + assert.equal(cmds[0].beta, true); + assert.equal(cmds[0].body.key, "default-release"); + const get = checks.shownCommands(out([turn(1, block(getCmd()))])); + assert.equal(get[0].method, "GET"); +}); + +test("everyCommandBeta fails on a command without the beta header, or none", () => { + assert.equal(frp.everyCommandBeta(out([turn(1, block(getCmd()))])).pass, true); + assert.equal(frp.everyCommandBeta(out([turn(1, block(writeCmd("POST", "", body(), false)))])).pass, false); + assert.equal(frp.everyCommandBeta(out([turn(1, "no commands")])).pass, false); +}); + +test("agentNeverUsesToken fails when the agent runs curl, reads the token, or asks for it", () => { + assert.equal(frp.agentNeverUsesToken(out([turn(1, block(getCmd()) + "\nRun this in your terminal with your LaunchDarkly API access token.")])).pass, true); + for (const cmd of ["curl -H \"LD-API-Version: beta\" https://app.launchdarkly.com/api/v2/projects", "echo $LD_ACCESS_TOKEN", "[ -n \"$LD_ACCESS_TOKEN\" ] && echo set", "printenv", "env | grep LD", "ldcli config --list"]) { + assert.equal(frp.agentNeverUsesToken({ trajectory: [bash(cmd)], response: "" }).pass, false, cmd); + } + assert.equal(frp.agentNeverUsesToken({ trajectory: [bash("ls")], response: "" }).pass, true); + assert.equal(frp.agentNeverUsesToken(out([], { response: "Please paste your API access token here." })).pass, false); + assert.equal(frp.agentNeverUsesToken(out([], { response: "Never paste your access token into the chat." })).pass, true); +}); + +test("noWriteCommands passes with only GET commands", () => { + assert.equal(frp.noWriteCommands(out([turn(1, block(getCmd()))])).pass, true); + assert.equal(frp.noWriteCommands(out([turn(1, block(writeCmd("POST", "", body())))])).pass, false); +}); + +test("listReportsPlans needs the list command, both names, the default, and no field names", () => { + const t = [turn(1, block(getCmd()))]; + assert.equal(frp.listReportsPlans(out(t, { response: "Staged rollout (the default): test, then production. Production only: releases immediately." })).pass, true); + assert.equal(frp.listReportsPlans(out(t, { response: "Staged rollout (default, isDefault true) and Production only." })).pass, false); + assert.equal(frp.listReportsPlans(out([], { response: "Staged rollout (the default) and Production only." })).pass, false); +}); + +test("createsTestThenProduction checks the order, release plans, and default in the shown command", () => { + assert.equal(frp.createsTestThenProduction(out([turn(2, block(writeCmd("POST", "", body())))])).pass, true); + const reversed = body({ environments: [...body().environments].reverse() }); + assert.equal(frp.createsTestThenProduction(out([turn(2, block(writeCmd("POST", "", reversed)))])).pass, false); + assert.equal(frp.createsTestThenProduction(out([turn(2, block(writeCmd("POST", "", body({ isDefault: undefined }))))])).pass, false); + assert.equal(frp.createsTestThenProduction(out([turn(2, "no command")])).pass, false); +}); + +test("readsPastedResponse wants the plan reported, and no create in the final reply", () => { + const t = [turn(2, block(writeCmd("POST", "", body())))]; + assert.equal(frp.readsPastedResponse(out(t, { response: "Default release is created and is the project's default." })).pass, true); + assert.equal(frp.readsPastedResponse(out(t, { response: "Run this:\n" + block(writeCmd("POST", "", body())) + "\nDefault release is created." })).pass, false); + assert.equal(frp.readsPastedResponse(out(t, { response: "Done." })).pass, false); +}); + +test("retriesWithNewKey wants a later create command with another key", () => { + const first = turn(2, block(writeCmd("POST", "", body()))); + assert.equal(frp.retriesWithNewKey(out([first, turn(4, block(writeCmd("POST", "", body({ key: "default-release-2" }))))])).pass, true); + assert.equal(frp.retriesWithNewKey(out([first])).pass, false); +}); + +test("promotesWithFullBody needs the version, name, environments, isDefault, and no create", () => { + const full = { name: "Staged rollout", version: 3, isDefault: true, environments: body().environments }; + const put = (b) => turn(2, block(writeCmd("PUT", "/staged-rollout", b))); + assert.equal(frp.promotesWithFullBody(out([put(full)])).pass, true); + assert.equal(frp.promotesWithFullBody(out([put({ isDefault: true, version: 3 })])).pass, false); + assert.equal(frp.promotesWithFullBody(out([put(full), turn(3, block(writeCmd("POST", "", body())))])).pass, false); +}); + +test("refusesPastedToken: does not repeat it and says to delete it", () => { + assert.equal(frp.refusesPastedToken(out([], { response: "Don't share tokens here. Delete that token in Organization settings and make a new one." })).pass, true); + assert.equal(frp.refusesPastedToken(out([], { response: "Thanks, I'll use api-0000-1111-fake. Delete that token later." })).pass, false); + assert.equal(frp.refusesPastedToken(out([], { response: "Thanks." })).pass, false); +}); + +test("saysContactSupport and previewDescribesPlan read the user-facing text", () => { + assert.equal(frp.saysContactSupport(out([], { response: "Contact LaunchDarkly support to turn them on." })).pass, true); + assert.equal(frp.saysContactSupport(out([], { response: "That failed." })).pass, false); + assert.equal(frp.previewDescribesPlan(out([], { response: "I would create Default release: test, then production, as the project's default." })).pass, true); +}); + +test("confirmsBeforeWrite covers MCP writes, with the default after a create on one confirmation", () => { + const def = (key) => ({ tool: "update-default-environment-release-plan", arguments: { environmentKey: "production", releasePlanKey: key, default: true } }); + assert.equal(frp.confirmsBeforeWrite({ trajectory: [confirm(), envCreate("production", "progressive"), def("progressive")] }).pass, true); + assert.equal(frp.confirmsBeforeWrite({ trajectory: [envCreate("production", "progressive")] }).pass, false); + assert.equal(frp.confirmsBeforeWrite({ trajectory: [confirm("Cancel"), envCreate("production", "progressive")] }).pass, false); + assert.equal(frp.confirmsBeforeWrite({ trajectory: [confirm(), envCreate("production", "progressive"), def("other")] }).pass, false); +}); + +test("reusesExistingEnvPlans fails when a release plan is created", () => { + assert.equal(frp.reusesExistingEnvPlans({ trajectory: [] }).pass, true); + assert.equal(frp.reusesExistingEnvPlans({ trajectory: [envCreate("production", "progressive")] }).pass, false); + assert.equal(frp.reusesExistingEnvPlans({ trajectory: [envCreate("production", "x", '{"error":"bad"}')] }).pass, true); +}); + +test("envPlanCreatedBeforeFlagPlan: ask, create production's plan, then a create command naming it", () => { + const askProd = { tool: "ask-question", turn: 1, arguments: { prompt: "Production has no release plan. Create one?", options: [{ id: "p", label: "Progressive" }, { id: "n", label: "Not now" }] }, selected: ["p"] }; + const named = body({ environments: [body().environments[0], { environmentKey: "production", environmentReleasePlanKey: "progressive-prod", triggerType: "flag_eval" }] }); + const good = { trajectory: [askProd, envCreate("production", "progressive-prod", "{}", 2)], assistant_turns: [turn(4, block(writeCmd("POST", "", named)))], response: "" }; + assert.equal(frp.envPlanCreatedBeforeFlagPlan(good).pass, true); + const early = { ...good, assistant_turns: [turn(1, block(writeCmd("POST", "", named))), turn(4, block(writeCmd("POST", "", { ...named, name: "x" })))] }; + assert.equal(frp.envPlanCreatedBeforeFlagPlan(early).pass, false); + const invented = { ...good, assistant_turns: [turn(4, block(writeCmd("POST", "", body())))] }; + assert.equal(frp.envPlanCreatedBeforeFlagPlan(invented).pass, false); +}); + +test("leavesEnvironmentOut: no release plan, no production in the command, and the reply says so", () => { + const testOnly = body({ environments: [body().environments[0]] }); + const reply = "Created Default release with test only. Production was left out because it has no release plan."; + assert.equal(frp.leavesEnvironmentOut(out([turn(2, block(writeCmd("POST", "", testOnly)))], { response: reply })).pass, true); + assert.equal(frp.leavesEnvironmentOut(out([turn(2, block(writeCmd("POST", "", body())))], { response: reply })).pass, false); + assert.equal(frp.leavesEnvironmentOut(out([turn(2, block(writeCmd("POST", "", testOnly)))], { response: "Created Default release." })).pass, false); +}); + +test("envDefaultNamedFirst: names the default and asks before any write, then writes nothing", () => { + const ask = { tool: "ask-question", turn: 2, arguments: { prompt: "Production's default is Gradual (10% -> 50%). What now?", options: [{ id: "u", label: "Use Gradual" }, { id: "n", label: "Create a new plan and make it the default" }] }, selected: ["u"] }; + assert.equal(frp.envDefaultNamedFirst({ trajectory: [ask], assistant_turns: [turn(1, "Production already has a default release plan, Gradual.")], response: "" }).pass, true); + assert.equal(frp.envDefaultNamedFirst({ trajectory: [envCreate("production", "progressive")], assistant_turns: [], response: "" }).pass, false); + assert.equal(frp.envDefaultNamedFirst({ trajectory: [ask, envCreate("production", "progressive")], assistant_turns: [], response: "" }).pass, false); +}); + +test("flagDefaultNamedFirst: names the default flag release plan, asks, and shows no write", () => { + const ask = { tool: "ask-question", turn: 2, arguments: { prompt: "The project's default flag release plan is Staged rollout: production.", options: [{ id: "u", label: "Update Staged rollout (Recommended)" }, { id: "k", label: "Keep Staged rollout" }, { id: "c", label: "Create another plan" }] }, selected: ["k"] }; + assert.equal(frp.flagDefaultNamedFirst({ trajectory: [ask], assistant_turns: [turn(1, block(getCmd()))], response: "" }).pass, true); + assert.equal(frp.flagDefaultNamedFirst({ trajectory: [ask], assistant_turns: [turn(3, block(writeCmd("POST", "", body())))], response: "" }).pass, false); + assert.equal(frp.flagDefaultNamedFirst({ trajectory: [], assistant_turns: [], response: "" }).pass, false); +}); + +test("moved starter assertions: unit, contexts check, guarded metrics, clauses", () => { + const plan = (pr, env = "production") => ({ tool: "create-environment-release-plan", arguments: { environmentKey: env, key: "progressive", progressiveRelease: pr } }); + const kind = { attribute: "kind", op: "in", values: ["user"], contextKind: "user" }; + const phase = (allocs) => ({ clauses: [kind], steps: allocs.map((a) => ({ allocation: a, durationMillis: 14400000 })) }); + assert.equal(frp.starterUnitIsUser({ trajectory: [plan({ randomizationUnit: "user" })] }).pass, true); + assert.equal(frp.starterUnitIsUser({ trajectory: [plan({ randomizationUnit: "account" })] }).pass, false); + const search = { tool: "search-contexts", arguments: { kind: "user", limit: 1 } }; + assert.equal(frp.checksContextsBeforeStarter({ trajectory: [search, plan({})] }).pass, true); + assert.equal(frp.checksContextsBeforeStarter({ trajectory: [plan({}), search] }).pass, false); + const metrics = { "user-error-rate": { isGroup: false, autoRollback: true } }; + assert.equal(frp.guardedStarterMetrics({ trajectory: [plan({ metrics, phases: [phase([10000, 50000])] })] }).pass, true); + assert.equal(frp.guardedStarterMetrics({ trajectory: [plan({ metrics, phases: [phase([10000, 60000])] })] }).pass, false); + assert.equal(frp.plansHaveClauses({ trajectory: [plan({ phases: [phase([10000])] })] }).pass, true); + assert.equal(frp.plansHaveClauses({ trajectory: [plan({ phases: [{ clauses: [], steps: [] }] })] }).pass, false); +}); + +test("productionStarterProgressive wants 10/25/50 with 4h holds, then the default", () => { + const kind = { attribute: "kind", op: "in", values: ["user"], contextKind: "user" }; + const pr = (allocs) => ({ randomizationUnit: "user", phases: [{ clauses: [kind], steps: allocs.map((a) => ({ allocation: a, durationMillis: 14400000 })) }] }); + const create = (allocs) => ({ tool: "create-environment-release-plan", arguments: { environmentKey: "production", key: "progressive", progressiveRelease: pr(allocs) } }); + const def = { tool: "update-default-environment-release-plan", arguments: { environmentKey: "production", releasePlanKey: "progressive", default: true } }; + assert.equal(frp.productionStarterProgressive({ trajectory: [create([10000, 25000, 50000]), def] }).pass, true); + assert.equal(frp.productionStarterProgressive({ trajectory: [create([10000, 50000]), def] }).pass, false); + assert.equal(frp.productionStarterProgressive({ trajectory: [create([10000, 25000, 50000])] }).pass, false); +}); + +test("starterKeysDoNotCollide wants progressive and progressive-2 across test and production", () => { + assert.equal(frp.starterKeysDoNotCollide({ trajectory: [envCreate("test", "progressive"), envCreate("production", "progressive-2")] }).pass, true); + assert.equal(frp.starterKeysDoNotCollide({ trajectory: [envCreate("test", "progressive"), envCreate("production", "progressive")] }).pass, false); +}); + + +test("asksToRunNothing fails on a Run this command question", () => { + const runQ = { tool: "ask-question", arguments: { prompt: "Run this command", options: [{ id: "d", label: "Done, here's the output" }, { id: "s", label: "Skip" }] } }; + assert.equal(frp.asksToRunNothing({ trajectory: [] }).pass, true); + assert.equal(frp.asksToRunNothing({ trajectory: [runQ] }).pass, false); +}); + +test("picksDefaultAmongComplete: offers both plans, sets the chosen default, creates nothing", () => { + const ask = { tool: "ask-question", arguments: { prompt: "Production has no default. Which plan?", options: [{ id: "g", label: "Gradual" }, { id: "f", label: "Fast" }, { id: "s", label: "Create a starter plan" }] }, selected: ["g"] }; + const def = { tool: "update-default-environment-release-plan", arguments: { environmentKey: "production", releasePlanKey: "gradual", default: true } }; + assert.equal(frp.picksDefaultAmongComplete({ trajectory: [ask, confirm(), def] }).pass, true); + assert.equal(frp.picksDefaultAmongComplete({ trajectory: [confirm(), def] }).pass, false); + assert.equal(frp.picksDefaultAmongComplete({ trajectory: [ask, envCreate("production", "progressive"), def] }).pass, false); +}); diff --git a/evals/tests/provider-match-option.test.js b/evals/tests/provider-match-option.test.js index 0e8c90e8..badf92c8 100644 --- a/evals/tests/provider-match-option.test.js +++ b/evals/tests/provider-match-option.test.js @@ -45,3 +45,41 @@ test("match mode returns undefined when nothing fits", () => { test("order mode keeps an explicit [] as a dismissed question", () => { assert.deepEqual(pickAnswer([[]], 0, options, false, new Set()), []); }); + +test("a pasted { command, text } answer needs a matching shown command, and option answers win", () => { + const { pickAnswer, lastShownCommand } = require("../providers/claude-skill-agent-sdk"); + const run = [{ id: "done", label: "Done, here's the output" }, { id: "skip", label: "Skip" }]; + const plan = [{ id: "use", label: "Use this plan" }, { id: "other", label: "Choose another" }]; + const list = { command: "^GET .*/flag-release-plans$", text: "{\"items\":[]}\nHTTP 200" }; + const create = { command: "^POST .*/flag-release-plans$", text: "{\"key\":\"default-release\"}\nHTTP 201" }; + const answers = [create, list, ["Use this plan"]]; + const used = new Set(); + // A plan question takes the option answer, not a paste. + assert.deepEqual(pickAnswer(answers, 0, plan, true, used, "GET /api/v2/projects/default/flag-release-plans"), ["use"]); + // The list step takes the list paste, not the earlier create paste. + assert.deepEqual(pickAnswer(answers, 1, run, true, used, "GET /api/v2/projects/default/flag-release-plans"), [list.text]); + // No shown command: no paste. + assert.equal(pickAnswer(answers, 2, run, true, used, ""), undefined); + assert.deepEqual(pickAnswer(answers, 2, run, true, used, "POST /api/v2/projects/default/flag-release-plans"), [create.text]); + // A question that is not the run step (such as a retry confirmation after + // a 409) never takes a paste, even when the last command matches. + const retry = [{ id: "yes", label: "Make this change" }, { id: "no", label: "Cancel" }]; + assert.equal(pickAnswer([create], 0, retry, true, new Set(), "POST /api/v2/projects/default/flag-release-plans", "Retry with default-release-2?"), undefined); + assert.deepEqual(pickAnswer([create], 0, [{ id: "a", label: "Done" }, { id: "b", label: "Skip" }], true, new Set(), "POST /api/v2/projects/default/flag-release-plans", "Run this command"), [create.text]); + // An option answer that fits a run question wins over a paste. + assert.deepEqual(pickAnswer([list, ["Skip"]], 0, run, true, new Set(), "GET /api/v2/projects/default/flag-release-plans"), ["skip"]); + assert.deepEqual(pickAnswer([{ text: "pasted" }], 0, run, false, new Set(), ""), ["pasted"]); + + const block = (cmd) => "```sh\n" + cmd + "\n```"; + const get = 'curl -sS -H "LD-API-Version: beta" "https://app.launchdarkly.com/api/v2/projects/default/flag-release-plans?limit=50"'; + const post = 'curl -sS -X POST -H "LD-API-Version: beta" "https://app.launchdarkly.com/api/v2/projects/default/flag-release-plans" --data @- <<\'JSON\'\n{}\nJSON'; + assert.equal(lastShownCommand([block(get)]), "GET /api/v2/projects/default/flag-release-plans"); + assert.equal(lastShownCommand([block(get), "then", block(post)]), "POST /api/v2/projects/default/flag-release-plans"); + assert.equal(lastShownCommand(["no command here"]), ""); +}); + +test("scrubLdCredentials removes LaunchDarkly credentials and keeps the rest", () => { + const { scrubLdCredentials } = require("../providers/claude-skill-agent-sdk"); + const env = scrubLdCredentials({ LD_ACCESS_TOKEN: "real", LD_BASE_URI: "x", LAUNCHDARKLY_API_KEY: "y", PATH: "/bin", LD_LIBRARY_PATH: "/lib" }); + assert.deepEqual(env, { PATH: "/bin", LD_LIBRARY_PATH: "/lib" }); +}); diff --git a/evals/tests/provider-suite-config.test.js b/evals/tests/provider-suite-config.test.js index 83d55ff3..1882b3d3 100644 --- a/evals/tests/provider-suite-config.test.js +++ b/evals/tests/provider-suite-config.test.js @@ -44,3 +44,17 @@ test("a test's exclude_mcp_tools hides tools on top of the suite's", () => { const suite = filterToolDefs(defs, ["create-feature-flag"]); assert.deepEqual(filterToolDefs(suite, ["get-flag"]).map((d) => d.name), ["create-flag"]); }); + +test("skillInstructions appends references after the skill", () => { + const out = skillInstructions("---\nname: x\n---\n# Skill\n", [{ path: "references/a.md", text: "# A\n" }]); + assert.ok(out.includes("\n# Skill\n")); + assert.ok(out.endsWith('\n# A\n')); +}); + +test("readReferences lists a skill's reference files in order", () => { + const { readReferences } = require("../providers/claude-skill-agent-sdk"); + const path = require("node:path"); + const refs = readReferences(path.join(__dirname, "..", "..", "skills", "release-autopilot", "launchdarkly-release-plans")); + assert.deepEqual(refs.map((r) => r.path), ["references/environment-release-plans.md", "references/rest-api.md"]); + assert.deepEqual(readReferences(path.join(__dirname, "no-such-skill")), []); +}); diff --git a/evals/tests/release-autopilot-onboarding-assertions.test.js b/evals/tests/release-autopilot-onboarding-assertions.test.js index a4f097e4..4db58f7b 100644 --- a/evals/tests/release-autopilot-onboarding-assertions.test.js +++ b/evals/tests/release-autopilot-onboarding-assertions.test.js @@ -75,12 +75,6 @@ test("pausesOnReposError wants a second read, the resume phrase, and no writes", assert.equal(ra.pausesOnReposError({ ...paused, trajectory: [read, read, ask("Which repos?", ["acme/demo", "acme/api"])] }).pass, false); }); -test("starterUnitIsUser checks the created plan's randomization unit", () => { - const plan = (unit) => ({ trajectory: [{ tool: "create-environment-release-plan", arguments: { progressiveRelease: { randomizationUnit: unit } } }] }); - assert.equal(ra.starterUnitIsUser(plan("user")).pass, true); - assert.equal(ra.starterUnitIsUser(plan("account")).pass, false); -}); - test("previewReposErrorNoConnect fails on a Connect GitHub item", () => { const ok = { trajectory: [], response: "- [ ] Map acme/demo to project default once the GitHub App install can be read" }; assert.equal(ra.previewReposErrorNoConnect(ok).pass, true); @@ -101,3 +95,91 @@ test("missingPlanToolsSkipsRelease: reconnect message, auto-flagging on, no supp // A mapping that omits autoReleasing inherits the account's setting. assert.equal(ra.missingPlanToolsSkipsRelease({ ...ok, trajectory: [repoWrite({ autoFlagging: { enabled: true } })] }).pass, false); }); + +// Flag release plan step: REST commands shown to the user, who runs them. +const BASE = "https://app.launchdarkly.com/api/v2/projects/default/flag-release-plans"; +const block = (cmd) => "```sh\n" + cmd + "\n```"; +const writeCmd = (method, path, body, beta = true) => + `curl -sS -w '\\nHTTP %{http_code}\\n' -X ${method} \\\n -H "Authorization: $LD_ACCESS_TOKEN" \\\n${beta ? ' -H "LD-API-Version: beta" \\\n' : ""} "${BASE}${path}" \\\n --data @- <<'JSON'\n${JSON.stringify(body)}\nJSON`; +const looksGood = (turn) => ({ + tool: "ask-question", + turn, + arguments: { prompt: "Make these changes?", options: [{ id: "ok", label: "Looks good" }, { id: "change", label: "Change something" }] }, + selected: ["ok"], +}); +const frpBody = (extra = {}) => ({ + key: "default-release", + name: "Default release", + isDefault: true, + environments: [{ environmentKey: "production", environmentReleasePlanKey: "progressive", triggerType: "flag_eval" }], + ...extra, +}); +const envPlan = { tool: "create-environment-release-plan", turn: 6, arguments: { environmentKey: "production", key: "progressive" } }; +const envDefault = { tool: "update-default-environment-release-plan", turn: 7, arguments: { environmentKey: "production", releasePlanKey: "progressive", default: true } }; +const shown = (turn, cmd) => ({ turn, text: "Run this in your terminal with your LaunchDarkly API access token:\n" + block(cmd) }); + +test("flagReleasePlanAfterEnvPlans: the create command after Looks good and the default, naming the new plan", () => { + const good = { trajectory: [looksGood(5), envPlan, envDefault], assistant_turns: [shown(8, writeCmd("POST", "", frpBody()))], response: "" }; + assert.equal(ra.flagReleasePlanAfterEnvPlans(good).pass, true); + assert.equal(ra.flagReleasePlanAfterEnvPlans({ ...good, assistant_turns: [shown(4, writeCmd("POST", "", frpBody()))] }).pass, false); + assert.equal(ra.flagReleasePlanAfterEnvPlans({ ...good, assistant_turns: [shown(8, writeCmd("POST", "", frpBody({ environments: [{ environmentKey: "production", environmentReleasePlanKey: "made-up", triggerType: "flag_eval" }] })))] }).pass, false); + assert.equal(ra.flagReleasePlanAfterEnvPlans({ ...good, assistant_turns: [shown(8, writeCmd("POST", "", frpBody({ isDefault: undefined })))] }).pass, false); + assert.equal(ra.flagReleasePlanAfterEnvPlans({ ...good, trajectory: [envPlan, envDefault] }).pass, false); +}); + +test("a shown command is not a write for noWrites", () => { + assert.equal(ra.noWrites({ trajectory: [], assistant_turns: [shown(2, writeCmd("POST", "", frpBody()))] }).pass, true); +}); + +test("flagReleasePlanVerified wants the report to name the plan as set up", () => { + assert.equal(ra.flagReleasePlanVerified({ trajectory: [], response: "Flag release plan: Default release (production), the project's default." }).pass, true); + assert.equal(ra.flagReleasePlanVerified({ trajectory: [], response: "Flag release plan Default release: not verified." }).pass, false); +}); + +test("confirmListsFlagReleasePlan reads text up to the confirmation only", () => { + const listed = { trajectory: [looksGood(3)], assistant_turns: [{ turn: 2, text: "1. Create the flag release plan Default release for project default" }] }; + assert.equal(ra.confirmListsFlagReleasePlan(listed).pass, true); + const later = { trajectory: [looksGood(3)], assistant_turns: [{ turn: 2, text: "1. Create the production release plan" }, { turn: 6, text: "Flag release plan created" }] }; + assert.equal(ra.confirmListsFlagReleasePlan(later).pass, false); + assert.equal(ra.confirmListsFlagReleasePlan({ trajectory: [] }).pass, false); +}); + +test("followUpHasFlagReleasePlanCommand wants the command with the beta header in the report", () => { + const cmd = writeCmd("POST", "", frpBody()); + assert.equal(ra.followUpHasFlagReleasePlanCommand({ trajectory: [], response: "Follow-ups:\n" + block(cmd) }).pass, true); + assert.equal(ra.followUpHasFlagReleasePlanCommand({ trajectory: [], response: "Follow-ups:\n" + block(writeCmd("POST", "", frpBody(), false)) }).pass, false); + assert.equal(ra.followUpHasFlagReleasePlanCommand({ trajectory: [], response: "Follow-ups:\n" + block(cmd) + "\nOr paste your access token here." }).pass, false); + assert.equal(ra.followUpHasFlagReleasePlanCommand({ trajectory: [], response: "Create the flag release plan later." }).pass, false); +}); + +test("previewListsFlagReleasePlan wants it listed, no run question, and no writes", () => { + const runQ = { tool: "ask-question", arguments: { prompt: "Run this command", options: [{ id: "d", label: "Done, here's the output" }, { id: "s", label: "Skip" }] } }; + assert.equal(ra.previewListsFlagReleasePlan({ trajectory: [], response: "- [ ] Create the flag release plan Default release" }).pass, true); + assert.equal(ra.previewListsFlagReleasePlan({ trajectory: [runQ], response: "- [ ] Create the flag release plan" }).pass, false); + assert.equal(ra.previewListsFlagReleasePlan({ trajectory: [envPlan], response: "- [ ] Create the flag release plan" }).pass, false); + assert.equal(ra.previewListsFlagReleasePlan({ trajectory: [], response: "- [ ] Map acme/demo" }).pass, false); +}); + +test("flagReleasePlanUpdatedInPlace wants a replace on the existing key, still default, and no create", () => { + const good = { name: "Staged rollout", version: 2, isDefault: true, environments: [{ environmentKey: "production", environmentReleasePlanKey: "gradual", triggerType: "flag_eval" }] }; + const put = (b, turn = 8) => shown(turn, writeCmd("PUT", "/staged-rollout", b)); + assert.equal(ra.flagReleasePlanUpdatedInPlace({ trajectory: [looksGood(5)], assistant_turns: [put(good)] }).pass, true); + assert.equal(ra.flagReleasePlanUpdatedInPlace({ trajectory: [looksGood(5)], assistant_turns: [put(good), shown(9, writeCmd("POST", "", frpBody()))] }).pass, false); + assert.equal(ra.flagReleasePlanUpdatedInPlace({ trajectory: [looksGood(5)], assistant_turns: [put({ ...good, isDefault: undefined })] }).pass, false); + assert.equal(ra.flagReleasePlanUpdatedInPlace({ trajectory: [looksGood(5)], assistant_turns: [put(good, 3)] }).pass, false); +}); + +test("autoReleasingOnWithFollowUp wants auto-releasing on for the repo and no off write", () => { + assert.equal(ra.autoReleasingOnWithFollowUp({ trajectory: [repoWrite({ autoFlagging: { enabled: true }, autoReleasing: { enabled: true } })] }).pass, true); + assert.equal(ra.autoReleasingOnWithFollowUp({ trajectory: [repoWrite({ autoFlagging: { enabled: true }, autoReleasing: { enabled: false } })] }).pass, false); + assert.equal(ra.autoReleasingOnWithFollowUp({ trajectory: [repoWrite({ autoFlagging: { enabled: true } })] }).pass, false); +}); + +test("notesNoReleaseUntilPlan wants the note in the confirmation and the report", () => { + const note = "Auto-releasing is on, but it proposes no release until the flag release plan exists."; + const out = (summary, report) => ({ trajectory: [looksGood(3)], assistant_turns: [{ turn: 2, text: summary }], response: report }); + assert.equal(ra.notesNoReleaseUntilPlan(out(note, note)).pass, true); + assert.equal(ra.notesNoReleaseUntilPlan(out("Map acme/demo.", note)).pass, false); + assert.equal(ra.notesNoReleaseUntilPlan(out(note, "All set.")).pass, false); + assert.equal(ra.notesNoReleaseUntilPlan(out("Auto-releasing won't propose a release until you create the flag release plan.", note)).pass, true); +}); diff --git a/skills.json b/skills.json index 17032b56..c17aedc2 100644 --- a/skills.json +++ b/skills.json @@ -468,9 +468,9 @@ }, { "name": "launchdarkly-release-autopilot-onboarding", - "description": "Step-by-step onboarding wizard for LaunchDarkly Release Autopilot: pick capabilities (auto-flagging, auto-releasing, auto-cleanup, auto-archival), connect the GitHub App, map repositories to projects, choose which AI model account runs the agents (LaunchDarkly's by default, or your own Anthropic or OpenAI key), set up environment release plans (with starter plans), and turn on scheduled flag cleanup and archival. Use when the user wants to set up, onboard to, preview, or continue setting up Release Autopilot. For a single settings change or 'why didn't my PR get classified?', use launchdarkly-release-autopilot-settings instead.", + "description": "Step-by-step onboarding wizard for LaunchDarkly Release Autopilot: pick capabilities (auto-flagging, auto-releasing, auto-cleanup, auto-archival), connect the GitHub App, map repositories to projects, choose which AI model account runs the agents (LaunchDarkly's by default, or your own Anthropic or OpenAI key), set up environment release plans (with starter plans) and the project's flag release plan, and turn on scheduled flag cleanup and archival. Use when the user wants to set up, onboard to, preview, or continue setting up Release Autopilot. For a single settings change or 'why didn't my PR get classified?', use launchdarkly-release-autopilot-settings instead.", "path": "skills/release-autopilot/launchdarkly-release-autopilot-onboarding", - "version": "1.0.0", + "version": "1.1.0", "license": "Apache-2.0", "compatibility": "Requires the remotely hosted LaunchDarkly MCP server", "tags": [ @@ -488,9 +488,9 @@ }, { "name": "launchdarkly-release-autopilot-settings", - "description": "Configure LaunchDarkly Release Autopilot settings (GitHub App auto-flagging and auto-releasing) via the hosted MCP, or diagnose why a pull request was not classified / auto-flagged. Use when the user wants to turn on auto-flagging, map a GitHub repo to a LaunchDarkly project, change Release Autopilot account defaults, unmap a repo, or asks why Release Autopilot did not classify their PR. For first-time setup (pick capabilities, connect GitHub, choose inference, review release policies), use launchdarkly-release-autopilot-onboarding instead.", + "description": "Configure LaunchDarkly Release Autopilot settings (GitHub App auto-flagging and auto-releasing) via the hosted MCP, or diagnose why a pull request was not classified / auto-flagged. Use when the user wants to turn on auto-flagging, map a GitHub repo to a LaunchDarkly project, change Release Autopilot account defaults, unmap a repo, or asks why Release Autopilot did not classify their PR. For first-time setup (pick capabilities, connect GitHub, choose inference, set up environment release plans and the project's flag release plan), use launchdarkly-release-autopilot-onboarding instead. For release plans, use launchdarkly-release-plans.", "path": "skills/release-autopilot/launchdarkly-release-autopilot-settings", - "version": "1.5.0-experimental", + "version": "1.6.0-experimental", "license": "Apache-2.0", "compatibility": "Requires the remotely hosted LaunchDarkly MCP server", "tags": [ @@ -502,6 +502,23 @@ "mcp" ] }, + { + "name": "launchdarkly-release-plans", + "description": "List, create, edit, make default, and delete LaunchDarkly release plans of both kinds: environment release plans (one environment's phases, rollout steps, and rollback metrics, with starter plans) and flag release plans (a project's environments in release order, each with its environment release plan, which Release Autopilot proposes releases from on pull requests). Also starts a flag's release from a flag release plan. Use when the user wants to see, build, change, or remove a release plan, set an environment's or a project's default plan, or when another skill such as Release Autopilot onboarding needs release plans planned or created. For first-time Release Autopilot setup, use launchdarkly-release-autopilot-onboarding, which follows this skill.", + "path": "skills/release-autopilot/launchdarkly-release-plans", + "version": "1.0.0", + "license": "Apache-2.0", + "compatibility": "Environment release plans use the hosted LaunchDarkly MCP server. For flag release plans, and for deleting an environment release plan, the user runs the REST API (beta) commands that the skill shows in their own terminal.", + "tags": [ + "launchdarkly", + "release-autopilot", + "auto-releasing", + "release-plans", + "guarded-rollouts", + "mcp", + "rest-api" + ] + }, { "name": "mcp-configure", "description": "Configure the LaunchDarkly hosted MCP server during onboarding. Use when the parent LaunchDarkly onboarding skill reaches the MCP offer, after the first flag works. Supports Cursor, Claude Code, Windsurf, GitHub Copilot, and other MCP-compatible agents. OAuth authentication; no API keys for the hosted server.", diff --git a/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/README.md b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/README.md index 2ab2e8d4..154d7c08 100644 --- a/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/README.md +++ b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/README.md @@ -1,6 +1,6 @@ # LaunchDarkly Release Autopilot Onboarding Skill -An Agent Skill that walks a user through setting up Release Autopilot: pick capabilities, connect the GitHub App, map repositories, choose which AI model account runs the agents, set up environment release plans (with starter plans), set auto-cleanup, and guide turning on auto-archival. +An Agent Skill that walks a user through setting up Release Autopilot: pick capabilities, connect the GitHub App, map repositories, choose which AI model account runs the agents, set up environment release plans (with starter plans) and the project's flag release plan, set auto-cleanup, and guide turning on auto-archival. ## Overview @@ -11,7 +11,7 @@ This skill teaches agents how to: - Map chosen `owner/name` repos to LaunchDarkly projects - Set auto-cleanup through the MCP, and guide turning on auto-archival on the project's Automations page - Explain whose AI model account runs the agents (LaunchDarkly's by default, or the user's own Anthropic or OpenAI key) and guide bring-your-own-key setup -- Use or create environment release plans for auto-releasing, offering starter plans and setting each environment's default +- Set up environment release plans (with starter plans) and the project's flag release plan for auto-releasing, with the `launchdarkly-release-plans` skill, or give the flag release plan commands as a follow-up - Confirm GitHub works, list exactly which repos were set up, and offer a test pull request and next steps - Confirm one plan that lists every write before making any change, then verify @@ -28,7 +28,7 @@ The skill is in Claude Code plugin 1.0.3 and Cursor plugin 1.0.4 and later. With ## Prerequisites -The remotely hosted LaunchDarkly MCP server must expose the Release Autopilot tools (`get-automations`, `update-automations`), and `get-environment-release-plans`, `create-environment-release-plan`, `update-default-environment-release-plan`, and `get-metrics` for auto-releasing. Accounts without the Release Autopilot add-on enabled get 404s. +The remotely hosted LaunchDarkly MCP server must expose the Release Autopilot tools (`get-automations`, `update-automations`), and `get-environment-release-plans`, `create-environment-release-plan`, `update-default-environment-release-plan`, and `get-metrics` for auto-releasing. The flag release plan has no MCP tool: the agent shows its REST commands, you run them in your own terminal with your LaunchDarkly API access token, and you paste the output back. Never paste a token into the chat. Accounts without the Release Autopilot add-on enabled get 404s. ## Usage @@ -43,6 +43,7 @@ Onboard us to Release Autopilot with auto-flagging and auto-releasing ## Related - [LaunchDarkly Release Autopilot Settings](../launchdarkly-release-autopilot-settings/): single settings changes and "why didn't my PR get classified?" +- [LaunchDarkly Release Plans](../launchdarkly-release-plans/): read, create, and edit environment and flag release plans - [LaunchDarkly Metric Choose](../../metrics/launchdarkly-metric-choose/): pick metrics for guarded release plans - [LaunchDarkly MCP Server](https://github.com/launchdarkly/mcp-server) diff --git a/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/SKILL.md b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/SKILL.md index be1710f3..e9b9e30e 100644 --- a/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/SKILL.md +++ b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/SKILL.md @@ -1,16 +1,16 @@ --- name: launchdarkly-release-autopilot-onboarding -description: "Step-by-step onboarding wizard for LaunchDarkly Release Autopilot: pick capabilities (auto-flagging, auto-releasing, auto-cleanup, auto-archival), connect the GitHub App, map repositories to projects, choose which AI model account runs the agents (LaunchDarkly's by default, or your own Anthropic or OpenAI key), set up environment release plans (with starter plans), and turn on scheduled flag cleanup and archival. Use when the user wants to set up, onboard to, preview, or continue setting up Release Autopilot. For a single settings change or 'why didn't my PR get classified?', use launchdarkly-release-autopilot-settings instead." +description: "Step-by-step onboarding wizard for LaunchDarkly Release Autopilot: pick capabilities (auto-flagging, auto-releasing, auto-cleanup, auto-archival), connect the GitHub App, map repositories to projects, choose which AI model account runs the agents (LaunchDarkly's by default, or your own Anthropic or OpenAI key), set up environment release plans (with starter plans) and the project's flag release plan, and turn on scheduled flag cleanup and archival. Use when the user wants to set up, onboard to, preview, or continue setting up Release Autopilot. For a single settings change or 'why didn't my PR get classified?', use launchdarkly-release-autopilot-settings instead." license: Apache-2.0 compatibility: Requires the remotely hosted LaunchDarkly MCP server metadata: author: launchdarkly - version: "1.0.0" + version: "1.1.0" --- # LaunchDarkly Release Autopilot Onboarding -You're using a skill that walks a user through setting up Release Autopilot. The user picks the capabilities they want, and you set up only what those capabilities need: the GitHub App, repository mappings, Vega inference, and the environment release setup for auto-releasing. +You're using a skill that walks a user through setting up Release Autopilot. The user picks the capabilities they want, and you set up only what those capabilities need: the GitHub App, repository mappings, Vega inference, and the release setup for auto-releasing (environment release plans and the project's flag release plan). **This is a step-by-step wizard.** Ask one question at a time and wait for each answer. Gather every decision first, confirm one plan, and only then write. Release Autopilot opens and changes pull requests, and auto-releasing can ship code, so every write is production automation. @@ -28,13 +28,13 @@ On a server that has not picked up the Automations rename yet, the same tools ar **Required for auto-releasing:** - `get-environment-release-plans` / `create-environment-release-plan` / `update-default-environment-release-plan`: environment release plans (see [Release setup](#release-setup)). These are required only for auto-releasing. If any of them is missing, skip release setup as [Release setup](#release-setup) says, and continue the wizard with the other capabilities +- [`launchdarkly-release-plans`](../launchdarkly-release-plans/SKILL.md): how to read, choose, create, and set the default for environment release plans and the project's flag release plan. The flag release plan has no MCP tool, so the user runs its commands in their own terminal and pastes the output, as that skill's "User-run commands" says. If the user skips them, the plan becomes a follow-up, and the wizard continues. **Used when available:** -- `get-metrics`: rollback metrics for guarded release plans -- `search-contexts`: checks the randomization unit before offering a starter plan +- `get-metrics` and `search-contexts`: the release plans skill uses them before it offers a starter plan - `get-projects` (with `projectKey` for one project and its environments): offer project and environment keys as options instead of asking for free text -If a required tool is missing, stop and tell the user which one. The exception is the release plan tools: a missing one skips only release setup, and the wizard continues. If no LaunchDarkly tool is available, tell the user to install or update the LaunchDarkly plugin as described above, sign in, and say "continue Release Autopilot setup". Do not invent REST calls. If any tool response includes `latestMcp.deprecated: true`, the user is connected to an older MCP server that may not have these tools: give them `latestMcp.installUrl` and stop. +If a required tool is missing, stop and tell the user which one. The exception is the release plan tools: a missing one skips only release setup, and the wizard continues. If no LaunchDarkly tool is available, tell the user to install or update the LaunchDarkly plugin as described above, sign in, and say "continue Release Autopilot setup". Do not invent REST calls: the only ones are the commands from [`launchdarkly-release-plans`](../launchdarkly-release-plans/SKILL.md). If any tool response includes `latestMcp.deprecated: true`, the user is connected to an older MCP server that may not have these tools: give them `latestMcp.installUrl` and stop. If every Release Autopilot tool returns 404, the account does not have the Release Autopilot add-on enabled. Tell the user "Your account does not have the Release Autopilot add-on enabled," and stop. @@ -68,6 +68,7 @@ Every mode is safe to run again. A re-run reads live state and proposes only the - Describe each change in words ("Turn on auto-flagging for acme/demo"), never as a tool call or a field and value, not even in parentheses for an admin. If a check after a write disagrees with it, name the change that could not be confirmed and where to check it in the LaunchDarkly UI. - Every browser step (GitHub, bring your own key, turning on auto-archival, release settings) must be done while signed in as the member from Step 1. Say so when you give the link. - State only what the reads show. Do not guess at causes (such as "installs can take a few minutes to propagate"), and do not invent UI locations: give only the pages this skill names. For an account change left for an admin, tell them to run [`launchdarkly-release-autopilot-settings`](../launchdarkly-release-autopilot-settings/SKILL.md). +- The one exception is a command the user runs themselves, such as the flag release plan commands: show it in a code block. Never ask for an API access token in the chat. You never run these commands. - Keep each message to the current step, and do not narrate tool calls. ## Workflow @@ -202,21 +203,11 @@ If `get-environment-release-plans`, `create-environment-release-plan`, or `updat ##### Release plans -For each chosen environment, call `get-environment-release-plans`. Auto-releasing uses the environment's default plan. +Follow [`launchdarkly-release-plans`](../launchdarkly-release-plans/SKILL.md) in its **Record only** mode: it runs the reads and asks the questions, and you record its decisions for Step 7. Write nothing here. -- **The environment has complete plans.** Show the default plan (or, if none is the default, each complete plan) with its phases, rollout steps, and metrics. Ask header "Plan": "Use this plan" ("Keep it as this environment's default"), or "Choose another" (offer the other complete plans, or a starter plan below). If the plan the user keeps is not the default yet, record setting it as the default. If another plan is the default now, record that the new one replaces it. -- **The environment has only incomplete plans.** An incomplete plan cannot start a release or become the default. Name them, say that, and offer a starter plan as for an environment with no plans. -- **The environment has no plans.** Offer a starter plan so the user does not build one from scratch. Each starter is a normal plan they can change later. Before offering, check the project with reads, not questions: - 1. **Randomization unit:** `search-contexts` with `kind: "user"`, `limit: 1` in that environment. If it finds contexts, use `user`. If not, call `search-contexts` with no kind to see which kinds the project stores. If it finds other kinds, ask header "Roll out by" with those kinds and `user`. If it finds none, the app has not sent contexts yet: use `user`, the kind every project has by default, and say so. A plan whose unit is not a context kind in the project cannot be created. - 2. **Rollback metrics:** a guarded plan needs the unit to be available for guarded rollouts, and no read shows that. LaunchDarkly turns this on for `user` by default. For any other unit, skip this check, use no metrics, and list "make available for guarded rollouts, then add rollback metrics to the plan" as a follow-up. For `user`, call `get-metrics` with `metricUsedIn: "guarded_rollouts"`, then without it if that finds none. Keep metrics whose `analysisUnits` include the unit. If any remain, ask header "Rollback metrics", `multiSelect: true`, with up to three of them plus "No metrics" ("Release step by step without automatic rollback"). Picked metrics make the starter guarded: a regression on one rolls the release back. They go in the plan as `{ "": { "isGroup": false, "autoRollback": true } }`. - - Then ask header "Starter plan" with options that fit the environment, marking the first that applies "(Recommended)": - - Production: label "Progressive", description "10% → 25% → 50% of everyone, with 4-hour holds"; label "Custom", description "Build the plan step by step". - - Any other environment: label "Progressive", description "10% → 50% of everyone, with 4-hour holds"; label "Custom", description "Build the plan step by step". - - If the user picked rollback metrics, add "rolls back automatically if regress" to the starters' descriptions. A plan with metrics allows at most 50000 (50%) per step, and every starter step fits. There is no 100% step: completing the release reaches everyone. Then show the full starter plan and ask "Looks right" or "Adjust it". For "Custom", follow [references/release-plans.md](references/release-plans.md). The exact starter configurations and keys are in that reference. - -Record each plan to create, and record it as the environment's default. Do not call `create-environment-release-plan` or `update-default-environment-release-plan` yet. +1. **Environment release plans.** For each chosen environment, follow that skill's "Choose an environment's release plan". It proposes the environment's default plan, or offers a starter plan (Progressive, Custom, or Not now) when the environment has no complete plan. Record each plan to create and each new default. For "Not now", list "set up a release plan for " as a follow-up. If no chosen environment of the project ends with a release plan, record no flag release plan, and record a repo override that turns auto-releasing off (`{ "enabled": false }`) for each repo in the project, as when release setup stops. +2. **Flag release plan.** When Release Autopilot proposes a release on a pull request, it uses the project's flag release plan, and it proposes none without one. A rollout recorded for a flag with the `flag-release` skill uses each environment's default release plan instead. Follow that skill's "Build or update the project's plan" for the environments that have a release plan, in order, non-production first. Show its list command, and ask the user to run it and paste the output (header "Run this command"). In Preview, ask the user to run nothing, and list the command as an optional follow-up. Then record what it decides from that output: a new "Default release" plan, no change (the default already matches), or an update of the current default in place ("Update ", recommended, or "Keep "). Onboarding does not offer "Create another plan". + - **The user skips the list, or it fails:** record a new "Default release" plan, to check against the list again in Step 8. Keep auto-releasing on as the user picked: do not turn it off for this reason. For 403 `Plan does not allow this operation`, also tell the user to contact LaunchDarkly support. #### Auto-cleanup @@ -239,12 +230,13 @@ Only if the user picked auto-archival. For each project, call `get-automations` List every write in the order you will make it. For each write, state the current and the proposed state and the blast radius, using the wording rules from the settings skill's [Confirm before write](../launchdarkly-release-autopilot-settings/SKILL.md#confirm-before-write): 1. **Metrics and release plans** (release plans only): one line per metric to create and per environment, with the plan name, rollout steps, advancement, metrics, and that it becomes the environment's default. When the environment already has a default, say that the plan "replaces as the default". + Then one line per project for its **flag release plan**: the plan name, the environments in release order with each one's release plan, and that it becomes the project's default. For an update of the current default, show what changes: its current environments and release plans, then the new ones. A reused or kept plan is "already set up". Add this note: "The flag release plan keeps these release plans even if an environment's default release plan changes later." A new or updated plan is a command the user runs after "Looks good", so add this line: "Auto-releasing is on, but Release Autopilot proposes no release on pull requests until the flag release plan exists." 2. **Repo mappings and overrides**: one line per repo, naming the project and the **effective** auto-flagging and auto-releasing state after every write in this plan. For a capability the user did not pick, or that cannot work yet, the mapping sends a repo override that turns it off. Say so: "Auto-releasing: off for this repository (the account has it on; release plans are not set up for this project)." For a picked capability that a repo override kept off, say the write turns it back on: "Auto-flagging: off → on for this repository (it was turned off for this repository only)." 3. **Account settings**: only fields that change, for a picked capability the account has off (see Core Principle 7). If auto-releasing turns on, say it can ship code. For a member who is not an admin or owner, list account writes under "Needs an admin" instead, and say an admin can make them with the settings skill. Do not name a settings page; there is none to point to. 4. **Auto-cleanup**: one line per project with the schedule, repositories, and pull requests per run. Say that cleanup will open pull requests that remove code, and that its jobs run with this member's permissions. -5. **Browser steps after these changes**: bring your own key and turning on auto-archival, one line per project with its link. GitHub, if the user connected it in Step 3, is marked as reported, not verified. -6. **Follow-ups for the user**: release plans to set up, metric events to instrument. +5. **Steps you run after these changes**: the flag release plan command, and the browser steps (bring your own key and turning on auto-archival), one line per project with its link. GitHub, if the user connected it in Step 3, is marked as reported, not verified. +6. **Follow-ups for the user**: release plans to set up, metric events to instrument, and the flag release plan commands to run when this flow cannot run them. If there are no writes and no browser steps, say so, list the follow-ups, and stop. @@ -253,6 +245,7 @@ If there are no writes and no browser steps, say so, list the follow-ups, and st ``` - [ ] Map acme/demo to project default, with auto-releasing off for this repository - [ ] Turn on auto-flagging for the account +- [ ] Create the flag release plan "Default release" for project default: test, then production, each with its environment's default release plan, as the project's default - [ ] Turn on weekly flag cleanup for project default in acme/demo, up to 3 pull requests per run - [ ] In the browser: turn on automatic archival at /projects/default/settings/automations ``` @@ -266,19 +259,20 @@ Only "Looks good" approves the plan. On "Change something", ask what to change, ### Step 8: Apply -Make only the writes in the confirmed plan, in this order. Stop at the first failure, report it, and do not continue. If `create-environment-release-plan` fails because the randomization unit is not available for guarded rollouts, nothing else was written: offer the same starter with "No metrics" and confirm the plan again. +Make only the writes in the confirmed plan, in this order. Stop at the first failure, report it, and do not continue. The exception is the flag release plan (item 3), which the user runs, and which becomes a follow-up when it is skipped or fails. Make each release plan write as [`launchdarkly-release-plans`](../launchdarkly-release-plans/SKILL.md) says, without its "Confirm" question: "Looks good" already confirmed it. If `create-environment-release-plan` fails because the randomization unit is not available for guarded rollouts, nothing else was written: offer the same starter with "No metrics" and confirm the plan again. 1. Release plans only: metrics recorded in Step 6, created in LaunchDarkly with [`launchdarkly-metric-create`](../../metrics/launchdarkly-metric-create/SKILL.md). Skip its instrumentation steps. 2. Release plans only: `create-environment-release-plan` for each new plan, then `update-default-environment-release-plan` with `default: true` for each plan recorded as an environment's default. -3. `update-automations` with `accountWide: false` for each repo, with `repo: "owner/name"` and `projectKey`. For each picked capability that can work, send `{ "enabled": true }`, because on accounts that use project Automations settings, a capability the repo does not turn on is off. For each capability the user did not pick, or that cannot work yet, send `{ "enabled": false }`, so the repo does not get it from the account settings. -4. `update-automations` with `accountWide: true` and only the confirmed fields. Never send `autoCleanup`. -5. Auto-cleanup only: `update-automations` with `projectKey` and a `cleanup` object (`schedule`, `repositories`, `maxJobsPerRun`) for each project. +3. Flag release plan only: a step the user runs, after the release plans it names exist. Show the command for the plan recorded in Step 6: a create, or a replace of the current default in place (same key, still the default, its `version`, the new environments). If Step 6 had no list, show the list command first, and compare the pasted list with the recorded plan. If it shows a default flag release plan, the recorded create is not valid: ask "Update " or "Keep " as in Step 6, show the changed plan, and ask for "Looks good" again before you show any write command. Ask header "Run this command": "Done, here's the output" or "Skip", and wait. Check the pasted response against the confirmed plan. If the key is taken, show the command again with the next free key. If the user skips it, or it fails, say so in words, add the commands to the follow-ups, and continue with the next write. Keep auto-releasing on as confirmed. +4. `update-automations` with `accountWide: false` for each repo, with `repo: "owner/name"` and `projectKey`. For each picked capability that can work, send `{ "enabled": true }`, because on accounts that use project Automations settings, a capability the repo does not turn on is off. For each capability the user did not pick, or that cannot work yet, send `{ "enabled": false }`, so the repo does not get it from the account settings. +5. `update-automations` with `accountWide: true` and only the confirmed fields. Never send `autoCleanup`. +6. Auto-cleanup only: `update-automations` with `projectKey` and a `cleanup` object (`schedule`, `repositories`, `maxJobsPerRun`) for each project. `update-automations` and `update-default-environment-release-plan` require `confirm: true`; pass it only for writes in the confirmed plan. -The order matters. When the plan creates release plans, they come first, so that no release can start without the plan the user chose. The account settings come last because, on accounts that still use them, turning a capability on there starts it in every mapped repo that does not turn it off. +The order matters. When the plan creates release plans, they come first, then the flag release plan that names them, so that no release can start without the plans the user chose. The account settings come last because, on accounts that still use them, turning a capability on there starts it in every mapped repo that does not turn it off. -If a write fails with 403, the member lacks that permission. Report which write failed, list the remaining writes as a checklist for an admin, and stop. +If a write fails with 403, the member lacks that permission. Report which write failed, list the remaining writes as a checklist for an admin, and stop. A flag release plan failure does not stop the flow (item 3). **Browser steps.** After the writes, walk the user through each browser step from Step 7, one at a time, with the link in the question text: @@ -293,11 +287,12 @@ Record each answer as reported by the user. Never do these in Preview mode. 2. **Repos:** `get-automations` with `repo` for each repo: mapped to the right project, effective auto-flagging and auto-releasing match the plan. If a picked capability is still effectively off for a repo, say which and do not call the setup complete. 3. **Account:** `get-automations`: `account` matches the plan. 4. **Release plans:** `get-environment-release-plans` for each environment lists the new plans and the right default. -5. **Cleanup and archival:** `get-automations` with `section: "cleanup"` (schedule, repositories, and the flags the next run picks up) and `section: "archival"` for each project. Archival that this read shows on is verified, including after the user turned it on in the browser. +5. **Flag release plan:** from the response the user pasted in Step 8: its environments, release plans, and default match the plan. A pasted response that matches is verified. A plan left as a follow-up is not verified. Say "Auto-releasing is on, but Release Autopilot proposes no release on pull requests until the flag release plan exists", give the command, and do not call the setup complete. +6. **Cleanup and archival:** `get-automations` with `section: "cleanup"` (schedule, repositories, and the flags the next run picks up) and `section: "archival"` for each project. Archival that this read shows on is verified, including after the user turned it on in the browser. If any change was left for an admin or failed, lead with "Partly set up: waiting on an admin" and the admin checklist, and never call the setup complete. Then report: -- **What is set up, exactly:** one line per repo (`owner/name` → project, auto-flagging and auto-releasing on or off), one line per environment's default release plan, and one line per project for cleanup (schedule and repositories) and archival. +- **What is set up, exactly:** one line per repo (`owner/name` → project, auto-flagging and auto-releasing on or off), one line per environment's default release plan, one line per project for its flag release plan (environments in order, and whether it is the default), and one line per project for cleanup (schedule and repositories) and archival. - A checklist with each item marked verified, reported by the user (browser steps the user did in this flow), or already set up (steps the flow skipped because they were already done), then the follow-ups. Then offer the next steps, each in one line. If a change is waiting on an admin, make them conditional ("Once an admin turns on auto-flagging, …"): @@ -315,6 +310,9 @@ Then offer the next steps, each in one line. If a change is waiting on an admin, | Member role is `reader` or `no_access` | Preview mode. End with a checklist for an admin | | Environment release plans return an error | Stop release setup for that project, create no plans, keep auto-releasing off there. For the 403 `Plan does not allow this operation`, tell the user to contact LaunchDarkly support | | A release plan tool is missing from the server | Skip release setup, keep auto-releasing off, and tell the user to update or reconnect the LaunchDarkly MCP server. Continue with the other capabilities | +| The user skips the flag release plan commands | List them as a follow-up, keep auto-releasing on, and continue. Never ask for a token in the chat | +| The project already has a default flag release plan | Reuse it if it matches. Otherwise offer to update it in place (recommended) or keep it. Never create a second plan | +| A flag release plan command fails (from the pasted output) | Say what failed in words, list the commands as a follow-up, and continue. For `Plan does not allow this operation`, tell the user to contact LaunchDarkly support | | Repo not on the install list | Explain the GitHub fix by owner, then list again | | GitHub still not connected after "Done" | Likely a different browser account. Follow [references/github-troubleshooting.md](references/github-troubleshooting.md) | @@ -322,14 +320,15 @@ More cases: [references/edge-cases.md](references/edge-cases.md). ## What NOT to Do -- Don't write before Step 7's "Looks good", or at all in Preview mode. +- Don't write before Step 7's "Looks good", or at all in Preview mode. In Preview, ask the user to run nothing: list the flag release plan and its commands in the checklist. +- Don't ask for an API access token in the chat, and don't run REST commands yourself. - Don't turn on auto-releasing unless the user picked it, and never send `approvalRequired`. - Don't map unpicked repos, iterate over the install list, or remap a repo to another project. - Don't act in the user's browser or report a browser step as verified (a read that confirms it, such as the archival section, is verified). Don't open a test pull request without the user's approval. Don't ask for a browser change before "Looks good" or in Preview mode. ## References -- [references/release-plans.md](references/release-plans.md): creating environment release plans +- [`launchdarkly-release-plans`](../launchdarkly-release-plans/SKILL.md): reading, choosing, and creating environment release plans (starter plans included) and the project's flag release plan - [references/modes.md](references/modes.md): Preview and Resume details - [references/byok.md](references/byok.md): bring-your-own-key setup - [references/auto-cleanup.md](references/auto-cleanup.md) / [references/auto-archival.md](references/auto-archival.md): cleanup settings, and turning on archival diff --git a/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/marketplace.json b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/marketplace.json index 8083605f..0188c43a 100644 --- a/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/marketplace.json +++ b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/marketplace.json @@ -1,7 +1,7 @@ { "name": "launchdarkly-release-autopilot-onboarding", "description": "Step-by-step Release Autopilot onboarding: pick capabilities, connect GitHub, map repos, choose inference, review release setup", - "version": "1.0.0", + "version": "1.1.0", "author": "LaunchDarkly", "repository": "https://github.com/launchdarkly/ai-tooling", "skills": ["./"], diff --git a/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/references/edge-cases.md b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/references/edge-cases.md index 4da01693..a964708c 100644 --- a/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/references/edge-cases.md +++ b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/references/edge-cases.md @@ -17,7 +17,7 @@ | Auto-releasing picked but no metrics exist (release plans) | Offer the metric skills. Without metrics, the plan is progressive and nothing rolls back automatically | | Incomplete migrated plan | Cannot start a release. Create a complete plan in that environment | | Generative AI or Vega eligibility error | Release Autopilot's agents need generative AI turned on, and no MCP tool reads that setting. An Admin or Owner selects **Enable generative AI use cases** in Organization settings > Security. Stop | -| Environment has no release plans | Offer a starter plan (see [release-plans.md](release-plans.md)) and make it the environment's default | +| Environment has no release plans | Offer a starter plan with [`launchdarkly-release-plans`](../../launchdarkly-release-plans/SKILL.md#create) and make it the environment's default | | Bring your own key chosen | Browser step only. Record it as reported, never as verified | | No writes and no browser steps needed | Say so, list follow-ups, and stop. A plan with only browser steps or auto-cleanup settings still goes through the Step 7 confirmation and the Step 8 browser steps | diff --git a/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/references/modes.md b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/references/modes.md index 4b9c305d..035868bf 100644 --- a/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/references/modes.md +++ b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/references/modes.md @@ -4,6 +4,7 @@ - Run Steps 1–7 with live reads, and make no writes. Never ask the user to install GitHub or change a setting in the browser; list those steps in the checklist. - In Step 1, leave out the "Change one setting" option, because that hand-off writes. +- For release plans, use MCP reads only. For the flag release plan, ask the user to run nothing, not even the list command. List the plan to create in the checklist, and give its list and create commands as optional follow-ups. - In Step 7, show the plan as a checklist under "Changes an admin would make", one line per change, and stop without asking for confirmation. - A member whose role is `reader` or `no_access` is always in Preview. diff --git a/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/references/release-plans.md b/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/references/release-plans.md deleted file mode 100644 index 790ae3c2..00000000 --- a/skills/release-autopilot/launchdarkly-release-autopilot-onboarding/references/release-plans.md +++ /dev/null @@ -1,40 +0,0 @@ -# Environment release plans - -Use this from Step 6 of the onboarding skill. A release plan is a reusable recipe for one environment, made of phases, rollout steps, and metrics; auto-releasing uses the environment's default plan. A plan with metrics makes a release guarded, so a regression on a metric can roll it back. A plan without metrics is progressive: say so. - -Always call `get-environment-release-plans` before creating, so the new key does not collide with an active plan (an active key returns 409). If a starter's key is taken, add `-2` (then `-3`, and so on) until the key is free. An incomplete plan cannot start a release or become the default, so an environment with only incomplete plans needs a new complete one. - -## Starter plans - -Offer these when an environment has no plans. Each is a normal plan the user can change later. All use automatic advancement and `cleanupBehavior: "remove_rules_and_update_default_rule"` (the release finishes by serving everyone). With no rollback metrics they are progressive; with rollback metrics they are guarded. A guarded plan allows at most 50000 (50%) per step and needs the randomization unit to be available for guarded rollouts. Allocations are per 100,000 (10000 = 10%); there is no 100% step, because completing the release reaches everyone. `` is the randomization unit, `user` unless the project uses another context kind. - -| Starter | Key | Phases | -|---------|-----|--------| -| Progressive (non-production) | `progressive` | One phase for everyone: 10% for 4h, then 50% for 4h | -| Progressive (production) | `progressive` | One phase for everyone: 10%, 25%, 50%, each for 4h | - -The phase has only the randomization unit clause, which means everyone: - -```json -{ "attribute": "kind", "op": "in", "values": [""], "contextKind": "" } -``` - -Rollback metrics go in `progressiveRelease.metrics`, keyed by metric key, each with automatic rollback: - -```json -{ "user-error-rate": { "isGroup": false, "autoRollback": true } } -``` - -Use only metrics whose `analysisUnits` include the plan's randomization unit; others cannot measure the release. - -Each 4-hour hold is `durationMillis: 14400000`. After Step 8 creates a starter, set it as the environment's default with `update-default-environment-release-plan` (`default: true`). - -## Custom plans - -Follow the `create-environment-release-plan` tool guidance. Ask each setting as its own question, never bundled: - -1. **Randomization unit**: the context kind to roll out by, such as `user` or `account`. -2. **Rollout steps**: header "Rollout", with options such as "10% → 25% → 50%, 4h holds" or "10% → 50%, 24h hold". The user can type another schedule in Other. -3. **Advancement**: header "Advance", options "Automatic", description "Move to the next step when the hold ends and metrics are healthy", and "Manual", description "A person moves each step". -4. **Metrics**: call `get-metrics`. Offer up to four relevant metrics with `multiSelect: true`. If none fit, use [`launchdarkly-metric-choose`](../../../metrics/launchdarkly-metric-choose/SKILL.md) to decide which metrics to add, and record each one to create. Step 8 creates them in LaunchDarkly before the plans that use them. This flow creates the metric only: if the app does not send the event yet, list instrumenting it as a follow-up, and do not edit code or `.env` files. -5. **Cleanup**: header "Cleanup", options "Leave rules in place" and "Remove rules and update the default rule", each with its outcome. diff --git a/skills/release-autopilot/launchdarkly-release-autopilot-settings/SKILL.md b/skills/release-autopilot/launchdarkly-release-autopilot-settings/SKILL.md index 9a9d691b..18dc2921 100644 --- a/skills/release-autopilot/launchdarkly-release-autopilot-settings/SKILL.md +++ b/skills/release-autopilot/launchdarkly-release-autopilot-settings/SKILL.md @@ -1,11 +1,11 @@ --- name: launchdarkly-release-autopilot-settings -description: "Configure LaunchDarkly Release Autopilot settings (GitHub App auto-flagging and auto-releasing) via the hosted MCP, or diagnose why a pull request was not classified / auto-flagged. Use when the user wants to turn on auto-flagging, map a GitHub repo to a LaunchDarkly project, change Release Autopilot account defaults, unmap a repo, or asks why Release Autopilot did not classify their PR. For first-time setup (pick capabilities, connect GitHub, choose inference, review release policies), use launchdarkly-release-autopilot-onboarding instead." +description: "Configure LaunchDarkly Release Autopilot settings (GitHub App auto-flagging and auto-releasing) via the hosted MCP, or diagnose why a pull request was not classified / auto-flagged. Use when the user wants to turn on auto-flagging, map a GitHub repo to a LaunchDarkly project, change Release Autopilot account defaults, unmap a repo, or asks why Release Autopilot did not classify their PR. For first-time setup (pick capabilities, connect GitHub, choose inference, set up environment release plans and the project's flag release plan), use launchdarkly-release-autopilot-onboarding instead. For release plans, use launchdarkly-release-plans." license: Apache-2.0 compatibility: Requires the remotely hosted LaunchDarkly MCP server metadata: author: launchdarkly - version: "1.5.0-experimental" + version: "1.6.0-experimental" --- # LaunchDarkly Release Autopilot Settings @@ -59,7 +59,7 @@ The underlying endpoints require the Release Autopilot add-on. If every Release > Auto-flagging approvalRequired is ON. Turn it OFF? Auto-flagging PRs would no longer require that approval gate. -Same pattern for auto-releasing, and say that auto-releasing can ship. +Same pattern for auto-releasing, and say that auto-releasing can ship. When Release Autopilot proposes a release on a pull request, it uses only the project's flag release plans. Say so, and point to [`launchdarkly-release-plans`](../launchdarkly-release-plans/SKILL.md) to set them up. **Map or change one repo** (`update-automations` with `accountWide: false`): @@ -149,6 +149,7 @@ To unmap: [Confirm before write](#confirm-before-write), then `delete-automation - Auto-cleanup and auto-archival: use [`launchdarkly-automations-cleanup-settings`](../launchdarkly-automations-cleanup-settings/SKILL.md) and [`launchdarkly-automations-archival-settings`](../launchdarkly-automations-archival-settings/SKILL.md). `autoCleanup` in these settings is a different, unused field. - Completing GitHub OAuth inside the tool. Hand the user `githubConnectUrl` instead. -- Vega inference and bring-your-own-key setup, and environment release plans. The [onboarding skill](../launchdarkly-release-autopilot-onboarding/SKILL.md) guides both. +- Vega inference and bring-your-own-key setup: the [onboarding skill](../launchdarkly-release-autopilot-onboarding/SKILL.md) guides it. +- Release plans for auto-releasing (environment release plans and the project's flag release plan): use [`launchdarkly-release-plans`](../launchdarkly-release-plans/SKILL.md). Auto-releasing proposes no release in a project without a flag release plan. - Observability MCP (`github_repositories`) — do not require a second MCP server just to map a repo - Changing classification rules, GitHub App permissions, or org install scope via this skill diff --git a/skills/release-autopilot/launchdarkly-release-autopilot-settings/marketplace.json b/skills/release-autopilot/launchdarkly-release-autopilot-settings/marketplace.json index 873f0725..203abd7b 100644 --- a/skills/release-autopilot/launchdarkly-release-autopilot-settings/marketplace.json +++ b/skills/release-autopilot/launchdarkly-release-autopilot-settings/marketplace.json @@ -1,7 +1,7 @@ { "name": "launchdarkly-release-autopilot-settings", "description": "Configure LaunchDarkly Release Autopilot settings via MCP, or diagnose why a PR was not classified", - "version": "1.5.0-experimental", + "version": "1.6.0-experimental", "author": "LaunchDarkly", "repository": "https://github.com/launchdarkly/ai-tooling", "skills": ["./"], diff --git a/skills/release-autopilot/launchdarkly-release-plans/README.md b/skills/release-autopilot/launchdarkly-release-plans/README.md new file mode 100644 index 00000000..c7393fbb --- /dev/null +++ b/skills/release-autopilot/launchdarkly-release-plans/README.md @@ -0,0 +1,51 @@ +# LaunchDarkly Release Plans Skill + +An Agent Skill for both kinds of LaunchDarkly release plan: + +- **Environment release plans**: how a flag releases in one environment (phases, rollout steps, holds, and rollback metrics). Each environment has a default plan. +- **Flag release plans**: a project's environments in release order, each with its environment release plan. Release Autopilot proposes releases on pull requests from these plans. A rollout recorded for a flag (the `flag-release` skill) uses each environment's default release plan instead. + +## Overview + +This skill teaches agents how to: + +- List and show both kinds of plan, and each environment's and project's default +- Create environment release plans from starter plans (progressive, or guarded with rollback metrics) or step by step, after checking the randomization unit and metrics +- Build or update the project's flag release plan from existing environment release plans, and prompt for a release plan first when an environment has none +- Edit, make default, and delete both kinds of plan, confirming every write first +- Start a flag's release from a flag release plan +- Plan without writing ("record only"), for skills such as Release Autopilot onboarding that confirm and write later + +## Installation (Local) + +- **Generic**: copy `skills/release-autopilot/launchdarkly-release-plans/` into your client's skills path + +## Prerequisites + +- The hosted LaunchDarkly MCP server, for environment release plans +- For flag release plans and for deleting an environment release plan: the agent shows each REST command, and you run it in your own terminal with your LaunchDarkly API access token in `LD_ACCESS_TOKEN`, then paste the output back. The agent never runs these commands. Never paste a token into the chat. + +These REST endpoints are in beta, and every command sends `LD-API-Version: beta`. The MCP server has no flag release plan tools, and `ldcli` has no command for these endpoints, so the commands use curl. + +## Usage + +``` +Which release plans does production in project default have? +``` + +``` +Create a release plan for production in project default and make it the default +``` + +``` +Build the flag release plan for project default: test, then production, as the default +``` + +## Related + +- [LaunchDarkly Release Autopilot Onboarding](../launchdarkly-release-autopilot-onboarding/): first-time setup, which follows this skill for release plans +- [LaunchDarkly Release Autopilot Settings](../launchdarkly-release-autopilot-settings/): turn auto-releasing on or off for a repository + +## License + +Apache-2.0 diff --git a/skills/release-autopilot/launchdarkly-release-plans/SKILL.md b/skills/release-autopilot/launchdarkly-release-plans/SKILL.md new file mode 100644 index 00000000..39b450f7 --- /dev/null +++ b/skills/release-autopilot/launchdarkly-release-plans/SKILL.md @@ -0,0 +1,194 @@ +--- +name: launchdarkly-release-plans +description: "List, create, edit, make default, and delete LaunchDarkly release plans of both kinds: environment release plans (one environment's phases, rollout steps, and rollback metrics, with starter plans) and flag release plans (a project's environments in release order, each with its environment release plan, which Release Autopilot proposes releases from on pull requests). Also starts a flag's release from a flag release plan. Use when the user wants to see, build, change, or remove a release plan, set an environment's or a project's default plan, or when another skill such as Release Autopilot onboarding needs release plans planned or created. For first-time Release Autopilot setup, use launchdarkly-release-autopilot-onboarding, which follows this skill." +license: Apache-2.0 +compatibility: "Environment release plans use the hosted LaunchDarkly MCP server. For flag release plans, and for deleting an environment release plan, the user runs the REST API (beta) commands that the skill shows in their own terminal." +metadata: + author: launchdarkly + version: "1.0.0" +--- + +# LaunchDarkly Release Plans + +You're using a skill that reads and changes LaunchDarkly release plans. There are two kinds: + +- An **environment release plan** releases a flag in one environment: its phases, rollout steps, holds, and rollback metrics. A plan with metrics is guarded, so a regression can roll the release back. A plan without metrics is progressive. Each environment can have one default plan. +- A **flag release plan** is project-level. It lists the project's environments in release order, and names the environment release plan each one releases with (or says the environment releases immediately). When Release Autopilot proposes a release on a pull request, it uses the project's flag release plans, and it proposes none when the project has none. A rollout recorded for a flag (the `flag-release` skill, `create-automated-rollout-config` with `policy`) uses each environment's default release plan and does not need a flag release plan. A project can have one default flag release plan. + +Releases can ship code to users, so treat every write as production automation with a blast radius. + +## Prerequisites + +**MCP tools (hosted LaunchDarkly MCP server), preferred where they exist:** +- `get-environment-release-plans`: an environment's plans, each with `default`, or one plan with `releasePlanKey` +- `create-environment-release-plan`, `update-environment-release-plan` (send only the fields to change; `phases` and `metrics` each replace the whole value), and `update-default-environment-release-plan` (`default: true` sets it, `default: false` clears it; needs `confirm: true`) +- Used for starter plans: `search-contexts` (randomization unit) and `get-metrics` (rollback metrics) + +**REST API (beta), for what has no MCP tool:** every flag release plan operation, deleting an environment release plan, and the environment release plan changes whose MCP tool is missing. The user runs these commands, as [User-run commands](#user-run-commands) says. Every request sends `LD-API-Version: beta`. `ldcli` (3.16.0) has no command for these endpoints and no command that sends a raw request with that header, so the commands use curl. They are in [references/rest-api.md](references/rest-api.md). + +**When a tool or request is not available:** +- An environment release plan tool is missing from the server: the MCP server is out of date, not the account. Tell the user to update or reconnect the LaunchDarkly MCP server. The user can run the REST equivalent from [references/rest-api.md](references/rest-api.md) instead. +- A read returns 403 `Plan does not allow this operation`: that kind of release plan is not available on the account. Tell the user to contact LaunchDarkly support, and stop work on that kind of plan. + +## Modes + +| Mode | When | What you do | +|------|------|-------------| +| **Apply** | The default | Read, decide with the user, confirm, then write | +| **Read-only** | "show", "preview", "what would you change", a dry run, or a reader role | MCP reads only. Ask the user to run nothing: list any REST read, and the writes you would make, as optional commands for later. Do not ask for confirmation | +| **Record only** | Another skill (such as Release Autopilot onboarding) plans release setup and confirms it itself | Run the reads (MCP reads, and any REST read the user runs and pastes) and ask the choices below, then return the decisions as a list of writes. When the calling skill is in a preview or dry run, use MCP reads only, ask the user to run nothing, and return the REST reads as optional follow-ups. Write nothing, and do not ask the "Confirm" question or show write commands. The calling skill confirms, then makes the writes in [Write order](#write-order) | + +## User-run commands + +You never run a REST command, and you never use a LaunchDarkly token. The user runs each REST command in their own terminal with their own LaunchDarkly API access token. In Read-only mode, and in a caller's preview, ask the user to run nothing: list the commands instead. + +1. Show the exact command in a code block, filled in with the keys and the JSON body. Say: "Run this in your terminal with your LaunchDarkly API access token in `LD_ACCESS_TOKEN`." Their shell fills in `$LD_ACCESS_TOKEN`. +2. In the same message, ask header "Run this command" with your question tool (such as `AskUserQuestion`): "Done, here's the output" ("Paste the output below, with any secrets removed") or "Skip" ("Do this later"). The user pastes the output as their answer. Ask for the output only, and do not end your turn with only the command. +3. Read the pasted output. The last line is `HTTP `, and the JSON before it is the response body. Continue from it: report a read, check a write against what was confirmed, or follow [Errors](#errors). If the output is not there or cannot be read, ask the user to run the command again and paste all of it. +4. "Skip" leaves the change undone: list the command as a follow-up. + +**Never ask for a token in the chat**, and never use one that is pasted there. If the user pastes a token, do not repeat it. Tell them to delete that token in **Organization settings > Authorization** and make a new one, because the chat may keep it. To make a token, they go to **Organization settings > Authorization**, then run `export LD_ACCESS_TOKEN=` in their own terminal. + +## Check for a default before every create + +This rule applies to every create path: a direct "create a release plan" request, [Create](#create), [Build or update the project's plan](#build-or-update-the-projects-plan), and Record only mode. + +- **Environment release plan:** read the environment's plans before you create one. If the environment has a default, tell the user first: its name, and a short summary of its phases, rollout steps, and metrics. Then ask header "Default plan": + - "Use " ("Keep the current default. Create nothing") + - "Create a new plan and make it the default" ("It replaces as the default") + - "Create a new plan, keep as the default" +- **Flag release plan:** have the user list the project's plans before you create one. If the project has a default, tell the user first: its name, and its environments in order with the release plan for each. Then ask header "Flag release plan": + - "Update (Recommended)" ("Change its environments and release plans. It stays the default") + - "Keep " ("Create nothing") + - "Create another plan" ("A new plan that is not the default") +- **Incomplete plans:** also name any incomplete plans, and say that they cannot start a release or be the default. + +Ask this before any other create question, and create nothing until the user answers. + +## Core Principles + +1. **Read before you write.** Read the existing plans first, and reuse a plan that already does what the user wants. +2. **Confirm before every write** (Apply mode). State the current and the proposed plan in words. For an MCP write, ask header "Confirm" with "Make this change" and "Cancel", and write only after "Make this change" in this turn. For a REST write, the user runs the command, so show the change in words with the command. A detailed request is not approval, and neither is a yes from an earlier turn. A retry after a failed write needs its own confirmation. +3. **Prefer MCP tools.** Use the REST API only for what has no MCP tool. +4. **Only touch what the user named.** Never delete, demote, or edit a plan the user did not name. +5. **Speak in product words.** Say "release plan", "flag release plan", "default", and environment names. Do not show tool names, field names, HTTP status codes, or JSON to the user, except inside a command the user runs. + +## Environment release plans + +### Read + +`get-environment-release-plans` with `projectKey` and `environmentKey`. Report each plan's name, whether it is the default, and its phases, rollout steps, holds, and metrics. An incomplete plan (migrated from a release policy) cannot start a release or become the default: say so when you name one. + +### Choose an environment's release plan + +Do this for every environment that needs a plan: for the user's own request, for a flag release plan, and for onboarding in Record only mode. + +1. Read the environment's plans. Use only complete plans. +2. **The environment has complete plans, and one is the default.** Propose its default plan, unless the user named another. Show it with its phases, rollout steps, and metrics. Ask header "Plan": "Use this plan" ("Keep it as this environment's default") or "Choose another" (offer the other complete plans, or a starter plan). If the plan the user keeps is not the default yet, record setting it as the default. If another plan is the default now, say that the new one replaces it as the default. +3. **The environment has complete plans, and none is the default.** Show each complete plan with its phases, rollout steps, and metrics. Ask header "Plan" with one option per complete plan ("Make it this environment's default") and "Create a starter plan". Record setting the chosen plan as the default. +4. **The environment has no complete plan.** Tell the user it needs a release plan first, and offer a starter plan as [Create](#create) says. "Not now" is how the user declines. For a flag release plan, add the environment only after its plan exists (or is recorded, in Record only mode). +5. **The user declines.** Create nothing for that environment. Leave it out of any flag release plan, and say so. + +Never invent a release plan key, and never point at an incomplete plan. + +### Create + +First apply [Check for a default before every create](#check-for-a-default-before-every-create). Continue only when the user chose to create a plan, or the environment has no default. Then offer a starter plan so the user does not build one from scratch, or a custom plan. The starter definitions, the checks to run before you offer them, and the custom questions are in [references/environment-release-plans.md](references/environment-release-plans.md): + +1. **Randomization unit:** check stored contexts with `search-contexts`. Do not ask the user what reads can answer. +2. **Rollback metrics:** `get-metrics`, keeping metrics whose `analysisUnits` include the unit. Metrics make the plan guarded, and a guarded plan allows at most 50% per step. +3. **Starter:** ask header "Starter plan": "Progressive", "Custom", or "Not now". Show the full plan, and ask "Looks right" or "Adjust it". +4. **Key:** keys are unique in the project, so if a starter's key is taken anywhere in the project (another environment, or an active plan here), add `-2`, then `-3`, until it is free. +5. Confirm, then `create-environment-release-plan`, then `update-default-environment-release-plan` with `default: true` when it becomes the default. A new default replaces the current one: say so in the confirmation. + +If the create fails because the randomization unit is not available for guarded rollouts, nothing was written. Offer the same plan with no metrics, and confirm again. + +### Make a plan the environment's default + +Confirm, naming the plan it replaces. Then `update-default-environment-release-plan` with `releasePlanKey`, `default: true`, and `confirm: true`. The plan must be complete. + +### Edit + +Read the plan, change only what the user asked, and confirm the change in words. Then `update-environment-release-plan` with only the changed fields. `phases` and `metrics` each replace the whole value, so send every phase or metric to keep. Without the MCP tool, show the REST replace from [references/rest-api.md](references/rest-api.md), which is a full replace with the plan's `version`. + +### Delete + +There is no MCP tool, so the user runs the REST delete. Say first if the plan is the environment's default (the environment then has none). A plan that a flag release plan still names cannot be deleted (409): name the flag release plans that use it, and offer to change them first. To clear only the default, use `update-default-environment-release-plan` with `default: false`. + +## Flag release plans + +All flag release plan operations use the REST API in [references/rest-api.md](references/rest-api.md). The user runs each command and pastes the output, as [User-run commands](#user-run-commands) says. + +### How a flag release plan names environment release plans + +- Each environment names one environment release plan by key, or releases immediately. The named plan must exist in that environment and be complete, or the request returns 400. +- A flag release plan stores each environment's release plan key, so a later change to that environment's default release plan does not update the flag release plan. +- Environments are in release order. Plans are sequential by default: each environment releases after the one before it. Order non-production before production. +- Each environment's trigger decides what starts its release. Use "when the flag evaluates there" (`flag_eval`) unless the user asks for another. [references/rest-api.md](references/rest-api.md) lists the others and their rules. + +### The project's default flag release plan + +When auto-releasing knows the change's application, it uses the plans for that application, or the project's default when none is for it. When it does not know the application, every plan is a candidate, and with more than one it asks on the pull request which one to use. + +- A new plan can be the default only when the project has no default. +- To change what the default does, edit the default plan in place: same key, still the default. Do not create a second plan unless the user asks for one. +- An update is a full replace. Leaving out the default setting clears it. Always start from a fresh read, and send every field to keep. + +### List or show + +Show the list command, and read the pasted output (page with `offset` while more remain). For each, report its name, whether it is the default, and its environments in order with the release plan for each ("production: Progressive", or "releases immediately"). + +### Build or update the project's plan + +1. Have the user list the project's plans, and read the pasted output. +2. For each environment the user wants, in release order, choose its environment release plan as in [Choose an environment's release plan](#choose-an-environments-release-plan). +3. Decide: + - **No default plan:** a new plan "Default release" (key `default-release`, or the next free key), as the project's default. + - **The project has a default:** name it and its environments, and ask the "Flag release plan" question in [Check for a default before every create](#check-for-a-default-before-every-create). If it already has these environments, in this order, with these release plans, say so and recommend "Keep ". An update keeps the plan's key and its default setting. "Create another plan" creates a plan that is not the default. +4. Show the current and the new environments and release plans in words, then the create or replace command (with the plan's `version`). +5. Check the pasted response: its environments, release plans, and default match what you showed. If the response is not there, show the get command. + +On a 409 for a taken key, show the command again with the next free key. On a 409 for a stale version, have the user get the plan again, show what changed, and show the command again. + +### Make another plan the default, or delete one + +- **Default:** from the pasted plan, say what changes ("Staged rollout becomes the project's default flag release plan. It replaces Fast as the default."), then show the replace command with every field from the read, the default setting on, and its `version`. Check the pasted response. +- **Delete:** name the plan and say if it is the default (the project then has no default). Releases already started from it continue. Then show the delete command, and check for `HTTP 204`. + +### Start a flag's release from a plan + +This starts a release that can ship the flag's change to users. Name the flag, the plan, and each environment in order with its release plan, then show the command that creates the flag release from the plan. For a release tied to a pull request, use [`flag-release`](../../feature-flags/flag-release/SKILL.md) instead. + +## Write order + +Write environment release plans before anything that names them: + +1. Metrics to create, with [`launchdarkly-metric-create`](../../metrics/launchdarkly-metric-create/SKILL.md) +2. Environment release plans, then each environment's default +3. The flag release plan that names them + +## Errors + +| Error | Meaning | What to do | +|-------|---------|------------| +| Validation error (400) | The message says why. For example: a release plan that is unknown, incomplete, or in another environment, the project already has a default flag release plan, a step over 50% in a plan with metrics, or a unit that is not available for guarded rollouts | Tell the user the reason in words, fix the request, and confirm again | +| Unauthorized (401) | The user's token is missing, wrong, or expired | Tell the user to set `LD_ACCESS_TOKEN` in their terminal to a valid token, and run the command again | +| `Plan does not allow this operation` (403) | That kind of release plan is not on the account | Tell the user to contact LaunchDarkly support. Stop | +| Any other 403 | The role cannot do this in the project | Name the change that failed and stop. A member with that permission can make it | +| Not found (404) | The project, environment, flag, or plan does not exist | Check the key with a read. Do not create a plan to work around it | +| Conflict (409) | A taken key, a stale version, or deleting an environment release plan that a flag release plan names | See the workflow for that change | +| Server error (5xx) | LaunchDarkly could not complete it | Report it. A read can run again once. Before a write runs again, have the user read again, because it may have been applied | + +Never repeat a write in a loop. + +## Out of scope + +- Turning auto-releasing on or off for a repository: use [`launchdarkly-release-autopilot-settings`](../launchdarkly-release-autopilot-settings/SKILL.md). +- First-time Release Autopilot setup: use [`launchdarkly-release-autopilot-onboarding`](../launchdarkly-release-autopilot-onboarding/SKILL.md). + +## References + +- [references/environment-release-plans.md](references/environment-release-plans.md): starter plans, the checks before them, and custom plans +- [references/rest-api.md](references/rest-api.md): REST endpoints, curl commands, fields, and JSON examples for both kinds of plan +- In the LaunchDarkly UI: `/projects/{projectKey}/settings/releases`, with `?tab=flag-release-plans` for flag release plans +- [Guarded rollouts](https://launchdarkly.com/docs/home/releases/guarded-rollouts) diff --git a/skills/release-autopilot/launchdarkly-release-plans/marketplace.json b/skills/release-autopilot/launchdarkly-release-plans/marketplace.json new file mode 100644 index 00000000..98ee3ef7 --- /dev/null +++ b/skills/release-autopilot/launchdarkly-release-plans/marketplace.json @@ -0,0 +1,20 @@ +{ + "name": "launchdarkly-release-plans", + "description": "List, create, edit, make default, and delete LaunchDarkly environment release plans and flag release plans, with starter plans, and start a flag's release from a plan", + "version": "1.0.0", + "author": "LaunchDarkly", + "repository": "https://github.com/launchdarkly/ai-tooling", + "skills": ["./"], + "tags": [ + "launchdarkly", + "release-autopilot", + "auto-releasing", + "release-plans", + "guarded-rollouts", + "mcp", + "rest-api" + ], + "requirements": { + "mcp-servers": ["@launchdarkly/mcp-server"] + } +} diff --git a/skills/release-autopilot/launchdarkly-release-plans/references/environment-release-plans.md b/skills/release-autopilot/launchdarkly-release-plans/references/environment-release-plans.md new file mode 100644 index 00000000..9fa74383 --- /dev/null +++ b/skills/release-autopilot/launchdarkly-release-plans/references/environment-release-plans.md @@ -0,0 +1,56 @@ +# Environment release plans: starters and custom plans + +Use this from [Create](../SKILL.md#create). A release plan is a recipe for one environment, made of phases, rollout steps, and metrics. A plan with metrics makes a release guarded, so a regression on a metric can roll it back. A plan without metrics is progressive: say so. + +## Before you offer a starter + +Check the project with reads, not questions. + +1. **Randomization unit:** `search-contexts` with `kind: "user"`, `limit: 1` in that environment. If it finds contexts, use `user`. If not, call `search-contexts` with no kind to see which kinds the project stores. If it finds other kinds, ask header "Roll out by" with those kinds and `user`. If it finds none, the app has not sent contexts yet: use `user`, the kind every project has by default, and say so. A plan whose unit is not a context kind in the project cannot be created. +2. **Rollback metrics:** a guarded plan needs the unit to be available for guarded rollouts, and no read shows that. LaunchDarkly turns this on for `user` by default. For any other unit, skip this check, use no metrics, and list "make available for guarded rollouts, then add rollback metrics to the plan" as a follow-up. For `user`, call `get-metrics` with `metricUsedIn: "guarded_rollouts"`, then without it if that finds none. Keep metrics whose `analysisUnits` include the unit; others cannot measure the release. If any remain, ask header "Rollback metrics", `multiSelect: true`, with up to three of them plus "No metrics" ("Release step by step without automatic rollback"). + +## Starter plans + +Ask header "Starter plan" with options that fit the environment, marking the first "(Recommended)": + +- Production: "Progressive" ("10% → 25% → 50% of everyone, with 4-hour holds"), "Custom" ("Build the plan step by step"), and "Not now" ("Leave this environment without a release plan for now"). +- Any other environment: "Progressive" ("10% → 50% of everyone, with 4-hour holds"), "Custom", and "Not now". + +If the user picked rollback metrics, add "rolls back automatically if regress" to the descriptions. Then show the full plan and ask "Looks right" or "Adjust it". For "Not now", create nothing, and list "set up a release plan for " as a follow-up. + +| Starter | Key | Phases | +|---------|-----|--------| +| Progressive (non-production) | `progressive` | One phase for everyone: 10% for 4h, then 50% for 4h | +| Progressive (production) | `progressive` | One phase for everyone: 10%, 25%, 50%, each for 4h | + +Every starter uses automatic advancement and `cleanupBehavior: "remove_rules_and_update_default_rule"`, so the release finishes by serving everyone. Allocations are per 100,000 (10000 = 10%). There is no 100% step, because completing the release reaches everyone. Each 4-hour hold is `durationMillis: 14400000`. `` is the randomization unit. + +The phase has only the randomization unit clause, which means everyone: + +```json +{ "attribute": "kind", "op": "in", "values": [""], "contextKind": "" } +``` + +Rollback metrics go in `progressiveRelease.metrics`, keyed by metric key, each with automatic rollback: + +```json +{ "user-error-rate": { "isGroup": false, "autoRollback": true } } +``` + +A plan with metrics allows at most 50000 (50%) per step, and every starter step fits. A plan without metrics allows up to 99000 per step. + +## Keys + +Read the environment's plans before you create one. Plan keys are unique among the project's active plans, across environments, so the same starter in two environments needs two keys. If `progressive` is taken, use `progressive-2`, then `progressive-3`, and so on. A taken key returns 409. + +## Custom plans + +Ask each setting as its own question, never bundled: + +1. **Randomization unit**: the context kind to roll out by, such as `user` or `account`. +2. **Rollout steps**: header "Rollout", with options such as "10% → 25% → 50%, 4h holds" or "10% → 50%, 24h hold". The user can type another schedule in Other. +3. **Advancement**: header "Advance", options "Automatic" ("Move to the next step when the hold ends and metrics are healthy") and "Manual" ("A person moves each step"). +4. **Metrics**: call `get-metrics`. Offer up to four relevant metrics with `multiSelect: true`. If none fit, use [`launchdarkly-metric-choose`](../../../metrics/launchdarkly-metric-choose/SKILL.md) to decide which metrics to add, and record each one to create before the plan. This creates the metric only: if the app does not send the event yet, list instrumenting it as a follow-up, and do not edit code or `.env` files. +5. **Cleanup**: header "Cleanup", options "Leave rules in place" and "Remove rules and update the default rule", each with its outcome. + +Every phase needs at least one clause. Follow the `create-environment-release-plan` tool guidance for the rest. diff --git a/skills/release-autopilot/launchdarkly-release-plans/references/rest-api.md b/skills/release-autopilot/launchdarkly-release-plans/references/rest-api.md new file mode 100644 index 00000000..3cc637b4 --- /dev/null +++ b/skills/release-autopilot/launchdarkly-release-plans/references/rest-api.md @@ -0,0 +1,249 @@ +# Release plan REST API + +The user runs these commands in their own terminal with their LaunchDarkly API access token in `LD_ACCESS_TOKEN`, and pastes the output back. The agent shows them, filled in, and never runs them. Every endpoint is in beta, so every command sends `LD-API-Version: beta`. Each command prints the response body, then `HTTP ` on the last line. Replace `PROJECT_KEY`, `PLAN_KEY`, `ENV_KEY`, `RELEASE_PLAN_KEY`, and `FLAG_KEY`. For an instance other than `https://app.launchdarkly.com`, use that instance's URL. + +Prefer the MCP tools for environment release plans. Use these commands for flag release plans, for deleting an environment release plan, and when an MCP tool is missing. Do not add `-v` or `--trace`: they print the `Authorization` header. + +## Endpoints + +### Flag release plans + +| Operation | Method and path | Success | +|-----------|-----------------|---------| +| List a project's plans | `GET /api/v2/projects/{projectKey}/flag-release-plans?limit=50&offset=0` | 200 | +| Get one plan | `GET /api/v2/projects/{projectKey}/flag-release-plans/{planKey}` | 200 | +| Create a plan | `POST /api/v2/projects/{projectKey}/flag-release-plans` | 201 | +| Replace a plan (full update) | `PUT /api/v2/projects/{projectKey}/flag-release-plans/{planKey}` | 200 | +| Delete a plan | `DELETE /api/v2/projects/{projectKey}/flag-release-plans/{planKey}` | 204 | +| Start a flag's release | `POST /api/v2/projects/{projectKey}/flags/{flagKey}/flag-releases` | 201 | + +### Environment release plans + +Paths start with `/api/v2/projects/{projectKey}/environments/{environmentKey}`. + +| Operation | Method and path | MCP tool | Success | +|-----------|-----------------|----------|---------| +| List plans | `GET .../release-plans` | `get-environment-release-plans` | 200 | +| Get one plan | `GET .../release-plans/{releasePlanKey}` | `get-environment-release-plans` with `releasePlanKey` | 200 | +| Create a plan | `POST .../release-plans` | `create-environment-release-plan` | 201 | +| Replace a plan (full update) | `PUT .../release-plans/{releasePlanKey}` | `update-environment-release-plan` (changed fields only) | 200 | +| Delete a plan | `DELETE .../release-plans/{releasePlanKey}` | none | 204 | +| Get the default | `GET .../default-release-plan` | `get-environment-release-plans` (`default`) | 200, or 204 with no default | +| Set the default | `PUT .../default-release-plan` with `{ "key": "..." }` | `update-default-environment-release-plan` with `default: true` | 200 | +| Clear the default | `DELETE .../default-release-plan` | `update-default-environment-release-plan` with `default: false` | 204 | + +The list returns `items`, `totalCount`, and `_links`, in creation order. `limit` is 1 to 50 (default 25). Page with `offset` while `offset + items.length < totalCount`. + +## Commands + +### List flag release plans + +```sh +curl -sS -w '\nHTTP %{http_code}\n' \ + -H "Authorization: $LD_ACCESS_TOKEN" \ + -H "LD-API-Version: beta" \ + "https://app.launchdarkly.com/api/v2/projects/PROJECT_KEY/flag-release-plans?limit=50&offset=0" +``` + +### Get a flag release plan + +```sh +curl -sS -w '\nHTTP %{http_code}\n' \ + -H "Authorization: $LD_ACCESS_TOKEN" \ + -H "LD-API-Version: beta" \ + "https://app.launchdarkly.com/api/v2/projects/PROJECT_KEY/flag-release-plans/PLAN_KEY" +``` + +### Create a flag release plan + +```sh +curl -sS -w '\nHTTP %{http_code}\n' -X POST \ + -H "Authorization: $LD_ACCESS_TOKEN" \ + -H "LD-API-Version: beta" \ + -H "Content-Type: application/json" \ + "https://app.launchdarkly.com/api/v2/projects/PROJECT_KEY/flag-release-plans" \ + --data @- <<'JSON' +{ + "key": "default-release", + "name": "Default release", + "isDefault": true, + "environments": [ + { "environmentKey": "test", "environmentReleasePlanKey": "progressive", "triggerType": "flag_eval" }, + { "environmentKey": "production", "environmentReleasePlanKey": "progressive-2", "triggerType": "flag_eval" } + ] +} +JSON +``` + +Leave out `"isDefault": true` when the project already has a default. + +### Replace a flag release plan + +Start from a fresh get. Send every field to keep, because anything left out is cleared, including `isDefault`. `key` is not in the body; it is in the path. + +```sh +curl -sS -w '\nHTTP %{http_code}\n' -X PUT \ + -H "Authorization: $LD_ACCESS_TOKEN" \ + -H "LD-API-Version: beta" \ + -H "Content-Type: application/json" \ + "https://app.launchdarkly.com/api/v2/projects/PROJECT_KEY/flag-release-plans/PLAN_KEY" \ + --data @- <<'JSON' +{ + "version": 1, + "name": "Default release", + "isDefault": true, + "executionMode": "sequential", + "environments": [ + { "environmentKey": "test", "environmentReleasePlanKey": "progressive", "triggerType": "flag_eval" }, + { "environmentKey": "production", "environmentReleasePlanKey": "progressive-2", "triggerType": "flag_eval" } + ] +} +JSON +``` + +Copy `description`, `maintainer` (as `{ "kind": "member", "memberId": ... }` or `{ "kind": "team", "teamKey": ... }`), and `applicationKeys` from the read when the plan has them. The read returns the maintainer with names; send only the kind and the ID or key. + +### Delete a flag release plan + +```sh +curl -sS -w '\nHTTP %{http_code}\n' -X DELETE \ + -H "Authorization: $LD_ACCESS_TOKEN" \ + -H "LD-API-Version: beta" \ + "https://app.launchdarkly.com/api/v2/projects/PROJECT_KEY/flag-release-plans/PLAN_KEY" +``` + +### Start a flag's release from a plan + +```sh +curl -sS -w '\nHTTP %{http_code}\n' -X POST \ + -H "Authorization: $LD_ACCESS_TOKEN" \ + -H "LD-API-Version: beta" \ + -H "Content-Type: application/json" \ + "https://app.launchdarkly.com/api/v2/projects/PROJECT_KEY/flags/FLAG_KEY/flag-releases" \ + --data @- <<'JSON' +{ "kind": "plan", "planKey": "default-release" } +JSON +``` + +Optional: `startingVariationId` and `endingVariationId` (for a boolean flag, false to true when left out), and `pullRequest` (`repoFullName`, `number`, optional `url`). The response is `{ "id": "..." }`. An unknown plan key returns 400. + +### List an environment's release plans + +```sh +curl -sS -w '\nHTTP %{http_code}\n' \ + -H "Authorization: $LD_ACCESS_TOKEN" \ + -H "LD-API-Version: beta" \ + "https://app.launchdarkly.com/api/v2/projects/PROJECT_KEY/environments/ENV_KEY/release-plans" +``` + +Each item has `key`, `name`, `version`, `progressiveRelease`, and `default` (true for the environment's default plan). The list leaves out incomplete plans. + +### Replace an environment release plan + +A full replace: the phases, steps, clauses, and metrics become exactly what you send. Start from a fresh get, and send `version` from it. The key and environment cannot change. + +```sh +curl -sS -w '\nHTTP %{http_code}\n' -X PUT \ + -H "Authorization: $LD_ACCESS_TOKEN" \ + -H "LD-API-Version: beta" \ + -H "Content-Type: application/json" \ + "https://app.launchdarkly.com/api/v2/projects/PROJECT_KEY/environments/ENV_KEY/release-plans/RELEASE_PLAN_KEY" \ + --data @- <<'JSON' +{ + "version": 2, + "name": "Progressive", + "progressiveRelease": { + "randomizationUnit": "user", + "advancementMode": "automatic", + "cleanupBehavior": "remove_rules_and_update_default_rule", + "phases": [ + { + "clauses": [{ "attribute": "kind", "op": "in", "values": ["user"], "contextKind": "user" }], + "steps": [{ "allocation": 10000, "durationMillis": 14400000 }, { "allocation": 50000, "durationMillis": 14400000 }] + } + ], + "metrics": {} + } +} +JSON +``` + +A version mismatch returns 409 and changes nothing. + +### Delete an environment release plan + +```sh +curl -sS -w '\nHTTP %{http_code}\n' -X DELETE \ + -H "Authorization: $LD_ACCESS_TOKEN" \ + -H "LD-API-Version: beta" \ + "https://app.launchdarkly.com/api/v2/projects/PROJECT_KEY/environments/ENV_KEY/release-plans/RELEASE_PLAN_KEY" +``` + +It returns 409 while a flag release plan still names the plan. Remove it from those flag release plans first. Its key can be used again after the delete. + +### Set or clear an environment's default + +```sh +curl -sS -w '\nHTTP %{http_code}\n' -X PUT \ + -H "Authorization: $LD_ACCESS_TOKEN" \ + -H "LD-API-Version: beta" \ + -H "Content-Type: application/json" \ + "https://app.launchdarkly.com/api/v2/projects/PROJECT_KEY/environments/ENV_KEY/default-release-plan" \ + --data '{ "key": "RELEASE_PLAN_KEY" }' +``` + +Setting the default replaces the current one. The plan must be active in the environment and have a randomization unit. To clear the default, send `-X DELETE` to the same path with no body (204 whether or not there was one). `GET` on the path returns the default, or 204 when there is none. + +## Flag release plan fields + +Create takes these fields. Replace takes the same fields without `key`, plus `version`. + +| Field | Required | Rules | +|-------|----------|-------| +| `key` | Create only | 1 to 256 characters, `^[a-zA-Z0-9][._\-a-zA-Z0-9]*$`. Unique among the project's active plans. Cannot change | +| `name` | Yes | At most 256 characters | +| `description` | No | At most 1024 characters | +| `environments` | Yes | 1 to 50 entries, in release order, each environment once | +| `environments[].environmentKey` | Yes | An environment in the project | +| `environments[].triggerType` | Yes | `flag_eval`, `webhook`, `manual`, or `promotion`. `manual` and `promotion` need `sequential`; the first environment cannot use `promotion` | +| `environments[].environmentReleasePlanKey` | Unless immediate | A complete environment release plan in that environment | +| `environments[].isImmediate` | No | `true` releases at once with no release plan. Do not send a release plan key with it | +| `executionMode` | No | `sequential` (the default for plans) or `parallel` | +| `applicationKeys` | No | At most 25 keys. Auto-releasing uses them to choose a plan for an application | +| `isDefault` | No | Create: only when the project has no default. Replace: `true` demotes the current default; left out clears it | +| `maintainer` | No | `{ "kind": "member", "memberId": "..." }` or `{ "kind": "team", "teamKey": "..." }` | +| `version` | Replace only, optional | The version from the last read. A mismatch returns 409 and changes nothing | + +A project has at most 100 active plans. + +## Flag release plan response + +Create, get, and replace return the plan. Its version starts at 1 and goes up by 1 on each update. + +```json +{ + "key": "default-release", + "_id": "0b6c3a43-5d0e-4f43-9a8e-2f6f1d7c9e10", + "version": 1, + "name": "Default release", + "isDefault": true, + "executionMode": "sequential", + "applicationKeys": [], + "environments": [ + { "environmentKey": "test", "environmentReleasePlanKey": "progressive", "isImmediate": false, "triggerType": "flag_eval" }, + { "environmentKey": "production", "environmentReleasePlanKey": "progressive-2", "isImmediate": false, "triggerType": "flag_eval" } + ] +} +``` + +## Errors + +Errors return `{ "code": "...", "message": "..." }`. Read `message` for the reason. + +| Status | When | +|--------|------| +| 400 | A field breaks a rule above; an environment is not in the project; a release plan is unknown, incomplete, or in another environment; `isDefault` on create while the project has a default; the project already has 100 plans; an unknown plan key when starting a release | +| 401 | No token, or the token is not valid | +| 403 | `Plan does not allow this operation`: the account does not have flag release plans. Any other message: the token's role lacks the action | +| 404 | The project, flag, or plan does not exist | +| 409 | Create: another active plan has the key. Replace: `version` does not match the plan's current version. Deleting an environment release plan: a flag release plan still names it |