From 428a18592c9d71a8dcbc953f142a77c3866f2f91 Mon Sep 17 00:00:00 2001 From: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Date: Fri, 6 Nov 2026 22:18:16 +0000 Subject: [PATCH 1/6] feat(agentcontrol): add legacy-sdk-migrate skill for moving to the handler-based AI SDKs Co-Authored-By: Traci Siebel --- CHANGELOG.md | 1 + README.md | 1 + skills.json | 8 + .../agentcontrol/legacy-sdk-migrate/SKILL.md | 139 ++++++ .../references/api-mapping.md | 445 ++++++++++++++++++ 5 files changed, 594 insertions(+) create mode 100644 skills/agentcontrol/legacy-sdk-migrate/SKILL.md create mode 100644 skills/agentcontrol/legacy-sdk-migrate/references/api-mapping.md diff --git a/CHANGELOG.md b/CHANGELOG.md index 4da3eee5..0562a42c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,7 @@ All notable changes to this repository will be documented in this file. ## Unreleased +- Add `agentcontrol/legacy-sdk-migrate` 0.1.0: migrate an app from the legacy AI SDKs (`launchdarkly-server-sdk-ai`, `@launchdarkly/server-sdk-ai`) to the current handler-based SDKs, following [Migrate from the legacy AI SDKs](https://launchdarkly.com/docs/sdk/ai/migration). Inventories legacy calls, maps each to a convenience function, `config().invoke()`, or `graph()`, moves tools into `tool_handlers` / `toolHandlers` or a `Registry`, replaces trackers with automatic metrics, ports `enabled=False` fallbacks to error handling and `inspect_config()` / `inspectConfig()`, and covers streaming, history, async judges, and graphs. `references/api-mapping.md` has before/after code in Python and Node.js using the handler factory names the packages export. - `observability/investigate` 0.3.0 and `observability/alert-investigation` 0.2.0: rewritten around the hosted MCP's current tools (`get-changes`, `get-trace`, `get-error-group`, `search-service-map`, `search-log-patterns`, `get-alert-firing-history`, `get-investigation`, `get-incidents`, `search-aggregations` with `compareTo` and breakdowns). `investigate` adds a "first five minutes" sequence: scope and onset, chronic or noise, what changed, what is over-represented against a baseline (by host, version, route and flag variation), evidence, blast radius; and warns about percentiles over few sampled rows. `alert-investigation` starts from `get-alerts` and `get-alert-firing-history` (replay, chronic check, automatic investigation) instead of alert variables injected into the conversation. The 50-row rule is gone; companions use the stable `search-*` tool names and current arguments. The guides also use the fields from launchdarkly/mcp#449 (firing-history `summary` and `chronic`, get-changes `alert_episode` and evaluated-only flag changes, get-trace `self_ms`/`critical_path`/`top_self_time`, project-wide `search-flag-evaluations` with `include` and the FLAG FILTERS keys, the `low_sample` warning, get-error-group `include` and `_pivots`, `sort: "spike"`, and the `ingest` block with `completeness_note`), each with a fallback for servers that predate it. - Claude Code plugin 1.0.6, Cursor plugin 1.0.7: Release Autopilot skills and evals use the MCP tool names that match the Automations settings UI: `get-automations`, `update-automations`, and `delete-automations-repo-settings` replace `get-release-autopilot`, `update-release-autopilot`, and `delete-release-autopilot-repo-settings` (`launchdarkly-release-autopilot-settings` 1.4.1-experimental, `launchdarkly-release-autopilot-onboarding` 0.3.1-experimental). Arguments are unchanged. On a server that still exposes the old names, the skills use those; if a call says the tool was removed, they ask the user to reconnect the MCP server. Skill names are unchanged. Pairs with the launchdarkly/mcp rename. - Claude Code plugin 1.0.4, Cursor plugin 1.0.5: resolve the directory policy finding "Uses a credential from the user's machine". `experiments/launchdarkly-experiment-guide` no longer looks for `LD_API_KEY` or `LAUNCHDARKLY_ACCESS_TOKEN` in the environment; for the REST path it asks the user to paste an API access token for the session. `flag-and-release-change` and `flag-release` no longer say credentials are injected by the environment; they use the user's existing git access and the MCP server's OAuth, and never read tokens from the machine. diff --git a/README.md b/README.md index 43c368c8..0069a97e 100644 --- a/README.md +++ b/README.md @@ -37,6 +37,7 @@ Agent Skills are modular, text-based playbooks that teach an agent how to perfor |-------|-------------| | `agentcontrol/configs-create` | Create configs with variations for agent or completion mode | | `agentcontrol/migrate` | Migrate an app with hardcoded LLM prompts to AgentControl in five stages (extract, wrap, tools, tracking, evals) | +| `agentcontrol/legacy-sdk-migrate` | Migrate from the legacy Python and Node.js AI SDKs to the current handler-based SDKs (packages, init, `config()`, tools, registries, judges, graphs, fallbacks) | | `agentcontrol/configs-update` | Update and delete configs, manage lifecycle | | `agentcontrol/configs-variations` | Manage config variations for A/B testing | | `agentcontrol/tools` | Create and attach tools for function calling | diff --git a/skills.json b/skills.json index dce3208d..3f4bb522 100644 --- a/skills.json +++ b/skills.json @@ -471,6 +471,14 @@ "mcp" ] }, + { + "name": "legacy-sdk-migrate", + "description": "Migrate an app from the legacy LaunchDarkly AI SDKs (Python `launchdarkly-server-sdk-ai` / `ldai`, Node.js `@launchdarkly/server-sdk-ai`) to the current handler-based AI SDKs (`launchdarkly-ai-server`, `@launchdarkly/ai-node` / `@launchdarkly/ai-server` plus handler packages). Use when code calls `LDAIClient`, `initAi`, `completion_config` / `completionConfig`, `agent_config` / `agentConfig`, `agent_configs`, `create_tracker` / `trackMetricsOf`, `create_judge`, `create_model`, or `agent_graph`, or when the user asks to upgrade to the turnkey AI SDKs.", + "path": "skills/agentcontrol/legacy-sdk-migrate", + "version": "0.1.0", + "license": "Apache-2.0", + "compatibility": "Python 3.12+ or Node.js. No LaunchDarkly MCP tools needed; the AgentControl configs already exist." + }, { "name": "mcp-configure", "description": "Configure the LaunchDarkly hosted MCP server during onboarding. Use when the parent LaunchDarkly onboarding skill reaches the MCP offer, after the first flag works. Supports Cursor, Claude Code, Windsurf, GitHub Copilot, and other MCP-compatible agents. OAuth authentication; no API keys for the hosted server.", diff --git a/skills/agentcontrol/legacy-sdk-migrate/SKILL.md b/skills/agentcontrol/legacy-sdk-migrate/SKILL.md new file mode 100644 index 00000000..adfee6e1 --- /dev/null +++ b/skills/agentcontrol/legacy-sdk-migrate/SKILL.md @@ -0,0 +1,139 @@ +--- +name: legacy-sdk-migrate +description: "Migrate an app from the legacy LaunchDarkly AI SDKs (Python `launchdarkly-server-sdk-ai` / `ldai`, Node.js `@launchdarkly/server-sdk-ai`) to the current handler-based AI SDKs (`launchdarkly-ai-server`, `@launchdarkly/ai-node` / `@launchdarkly/ai-server` plus handler packages). Use when code calls `LDAIClient`, `initAi`, `completion_config` / `completionConfig`, `agent_config` / `agentConfig`, `agent_configs`, `create_tracker` / `trackMetricsOf`, `create_judge`, `create_model`, or `agent_graph`, or when the user asks to upgrade to the turnkey AI SDKs." +license: Apache-2.0 +compatibility: Python 3.12+ or Node.js. No LaunchDarkly MCP tools needed; the AgentControl configs already exist. +metadata: + author: launchdarkly + version: "0.1.0" +--- + +# Migrate from the legacy AI SDKs + +You're using a skill that will guide you through moving an application from the legacy LaunchDarkly AI SDKs to the current handler-based AI SDKs. Your job is to inventory every legacy call site, map each one to its current equivalent, rewrite the code, and verify that the app still behaves the same, including when LaunchDarkly is unreachable. + +The current SDKs are **not drop-in replacements**. In the legacy SDKs the app evaluates a config, builds a provider client, merges messages, calls the provider, and wraps the call in a tracker. In the current SDKs the app registers handlers and tools once, then calls `config()`, `graph()`, or a handler package's convenience function. The SDK evaluates the config, routes to the handler that matches the variation's provider and mode, calls the provider, and records metrics and OpenTelemetry spans. + +Source guide: [Migrate from the legacy AI SDKs](https://launchdarkly.com/docs/sdk/ai/migration). Full before/after code for every pattern: [references/api-mapping.md](references/api-mapping.md). + +## Prerequisites + +- The app already uses a legacy AI SDK, and its AgentControl configs (formerly AI Configs) already exist in LaunchDarkly. The config keys, variations, tools, and judges carry over unchanged; this is a code migration only. +- `LD_SDK_KEY` (server-side SDK key) and the provider API keys (`OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, ...) available as environment variables. The current SDKs read them from the environment. +- **Python:** the current packages require **Python 3.12 or later**, and the API is **async** (`await config(...).invoke(...)`). Check both before you start. +- **Node.js:** `@launchdarkly/ai-node` for Node; `@launchdarkly/ai-server` plus a pre-initialized client for edge or custom runtimes. + +## Core Principles + +1. **Inventory before you edit.** Find every legacy import and call site first, and confirm the list with the user. +2. **Use the real export names.** Handler factories are `create___handler()` in Python and `createHandler()` in Node (`createOpenAIMessagesHandler`, `createOpenAIAgentsHandler`, `create_openai_agents_handler`). Check the installed package's exports before writing an import; don't guess. +3. **Keep the fallback.** The current SDKs take no fallback argument. `invoke()` raises when the variation is disabled (targeting off) or LaunchDarkly is unreachable, so every call site that relied on `enabled=False` defaults needs an explicit `try`/`except` (or `try`/`catch`) that preserves today's degraded behavior. +4. **Delete, don't wrap.** Remove trackers, `track_*` calls, provider-client construction, message merging, and `model=config.model.name` plumbing. Leaving them in double-counts metrics. +5. **Register once.** Put handlers and tool implementations in one `Registry` (or the global registry) instead of rebuilding them at every call site. +6. **Ask before rewriting business logic.** If the app post-processes provider-specific response objects (for example `completion.choices[0].message.content`), switch to `result.response`, and ask before changing anything downstream that depends on the raw provider object. + +## Workflow + +### Step 1: Explore (read-only) + +Find every legacy usage. Search for these patterns and record file and line for each hit: + +| Language | Imports and packages | Calls | +|----------|----------------------|-------| +| Python | `launchdarkly-server-sdk-ai`, `launchdarkly-server-sdk-ai-openai`, `launchdarkly-server-sdk-ai-langchain`; `from ldai`, `from ldai_openai`, `from ldai_langchain` | `LDAIClient(`, `completion_config(`, `agent_config(`, `agent_configs(`, `judge_config(`, `create_judge(`, `create_model(`, `create_agent(`, `agent_graph(`, `create_agent_graph(`, `create_tracker(`, `track_metrics_of`, `track_duration`, `track_tokens`, `track_success`, `track_error`, `track_feedback`, `track_tool_call`, `get_ai_metrics_from_response` | +| Node.js | `@launchdarkly/server-sdk-ai`, `@launchdarkly/server-sdk-ai-openai`, `@launchdarkly/server-sdk-ai-langchain`, `@launchdarkly/server-sdk-ai-vercel` | `initAi(`, `completionConfig(`, `agentConfig(`, `agentConfigs(`, `judgeConfig(`, `createJudge(`, `createModel(`, `createAgent(`, `agentGraph(`, `createTracker(`, `.tracker`, `trackMetricsOf`, `trackStreamMetricsOf`, `trackDuration`, `trackTokens`, `trackSuccess`, `trackError`, `trackFeedback`, `getAIMetricsFromResponse`, `getAIMetricsFromStream`, `createLangChainModel` | + +For each call site, also record: + +- **Provider and SDK** used for the actual model call (OpenAI, Anthropic, LangChain/LangGraph, Vercel AI SDK, LiteLLM, Bedrock, Gemini, other). +- **Mode**: completion (`completion_config`) or agent (`agent_config`), and whether it streams. +- **Tools**: any function-calling loop, and the tool names it dispatches. +- **Fallback behavior**: what the code does when `enabled` is false. +- **Extras**: template variables, conversation history, judges, agent graphs, custom tracker calls such as `track_feedback`. + +Present the inventory and **stop** until the user confirms it. + +### Step 2: Assess + +Pick a target for each call site: + +| Legacy pattern | Current replacement | +|----------------|---------------------| +| One provider, one mode, manual completion call | That handler package's convenience function, for example `openai_messages()` / `openaiMessages()` | +| Config can serve several providers or modes | `config(key=..., handler=[...])` then `.invoke(user_input, context, variables)` | +| Agent mode with a tool loop | `config()` with an agent handler and `tool_handlers` / `toolHandlers`; mode comes from the variation | +| Streaming (`trackStreamMetricsOf`, manual TTFT) | `config(...).stream(...)`; iterate events (`chunk` events carry `text`; the final `done` event carries `response` and `usage`) | +| Multi-turn chat (`createChat`, message arrays you maintain) | Pass prior turns as the `history` argument (fourth positional argument) to `invoke()` / `stream()` | +| `agent_configs()` / `agentConfigs()`, `agent_graph()`, `create_agent_graph()`, managed graph runners | `graph(key, handlers=[...]).invoke(...)`, or `resolve_graph()` / `resolveGraph()` plus a native runner (`to_lang_graph` / `toLangGraph`, `to_openai_agents` / `toOpenAIAgents`, `to_claude_agents` / `toClaudeAgents`) | +| `create_judge()` / `createJudge()` / `judge_config()` | Judges attached to the config variation run automatically. For background scoring, pass `skip_judges=True` / `skipJudges: true` and hand each `judge_tasks` / `judgeTasks` item to `run_judge()` / `runJudge()` | +| `create_model().run()` / `createModel().run()`, `create_agent()` | Same convenience function or `config().invoke()` path | +| Same handlers and tools at many call sites | One shared `Registry` passed as `registry=` | + +Handler packages (each `launchdarkly-ai-` in Python, `@launchdarkly/ai-` in Node): + +| Package suffix | Provider | Mode | +|----------------|----------|------| +| `openai-messages` | OpenAI | completion (`messages`) | +| `openai-agents` | OpenAI | agent | +| `claude-messages` | Anthropic | completion (`messages`) | +| `claude-agents` | Anthropic (Claude Agent SDK) | agent | +| `langchain-messages` / `langchain-agents` | Any LangChain chat model / LangGraph | completion / agent | +| `litellm-messages` / `litellm-agents` | Any LiteLLM provider (use for Bedrock, Gemini, and others without a dedicated package) | completion / agent | +| `vercel-messages` / `vercel-agents` | Any Vercel AI SDK provider | completion / agent | + +A variation whose provider has no matching registered handler raises `Handler for provider not found`. Register a handler for every provider and mode your variations can serve, or write one with `createHandler` / `create_handler` for an internal endpoint. + +Confirm the plan with the user: target per call site, packages to add and remove, and how each fallback will be preserved. + +### Step 3: Execute + +Work one call site at a time. Full code for each step is in [references/api-mapping.md](references/api-mapping.md). + +1. **Swap packages.** Remove the legacy AI packages and install the current core package plus one handler package per provider and mode. + - Python: `pip install "launchdarkly-ai-python[ld]" launchdarkly-ai-openai-messages` (or `launchdarkly-server-sdk launchdarkly-ai-server ...`). Remove `launchdarkly-server-sdk-ai` and `launchdarkly-server-sdk-ai-*`. + - Node: `npm install @launchdarkly/ai-node @launchdarkly/ai-openai-messages`. Remove `@launchdarkly/server-sdk-ai` and `@launchdarkly/server-sdk-ai-*`. Edge runtimes install `@launchdarkly/ai-server` and pass their client to `initClient(client)`. + - Keep the base server SDK (`launchdarkly-server-sdk` / `@launchdarkly/node-server-sdk`) only if the app still evaluates ordinary feature flags with it. +2. **Replace initialization.** Delete `LDAIClient(...)` / `initAi(...)`. The SDK initializes lazily from `LD_SDK_KEY` on the first AI call. Call `init_client()` / `initClient()` only to pass options or pre-warm, and call `shutdown()` on exit to flush events and spans. +3. **Rewrite each call.** Replace evaluate → build messages → call provider → track with one convenience function or `config(...).invoke(user_input, context, variables)`. Read the output from `result.response`. Python contexts are plain dicts (`{"kind": "user", "key": "user-123"}`). +4. **Move template variables** from the `completion_config` / `agent_config` argument to the `variables` keyword (convenience functions, Python), the `{ variables }` option (convenience functions, Node), or the third argument to `invoke()`. +5. **Move tools.** Register implementations by name in `tool_handlers` / `toolHandlers` or a `Registry`. Keys must match the tool names in the config variation exactly, including case. Delete the hand-written tool loop. +6. **Preserve fallbacks.** Wrap `invoke()` and return the app's previous fallback response on error. Use `inspect_config()` / `inspectConfig()` (never raises; returns `enabled`, `config`, `meta`) for health checks or to branch before invoking. +7. **Delete tracking code.** Remove `create_tracker`, every `track_*` / `track*` call, and `get_ai_metrics_from_response` / `getAIMetricsFromResponse` helpers. Metrics and spans are automatic. +8. **Centralize.** If two or more call sites share handlers or tools, create one `Registry` (or register on `global_registry` / `globalRegistry` at startup) and pass `registry=`. +9. **Remove dead imports and packages** once nothing references them. + +### Step 4: Verify + +1. **No legacy references remain:** re-run the Step 1 search. Expect zero hits for `ldai`, `server-sdk-ai`, `create_tracker`, `createTracker`, `track_metrics_of`, and `trackMetricsOf`. +2. **Build and tests pass:** run the project's type check, linter, and test suite. Update mocks that patched `LDAIClient` / `initAi` to patch `config` or the convenience function instead. +3. **Happy path:** with `LD_SDK_KEY` set, run each migrated call and confirm it returns a response from the provider the variation names. +4. **Fallback path:** with targeting off for the config (or an invalid `LD_SDK_KEY`), confirm the app returns its fallback instead of crashing. +5. **Metrics:** after a few calls, the config's Monitoring tab in LaunchDarkly shows generations, tokens, and duration for the new SDK version. +6. **Report:** list each migrated call site (file and line), packages added and removed, and anything left unmigrated with the reason. + +## Edge Cases + +| Situation | Action | +|-----------|--------| +| Python below 3.12 | Stop and tell the user. The current Python packages need 3.12+. | +| Synchronous Python code path (Flask, Django views, scripts) | Run the async call with `asyncio.run(...)` at the boundary, or move the handler to an async framework. Ask which. | +| App reads provider-native fields (`choices`, `tool_calls`, `stop_reason`) | Use `result.response` and `result.usage`; ask before changing downstream logic that needs the raw object. | +| Provider has no handler package (Bedrock, Gemini, Azure, internal gateway) | Use `litellm-*`, `langchain-*`, or `vercel-*`, or build a custom handler with `createHandler` / `create_handler`. | +| App calls `track_feedback` / `trackFeedback` for thumbs up or down | The current SDKs have no feedback helper. Don't silently delete the call: tell the user, and leave it in place on the legacy client or ask how they want to record feedback. | +| Large app, gradual rollout | Migrate one call site at a time, verify it, and remove the legacy packages only after the last call site moves. | +| Judges must not add latency | `skip_judges=True` / `skipJudges: true`, then run `judge_tasks` / `judgeTasks` in a background task or worker with `run_judge()` / `runJudge()`. | +| Agent Skills from AgentControl are needed | Agent Skills reach apps only through the current SDKs; the legacy SDKs never deliver them. | + +## What NOT to Do + +- Don't keep the legacy tracker around the new call. The SDK already records metrics; you'll double-count. +- Don't pass a default or fallback config object to `config()`; it has no such argument. Handle errors instead. +- Don't construct an OpenAI or Anthropic client to pass in. Handlers build their own clients from environment variables. +- Don't rename config keys, tools, or variations in LaunchDarkly as part of this migration. +- Don't invent export names. If an import fails, list the package's exports and use the real name. + +## References + +- [references/api-mapping.md](references/api-mapping.md): before/after code for initialization, completion calls, agent mode and tools, registries, streaming, history, judges, and graphs, in Python and Node.js +- [Migrate from the legacy AI SDKs](https://launchdarkly.com/docs/sdk/ai/migration) +- [Python AI SDK reference](https://launchdarkly.com/docs/sdk/ai/python) and [Node.js (server-side) AI SDK reference](https://launchdarkly.com/docs/sdk/ai/node-js) diff --git a/skills/agentcontrol/legacy-sdk-migrate/references/api-mapping.md b/skills/agentcontrol/legacy-sdk-migrate/references/api-mapping.md new file mode 100644 index 00000000..1e842596 --- /dev/null +++ b/skills/agentcontrol/legacy-sdk-migrate/references/api-mapping.md @@ -0,0 +1,445 @@ +# Legacy to current AI SDK: before and after + +Each section shows the legacy pattern and its replacement in Python and Node.js. All current examples assume `LD_SDK_KEY` and the provider API keys are set in the environment. + +Handler factory names follow `create___handler` (Python) and `createHandler` (Node.js), for example `create_openai_messages_handler`, `create_openai_agents_handler`, `createOpenAIMessagesHandler`, `createOpenAIAgentsHandler`, `createClaudeMessagesHandler`, `createClaudeAgentsHandler`. Some published examples use other names, such as `createOpenAIHandler` or `create_openai_agent_handler`; these are not exported by the current packages. + +## Contents + +1. [API mapping](#api-mapping) +2. [Packages](#packages) +3. [Initialization](#initialization) +4. [Completion call: convenience function](#completion-call-convenience-function) +5. [Completion call: multi-provider handlers](#completion-call-multi-provider-handlers) +6. [Fallbacks and inspection](#fallbacks-and-inspection) +7. [Agent mode and tools](#agent-mode-and-tools) +8. [Registries](#registries) +9. [Streaming and history](#streaming-and-history) +10. [Judges](#judges) +11. [Agent graphs](#agent-graphs) + +## API mapping + +| Concern | Legacy | Current | +|---------|--------|---------| +| Init | `LDAIClient(ld_client)` / `initAi(ldClient)` | Lazy init from `LD_SDK_KEY`, or `init_client()` / `initClient()` | +| Completion | `completion_config()` / `completionConfig()` | `config().invoke()` or a convenience function | +| Agent | `agent_config()` / `agentConfig()` | `config().invoke()`; mode comes from the variation | +| Multi-agent | `agent_configs()` / `agentConfigs()` | `graph()` or a native framework runner | +| Managed objects | `create_model()`, `create_agent()` / `createModel().run()`, `createAgent()` | Convenience function or `config().invoke()` | +| Metrics | `create_tracker()`, `track_metrics_of`, `track_*` / `createTracker()`, `trackMetricsOf`, `track*` | Automatic on every call, plus OpenTelemetry spans | +| Fallback | Default config object with `enabled=False` | `try`/`except` around `invoke()`; `inspect_config()` / `inspectConfig()` to read state | +| Template variables | Fourth argument to `completion_config` / `completionConfig` | `variables=` (convenience, Python), `{ variables }` (convenience, Node), or third argument to `invoke()` | +| Judges | `create_judge()`, `judge_config()` / `createJudge()`, `judgeConfig()` | Judges attached to the variation; `skip_judges` / `skipJudges` plus `run_judge()` / `runJudge()` for async work | +| Graphs | `agent_graph()`, `create_agent_graph()`, traversal helpers | `graph()`, `resolve_graph()` / `resolveGraph()` | +| Shutdown | `ld_client.close()` / `ldClient.close()` | `shutdown()` | + +## Packages + +| Language | Remove | Add | +|----------|--------|-----| +| Python | `launchdarkly-server-sdk-ai`, `launchdarkly-server-sdk-ai-openai`, `launchdarkly-server-sdk-ai-langchain` (imports `ldai`, `ldai_openai`, `ldai_langchain`) | `launchdarkly-server-sdk` + `launchdarkly-ai-server`, or `"launchdarkly-ai-python[ld]"`; plus one `launchdarkly-ai--` per handler | +| Node.js | `@launchdarkly/server-sdk-ai`, `@launchdarkly/server-sdk-ai-openai`, `@launchdarkly/server-sdk-ai-langchain`, `@launchdarkly/server-sdk-ai-vercel` | `@launchdarkly/ai-node` (bundles `@launchdarkly/node-server-sdk`), or `@launchdarkly/ai-server` for edge runtimes; plus one `@launchdarkly/ai--` per handler | + +```bash +# Python (3.12+) +pip uninstall launchdarkly-server-sdk-ai launchdarkly-server-sdk-ai-openai +pip install launchdarkly-server-sdk launchdarkly-ai-server launchdarkly-ai-openai-messages + +# Node.js +npm uninstall @launchdarkly/server-sdk-ai @launchdarkly/server-sdk-ai-openai +npm install @launchdarkly/ai-node @launchdarkly/ai-openai-messages +``` + +Python modules are the package name with underscores (`launchdarkly_ai_server`, `launchdarkly_ai_openai_messages`). If you install `launchdarkly-ai-python`, import the core API from `launchdarkly_ai_python` instead. + +## Initialization + +Legacy: + +```python +from ldclient import set_config, get +from ldclient.config import Config +from ldai import LDAIClient + +set_config(Config("YOUR_SDK_KEY")) +aiclient = LDAIClient(get()) +``` + +```typescript +import { init } from '@launchdarkly/node-server-sdk'; +import { initAi } from '@launchdarkly/server-sdk-ai'; + +const ldClient = init('YOUR_SDK_KEY'); +await ldClient.waitForInitialization({ timeout: 10 }); +const aiClient = initAi(ldClient); +``` + +Current: delete both. The first AI call initializes the client from `LD_SDK_KEY`. Pre-warm or pass options only if needed, and always flush on exit: + +```python +from launchdarkly_ai_server import init_client, shutdown + +await init_client() # optional +... +await shutdown() +``` + +```typescript +import { initClient, shutdown } from '@launchdarkly/ai-node'; + +await initClient(); // optional +// Edge or custom runtime: await initClient(preInitializedClient); +... +await shutdown(); +``` + +If the app also evaluates ordinary feature flags, keep its existing server SDK client for those. To share one client, pass it to `init_client(client=ld_client)` / `initClient(ldClient)`. + +## Completion call: convenience function + +Use when the config serves one provider in one mode. + +Legacy: + +```python +from ldai import AICompletionConfigDefault +from ldai_openai import get_ai_metrics_from_response + +config = aiclient.completion_config( + "example-config-key", + context, + AICompletionConfigDefault(enabled=False), + {"example_custom_variable": "example_custom_value"}, +) +tracker = config.create_tracker() + +if config.enabled: + messages = [] if config.messages is None else config.messages + completion = tracker.track_metrics_of( + get_ai_metrics_from_response, + lambda: openai_client.chat.completions.create( + model=config.model.name, + messages=[m.to_dict() for m in messages] + [{"role": "user", "content": user_input}], + ), + ) + text = completion.choices[0].message.content +``` + +```typescript +import { getAIMetricsFromResponse } from '@launchdarkly/server-sdk-ai-openai'; + +const aiConfig = await aiClient.completionConfig( + 'example-config-key', + context, + { enabled: false }, + { exampleCustomVariable: 'exampleCustomValue' }, +); +const tracker = aiConfig.createTracker(); +const response = await tracker.trackMetricsOf(getAIMetricsFromResponse, () => + openai.chat.completions.create({ + model: aiConfig.model?.name || 'gpt-4', + messages: [...(aiConfig.messages || []), { role: 'user', content: userInput }], + }), +); +const text = response.choices[0].message.content; +``` + +Current: + +```python +from launchdarkly_ai_openai_messages import openai_messages + +result = await openai_messages( + "example-config-key", + user_input, + {"kind": "user", "key": "user-123"}, + variables={"example_custom_variable": "example_custom_value"}, +) +text = result.response +``` + +```typescript +import { openaiMessages } from '@launchdarkly/ai-openai-messages'; + +const result = await openaiMessages( + 'example-config-key', + userInput, + { kind: 'user', key: 'user-123' }, + { variables: { exampleCustomVariable: 'exampleCustomValue' } }, +); +const text = result.response; +``` + +Other convenience functions follow the same shape: `openai_agents` / `openaiAgents`, `claude_messages` / `claudeMessages`, `claude_agents` / `claudeAgents`, and so on. Check the handler package's exports. + +## Completion call: multi-provider handlers + +Use when variations of one config can target different providers or modes. The SDK routes to the handler that matches the variation's provider and mode. + +```python +from launchdarkly_ai_server import config +from launchdarkly_ai_openai_messages import create_openai_messages_handler +from launchdarkly_ai_claude_messages import create_claude_messages_handler + +result = await config( + key="example-config-key", + handler=[create_openai_messages_handler(), create_claude_messages_handler()], +).invoke( + user_input, + {"kind": "user", "key": "user-123"}, + {"example_custom_variable": "example_custom_value"}, +) +print(result.response, result.usage) +``` + +```typescript +import { config } from '@launchdarkly/ai-node'; +import { createOpenAIMessagesHandler } from '@launchdarkly/ai-openai-messages'; +import { createClaudeMessagesHandler } from '@launchdarkly/ai-claude-messages'; + +const result = await config({ + key: 'example-config-key', + handler: [createOpenAIMessagesHandler(), createClaudeMessagesHandler()], +}).invoke( + userInput, + { kind: 'user', key: 'user-123' }, + { exampleCustomVariable: 'exampleCustomValue' }, +); +console.log(result.response, result.usage); +``` + +## Fallbacks and inspection + +There is no fallback argument. `invoke()` raises when the variation is disabled (`RuntimeError: Variation '' is disabled.` in Python, `Error: Variation is not enabled` in Node.js) or when LaunchDarkly cannot be reached. Port the legacy `if not config.enabled:` branch into an error handler: + +```python +FALLBACK = "Sorry, the assistant is unavailable right now." + +try: + result = await openai_messages("example-config-key", user_input, context) + text = result.response +except Exception: + logger.exception("AI config call failed; using fallback") + text = FALLBACK +``` + +```typescript +let text: string; +try { + text = (await openaiMessages('example-config-key', userInput, context)).response; +} catch (err) { + logger.error('AI config call failed; using fallback', err); + text = FALLBACK; +} +``` + +To read state without raising and without calling a provider or emitting metrics, use `inspect_config()` / `inspectConfig()`. It returns `enabled`, `config` (the evaluated variation, or `None` / `null`), and `meta`: + +```python +from launchdarkly_ai_server import inspect_config + +state = await inspect_config("example-config-key", context) +if not state["enabled"]: + return FALLBACK +``` + +```typescript +import { inspectConfig } from '@launchdarkly/ai-node'; + +const state = await inspectConfig('example-config-key', context); +if (!state.enabled) return FALLBACK; +``` + +Inspecting first and then invoking evaluates the flag twice; keep the `try`/`except` either way, because LaunchDarkly can become unreachable between the two calls. + +## Agent mode and tools + +Legacy agent mode used a separate evaluation call, and the app ran its own tool loop and tracked metrics: + +```python +from ldai import AIAgentConfigDefault + +agent = aiclient.agent_config( + "example-config-key", context, AIAgentConfigDefault(enabled=False), + {"example_custom_variable": "example_custom_value"}, +) +if agent.enabled: + tracker = agent.create_tracker() + result = run_my_tool_loop(agent.instructions, tools={"get-prefs": get_preferences_fn}) + tracker.track_success() +``` + +```typescript +const agent = await aiClient.agentConfig( + 'example-config-key', context, { enabled: false }, + { exampleCustomVariable: 'exampleCustomValue' }, +); +if (agent.enabled) { + // agent.instructions + hand-written tool loop + agent.tracker.track*() +} +``` + +Current: completion and agent mode share `config()`. Register tool implementations by name; the handler runs the tool loop. Keys must match the tool names in the variation exactly, including case. + +```python +from launchdarkly_ai_server import config +from launchdarkly_ai_openai_agents import create_openai_agents_handler + +result = await config( + key="example-config-key", + handler=[create_openai_agents_handler()], + tool_handlers={"get-prefs": get_preferences_fn}, +).invoke( + "What are my preferences?", + {"kind": "user", "key": "user-123"}, + {"example_custom_variable": "example_custom_value"}, +) +``` + +```typescript +import { config } from '@launchdarkly/ai-node'; +import { createOpenAIAgentsHandler } from '@launchdarkly/ai-openai-agents'; + +const result = await config({ + key: 'example-config-key', + handler: [createOpenAIAgentsHandler()], + toolHandlers: { 'get-prefs': getPreferencesFn }, +}).invoke( + 'What are my preferences?', + { kind: 'user', key: 'user-123' }, + { exampleCustomVariable: 'exampleCustomValue' }, +); +``` + +Delete the hand-written loop, the provider tool-schema construction, and the manual `track_tool_call` / `trackToolCall` calls. + +## Registries + +Register handlers and tools once, then pass the registry to every call site. Use a scoped `Registry` when parts of the app need different handlers or tools, and `global_registry` / `globalRegistry` when one set fits the whole app. + +```python +from launchdarkly_ai_server import Registry, config +from launchdarkly_ai_openai_messages import create_openai_messages_handler +from launchdarkly_ai_openai_agents import create_openai_agents_handler + +openai_registry = Registry( + handlers=[create_openai_messages_handler(), create_openai_agents_handler()], + tools={"get-prefs": get_preferences_fn}, +) + +result = await config(key="example-config-key", registry=openai_registry).invoke(user_input, context) +``` + +```typescript +import { Registry, config } from '@launchdarkly/ai-node'; +import { createOpenAIMessagesHandler } from '@launchdarkly/ai-openai-messages'; +import { createOpenAIAgentsHandler } from '@launchdarkly/ai-openai-agents'; + +const openaiRegistry = new Registry({ + handlers: [createOpenAIMessagesHandler(), createOpenAIAgentsHandler()], + tools: { 'get-prefs': getPreferencesFn }, +}); + +const result = await config({ key: 'example-config-key', registry: openaiRegistry }).invoke(userInput, context); +``` + +Some providers expose built-in tools (for example Claude web search) that you assign as tool implementations instead of writing a function. + +## Streaming and history + +Replace `trackStreamMetricsOf` / manual time-to-first-token tracking with `stream()`. It takes the same arguments as `invoke()`. + +```python +async for event in config(key="example-config-key", registry=openai_registry).stream(user_input, context): + if event["type"] == "chunk": + print(event["text"], end="") + elif event["type"] == "done": + final_text, usage = event["response"], event["usage"] +``` + +```typescript +for await (const event of config({ key: 'example-config-key', registry: openaiRegistry }).stream(userInput, context)) { + if (event.type === 'chunk') process.stdout.write(event.text); + if (event.type === 'done') console.log(event.usage); +} +``` + +For multi-turn chat, stop appending to a message array you merge with `config.messages`. Pass earlier turns as the fourth argument, `history`: + +```python +history = [ + {"role": "user", "content": "Hi"}, + {"role": "assistant", "content": "Hello! How can I help?"}, +] +result = await config(key="example-config-key", registry=openai_registry).invoke( + user_input, context, None, history +) +``` + +## Judges + +Legacy apps created a judge and called it after each response. Now judges are attached to the config variation in LaunchDarkly and run inline by default on every `invoke()`. Delete `create_judge` / `createJudge` call sites once the judges are attached to the variation (ask the user to attach them if they aren't). + +To keep latency low, skip inline judges and run them in the background: + +```python +import asyncio +from launchdarkly_ai_server import config, global_registry, run_judge + +result = await config( + key="example-config-key", registry=global_registry, skip_judges=True +).invoke(user_input, context) + +async def judge_worker(task): + await run_judge(task, handlers) + +for task in result.judge_tasks or []: + asyncio.create_task(judge_worker(task)) +``` + +```typescript +import { config, globalRegistry, runJudge } from '@launchdarkly/ai-node'; + +const { response, judgeTasks } = await config({ + key: 'example-config-key', + registry: globalRegistry, + skipJudges: true, +}).invoke(userInput, context); + +for (const task of judgeTasks ?? []) { + void runJudge(task, handlers); // or hand `task` to a worker thread or queue +} +``` + +`handlers` is the list of provider handlers the judge configs can use. Delete manual `track_judge_result` / `trackJudgeResult` calls. + +## Agent graphs + +Replace `agent_configs` / `agentConfigs`, `agent_graph`, `create_agent_graph`, and hand-written traversal with `graph().invoke()`: + +```python +from launchdarkly_ai_server import graph +from launchdarkly_ai_claude_agents import create_claude_agents_handler +from launchdarkly_ai_openai_agents import create_openai_agents_handler + +result = await graph( + "support-graph", + handlers=[create_claude_agents_handler(), create_openai_agents_handler()], + tool_handlers={"lookup-invoice": lookup_invoice}, +).invoke("I was double charged", {"kind": "user", "key": "user-123"}, {"account_tier": "pro"}) +print(result.response) +``` + +```typescript +import { graph } from '@launchdarkly/ai-node'; +import { createClaudeAgentsHandler } from '@launchdarkly/ai-claude-agents'; + +const result = await graph('support-graph', { + handlers: [createClaudeAgentsHandler()], + toolHandlers: { 'lookup-invoice': lookupInvoice }, +}).invoke('I was double charged', { kind: 'user', key: 'user-123' }, { account_tier: 'pro' }); +console.log(result.response); +``` + +To keep running on a framework the app already uses, call `resolve_graph()` / `resolveGraph()` and pass the result to that framework's runner: `to_lang_graph` / `toLangGraph`, `to_openai_agents` / `toOpenAIAgents`, or `to_claude_agents` / `toClaudeAgents`. Check the handler package's exports and README for the exact call. From e9e223167665d1a0c15446f5d64fded2aa7c2322 Mon Sep 17 00:00:00 2001 From: Scarlett Attensil Date: Mon, 9 Nov 2026 11:00:51 -0800 Subject: [PATCH 2/6] fix(legacy-sdk-migrate): add feedback tracking, judge metric tracking, and tool/template details Co-authored-by: Cursor --- CHANGELOG.md | 2 +- README.md | 4 +- skills.json | 2 +- .../agentcontrol/legacy-sdk-migrate/SKILL.md | 16 ++-- .../references/api-mapping.md | 82 ++++++++++++++++--- 5 files changed, 84 insertions(+), 22 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index b6c8dc66..17892c5b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,7 +4,7 @@ All notable changes to this repository will be documented in this file. ## Unreleased -- Add `agentcontrol/legacy-sdk-migrate` 0.1.0: migrate an app from the legacy AI SDKs (`launchdarkly-server-sdk-ai`, `@launchdarkly/server-sdk-ai`) to the current handler-based SDKs, following [Migrate from the legacy AI SDKs](https://launchdarkly.com/docs/sdk/ai/migration). Inventories legacy calls, maps each to a convenience function, `config().invoke()`, or `graph()`, moves tools into `tool_handlers` / `toolHandlers` or a `Registry`, replaces trackers with automatic metrics, ports `enabled=False` fallbacks to error handling and `inspect_config()` / `inspectConfig()`, and covers streaming, history, async judges, and graphs. `references/api-mapping.md` has before/after code in Python and Node.js using the handler factory names the packages export. +- Add `agentcontrol/legacy-sdk-migrate` 0.1.0: migrate an app from the legacy AI SDKs (`launchdarkly-server-sdk-ai`, `@launchdarkly/server-sdk-ai`) to the current handler-based SDKs, following [Migrate from the legacy AI SDKs](https://launchdarkly.com/docs/sdk/ai/migration). Inventories legacy calls, maps each to a convenience function, `config().invoke()`, or `graph()`, moves tools into `tool_handlers` / `toolHandlers` or a `Registry`, replaces trackers with automatic metrics (and `track_feedback` with `result.track_data`), ports `enabled=False` fallbacks to error handling and `inspect_config()` / `inspectConfig()`, and covers streaming, history, async judges, and graphs. `references/api-mapping.md` has before/after code in Python and Node.js using the handler factory names the packages export. - Add `release-autopilot/launchdarkly-automations-cleanup-settings` and `release-autopilot/launchdarkly-automations-archival-settings` 0.1.0-experimental (Claude Code plugin 1.0.7, Cursor plugin 1.0.8): read and change the auto-cleanup and auto-archival sections of the project Automations page through the hosted MCP tools `get-automations` (`section: "cleanup"` or `"archival"`) and `update-automations` (a `cleanup` or `archival` object), and explain why cleanup did not open a pull request or a flag was not archived. Every write is confirmed first. Turning auto-archival on stays with a person, who confirms its risk on the Automations page. On a server without the tools, both skills guide the browser steps. Onboarding and the settings skill point to the new skills for changes after setup. Pairs with launchdarkly/mcp#462. - `release-autopilot/launchdarkly-release-autopilot-onboarding` 0.4.0-experimental and `release-autopilot/launchdarkly-release-autopilot-settings` 1.5.0-experimental: each repository's own settings decide what runs in it. On accounts that use the project Automations settings page, the account settings are not read, and a capability a repo does not turn on is off. Accounts that have not moved yet still use the account settings too, so the account tools and the account write stay. Onboarding now turns on each picked capability in the repo write, and the settings skill sends both capabilities on a new mapping and diagnoses a PR from the repo's effective setting and `enabledOverride` instead of stopping at the account setting (EMSR-2650). Bring your own key, auto-cleanup, and auto-archival link to `/projects/{projectKey}/settings/automations` instead of `/settings/vega` and `/settings/lifecycle`, which stay as the fallback on accounts without the Automations page (EMSR-2676). The onboarding prerequisites and README say to install the LaunchDarkly plugin first, with the Claude Code and Cursor steps and the first plugin versions that include the skill (EMSR-2634). Evals check the Automations link and that the mapping turns auto-flagging on. - `observability/investigate` 0.3.0 and `observability/alert-investigation` 0.2.0: rewritten around the hosted MCP's current tools (`get-changes`, `get-trace`, `get-error-group`, `search-service-map`, `search-log-patterns`, `get-alert-firing-history`, `get-investigation`, `get-incidents`, `search-aggregations` with `compareTo` and breakdowns). `investigate` adds a "first five minutes" sequence: scope and onset, chronic or noise, what changed, what is over-represented against a baseline (by host, version, route and flag variation), evidence, blast radius; and warns about percentiles over few sampled rows. `alert-investigation` starts from `get-alerts` and `get-alert-firing-history` (replay, chronic check, automatic investigation) instead of alert variables injected into the conversation. The 50-row rule is gone; companions use the stable `search-*` tool names and current arguments. The guides also use the fields from launchdarkly/mcp#449 (firing-history `summary` and `chronic`, get-changes `alert_episode` and evaluated-only flag changes, get-trace `self_ms`/`critical_path`/`top_self_time`, project-wide `search-flag-evaluations` with `include` and the FLAG FILTERS keys, the `low_sample` warning, get-error-group `include` and `_pivots`, `sort: "spike"`, and the `ingest` block with `completeness_note`), each with a fallback for servers that predate it. diff --git a/README.md b/README.md index c6894e53..0c5e58d4 100644 --- a/README.md +++ b/README.md @@ -38,8 +38,8 @@ Agent Skills are modular, text-based playbooks that teach an agent how to perfor | Skill | Description | |-------|-------------| | `agentcontrol/configs-create` | Create configs with variations for agent or completion mode | -| `agentcontrol/migrate` | Migrate an app with hardcoded LLM prompts to AgentControl in five stages (extract, wrap, tools, tracking, evals) | -| `agentcontrol/legacy-sdk-migrate` | Migrate from the legacy Python and Node.js AI SDKs to the current handler-based SDKs (packages, init, `config()`, tools, registries, judges, graphs, fallbacks) | +| `agentcontrol/migrate` | Migrate an app with hardcoded LLM prompts (not on AgentControl yet) to the current AI SDKs in five stages | +| `agentcontrol/legacy-sdk-migrate` | Migrate between LaunchDarkly AI SDKs: from the legacy Python and Node.js AI SDKs to the current handler-based SDKs (packages, init, `config()`, tools, registries, judges, graphs, fallbacks) | | `agentcontrol/configs-update` | Update and delete configs, manage lifecycle | | `agentcontrol/configs-variations` | Manage config variations for A/B testing | | `agentcontrol/tools` | Create and attach tools for function calling | diff --git a/skills.json b/skills.json index 1d9b715d..c8ebf210 100644 --- a/skills.json +++ b/skills.json @@ -504,7 +504,7 @@ }, { "name": "legacy-sdk-migrate", - "description": "Migrate an app from the legacy LaunchDarkly AI SDKs (Python `launchdarkly-server-sdk-ai` / `ldai`, Node.js `@launchdarkly/server-sdk-ai`) to the current handler-based AI SDKs (`launchdarkly-ai-server`, `@launchdarkly/ai-node` / `@launchdarkly/ai-server` plus handler packages). Use when code calls `LDAIClient`, `initAi`, `completion_config` / `completionConfig`, `agent_config` / `agentConfig`, `agent_configs`, `create_tracker` / `trackMetricsOf`, `create_judge`, `create_model`, or `agent_graph`, or when the user asks to upgrade to the turnkey AI SDKs.", + "description": "Migrate an app from the legacy LaunchDarkly AI SDKs (Python `launchdarkly-server-sdk-ai` / `ldai`, Node.js `@launchdarkly/server-sdk-ai`) to the current handler-based AI SDKs (`launchdarkly-ai-server`, `@launchdarkly/ai-node` / `@launchdarkly/ai-server` plus handler packages). Use when code calls `LDAIClient`, `initAi`, `completion_config` / `completionConfig`, `agent_config` / `agentConfig`, `agent_configs`, `create_tracker` / `trackMetricsOf`, `create_judge`, `create_model`, or `agent_graph`, or when the user asks to upgrade between LaunchDarkly AI SDKs. For apps with hardcoded LLM prompts that aren't on LaunchDarkly yet, use `migrate` instead.", "path": "skills/agentcontrol/legacy-sdk-migrate", "version": "0.1.0", "license": "Apache-2.0", diff --git a/skills/agentcontrol/legacy-sdk-migrate/SKILL.md b/skills/agentcontrol/legacy-sdk-migrate/SKILL.md index adfee6e1..69f645a4 100644 --- a/skills/agentcontrol/legacy-sdk-migrate/SKILL.md +++ b/skills/agentcontrol/legacy-sdk-migrate/SKILL.md @@ -1,6 +1,6 @@ --- name: legacy-sdk-migrate -description: "Migrate an app from the legacy LaunchDarkly AI SDKs (Python `launchdarkly-server-sdk-ai` / `ldai`, Node.js `@launchdarkly/server-sdk-ai`) to the current handler-based AI SDKs (`launchdarkly-ai-server`, `@launchdarkly/ai-node` / `@launchdarkly/ai-server` plus handler packages). Use when code calls `LDAIClient`, `initAi`, `completion_config` / `completionConfig`, `agent_config` / `agentConfig`, `agent_configs`, `create_tracker` / `trackMetricsOf`, `create_judge`, `create_model`, or `agent_graph`, or when the user asks to upgrade to the turnkey AI SDKs." +description: "Migrate an app from the legacy LaunchDarkly AI SDKs (Python `launchdarkly-server-sdk-ai` / `ldai`, Node.js `@launchdarkly/server-sdk-ai`) to the current handler-based AI SDKs (`launchdarkly-ai-server`, `@launchdarkly/ai-node` / `@launchdarkly/ai-server` plus handler packages). Use when code calls `LDAIClient`, `initAi`, `completion_config` / `completionConfig`, `agent_config` / `agentConfig`, `agent_configs`, `create_tracker` / `trackMetricsOf`, `create_judge`, `create_model`, or `agent_graph`, or when the user asks to upgrade between LaunchDarkly AI SDKs. For apps with hardcoded LLM prompts that aren't on LaunchDarkly yet, use `migrate` instead." license: Apache-2.0 compatibility: Python 3.12+ or Node.js. No LaunchDarkly MCP tools needed; the AgentControl configs already exist. metadata: @@ -18,7 +18,7 @@ Source guide: [Migrate from the legacy AI SDKs](https://launchdarkly.com/docs/sd ## Prerequisites -- The app already uses a legacy AI SDK, and its AgentControl configs (formerly AI Configs) already exist in LaunchDarkly. The config keys, variations, tools, and judges carry over unchanged; this is a code migration only. +- The app already uses a legacy AI SDK, and its AgentControl configs (formerly AI Configs) already exist in LaunchDarkly. The config keys, variations, tools, and judges carry over unchanged; this is a code migration only. (If the app calls an LLM provider directly with hardcoded prompts and does not use LaunchDarkly yet, use the `migrate` skill instead.) - `LD_SDK_KEY` (server-side SDK key) and the provider API keys (`OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, ...) available as environment variables. The current SDKs read them from the environment. - **Python:** the current packages require **Python 3.12 or later**, and the API is **async** (`await config(...).invoke(...)`). Check both before you start. - **Node.js:** `@launchdarkly/ai-node` for Node; `@launchdarkly/ai-server` plus a pre-initialized client for edge or custom runtimes. @@ -95,10 +95,10 @@ Work one call site at a time. Full code for each step is in [references/api-mapp - Keep the base server SDK (`launchdarkly-server-sdk` / `@launchdarkly/node-server-sdk`) only if the app still evaluates ordinary feature flags with it. 2. **Replace initialization.** Delete `LDAIClient(...)` / `initAi(...)`. The SDK initializes lazily from `LD_SDK_KEY` on the first AI call. Call `init_client()` / `initClient()` only to pass options or pre-warm, and call `shutdown()` on exit to flush events and spans. 3. **Rewrite each call.** Replace evaluate → build messages → call provider → track with one convenience function or `config(...).invoke(user_input, context, variables)`. Read the output from `result.response`. Python contexts are plain dicts (`{"kind": "user", "key": "user-123"}`). -4. **Move template variables** from the `completion_config` / `agent_config` argument to the `variables` keyword (convenience functions, Python), the `{ variables }` option (convenience functions, Node), or the third argument to `invoke()`. -5. **Move tools.** Register implementations by name in `tool_handlers` / `toolHandlers` or a `Registry`. Keys must match the tool names in the config variation exactly, including case. Delete the hand-written tool loop. +4. **Move template variables** from the `completion_config` / `agent_config` argument to the `variables` keyword (convenience functions, Python), the `{ variables }` option (convenience functions, Node), or the third argument to `invoke()`. Note that the current SDKs inject the evaluation context as `{{ldContext.*}}` (replacing legacy `{{ldctx.*}}`) and require Mustache placeholders without spaces (`{{name}}`, not `{{ name }}`). +5. **Move tools.** Register implementations by name in `tool_handlers` / `toolHandlers` or a `Registry`. Keys must match the tool names in the config variation exactly, including case. Each tool function is called with a single `args` dict/object (not unpacked keyword arguments), so wrap multi-argument functions (`lambda args: fn(**args)`) or LangChain `@tool` objects (`lambda args: tool.invoke(args)`). Delete the hand-written tool loop. 6. **Preserve fallbacks.** Wrap `invoke()` and return the app's previous fallback response on error. Use `inspect_config()` / `inspectConfig()` (never raises; returns `enabled`, `config`, `meta`) for health checks or to branch before invoking. -7. **Delete tracking code.** Remove `create_tracker`, every `track_*` / `track*` call, and `get_ai_metrics_from_response` / `getAIMetricsFromResponse` helpers. Metrics and spans are automatic. +7. **Delete tracking code.** Remove `create_tracker`, every `track_*` / `track*` call, and `get_ai_metrics_from_response` / `getAIMetricsFromResponse` helpers. Metrics and spans are automatic. If the app records user feedback (`track_feedback` / `trackFeedback`), replace it with `get_client().track()` / `getClient().track()` using `result.track_data` / `result.trackData` (see [references/api-mapping.md](references/api-mapping.md#user-feedback)). 8. **Centralize.** If two or more call sites share handlers or tools, create one `Registry` (or register on `global_registry` / `globalRegistry` at startup) and pass `registry=`. 9. **Remove dead imports and packages** once nothing references them. @@ -118,10 +118,10 @@ Work one call site at a time. Full code for each step is in [references/api-mapp | Python below 3.12 | Stop and tell the user. The current Python packages need 3.12+. | | Synchronous Python code path (Flask, Django views, scripts) | Run the async call with `asyncio.run(...)` at the boundary, or move the handler to an async framework. Ask which. | | App reads provider-native fields (`choices`, `tool_calls`, `stop_reason`) | Use `result.response` and `result.usage`; ask before changing downstream logic that needs the raw object. | -| Provider has no handler package (Bedrock, Gemini, Azure, internal gateway) | Use `litellm-*`, `langchain-*`, or `vercel-*`, or build a custom handler with `createHandler` / `create_handler`. | -| App calls `track_feedback` / `trackFeedback` for thumbs up or down | The current SDKs have no feedback helper. Don't silently delete the call: tell the user, and leave it in place on the legacy client or ask how they want to record feedback. | +| Provider has no handler package (Bedrock, Gemini, Azure, internal gateway) | Use `litellm-*`, `langchain-*`, or `vercel-*`, or build a custom handler with `createHandler` / `create_handler`. In Python, `provides_for` must be a tuple `("ProviderName", "messages")`, and custom handlers must call `parse_template` / `parseTemplate` on raw prompt strings. | +| App calls `track_feedback` / `trackFeedback` for thumbs up or down | Replace `tracker.track_feedback` with `get_client().track("$ld:ai:feedback:user:positive" or ":negative", Context.from_dict(context), result.track_data, 1)` in Python or `getClient().track(event, context, result.trackData, 1)` in Node.js (note: `stream()` does not currently return `track_data`). | | Large app, gradual rollout | Migrate one call site at a time, verify it, and remove the legacy packages only after the last call site moves. | -| Judges must not add latency | `skip_judges=True` / `skipJudges: true`, then run `judge_tasks` / `judgeTasks` in a background task or worker with `run_judge()` / `runJudge()`. | +| Judges must not add latency | `skip_judges=True` / `skipJudges: true`, then run `judge_tasks` / `judgeTasks` in a background task or worker with `run_judge()` / `runJudge()` and record the score with `get_client().track()` / `getClient().track()`. | | Agent Skills from AgentControl are needed | Agent Skills reach apps only through the current SDKs; the legacy SDKs never deliver them. | ## What NOT to Do diff --git a/skills/agentcontrol/legacy-sdk-migrate/references/api-mapping.md b/skills/agentcontrol/legacy-sdk-migrate/references/api-mapping.md index 1e842596..4bcee7e9 100644 --- a/skills/agentcontrol/legacy-sdk-migrate/references/api-mapping.md +++ b/skills/agentcontrol/legacy-sdk-migrate/references/api-mapping.md @@ -15,8 +15,9 @@ Handler factory names follow `create___handler` (Python) and `cr 7. [Agent mode and tools](#agent-mode-and-tools) 8. [Registries](#registries) 9. [Streaming and history](#streaming-and-history) -10. [Judges](#judges) -11. [Agent graphs](#agent-graphs) +10. [User feedback](#user-feedback) +11. [Judges](#judges) +12. [Agent graphs](#agent-graphs) ## API mapping @@ -28,8 +29,9 @@ Handler factory names follow `create___handler` (Python) and `cr | Multi-agent | `agent_configs()` / `agentConfigs()` | `graph()` or a native framework runner | | Managed objects | `create_model()`, `create_agent()` / `createModel().run()`, `createAgent()` | Convenience function or `config().invoke()` | | Metrics | `create_tracker()`, `track_metrics_of`, `track_*` / `createTracker()`, `trackMetricsOf`, `track*` | Automatic on every call, plus OpenTelemetry spans | +| Feedback | `tracker.track_feedback()` / `tracker.trackFeedback()` | `get_client().track()` / `getClient().track()` with `result.track_data` / `result.trackData` | | Fallback | Default config object with `enabled=False` | `try`/`except` around `invoke()`; `inspect_config()` / `inspectConfig()` to read state | -| Template variables | Fourth argument to `completion_config` / `completionConfig` | `variables=` (convenience, Python), `{ variables }` (convenience, Node), or third argument to `invoke()` | +| Template variables | Fourth argument to `completion_config` / `completionConfig` | `variables=` (convenience, Python), `{ variables }` (convenience, Node), or third argument to `invoke()` (`{{name}}` without spaces; `{{ldContext.*}}` replaces `{{ldctx.*}}`) | | Judges | `create_judge()`, `judge_config()` / `createJudge()`, `judgeConfig()` | Judges attached to the variation; `skip_judges` / `skipJudges` plus `run_judge()` / `runJudge()` for async work | | Graphs | `agent_graph()`, `create_agent_graph()`, traversal helpers | `graph()`, `resolve_graph()` / `resolveGraph()` | | Shutdown | `ld_client.close()` / `ldClient.close()` | `shutdown()` | @@ -280,7 +282,7 @@ if (agent.enabled) { } ``` -Current: completion and agent mode share `config()`. Register tool implementations by name; the handler runs the tool loop. Keys must match the tool names in the variation exactly, including case. +Current: completion and agent mode share `config()`. Register tool implementations by name; the handler runs the tool loop. Keys must match the tool names in the variation exactly, including case. Each tool function receives a single `args` dict (Python) or object (Node.js), not unpacked keyword arguments—wrap multi-argument functions (`lambda args: get_preferences_fn(**args)`) or LangChain `@tool` objects (`lambda args: get_preferences_tool.invoke(args)`). ```python from launchdarkly_ai_server import config @@ -377,29 +379,84 @@ result = await config(key="example-config-key", registry=openai_registry).invoke ) ``` +## User feedback + +The current SDKs do not have a `track_feedback` / `trackFeedback` helper on a tracker, because there is no tracker. Instead, `invoke()` returns `result.track_data` (Python) / `result.trackData` (Node.js) identifying the config, variation, version, and model. Pass it to `get_client().track()` / `getClient().track()` when the user gives thumbs up or down: + +Legacy: + +```python +from ldai.tracker import FeedbackKind + +tracker.track_feedback({"kind": FeedbackKind.Positive}) +``` + +```typescript +import { LDFeedbackKind } from '@launchdarkly/server-sdk-ai'; + +tracker.trackFeedback({ kind: LDFeedbackKind.Positive }); +``` + +Current: + +```python +from ldclient import Context +from launchdarkly_ai_server import config, get_client + +result = await config(key="example-config-key", registry=openai_registry).invoke(user_input, context) + +# When the user responds with thumbs up / down: +event = "$ld:ai:feedback:user:positive" if thumbs_up else "$ld:ai:feedback:user:negative" +get_client().track(event, Context.from_dict(context), result.track_data, 1) +``` + +```typescript +import { config, getClient } from '@launchdarkly/ai-node'; + +const result = await config({ key: 'example-config-key', registry: openaiRegistry }).invoke(userInput, context); + +// When the user responds with thumbs up / down: +const event = thumbsUp ? '$ld:ai:feedback:user:positive' : '$ld:ai:feedback:user:negative'; +getClient().track(event, context, result.trackData, 1); +``` + +`stream()` does not currently return `track_data`, so feedback on streamed responses cannot yet be attributed to a variation. + ## Judges Legacy apps created a judge and called it after each response. Now judges are attached to the config variation in LaunchDarkly and run inline by default on every `invoke()`. Delete `create_judge` / `createJudge` call sites once the judges are attached to the variation (ask the user to attach them if they aren't). -To keep latency low, skip inline judges and run them in the background: +To keep latency low, skip inline judges and run them in the background. `run_judge()` / `runJudge()` runs the judge model call and returns the score and merged `track_data` / `trackData`; record the evaluation metric with `get_client().track()` / `getClient().track()`: ```python import asyncio -from launchdarkly_ai_server import config, global_registry, run_judge +from ldclient import Context +from launchdarkly_ai_server import config, get_client, global_registry, run_judge result = await config( key="example-config-key", registry=global_registry, skip_judges=True ).invoke(user_input, context) +_background_tasks = set() + async def judge_worker(task): - await run_judge(task, handlers) + judge_result = await run_judge(task, handlers) + if judge_result and task.evaluation_metric_key: + get_client().track( + task.evaluation_metric_key, + Context.from_dict(task.user_context), + judge_result.track_data, + judge_result.score, + ) for task in result.judge_tasks or []: - asyncio.create_task(judge_worker(task)) + t = asyncio.create_task(judge_worker(task)) + _background_tasks.add(t) + t.add_done_callback(_background_tasks.discard) ``` ```typescript -import { config, globalRegistry, runJudge } from '@launchdarkly/ai-node'; +import { config, getClient, globalRegistry, runJudge } from '@launchdarkly/ai-node'; const { response, judgeTasks } = await config({ key: 'example-config-key', @@ -408,7 +465,12 @@ const { response, judgeTasks } = await config({ }).invoke(userInput, context); for (const task of judgeTasks ?? []) { - void runJudge(task, handlers); // or hand `task` to a worker thread or queue + void (async () => { + const judgeResult = await runJudge(task, handlers); + if (judgeResult && task.evaluationMetricKey) { + getClient().track(task.evaluationMetricKey, task.userContext, judgeResult.trackData, judgeResult.score); + } + })(); // or hand `task` to a worker thread or queue } ``` From d7c0bf3c6993aa0a1f1a0462deaca4e1d544b00e Mon Sep 17 00:00:00 2001 From: Scarlett Attensil Date: Mon, 9 Nov 2026 11:02:23 -0800 Subject: [PATCH 3/6] fix(legacy-sdk-migrate): preserve background judge metrics Co-authored-by: Cursor --- .../agentcontrol/legacy-sdk-migrate/SKILL.md | 9 +- .../references/api-mapping.md | 102 ++++++++++++++---- tests/test_legacy_sdk_migrate.py | 31 ++++++ 3 files changed, 118 insertions(+), 24 deletions(-) create mode 100644 tests/test_legacy_sdk_migrate.py diff --git a/skills/agentcontrol/legacy-sdk-migrate/SKILL.md b/skills/agentcontrol/legacy-sdk-migrate/SKILL.md index 69f645a4..0a04202e 100644 --- a/skills/agentcontrol/legacy-sdk-migrate/SKILL.md +++ b/skills/agentcontrol/legacy-sdk-migrate/SKILL.md @@ -65,7 +65,7 @@ Pick a target for each call site: | Streaming (`trackStreamMetricsOf`, manual TTFT) | `config(...).stream(...)`; iterate events (`chunk` events carry `text`; the final `done` event carries `response` and `usage`) | | Multi-turn chat (`createChat`, message arrays you maintain) | Pass prior turns as the `history` argument (fourth positional argument) to `invoke()` / `stream()` | | `agent_configs()` / `agentConfigs()`, `agent_graph()`, `create_agent_graph()`, managed graph runners | `graph(key, handlers=[...]).invoke(...)`, or `resolve_graph()` / `resolveGraph()` plus a native runner (`to_lang_graph` / `toLangGraph`, `to_openai_agents` / `toOpenAIAgents`, `to_claude_agents` / `toClaudeAgents`) | -| `create_judge()` / `createJudge()` / `judge_config()` | Judges attached to the config variation run automatically. For background scoring, pass `skip_judges=True` / `skipJudges: true` and hand each `judge_tasks` / `judgeTasks` item to `run_judge()` / `runJudge()` | +| `create_judge()` / `createJudge()` / `judge_config()` | Judges attached to the config variation run automatically. For background scoring, pass `skip_judges=True` / `skipJudges: true`, hand each `judge_tasks` / `judgeTasks` item to a durable worker, run `run_judge()` / `runJudge()`, and explicitly track the returned score(s) | | `create_model().run()` / `createModel().run()`, `create_agent()` | Same convenience function or `config().invoke()` path | | Same handlers and tools at many call sites | One shared `Registry` passed as `registry=` | @@ -98,7 +98,7 @@ Work one call site at a time. Full code for each step is in [references/api-mapp 4. **Move template variables** from the `completion_config` / `agent_config` argument to the `variables` keyword (convenience functions, Python), the `{ variables }` option (convenience functions, Node), or the third argument to `invoke()`. Note that the current SDKs inject the evaluation context as `{{ldContext.*}}` (replacing legacy `{{ldctx.*}}`) and require Mustache placeholders without spaces (`{{name}}`, not `{{ name }}`). 5. **Move tools.** Register implementations by name in `tool_handlers` / `toolHandlers` or a `Registry`. Keys must match the tool names in the config variation exactly, including case. Each tool function is called with a single `args` dict/object (not unpacked keyword arguments), so wrap multi-argument functions (`lambda args: fn(**args)`) or LangChain `@tool` objects (`lambda args: tool.invoke(args)`). Delete the hand-written tool loop. 6. **Preserve fallbacks.** Wrap `invoke()` and return the app's previous fallback response on error. Use `inspect_config()` / `inspectConfig()` (never raises; returns `enabled`, `config`, `meta`) for health checks or to branch before invoking. -7. **Delete tracking code.** Remove `create_tracker`, every `track_*` / `track*` call, and `get_ai_metrics_from_response` / `getAIMetricsFromResponse` helpers. Metrics and spans are automatic. If the app records user feedback (`track_feedback` / `trackFeedback`), replace it with `get_client().track()` / `getClient().track()` using `result.track_data` / `result.trackData` (see [references/api-mapping.md](references/api-mapping.md#user-feedback)). +7. **Replace tracking code.** Remove legacy `create_tracker`, generation/token/duration tracking, and `get_ai_metrics_from_response` / `getAIMetricsFromResponse` helpers; `invoke()` and `stream()` record those automatically. Preserve explicit app events: replace user feedback calls with `get_client().track()` / `getClient().track()` using `result.track_data` / `result.trackData` (see [references/api-mapping.md](references/api-mapping.md#user-feedback)). **Background judges are the exception:** `run_judge()` / `runJudge()` returns score data but deliberately does not call the LaunchDarkly client's `track()` method. Track every returned metric from the worker as shown in the reference. 8. **Centralize.** If two or more call sites share handlers or tools, create one `Registry` (or register on `global_registry` / `globalRegistry` at startup) and pass `registry=`. 9. **Remove dead imports and packages** once nothing references them. @@ -108,7 +108,7 @@ Work one call site at a time. Full code for each step is in [references/api-mapp 2. **Build and tests pass:** run the project's type check, linter, and test suite. Update mocks that patched `LDAIClient` / `initAi` to patch `config` or the convenience function instead. 3. **Happy path:** with `LD_SDK_KEY` set, run each migrated call and confirm it returns a response from the provider the variation names. 4. **Fallback path:** with targeting off for the config (or an invalid `LD_SDK_KEY`), confirm the app returns its fallback instead of crashing. -5. **Metrics:** after a few calls, the config's Monitoring tab in LaunchDarkly shows generations, tokens, and duration for the new SDK version. +5. **Metrics:** after a few calls, the config's Monitoring tab in LaunchDarkly shows generations, tokens, and duration for the new SDK version. If judges run in a worker, also confirm their evaluation metrics appear. 6. **Report:** list each migrated call site (file and line), packages added and removed, and anything left unmigrated with the reason. ## Edge Cases @@ -121,12 +121,13 @@ Work one call site at a time. Full code for each step is in [references/api-mapp | Provider has no handler package (Bedrock, Gemini, Azure, internal gateway) | Use `litellm-*`, `langchain-*`, or `vercel-*`, or build a custom handler with `createHandler` / `create_handler`. In Python, `provides_for` must be a tuple `("ProviderName", "messages")`, and custom handlers must call `parse_template` / `parseTemplate` on raw prompt strings. | | App calls `track_feedback` / `trackFeedback` for thumbs up or down | Replace `tracker.track_feedback` with `get_client().track("$ld:ai:feedback:user:positive" or ":negative", Context.from_dict(context), result.track_data, 1)` in Python or `getClient().track(event, context, result.trackData, 1)` in Node.js (note: `stream()` does not currently return `track_data`). | | Large app, gradual rollout | Migrate one call site at a time, verify it, and remove the legacy packages only after the last call site moves. | -| Judges must not add latency | `skip_judges=True` / `skipJudges: true`, then run `judge_tasks` / `judgeTasks` in a background task or worker with `run_judge()` / `runJudge()` and record the score with `get_client().track()` / `getClient().track()`. | +| Judges must not add latency | `skip_judges=True` / `skipJudges: true`, then put `judge_tasks` / `judgeTasks` on a durable worker or queue. Run `run_judge()` / `runJudge()` there and track the returned score(s) with their supplied track data. | | Agent Skills from AgentControl are needed | Agent Skills reach apps only through the current SDKs; the legacy SDKs never deliver them. | ## What NOT to Do - Don't keep the legacy tracker around the new call. The SDK already records metrics; you'll double-count. +- Do not fire and forget `run_judge()` / `runJudge()`. A request-scoped task can be cancelled before it finishes, and the runner returns metrics for the worker to track rather than tracking them itself. - Don't pass a default or fallback config object to `config()`; it has no such argument. Handle errors instead. - Don't construct an OpenAI or Anthropic client to pass in. Handlers build their own clients from environment variables. - Don't rename config keys, tools, or variations in LaunchDarkly as part of this migration. diff --git a/skills/agentcontrol/legacy-sdk-migrate/references/api-mapping.md b/skills/agentcontrol/legacy-sdk-migrate/references/api-mapping.md index 4bcee7e9..eec78470 100644 --- a/skills/agentcontrol/legacy-sdk-migrate/references/api-mapping.md +++ b/skills/agentcontrol/legacy-sdk-migrate/references/api-mapping.md @@ -28,11 +28,11 @@ Handler factory names follow `create___handler` (Python) and `cr | Agent | `agent_config()` / `agentConfig()` | `config().invoke()`; mode comes from the variation | | Multi-agent | `agent_configs()` / `agentConfigs()` | `graph()` or a native framework runner | | Managed objects | `create_model()`, `create_agent()` / `createModel().run()`, `createAgent()` | Convenience function or `config().invoke()` | -| Metrics | `create_tracker()`, `track_metrics_of`, `track_*` / `createTracker()`, `trackMetricsOf`, `track*` | Automatic on every call, plus OpenTelemetry spans | +| Metrics | `create_tracker()`, `track_metrics_of`, `track_*` / `createTracker()`, `trackMetricsOf`, `track*` | Generation metrics are automatic on `invoke()` / `stream()`; a background `run_judge()` / `runJudge()` returns judge metrics that the worker must track | | Feedback | `tracker.track_feedback()` / `tracker.trackFeedback()` | `get_client().track()` / `getClient().track()` with `result.track_data` / `result.trackData` | | Fallback | Default config object with `enabled=False` | `try`/`except` around `invoke()`; `inspect_config()` / `inspectConfig()` to read state | | Template variables | Fourth argument to `completion_config` / `completionConfig` | `variables=` (convenience, Python), `{ variables }` (convenience, Node), or third argument to `invoke()` (`{{name}}` without spaces; `{{ldContext.*}}` replaces `{{ldctx.*}}`) | -| Judges | `create_judge()`, `judge_config()` / `createJudge()`, `judgeConfig()` | Judges attached to the variation; `skip_judges` / `skipJudges` plus `run_judge()` / `runJudge()` for async work | +| Judges | `create_judge()`, `judge_config()` / `createJudge()`, `judgeConfig()` | Judges attached to the variation; `skip_judges` / `skipJudges` plus a durable worker that runs and tracks `run_judge()` / `runJudge()` results | | Graphs | `agent_graph()`, `create_agent_graph()`, traversal helpers | `graph()`, `resolve_graph()` / `resolveGraph()` | | Shutdown | `ld_client.close()` / `ldClient.close()` | `shutdown()` | @@ -426,37 +426,77 @@ getClient().track(event, context, result.trackData, 1); Legacy apps created a judge and called it after each response. Now judges are attached to the config variation in LaunchDarkly and run inline by default on every `invoke()`. Delete `create_judge` / `createJudge` call sites once the judges are attached to the variation (ask the user to attach them if they aren't). -To keep latency low, skip inline judges and run them in the background. `run_judge()` / `runJudge()` runs the judge model call and returns the score and merged `track_data` / `trackData`; record the evaluation metric with `get_client().track()` / `getClient().track()`: +To keep latency low, skip inline judges and put the returned tasks on a durable +worker or queue. `run_judge()` / `runJudge()` executes the judge call but does +**not** emit the judge's evaluation metric. The worker must track the returned +score and supplied track data. Do not use a request-scoped `create_task()` or an +unobserved promise: the request can end before either the judge or its metric +finishes. + +Python example for a long-lived process (use a durable queue instead when the +process can be suspended or terminated after the request): ```python import asyncio +import threading from ldclient import Context -from launchdarkly_ai_server import config, get_client, global_registry, run_judge +from launchdarkly_ai_server import ( + config, + get_client, + global_registry, + resolve_handlers, + run_judge, +) result = await config( key="example-config-key", registry=global_registry, skip_judges=True ).invoke(user_input, context) -_background_tasks = set() - -async def judge_worker(task): - judge_result = await run_judge(task, handlers) - if judge_result and task.evaluation_metric_key: +handlers = resolve_handlers(global_registry, None) or [] + +def run_and_track_judge(task): + judge_result = asyncio.run(run_judge(task, handlers)) + if judge_result is None: + return + + ld_context = Context.from_dict(task.user_context) + if judge_result.metrics: + for metric in judge_result.metrics: + get_client().track( + metric["eventKey"], + ld_context, + { + **judge_result.track_data, + "judgeConfigKey": metric["judgeConfigKey"], + }, + metric["score"], + ) + elif task.evaluation_metric_key: get_client().track( task.evaluation_metric_key, - Context.from_dict(task.user_context), + ld_context, judge_result.track_data, judge_result.score, ) for task in result.judge_tasks or []: - t = asyncio.create_task(judge_worker(task)) - _background_tasks.add(t) - t.add_done_callback(_background_tasks.discard) + threading.Thread( + target=run_and_track_judge, + args=(task,), + daemon=False, + ).start() ``` +Node.js worker example. Enqueue each `judgeTask` from the request handler, then +run this function in the queue consumer: + ```typescript -import { config, getClient, globalRegistry, runJudge } from '@launchdarkly/ai-node'; +import { + config, + getClient, + globalRegistry, + runJudge, +} from '@launchdarkly/ai-node'; const { response, judgeTasks } = await config({ key: 'example-config-key', @@ -464,17 +504,39 @@ const { response, judgeTasks } = await config({ skipJudges: true, }).invoke(userInput, context); +// Request path: enqueue the serializable task; do not run it as an unobserved promise. for (const task of judgeTasks ?? []) { - void (async () => { - const judgeResult = await runJudge(task, handlers); - if (judgeResult && task.evaluationMetricKey) { - getClient().track(task.evaluationMetricKey, task.userContext, judgeResult.trackData, judgeResult.score); + await judgeQueue.enqueue(task); +} + +// Durable worker / queue consumer: +async function processJudgeTask(task) { + const judgeResult = await runJudge(task, handlers); + if (!judgeResult) return; + + if (judgeResult.metrics?.length) { + for (const metric of judgeResult.metrics) { + getClient().track( + metric.eventKey, + task.userContext, + { ...judgeResult.trackData, judgeConfigKey: metric.judgeConfigKey }, + metric.score, + ); } - })(); // or hand `task` to a worker thread or queue + } else if (task.evaluationMetricKey) { + getClient().track( + task.evaluationMetricKey, + task.userContext, + judgeResult.trackData, + judgeResult.score, + ); + } } ``` -`handlers` is the list of provider handlers the judge configs can use. Delete manual `track_judge_result` / `trackJudgeResult` calls. +`handlers` is the list of provider handlers the judge configs can use. Remove +the legacy `track_judge_result` / `trackJudgeResult` calls, but preserve their +behavior with the current client's `track()` call shown above. ## Agent graphs diff --git a/tests/test_legacy_sdk_migrate.py b/tests/test_legacy_sdk_migrate.py new file mode 100644 index 00000000..0537fe8b --- /dev/null +++ b/tests/test_legacy_sdk_migrate.py @@ -0,0 +1,31 @@ +import unittest +from pathlib import Path + + +ROOT = Path(__file__).resolve().parents[1] +SKILL = ROOT / "skills/agentcontrol/legacy-sdk-migrate/SKILL.md" +API_MAPPING = ( + ROOT + / "skills/agentcontrol/legacy-sdk-migrate/references/api-mapping.md" +) + + +class LegacySdkMigrateTests(unittest.TestCase): + def test_background_judge_examples_track_returned_metrics(self): + mapping = API_MAPPING.read_text(encoding="utf-8") + + self.assertIn("get_client().track", mapping) + self.assertIn("getClient().track", mapping) + self.assertNotIn("asyncio.create_task(judge_worker(task))", mapping) + self.assertNotIn("void runJudge(task, handlers)", mapping) + + def test_skill_preserves_tracking_for_background_judges(self): + skill = SKILL.read_text(encoding="utf-8") + + self.assertIn("Background judges are the exception", skill) + self.assertIn("Do not fire and forget", skill) + self.assertIn("returned score", skill) + + +if __name__ == "__main__": + unittest.main() From 335fb420e486bfd539c597130eafeec6a1fe890c Mon Sep 17 00:00:00 2001 From: Scarlett Attensil Date: Mon, 9 Nov 2026 11:32:19 -0800 Subject: [PATCH 4/6] fix(legacy-sdk-migrate): address review findings on [ld] extra, fallback re-raise, and judge track_data Co-authored-by: Cursor --- skills.json | 2 +- .../agentcontrol/legacy-sdk-migrate/SKILL.md | 16 +++--- .../references/api-mapping.md | 57 ++++++------------- tests/test_legacy_sdk_migrate.py | 2 + 4 files changed, 29 insertions(+), 48 deletions(-) diff --git a/skills.json b/skills.json index c8ebf210..53eb572a 100644 --- a/skills.json +++ b/skills.json @@ -508,7 +508,7 @@ "path": "skills/agentcontrol/legacy-sdk-migrate", "version": "0.1.0", "license": "Apache-2.0", - "compatibility": "Python 3.12+ or Node.js. No LaunchDarkly MCP tools needed; the AgentControl configs already exist." + "compatibility": "Python 3.12+ (`launchdarkly-ai-server` 1.0.0+ with `launchdarkly-server-sdk`) or Node.js (`@launchdarkly/ai-node` 1.0.0+). No LaunchDarkly MCP tools needed; the AgentControl configs already exist." }, { "name": "mcp-configure", diff --git a/skills/agentcontrol/legacy-sdk-migrate/SKILL.md b/skills/agentcontrol/legacy-sdk-migrate/SKILL.md index 0a04202e..b20ba147 100644 --- a/skills/agentcontrol/legacy-sdk-migrate/SKILL.md +++ b/skills/agentcontrol/legacy-sdk-migrate/SKILL.md @@ -2,7 +2,7 @@ name: legacy-sdk-migrate description: "Migrate an app from the legacy LaunchDarkly AI SDKs (Python `launchdarkly-server-sdk-ai` / `ldai`, Node.js `@launchdarkly/server-sdk-ai`) to the current handler-based AI SDKs (`launchdarkly-ai-server`, `@launchdarkly/ai-node` / `@launchdarkly/ai-server` plus handler packages). Use when code calls `LDAIClient`, `initAi`, `completion_config` / `completionConfig`, `agent_config` / `agentConfig`, `agent_configs`, `create_tracker` / `trackMetricsOf`, `create_judge`, `create_model`, or `agent_graph`, or when the user asks to upgrade between LaunchDarkly AI SDKs. For apps with hardcoded LLM prompts that aren't on LaunchDarkly yet, use `migrate` instead." license: Apache-2.0 -compatibility: Python 3.12+ or Node.js. No LaunchDarkly MCP tools needed; the AgentControl configs already exist. +compatibility: Python 3.12+ (`launchdarkly-ai-server` 1.0.0+ with `launchdarkly-server-sdk`) or Node.js (`@launchdarkly/ai-node` 1.0.0+). No LaunchDarkly MCP tools needed; the AgentControl configs already exist. metadata: author: launchdarkly version: "0.1.0" @@ -20,8 +20,8 @@ Source guide: [Migrate from the legacy AI SDKs](https://launchdarkly.com/docs/sd - The app already uses a legacy AI SDK, and its AgentControl configs (formerly AI Configs) already exist in LaunchDarkly. The config keys, variations, tools, and judges carry over unchanged; this is a code migration only. (If the app calls an LLM provider directly with hardcoded prompts and does not use LaunchDarkly yet, use the `migrate` skill instead.) - `LD_SDK_KEY` (server-side SDK key) and the provider API keys (`OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, ...) available as environment variables. The current SDKs read them from the environment. -- **Python:** the current packages require **Python 3.12 or later**, and the API is **async** (`await config(...).invoke(...)`). Check both before you start. -- **Node.js:** `@launchdarkly/ai-node` for Node; `@launchdarkly/ai-server` plus a pre-initialized client for edge or custom runtimes. +- **Python:** the current packages require **Python 3.12 or later** (`launchdarkly-ai-server` 1.0.0+), and the API is **async** (`await config(...).invoke(...)`). `launchdarkly-server-sdk` is required at runtime for standard initialization and `ldclient.Context` (installed via `"launchdarkly-ai-server[ld]"` / `"launchdarkly-ai-python[ld]"` or by keeping `launchdarkly-server-sdk`). Check Python 3.12+ before you start. +- **Node.js:** `@launchdarkly/ai-node` 1.0.0+ for Node (bundles `@launchdarkly/node-server-sdk`); `@launchdarkly/ai-server` plus a pre-initialized client for edge or custom runtimes. ## Core Principles @@ -90,14 +90,13 @@ Confirm the plan with the user: target per call site, packages to add and remove Work one call site at a time. Full code for each step is in [references/api-mapping.md](references/api-mapping.md). 1. **Swap packages.** Remove the legacy AI packages and install the current core package plus one handler package per provider and mode. - - Python: `pip install "launchdarkly-ai-python[ld]" launchdarkly-ai-openai-messages` (or `launchdarkly-server-sdk launchdarkly-ai-server ...`). Remove `launchdarkly-server-sdk-ai` and `launchdarkly-server-sdk-ai-*`. - - Node: `npm install @launchdarkly/ai-node @launchdarkly/ai-openai-messages`. Remove `@launchdarkly/server-sdk-ai` and `@launchdarkly/server-sdk-ai-*`. Edge runtimes install `@launchdarkly/ai-server` and pass their client to `initClient(client)`. - - Keep the base server SDK (`launchdarkly-server-sdk` / `@launchdarkly/node-server-sdk`) only if the app still evaluates ordinary feature flags with it. + - Python: `pip install "launchdarkly-ai-server[ld]" launchdarkly-ai-openai-messages` (or `"launchdarkly-ai-python[ld]"`). Remove `launchdarkly-server-sdk-ai` and `launchdarkly-server-sdk-ai-*`. Keep `launchdarkly-server-sdk` installed (either directly or via the `[ld]` extra) unless the app runs on a custom runtime and passes its own client to `init_client(client=...)`. + - Node: `npm install @launchdarkly/ai-node @launchdarkly/ai-openai-messages`. Remove `@launchdarkly/server-sdk-ai` and `@launchdarkly/server-sdk-ai-*`. `@launchdarkly/ai-node` bundles `@launchdarkly/node-server-sdk`, so keep a direct `@launchdarkly/node-server-sdk` dependency only if the app still imports it for ordinary feature flags. Edge runtimes install `@launchdarkly/ai-server` and pass their client to `initClient(client)`. 2. **Replace initialization.** Delete `LDAIClient(...)` / `initAi(...)`. The SDK initializes lazily from `LD_SDK_KEY` on the first AI call. Call `init_client()` / `initClient()` only to pass options or pre-warm, and call `shutdown()` on exit to flush events and spans. 3. **Rewrite each call.** Replace evaluate → build messages → call provider → track with one convenience function or `config(...).invoke(user_input, context, variables)`. Read the output from `result.response`. Python contexts are plain dicts (`{"kind": "user", "key": "user-123"}`). 4. **Move template variables** from the `completion_config` / `agent_config` argument to the `variables` keyword (convenience functions, Python), the `{ variables }` option (convenience functions, Node), or the third argument to `invoke()`. Note that the current SDKs inject the evaluation context as `{{ldContext.*}}` (replacing legacy `{{ldctx.*}}`) and require Mustache placeholders without spaces (`{{name}}`, not `{{ name }}`). 5. **Move tools.** Register implementations by name in `tool_handlers` / `toolHandlers` or a `Registry`. Keys must match the tool names in the config variation exactly, including case. Each tool function is called with a single `args` dict/object (not unpacked keyword arguments), so wrap multi-argument functions (`lambda args: fn(**args)`) or LangChain `@tool` objects (`lambda args: tool.invoke(args)`). Delete the hand-written tool loop. -6. **Preserve fallbacks.** Wrap `invoke()` and return the app's previous fallback response on error. Use `inspect_config()` / `inspectConfig()` (never raises; returns `enabled`, `config`, `meta`) for health checks or to branch before invoking. +6. **Preserve fallbacks.** Wrap `invoke()` and return the app's previous fallback response when the config is disabled or LaunchDarkly is unreachable. In the `except` / `catch` block, check `(await inspect_config(key, context))["enabled"]` / `(await inspectConfig(key, context)).enabled` and re-raise when `True` so mid-run provider or tool-loop errors aren't mistaken for a disabled config. Use `inspect_config()` / `inspectConfig()` (never raises; returns `enabled`, `config`, `meta`) for health checks or to branch before invoking. 7. **Replace tracking code.** Remove legacy `create_tracker`, generation/token/duration tracking, and `get_ai_metrics_from_response` / `getAIMetricsFromResponse` helpers; `invoke()` and `stream()` record those automatically. Preserve explicit app events: replace user feedback calls with `get_client().track()` / `getClient().track()` using `result.track_data` / `result.trackData` (see [references/api-mapping.md](references/api-mapping.md#user-feedback)). **Background judges are the exception:** `run_judge()` / `runJudge()` returns score data but deliberately does not call the LaunchDarkly client's `track()` method. Track every returned metric from the worker as shown in the reference. 8. **Centralize.** If two or more call sites share handlers or tools, create one `Registry` (or register on `global_registry` / `globalRegistry` at startup) and pass `registry=`. 9. **Remove dead imports and packages** once nothing references them. @@ -118,7 +117,8 @@ Work one call site at a time. Full code for each step is in [references/api-mapp | Python below 3.12 | Stop and tell the user. The current Python packages need 3.12+. | | Synchronous Python code path (Flask, Django views, scripts) | Run the async call with `asyncio.run(...)` at the boundary, or move the handler to an async framework. Ask which. | | App reads provider-native fields (`choices`, `tool_calls`, `stop_reason`) | Use `result.response` and `result.usage`; ask before changing downstream logic that needs the raw object. | -| Provider has no handler package (Bedrock, Gemini, Azure, internal gateway) | Use `litellm-*`, `langchain-*`, or `vercel-*`, or build a custom handler with `createHandler` / `create_handler`. In Python, `provides_for` must be a tuple `("ProviderName", "messages")`, and custom handlers must call `parse_template` / `parseTemplate` on raw prompt strings. | +| Provider has no handler package (Bedrock, Gemini, Azure, internal gateway) | Use `litellm-*`, `langchain-*`, or `vercel-*`, or build a custom handler with `createHandler` / `create_handler` (match the config's provider name case-sensitively, such as `"OpenAI"`, `"Anthropic"`, `"Bedrock"`, `"Gemini"`, and call `parse_template` / `parseTemplate` on raw prompt strings). | +| App uses `claude-agents` and relies on Claude Code built-ins or `~/.claude/settings.json` | In 1.0+, `create_claude_agents_handler()` / `createClaudeAgentsHandler()` disables built-in tools unless a `NativeTool` sentinel is attached to the variation, and loads no host settings (`setting_sources=[]`). Pass `setting_sources=["user"]` or set env vars if the app routes to Bedrock/Vertex/a gateway through `~/.claude/settings.json`. | | App calls `track_feedback` / `trackFeedback` for thumbs up or down | Replace `tracker.track_feedback` with `get_client().track("$ld:ai:feedback:user:positive" or ":negative", Context.from_dict(context), result.track_data, 1)` in Python or `getClient().track(event, context, result.trackData, 1)` in Node.js (note: `stream()` does not currently return `track_data`). | | Large app, gradual rollout | Migrate one call site at a time, verify it, and remove the legacy packages only after the last call site moves. | | Judges must not add latency | `skip_judges=True` / `skipJudges: true`, then put `judge_tasks` / `judgeTasks` on a durable worker or queue. Run `run_judge()` / `runJudge()` there and track the returned score(s) with their supplied track data. | diff --git a/skills/agentcontrol/legacy-sdk-migrate/references/api-mapping.md b/skills/agentcontrol/legacy-sdk-migrate/references/api-mapping.md index eec78470..2e086c23 100644 --- a/skills/agentcontrol/legacy-sdk-migrate/references/api-mapping.md +++ b/skills/agentcontrol/legacy-sdk-migrate/references/api-mapping.md @@ -40,13 +40,13 @@ Handler factory names follow `create___handler` (Python) and `cr | Language | Remove | Add | |----------|--------|-----| -| Python | `launchdarkly-server-sdk-ai`, `launchdarkly-server-sdk-ai-openai`, `launchdarkly-server-sdk-ai-langchain` (imports `ldai`, `ldai_openai`, `ldai_langchain`) | `launchdarkly-server-sdk` + `launchdarkly-ai-server`, or `"launchdarkly-ai-python[ld]"`; plus one `launchdarkly-ai--` per handler | +| Python | `launchdarkly-server-sdk-ai`, `launchdarkly-server-sdk-ai-openai`, `launchdarkly-server-sdk-ai-langchain` (imports `ldai`, `ldai_openai`, `ldai_langchain`) | `"launchdarkly-ai-server[ld]"` (or `launchdarkly-server-sdk` + `launchdarkly-ai-server`, or `"launchdarkly-ai-python[ld]"`); plus one `launchdarkly-ai--` per handler | | Node.js | `@launchdarkly/server-sdk-ai`, `@launchdarkly/server-sdk-ai-openai`, `@launchdarkly/server-sdk-ai-langchain`, `@launchdarkly/server-sdk-ai-vercel` | `@launchdarkly/ai-node` (bundles `@launchdarkly/node-server-sdk`), or `@launchdarkly/ai-server` for edge runtimes; plus one `@launchdarkly/ai--` per handler | ```bash # Python (3.12+) pip uninstall launchdarkly-server-sdk-ai launchdarkly-server-sdk-ai-openai -pip install launchdarkly-server-sdk launchdarkly-ai-server launchdarkly-ai-openai-messages +pip install "launchdarkly-ai-server[ld]" launchdarkly-ai-openai-messages # Node.js npm uninstall @launchdarkly/server-sdk-ai @launchdarkly/server-sdk-ai-openai @@ -213,42 +213,47 @@ console.log(result.response, result.usage); ## Fallbacks and inspection -There is no fallback argument. `invoke()` raises when the variation is disabled (`RuntimeError: Variation '' is disabled.` in Python, `Error: Variation is not enabled` in Node.js) or when LaunchDarkly cannot be reached. Port the legacy `if not config.enabled:` branch into an error handler: +There is no fallback argument. `invoke()` raises when the variation is disabled (`RuntimeError: Variation '' is disabled.` in Python, `Error: Variation is not enabled` in Node.js) or when LaunchDarkly cannot be reached. Because handler packages also raise `RuntimeError` / `Error` when a run fails after starting (for example, hitting the tool-loop step limit), check `inspect_config()` / `inspectConfig()` in the error handler and re-raise when `enabled` is true so real run failures are not swallowed by the disabled-config fallback: ```python +from launchdarkly_ai_server import inspect_config +from launchdarkly_ai_openai_messages import openai_messages + FALLBACK = "Sorry, the assistant is unavailable right now." try: result = await openai_messages("example-config-key", user_input, context) text = result.response except Exception: - logger.exception("AI config call failed; using fallback") + if (await inspect_config("example-config-key", context))["enabled"]: + raise + logger.warning("AI config disabled or unreachable; using fallback") text = FALLBACK ``` ```typescript +import { inspectConfig } from '@launchdarkly/ai-node'; +import { openaiMessages } from '@launchdarkly/ai-openai-messages'; + let text: string; try { text = (await openaiMessages('example-config-key', userInput, context)).response; } catch (err) { - logger.error('AI config call failed; using fallback', err); + if ((await inspectConfig('example-config-key', context)).enabled) throw err; + logger.warn('AI config disabled or unreachable; using fallback', err); text = FALLBACK; } ``` -To read state without raising and without calling a provider or emitting metrics, use `inspect_config()` / `inspectConfig()`. It returns `enabled`, `config` (the evaluated variation, or `None` / `null`), and `meta`: +To read state up front without raising and without calling a provider or emitting metrics, use `inspect_config()` / `inspectConfig()`. It returns `enabled`, `config` (the evaluated variation, or `None` / `null`), and `meta`: ```python -from launchdarkly_ai_server import inspect_config - state = await inspect_config("example-config-key", context) if not state["enabled"]: return FALLBACK ``` ```typescript -import { inspectConfig } from '@launchdarkly/ai-node'; - const state = await inspectConfig('example-config-key', context); if (!state.enabled) return FALLBACK; ``` @@ -456,25 +461,10 @@ handlers = resolve_handlers(global_registry, None) or [] def run_and_track_judge(task): judge_result = asyncio.run(run_judge(task, handlers)) - if judge_result is None: - return - - ld_context = Context.from_dict(task.user_context) - if judge_result.metrics: - for metric in judge_result.metrics: - get_client().track( - metric["eventKey"], - ld_context, - { - **judge_result.track_data, - "judgeConfigKey": metric["judgeConfigKey"], - }, - metric["score"], - ) - elif task.evaluation_metric_key: + if judge_result and task.evaluation_metric_key: get_client().track( task.evaluation_metric_key, - ld_context, + Context.from_dict(task.user_context), judge_result.track_data, judge_result.score, ) @@ -512,18 +502,7 @@ for (const task of judgeTasks ?? []) { // Durable worker / queue consumer: async function processJudgeTask(task) { const judgeResult = await runJudge(task, handlers); - if (!judgeResult) return; - - if (judgeResult.metrics?.length) { - for (const metric of judgeResult.metrics) { - getClient().track( - metric.eventKey, - task.userContext, - { ...judgeResult.trackData, judgeConfigKey: metric.judgeConfigKey }, - metric.score, - ); - } - } else if (task.evaluationMetricKey) { + if (judgeResult && task.evaluationMetricKey) { getClient().track( task.evaluationMetricKey, task.userContext, diff --git a/tests/test_legacy_sdk_migrate.py b/tests/test_legacy_sdk_migrate.py index 0537fe8b..fdb7e240 100644 --- a/tests/test_legacy_sdk_migrate.py +++ b/tests/test_legacy_sdk_migrate.py @@ -25,6 +25,8 @@ def test_skill_preserves_tracking_for_background_judges(self): self.assertIn("Background judges are the exception", skill) self.assertIn("Do not fire and forget", skill) self.assertIn("returned score", skill) + self.assertIn("launchdarkly-server-sdk", skill) + self.assertNotIn("must be a tuple", skill) if __name__ == "__main__": From 784614aea350a2470ee012777b092ffc43ca1b44 Mon Sep 17 00:00:00 2001 From: Scarlett Attensil Date: Mon, 9 Nov 2026 13:35:23 -0800 Subject: [PATCH 5/6] fix(legacy-sdk-migrate): cover parameter forwarding, track() contexts, custom graph walks, and content capture Co-authored-by: Cursor --- .../agentcontrol/legacy-sdk-migrate/SKILL.md | 11 +++++---- .../references/api-mapping.md | 24 +++++++++++++++++-- 2 files changed, 29 insertions(+), 6 deletions(-) diff --git a/skills/agentcontrol/legacy-sdk-migrate/SKILL.md b/skills/agentcontrol/legacy-sdk-migrate/SKILL.md index b20ba147..f377e0fb 100644 --- a/skills/agentcontrol/legacy-sdk-migrate/SKILL.md +++ b/skills/agentcontrol/legacy-sdk-migrate/SKILL.md @@ -92,12 +92,12 @@ Work one call site at a time. Full code for each step is in [references/api-mapp 1. **Swap packages.** Remove the legacy AI packages and install the current core package plus one handler package per provider and mode. - Python: `pip install "launchdarkly-ai-server[ld]" launchdarkly-ai-openai-messages` (or `"launchdarkly-ai-python[ld]"`). Remove `launchdarkly-server-sdk-ai` and `launchdarkly-server-sdk-ai-*`. Keep `launchdarkly-server-sdk` installed (either directly or via the `[ld]` extra) unless the app runs on a custom runtime and passes its own client to `init_client(client=...)`. - Node: `npm install @launchdarkly/ai-node @launchdarkly/ai-openai-messages`. Remove `@launchdarkly/server-sdk-ai` and `@launchdarkly/server-sdk-ai-*`. `@launchdarkly/ai-node` bundles `@launchdarkly/node-server-sdk`, so keep a direct `@launchdarkly/node-server-sdk` dependency only if the app still imports it for ordinary feature flags. Edge runtimes install `@launchdarkly/ai-server` and pass their client to `initClient(client)`. -2. **Replace initialization.** Delete `LDAIClient(...)` / `initAi(...)`. The SDK initializes lazily from `LD_SDK_KEY` on the first AI call. Call `init_client()` / `initClient()` only to pass options or pre-warm, and call `shutdown()` on exit to flush events and spans. -3. **Rewrite each call.** Replace evaluate → build messages → call provider → track with one convenience function or `config(...).invoke(user_input, context, variables)`. Read the output from `result.response`. Python contexts are plain dicts (`{"kind": "user", "key": "user-123"}`). +2. **Replace initialization and check content capture.** Delete `LDAIClient(...)` / `initAi(...)`. The SDK initializes lazily from `LD_SDK_KEY` on the first AI call. Call `init_client()` / `initClient()` only to pass options or pre-warm, and call `shutdown()` on exit to flush events and spans. In 1.0+, OpenTelemetry spans capture prompts, model output, tool payloads, and judge reasoning **by default** (`capture_content=True`). Ask if the app handles sensitive data and needs to opt out via `configure(options=SDKOptions(capture_content=False))` (Python) or `configure({ options: { captureContent: false } })` (Node.js). +3. **Rewrite each call and check parameter forwarding.** Replace evaluate → build messages → call provider → track with one convenience function or `config(...).invoke(user_input, context, variables)`. Read the output from `result.response`. Python contexts passed to `invoke()` / `stream()` / `graph()` are plain dicts (`{"kind": "user", "key": "user-123"}`). Handler packages forward a fixed per-handler allowlist of `model.parameters` (mapping `max_tokens` to `max_output_tokens` on OpenAI) and drop unknown keys; move app-scoped knobs to `model.custom`. If writing a custom handler (`create_handler` / `createHandler`), forward `model.parameters` through an explicit allowlist (never a raw `**params` spread or denylist alone, and never forward `tools`). 4. **Move template variables** from the `completion_config` / `agent_config` argument to the `variables` keyword (convenience functions, Python), the `{ variables }` option (convenience functions, Node), or the third argument to `invoke()`. Note that the current SDKs inject the evaluation context as `{{ldContext.*}}` (replacing legacy `{{ldctx.*}}`) and require Mustache placeholders without spaces (`{{name}}`, not `{{ name }}`). 5. **Move tools.** Register implementations by name in `tool_handlers` / `toolHandlers` or a `Registry`. Keys must match the tool names in the config variation exactly, including case. Each tool function is called with a single `args` dict/object (not unpacked keyword arguments), so wrap multi-argument functions (`lambda args: fn(**args)`) or LangChain `@tool` objects (`lambda args: tool.invoke(args)`). Delete the hand-written tool loop. 6. **Preserve fallbacks.** Wrap `invoke()` and return the app's previous fallback response when the config is disabled or LaunchDarkly is unreachable. In the `except` / `catch` block, check `(await inspect_config(key, context))["enabled"]` / `(await inspectConfig(key, context)).enabled` and re-raise when `True` so mid-run provider or tool-loop errors aren't mistaken for a disabled config. Use `inspect_config()` / `inspectConfig()` (never raises; returns `enabled`, `config`, `meta`) for health checks or to branch before invoking. -7. **Replace tracking code.** Remove legacy `create_tracker`, generation/token/duration tracking, and `get_ai_metrics_from_response` / `getAIMetricsFromResponse` helpers; `invoke()` and `stream()` record those automatically. Preserve explicit app events: replace user feedback calls with `get_client().track()` / `getClient().track()` using `result.track_data` / `result.trackData` (see [references/api-mapping.md](references/api-mapping.md#user-feedback)). **Background judges are the exception:** `run_judge()` / `runJudge()` returns score data but deliberately does not call the LaunchDarkly client's `track()` method. Track every returned metric from the worker as shown in the reference. +7. **Replace tracking code and convert Python `track()` contexts.** Remove legacy `create_tracker`, generation/token/duration tracking, and `get_ai_metrics_from_response` / `getAIMetricsFromResponse` helpers; `invoke()` and `stream()` record those automatically. Preserve explicit app events: replace user feedback calls with `get_client().track()` / `getClient().track()` using `result.track_data` / `result.trackData` (see [references/api-mapping.md](references/api-mapping.md#user-feedback)). **In Python, `get_client().track()` requires an `ldclient.Context` (`Context.from_dict(context)`), not the plain dict passed to `invoke()`**—passing a raw dict to `get_client().track()` silently drops the event. **Background judges are the exception:** `run_judge()` / `runJudge()` returns score data but deliberately does not call the LaunchDarkly client's `track()` method. Track every returned metric from the worker as shown in the reference. 8. **Centralize.** If two or more call sites share handlers or tools, create one `Registry` (or register on `global_registry` / `globalRegistry` at startup) and pass `registry=`. 9. **Remove dead imports and packages** once nothing references them. @@ -117,7 +117,10 @@ Work one call site at a time. Full code for each step is in [references/api-mapp | Python below 3.12 | Stop and tell the user. The current Python packages need 3.12+. | | Synchronous Python code path (Flask, Django views, scripts) | Run the async call with `asyncio.run(...)` at the boundary, or move the handler to an async framework. Ask which. | | App reads provider-native fields (`choices`, `tool_calls`, `stop_reason`) | Use `result.response` and `result.usage`; ask before changing downstream logic that needs the raw object. | -| Provider has no handler package (Bedrock, Gemini, Azure, internal gateway) | Use `litellm-*`, `langchain-*`, or `vercel-*`, or build a custom handler with `createHandler` / `create_handler` (match the config's provider name case-sensitively, such as `"OpenAI"`, `"Anthropic"`, `"Bedrock"`, `"Gemini"`, and call `parse_template` / `parseTemplate` on raw prompt strings). | +| Provider has no handler package (Bedrock, Gemini, Azure, internal gateway) | Use `litellm-*`, `langchain-*`, or `vercel-*`, or build a custom handler with `createHandler` / `create_handler` (match the config's provider name case-sensitively, such as `"OpenAI"`, `"Anthropic"`, `"Bedrock"`, `"Gemini"`, call `parse_template` / `parseTemplate` on raw prompt strings, and forward `model.parameters` through an explicit allowlist rather than spreading `**params`). | +| App stores custom app/tool knobs in `model.parameters` | Handler packages forward only an allowlist of model parameters and drop unknown keys. Move app-scoped knobs to `model.custom` and read them via `inspect_config()` / `inspectConfig()` or inside a custom handler (`ld_config["model"].get("custom")`). | +| App walks an agent graph by hand (`agent_graph().traverse()` / `get_edges()`) | Prefer `graph().invoke()` or a native runner (`to_lang_graph`, `to_openai_agents`, `to_claude_agents`). If keeping a custom walk with `resolve_graph()` / `resolveGraph()`, note that it returns topology only (`enabled`, `root`, `node.config`, `node.edges`, `edge.target_key`) and emits **no** telemetry unless you execute hops via `graph.run_node()` / `graph.route()` (passing `handlers=` to `resolve_graph()`) or emit the 1.0 `$ld:ai:graph:*` events yourself. Warn the user that legacy graph event keys changed (`path` → `node`, `handoff` → `handoff_success`/`handoff_failure`, `invocation` → `invocation_success`/`invocation_failure`, `total` → `total_tokens`, `tool` → `tool_call`). | +| App handles sensitive/regulated prompts or outputs (PII, PHI) | Content capture on OpenTelemetry spans is **on by default** in 1.0+ (`capture_content=True`). Disable it process-wide with `configure(options=SDKOptions(capture_content=False))` (Python) or `configure({ options: { captureContent: false } })` (Node.js), or pass `capture_content=False` / `{ captureContent: false }` to handler factories. | | App uses `claude-agents` and relies on Claude Code built-ins or `~/.claude/settings.json` | In 1.0+, `create_claude_agents_handler()` / `createClaudeAgentsHandler()` disables built-in tools unless a `NativeTool` sentinel is attached to the variation, and loads no host settings (`setting_sources=[]`). Pass `setting_sources=["user"]` or set env vars if the app routes to Bedrock/Vertex/a gateway through `~/.claude/settings.json`. | | App calls `track_feedback` / `trackFeedback` for thumbs up or down | Replace `tracker.track_feedback` with `get_client().track("$ld:ai:feedback:user:positive" or ":negative", Context.from_dict(context), result.track_data, 1)` in Python or `getClient().track(event, context, result.trackData, 1)` in Node.js (note: `stream()` does not currently return `track_data`). | | Large app, gradual rollout | Migrate one call site at a time, verify it, and remove the legacy packages only after the last call site moves. | diff --git a/skills/agentcontrol/legacy-sdk-migrate/references/api-mapping.md b/skills/agentcontrol/legacy-sdk-migrate/references/api-mapping.md index 2e086c23..84d0d518 100644 --- a/skills/agentcontrol/legacy-sdk-migrate/references/api-mapping.md +++ b/skills/agentcontrol/legacy-sdk-migrate/references/api-mapping.md @@ -29,8 +29,10 @@ Handler factory names follow `create___handler` (Python) and `cr | Multi-agent | `agent_configs()` / `agentConfigs()` | `graph()` or a native framework runner | | Managed objects | `create_model()`, `create_agent()` / `createModel().run()`, `createAgent()` | Convenience function or `config().invoke()` | | Metrics | `create_tracker()`, `track_metrics_of`, `track_*` / `createTracker()`, `trackMetricsOf`, `track*` | Generation metrics are automatic on `invoke()` / `stream()`; a background `run_judge()` / `runJudge()` returns judge metrics that the worker must track | -| Feedback | `tracker.track_feedback()` / `tracker.trackFeedback()` | `get_client().track()` / `getClient().track()` with `result.track_data` / `result.trackData` | -| Fallback | Default config object with `enabled=False` | `try`/`except` around `invoke()`; `inspect_config()` / `inspectConfig()` to read state | +| Feedback | `tracker.track_feedback()` / `tracker.trackFeedback()` | `get_client().track()` / `getClient().track()` with `result.track_data` / `result.trackData` (in Python, convert the context dict with `ldclient.Context.from_dict(context)`) | +| Model parameters | Passed manually to provider client (`model=config.model.name`, `**config.model.parameters`) | Handler packages forward a per-handler allowlist of `model.parameters` (mapping `max_tokens` → `max_output_tokens` on OpenAI) and drop unknown keys; move app-scoped knobs to `model.custom`, and use an explicit allowlist in custom handlers | +| Content capture | N/A (app owned any tracing) | Prompts, outputs, tool payloads, and judge reasoning are captured on OpenTelemetry spans by default in 1.0+ (`capture_content=True`); opt out with `configure(options=SDKOptions(capture_content=False))` / `configure({ options: { captureContent: false } })` | +| Fallback | Default config object with `enabled=False` | `try`/`except` around `invoke()` (checking `inspect_config()` / `inspectConfig()` before falling back); `inspect_config()` / `inspectConfig()` to read state | | Template variables | Fourth argument to `completion_config` / `completionConfig` | `variables=` (convenience, Python), `{ variables }` (convenience, Node), or third argument to `invoke()` (`{{name}}` without spaces; `{{ldContext.*}}` replaces `{{ldctx.*}}`) | | Judges | `create_judge()`, `judge_config()` / `createJudge()`, `judgeConfig()` | Judges attached to the variation; `skip_judges` / `skipJudges` plus a durable worker that runs and tracks `run_judge()` / `runJudge()` results | | Graphs | `agent_graph()`, `create_agent_graph()`, traversal helpers | `graph()`, `resolve_graph()` / `resolveGraph()` | @@ -98,6 +100,20 @@ await shutdown(); If the app also evaluates ordinary feature flags, keep its existing server SDK client for those. To share one client, pass it to `init_client(client=ld_client)` / `initClient(ldClient)`. +In 1.0+, OpenTelemetry spans capture prompts, model output, tool arguments/results, and judge reasoning by default (`capture_content=True`). To opt out process-wide, call `configure()` before the first AI call: + +```python +from launchdarkly_ai_server import SDKOptions, configure + +configure(options=SDKOptions(capture_content=False)) +``` + +```typescript +import { configure } from '@launchdarkly/ai-node'; + +configure({ options: { captureContent: false } }); +``` + ## Completion call: convenience function Use when the config serves one provider in one mode. @@ -546,3 +562,7 @@ console.log(result.response); ``` To keep running on a framework the app already uses, call `resolve_graph()` / `resolveGraph()` and pass the result to that framework's runner: `to_lang_graph` / `toLangGraph`, `to_openai_agents` / `toOpenAIAgents`, or `to_claude_agents` / `toClaudeAgents`. Check the handler package's exports and README for the exact call. + +If the app walks the resolved topology by hand instead of using `graph().invoke()` or a native runner: +- Node and edge accessors are attributes now: `graph.enabled`, `graph.root`, `node.key`, `node.config` (a plain dict), `node.edges`, and `edge.target_key` (replacing `is_enabled()`, `root()`, `get_key()`, `get_config()`, `get_edges()`, and `edge.target_config`). +- `resolve_graph()` / `resolveGraph()` returns topology only and emits **no** telemetry on its own. Pass `handlers=` to `resolve_graph()` and execute nodes with `await graph.run_node(node, input, opts)` / `await graph.route(node, input, opts)`, or emit the 1.0 `$ld:ai:graph:*` events yourself (`$ld:ai:graph:node`, `$ld:ai:graph:handoff_success`, `$ld:ai:graph:handoff_failure`, `$ld:ai:graph:invocation_success`, `$ld:ai:graph:invocation_failure`, `$ld:ai:graph:duration:total`, `$ld:ai:graph:total_tokens`). Note that legacy event keys (`$ld:ai:graph:path`, `$ld:ai:graph:handoff`, `$ld:ai:graph:invocation`, `$ld:ai:graph:total`, `$ld:ai:tool`) changed in 1.0, so existing LaunchDarkly metrics built on the old keys must be updated. From f4f701546be9b305f9a72f81e155939d1e6c4fb6 Mon Sep 17 00:00:00 2001 From: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Date: Fri, 13 Nov 2026 14:31:24 +0000 Subject: [PATCH 6/6] fix(legacy-sdk-migrate): address jeffdupont review (graph keys, Node names, judge handlers, tool checks) Co-Authored-By: Traci Siebel --- skills/agentcontrol/legacy-sdk-migrate/SKILL.md | 10 +++++----- .../legacy-sdk-migrate/references/api-mapping.md | 12 ++++++++---- 2 files changed, 13 insertions(+), 9 deletions(-) diff --git a/skills/agentcontrol/legacy-sdk-migrate/SKILL.md b/skills/agentcontrol/legacy-sdk-migrate/SKILL.md index f377e0fb..d8810c36 100644 --- a/skills/agentcontrol/legacy-sdk-migrate/SKILL.md +++ b/skills/agentcontrol/legacy-sdk-migrate/SKILL.md @@ -40,8 +40,8 @@ Find every legacy usage. Search for these patterns and record file and line for | Language | Imports and packages | Calls | |----------|----------------------|-------| -| Python | `launchdarkly-server-sdk-ai`, `launchdarkly-server-sdk-ai-openai`, `launchdarkly-server-sdk-ai-langchain`; `from ldai`, `from ldai_openai`, `from ldai_langchain` | `LDAIClient(`, `completion_config(`, `agent_config(`, `agent_configs(`, `judge_config(`, `create_judge(`, `create_model(`, `create_agent(`, `agent_graph(`, `create_agent_graph(`, `create_tracker(`, `track_metrics_of`, `track_duration`, `track_tokens`, `track_success`, `track_error`, `track_feedback`, `track_tool_call`, `get_ai_metrics_from_response` | -| Node.js | `@launchdarkly/server-sdk-ai`, `@launchdarkly/server-sdk-ai-openai`, `@launchdarkly/server-sdk-ai-langchain`, `@launchdarkly/server-sdk-ai-vercel` | `initAi(`, `completionConfig(`, `agentConfig(`, `agentConfigs(`, `judgeConfig(`, `createJudge(`, `createModel(`, `createAgent(`, `agentGraph(`, `createTracker(`, `.tracker`, `trackMetricsOf`, `trackStreamMetricsOf`, `trackDuration`, `trackTokens`, `trackSuccess`, `trackError`, `trackFeedback`, `getAIMetricsFromResponse`, `getAIMetricsFromStream`, `createLangChainModel` | +| Python | `launchdarkly-server-sdk-ai`, `launchdarkly-server-sdk-ai-openai`, `launchdarkly-server-sdk-ai-langchain`; `from ldai`, `from ldai_openai`, `from ldai_langchain` | `LDAIClient(`, `completion_config(`, `agent_config(`, `agent_configs(`, `judge_config(`, `create_judge(`, `create_model(`, `create_agent(`, `agent_graph(`, `create_agent_graph(`, `create_tracker(`, `track_metrics_of`, `track_duration`, `track_tokens`, `track_success`, `track_error`, `track_feedback`, `track_tool_call`, `track_judge_result`, `FeedbackKind`, `AICompletionConfigDefault`, `AIAgentConfigDefault`, `get_ai_metrics_from_response` | +| Node.js | `@launchdarkly/server-sdk-ai`, `@launchdarkly/server-sdk-ai-openai`, `@launchdarkly/server-sdk-ai-langchain`, `@launchdarkly/server-sdk-ai-vercel` | `initAi(`, `completionConfig(`, `agentConfig(`, `agentConfigs(`, `judgeConfig(`, `createJudge(`, `createModel(`, `createAgent(`, `agentGraph(`, `createTracker(`, `.tracker`, `trackMetricsOf`, `trackStreamMetricsOf`, `trackDuration`, `trackTokens`, `trackSuccess`, `trackError`, `trackFeedback`, `trackToolCall`, `trackToolCalls`, `trackJudgeResult`, `LDFeedbackKind`, `getAIMetricsFromResponse`, `getAIMetricsFromStream`, `createLangChainModel` | For each call site, also record: @@ -97,7 +97,7 @@ Work one call site at a time. Full code for each step is in [references/api-mapp 4. **Move template variables** from the `completion_config` / `agent_config` argument to the `variables` keyword (convenience functions, Python), the `{ variables }` option (convenience functions, Node), or the third argument to `invoke()`. Note that the current SDKs inject the evaluation context as `{{ldContext.*}}` (replacing legacy `{{ldctx.*}}`) and require Mustache placeholders without spaces (`{{name}}`, not `{{ name }}`). 5. **Move tools.** Register implementations by name in `tool_handlers` / `toolHandlers` or a `Registry`. Keys must match the tool names in the config variation exactly, including case. Each tool function is called with a single `args` dict/object (not unpacked keyword arguments), so wrap multi-argument functions (`lambda args: fn(**args)`) or LangChain `@tool` objects (`lambda args: tool.invoke(args)`). Delete the hand-written tool loop. 6. **Preserve fallbacks.** Wrap `invoke()` and return the app's previous fallback response when the config is disabled or LaunchDarkly is unreachable. In the `except` / `catch` block, check `(await inspect_config(key, context))["enabled"]` / `(await inspectConfig(key, context)).enabled` and re-raise when `True` so mid-run provider or tool-loop errors aren't mistaken for a disabled config. Use `inspect_config()` / `inspectConfig()` (never raises; returns `enabled`, `config`, `meta`) for health checks or to branch before invoking. -7. **Replace tracking code and convert Python `track()` contexts.** Remove legacy `create_tracker`, generation/token/duration tracking, and `get_ai_metrics_from_response` / `getAIMetricsFromResponse` helpers; `invoke()` and `stream()` record those automatically. Preserve explicit app events: replace user feedback calls with `get_client().track()` / `getClient().track()` using `result.track_data` / `result.trackData` (see [references/api-mapping.md](references/api-mapping.md#user-feedback)). **In Python, `get_client().track()` requires an `ldclient.Context` (`Context.from_dict(context)`), not the plain dict passed to `invoke()`**—passing a raw dict to `get_client().track()` silently drops the event. **Background judges are the exception:** `run_judge()` / `runJudge()` returns score data but deliberately does not call the LaunchDarkly client's `track()` method. Track every returned metric from the worker as shown in the reference. +7. **Replace tracking code and convert Python `track()` contexts.** Remove legacy `create_tracker`, generation/token/duration tracking, and `get_ai_metrics_from_response` / `getAIMetricsFromResponse` helpers; `invoke()` and `stream()` record those automatically. Preserve explicit app events: replace user feedback calls with `get_client().track()` / `getClient().track()` using `result.track_data` / `result.trackData` (see [references/api-mapping.md](references/api-mapping.md#user-feedback)). **In Python, `get_client().track()` requires an `ldclient.Context` (`Context.from_dict(context)`), not the plain dict passed to `invoke()`**—passing a raw dict raises `AttributeError: 'dict' object has no attribute 'valid'`. **In Node.js, pass a plain object literal as the context;** the `LDContext` type from `@launchdarkly/node-server-sdk` doesn't type-check as the context argument to `config().invoke()`. **Background judges are the exception:** `run_judge()` / `runJudge()` returns score data but deliberately does not call the LaunchDarkly client's `track()` method. Track every returned metric from the worker as shown in the reference. 8. **Centralize.** If two or more call sites share handlers or tools, create one `Registry` (or register on `global_registry` / `globalRegistry` at startup) and pass `registry=`. 9. **Remove dead imports and packages** once nothing references them. @@ -119,9 +119,9 @@ Work one call site at a time. Full code for each step is in [references/api-mapp | App reads provider-native fields (`choices`, `tool_calls`, `stop_reason`) | Use `result.response` and `result.usage`; ask before changing downstream logic that needs the raw object. | | Provider has no handler package (Bedrock, Gemini, Azure, internal gateway) | Use `litellm-*`, `langchain-*`, or `vercel-*`, or build a custom handler with `createHandler` / `create_handler` (match the config's provider name case-sensitively, such as `"OpenAI"`, `"Anthropic"`, `"Bedrock"`, `"Gemini"`, call `parse_template` / `parseTemplate` on raw prompt strings, and forward `model.parameters` through an explicit allowlist rather than spreading `**params`). | | App stores custom app/tool knobs in `model.parameters` | Handler packages forward only an allowlist of model parameters and drop unknown keys. Move app-scoped knobs to `model.custom` and read them via `inspect_config()` / `inspectConfig()` or inside a custom handler (`ld_config["model"].get("custom")`). | -| App walks an agent graph by hand (`agent_graph().traverse()` / `get_edges()`) | Prefer `graph().invoke()` or a native runner (`to_lang_graph`, `to_openai_agents`, `to_claude_agents`). If keeping a custom walk with `resolve_graph()` / `resolveGraph()`, note that it returns topology only (`enabled`, `root`, `node.config`, `node.edges`, `edge.target_key`) and emits **no** telemetry unless you execute hops via `graph.run_node()` / `graph.route()` (passing `handlers=` to `resolve_graph()`) or emit the 1.0 `$ld:ai:graph:*` events yourself. Warn the user that legacy graph event keys changed (`path` → `node`, `handoff` → `handoff_success`/`handoff_failure`, `invocation` → `invocation_success`/`invocation_failure`, `total` → `total_tokens`, `tool` → `tool_call`). | +| App walks an agent graph by hand (`agent_graph().traverse()` / `get_edges()`) | Prefer `graph().invoke()` or a native runner (`to_lang_graph`, `to_openai_agents`, `to_claude_agents`). If keeping a custom walk with `resolve_graph()` / `resolveGraph()`, note that it returns topology only (`enabled`, `root`, `node.config`, `node.edges`, `edge.target_key` / `edge.targetKey`) and emits **no** telemetry unless you execute hops via `graph.run_node()` / `graph.runNode()` or `graph.route()` (passing handlers to `resolve_graph()` / `resolveGraph()`) or emit the `$ld:ai:graph:*` events yourself. Graph event keys are mostly unchanged; only `$ld:ai:graph:path` became `$ld:ai:graph:node`, and `$ld:ai:graph:redirect` is gone. Tell the user to update only metrics built on those two keys. | | App handles sensitive/regulated prompts or outputs (PII, PHI) | Content capture on OpenTelemetry spans is **on by default** in 1.0+ (`capture_content=True`). Disable it process-wide with `configure(options=SDKOptions(capture_content=False))` (Python) or `configure({ options: { captureContent: false } })` (Node.js), or pass `capture_content=False` / `{ captureContent: false }` to handler factories. | -| App uses `claude-agents` and relies on Claude Code built-ins or `~/.claude/settings.json` | In 1.0+, `create_claude_agents_handler()` / `createClaudeAgentsHandler()` disables built-in tools unless a `NativeTool` sentinel is attached to the variation, and loads no host settings (`setting_sources=[]`). Pass `setting_sources=["user"]` or set env vars if the app routes to Bedrock/Vertex/a gateway through `~/.claude/settings.json`. | +| App uses `claude-agents` and relies on Claude Code built-ins or `~/.claude/settings.json` | In 1.0+, `create_claude_agents_handler()` / `createClaudeAgentsHandler()` disables built-in tools unless a `NativeTool` sentinel is attached to the variation, and loads no host settings (`setting_sources=[]` / `settingSources: []`). Pass `setting_sources=["user"]` / `settingSources: ['user']` or set env vars if the app routes to Bedrock/Vertex/a gateway through `~/.claude/settings.json`. | | App calls `track_feedback` / `trackFeedback` for thumbs up or down | Replace `tracker.track_feedback` with `get_client().track("$ld:ai:feedback:user:positive" or ":negative", Context.from_dict(context), result.track_data, 1)` in Python or `getClient().track(event, context, result.trackData, 1)` in Node.js (note: `stream()` does not currently return `track_data`). | | Large app, gradual rollout | Migrate one call site at a time, verify it, and remove the legacy packages only after the last call site moves. | | Judges must not add latency | `skip_judges=True` / `skipJudges: true`, then put `judge_tasks` / `judgeTasks` on a durable worker or queue. Run `run_judge()` / `runJudge()` there and track the returned score(s) with their supplied track data. | diff --git a/skills/agentcontrol/legacy-sdk-migrate/references/api-mapping.md b/skills/agentcontrol/legacy-sdk-migrate/references/api-mapping.md index 84d0d518..85d76018 100644 --- a/skills/agentcontrol/legacy-sdk-migrate/references/api-mapping.md +++ b/skills/agentcontrol/legacy-sdk-migrate/references/api-mapping.md @@ -335,7 +335,7 @@ const result = await config({ ); ``` -Delete the hand-written loop, the provider tool-schema construction, and the manual `track_tool_call` / `trackToolCall` calls. +Before deleting the loop, confirm that every tool it defined in code exists on each variation in LaunchDarkly. A tool missing from the variation is silently unavailable to the migrated agent; nothing fails. Do the same for a step cap or default model the old loop set: move it to the variation (or a custom handler) or tell the user. Then delete the hand-written loop, the provider tool-schema construction, and the manual `track_tool_call` / `trackToolCall` calls. ## Registries @@ -445,7 +445,7 @@ getClient().track(event, context, result.trackData, 1); ## Judges -Legacy apps created a judge and called it after each response. Now judges are attached to the config variation in LaunchDarkly and run inline by default on every `invoke()`. Delete `create_judge` / `createJudge` call sites once the judges are attached to the variation (ask the user to attach them if they aren't). +Legacy apps created a judge and called it after each response. Now judges are attached to the config variation in LaunchDarkly and run inline by default on every `invoke()`. If the legacy app judged only some calls on a config, it will now judge all of them; pass `skip_judges=True` / `skipJudges: true` on the calls that shouldn't be judged. Delete `create_judge` / `createJudge` call sites once the judges are attached to the variation (ask the user to attach them if they aren't). To keep latency low, skip inline judges and put the returned tasks on a durable worker or queue. `run_judge()` / `runJudge()` executes the judge call but does @@ -503,6 +503,10 @@ import { globalRegistry, runJudge, } from '@launchdarkly/ai-node'; +import { createOpenAIMessagesHandler } from '@launchdarkly/ai-openai-messages'; + +// The handlers your judge configs' providers need. +const handlers = [createOpenAIMessagesHandler()]; const { response, judgeTasks } = await config({ key: 'example-config-key', @@ -564,5 +568,5 @@ console.log(result.response); To keep running on a framework the app already uses, call `resolve_graph()` / `resolveGraph()` and pass the result to that framework's runner: `to_lang_graph` / `toLangGraph`, `to_openai_agents` / `toOpenAIAgents`, or `to_claude_agents` / `toClaudeAgents`. Check the handler package's exports and README for the exact call. If the app walks the resolved topology by hand instead of using `graph().invoke()` or a native runner: -- Node and edge accessors are attributes now: `graph.enabled`, `graph.root`, `node.key`, `node.config` (a plain dict), `node.edges`, and `edge.target_key` (replacing `is_enabled()`, `root()`, `get_key()`, `get_config()`, `get_edges()`, and `edge.target_config`). -- `resolve_graph()` / `resolveGraph()` returns topology only and emits **no** telemetry on its own. Pass `handlers=` to `resolve_graph()` and execute nodes with `await graph.run_node(node, input, opts)` / `await graph.route(node, input, opts)`, or emit the 1.0 `$ld:ai:graph:*` events yourself (`$ld:ai:graph:node`, `$ld:ai:graph:handoff_success`, `$ld:ai:graph:handoff_failure`, `$ld:ai:graph:invocation_success`, `$ld:ai:graph:invocation_failure`, `$ld:ai:graph:duration:total`, `$ld:ai:graph:total_tokens`). Note that legacy event keys (`$ld:ai:graph:path`, `$ld:ai:graph:handoff`, `$ld:ai:graph:invocation`, `$ld:ai:graph:total`, `$ld:ai:tool`) changed in 1.0, so existing LaunchDarkly metrics built on the old keys must be updated. +- Node and edge accessors are attributes now: `graph.enabled`, `graph.root`, `node.key`, `node.config` (a plain dict), `node.edges`, and `edge.target_key` / `edge.targetKey` (replacing `is_enabled()`, `root()`, `get_key()`, `get_config()`, `get_edges()`, and `edge.target_config`). +- `resolve_graph()` / `resolveGraph()` returns topology only and emits **no** telemetry on its own. Pass `handlers=` to `resolve_graph()` and execute nodes with `await graph.run_node(node, input, opts)` / `await graph.runNode(node, input, opts)` or `graph.route(node, input, opts)`, or emit the `$ld:ai:graph:*` events yourself (`$ld:ai:graph:node`, `$ld:ai:graph:handoff_success`, `$ld:ai:graph:handoff_failure`, `$ld:ai:graph:invocation_success`, `$ld:ai:graph:invocation_failure`, `$ld:ai:graph:duration:total`, `$ld:ai:graph:total_tokens`, plus `$ld:ai:tool_call`). These match the legacy SDKs except that `$ld:ai:graph:path` became `$ld:ai:graph:node` and `$ld:ai:graph:redirect` is gone, so only LaunchDarkly metrics built on those two keys need updating.