[RamazTs]

🎯 What problem does this PR solve?

• @qvac/tts-ggml 0.10+ (the SDK pins ^0.15.0) ships engine controls and runtime statistics that the inference schema and the tts-ggml plugin never accepted or forwarded, so SDK users could not reach them while the addon's own API could:
  • Chatterbox: T3 sampling (temperature, topK, topP, repeatPenalty), the multilingual prosody controls (exaggeration, cfgWeight, minP), length / streaming-cost controls (nPredict, maxSentenceChars, crossfadeMs, batchCfmSteps, streamLeftContextTokens) and pre-baked voice profiles (voiceDir).
  • Supertonic: native chunk streaming (streamChunkTokens, streamFirstChunkTokens, streamChunkTolerancePct, streamMinChunkTokens), external voices (voiceJsonPath) and the deployment controls prewarmText, noiseNpyPath, vulkanDevice. The SDK only had vulkanCacheDir.
  • CosyVoice3: instruct was load-time only although the addon takes it per call; flowCutPrompt and vulkanDevice were absent.
  • Audio8: voice cloning was load-time only although the addon takes referenceAudio / referenceText per call.
  • Statistics: the SDK forwarded an allowlist that dropped firstAudioMs, the Chatterbox t3Ms / s3genMs / t3Tokens, the CosyVoice3 stage timings and work counters, the Audio8 stage timings and stageTotalMs.

📝 How does it solve it?

• packages/inference/src/schemas/text-to-speech.ts: every row above becomes a described schema key on the engine arm the addon's option validator (lib/optionValidation.ts, assertEngineScopedOptions) scopes it to, with the addon's ranges (int32 knobs, topP in (0, 1], minP in [0, 1], vulkanDevice >= -1). Supertonic moves to the shape + load-refinement pattern Parler and CosyVoice3 use, so its load schema can reject native chunk streaming together with either LavaSR stage, as the addon does (assertChunkStreamingSupport). Supertonic's voice JSON and noise tensor are single files, so they are voiceJsonSrc / noiseNpySrc model sources resolved like the LavaSR GGUFs.

• Per-request voice controls (instruct, referenceAudioSrc, referenceText) are added to both the textToSpeech and textToSpeechStream request schemas with the engine-independent rules (a request-level recording needs its transcript; instruct conflicts with emotion / a non-moderate pace). The new ops/voice-options.ts does the engine check once the model is known (CosyVoice3 for instruct, Audio8 for the reference pair), resolves referenceAudioSrc through the same source resolution as the load-time field, and both ops forward the result on run(), runStream() and runStreaming(). The engine check runs before the op queues for the model; the recording is resolved inside the op's request context with that context as the download's cancellation binding, so cancel({ modelId }) and cancel({ requestId }) stop an uncached remote recording mid-download instead of being followed by synthesis. The duplex request builders in packages/inference/src/api and packages/sdk/src/client/api forward the three fields (they whitelist).

• plugin.ts forwards the new constructor options; resolveSupertonicConfig resolves the two new sources into voiceJsonPath / noiseNpyPath artifacts. toAddonInstruct moves to ops/voice-options.ts so the load and request paths render the structured control the same way.

• Stats: ttsStatsSchema and the addon stats type gain the 28 fields; collectTtsStats is table-driven over a satisfies (keyof AddonTtsStats & keyof TtsStats)[] list, and an exhaustiveness type requires every key of either type (apart from the renamed audioDurationMs / audioDuration pair) to be in that list, so a field added to one side only, or to both without a table entry, fails to compile naming the field instead of being dropped.

• qvac serve: /v1/audio/speech forwards OpenAI's instructions as the per-request instruct (a raw instruction string) to a CosyVoice3 alias that configures no conditioning of its own. A per-request instruct replaces the alias's conditioning, so an alias whose config sets instruct, emotion or a non-moderate pace keeps the operator's choice and instructions stays an ignored param there, as on the other engines; docs and the OpenAPI description updated. The Audio8 request-level reference pair has no OpenAI analogue (a server-side recording), so it is library-only; the load-time options reach qvac serve through the model config as before.

• Docs ((v0.23)/ai-capabilities/text-to-speech.mdx): Chatterbox, Supertonic, CosyVoice3 and Audio8 sections, the stats table and the LavaSR callout.

• Contracts regenerated: packages/sdk/contract/*.json, the Python models, the Kotlin SchemaTypes.kt.

• CI: two fixes for breakage this PR ran into rather than caused. The fake AudioGenResult in packages/cli/test/e2e/default/music.test.ts gets the chunkStream and encoded fields QVAC-27588 feat[api]: add chunkStream, encoded output and MiniMax Core ML to audioGen #5604 added. And the cli entry of .github/sdk-pod-checks.json now has depends_on: [packages/sdk, packages/inference]: SDK Pod Checks selects packages by changed path, so an SDK type change never ran the CLI typecheck and could break main unnoticed, as QVAC-27588 feat[api]: add chunkStream, encoded output and MiniMax Core ML to audioGen #5604 did. A policy test in .github/scripts/test/ci-trust-policy.test.mjs pins the dependency and the selection rule, and the CLI README says when the CLI checks run.

Not in this PR

• Pocket: the addon's pocket engine is a separate engine arm (its own model files and lifecycle); exposing it needs a new ttsEngine arm, model-source resolution for four companion files, plugin dispatch, CLI preset and docs. The earlier attempt was QVAC-24306 feat[api]: expose Pocket TTS through inference, SDK and CLI #4698 (closed). It is better as its own PR.
• Parler's Core ML sidecar stats (decoderSidecarLoaded, dacSidecarLoaded, decoderOnCoreml, dacOnCoreml) are in the pinned 0.15 addon, but the SDK neither downloads Parler Core ML sidecars nor exposes coremlDecoder, so no SDK load could report them. They belong with the change that adds Parler Core ML to the SDK.
• A request-level referenceAudioSrc stays a model source (same spelling and resolver as the load-time field). A remote source that is not cached downloads inside the request, holding the model until it finishes; it is cancellable, and the docs and the schema description tell callers to pre-fetch with downloadAsset() or pass a local path.
• streamLeftContextTokens is exposed on Chatterbox only; the addon accepts it on CosyVoice3 but documents it as not yet read by the pinned engine.

🧪 How was it tested?

• packages/inference: tsc -p tsconfig.build.json reports the same 42 pre-existing errors as main (stale addon installs, none in TTS files); npm run lint and npx prettier --check . clean; tts-schemas (105 tests), tts-stats, tts-resolve-config, the new tts-voice-options (per-request forwarding on run() / runStream() / runStreaming(), Audio8 path resolution, engine rejections, and a model-scoped and a request-scoped cancel each stopping a reference download parked on a stalled loopback server, with no engine call and the model slot released; both fail against the pre-fix ops), tts-audio8-stream, tts-parler-options, tts-cancel, tts-moss, tts-moss-stream, tts-multicast, tts-side-channel, config-description-guard, registry-schema-coverage all pass.
• packages/sdk: lint, tsc --noEmit (both tsconfigs), contract-export / contract-schemas / plugin-contract tests, text-to-speech-stream-client (new per-request rule cases).
• packages/cli: test/unit/extensions/openai/audio.test.ts (new speechInstructFor cases); typecheck is CI-only locally.
• packages/sdk-python: generate.py then --check, pytest tests/test_api.py.
• packages/sdk-kotlin: generate-contract.py then --check, python3 -m unittest discover -s scripts -p 'test_*.py', check-wire-literals.py.
• Cancellation: tts-voice-options cancels a request while a cached referenceAudioSrc resolves, through textToSpeech and textToSpeechStream; both cases fail without the post-resolution abort check. Workflow policy tests: the policy-tests job's 29 files pass locally (559 tests), and the workflow's own jq selection picks sdk, inference and cli for a change under packages/inference.
• Not run: synthesis against real models (no local GGUFs); every forwarded value is asserted on the addon's resolved _setup.settings / the job object instead.

🔌 API Changes

// Chatterbox (load): sampling, prosody, length / streaming cost, pre-baked voice
await loadModel({
  modelSrc: TTS_T3_MTL_CHATTERBOX_Q8_0,
  modelType: 'tts',
  modelConfig: {
    ttsEngine: 'chatterbox',
    language: 'de',
    s3genModelSrc: TTS_S3GEN_MTL_CHATTERBOX,
    temperature: 0.8, topK: 1000, topP: 0.95, repeatPenalty: 1.2,
    exaggeration: 0.7, cfgWeight: 0.5, minP: 0.05,
    nPredict: 1500, maxSentenceChars: 180, crossfadeMs: 30,
    batchCfmSteps: 4, streamLeftContextTokens: 32,
    voiceDir: '/voices/me'
  }
})

// Supertonic (load): native chunk streaming, external voice, deployment controls
await loadModel({
  modelSrc: TTS_MULTILINGUAL_SUPERTONIC3_Q8_0,
  modelType: 'tts',
  modelConfig: {
    ttsEngine: 'supertonic',
    language: 'en',
    streamChunkTokens: 50, streamFirstChunkTokens: 20,
    streamChunkTolerancePct: 20, streamMinChunkTokens: 30,
    voiceJsonSrc: '/voices/cloned.json', noiseNpySrc: '/voices/noise.npy',
    prewarmText: 'A sentence of representative length.', vulkanDevice: -1
  }
})

// CosyVoice3: flowCutPrompt / vulkanDevice at load, instruct per request
const cosy = textToSpeech({ modelId, text: '今天天气真好。', stream: false, instruct: { dialect: 'cantonese' } })

// Audio8: switch the cloned voice per request
const audio8 = textToSpeech({
  modelId, text: 'Spoken in another voice.', stream: false,
  referenceAudioSrc: '/voices/someone-else.wav',
  referenceText: 'Exactly what that recording says.'
})

// Stats: firstAudioMs plus the engines' stage timings and counters
const stats = await cosy.stats
console.log(stats?.firstAudioMs, stats?.lmDecodeMs, stats?.speechTokens, stats?.stageTotalMs)

qvac serve: POST /v1/audio/speech forwards instructions as instruct to a CosyVoice3 alias without configured conditioning (instruct, emotion or a non-moderate pace); otherwise it stays an ignored param.

[github-actions]

Review Status

Current Status: ❌ PENDING
Approvals so far: none

Pending reviews: Needs 1 Management or Team Lead, and 1 more from Management, Team Lead, or Member.

[github-actions]

License compliance — clean

No new dependency license findings in this PR.

Warn-only (shadow) mode — this check does not block merges yet.

Updated automatically by the canonical license compliance workflow.

NOTICE presence (advisory)

Missing NOTICE (advisory, does not block):

• ./docs/website
• ./packages/asr-ggml/benchmarks/server
• ./packages/fabric/test/integration
• ./packages/llm-llamacpp/benchmarks/performance
• ./packages/llm-llamacpp/benchmarks/server
• ./packages/inference-addon-cpp/mobile
• ./packages/sdk-e2e
• ./packages/embed-llamacpp/benchmarks/performance
• ./packages/embed-llamacpp/benchmarks/server
• ./packages/vla-ggml/sim/server
• ./.github/actions/release-merge-guard
• ./.github/bare-toolchain

[greptile-apps]

Confidence Score: 5/5

[Medium impact] The PR appears safe to merge; no new actionable issue was found.

Summary

Exposes more speech controls and engine statistics through the SDK.

• Chatterbox, Supertonic, and CosyVoice3 accept more engine controls at load time.
• CosyVoice3 and Audio8 can use voice controls sent with each speech request.
• Speech results include more engine timings and work counters.
• CosyVoice3 aliases can use OpenAI instructions as per-request voice controls.
• Generated SDK contracts now include the expanded speech fields.
• CLI checks now run when SDK or inference changes can affect CLI types.

The earlier download-cancellation, cached-reference cancellation, and stats-table findings are addressed in the current code. No new actionable issue was established. Synthesis with real models was not run during this review.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[SDK speech request] --> B[Check request fields and engine]
  C[OpenAI speech instructions] --> D{CosyVoice3 without configured conditioning?}
  D -->|Yes| A
  D -->|No| E[Keep instructions ignored]
  B --> F[Open cancellable request]
  F --> G[Resolve optional reference recording]
  G --> H{Request cancelled?}
  H -->|Yes| I[Return cancelled and release model]
  H -->|No| J[Forward voice options to addon]
  J --> K[Return audio and available stats]

Loading

_{Reviews (15) · Last reviewed commit: "Merge remote-tracking branch 'upstream/m..." · Reviewed by Greptile}

[opaninakuffo]

the description is stale after the main merge. it says the SDK pins ^0.14.0 and that the parler Core ML sidecar stats wait for the next pin bump, but main now pins ^0.15.0 and those stats are in it. still fine to leave them out, since the SDK doesn't download parler sidecars or expose coremlDecoder. just update the reason.

[RamazTs]

@opaninakuffo on the stale description: updated. It now says the SDK pins ^0.15.0, and the Parler Core ML sidecar stats are left out because the SDK neither downloads Parler sidecars nor exposes coremlDecoder, so no SDK load could report them; they belong with the change that adds Parler Core ML. The description also reflects the two fixes in d43c8b2 (the instructions forwarding rule and the pre-fetch note for a remote request-level referenceAudioSrc).