[RamazTs]

🎯 What problem does this PR solve?

• The SDK pins @qvac/tts-ggml ^0.15.0, but the inference schema and the tts-ggml plugin never accepted or forwarded what the 0.15 addon added on top of what QVAC-27586 feat[api]: expose the tts-ggml controls and stats the SDK dropped #5554 exposed, so SDK users could not reach it while the addon's own API could:
  • Every engine: the explicit native backend / device selector (config.backend: auto, cpu, opencl, webgpu, MTL0, CUDA0, …), also taken by assessFit.
  • Supertonic: weight precision, the f16Attn / f16Weights / kvAttnType precision overrides, f16WeightsDenyList, vulkanEnvOverrides, and in-memory voice tensors (voiceStyleTtl / voiceStyleDp).
  • Parler: the Core ML sidecar controls coremlDecoder, coremlComputeUnits, coremlStrict, and the four stats they produce (decoderSidecarLoaded, dacSidecarLoaded, decoderOnCoreml, dacOnCoreml), which collectTtsStats dropped.
  • CosyVoice3: greedy decoding, forceSpeechTokens (replay a trajectory), and the speechTokenIds the engine hands back on its output, which the ops dropped; under native chunk streaming they arrive on an empty final chunk the streaming paths skipped as an end marker.
  • MOSS / MOSS-TTSD: the separate text- and audio-channel samplers (textTemperature / textTopP / textTopK, audioTemperature / audioTopP / audioTopK / audioRepetitionPenalty) and the nCtx / maxNewTokens budgets; durationTokens was bounded by the default budget only.
  • LavaSR: the per-stage Core ML model and compute policy (enhancer / denoiser coremlModelPath + coremlComputeUnits).
• Remaining SDK items of the QVAC-27970 epic for tts-ggml 0.15.0. Chatterbox controls, Supertonic native streaming / voices / pre-warm, per-request CosyVoice3 instruct and Audio8 reference voice landed in QVAC-27586 feat[api]: expose the tts-ggml controls and stats the SDK dropped #5554; Audio8 incremental output, chunkStream and the binary PCM transport in QVAC-27326 feat[api]: add chunked Int16Array output to textToSpeech #5348 / QVAC-27325 feat[api]: replace JSON PCM transport with binary audio transport #5380; the addon-side fixes (unconditioned MOSS output, backend fields on streamed stats, hardware-acceleration memory) arrive through the existing pin.

📝 How does it solve it?

• packages/inference/src/schemas/text-to-speech.ts: each row above is a described schema key on the engine arm the addon scopes it to (lib/fabricOptions.js ENGINE_OPTION_KEYS, lib/engines/*), with the addon's ranges (int32 knobs, topP in (0, 1], positive counts). backend rides the shared backend shape onto all eight arms; a webgpu selector is refused on Chatterbox and the MOSS engines, which the native library has no WebGPU support for (validateBackendOption). Supertonic's voice tensors come as a pair; MOSS durationTokens must fit maxNewTokens − 33 (2015 by default, as before); the LavaSR Core ML path and policy come together and need the stage's GGUF source (validateLavaSRCoreML). Three new vocabularies (TTS_PARLER_COREML_COMPUTE_UNITS, TTS_LAVASR_COREML_COMPUTE_UNITS, TTS_SUPERTONIC_PRECISIONS) are exported from the SDK root and registered for codegen.
• plugin.ts forwards the new constructor options; the LavaSR Core ML fields are load-only (compiled .mlmodelc directories on the host, not sources), so resolveConfig moves them into artifacts and createModel builds the addon's enhancer / denoiser option objects beside the GGUF files. fit.ts forwards backend, Supertonic precision / f16Weights, Parler coremlDecoder and MOSS nCtx / maxNewTokens, so the projection matches the load.
• speechTokenIds: added to both response schemas and to TtsOutputChunk / TtsOpYield / chunkMetadata / ttsFrameMetadata. The batch paths collect it with the PCM; the chunked paths keep an empty chunk that carries it instead of skipping it as an end marker (Audio8's empty markers, which carry nothing, are still skipped). The SDK client and the in-process @qvac/inference client expose it as result.speechTokenIds (the arrays of a run concatenated, one per native synthesis) and on each chunkUpdates entry.
• Stats: ttsStatsSchema, the addon stats type and the table-driven TTS_FORWARDED_STATS gain the four Parler fields (the exhaustiveness type required all three). backendId documents 5 = WebGPU.
• qvac serve / qvac configure: the new load keys reach both through the shared modelConfig schema with no CLI code. speechTokenIds has no OpenAI analogue, so it is library-only. The route's sample-rate fallback table (used only when a run produced no audio) now knows every tts-ggml engine instead of Chatterbox and Supertonic alone.
• Docs: _unpublished/sdk/next/ai-capabilities/text-to-speech.mdx (the development line) is first brought in line with the published (v0.23) page, which QVAC-27326 feat[api]: add chunked Int16Array output to textToSpeech #5348 / QVAC-27329 fix[api]: keep backend fields on streamed TTS stats and put the audiogen GPU fallback code on the wire #5384 / QVAC-27586 feat[api]: expose the tts-ggml controls and stats the SDK dropped #5554 had edited directly, then gains a Backend selection section, the Supertonic, Parler, CosyVoice3, MOSS and LavaSR additions, and the new stats rows.
• Contracts regenerated: packages/sdk/contract/schema.json, the Python models, the Kotlin SchemaTypes.kt.

Not in this PR

• Pocket TTS (streaming, incremental text, cancellation, memory preflight): a separate engine arm with four companion files, no registry model, a text fit workload and memoryBudgetBytes; the addon's own docs say SDK integration follows separately, QVAC-27586 feat[api]: expose the tts-ggml controls and stats the SDK dropped #5554 deferred it for the same reason, QVAC-24306 feat[api]: expose Pocket TTS through inference, SDK and CLI #4698 (the earlier attempt) is closed and no Pocket SDK PR is open. The CLI's qvac configure templates are keyed by TtsEngine and need a registry model per engine, which Pocket does not have yet. It stays a PR of its own.
• Supertonic's callable warmUp(text) is a new RPC method rather than a pass-through option; the load-time prewarmText already warms the engine up, so the callable form is left out (noted in the docs).
• TTSGgml.getVoiceControls() (not in the epic).

🧪 How was it tested?

• packages/inference: tsc -p tsconfig.build.json reports the same 13 pre-existing environmental errors as main (stale addon installs, none in TTS files); lunte and prettier --check clean on the touched files. Under Bare: tts-schemas (119 tests: backend on every arm and the WebGPU rule, Supertonic Fabric controls and the voice-tensor pair, Parler Core ML, CosyVoice3 greedy / forceSpeechTokens, MOSS samplers and the durationTokens / maxNewTokens budget, LavaSR Core ML pairing rules, speechTokenIds on both response schemas, Parler stats), tts-resolve-config (38: every forwarded option read back from the addon's _setup, LavaSR Core ML fields stripped into artifacts and rebuilt as enhancer / denoiser options), tts-fit-request (12), tts-stats (13), the new tts-speech-token-ids (8: batch collect, batch-over-native-stream, and the plain / sentence / duplex paths forwarding the empty token chunk while still skipping Audio8's empty end markers), tts-side-channel (6), plus tts-voice-options, tts-audio8-stream, tts-chunk-stream, tts-chunk-stream-public, tts-moss, tts-moss-stream, tts-handler-pcm, tts-cancel, tts-multicast, tts-parler-options, config-description-guard, registry-schema-coverage all pass.
• packages/sdk: tsc --noEmit on sources clean, on tests only the known local bare-link typing gap; lunte and prettier clean; tts-constants-exports (new vocabularies at the root and in the constants registry), the new tts-speech-token-ids (batch, plain stream, sentence stream chunkUpdates, engines without tokens), text-to-speech-stream-client, tts-chunk-stream, tts-batch-terminal, tts-chunk-stream-public, tts-multicast, tts-pcm-wire, contract-schemas, contract-export, plugin-contract pass; export-contract.ts --check passes.
• packages/cli: test/unit/extensions/openai/audio.test.ts (41, with the new per-engine sample-rate case); lunte and prettier clean; typecheck is CI-only locally.
• packages/sdk-python: generate.py then --check; pytest tests passes 512 tests, the 23 that spawn a real worker fail locally on the known Bare 1.28.1 vs >=1.32.0 engine gap (CI covers them); mypy -p tetherto.qvac_sdk reports the same two pre-existing errors as main (sdcpp mode, vector-index storage). packages/sdk-kotlin: generate-contract.py then --check, the generator's unittest suite (16) and check-wire-literals.py pass; vulkanEnvOverrides renders as Map<String, String>, the token arrays as List<Long>.
• Not run: synthesis against real models (no local GGUFs or Core ML models); every forwarded value is asserted on the addon's resolved _setup or the forwarded job / frame instead.

🔌 API Changes

// Every engine: name the backend instead of asking for "the GPU"
await loadModel({
  modelSrc: TTS_MULTILINGUAL_SUPERTONIC3_Q8_0,
  modelType: 'tts',
  modelConfig: {
    ttsEngine: 'supertonic',
    language: 'en',
    backend: 'CUDA0',
    // Supertonic precision / environment / in-memory voice
    precision: 'f16', f16Attn: 1, f16Weights: -1, kvAttnType: 0,
    f16WeightsDenyList: ['vocoder.*'],
    vulkanEnvOverrides: { GGML_VK_PIPELINE_CACHE_DIR: '/cache/vk' },
    voiceStyleTtl: ttl, voiceStyleDp: dp,
    // LavaSR enhancer on Core ML (macOS / iOS)
    lavasrEnhancerModelSrc: TTS_ENHANCER_LAVASR_FP16.src,
    lavasrEnhancerCoremlModelPath: '/models/lavasr-enhancer.mlmodelc',
    lavasrEnhancerCoremlComputeUnits: 'CPU_AND_NE'
  }
})

// Parler Core ML sidecars
await loadModel({
  modelSrc: TTS_MINI_V1_EN_PARLER_TTS_Q8_0,
  modelType: 'tts',
  modelConfig: { ttsEngine: 'parler', coremlDecoder: true, coremlComputeUnits: 'cpu_and_ane', coremlStrict: false }
})
const stats = await result.stats
console.log(stats?.decoderSidecarLoaded, stats?.dacOnCoreml)

// CosyVoice3: reproducible takes and replay
const take = textToSpeech({ modelId, text: 'Hello.', stream: false })
await take.buffer
const tokens = await take.speechTokenIds // number[] | undefined
await loadModel({
  modelSrc: TTS_COSYVOICE3_LLM_COSYVOICE_Q8_0,
  modelType: 'tts',
  modelConfig: { ttsEngine: 'cosyvoice3', greedy: true, forceSpeechTokens: tokens }
})

// MOSS: budgets and separate text / audio samplers
await loadModel({
  modelSrc: TTS_DELAY_LLM_MOSS_TTS_F16,
  modelType: 'tts',
  modelConfig: {
    ttsEngine: 'moss',
    mossCodecDecoderModelSrc: TTS_CODEC_DECODER_MOSS_TTS_F16,
    nCtx: 8192, maxNewTokens: 4096, durationTokens: 3000,
    textTemperature: 0.7, textTopP: 0.9, textTopK: 40,
    audioTemperature: 0.8, audioTopP: 0.95, audioTopK: 50, audioRepetitionPenalty: 1.1
  }
})

New root exports: TTS_PARLER_COREML_COMPUTE_UNITS, TTS_LAVASR_COREML_COMPUTE_UNITS, TTS_SUPERTONIC_PRECISIONS. TextToSpeechStreamResult and TtsSentenceChunkUpdate gain speechTokenIds; both response frames carry speechTokenIds.

Replay is scoped to one native synthesis: forceSpeechTokens is fixed at load, so a sentence stream's concatenated speechTokenIds cannot be replayed as one (replay it sentence by sentence from chunkUpdates). The token item is one titled int32 scalar (TtsSpeechTokenId) on both the response and forceSpeechTokens, so a returned list feeds the replay config unchanged in every generated client: Python list[TtsSpeechTokenId] on both sides (one shared class), Kotlin List<Long>, TypeScript number[].

[github-actions]

Review Status

Current Status: ✅ APPROVED
Approvals so far: Team Lead: 1, Member: 1

[github-actions]

License compliance — clean

No new dependency license findings in this PR.

Warn-only (shadow) mode — this check does not block merges yet.

Updated automatically by the canonical license compliance workflow.

NOTICE presence (advisory)

Missing NOTICE (advisory, does not block):

• ./docs/website
• ./packages/asr-ggml/benchmarks/server
• ./packages/fabric/test/integration
• ./packages/llm-llamacpp/benchmarks/performance
• ./packages/llm-llamacpp/benchmarks/server
• ./packages/inference-addon-cpp/mobile
• ./packages/sdk-e2e
• ./packages/embed-llamacpp/benchmarks/performance
• ./packages/embed-llamacpp/benchmarks/server
• ./packages/vla-ggml/sim/server
• ./.github/actions/release-merge-guard
• ./.github/bare-toolchain

[greptile-apps]

Confidence Score: 5/5

[Medium impact] The PR appears safe to merge; no new actionable issue was established.

Summary

Exposes the tts-ggml 0.15 controls through inference and the SDK.

• TTS models accept more backend and engine-specific settings.
• CosyVoice3 speech tokens reach callers and can replay a synthesis.
• OpenAI audio fallback rates now cover every TTS engine.

Both earlier findings are addressed: replay docs now limit replay to one native synthesis, and both clients append tokens into one owned array. The supplied threads have no comment numbers, so they have no numbered previous-finding entries.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  A[Load config] --> B[Engine schema]
  B --> C[Resolve GGUF sources]
  C --> D[Build addon options]
  B --> E[Build fit request]
  D --> F[Native synthesis]
  F --> G[Audio and speech tokens]
  G --> H[Response frames]
  H --> I[SDK audio streams]
  H --> J[Speech-token result and chunk updates]

Loading

_{Reviews (9) · Last reviewed commit: "Merge branch 'main' into feat/QVAC-27970..." · Reviewed by Greptile}

[GustavoA1604]

[medium] packages/sdk-python/src/tetherto/qvac_sdk/_generated/models/_internal.py — line 26738 (speech_token_ids: Annotated[)

Responses expose list[SpeechTokenId], but force_speech_tokens expects list[ForceSpeechToken]. Passing response.speech_token_ids directly into the CosyVoice3 replay configuration raises a Pydantic ValidationError for every token. Use a compatible scalar representation for both fields and add a Python regression test that feeds returned tokens into the replay configuration.

[RamazTs]

Thanks, valid: the response arrays carried safe-integer items while forceSpeechTokens carried int32 items, so datamodel-code-generator emitted two unrelated RootModel[int] classes and a returned trajectory failed validation token by token.

Fixed at the schema source in 7b37565: both arrays now take one titled int32 scalar, TtsSpeechTokenId, defined once in packages/inference/src/schemas/text-to-speech.ts. The Python generator turns that into a single shared class, so response.speech_token_ids is accepted by force_speech_tokens as is (list[TtsSpeechTokenId] on both sides); Kotlin stays List<Long> and TypeScript number[] on both. schema.json and the Python models are regenerated, the Kotlin types did not change.

Regression test: packages/sdk-python/tests/test_tts_speech_tokens.py validates TextToSpeechResponse and TextToSpeechStreamResponse frames carrying speechTokenIds, feeds the returned list straight into LoadModelSrcRequestTtsGgmlModelConfigCosyvoice3(force_speech_tokens=...), checks the wire dump, that plain ints still work, and that both fields share one item annotation. Ran pytest tests, mypy -p tetherto.qvac_sdk, mypy scripts, mypy tests, black and ruff in a fresh venv as CI does: all green.

The branch also carries the main merge (a2add92, regenerated SchemaTypes.kt) on top of your merge commit.