QVAC-27586 feat[api]: expose the tts-ggml controls and stats the SDK dropped

docs/website/content/docs/sdk/(v0.23)/ai-capabilities/text-to-speech.mdx

@@ -97,14 +97,14 @@ A cancelled run ends cleanly rather than as an error: `done` resolves `true`,
 statistics from the final frame (`undefined` if the run reported none).
 
 <Callout type="info">
-  The five backend fields (`backendDevice`, `backendId`, `gpuUnsupported`,
-  `enhancerBackendDevice`, `enhancerBackendId`) are reported by batch
-  synthesis (`stream: false`) only. The chunked paths — `stream: true` (the
-  default), `sentenceStream: true`, and `textToSpeechStream()` — run one
-  native job per sentence and aggregate only the timing and sample counts,
-  plus Audio8's two Core ML fields, which keep the last sentence's value.
-  MOSS is the exception for `stream: true`: it runs the whole text as one
-  native job, so it reports them there too.
+  The chunked paths — `stream: true` (the default), `sentenceStream: true`,
+  and `textToSpeechStream()` — run one native job per sentence. Their
+  aggregate sums the timing and sample counts, keeps `firstAudioMs` from the
+  first sentence and the backend and Core ML fields from the last sentence
+  that reported them, and carries no per-stage timings or counters: those
+  describe one native synthesis, so they are reported by batch synthesis
+  (`stream: false`) only. MOSS is the exception for `stream: true`: it runs
+  the whole text as one native job, so it reports everything there too.
 </Callout>
 
 | Field | Meaning |
@@ -122,6 +122,12 @@ statistics from the final frame (`undefined` if the run reported none).
 | `enhancerBackendId` | LavaSR enhancer backend family, same codes as `backendId`. |
 | `codecSidecarLoaded` | Audio8 on macOS/iOS: `1` while the Core ML codec sidecar is attached, `0` without one or once a failing sidecar was retired. |
 | `codecOnCoreml` | Audio8: `1` when the synthesis ran its codec on the Core ML sidecar, `0` when it ran on the ggml backend `backendId` names. |
+| `firstAudioMs` | Milliseconds from the start of the run to the first audio, where the engine reports it. |
+| `t3Ms`, `s3genMs`, `t3Tokens` | Chatterbox: T3 decode and S3Gen + HiFT wall times in milliseconds, and the speech tokens T3 emitted. |
+| `stageTotalMs` | CosyVoice3 and Audio8: the engine's own wall time in milliseconds, the sum of its stages, excluding the addon's post-processing. |
+| `lmPrefillMs`, `lmDecodeMs`, `flowFrontendMs`, `ditEulerMs`, `hiftF0Ms`, `hiftSourceMs`, `hiftStftMs`, `hiftDecodeMs` | CosyVoice3 stage wall times, in milliseconds. |
+| `decodeSteps`, `speechTokens`, `flowFrames`, `melFrames`, `textIds`, `promptSpeechTokens` | CosyVoice3 work counters: LM decode steps, speech tokens generated, flow frames in (prompt + generated), mel frames out, text ids in the LM prefill, and the prompt speech tokens the LM continued from. |
+| `voiceEncodeMs`, `promptMs`, `prefillMs`, `sampleMs`, `fastDecodeMs`, `slowDecodeMs`, `codecLatentMs`, `codecSynthMs`, `resampleMs` | Audio8 stage wall times, in milliseconds. |
 
 ```ts
 const stats = await result.stats;
@@ -163,6 +169,31 @@ as on the other engines; `openclCacheDir` (the Android OpenCL program cache)
 is accepted by Chatterbox, Supertonic and CosyVoice3 only. Chatterbox supports
 [LavaSR](#lavasr) post-processing.
 
+Instead of (or alongside) `referenceAudioSrc`, `voiceDir` points at a
+directory of pre-baked voice-conditioning tensors — the upstream CLI's
+`--save-voice` output — which skips the reference preprocessing; when both are
+set, tensors missing from the directory are backfilled from the recording.
+
+T3 sampling is tunable: `temperature` (engine default `0.8`; `0` decodes
+greedily), `topK` (`1000`; `0` disables the cutoff), `topP` (`0.95`) and
+`repeatPenalty` (`1.2`; `1` turns it off). The multilingual GGUFs also take
+`exaggeration` (prosody intensity, default `0.5`; above `0` more expressive,
+below `0` flatter), `cfgWeight` (T3 classifier-free-guidance scale, default
+`0.5`; `0` disables it, and it is separate from `cfgRate`, which steers S3Gen)
+and `minP` (min-p threshold, `0` = off); the Turbo variant ignores the three.
+
+Length and streaming cost: `nPredict` caps the speech tokens T3 decodes per
+synthesis (default `1000`, about 40 s); longer single calls also need `nCtx`
+to cover the prompt plus the generated tokens, or set `maxSentenceChars` so
+T3 + S3Gen run per segment of at most that many bytes (the reference CLI uses
+`180`), with `crossfadeMs` (default `30`) blending the segments in batch
+synthesis. `batchCfmSteps` sets the CFM Euler steps for batch synthesis (`0`
+= the library default: 2-step meanflow on Turbo, the model's full schedule on
+multilingual) while `cfmSteps` keeps driving native streaming, and
+`streamLeftContextTokens` bounds the per-chunk cost of long native streams by
+keeping only that many already-emitted speech tokens as S3Gen left context
+(typically 25–50; `0` = the whole emitted prefix).
+
 ### Supertonic
 
 Supertonic uses a **single GGUF** via top-level `modelSrc`. Set `voice`, `ttsSpeed`, and `ttsNumInferenceSteps` in `modelConfig` as needed. Multilingual output is selected by the GGUF (e.g. `TTS_MULTILINGUAL_SUPERTONIC2_Q8_0`) plus `language`.
@@ -189,7 +220,32 @@ per request — set it in `modelConfig`.
 `seed`, `threads`, `nGpuLayers`, `outputSampleRate` and `backendsDir` work
 as on the other engines; `openclCacheDir` is accepted by Chatterbox,
 Supertonic and CosyVoice3 only, and `vulkanCacheDir` additionally persists
-the compiled Vulkan pipeline cache when `useGPU` is set.
+the compiled Vulkan pipeline cache when `useGPU` is set. `vulkanDevice` picks
+the Vulkan adapter (`0` = the first, `N` = the Nth, `-1` = auto-pick by free
+VRAM preferring a discrete GPU) and is also used by the LavaSR enhancer;
+`prewarmText` is synthesized once at load so GPU pipelines (and a Core ML
+vocoder sidecar) are ready before the first request.
+
+Voices: `voice` selects a preset baked into the GGUF (`F1`, `M1`, …), and
+`voiceJsonSrc` loads an external voice JSON (`{ style_ttl, style_dp }`, e.g.
+a cloned voice) that overrides it; the engine checks the tensor sizes against
+the loaded model. `noiseNpySrc` supplies a fixed CFM initial-noise `.npy`
+tensor for reproducible output.
+
+Supertonic also streams natively: set `streamChunkTokens` above zero to emit
+audio per chunk of that many text tokens (Unicode code points — about 50 for
+English, 25–30 for CJK). `streamFirstChunkTokens` shrinks the first chunk for
+lower first-audio latency, `streamChunkTolerancePct` (default `20`) is the
+window, as a percentage of the chunk target, within which a boundary snaps to
+a sentence end, clause or space, and `streamMinChunkTokens` (default `30`)
+merges a shorter trailing chunk into the previous one.
+
+<Callout type="warn">
+  Supertonic native chunk streaming cannot be combined with the LavaSR
+  enhancer or denoiser; the pair is rejected at load. Sentence streaming
+  (`stream: true` without `streamChunkTokens`, `sentenceStream: true`, or
+  `textToSpeechStream()`) works with either stage loaded.
+</Callout>
 
 ### Parler-TTS
 
@@ -334,18 +390,34 @@ Conditioning uses one of three controls:
 <Callout type="warn">
   CosyVoice3 accepts **one conditioning control per synthesis** — an
   `emotion`, a non-`moderate` `pace`, or an `instruct`. Combining them is
-  rejected at load and request validation. `emotion` and `pace` can also be
-  set per request on `textToSpeech()`; `instruct` is load-time only.
+  rejected at load and request validation. All three can also be set per
+  request on `textToSpeech()` and `textToSpeechStream()`, where the request's
+  control replaces the loaded one for that synthesis.
 </Callout>
 
+```ts
+const result = textToSpeech({
+  modelId,
+  text: "今天天气真好。",
+  stream: false,
+  // Per-request control; a raw string is passed to the engine verbatim.
+  instruct: { dialect: "cantonese" },
+});
+```
+
 Set `streamChunkTokens` above zero for native chunk streaming
 (`streamFirstChunkTokens` tunes the first chunk only). Native streaming emits
 at 24 kHz, so omit `outputSampleRate` or set it to `24000` unless the LavaSR
 enhancer is loaded. `useGPU` offloads to Metal on macOS/iOS, Vulkan on
-desktop Linux/Windows, and OpenCL on Adreno Android devices; `threads`,
-`nGpuLayers`, `seed`, and `outputSampleRate` (8,000–192,000 Hz) work as on
-the other engines. CosyVoice3 also supports [LavaSR](#lavasr)
-post-processing via `lavasrEnhancerModelSrc` (48 kHz bandwidth extension) and
+desktop Linux/Windows, and OpenCL on Adreno Android devices; `vulkanDevice`
+picks the Vulkan adapter (`0` = the first, `N` = the Nth, `-1` = auto-pick by
+free VRAM preferring a discrete GPU) and is also used by the LavaSR enhancer.
+`flowCutPrompt` treats the baked-voice prompt frames as attention-only
+conditioning in the flow — a faster flow whose output deviates slightly from
+the reference (off by default). `threads`, `nGpuLayers`, `seed`, and
+`outputSampleRate` (8,000–192,000 Hz) work as on the other engines.
+CosyVoice3 also supports [LavaSR](#lavasr) post-processing via
+`lavasrEnhancerModelSrc` (48 kHz bandwidth extension) and
 `lavasrDenoiserModelSrc` (batch-only; runs before the enhancer).
 
 #### CosyVoice3 voice cloning
@@ -428,6 +500,25 @@ supply `referenceAudioSrc` (a reference recording) together with
 together — a reference without its transcript or without the encoder is
 rejected at load.
 
+With the encoder loaded, the voice can also change per request:
+`textToSpeech()` and `textToSpeechStream()` take `referenceAudioSrc` and
+`referenceText` too. A request-level recording replaces the loaded voice for
+that synthesis and must bring its own `referenceText` (the loaded transcript
+describes a different recording, so it is not inherited); a request-level
+`referenceText` on its own corrects the transcript of the loaded recording.
+The engine caches the codes of the most recent reference, so repeating one
+across requests skips the encoder.
+
+```ts
+const result = textToSpeech({
+  modelId,
+  text: "Spoken in another voice.",
+  stream: false,
+  referenceAudioSrc: "/path/to/someone-else.wav",
+  referenceText: "Exactly what that recording says.",
+});
+```
+
 Generation is sampled by default: `temperature` (default `0.7`), `topK`
 (`50`), and `topP` (`0.9`) tune it, `greedy: true` takes the argmax instead
 and ignores all three, and `maxFrames` caps generation in codec frames
@@ -451,7 +542,9 @@ synthesis fails on it, Audio8 runs the codec on ggml instead; the
   Audio8 has no `emotion`, `pace`, or description fields (at load or per
   request), no LavaSR post-processing, and no native chunk streaming.
   Sentence streaming (`sentenceStream: true`) and duplex
-  `textToSpeechStream()` work as with every engine.
+  `textToSpeechStream()` work as with every engine. The request-level
+  `referenceAudioSrc` / `referenceText` pair is Audio8-only; other engines
+  reject it.
 </Callout>
 
 For model constants, see [SDK — Models](/sdk#models).
@@ -635,7 +728,9 @@ With Chatterbox, Supertonic, Parler, or CosyVoice3, you can choose to perform po
   The denoiser needs the whole utterance, so it cannot be combined with native
   chunk streaming (`streamChunkTokens > 0`) on Chatterbox, Parler, or
   CosyVoice3. The combination is rejected at load. The enhancer has no such
-  restriction — it resamples seam-free across chunk boundaries.
+  restriction there — it resamples seam-free across chunk boundaries.
+  Supertonic native chunk streaming rejects both stages; use sentence
+  streaming for enhanced or denoised Supertonic output.
 </Callout>
 
 ```ts

packages/cli/docs/serve/openai.md

@@ -909,7 +909,8 @@ When `voice` is omitted, the configured **`serve.openai.audio.speech.defaultVoic
   - `voice` (optional, defaults to `defaultVoice`) — an OpenAI built-in voice name (`alloy`, `ash`, `ballad`, `coral`, `echo`, `fable`, `onyx`, `nova`, `sage`, `shimmer`, `verse`, `marin`, `cedar`), any other string, or a custom voice reference `{ "id": "voice_1234" }`. The object form routes the same way as its `id` string.
   - `response_format` (optional) — `wav` (default), `pcm` (raw 16-bit signed little-endian PCM, mono), or `mp3` / `opus` / `aac` / `flac`. The encoded formats are produced by transcoding the synthesized audio through **`ffmpeg`**, which must be on the server's `PATH`; when ffmpeg is absent they return `503 transcode_unavailable` (use `wav`/`pcm` or install ffmpeg — see `qvac doctor`). The default stays `wav` so synthesis works on hosts without ffmpeg.
   - `stream_format` (optional) — `audio` (default, one audio body) or `sse` (see [Streaming](#streaming-stream_format-sse)). Other values return `400 invalid_stream_format`.
-- **Accepted but ignored:** `speed`, `instructions` (a warning is logged; ignored keys are echoed in the success response via the `X-QVAC-Ignored-Params` header).
+  - `instructions` (optional) — forwarded to CosyVoice3 aliases (`ttsEngine: "cosyvoice3"`) as the engine's per-request `instruct` control: a raw natural-language instruction such as a Chinese dialect, emotion or speaking style, applied to this synthesis only. Other engines have no free-text voice control, so there it is an ignored param.
+- **Accepted but ignored:** `speed`, and `instructions` on every engine but CosyVoice3 (a warning is logged; ignored keys are echoed in the success response via the `X-QVAC-Ignored-Params` header).
 
 ### Response
 

packages/cli/src/serve/extensions/openai/audio.ts

@@ -185,6 +185,16 @@ export function buildWavBuffer(samples: number[], sampleRate: number): Buffer {
   return Buffer.concat([header, data])
 }
 
+// OpenAI's `instructions` steers the voice in natural language, which is what
+// CosyVoice3's per-request `instruct` control does, so it is forwarded there
+// verbatim (a raw instruction string). No other engine takes a free-text
+// control, so elsewhere the route keeps treating it as an ignored param.
+export function speechInstructFor(ttsEngine: unknown, instructions: unknown): string | undefined {
+  if (ttsEngine !== 'cosyvoice3' || typeof instructions !== 'string') return undefined
+  const trimmed = instructions.trim()
+  return trimmed.length > 0 ? trimmed : undefined
+}
+
 // Voice + model → alias lookup key. The route tries this first and falls
 // back to the bare model name when no <model>-<voice> alias exists, so
 // existing single-alias TTS configs continue to work even when callers omit

packages/cli/src/serve/extensions/openai/routes/audio.ts

@@ -32,7 +32,8 @@ import {
   mapResponseFormat,
   pcmContentType,
   resolveSampleRate,
-  speechAliasKey
+  speechAliasKey,
+  speechInstructFor
 } from '@/serve/extensions/openai/audio'
 import {
   transcodeWav,
@@ -140,8 +141,13 @@ as usual; a failure after it arrives as a final \`{ "error": ... }\` event.
 \`model_not_found\`; \`502\` \`speech_empty\`, \`transcode_failed\`; \`503\`
 \`transcode_unavailable\`, \`model_not_ready\`.
 
+**\`instructions\`** reaches CosyVoice3 aliases (\`ttsEngine\` \`cosyvoice3\`)
+as the engine's per-request \`instruct\` control, a raw natural-language
+instruction such as a Chinese dialect, emotion or speaking style. Other engines
+have no free-text voice control, so there it is an ignored param.
+
 **Ignored params** (logged, returned in \`X-QVAC-Ignored-Params\` header):
-\`speed\`, \`instructions\`.
+\`speed\`; \`instructions\` on every engine but CosyVoice3.
 `.trim(),
   voices: `
 List the configured TTS voices. Returns the OpenAI \`voice\` names mapped under
@@ -465,6 +471,11 @@ const plugin: FastifyPluginAsyncZod = async (app) => {
       for (const key of SPEECH_UNSUPPORTED_PARAMS) {
         if ((body as Record<string, unknown>)[key] !== undefined) ignoredParams.push(key)
       }
+      const instruct = speechInstructFor(ttsEngine, body.instructions)
+      if (body.instructions !== undefined && ttsEngine !== 'cosyvoice3') {
+        ctx.logger.warn('Ignoring unsupported param: instructions')
+        ignoredParams.push('instructions')
+      }
 
       ctx.logger.info(
         `  speech model=${alias} voice=${voice} format=${formatMapping.format} chars=${input.length} ` +
@@ -478,6 +489,7 @@ const plugin: FastifyPluginAsyncZod = async (app) => {
           format: formatMapping.format === 'pcm' ? 'pcm' : 'wav',
           fallbackSampleRate: resolveSampleRate(registryEntry.config),
           ignoredParams,
+          ...(instruct !== undefined ? { instruct } : {}),
           ...(openai.textToSpeechStreamOverride !== undefined
             ? { streamFn: openai.textToSpeechStreamOverride }
             : {})
@@ -489,7 +501,8 @@ const plugin: FastifyPluginAsyncZod = async (app) => {
         modelId: sdkModelId,
         text: input,
         inputType: 'text',
-        stream: true
+        stream: true,
+        ...(instruct !== undefined ? { instruct } : {})
       })
       // A client that disconnects mid-synthesis should stop the engine, not
       // just stop receiving — same binding completions and transcriptions use.
@@ -630,6 +643,8 @@ interface StreamSpeechOptions {
   format: 'wav' | 'pcm'
   fallbackSampleRate: number
   ignoredParams: string[]
+  /** CosyVoice3 per-request natural-language control (OpenAI `instructions`). */
+  instruct?: string
   streamFn?: typeof textToSpeechStream
 }
 
@@ -641,7 +656,8 @@ async function streamSpeech(
   const logger = req.server.qvac.logger
   const session = await (opts.streamFn ?? textToSpeechStream)({
     modelId: opts.sdkModelId,
-    inputType: 'text'
+    inputType: 'text',
+    ...(opts.instruct !== undefined ? { instruct: opts.instruct } : {})
   })
   req.bindCancel(session.requestId)
   session.write(opts.input)

packages/cli/src/serve/extensions/openai/schemas/audio.ts

@@ -109,7 +109,9 @@ export const audioSpeechBody = z
   })
   .passthrough()
 
-export const SPEECH_UNSUPPORTED_PARAMS = ['speed', 'instructions'] as const
+// `instructions` is not here: the speech route forwards it to CosyVoice3 as
+// the per-request `instruct` control and only ignores it on other engines.
+export const SPEECH_UNSUPPORTED_PARAMS = ['speed'] as const
 
 export const voiceObject = z.object({
   id: z.string(),

packages/cli/test/unit/extensions/openai/audio.test.ts

@@ -10,7 +10,8 @@ import {
   buildWavHeader,
   buildStreamingWavHeader,
   buildWavBuffer,
-  speechAliasKey
+  speechAliasKey,
+  speechInstructFor
 } from '@/serve/extensions/openai/audio'
 import { parseOpenAIOptions } from '@/serve/extensions/openai/config'
 import { speechEncodeArgs, ENCODED_SPEECH_FORMATS } from '@/serve/lib/audio-transcode'
@@ -294,3 +295,17 @@ describe('parseOpenAIOptions — audio.speech.maxInputChars', () => {
     }
   })
 })
+
+describe('speechInstructFor', () => {
+  it('forwards a trimmed instructions string to CosyVoice3 models', () => {
+    assert.equal(speechInstructFor('cosyvoice3', '  请用广东话表达。 '), '请用广东话表达。')
+  })
+
+  it('returns undefined for other engines, blank text, and non-strings', () => {
+    assert.equal(speechInstructFor('chatterbox', 'speak softly'), undefined)
+    assert.equal(speechInstructFor(undefined, 'speak softly'), undefined)
+    assert.equal(speechInstructFor('cosyvoice3', '   '), undefined)
+    assert.equal(speechInstructFor('cosyvoice3', undefined), undefined)
+    assert.equal(speechInstructFor('cosyvoice3', 42), undefined)
+  })
+})

packages/inference/src/api/text-to-speech.ts

@@ -184,7 +184,13 @@ function buildTextToSpeechStreamRequest(
     ...(params.expressivity !== undefined && { expressivity: params.expressivity }),
     ...(params.noise !== undefined && { noise: params.noise }),
     ...(params.reverb !== undefined && { reverb: params.reverb }),
-    ...(params.quality !== undefined && { quality: params.quality })
+    ...(params.quality !== undefined && { quality: params.quality }),
+    // Per-request voice controls: CosyVoice3's instruct, Audio8's reference pair.
+    ...(params.instruct !== undefined && { instruct: params.instruct }),
+    ...(params.referenceAudioSrc !== undefined && {
+      referenceAudioSrc: params.referenceAudioSrc
+    }),
+    ...(params.referenceText !== undefined && { referenceText: params.referenceText })
   }
 }