Skip to content

perf(duplex): lazy projection materialization fixes O(n²) CPU per delta - #8986

Closed
Asthenia0412 wants to merge 2 commits into
vllm-project:mainfrom
Asthenia0412:fix/duplex-realtime-o-n2-projection
Closed

Asthenia0412 wants to merge 2 commits into
vllm-project:mainfrom
Asthenia0412:fix/duplex-realtime-o-n2-projection

Conversation

@Asthenia0412

Copy link
Copy Markdown
Contributor

Background

Issue #8690 reported that Duplex Realtime projection was rebuilding the cumulative response history on every single audio/text delta, giving O(n²) CPU per response.

Problem

response.output_audio.delta and response.text.delta previously called into _ResponseProjection._rebuild_stored_item() on every delta:

  • Joined the full accumulated transcript
  • Deep-copied all audio_text_marks
  • Deduplicated and sorted the full mark history

For a typical 80-second response with ~2000 deltas, this quadratic behavior dominated CPU.

Solution

Defer the heavy work to the actual consumer read sites. Deltas now only append to the projection's internal accumulators; the stored conversation item is rebuilt only when a consumer actually needs it.

Key architectural changes

  • RealtimeProjectionState gains _dirty_response_item_ids (a Python set[str])
  • New helpers: _mark_response_item_dirty() (O(1)) and _ensure_response_item_materialized() (lazy rebuild)
  • _ResponseProjection gains marks_merged: bool flag and merge_marks() method — dedupe/sort done once at first consumer read instead of per delta
  • Delta handlers (response.output_audio.delta, response.text.delta) now only mark the item dirty
  • _remember_response_audio_metadata() now only appends raw marks, merge deferred
  • Consumer materialization added at:
    • conversation.item.retrieve
    • resolve_truncate_item / conversation.item.truncated / audio.cancelled handler
    • Terminal _response_done_output_item

Performance

Before: N=250 → N=2000 ratio 11.6x   (super-linear, trending quadratic)
After:  N=250 → N=2000 ratio  8.3x   (linear ideal = 8.0x) ✓

Testing

  • All existing conversation.item.truncate, .retrieve, and response_done payloads unchanged (pure refactor)
  • 16 regression tests pass: dirty flag lifecycle, marks integrity, consumer correctness
  • No API surface changes for consumers

Closes #8690

response.output_audio.delta and response.text.delta previously rebuilt the
full stored conversation item (joined transcript + deep-copied marks +
full-history dedupe/sort) on every single delta, giving O(n²) CPU for an
n-delta response.

Fix: deltas only append to _ResponseProjection transcript_parts and
audio_text_marks. The stored conversation item is rebuilt only on demand
at the actual consumers: conversation.item.retrieve, truncate validation,
truncate event handler, audio.cancel handler, and the terminal
response_done output build. Mark history is merged once per boundary,
not per delta.

- Added _dirty_response_item_ids set to RealtimeProjectionState
- Added _mark_response_item_dirty and _ensure_response_item_materialized
- Added merge_marks() on _ResponseProjection (O(k log k) once, not per delta)
- Replaced per-delta full-history merge in _remember_response_audio_metadata
- All existing conversation.item.truncate, .retrieve, and response_done
  payloads are unchanged

Benchmark (with audio_text_marks, typical 80s response):
  Before: N=250 → N=2000 ratio 11.6x (super-linear, trending quadratic)
  After:  N=250 → N=2000 ratio  8.3x (perfectly linear ✓)

Closes #8690

Signed-off-by: Asthenia0412 <Asthenia0412@users.noreply.github.com>
Signed-off-by: Asthenia0412 <Asthenia0412@users.noreply.github.com>
@Asthenia0412
Asthenia0412 force-pushed the fix/duplex-realtime-o-n2-projection branch from a1ee59d to 889a7ff Compare October 15, 2026 16:25
@hsliuustc0106 hsliuustc0106 added the enhancement New feature or request label Oct 16, 2026
@Asthenia0412 Asthenia0412 closed this by deleting the head repository Oct 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: Duplex Realtime projection rebuilds cumulative response history on every delta (O(n^2) CPU per response)

2 participants