Repository navigation
perf(duplex): lazy projection materialization fixes O(n²) CPU per delta - #8986
Closed
Asthenia0412 wants to merge 2 commits into
Closed
Asthenia0412 wants to merge 2 commits into
Asthenia0412 wants to merge 2 commits into
Conversation
response.output_audio.delta and response.text.delta previously rebuilt the full stored conversation item (joined transcript + deep-copied marks + full-history dedupe/sort) on every single delta, giving O(n²) CPU for an n-delta response. Fix: deltas only append to _ResponseProjection transcript_parts and audio_text_marks. The stored conversation item is rebuilt only on demand at the actual consumers: conversation.item.retrieve, truncate validation, truncate event handler, audio.cancel handler, and the terminal response_done output build. Mark history is merged once per boundary, not per delta. - Added _dirty_response_item_ids set to RealtimeProjectionState - Added _mark_response_item_dirty and _ensure_response_item_materialized - Added merge_marks() on _ResponseProjection (O(k log k) once, not per delta) - Replaced per-delta full-history merge in _remember_response_audio_metadata - All existing conversation.item.truncate, .retrieve, and response_done payloads are unchanged Benchmark (with audio_text_marks, typical 80s response): Before: N=250 → N=2000 ratio 11.6x (super-linear, trending quadratic) After: N=250 → N=2000 ratio 8.3x (perfectly linear ✓) Closes #8690 Signed-off-by: Asthenia0412 <Asthenia0412@users.noreply.github.com>
Signed-off-by: Asthenia0412 <Asthenia0412@users.noreply.github.com>
Asthenia0412
force-pushed
the
fix/duplex-realtime-o-n2-projection
branch
from
October 15, 2026 16:25
a1ee59d to
889a7ff
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Background
Issue #8690 reported that
Duplex Realtimeprojection was rebuilding the cumulative response history on every single audio/text delta, giving O(n²) CPU per response.Problem
response.output_audio.deltaandresponse.text.deltapreviously called into_ResponseProjection._rebuild_stored_item()on every delta:audio_text_marksFor a typical 80-second response with ~2000 deltas, this quadratic behavior dominated CPU.
Solution
Defer the heavy work to the actual consumer read sites. Deltas now only append to the projection's internal accumulators; the stored conversation item is rebuilt only when a consumer actually needs it.
Key architectural changes
RealtimeProjectionStategains_dirty_response_item_ids(a Python set[str])_mark_response_item_dirty()(O(1)) and_ensure_response_item_materialized()(lazy rebuild)_ResponseProjectiongainsmarks_merged: boolflag andmerge_marks()method — dedupe/sort done once at first consumer read instead of per deltaresponse.output_audio.delta,response.text.delta) now only mark the item dirty_remember_response_audio_metadata()now only appends raw marks, merge deferredconversation.item.retrieveresolve_truncate_item/conversation.item.truncated/audio.cancelledhandler_response_done_output_itemPerformance
Testing
Closes #8690