Skip to content

fix(telemetry): publish failed operation metrics - #6237

Draft
ehz0ah wants to merge 1 commit into
volcengine:mainfrom
ehz0ah:fix/failed-operation-metrics
Draft

ehz0ah wants to merge 1 commit into
volcengine:mainfrom
ehz0ah:fix/failed-operation-metrics

Conversation

@ehz0ah

@ehz0ah ehz0ah commented Oct 18, 2026

Copy link
Copy Markdown
Contributor

Summary

  • publish finished telemetry summaries on both success and failure paths
  • keep bridge publication best-effort so metrics failures cannot replace the operation result
  • verify failed operations increment the existing operation request and duration metrics with status="error"

Why

run_with_telemetry() already finishes and logs an error snapshot, but it re-raises before the existing metrics bridge call. As a result, failed search operations appear in the generic HTTP 500 metric but are absent from operation-level metrics.

This was reproduced through POST /api/v1/search/search at 0fcc1663811914484163993735dfbef2c3ca2080: the request emitted one generic http.request event and zero telemetry.summary events. The same probe after this change emits exactly one operation summary with operation="search.search", status="error", and RuntimeError, while preserving the public 500 response.

The investigation was prompted by Cognee issue #5991, which reports elevated production recall failures and the need for operation-level error telemetry. The implementation here is independent and limited to OpenViking's existing telemetry bridge contract.

Closes #6236.

Validation

  • pytest -q tests/metrics/integration/test_telemetry_bridge.py tests/telemetry/test_execution.py — 16 passed
  • source-pinned endpoint probe — baseline reproduced twice; fixed result reproduced twice
  • fixed probe — response unchanged at generic 500, one telemetry.summary event with status="error"
  • bridge-failure regression — the original operation exception remains authoritative
  • ruff check on all changed files
  • ruff format --check on all changed files
  • git diff --check
  • focused mypy comparison — both base and changed revisions report only the same five pre-existing optional OpenTelemetry fallback errors

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Backlog

Development

Successfully merging this pull request may close these issues.

[Bug]: Failed operations are absent from operation metrics

1 participant