[agent-job-health] Agent Job Health Monitor — 2026-10-13: fleet rate 2.36%, no blind spots, no new issue

[github-actions[bot]]

Note

Fleet is healthy. Failure rate dropped vs. the last two reports and no novel, actionable cluster meets the issue-creation bar.

Summary

• Window evaluated: intended last 24h ending 2026-10-13T23:16 UTC, but the pre-downloaded local log cache only covered ~8 hours (2026-10-13T15:21–23:17 UTC) before hitting the collection budget — same partial-coverage pattern seen on 2026-10-09 and 2026-10-12. Numbers below describe that ~8h subset, not the full 24h window.
• total_runs: 302 (218 with full per-job detail, 84 with only top-level run metadata) · distinct_workflows: 53 · agent_failures: 7 (6 runs still in_progress at collection time were excluded from the rate)
• Run-weighted fleet failure rate: 7 / 296 resolved runs = 2.36%
• Workflow-weighted rate: median 0.0%, mean 8.96% (5 of 53 workflows had any failure) — the gap between the two confirms the rate is driven by a handful of chronically-flaky daily workflows, not a broad regression.
• Tracked vs. novel: GitHub issue search/read was blocked by an integrity policy in this session (3rd consecutive run with this block — see §38483235574), so Tracked/Novel could not be verified live. Cross-referencing this workflow's own history log instead (see Novel Clusters below).
• Trend: rate decreased from 6.06% (2026-10-12) → 2.36% (today). No >10pp regression.

Failure Rate by Step

View per-step breakdown (7 confirmed agent-job failures)

Failing step	Runs	Share of agent_failures
Execute Codex CLI	3	43%
Execute GitHub Copilot CLI	2	29%
Warm up model	1	14%
Pre-compute recent changes governance context	1	14%

Tracked Failures

Issue search/read was blocked by integrity policy this run (resources below "approved" integrity could not be opened). Based on this workflow's own history.jsonl, "Execute Codex CLI" and "Execute GitHub Copilot CLI" step failures are chronic signatures that have recurred in nearly every prior report (tracked historically under issues #54242/#65950 for Codex, #54186/#55413 lineage for Copilot CLI) — treating these as likely-tracked rather than novel, pending maintainer confirmation.

Workflow	Failing runs	Likely covers
Daily Go Test Parallelizer	3	Chronic "Execute Codex CLI" cluster (historical #54242 / #65950)
Daily Security Observability Report, Daily Reliability Review	2	Chronic "Execute GitHub Copilot CLI" cluster (historical #54186 lineage)

Novel Failure Clusters

Cluster	Runs	Workflows	Engine	Representative run	Error excerpt	Suspected cause
Warm up model step failure	1	Daily BYOK Ollama Test	copilot	§38477676843	Agent job failed at the "Warm up model" step before the CLI ran	Likely Ollama backend unavailable/slow to respond; single-workflow, single-occurrence — blast radius: one workflow
Pre-compute recent changes governance context step failure	1	UK AI Operational Resilience	copilot	§38423321589	Agent job failed in a workflow-specific pre-processing step (not the CLI step itself)	Workflow-specific data-prep script issue; single-workflow, single-occurrence — blast radius: one workflow

Neither cluster affects 2+ workflows or ≥10% of runs, so no new issue is being opened (see Issue Creation criteria). Recommend the maintainer spot-check these two run IDs directly.

Schedule Heartbeat

The repo has ~210 schedule-triggered workflows (205 object-form on.schedule, 4 shorthand on: "daily" aliases, 1 cooldown-prefixed form) — far more than fit in an exhaustive per-workflow Actions API check this run. Only 53 appeared in the local 24h sample; the rest were not downloaded due to the storage/count budget, not necessarily silent. A 10-workflow stratified sample (mixing daily/weekly cadence, including categories similar to the historical #53252 "audit-workflows" blind-spot precedent) was checked live via the Actions API:

Workflow	Last observed run	Expected cadence	Gap
Safe Output Health Monitor	2026-10-13T04:31 UTC	daily	~19h — healthy
Organization Health Report	2026-10-12T08:47 UTC	weekly	~1d — healthy
Repository Quality Improvement Agent	2026-10-13T13:06 UTC	daily	~10h — healthy
Static Analysis Report	2026-10-13T04:54 UTC	daily	~18h — healthy
Stale PR Cleanup	2026-10-10T08:26 UTC	weekly	~3d — healthy
Weekly Issue Summary	2026-10-12T15:01 UTC	weekly	~1d — healthy
Daily Documentation Healer	2026-10-12T23:45 UTC	daily	~0.5d — healthy
Daily Malicious Code Scan Agent	2026-10-13T13:43 UTC	daily	~10h — healthy
Daily Workflow Updater	2026-10-13T07:13 UTC	daily	~16h — healthy
Weekly Network Domain Set Audit	2026-10-12T09:59 UTC	weekly	~1d — healthy

No blind spots detected in the sampled workflows. Full 210-workflow verification was not exhaustive given repo scale — this is the same scoping limitation noted in the 2026-10-09 and 2026-10-12 reports.

View per-workflow breakdown (failures only; full 53-workflow table omitted for brevity — all others had 0 failures)

Workflow	Runs seen	Agent failures	Failure rate
Daily Go Test Parallelizer	4	3	75%
Daily BYOK Ollama Test	1	1	100%
UK AI Operational Resilience	1	1	100%
Daily Security Observability Report	1	1	100%
Daily Reliability Review	1	1	100%
(48 other workflows)	—	0	0%

Also observed (out of scope, noted for context)

• 10 runs had an overall failure conclusion while the agent job itself reported success or skipped (e.g. Daily Spec Coverage Review, Agentic Workflow Audit Agent, PR Sous Chef, Daily Code Debt Cleanup — Aider) — these are safe-output/other-job failures, owned by the Safe Output Health Monitor.
• 16 runs (Design Decision Gate, Test Quality Sentinel, PR Code Quality Reviewer, Matt Pocock Skills Reviewer, Ponytail Reviewer — all PR-triggered, clustered in 3 bursts at 15:56, 15:57, and 18:13 UTC) failed with 0 jobs ever created per the Actions API, i.e. the run never reached job scheduling. This is the identical workflow set and signature recorded as "indeterminate" on 2026-10-09, reinforcing it's a recurring PR-branch-validation artifact (likely an invalid .lock.yml on the PR's own branch) rather than a fleet-wide agent regression. Excluded from the agent-job failure count.

Recommendations

1. No new issue opened this cycle — both novel single-workflow clusters (BYOK Ollama Warm up model, UK AI Operational Resilience governance-context step) are below the 2-workflow/10% threshold.
2. Maintainer should confirm whether issues tracking "Execute Codex CLI" ([aw-failures] [P0] Codex OpenAI invalid_project 401 auth failure — unfixed 3+ days, Daily Cache Strategy Analyzer failing every #54242/[aw] Daily Go Test Parallelizer engine driver failed before emitting a terminal safe output #65950) and "Execute GitHub Copilot CLI" failures are still open — this workflow could not verify live due to the integrity-policy block, which has now occurred on 3 consecutive runs and may itself be worth a quick manual check.
3. Consider widening the local log storage/count budget for this workflow — 3 of the last 4 reports (2026-10-09, 2026-10-12, today) covered well under 24h of actual window.

References:

• §38478313671 — Daily Go Test Parallelizer, Execute Codex CLI failure
• §38480674694 — Daily Reliability Review, Execute GitHub Copilot CLI failure
• §38477676843 — Daily BYOK Ollama Test, Warm up model failure (novel)

Generated by 🩺 Agent Job Health Monitor · claude · agent · 692.6 AIC · ⊞ 6.8K · ◷

• expires on Oct 16, 2026, 3:44 PM UTC-08:00