Skip to content

feat: correlate enclave handoff and VMM launch diagnostics - #10066

Merged
lpcox merged 2 commits into
mainfrom
lpcox-enclave-handoff-diagnostics
Oct 16, 2026
Merged

lpcox merged 2 commits into
mainfrom
lpcox-enclave-handoff-diagnostics

Conversation

@lpcox

@lpcox lpcox commented Oct 16, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Correlate protected host-control admission, registration, backend, settlement, and rejection events with validated invocation IDs and event sequence numbers.
  • Split the shared Cloud Hypervisor manager path into allowlisted launch milestones: preflight/recovery, identity, network (including script enclaves' empty network namespace), rootfs/staging/cgroup, VMM spawn, API readiness, confinement, virtiofsd, VM create, boot, and guest-supervisor connection.
  • Reuse the durable protected invocation snapshots and existing failed-shutdown export. Preserve the original failure independently from secondary partial-start cleanup/reconcile failures.
  • Document evidence interpretation and deployment requirements. No caller protocol/error, retry, isolation, artifact acceptance, release/pin, or workflow changes.

Validation

  • npm run build and npm run type-check pass.
  • Latest focused run: 8 suites / 436 tests pass, covering manager launch/connectivity/cleanup, launch evidence, progress catalog, broker correlation, diagnostic bounds, and protocol.
  • Additional focused run: 5 suites / 113 tests pass, including host backend, invocation persistence, and protected export during failed shutdown.
  • Changed-file ESLint: zero errors (security-rule warnings remain); git diff --check passes.
  • No VM/inference execution or workflow dispatch/rerun performed.

Activation limitations

These new host checks require an AWF host binary built with this instrumentation. A main checkout SHA does not update a release-pinned binary. The existing broker exchange audit requires its corresponding updated enclave MCP image; existing script guest milestones still require a rebuilt, release-attested guest rootfs. The new host diagnostics themselves need no guest-rootfs rebuild. A run pinned to v0.28.59 cannot emit these new diagnostics, and this PR does not claim a root cause for run 38842179031 or retroactively add evidence to it.

Preserve protected invocation correlation and shared launch milestones without changing caller errors or runtime policy.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 417ff0fc-9c06-4eb0-9f53-a998f99d7fd3
Copilot AI max review requested due to automatic review settings October 16, 2026 03:34
@github-actions

github-actions Bot commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

Documentation Preview

Documentation has been built for this PR.

Download preview artifact

To view locally:

  1. Download the docs-preview-pr-10066 artifact from the workflow run
  2. Unzip and open index.html in your browser

Built from commit bb81255

@github-actions

github-actions Bot commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

✅ Coverage Check Passed

Overall Coverage

Metric Base PR Delta
Lines 93.18% 93.21% 📈 +0.03%
Statements 91.76% 91.80% 📈 +0.04%
Functions 90.68% 90.73% 📈 +0.05%
Branches 85.62% 85.66% 📈 +0.04%
📁 Per-file Coverage Changes (8 files)
File Lines (Before → After) Statements (Before → After)
src/cloud-hypervisor/diagnostics.ts 88.4% → 88.0% (-0.44%) 86.9% → 86.9% (+0.00%)
src/cloud-hypervisor/host-preflight-progress.ts 100.0% → 100.0% (+0.00%) 96.2% → 96.3% (+0.09%)
src/enclave/host-executor-server.ts 90.8% → 90.8% (+0.00%) 89.5% → 89.2% (-0.30%)
src/cloud-hypervisor/host-enclave-executor.ts 94.6% → 94.6% (+0.02%) 91.0% → 91.0% (+0.03%)
src/cloud-hypervisor/manager.ts 90.2% → 90.6% (+0.46%) 88.8% → 89.4% (+0.63%)
src/cloud-hypervisor/manager-start.ts 86.3% → 88.5% (+2.22%) 85.8% → 89.9% (+4.03%)
src/log-directory-setup.ts 96.8% → 100.0% (+3.18%) 96.9% → 100.0% (+3.13%)
src/cloud-hypervisor/manager-stop.ts 92.5% → 95.7% (+3.22%) 86.8% → 90.9% (+4.13%)

Coverage comparison generated by scripts/ci/compare-coverage.ts

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Diagnostic records can misreport launch outcomes and lose or mix original failure details.

5 open findings
What changed in this PR

Adds protected, correlated diagnostics across AWF enclave handoffs and Cloud Hypervisor startup.

Changes:

  • Correlates host-control events with invocation IDs and sequence numbers.
  • Adds launch milestones and partial-start cleanup reporting.
  • Expands regression coverage and documents activation requirements.
File Description
src/​enclave/​host-executor-server.ts Attaches invocation IDs to control events.
src/​enclave/​host-executor-journal.ts Adds validated event identity and sequencing.
src/​enclave/​host-executor-invocation-diagnostics.test.ts Tests correlation, filtering, and bounds.
src/​enclave/​host-executor-broker.test.ts Checks handoff and rejection correlation.
src/​enclave/​cloud-hypervisor-lifecycle.test.ts Covers protected evidence export during failed shutdown.
src/​cloud-hypervisor/​manager.ts Instruments boot and guest connection.
src/​cloud-hypervisor/​manager-start.ts Records launch and cleanup milestones.
src/​cloud-hypervisor/​manager-launch-evidence.test.ts Adds launch-evidence regression cases.
src/​cloud-hypervisor/​host-preflight-schema.ts Defines launch checks.
src/​cloud-hypervisor/​host-enclave-executor.ts Passes the launch reporter into managers.
src/​cloud-hypervisor/​host-enclave-executor.test.ts Verifies reporter wiring.
src/​cloud-hypervisor/​enclave-executor-types.ts Extends the manager factory interface.
docs/​cloud-hypervisor-foundation.md Explains evidence interpretation and deployment requirements.

🧠 Review effort: Max


💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment on lines +140 to +151
(deps.rm as jest.Mock).mockImplementation(async () => {
expect(JSON.parse(readHostExecutorInvocationDiagnostics(run)[0].contents)).toMatchObject({
failure: { stage: 'vm-launch/vm-create', reason: 'EACCES' },
});
throw Object.assign(new Error('PRIVATE_CLEANUP'), { code: 'EIO' });
});
const manager = new CloudHypervisorManager(
config(), '/tmp/awf', deps, 'evidence', networkConfig(), undefined, undefined, false,
report.fork('vm-launch'),
);
await expect(diagnostic.check(report, 'manager-start', () => manager.start()))
.rejects.toThrow('partial-start cleanup also failed');
Comment thread src/cloud-hypervisor/manager-start.ts Outdated
Comment thread src/cloud-hypervisor/manager-start.ts Outdated
Comment on lines +293 to +294
const child = context.launchReport
? context.launchReport.checkSync('vmm-launch', launch) : launch();
Comment on lines +380 to +381
if (context.launchReport) {
await context.launchReport.fork('host-invocation-cleanup').check('manager-stop', () => context.stop());
try {
await context.stop();
if (context.launchReport) {
await context.launchReport.fork('host-invocation-cleanup').check('manager-stop', () => context.stop());
@lpcox

lpcox commented Oct 16, 2026

Copy link
Copy Markdown
Collaborator Author

@copilot adddress review feedback

Copilot AI commented Oct 16, 2026

Copy link
Copy Markdown
Contributor

@copilot adddress review feedback

Addressed in commit f575c35: launch milestones now reflect only executed checks, startup failures are recorded before cleanup, and first cleanup-error classification is retained across retries. Added regressions; focused tests, build, and type-check pass.

@github-actions

Copy link
Copy Markdown
Contributor

✅ Copilot review passed with no inline comments.

@lpcox Add the ready-for-aw label to this PR to trigger agentic CI smoke tests.

@lpcox
lpcox enabled auto-merge (squash) October 16, 2026 04:35
@github-actions

github-actions Bot commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

📡 Smoke OTel Tracing completed. All tracing scenarios validated. ✅

📡 OTel tracing validated by Smoke OTel Tracing

@github-actions

github-actions Bot commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

✅ Smoke Claude passed

Generated by Smoke Claude for #10066

@github-actions

github-actions Bot commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

✅ Smoke Copilot BYOK completed. Copilot BYOK mode operational. 🔓

🔑 BYOK report filed by Smoke Copilot BYOK

@github-actions

github-actions Bot commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

Smoke Cloud Hypervisor completed. Cloud Hypervisor + Copilot passed.

Cloud Hypervisor + Copilot smoke test by Smoke Cloud Hypervisor

@github-actions

Copy link
Copy Markdown
Contributor

🚀 Security Guard has started processing this pull request

@github-actions

github-actions Bot commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

Chroot tests passed! Smoke Chroot - All security and functionality tests succeeded.

Tested by Smoke Chroot

@github-actions

github-actions Bot commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

✅ Build Test Suite completed successfully!

Generated by Build Test Suite for #10066

@github-actions

Copy link
Copy Markdown
Contributor

💎 Smoke Gemini is crystallizing results on this pull request...

@github-actions

github-actions Bot commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

🛡️ Smoke Copilot Network Isolation confirmed the egress allowlist is enforced. ✅

🛡️ Egress verdict from Smoke Copilot Network Isolation

@github-actions

github-actions Bot commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

🔌 Smoke Services — All services reachable! ✅

🔌 Service connectivity validated by Smoke Services

@github-actions

Copy link
Copy Markdown
Contributor

🔑 Smoke Copilot BYOK AOAI (api-key) is testing Azure OpenAI BYOK (api-key) mode on this pull request...

@github-actions

github-actions Bot commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

✨ The prophecy is fulfilled... Smoke Codex has completed its mystical journey. The stars align. 🌟

🔮 The oracle has spoken through Smoke Codex

@github-actions

Copy link
Copy Markdown
Contributor

🪪 Smoke Copilot BYOK AOAI (Entra) is testing Azure OpenAI BYOK (Entra / GitHub OIDC) mode on this pull request...

@github-actions

github-actions Bot commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

📰 VERDICT: Smoke Copilot has concluded. All systems operational. This is a developing story. 🎤

📰 BREAKING: Report filed by Smoke Copilot

@github-actions

Copy link
Copy Markdown
Contributor

Smoke Test: Claude Engine Validation

Check Status
API ✅ PASS
GH CLI ✅ PASS
File ✅ PASS

Overall result: PASS

Generated by Smoke Claude for #10066 · claude · haiku45 · 49.2 AIC · ⊞ 6.1K · ◷
Add label ready-for-aw to run again

@github-actions

Copy link
Copy Markdown
Contributor

Services smoke test: ✅ Redis PONG · ✅ pg_isready accepting connections · ✅ SELECT 1 = 1. PASS

🔌 Service connectivity validated by Smoke Services
Add label ready-for-aw to run again

@github-actions

Copy link
Copy Markdown
Contributor

Cloud Hypervisor + Copilot smoke test

  1. Merged PR lookup: PASS
  2. github.com connectivity: PASS (HTTP 200)
  3. File write/read: PASS
  4. example.com blocked: PASS (HTTP 000)
    Overall: PASS

Cloud Hypervisor + Copilot smoke test by Smoke Cloud Hypervisor
Add label ready-for-aw to run again

@github-actions

Copy link
Copy Markdown
Contributor

🟢 Smoke Test: Copilot BYOK (Direct) Mode — PASS

Tests:

Running in direct BYOK mode via COPILOT_PROVIDER_API_KEY.

cc/ @lpcox

🔑 BYOK report filed by Smoke Copilot BYOK
Add label ready-for-aw to run again

@github-actions

Copy link
Copy Markdown
Contributor

EGRESS_RESULT allow=pass deny=pass

  • ✅ Allowed domain (api.github.com): HTTP 403 status returned (reachable)
  • ✅ Blocked domain (example.com): blocked (TLS/proxy denial)

Overall: PASS — cc @lpcox

🛡️ Egress verdict from Smoke Copilot Network Isolation
Add label ready-for-aw to run again

@github-actions github-actions Bot added the smoke-copilot-network-isolation Copilot network-isolation egress smoke test label Oct 16, 2026
@github-actions

Copy link
Copy Markdown
Contributor

OTEL smoke test

  • ✅ S1 Module loading: otel.js loads; exports startRequestSpan, setTokenAttributes, setBudgetAttributes, endSpan, endSpanError, shutdown, isEnabled (+ internals)
  • ✅ S2 Tests: 3 suites, 68/68 passed
  • ✅ S3 Env forwarding: OTEL vars present in env-passthrough.ts and api-proxy-env-config.ts
  • ✅ S4 Token tracker: onUsage hook present in token-tracker-http.js
  • ✅ S5 Diagnostics: otel.jsonl exists (1 line); no export errors observed

📡 OTel tracing validated by Smoke OTel Tracing
Add label ready-for-aw to run again

@github-actions

Copy link
Copy Markdown
Contributor

Smoke Copilot — PR: "feat: correlate enclave handoff and VMM launch diagnostics"

📰 BREAKING: Report filed by Smoke Copilot
Add label ready-for-aw to run again

@github-actions

Copy link
Copy Markdown
Contributor

Smoke Test

  • docs: define configurable, offline-first model information architecture ✅
  • chore(deps): bump the all-github-actions group with 3 updates ✅
  • PR detail read ✅
  • Playwright title check ❌
  • File write/read ✅
  • AWF build ✅
  • Overall: FAIL

🔮 The oracle has spoken through Smoke Codex
Add label ready-for-aw to run again

@github-actions

Copy link
Copy Markdown
Contributor

Chroot version comparison

Runtime Host Version Chroot Version Match?
Python 3.12.15 3.12.15 ✅
Node.js v24.21.0 v22.23.2 ❌
Go go1.22.12 go1.22.12 ✅

Not all tests passed: the Node.js versions differ, so the smoke-chroot label was not added.

Tested by Smoke Chroot
Add label ready-for-aw to run again

@github-actions

Copy link
Copy Markdown
Contributor

🏗️ Build Test Suite Results

Ecosystem Project Build/Install Tests Status
Bun elysia ❌ not run ❌ FAIL
Bun hono ❌ not run ❌ FAIL
C++ fmt ✅ N/A ✅ PASS
C++ json ✅ N/A ✅ PASS
Deno oak ❌ not run ❌ FAIL
Deno std ❌ not run ❌ FAIL
.NET hello-world ✅ N/A ✅ PASS
.NET json-parse ✅ N/A ✅ PASS
Go color ✅ 1/1 passed ✅ PASS
Go env ✅ 1/1 passed ✅ PASS
Go uuid ✅ 1/1 passed ✅ PASS
Java gson ✅ 1/1 passed ✅ PASS
Java caffeine ✅ 1/1 passed ✅ PASS
Node.js clsx ✅ all passed ✅ PASS
Node.js execa ✅ all passed ✅ PASS
Node.js p-limit ✅ all passed ✅ PASS
Rust fd ✅ 1/1 passed ✅ PASS
Rust zoxide ✅ 1/1 passed ✅ PASS

Overall: 6/8 ecosystems passed — FAIL

Failures:

  • Bun / Deno: the curl | bash / curl | sh installer commands (bun.sh, deno.land) were denied by the sandbox permission policy ("Permission denied because no interactive user response was available"), so neither runtime was installed and no tests ran. All repos cloned successfully.
  • Note: .NET and Maven needed relocated cache/runtime dirs (default $HOME/.m2 and /run/user/1001 were not accessible); both then succeeded.

The build-test label was not added because not all ecosystems passed.

Generated by Build Test Suite for #10066 · copilot · auto · 25.7 AIC · ⊞ 12K · ◷
Add label ready-for-aw to run again

@lpcox
lpcox merged commit 50d5c21 into main Oct 16, 2026
145 of 151 checks passed
@lpcox
lpcox deleted the lpcox-enclave-handoff-diagnostics branch October 16, 2026 04:55

This branch had an error being deployed

1 failed deployment
aoai-model — f575c351 Deployed Oct 16, 2026 by lpcox via conclusion #1974
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants