Repository navigation
feat(conclude): emit verbose diagnostics at parity with gh-aw inline step - #749
Merged
Merged
Conversation
…step `threat-detect conclude` previously emitted at most a single line, making detection triage on the external path require downloading artifacts. It now mirrors gh-aw's parse_threat_detection_results.cjs job-log output: - Opening/closing banners framing the conclusion section - Echo of RUN_DETECTION, continue-on-error, execution outcome, and the resolved result-file / detection-log paths - Per-field verdict breakdown on both the safe and threat paths, with an indexed reasons list (or "(none)") - Recursive listing of the result directory plus detection-log stats and echoed THREAT_DETECTION_* marker lines when the result is missing or unusable (bounded so job logs stay readable) - New --detection-log and --log-file flags; the conclusion, verdict, and diagnostics are mirrored into the JSONL run log The detection log remains diagnostic-only and is never parsed for a verdict. Spec (TD-20c, U-18a) and README updated. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: David Slater <12449447+davidslater@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Adds verbose diagnostics to threat-detect conclude, aligning the released-binary path with gh-aw’s inline conclusion reporting.
Changes:
- Adds banners, input/path reporting, verdict breakdowns, and bounded failure diagnostics.
- Adds JSONL conclusion events and detection-log overrides.
- Updates specifications, documentation, and tests.
Show a summary per file
| File | Description |
|---|---|
cmd/threat-detect/conclude.go |
Implements diagnostic and JSONL reporting. |
cmd/threat-detect/conclude_test.go |
Tests conclusion diagnostics and logging. |
README.md |
Documents output and flags. |
specs/threat-detection-spec.md |
Defines diagnostic requirements. |
specs/usage-spec.md |
Defines host integration behavior. |
Review details
Tip
Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- Files reviewed: 5/5 changed files
- Comments generated: 5
- Review effort level: Balanced
| // not change the conclusion, so it degrades to a warning on stderr. | ||
| var logger *runlog.Logger | ||
| if logFile != "" { | ||
| opened, err := runlog.Open(logFile) |
Comment on lines
+90
to
+92
| if err != nil { | ||
| fmt.Fprintf(os.Stderr, "conclude: failed to open log file: %v\n", err) | ||
| } else { |
| if len(result.Reasons) > 0 { | ||
| c.info(fmt.Sprintf(" reasons (%d):", len(result.Reasons))) | ||
| for i, reason := range result.Reasons { | ||
| c.info(fmt.Sprintf(" [%d] %s", i+1, reason)) |
| } | ||
| c.info(fmt.Sprintf(" Found %d file(s):", len(files))) | ||
| for _, file := range shown { | ||
| c.info(" - " + file) |
Comment on lines
+329
to
+330
| lines := strings.Split(string(data), "\n") | ||
| c.info(fmt.Sprintf("📊 Detection log stats: %d lines, %d bytes", len(lines), len(data))) |
Addresses review feedback on #749: - Reject --log-file colliding with --result-file or the resolved detection log. runlog.Open truncates, so a collision erased the verdict (producing a bogus parse_error) or the failure diagnostics. Uses the existing samePath helper and checks the default sibling detection-log path too. - Treat an unopenable --log-file as a configuration error (TD-20a) instead of silently concluding without the required JSONL mirroring. - Escape control characters in untrusted values echoed to the job log (model-authored reasons, artifact filenames, detection-log lines). An embedded newline previously let a reason such as "text\n::add-mask::x" emit a live GitHub Actions workflow command; regression tests confirm this. - Report the detection log's true byte size and explicitly label truncation rather than presenting the scanned 8 MiB prefix as the whole file. Spec TD-20c and README updated to require truncation labelling and escaping. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: David Slater <12449447+davidslater@users.noreply.github.com>
main gained an overlapping `--detection-log` flag and defaultDetectionLogName (#737), plus detectionFailureReason/lastDetectionStatusReason which map the run's terminal THREAT_DETECTION_STATUS: reason onto agent_failure/parse_error. Resolution keeps both features and makes them reinforce each other: - Single defaultDetectionLogName and one --detection-log flag, whose help text now covers both the reason refinement and the diagnostics. - The missing/unreadable-result path emits the directory listing and detection-log diagnostics, then defers to main's detectionFailureReason for the reason and error code, so the log output shows exactly which status line drove the verdict. - conclude() resolves the detection-log path once into c.detectionLog so the diagnostics and detectionFailureReason can never disagree, including for a concluder constructed directly in tests. - --log-file collision checks cover the resolved detection-log path. Docs de-duplicated: the README --detection-log bullet no longer contradicts main's reason table, TD-20c now says the log must not derive the *verdict* (TD-20b legitimately derives the *reason* from it), and a dropped blank line before the U-19 table is restored. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: David Slater <12449447+davidslater@users.noreply.github.com>
main's #747 introduced a TD-20c (append the artifact inventory to GITHUB_STEP_SUMMARY) while this branch had already defined TD-20c for the conclude diagnostics — two different normative requirements under one ID. main's TD-20c keeps the ID since it is already published; the conclude diagnostics requirement is renumbered to TD-20d, and the U-18a cross-reference in specs/usage-spec.md is updated to match. The unrelated TD-20c reference at U-09 (step-summary inventory) is intentionally left pointing at TD-20c. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: David Slater <12449447+davidslater@users.noreply.github.com>
davidslater
added a commit
that referenced
this pull request
Aug 4, 2026
Merge origin/main into the step-summary branch (PR #749 landed: verbose conclude diagnostics parity with gh-aw's inline step). - README.md: kept both the --step-summary verdict-block bullet and main's verbose job-log diagnostics description. - conclude.go: combined the --step-summary flag, rejectPathCollisions validation, and writeVerdictSummary calls (mine) with the --log-file/runlog.Logger wiring, samePath collision checks, and logger.Info/Error event emission (main's) at every conflicting call site (flag parsing, concluder construction, struct fields, skipped/ success/warning/failure paths). - conclude_test.go: kept all tests from both branches (diagnostic output, safe/skipped verdict breakdown, run-log, log collision, and step-summary collision tests). - threat-detection-spec.md: renumbered the step-summary requirements to TD-20g/TD-20h to avoid colliding with main's newly landed TD-20d (verbose diagnostic output requirement). make build/vet/fmt/test all pass after the merge. Co-authored-by: David Slater <12449447+davidslater@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #692.
Problem
threat-detect concludeemitted at most a single line (✅ No security threats detected...) plus a bare::error::/::warning::. gh-aw's inlineparse_threat_detection_results.cjsis deliberately verbose, so debugging detection on the external (released-binary) path required downloading the artifact.Changes
cmd/threat-detect/conclude.gonow writes a self-contained diagnostic section:RUN_DETECTION,continue-on-error,DETECTION_AGENTIC_EXECUTION_OUTCOME, the detection directory, and the resolved detection-log and structured-result paths.prompt_injection/secret_leak/malicious_patch) on both the safe and threat paths, with an indexed reasons list ([1] …) orreasons : (none).THREAT_DETECTION_STATUS:/THREAT_DETECTION_RESULT:marker, on the failure paths.--log-fileflag (envTHREAT_DETECTION_LOG_FILE):conclude_start,conclude_verdict,conclude_directory_listing,conclude_detection_log,conclude_outcome.--detection-log <path>flag; defaults todetection.logbeside the result file.Diagnostic output is bounded (200 files, 50 marker lines, 500 runes/line, 8 MiB log read) so a pathological run can't flood the job log. The detection log remains diagnostic-only — it is never parsed for a verdict, preserving the "no log-scraping fallback" contract.
Contract
No change to the job-output contract, reason categories, exit codes, or warn/strict semantics — all existing
TestConcludeContractcases pass unchanged.Spec updated: TD-20c (
specs/threat-detection-spec.md) and U-18a (specs/usage-spec.md); READMEconcludesection documents the new flags and output.Upstream follow-up
gh-aw's
actions/setup/sh/conclude_threat_detection.shcurrently short-circuits theRUN_DETECTION != "true"and missing-result-file cases in bash, exiting beforethreat-detect concluderuns — so the new failure-path diagnostics won't surface on thefeatures: gh-aw-detection: truepath until that wrapper delegates to the detector. Tracked as items 5–8 in #745.This is not a blocker: the happy path (the only one the wrapper reaches today) already gains the banner, echoed inputs, and verdict breakdown, and nothing here regresses current behavior. The detector's default detection-log path was verified byte-identical to
constants.ThreatDetectionLogPath, so no upstream change is required for the default case.Testing
make fmt lint build testgreen. Seven new tests cover the banners, both verdict-breakdown paths, the skip path, the missing-result diagnostics, a missing detection log, the--detection-logoverride, and the JSONL records.