[adel-pplx]

Adds opt-in, model-backed decision rules for agent actions that need content or session context, without replacing deterministic rules or native hook enforcement. The feature remains experimental and explicitly configured.

What changes

• Third rule type: decision: uses the ordinary rule catalog, with per-rule selectors, instructions, choices and thresholds. One provider configuration supports the Perplexity and OpenRouter Decisions APIs, using either an environment variable or a protected token file.
• Action or context: classify a single call/result or include bounded local history. Partial context can produce advisory findings, but cannot authorize a contextual deny.
• Existing enforcement: deterministic and sequence rules run first; a clean deny skips further model calls. Up to two independent model requests share one deadline, with eligibility rechecked when each result is recorded. A clean model deny cancels outstanding work; hooks retain native enforcement gates, while scan and collect remain advisory.
• Audit records: each evaluated rule emits a structured assessment, including non-matches, abstentions, errors and skips. Assessments carry outcome and input references/digests, not complete inference requests; model-backed findings identify each evaluation separately.
• Collector isolation: optional inference runs in a best-effort background worker, not the export acknowledgment path. Queued/in-flight work is bounded to 64 events and 8 MiB of serialized evidence/metadata; work waiting over five seconds is skipped. Collector shutdown cancels work, and background output errors cannot be attributed to an unrelated export.
• Provider routing: an optional explicit HTTP/HTTPS CONNECT proxy supports managed egress. Ambient proxy variables remain ignored, proxy authentication is unsupported, and outages do not cause direct fallback or fail-closed blocking.
• Disabled starter: a monitor-only prompt-injection rule under examples/decision-rules/ can be enabled without site-specific policy or allowlists. It is outside the embedded catalog and reviews observed tool results, not whether an attack succeeded.

Operator impact

All records use schema 0.6.0, even without decision evaluation enabled. Update strict downstream validators before upgrading; assessments and their links to findings/enforcement are new, and case tools do not migrate earlier record schemas. Builds require Go 1.26.10 or later.

--decision-config explicitly enables sending projected tool evidence to the configured provider. Raw content requires a separate opt-in; redaction is not a data-loss-prevention guarantee. Shared sequence/history state must satisfy the stricter private-state checks when decision evaluation is configured.

Claude hook child identities now split timeline groups even without decision evaluation; payloads without agent_id retain their existing grouping. Contextual evidence gaps remain visible and prevent contextual blocking, not advisory findings.

Recommended use is selective, tested local enforcement plus a few narrowly selected model policies, initially monitored. Broader correlation belongs in the SIEM/data lake. Questions are not automatically batched because grouping can change model answers; concurrency reduces serial waiting, not request count or capacity requirements.

Try it

Build with go build ./cmd/numbat, then follow the decision rule quick start or rule-authoring guide. Start in monitor mode with synthetic data.

Synthetic checks exercise single-call and contextual read/upload flows, native response generation, slow-provider isolation, bounded scheduling and proxy routing over real HTTP. These are functional checks, not production accuracy claims; probability thresholds still need calibration.

Installed-host blocking and fleet behavior remain unverified. Cross-platform tests do not establish installed-host acceptance, and a logged deny request does not prove that the host blocked the action.

[homanp]

@adel-pplx a couple of suggestions on the provider config:

• Could we support reading the API token from an environment variable, such as PERPLEXITY_API_KEY, as an alternative to token_file? An explicit option like token_env: PERPLEXITY_API_KEY would let us reuse existing credentials without managing a separate token file.
• Could we expose headers as a config option, with environment variable substitution in header values? This would support custom authentication and additional headers when routing through OpenRouter or other gateways.

For example, the proposed header syntax could look like:

provider:
  endpoint: https://openrouter.ai/api/alpha/decisions
  model: perplexity/pplx-decider-v1.2
  headers:
    Authorization: "Bearer ${OPENROUTER_API_KEY}"
    HTTP-Referer: "https://example.com"
    X-Title: "My application"

If Authorization is explicitly configured, it should take precedence over the generated bearer header.

[ronheichman]

I think it would be better to pack questions. questions is a map, but the client always sends exactly one ("policy") and answers is rejected unless it contains only that key (L275-279). So N decision rules on the same event = N sequential requests, each re-sending the same state.

If rules with the same view were grouped into one request keyed by rule name, latency and payload would stay ~1x as rules are added instead of growing linearly inside the shared 1 s budget. Each questions still returns a probability, also.

[adel-pplx]

Keeping questions separate because provider docs say grouping can change answers. 7b6bc53 uses bounded concurrency instead; grouped policies need separate accuracy validation.

[ronheichman]

See: https://docs.perplexity.ai/docs/decisions/quickstart#grouping-questions-in-one-request:~:text=Grouping%20helps%20when,or%20duplicate%20detection.

Grouping helps when questions are facets of one decisions. Here evaluation is on one tool-call, so likely it would help. That being said, some benchmarking can resolve this empirically.

[adel-pplx]

Agreed that same-action policies are a plausible grouping case. Kept independent requests for this revision; the follow-up benchmark should use distinct policies and fixed groups, checking answer completeness, threshold crossings, latency, and tokens.

[ronheichman]

Selected decision rules still emit skipped assessments after a deterministic deny. If one optional assessment or finding write fails, global RecordErrors makes runHookEvent fail open and removes that clean deterministic deny. Can decision-record errors be isolated here as they are in the collector?

[adel-pplx]

I kept the documented clean-output gate: an undelivered assessment is actual audit loss, distinct from an inference failure, so it still suppresses a local deny. Added an explicit regression and clarified that distinction in 36d5ee1 rather than silently ignoring failed records.

[ronheichman]

After a sibling deny, this overwrites even a reply that already completed as prior_semantic_deny; delivery order can therefore erase a paid-for matched finding. Can completed replies be preserved and only unstarted or canceled work be marked skipped?

[adel-pplx]

Fixed in 36d5ee1. Cancelling remaining provider work no longer replaces completed replies or terminal errors; completed siblings retain the original deadline and freshness checks and cannot replace the winning deny.

[ronheichman]

This budgets one HTTP sink timeout, but --http-batch-size=1 flushes each assessment and finding synchronously. Multiple selected rules can spend several sink timeouts after inference and exceed the native-hook deadline. Can enforcement require one batch or budget every possible flush?

[adel-pplx]

Fixed in 36d5ee1: native HTTP delivery now shares one absolute nine-second deadline across flushes and Close, rather than budgeting one request. Also recheck the winning contextual decision after delivery, since session state can change during those writes.

[ronheichman]

advisoryErrors captures only immediate Observe failures. Later background assessment or finding writes increment RecordErrors but not this counter, so runCollect exits 1 on shutdown after an optional evaluator write failure. Can the background writer account for its own failures too?

[adel-pplx]

Kept shutdown exit 1 for actual audit-record loss; that is separate from accepting an OTLP export and from best-effort model execution. Clarified the distinction in 36d5ee1 rather than hiding background write failures.

[ronheichman]

lab.check is reassigned after the first invocation launches two provider requests. A canceled straggler can enter the handler afterward, racing this assignment and skewing lab.hits; the focused race test reproduced it. Can the checker be installed before the first call and filter the second command?

[adel-pplx]

Fixed in 36d5ee1: the checker is installed before any requests and filters/counts the second action by structured input, removing the callback reassignment and aggregate-hit baseline. Focused race regressions pass.