Facility Management AgentOps — Enterprise Thin Slice

Enterprise AgentOps on GCP: an end-to-end production-level agentic application with orchestration, tool use, state, tracing, metrics eval, release gating, and Cloud Run deployment.

Floor 3 HVAC keeps recurring → investigate → policy → work order → audit → structured JSON — with traces and a fail-closed gate that can block a bad agent version.

Built with Google ADK + Vertex AI (Gemini). Phase 1 is a closed thin slice: serving + measurement + release discipline — not a chatbot prototype.

---

What makes this more than an agent demo?

1. Eval + metrics — 5 golden cases, rule judge + LLM-as-a-judge, scored metrics (tool_call_accuracy, task_success, groundedness, …)
2. Tracing / observability — every request writes JSONL spans (agent turns + tool calls/results)
3. Regression / release gate — YAML SLOs; exit 0 = PASS, exit 1 = BLOCK ship (week3_4_regression.sh proves it)
4. Deployment — same investigate path on Docker and Cloud Run (CI/CD-ready exit-code gate)

Also in V1: session + case memory, context engineering (policy + similar cases), online guardrails (auditor), harness retries.

Samples (no Vertex needed): docs/samples/

---

System architecture

What you score is what you deploy — API, CLI, and eval share one investigate pipeline:

flowchart LR
  subgraph Runtime["Enterprise MAS — Google ADK on GCP"]
    U[Client / API] --> GW[FastAPI<br/>POST /v1/investigate]
    GW --> ROOT[RootAgent<br/>SequentialAgent]
    ROOT --> AGENTS[Specialist agents + MCP-ready tools]
    AGENTS --> MEM[(Context: session + cases)]
    AGENTS --> TRACE[(Traces / observability)]
    AGENTS --> GUARD[Auditor guardrail]
  end

  subgraph Quality["Harness — metrics eval — regression gate"]
    GOLD[Golden dataset] --> HARNESS[Eval harness]
    TRACE --> HARNESS
    HARNESS --> RULE[Rule metrics]
    HARNESS --> LLM[LLM-as-a-judge + retry]
    RULE --> METRICS[Metrics scorer]
    LLM --> METRICS
    METRICS --> GATE{Release gate}
    GATE -->|PASS| CR[Cloud Run]
    GATE -->|BLOCK| STOP[Hold release]
  end

  AGENTS -.-> HARNESS

Loading

Multi-agent investigate pipeline

flowchart TD
  Q["Query: Floor 3 HVAC keeps recurring…"] --> R[RootAgent]

  R --> A[AlarmAgent]
  A --> A1["search_alarm_history"]
  A --> A2["search_similar_cases"]

  R --> K[KBAgent]
  K --> K1["search_policy_doc"]

  R --> W[WorkOrderAgent]
  W --> W1["recommend_work_order"]

  R --> AU[AuditorAgent]
  AU --> AU1["audit_recommendation"]

  AU --> OUT["Structured response<br/>WO action · priority · auditor verdict"]
  R --> T["Observability → eval_harness/reports/traces/"]
  R --> S["Sessions → data/memory/sessions/"]

Loading

Phase 1 uses a fixed SequentialAgent pipeline (Router/Planner in Phase 2) so the thin slice stays closed, measurable, and shippable.

Artifacts: docs/samples/ — sample trace + sample eval report.

---

Enterprise harness engineering, metrics eval & regression gate

Enterprise agent versions do not ship on vibe checks. The eval harness runs golden cases, computes metrics, and the regression / release gate fails closed unless aggregate scores clear criteria.yaml.

Metrics evaluated

Layer	Metrics	Source
Rule-based	tool_call_accuracy, agent_coverage, answer_keyword_score, must_not_violations, latency_seconds	Deterministic checks on traces
LLM-as-a-judge	task_success, groundedness, plus tool-use / risk / clarity	Vertex Gemini judge (+ retry)
Aggregate report	pass_rate, avg_* rollups, critical_failures	eval_harness/metrics/scorer.py
Release gate SLOs	Thresholds on the aggregates above	eval_harness/release_gate/criteria.yaml

sequenceDiagram
  participant CICDish as CI / eval job
  participant Harness as Eval harness
  participant Metrics as Metrics scorer
  participant Gate as Regression gate
  participant Run as GCP Cloud Run

  CICDish->>Harness: Golden cases (rule + LLM judges, with retry)
  Harness->>Metrics: Per-case scores
  Metrics->>Gate: eval_report.json (aggregates)
  Note over Metrics: tool_call_accuracy · task_success · groundedness · …

  alt Meets release criteria
    Gate-->>Run: PASS (exit 0) → deploy
  else Prompt regression / metric miss / policy violation
    Gate-->>CICDish: BLOCK (exit 1) → hold release
  end

  Note over CICDish,Gate: week3_4_regression.sh proves BLOCK then PASS

Loading

• Gate thresholds: eval_harness/release_gate/criteria.yaml — e.g. pass_rate >= 0.85, avg_tool_call_accuracy >= 0.90, avg_task_success >= 0.85, avg_groundedness >= 0.90, critical_failures == 0
• Judge retry: EVAL_JUDGE_MAX_RETRIES / EVAL_JUDGE_RETRY_DELAY_SECONDS (see .env.example)
• Sample scored report: docs/samples/sample_eval_report.json
• CI/CD path (V1 → V2): metrics + gate are already an exit-code contract; Phase 2 plugs them into Cloud Build so push → eval → metrics → gate → deploy

---

Quick start

# 1) Agentic investigate (synthetic DemoCorp / Toronto mock data)
uv run python scripts/run_investigate_cli.py \
  "The HVAC alarm on Floor 3 keeps recurring. Check the issue, search policy, decide if we should create a work order, and recommend priority."

# or: ./scripts/run_demo.sh

# 2) Enterprise API
uv run fm-agentops-api
curl -s -X POST http://localhost:8080/v1/investigate \
  -H "Content-Type: application/json" \
  -d '{"query": "The HVAC alarm on Floor 3 keeps recurring. Check the issue, search policy, decide if we should create a work order, and recommend priority."}' \
  | jq .

# 3) Harness + release gate
./scripts/run_eval.sh --case-id alarm_001 --skip-llm-judge   # fast
./scripts/run_eval.sh                                         # full (~5–10 min)

# 4) Regression gate demo
./scripts/week3_4_regression.sh

---

Setup (GCP)

Requirements: Python 3.11+, uv, gcloud CLI, GCP project with Vertex AI enabled.

cp .env.example .env
# set GOOGLE_CLOUD_PROJECT=

gcloud auth application-default login
gcloud services enable aiplatform.googleapis.com --project=$GOOGLE_CLOUD_PROJECT

uv sync --extra dev
uv run pytest -q

Docker

./scripts/docker_run_local.sh

Google Cloud Run

# Runtime service account needs roles/aiplatform.user
./scripts/deploy.sh

---

Repository layout

Path	Role
api/	FastAPI gateway (POST /v1/investigate)
agents/	Alarm, KB, WorkOrder, Auditor, Root
tools/	In-process tool registry (MCP-shaped surface)
services/	Shared investigate pipeline (API / CLI / harness)
memory/	Session + long-term case store
observability/	Production-style JSONL tracing
eval_harness/	Harness engineering: golden cases, judges, metrics scorer, release gate
data/mock/	Synthetic FM data (DemoCorp / Toronto)
deployment/	Dockerfile + Cloud Build (CI/CD foundation)
docs/samples/	Sample trace + eval report
scripts/	CLI, eval, deploy, regression gate, secret check

---

Platform roadmap

Phase	Question
V1 v1.0-thin-slice (this)	Can it work — harness, metrics eval, observability, regression gate, GCP deploy?
V2 platform reliability	Can I trust it — full MCP, A2A, Cloud Build CI/CD, vector memory, OTel / Cloud Trace?
V3+	Can it reason about evidence sufficiency and execution state?

Optional deep dive (not the homepage): plan_phase1.md · doc/plan.md · 2026 term map

---

2026 stack alignment (optional)

Honest map — no vapor features claimed as done:

Topic	V1?	Notes
AgentOps / LLMOps, harness & context engineering	✅	Shared investigate path + memory + eval harness
MAS, Google ADK, Vertex / Gemini	✅	4 specialists + Root SequentialAgent
LLM-as-a-judge, metrics eval, release gate	✅	Rule + LLM judges → metrics/scorer.py → gate
Trace → eval loop, guardrails	✅	JSONL traces; auditor agent
MCP	🟡	MCP-shaped registry now; full servers in V2
A2A, OTel / Cloud Trace, Agent Skills, debate eval	🔜	V2 roadmap

---

Security

• No API keys in the repo. Vertex AI on GCP + Application Default Credentials (ADC).
• Local config: copy .env.example → .env (gitignored). Never commit credentials or SA JSON.
• Before push: ./scripts/check_secrets.sh
• Full policy: SECURITY.md

Notes

• Building data is synthetic (DemoCorp, BLD-001, fictional WO/policy IDs) for safe local/dev runs.
• Runtime sessions/traces are gitignored; sample outputs live in docs/samples/.
• Implementation notes: docs/phase1_notes.md