metrics: add metrics to count job minutes

README.MD

@@ -135,6 +135,10 @@ See [charts/cattery/values.yaml](charts/cattery/values.yaml) for the available v
 
 See [docs/configuration.md](docs/configuration.md) for the full configuration reference covering server, database, stale tray cleanup, leader election, GitHub App, providers, and tray types.
 
+## Metrics
+
+See [docs/metrics.md](docs/metrics.md) for the metrics exposed on `/metrics`, including per-repository runner consumption.
+
 ## License
 
 [MIT](LICENSE)

docs/configuration.md

@@ -93,7 +93,7 @@ trayTypes:
 | Key                  | Type   | Required | Description                                                                                                              |
 |----------------------|--------|----------|--------------------------------------------------------------------------------------------------------------------------|
 | listenAddress        | string | yes      | Host:port for the HTTP server to bind (e.g., 0.0.0.0:5137).                                                             |
-| statusListenAddress  | string | no       | Separate host:port for the /status and /metrics endpoints. If empty or equal to listenAddress, served on the agent port. |
+| statusListenAddress  | string | no       | Separate host:port for the /status and /metrics endpoints. If empty or equal to listenAddress, served on the agent port. See [metrics.md](metrics.md). |
 | advertiseUrl         | string | yes      | Public base URL where the server is reachable. Passed to agents.                                                         |
 | agentSecret          | string | no       | Bearer token the server requires on every `/agent/*` request. **Leave empty:** the agent does not send this token yet, so setting it rejects every agent registration. |
 

docs/metrics.md

@@ -0,0 +1,90 @@
+# Metrics
+
+Cattery exposes Prometheus metrics on `/metrics`, alongside `/status`. By
+default both are served on the agent port; set `server.statusListenAddress` to
+move them to a separate port (see [configuration.md](configuration.md)). The
+Helm chart ships a ServiceMonitor that scrapes this endpoint.
+
+The standard Go process and runtime collectors are exposed too. Everything
+below is Cattery's own.
+
+## Labels
+
+| Label | Meaning |
+|-------|---------|
+| `org` | GitHub organisation the tray type belongs to |
+| `tray_type` | Tray type name from the config |
+| `repository` | Bare repository name, without the owner — `org` already carries that |
+| `provider` | Provider name (`google`, `docker`, `nomad`, …) |
+
+A tray that has not been assigned a job yet has no repository, and appears
+under an empty `repository` label rather than being dropped. It still occupies
+a runner, so it still counts.
+
+## Counters
+
+| Metric | Labels | Meaning |
+|--------|--------|---------|
+| `cattery_job_seconds_total` | `org`, `tray_type`, `repository` | Runner seconds consumed by finished jobs |
+| `cattery_jobs_total` | `org`, `tray_type`, `repository` | Number of finished jobs |
+| `cattery_stale_trays_count` | `org`, `tray_type` | Trays cleaned up by the stale handler |
+| `cattery_preempted_trays_count` | `org`, `tray_type` | Trays lost to preemption |
+| `cattery_tray_provider_errors` | `org`, `provider`, `tray_type`, `operation_type` | Provider errors during tray `create` / `delete` |
+| `cattery_scaleset_poll_errors` | `org`, `tray_type` | Errors handling scale set messages |
+
+`cattery_job_seconds_total` and `cattery_jobs_total` are recorded in
+`TrayManager.DeleteTray`, which every teardown path funnels through — job
+completed, agent unregister on preemption or SigTerm, the stale reaper, and
+creation-failure cleanup. Preempted and lost jobs are therefore counted, not
+just cleanly finished ones: they consumed runner time regardless.
+
+Each job is counted exactly once. The tray's `jobStartedAt` is read and cleared
+atomically, so a tray whose provider cleanup failed and is later retried by the
+stale handler is not billed twice. Trays that never ran a job have no start
+time and are not counted at all.
+
+## Gauges
+
+| Metric | Labels | Meaning |
+|--------|--------|---------|
+| `cattery_registered_trays` | `org`, `tray_type`, `repository` | Trays currently registered (any status but `deleting`) |
+| `cattery_scaleset_pending_jobs` | `org`, `tray_type` | Available (queued) jobs |
+| `cattery_scaleset_assigned_jobs` | `org`, `tray_type` | Jobs assigned to runners |
+| `cattery_scaleset_running_jobs` | `org`, `tray_type` | Currently running jobs |
+| `cattery_scaleset_busy_runners` | `org`, `tray_type` | Busy runners |
+| `cattery_scaleset_idle_runners` | `org`, `tray_type` | Idle runners |
+| `cattery_scaleset_registered_runners` | `org`, `tray_type` | Registered runners |
+
+`cattery_registered_trays` is computed from the database on each scrape. If the
+query fails the collector logs and emits nothing for that scrape, so the series
+goes stale rather than reporting a false zero.
+
+The `cattery_scaleset_*` gauges come from GitHub's scale set statistics, which
+are reported per scale set. GitHub does not break them down by repository, so
+these cannot carry a `repository` label.
+
+## Per-repository usage
+
+```promql
+# Runner hours per repository over the last 7 days.
+sum by (repository) (increase(cattery_job_seconds_total[7d])) / 3600
+
+# Biggest movers: this week's consumption over last week's.
+sum by (repository) (increase(cattery_job_seconds_total[7d]))
+  / sum by (repository) (increase(cattery_job_seconds_total[7d] offset 7d))
+
+# Is a repository running more jobs, or slower ones?
+sum by (repository) (increase(cattery_job_seconds_total[7d]))
+  / sum by (repository) (increase(cattery_jobs_total[7d]))
+
+# Runners held right now.
+sum by (repository) (cattery_registered_trays)
+```
+
+### Cardinality
+
+The two per-repository counters keep a series for every
+`org` × `tray_type` × `repository` combination ever seen, including repositories
+that have since gone quiet. The gauge only has series for live trays. If this
+becomes a problem, drop or bucket repositories with `metric_relabel_configs` in
+the ServiceMonitor — no Cattery change or redeploy needed.

src/go.mod

@@ -70,6 +70,7 @@ require (
 	github.com/inconshreveable/mousetrap v1.1.0 // indirect
 	github.com/json-iterator/go v1.1.12 // indirect
 	github.com/klauspost/compress v1.19.1 // indirect
+	github.com/kylelemons/godebug v1.1.0 // indirect
 	github.com/leodido/go-urn v1.5.0 // indirect
 	github.com/mitchellh/go-homedir v1.1.0 // indirect
 	github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect

src/lib/metrics/metrics.go

@@ -38,6 +38,16 @@ var (
 		Help: "Number of errors while handling scale set messages (job started/completed, scaling)",
 	}, []string{"org", "tray_type"})
 
+	jobSecondsTotal = promauto.NewCounterVec(prometheus.CounterOpts{
+		Name: "cattery_job_seconds_total",
+		Help: "Runner seconds consumed by finished jobs, by repository",
+	}, []string{"org", "tray_type", "repository"})
+
+	jobsTotal = promauto.NewCounterVec(prometheus.CounterOpts{
+		Name: "cattery_jobs_total",
+		Help: "Number of finished jobs, by repository",
+	}, []string{"org", "tray_type", "repository"})
+
 	// Gauges
 
 	scaleSetPendingJobs = promauto.NewGaugeVec(prometheus.GaugeOpts{
@@ -73,7 +83,7 @@ var (
 	registeredTraysDesc = prometheus.NewDesc(
 		"cattery_registered_trays",
 		"Number of currently registered trays",
-		[]string{"org", "tray_type"}, nil,
+		[]string{"org", "tray_type", "repository"}, nil,
 	)
 )
 
@@ -133,6 +143,14 @@ func ScaleSetRegisteredRunnersSet(org string, trayType string, count int) {
 	scaleSetRegisteredRunners.WithLabelValues(org, trayType).Set(float64(count))
 }
 
+// JobFinished records one finished job's runner time against its repository.
+// Called once per tray from TrayManager.DeleteTray, which every teardown path
+// funnels through, so preempted and stale-reaped jobs are counted too.
+func JobFinished(org string, trayType string, repository string, seconds float64) {
+	jobSecondsTotal.WithLabelValues(org, trayType, repository).Add(seconds)
+	jobsTotal.WithLabelValues(org, trayType, repository).Inc()
+}
+
 // trayCollector queries the database on each Prometheus scrape.
 type trayCollector struct {
 	lister TrayLister
@@ -152,15 +170,18 @@ func (c *trayCollector) Collect(ch chan<- prometheus.Metric) {
 		return
 	}
 
-	counts := make(map[[2]string]int)
+	// A tray that has not been assigned a job yet has no repository; it is
+	// reported under the empty label rather than dropped, since it still
+	// occupies a runner.
+	counts := make(map[[3]string]int)
 	for _, t := range allTrays {
 		if t.Status != trays.TrayStatusDeleting {
-			counts[[2]string{t.GitHubOrgName, t.TrayTypeName}]++
+			counts[[3]string{t.GitHubOrgName, t.TrayTypeName, t.Repository}]++
 		}
 	}
 
 	for key, count := range counts {
-		ch <- prometheus.MustNewConstMetric(registeredTraysDesc, prometheus.GaugeValue, float64(count), key[0], key[1])
+		ch <- prometheus.MustNewConstMetric(registeredTraysDesc, prometheus.GaugeValue, float64(count), key[0], key[1], key[2])
 	}
 }
 

src/lib/metrics/metrics_test.go

@@ -0,0 +1,87 @@
+package metrics
+
+import (
+	"cattery/lib/trays"
+	"context"
+	"strings"
+	"testing"
+
+	"github.com/prometheus/client_golang/prometheus/testutil"
+	"github.com/stretchr/testify/assert"
+)
+
+type stubLister struct {
+	trays []*trays.Tray
+	err   error
+}
+
+func (s *stubLister) ListTrays(_ context.Context) ([]*trays.Tray, error) {
+	return s.trays, s.err
+}
+
+func TestJobFinished_RecordsSecondsAndCount(t *testing.T) {
+	JobFinished("test-org", "test-type", "test-repo", 12.5)
+
+	assert.Equal(t, 12.5, testutil.ToFloat64(
+		jobSecondsTotal.WithLabelValues("test-org", "test-type", "test-repo")))
+	assert.Equal(t, 1.0, testutil.ToFloat64(
+		jobsTotal.WithLabelValues("test-org", "test-type", "test-repo")))
+}
+
+func TestJobFinished_AccumulatesPerRepository(t *testing.T) {
+	JobFinished("acc-org", "acc-type", "repo-a", 10)
+	JobFinished("acc-org", "acc-type", "repo-a", 5)
+	JobFinished("acc-org", "acc-type", "repo-b", 3)
+
+	assert.Equal(t, 15.0, testutil.ToFloat64(
+		jobSecondsTotal.WithLabelValues("acc-org", "acc-type", "repo-a")))
+	assert.Equal(t, 2.0, testutil.ToFloat64(
+		jobsTotal.WithLabelValues("acc-org", "acc-type", "repo-a")))
+	assert.Equal(t, 3.0, testutil.ToFloat64(
+		jobSecondsTotal.WithLabelValues("acc-org", "acc-type", "repo-b")))
+}
+
+func TestTrayCollector_GroupsByRepository(t *testing.T) {
+	c := &trayCollector{lister: &stubLister{trays: []*trays.Tray{
+		{GitHubOrgName: "org", TrayTypeName: "big", Repository: "repo-a", Status: trays.TrayStatusRunning},
+		{GitHubOrgName: "org", TrayTypeName: "big", Repository: "repo-a", Status: trays.TrayStatusRunning},
+		{GitHubOrgName: "org", TrayTypeName: "big", Repository: "repo-b", Status: trays.TrayStatusRunning},
+	}}}
+
+	expected := `
+# HELP cattery_registered_trays Number of currently registered trays
+# TYPE cattery_registered_trays gauge
+cattery_registered_trays{org="org",repository="repo-a",tray_type="big"} 2
+cattery_registered_trays{org="org",repository="repo-b",tray_type="big"} 1
+`
+	assert.NoError(t, testutil.CollectAndCompare(c, strings.NewReader(expected)))
+}
+
+func TestTrayCollector_UnassignedTrayReportsEmptyRepository(t *testing.T) {
+	// A tray that has not been given a job yet has no repository. It still
+	// occupies a runner, so it must be reported rather than dropped.
+	c := &trayCollector{lister: &stubLister{trays: []*trays.Tray{
+		{GitHubOrgName: "org", TrayTypeName: "big", Repository: "", Status: trays.TrayStatusRegistered},
+	}}}
+
+	expected := `
+# HELP cattery_registered_trays Number of currently registered trays
+# TYPE cattery_registered_trays gauge
+cattery_registered_trays{org="org",repository="",tray_type="big"} 1
+`
+	assert.NoError(t, testutil.CollectAndCompare(c, strings.NewReader(expected)))
+}
+
+func TestTrayCollector_ExcludesDeletingTrays(t *testing.T) {
+	c := &trayCollector{lister: &stubLister{trays: []*trays.Tray{
+		{GitHubOrgName: "org", TrayTypeName: "big", Repository: "repo-a", Status: trays.TrayStatusRunning},
+		{GitHubOrgName: "org", TrayTypeName: "big", Repository: "repo-a", Status: trays.TrayStatusDeleting},
+	}}}
+
+	expected := `
+# HELP cattery_registered_trays Number of currently registered trays
+# TYPE cattery_registered_trays gauge
+cattery_registered_trays{org="org",repository="repo-a",tray_type="big"} 1
+`
+	assert.NoError(t, testutil.CollectAndCompare(c, strings.NewReader(expected)))
+}

src/lib/testutil/mock_tray_repository.go

@@ -22,6 +22,7 @@ type MockTrayRepository struct {
 	SetErr      error
 	DeleteErr   error
 	GetErr      error
+	FinishErr   error
 	StaleTrays  []*trays.Tray
 	StaleErr    error
 }
@@ -91,9 +92,29 @@ func (m *MockTrayRepository) UpdateStatus(_ context.Context, trayId string, stat
 		tray.WorkflowName = workflowName
 	}
 	tray.StatusChanged = time.Now()
+	if status == trays.TrayStatusRunning {
+		tray.JobStartedAt = time.Now()
+	}
 	return tray, nil
 }
 
+// FinishJob mirrors the real read-and-clear: the returned copy carries the
+// jobStartedAt that was there, and the stored row no longer has it.
+func (m *MockTrayRepository) FinishJob(_ context.Context, trayId string) (*trays.Tray, error) {
+	m.mu.Lock()
+	defer m.mu.Unlock()
+	if m.FinishErr != nil {
+		return nil, m.FinishErr
+	}
+	tray, ok := m.Trays[trayId]
+	if !ok {
+		return nil, nil
+	}
+	before := *tray
+	tray.JobStartedAt = time.Time{}
+	return &before, nil
+}
+
 func (m *MockTrayRepository) SetProviderData(_ context.Context, trayId string, data map[string]string) (*trays.Tray, error) {
 	m.mu.Lock()
 	defer m.mu.Unlock()

src/lib/trayManager/trayManager.go

@@ -197,6 +197,8 @@ func (tm *TrayManager) SetJob(ctx context.Context, trayId string, jobRunId int64
 // caller. Callers (unregister handler, stale loop) should treat a non-error
 // return as "deletion was requested," not "the upstream resource is gone."
 func (tm *TrayManager) DeleteTray(ctx context.Context, trayId string) (*trays.Tray, error) {
+	tm.accountFinishedJob(ctx, trayId)
+
 	tray, err := tm.trayRepository.UpdateStatus(ctx, trayId, trays.TrayStatusDeleting, 0, 0, 0, "", "", "")
 	if err != nil {
 		return nil, err
@@ -224,6 +226,36 @@ func (tm *TrayManager) DeleteTray(ctx context.Context, trayId string) (*trays.Tr
 	return tray, nil
 }
 
+// accountFinishedJob bills the tray's elapsed runner time to its repository.
+//
+// It sits in DeleteTray because every teardown path funnels through there —
+// job completed, agent unregister on preemption or SigTerm, the stale reaper,
+// and creation-failure cleanup. Accounting in the job-completed handler alone
+// would silently undercount exactly the repositories losing the most work to
+// preemption.
+//
+// FinishJob clears the start time as it reads it, so a tray whose cleanup
+// failed and is retried by the stale handler is counted once. Trays that never
+// ran a job have no start time and are skipped. A failure here must not block
+// deletion: the metric is worth less than the teardown.
+func (tm *TrayManager) accountFinishedJob(ctx context.Context, trayId string) {
+	finished, err := tm.trayRepository.FinishJob(ctx, trayId)
+	if err != nil {
+		log.Errorf("Failed to record finished job for tray %s: %v", trayId, err)
+		return
+	}
+	if finished == nil || finished.JobStartedAt.IsZero() {
+		return
+	}
+
+	metrics.JobFinished(
+		finished.GitHubOrgName,
+		finished.TrayTypeName,
+		finished.Repository,
+		time.Since(finished.JobStartedAt).Seconds(),
+	)
+}
+
 // HandleStale periodically deletes trays stuck in a non-running status for
 // longer than the configured threshold. It blocks until ctx is done.
 //