Skip to content

metrics: add metrics to count job minutes - #76

Open
alvicsam wants to merge 1 commit into
mainfrom
as-prom
Open

alvicsam wants to merge 1 commit into
mainfrom
as-prom

Conversation

@alvicsam

@alvicsam alvicsam commented Sep 23, 2026 •

Copy link
Copy Markdown
Collaborator

New counters (metrics.go), labeled org, tray_type, repository:

  • cattery_job_seconds_total — runner seconds consumed
  • cattery_jobs_total — finished job count

cattery_registered_trays gained a repository label; unassigned trays report as repository="" rather than being dropped.

Plumbing:

  • Tray.JobStartedAt — stamped on the Running transition (the only one SetJob makes). It can't reuse StatusChanged, which DeleteTray's own status update overwrites.
  • TrayRepository.FinishJob — FindOneAndUpdate with $unset and ReturnDocument(Before): an atomic read-and-clear.
  • TrayManager.accountFinishedJob, called from DeleteTray — the single funnel all four teardown paths pass through, so preempted and stale-reaped jobs get billed too, not just clean completions.

DeleteTray can run more than once on the same tray. If provider cleanup fails, the row deliberately stays in deleting state, and the stale handler retries it ~15 minutes later. Without clearing jobStartedAt, each retry re-bills time.Since(jobStartedAt) — and the second charge is larger than the first, since the clock keeps running. The atomic read-and-clear also settles the race between the poller's JobCompleted and the agent's unregister, which can hit the same tray concurrently across replicas. Covered by TestDeleteTray_RetriedDeleteRecordsJobOnce.

cc https://github.com/paritytech/devops/issues/5688

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant