Skip to content

Bug fixes & doc update #74

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
AndWeHaveAPlan merged 2 commits into from
Sep 14, 2026
Merged

Bug fixes & doc update #74

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
8345e46
fixes
AndWeHaveAPlan Sep 14, 2026
41206eb
docs
AndWeHaveAPlan Sep 14, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 11 additions & 8 deletions README.MD
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ Cattery follows a **server/agent** architecture:

```
GitHub Actions Cattery Server Infrastructure
(Scale Set API) | (Docker / GCE)
(Scale Set API) | (Docker / GCE / Nomad)
| | |
|-- job demand ----------->| |
| |-- provision tray ----------->|
Expand All @@ -28,23 +28,24 @@ GitHub Actions Cattery Server Infrastructure

A tray progresses through: **Creating** -> **Registering** -> **Registered** -> **Running** -> **Deleting**

Stale trays (no heartbeat for 2+ minutes) are automatically cleaned up.
A tray that sits in any non-running status for longer than a configurable threshold is treated as stale and deleted (defaults: 5 minutes for Creating and Registering, 15 minutes for Registered and Deleting; see the `stale` section in [Configuration](docs/configuration.md)). An idle Registered tray is also told to shut itself down after 15 minutes.

## Features

- **Multi-provider** — Ships with Docker and Google Compute Engine providers, new providers can be added
- **Multi-provider** — Ships with Docker, Google Compute Engine and Nomad providers, new providers can be added
- **Scale-to-zero** — Runners are only provisioned when jobs are queued
- **Ephemeral runners** — Each job gets a fresh, isolated environment via JIT configuration
- **Multi-org support** — Manage runners across multiple GitHub organizations from a single server
- **Automatic failed job restart** — Agents can request reruns of failed workflow jobs
- **Preemption recovery** — Jobs lost to a preempted or terminated VM are re-run once their workflow run completes (skipped for merge queue runs, closed pull requests, and runs cancelled by a person or a concurrency group)
- **High availability** — Multiple server replicas with leader election backed by MongoDB or Kubernetes Leases
- **Monitoring** — Built-in status page (`/status`) and Prometheus metrics (`/metrics`)

## Prerequisites

- Go 1.25+
- Go 1.26+
- MongoDB
- A [GitHub App](https://docs.github.com/en/apps/creating-github-apps) with Actions read/write, Checks read and Pull requests read permissions, installed on your organization
- Docker and/or GCP credentials depending on your provider choice
- A [GitHub App](https://docs.github.com/en/apps/creating-github-apps) installed on your organization with these organization permissions: Self-hosted runners (read and write, for the runner scale set and JIT runner configuration), Actions (read and write, to re-run failed jobs), Checks (read) and Pull requests (read) for the preemption recovery checks
- Docker, GCP credentials, or a Nomad cluster depending on your provider choice

## Quick start

Expand Down Expand Up @@ -95,6 +96,8 @@ trayTypes:
image: my-runner-image:latest
```

`maxTrays` must be set: with the default of 0 no trays are ever created. A complete walkthrough with a local docker runner is in [examples/example.md](examples/example.md).

### Run

**Server:**
Expand Down Expand Up @@ -130,7 +133,7 @@ See [charts/cattery/values.yaml](charts/cattery/values.yaml) for the available v

## Configuration

See [docs/configuration.md](docs/configuration.md) for the full configuration reference covering server, database, GitHub App, providers, and tray types.
See [docs/configuration.md](docs/configuration.md) for the full configuration reference covering server, database, stale tray cleanup, leader election, GitHub App, providers, and tray types.

## License

Expand Down
5 changes: 3 additions & 2 deletions charts/cattery/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -180,8 +180,9 @@ CRDs installed in the cluster).
The Deployment defaults to `strategy: Recreate` with a single replica: each tray
type's poller holds a long-running GitHub session, so two overlapping pods would
double-poll. To run multiple replicas, set `config.coordination.backend` to
`mongo` or `k8s` — which leases each tray type's session to one replica at a
time — and switch `updateStrategy` to RollingUpdate. The tray API is served by
`mongo` or `k8s` — which leases each tray type's session, the workflow
restarter, and the stale tray cleanup to one replica at a time — and switch
`updateStrategy` to RollingUpdate. The tray API is served by
every replica regardless, so trays stay served throughout rollouts and
failovers. The `k8s` backend additionally needs `rbac.create` (default `true`)
for Lease access and mounts the service account token automatically.
Expand Down
64 changes: 48 additions & 16 deletions docs/configuration.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,6 @@
# Configuration

Cattery uses YAML configuration files and searches for the file named config.yaml first in the working directory and then in the /etc/cattery directory.
Cattery will use the first config file found.
Cattery reads a YAML configuration file. Pass its path with `cattery server -c /path/to/config.yaml`; without `-c`, cattery looks for `config.yaml` first in `/etc/cattery/` and then in the working directory, and uses the first one found.

### Example:

Expand All @@ -10,12 +9,22 @@ server:
listenAddress: "0.0.0.0:5137"
statusListenAddress: "0.0.0.0:5138"
advertiseUrl: https://example.org
agentSecret: my-secret-token

database:
uri: mongodb://localhost:27017/
database: cattery

stale:
pollInterval: 1m
thresholds:
creating: 5m
registering: 5m
registered: 15m
deleting: 15m

coordination:
backend: memory

github:
- name: my-org
appId: 123456
Expand Down Expand Up @@ -48,7 +57,6 @@ trayTypes:
maxParallelCreation: 5
config:
image: cattery-runner-tiny:latest
namePrefix: cattery

- name: cattery-gce-prod
provider: gce-prod
Expand All @@ -59,7 +67,6 @@ trayTypes:
extraMetadata:
cattery-agent-version: 0.0.4
config:
project: my-gcp-project
zones:
- europe-west1-c
- europe-west1-d
Expand Down Expand Up @@ -88,7 +95,7 @@ trayTypes:
| listenAddress | string | yes | Host:port for the HTTP server to bind (e.g., 0.0.0.0:5137). |
| statusListenAddress | string | no | Separate host:port for the /status and /metrics endpoints. If empty or equal to listenAddress, served on the agent port. |
| advertiseUrl | string | yes | Public base URL where the server is reachable. Passed to agents. |
| agentSecret | string | no | Bearer token that agents must present to register/unregister. If empty, agent auth is disabled. |
| agentSecret | string | no | Bearer token the server requires on every `/agent/*` request. **Leave empty:** the agent does not send this token yet, so setting it rejects every agent registration. |

#### database

Expand All @@ -97,6 +104,30 @@ trayTypes:
| uri | string | yes | MongoDB connection string (e.g., mongodb://localhost:27017/). |
| database | string | yes | Database name (e.g., cattery). |

#### stale
Optional. Configures the cleanup loop that deletes trays stuck in a non-running status. A tray whose status has not changed for longer than its threshold is deleted (the provider resource is cleaned up and the record removed). `running` trays are never stale.

| Key | Type | Required | Description |
|--------------|-------------------------|----------|-------------------------------------------------------------------------------------------------------------------------------------------------|
| pollInterval | duration | no | How often stale trays are looked for. Default `1m`. |
| thresholds | map[status]duration | no | Per-status age after which a tray is stale. Keys: `creating`, `registering`, `registered`, `deleting`. Defaults: `5m`, `5m`, `15m`, `15m`. Statuses missing from the map are not checked. |

Durations use Go syntax (`30s`, `5m`, `1h30m`). Independently of this loop, an agent whose tray has sat in `registered` (idle, no job) for 15 minutes is told to shut down on its next ping.

#### coordination
Optional. Selects the leader-election backend for running more than one server replica. Each tray type's scale set poller (the GitHub session), the workflow restarter and the stale tray cleanup each run on exactly one replica at a time; every replica serves the agent HTTP API regardless.

| Key | Type | Required | Description |
|-----------------------------|----------|----------|----------------------------------------------------------------------------------------------------------------------------------|
| backend | enum | no | `memory` (default), `mongo` or `k8s`. `memory` always leads and is correct **only for a single replica**. `mongo` stores leases in the configured database. `k8s` uses `coordination.k8s.io` Leases and must run in-cluster with RBAC to get/create/update leases. |
| lease.ttl | duration | no | Lease validity; bounds worst-case failover after a leader dies. Default `30s`. Ignored by `memory`. |
| lease.renewInterval | duration | no | How often a leader renews. Default `ttl/3`. Must stay well below `ttl`. |
| lease.retryInterval | duration | no | How often a non-leader retries acquisition. Default `5s`. |
| kubernetes.namespace | string | no | `k8s` only. Namespace for Lease objects. Defaults to the pod's namespace. |
| kubernetes.leaseNamePrefix | string | no | `k8s` only. Prefix prepended to the sanitized lease key to form the Lease name. |

Lease keys are the tray type names plus `cattery-restarter` and `cattery-stale-trays`; do not name a tray type after those two.

#### github
A list of GitHub organizations/accounts the server manages via a GitHub App.

Expand Down Expand Up @@ -142,7 +173,7 @@ Provider-specific fields:
| namespace | string | no | Nomad namespace to dispatch into. Defaults to `default`. |
| region | string | no | Nomad region. Defaults to the agent's region. |
| tlsCaFile | string | no | Path to a PEM CA bundle for verifying the Nomad agent's TLS certificate. |
| insecure | bool | no | Skip TLS verification. Dev-only. |
| insecure | bool | no | Skip TLS verification. Dev-only. Accepts `true`/`false`, quoted or not. |

#### trayTypes
Defines one or more tray "profiles" that the Tray Manager can maintain.
Expand All @@ -155,29 +186,30 @@ Defines one or more tray "profiles" that the Tray Manager can maintain.
| runnerGroupId | int | yes | GitHub Runner Group ID to register runners into. |
| githubOrg | string | yes | The GitHub org key, matching one of the entries under `github`. |
| shutdown | bool | no | Whether instances should self-terminate when the job completes. |
| maxTrays | int | no | Maximum number of concurrent trays of this type. |
| maxTrays | int | yes | Maximum number of concurrent trays of this type. Also the scale set's runner capacity reported to GitHub. Must be greater than 0: with the default of 0 no trays are ever created. |
| maxParallelCreation | int | no | Maximum number of trays to create in parallel. Defaults to 10. |
| extraMetadata | map[string]string | no | Extra key-value metadata passed to the provider (e.g., GCE instance metadata). |
| extraMetadata | map[string]string | no | Extra key-value metadata passed to the provider (GCE instance metadata, Nomad dispatch meta; ignored by docker). Keys are lowercased when the file is read. |
| config | provider-dependent | yes | Provider-specific configuration for how to create a tray (see below). |

Provider-specific config under trayType.config:

- docker config


Containers are named after the tray id and started as `<image> /action-runner/cattery/cattery agent -i <tray-id> -s <advertiseUrl> --runner-folder /action-runner` with `host.docker.internal` mapped to the host, so the image must contain the cattery binary and the Actions runner at those paths (see [examples/Dockerfile-cattery-tiny](../examples/Dockerfile-cattery-tiny)).

| Key | Type | Required | Description |
|------------|--------|----------|-----------------------------------------------------------------------------|
| image | string | yes | Docker image to run for the agent/runner (e.g., cattery-runner-tiny:latest) |
| namePrefix | string | no | Prefix for container names |

- google (GCE) config


Instances are named after the tray id and receive `cattery-url` (the `advertiseUrl`) and `cattery-agent-id` (the tray id) as instance metadata, plus any `extraMetadata`. The GCP project comes from the provider.

| Key | Type | Required | Description |
|------------------|----------|----------|---------------------------------------------------------------------------------|
| project | string | no | GCP project ID (can also be set at provider level) |
| zones | []string | yes | List of zones to create instances in (e.g. `europe-west1-c`) |
| zones | []string | yes | List of zones to create instances in (e.g. `europe-west1-c`); one is picked at random per tray |
| machineType | string | yes | Instance machine type (e.g. `e2-standard-4`) |
| instanceTemplate | string | yes | Template to base instances on (e.g. `global/instanceTemplates/cattery-default`) |
| namePrefix | string | no | Prefix for VM names |

- nomad config

Expand All @@ -189,7 +221,7 @@ Provider-specific config under trayType.config:

**Bootstrap composition.** The provider builds the dispatched payload from three pieces:

1. A fixed prelude that downloads the cattery agent binary from `$CATTERY_URL/agent/binary` to `/usr/local/bin/cattery`.
1. A fixed prelude that downloads the cattery agent binary from `$CATTERY_URL/agent/download` (the server serves its own executable) to `/usr/local/bin/cattery`.
2. The optional `script` field, executed as a pre-agent hook.
3. An `exec /usr/local/bin/cattery agent -i "$TRAY_NAME" -s "$CATTERY_URL" --runner-folder <runnerFolder>`, where `<runnerFolder>` defaults to `/cattery`.

Expand Down
40 changes: 23 additions & 17 deletions examples/Dockerfile-cattery-tiny
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,24 +1,30 @@
FROM ubuntu:24.04

RUN apt-get update && apt-get install -y \
git dotnet-runtime-8.0 golang-go tar curl

WORKDIR /action-runner
# Minimal runner image for the docker provider. Build from the repository root:
# docker build -t cattery-runner-tiny:latest -f examples/Dockerfile-cattery-tiny .
#
# The docker provider starts the container with
# /action-runner/cattery/cattery agent -i <tray-id> -s <advertiseUrl> --runner-folder /action-runner
# so the agent binary and the GitHub Actions runner must live at those paths.

ARG RUNNER_VERISON='2.333'
ENV RUNNER_VERISON=$RUNNER_VERISON
ARG GO_VERSION=1.26

RUN curl -sL -o actions-runner-linux-x64-${RUNNER_VERISON}.0.tar.gz https://github.com/actions/runner/releases/download/v${RUNNER_VERISON}.0/actions-runner-linux-x64-${RUNNER_VERISON}.0.tar.gz
RUN ls -al
RUN tar xzf ./actions-runner-linux-x64-${RUNNER_VERISON}.0.tar.gz
FROM golang:${GO_VERSION} AS build
WORKDIR /cattery
COPY src/ .
RUN CGO_ENABLED=0 go build -o /out/cattery .

WORKDIR /cattery-agent
FROM ubuntu:24.04

COPY . .
RUN apt-get update && apt-get install -y --no-install-recommends \
git curl ca-certificates tar \
&& rm -rf /var/lib/apt/lists/*

RUN ls -al /action-runner
RUN ls -al /cattery-agent
RUN ls -al .
WORKDIR /action-runner

ARG RUNNER_VERSION='2.333.0'
RUN curl -sL -o actions-runner.tar.gz \
https://github.com/actions/runner/releases/download/v${RUNNER_VERSION}/actions-runner-linux-x64-${RUNNER_VERSION}.tar.gz \
&& tar xzf actions-runner.tar.gz \
&& rm actions-runner.tar.gz \
&& ./bin/installdependencies.sh

RUN cd src && go build -o /action-runner/cattery/cattery
COPY --from=build /out/cattery /action-runner/cattery/cattery
Loading
Loading