Skip to content

Add the upgrade flag to skupper system install command - #2597

Merged
nluaces merged 10 commits into
skupperproject:mainfrom
nluaces:handle-system-updates
Oct 26, 2026
Merged

nluaces merged 10 commits into
skupperproject:mainfrom
nluaces:handle-system-updates

Conversation

@nluaces

@nluaces nluaces commented Oct 7, 2026 •

Copy link
Copy Markdown
Member

Resolves #2390

Summary by CodeRabbit

  • New Features
    • Added an --upgrade option to skupper system install to update the system controller and routers to their configured images.
    • Components remain unchanged when no image updates are needed.
  • Bug Fixes
    • Upgrade errors are reported, and affected components are restored where possible.
    • The --upgrade option cannot be combined with --reload-type.
    • Extracted files now replace existing contents completely, including leftover data from longer files.
    • Site cleanup skips containers that are not running.

@nluaces nluaces self-assigned this Oct 7, 2026
@coderabbitai

coderabbitai Bot commented Oct 7, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review
Walkthrough

The skupper system install command adds an --upgrade flag. For non-Kubernetes installations, upgrade checks controller and router image drift, backs up affected containers, and upgrades drifted components. The change also isolates command test data and adjusts container cleanup and tar extraction behavior.

Changes: Non-Kubernetes system upgrade

Layer / File(s) Summary
Upgrade flag and command dispatch
internal/cmd/skupper/common/flags.go, internal/cmd/skupper/system/system.go, internal/cmd/skupper/system/system_test.go, internal/cmd/skupper/system/nonkube/system_install.go, internal/cmd/skupper/system/nonkube/system_install_test.go
The install command adds the --upgrade flag, which defaults to false. The non-Kubernetes command rejects a reload type when upgrade is selected and dispatches to either the install or upgrade callback. Tests cover flag propagation, validation, and callback results.
Controller and router image drift checks
internal/nonkube/bootstrap/image_check.go, internal/nonkube/bootstrap/image_check_test.go, internal/nonkube/bootstrap/upgrade.go, internal/nonkube/bootstrap/upgrade_test.go
Controller and router checks compare inspected container images with configured images. The namespace scan checks directories and router drift. Tests cover scan results, skipped files, and errors.
Back up and upgrade drifted containers
internal/nonkube/bootstrap/upgrade.go, internal/nonkube/bootstrap/upgrade_test.go
Upgrade backs up affected containers, upgrades drifted routers before installing a drifted controller, and restores backups if an upgrade step fails. It bootstraps backed-up router namespaces using stored site-state paths and collects namespace failures.

Non-Kubernetes command test data isolation

Layer / File(s) Summary
Use temporary command test data
internal/cmd/skupper/connector/nonkube/connector_create_test.go, internal/cmd/skupper/connector/nonkube/connector_update_test.go, internal/cmd/skupper/listener/nonkube/listener_create_test.go, internal/cmd/skupper/listener/nonkube/listener_update_test.go, internal/cmd/skupper/site/nonkube/site_create_test.go
Connector, listener, and site tests direct data storage to temporary locations based on whether the process runs as root. Some run tests no longer defer deletion of test entries.

Namespace container cleanup

Layer / File(s) Summary
Skip stopped site containers
internal/nonkube/compat/site_state_renderer.go
Namespace cleanup skips matching site containers that are not running. Running matches follow the existing stop-and-rename path.

Tar extraction file handling

Layer / File(s) Summary
Truncate regular-file targets
internal/utils/tarball.go
Tar extraction opens regular-file targets with truncation enabled.

Priority: Low

Estimated code review effort: 3 (Moderate) | ~25 minutes

Change: Feature

Sequence Diagram(s)

sequenceDiagram
  participant CmdSystemInstall
  participant Upgrade
  participant ImageDriftChecks
  participant BootstrapSite
  CmdSystemInstall->>Upgrade: invoke when upgrade is selected
  Upgrade->>ImageDriftChecks: check controller and router image drift
  Upgrade->>BootstrapSite: bootstrap backed-up router namespaces
Loading
Merge Risk: Moderate · up to 9060f

Inspection failures can leave an upgrade incomplete or prevent controller recovery, and a stopped router can block reload. Address those paths before merging.

Security architecture risk: Moderate · up to 9060f

The upgrade’s success and recovery guarantees are incomplete. Failures can leave older versions running or leave sites unavailable, and rollback may not restore the full prior state. No new unauthorized access path was established.

Retained concerns

  • Medium · security · observed: The upgrade can report success without replacing an outdated component. Router backup suppresses subsequent drift-check and inspection errors and omits affected namespaces; an empty backup map then upgrades nothing successfully. Controller inspection failures likewise become a drift-positive result followed by no backup, rather than a confirmed replacement decision. This undermines upgrade completion, including delivery of security updates.
  • Medium · reliability · observed: Recovery ownership is not preserved through every failure or reconstructed after interruption. Failure to remove a later namespace’s stale backup returns nil backup records even when earlier routers were already stopped and renamed, preventing their restoration. An interruption after rename also strands backups: a subsequent router check treats the absent original container as no drift and can declare the system up to date. The newly invoked renderer additionally calls os.Exit on configuration-rendering failure, bypassing both deferred and outer recovery.
  • Medium · reliability · inferred: Rollback restores container identities but not necessarily their matching configuration and certificates. Each successful namespace render discards its local filesystem snapshot before the overall upgrade commits. If another namespace or controller installation later fails, outer recovery restarts the old router containers against the newly rendered host-mounted state. Compatibility and security-configuration consistency of that mixed state are not established.

Security review details

Security Blast Radius

  • inferred — The effective mutation scope is the selected container endpoint’s accessible, namespace-named routers represented in the local namespace directory, plus the current user’s named controller. Failures can therefore affect multiple sites in one invocation; the evidence does not establish access across independent engines or environments.

Security Findings and Attack Paths

  • inferred — The supported security-relevant failure path is incomplete update delivery, not an established unauthorized caller: a router is initially detected as outdated, a later inspection fails, backup silently omits it, and the command can finish successfully. No specific exploitable image vulnerability or unprivileged mechanism for inducing that failure was established.

Trust Boundaries and Controls

  • observed — The concrete entrypoint is the existing operator CLI. Endpoint and image configuration remain the same authorities used by installation, and container removal checks Skupper ownership labels. These controls constrain exposure; they do not make upgrade selection and recovery atomic.

Resilience and Maintainability Implications

  • observed — Backup identity uses shared deterministic names and recovery records live in process memory. The inspected root, shared command dispatch, and upgrade implementation contain no cross-process lifecycle guard. Cleanup failures are logged rather than changing a successful result, so backup disposal is not part of the success guarantee.

Hardening Proposals

  • proposed — Make upgrade a recoverable transaction: serialize operations for the same endpoint and state scope, persist backup ownership before destructive transitions, reconcile interrupted backups before drift-based early exit, retain configuration snapshots until overall commit, and require explicit completion or failure for every selected component.
Pre-merge checks | Passed 4 | Failed 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 48 functions across 16 files. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check Passed The title clearly and concisely describes the primary change: adding the upgrade flag to the Skupper system install command.
Linked Issues check Passed Issue #2390 requests a CLI path to update the System controller and controlled router sites. The PR adds skupper system install --upgrade, validates its flag combination, detects controller and rout…
Out of Scope Changes check Passed The production changes implement the issue #2390 upgrade flow. Backup restoration, stopped-container cleanup, tar extraction truncation, and isolated non-Kubernetes test data support that flow or its …
✨ Finishing Touches: 💡 1
🛠️ Fix failing CI checks: 💡
  • Commit to this branch
  • Create a new PR
  • Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 6


ℹ️ Review info
⚙️ Run configuration
  • Configuration used: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 27b7c91b-3675-49a0-848c-a4f611a6eb05
📥 Commits

Reviewing files that changed from the base of the PR and between 7e3dd8f and d3169ee.

📒 Files selected for processing (9)
  • internal/cmd/skupper/common/flags.go
  • internal/cmd/skupper/system/nonkube/system_install.go
  • internal/cmd/skupper/system/nonkube/system_install_test.go
  • internal/cmd/skupper/system/system.go
  • internal/cmd/skupper/system/system_test.go
  • internal/nonkube/bootstrap/image_check.go
  • internal/nonkube/bootstrap/image_check_test.go
  • internal/nonkube/bootstrap/upgrade.go
  • internal/nonkube/bootstrap/upgrade_test.go

Included review availability: This review used your included allowance. Your plan provides up to 4 included reviews per hour; 3 remain after this review.

Comment thread internal/nonkube/bootstrap/upgrade.go Resolved
Comment thread internal/nonkube/bootstrap/upgrade.go Resolved
Comment thread internal/nonkube/bootstrap/upgrade.go Resolved

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1


ℹ️ Review info
⚙️ Run configuration
  • Configuration used: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 78fb89b3-db34-48de-876d-d9d6db5fc679
📥 Commits

Reviewing files that changed from the base of the PR and between 1999450 and a97d3e6.

📒 Files selected for processing (8)
  • internal/cmd/skupper/connector/nonkube/connector_create_test.go
  • internal/cmd/skupper/connector/nonkube/connector_update_test.go
  • internal/cmd/skupper/listener/nonkube/listener_create_test.go
  • internal/cmd/skupper/listener/nonkube/listener_update_test.go
  • internal/cmd/skupper/site/nonkube/site_create_test.go
  • internal/cmd/skupper/system/nonkube/system_install.go
  • internal/cmd/skupper/system/nonkube/system_install_test.go
  • internal/nonkube/bootstrap/upgrade.go

Included review availability: This review used your included allowance. Your plan provides up to 4 included reviews per hour; 3 remain after this review.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 4


ℹ️ Review info
⚙️ Run configuration
  • Configuration used: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 798b666f-1597-4cfa-8ab6-ac39a2c53367
📥 Commits

Reviewing files that changed from the base of the PR and between a97d3e6 and f99fe0e.

📒 Files selected for processing (4)
  • internal/nonkube/bootstrap/upgrade.go
  • internal/nonkube/bootstrap/upgrade_test.go
  • internal/nonkube/compat/site_state_renderer.go
  • internal/utils/tarball.go

Included review availability: This review used your included allowance. Your plan provides up to 4 included reviews per hour; 3 remain after this review.

Comment thread internal/nonkube/bootstrap/upgrade.go Resolved
Comment thread internal/nonkube/bootstrap/upgrade.go Resolved
Comment thread internal/nonkube/bootstrap/upgrade.go Resolved

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

Caution

Some comments are outside the diff and can’t be posted inline due to GitHub limitations.

⚠️ Outside diff range comments (1)

🟡 Minor · Propagate operational errors from router inspection. · upgrade.go:107-120

internal/nonkube/bootstrap/upgrade.go:107-120
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Propagate operational errors from router inspection.

Upgrade checks router drift, then inspects each router again during backup. If the second ContainerInspect returns an operational error, backupRouterContainers skips that namespace. upgradeNamespaces then has no entry to upgrade and returns success. Preserve the existing not-found handling, but return other inspection errors.

Suggested fix
 import (
 	"errors"
 	"fmt"
 	"log/slog"
 	"os"
 	"os/user"
+	"strings"
@@
 		c, err := cli.ContainerInspect(containerName)
-		if err != nil || c == nil {
+		if err != nil {
+			if strings.Contains(err.Error(), "not found") {
+				continue
+			}
+			return backups, backupContainerNames, fmt.Errorf("namespace %q: failed to inspect router container: %w", ns, err)
+		}
+		if c == nil {
 			continue
 		}

ℹ️ Review info
⚙️ Run configuration
  • Configuration used: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 9157acd1-165a-445d-b285-bbcccccd5a4a
📥 Commits

Reviewing files that changed from the base of the PR and between f99fe0e and 9060f7a.

📒 Files selected for processing (1)
  • internal/nonkube/bootstrap/upgrade.go

Included review availability: This review used your included allowance. Your plan provides up to 4 included reviews per hour; 2 remain after this review.


// Remove the partially created controller if Install created it before
// failing, so the rename below does not hit a name collision.
if existing, inspectErr := cli.ContainerInspect(originalName); inspectErr == nil && existing != nil {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🟠 Major | ⚡ Quick win

Handle controller inspection errors before rollback.

If the replacement controller still has originalName and ContainerInspect returns an error, the current condition skips ContainerRemove. The subsequent backup rename can fail because the replacement still owns the name, while the original controller remains stopped. Distinguish a missing container from other inspection errors. Preserve the backup and return an error that identifies the failed inspection.

@nluaces
nluaces force-pushed the handle-system-updates branch from 9060f7a to a5b9363 Compare October 13, 2026 15:47

@fgiorgetti fgiorgetti left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One last comment. If the system controller is running with docker and the reload-type is set to auto. The upgrade is not working due to permission issues.

Comment thread internal/nonkube/bootstrap/upgrade.go Resolved
Comment thread internal/nonkube/bootstrap/upgrade.go Resolved
@nluaces
nluaces requested a review from fgiorgetti October 22, 2026 12:39
Comment thread internal/nonkube/bootstrap/upgrade.go Resolved
Platform: types.Platform(nsPlatform),
InputPath: sourcesPath,
}
siteState, err := w.bootstrap(cfg)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am seeing one issue when the controller reloads a site due to a drifted router (reload-type=auto).
If a site is started with --enable-link-access, the allocated router access port is 55671 and 45671.
When the router image drifts, the new container is allocating 55672 and 45672, therefore existing incoming links stop working.
I believe we might need a change to detect currently allocated router access ports if enable-link-access has been defined and reuse them.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually this is also happening when reload-type is manual.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you! I tested this before adding the backup part for the upgrade!

@nluaces
nluaces requested a review from fgiorgetti October 26, 2026 11:06

@fgiorgetti fgiorgetti left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@nluaces
nluaces merged commit 074d0c8 into skupperproject:main Oct 26, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Handle System controller updates

3 participants