Skip to content

fix: resolve data race on context in CollectorLifecycleHandler - #2617

Open
dalyk24 wants to merge 4 commits into
skupperproject:mainfrom
dalyk24:contribution/collector-handler-race
Open

dalyk24 wants to merge 4 commits into
skupperproject:mainfrom
dalyk24:contribution/collector-handler-race

Conversation

@dalyk24

@dalyk24 dalyk24 commented Oct 30, 2026 •

Copy link
Copy Markdown

Summary

Resolves a data race on CollectorLifecycleHandler.ctx between concurrent calls to Start() and asynchronous shutdown handling / collector startup.

Problem

In CollectorLifecycleHandler, Start(ctx context.Context, ...) assigns c.ctx = ctx under c.mux. However, handleShutdown (and startCollectorLite) running in background goroutines accessed c.ctx directly through the struct pointer without holding c.mux.

When Start() is called again concurrently or during restart sequences (e.g. tested in TestCollectorHandler/restart-collector), Go's race detector reports a data race:

WARNING: DATA RACE
Read at 0x... by goroutine ...:
  github.com/skupperproject/skupper/internal/nonkube/controller.(*CollectorLifecycleHandler).handleShutdown()
      internal/nonkube/controller/collector_handler.go:102
Previous write at 0x... by goroutine ...:
  github.com/skupperproject/skupper/internal/nonkube/controller.(*CollectorLifecycleHandler).Start()
      internal/nonkube/controller/collector_handler.go:73

Fix

  • In CollectorLifecycleHandler.Start(), capture ctx := c.ctx locally under c.mux.
  • Pass ctx by value to c.startMethod(ctx) and c.handleShutdown(ctx, stopCh).
  • Update CollectorLifecycleHandler.handleShutdown and startMethod signature to accept ctx context.Context as a parameter instead of dereferencing c.ctx.
  • Update fake collector implementations in collector_handler_test.go to match the signature.

Testing

Verified with go test -race across 100 consecutive runs:

# Before fix:
$ go test -race -count=100 -run TestCollectorHandler ./internal/nonkube/controller/...
--- FAIL: TestCollectorHandler (0.01s)
    --- FAIL: TestCollectorHandler/restart-collector (0.00s)
    testing.go:1464: race detected during execution of test

# After fix:
$ go test -race -count=100 -run TestCollectorHandler ./internal/nonkube/controller/...
PASS
ok      github.com/skupperproject/skupper/internal/nonkube/controller   1.218s

Changes

  • internal/nonkube/controller/collector_handler.go: Pass ctx by value to startMethod and handleShutdown, eliminating unprotected struct field reads.
  • internal/nonkube/controller/collector_handler_test.go: Update mock/fake test handler to accept ctx context.Context.

Summary by CodeRabbit

  • Bug Fixes
    • Collector startup and shutdown now follow a consistent lifecycle, and startup is skipped if cancellation has already occurred.
    • Rapid restarts wait for the previous collector run to finish, helping prevent overlapping runs.
    • Collector status data is cleaned up when a run stops or setup fails, reducing the chance of stale status remaining.

The handleShutdown goroutine was reading c.ctx through the struct pointer
without holding c.mux. A concurrent Start() call writes c.ctx under the
lock, creating an unsynchronised read/write pair detected by -race on the
restart-collector sub-test.

Capture ctx := c.ctx inside Start() under the lock before launching the
goroutines, then pass ctx by value to startMethod(ctx) and
handleShutdown(ctx, stopCh). Update the startMethod signature and test
fakes accordingly.

Signed-off-by: Killian Daly <Killian.Daly@ibm.com>
@coderabbitai

coderabbitai Bot commented Oct 30, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

Important

Review skipped

Review was skipped as selected files did not have any reviewable changes.

Run configuration
  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 136bcb4d-6c3d-4ca8-883b-662f305e5772
Commits

Reviewing files that changed from the base of the PR and between b228d01 and 33d9f09.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • Trigger review

No actionable comments were generated in the recent review.

Recent review info
Run configuration
  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 38deb4ce-c32b-421e-ac8c-6b1ed72bc7c0
Commits

Reviewing files that changed from the base of the PR and between da4c734 and ba83af3.

Files selected for processing: 4
  • internal/nonkube/controller/collector_handler.go
  • internal/nonkube/controller/collector_handler_generation_test.go
  • internal/nonkube/controller/collector_handler_test.go
  • internal/nonkube/flow/collector.go
Files skipped from review as they are similar to previous changes: 3
  • internal/nonkube/controller/collector_handler_test.go
  • internal/nonkube/flow/collector.go
  • internal/nonkube/controller/collector_handler.go

Included review availability: This review used your included allowance. Your plan provides up to 4 included reviews per hour; 3 remain after this review.

Walkthrough

The collector handler passes each generation’s context to startup and shutdown handling and waits for the previous generation to finish before starting another. The collector waits for status synchronization to stop before removing its status ConfigMap and returning.

Changes: Collector lifecycle

Layer / File(s) Summary
Pass context and serialize generations
internal/nonkube/controller/collector_handler.go, internal/nonkube/controller/collector_handler_test.go, internal/nonkube/controller/collector_handler_generation_test.go
Start captures the generation context and start method, then waits for the previous generation to finish. Startup and shutdown handling use the captured context. Tests cover context cancellation and repeated restarts.
Wait for collector cleanup
internal/nonkube/flow/collector.go
siteCollector defers status ConfigMap deletion and waits for statusSync.Run to stop after context cancellation. StartCollector documents its blocking and cleanup behavior.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Change: Bug fix

Sequence Diagram(s)

sequenceDiagram
  participant CollectorHandler
  participant StartCollector
  participant StatusSync
  CollectorHandler->>StartCollector: Start with captured generation context
  StartCollector->>StatusSync: Run status synchronization
  CollectorHandler->>StartCollector: Cancel generation context
  StartCollector->>StatusSync: Wait for Run to stop
  StartCollector->>CollectorHandler: Return after ConfigMap cleanup
Loading
Merge Risk: Minimal · previous review b228d

Merge Risk: ⚪ Minimal · up to b228d

Collector generations are serialized and cleanup waits for status synchronization. No merge-blocking issue was established.

Pre-merge checks: Passed 5
Check name Status Explanation
Description Check Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check Passed The title clearly and concisely describes the main change: resolving a data race involving context handling in CollectorLifecycleHandler.
Docstring Coverage Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentat…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Docstring Coverage

Explanation

Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 5 functions across 4 files.

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1


Review info
Run configuration
  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: ef1f167c-8729-496b-b971-532601150fa9
Commits

Reviewing files that changed from the base of the PR and between 3981c91 and a4395ec.

Files selected for processing: 2
  • internal/nonkube/controller/collector_handler.go
  • internal/nonkube/controller/collector_handler_test.go

Included review availability: This review used your included allowance. Your plan provides up to 4 included reviews per hour; 3 remain after this review.

If Stop() and a new Start() occur before a generation-1 startCollectorLite
goroutine is scheduled, the goroutine receives an already-cancelled ctx.
flow.StartCollector does not check ctx.Err() at entry: siteCollector adds
the shared status ConfigMap and registers a cleanup goroutine that deletes
it on ctx.Done(). Because ctx is already done, the deletion fires
immediately, removing the ConfigMap that the new generation is actively
using and preventing further status updates.

Add an ctx.Err() guard at the top of startCollectorLite. If the context is
already cancelled, log and return before siteCollector is ever called, so
the cleanup goroutine is never registered and the shared ConfigMap is never
deleted by a stale generation.
…t status

Each collector generation writes the same network status ConfigMap file.
Its removal was fire-and-forget: siteCollector returned right after
spawning a goroutine that waited for ctx.Done() and then deleted the file.
Nothing made the next generation wait for that delete, so an immediate
Stop() followed by Start() could let the new generation write the file
before the old generation's delete landed, removing the new generation's
status and stopping status updates.

Make the collector own its lifetime:

- flow.StartCollector now blocks until ctx is canceled and the status
  sync has stopped, then removes the ConfigMap via defer. The delete also
  runs if setup fails after the ConfigMap was added.
- CollectorLifecycleHandler.Start keeps the previous generation's done
  channel and makes the new generation wait for it before calling
  startMethod. Start still returns immediately and the wait happens in
  the goroutine, so c.mux is never held while waiting.

The ctx.Err() guard in startCollectorLite is kept; it now also skips a
generation that was canceled while queued behind its predecessor.

Add a regression test that restarts the real collector back to back
against a temporary data directory and asserts the current generation's
ConfigMap is present. It fails without the change.

Signed-off-by: Killian Daly <Killian.Daly@ibm.com>

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1


Review info
Run configuration
  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 432d89aa-b417-43d8-b50d-d7433082e4ff
Commits

Reviewing files that changed from the base of the PR and between 1e9c7ae and da4c734.

Files selected for processing: 3
  • internal/nonkube/controller/collector_handler.go
  • internal/nonkube/controller/collector_handler_generation_test.go
  • internal/nonkube/flow/collector.go
Files skipped from review as they are similar to previous changes: 1
  • internal/nonkube/controller/collector_handler.go

Included review availability: This review used your included allowance. Your plan provides up to 4 included reviews per hour; 3 remain after this review.

The restart regression test slept for 100ms and then checked that the
status ConfigMap existed. If generation 1 had not finished by then, its
file could satisfy the check and generation 1 could delete it after the
test reported success, while a slow but correct generation 2 could make
the test fail.

Capture each generation's lastDone channel around the restart. Wait for
generation 1 to finish (it removes its ConfigMap before returning), then
poll for generation 2's file. After the final Stop, wait for generation
2 to finish and assert the file is gone before starting the next round.
All waits have a timeout.

Also add doc comments to the functions touched by this change and fix the
comment in Start, which said ctx is captured before the lock is
released; the lock is held at that point.

Signed-off-by: Killian Daly <Killian.Daly@ibm.com>
@dalyk24
dalyk24 force-pushed the contribution/collector-handler-race branch from b228d01 to 33d9f09 Compare November 1, 2026 16:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant