Skip to content

Make coalesce = TRUE output reproducible - #31

Open
gpt-cmdr wants to merge 1 commit into
walkerke:mainfrom
gpt-cmdr:upstream/deterministic-coalesce
Open

gpt-cmdr wants to merge 1 commit into
walkerke:mainfrom
gpt-cmdr:upstream/deterministic-coalesce

Conversation

@gpt-cmdr

Copy link
Copy Markdown

With coalesce = TRUE, building the same archive twice gives different bytes. coalesce_features() grouped features by property hash in a HashMap and emitted the groups in iteration order. std's HashMap is seeded randomly per map, so the order of the coalesced features in each tile changes between runs, and with it the archive bytes and size. Rendering is the same, but caching, checksums and output diffs break.

Groups now keep the order of their first feature. Tiles are Morton-sorted before coalescing, so this is the existing spatial order. Lookups still go through the same hash, so it costs nothing extra.

  • coalesce.rs: merge_lines() and group_polygons() share a group_by_properties() helper that returns groups in first-appearance order. Grouping is otherwise unchanged.
  • New Rust integration test tests/coalesce_order.rs, run as a CI step. It checks first-appearance order for lines and polygons, and repeats the call 32 times in one process. On main it fails on the first assertion. Each HashMap has its own seed, so repeated calls in one process are enough to show the problem.
  • New R test in test-coalesce.R: two freestile(coalesce = TRUE) runs on nc.shp, with a 3-value attribute, must produce identical bytes.

Validation:

  • On main, the new R test fails (archive lengths 36,496 vs 36,512 bytes, 36,166 mismatched bytes). With this change it passes, and the rest of test-coalesce.R still passes. R 4.6.1 on Windows, Rtools45, built from source with the CRAN feature set.
  • tests/coalesce_order.rs fails on main and passes with the fix. I ran it through a small harness crate that path-depends on freestiler-core. On my Windows machine, the crate's dev-dependency mlt-core -> fastpfor can't download CPM.cmake (TLS error), so a plain cargo test doesn't build there. The new CI step runs the test directly.
  • rustfmt --check is clean on the changed files. No dependency or lockfile changes.

I prepared this change with AI assistance (Claude Code) and reviewed it before submitting.

coalesce_features() grouped features by property hash in a HashMap and
emitted the groups in iteration order. std's HashMap is seeded per map, so
the order of coalesced features in a tile, and therefore the archive bytes,
changed from run to run. Groups now keep the order in which their first
feature appears, which is the Morton order the tile was sorted in.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant