Skip to content

Add gzip_level to freestile() - #32

Open
gpt-cmdr wants to merge 1 commit into
walkerke:mainfrom
gpt-cmdr:upstream/gzip-level
Open

gpt-cmdr wants to merge 1 commit into
walkerke:mainfrom
gpt-cmdr:upstream/gzip-level

Conversation

@gpt-cmdr

Copy link
Copy Markdown

Tile data is always gzipped at the fast level 1. That's a good default for build time, but for archives that are built once and served many times, a higher level gives a smaller file for a one-time cost. This adds a gzip_level option (0-9) to freestile() in R and Python. The default is unchanged, so existing output is byte-identical. Other levels change only the compression: the decompressed tiles are identical.

  • Core: TileConfig::gzip_level: Option<u32> (None = level 1) and pmtiles_writer::gzip_compress_level(). gzip_compress() keeps its signature and behavior. Levels above 9 are refused before any output is written.
  • R: freestile(gzip_level = NULL), validated as a whole number from 0 to 9 and passed to rust_freestile(). The Rd files and extendr-wrappers.R are updated by hand to match the new argument.
  • Python: freestile(gzip_level=None), which rejects non-integers, booleans and out-of-range values.
  • Both bindings refuse gzip_level together with categorical clustering, which writes through its own path. freestile_file(), freestile_query() and the streaming writer pass None and are unchanged. Exposing the option there too is a small follow-up if you want it everywhere.

Measured with Python on synthetic data (the best of 3 runs; Windows, 8 cores, a shared machine, so the times are approximate):

Input Zooms Level 1 (default) Level 6 Level 9
300 x 300 grid of square cells, cell_id + one float column 10-15 12.44 MB, 7.8 s 10.66 MB (-14%), 9.1 s 10.62 MB (-15%), 11.2 s
200,000 random points, one float column 0-12 48.3 MB 45.3 MB (-6%) 45.1 MB (-7%)

The point timings varied by more than the level effect on this machine, so I left them out. Level 6 looks like the useful setting; level 9 adds little size reduction for the extra time.

Validation:

  • New Rust integration test tests/gzip_level.rs, added as a CI step. It checks level round trips and bounds, that level 9 makes a smaller archive with identical decompressed tiles, and that level 12 is refused with no file written.
  • New R test test-gzip-level.R: level 9 is smaller than the default on nc, and -1, 10, 2.5, "6", NA and c(1, 2) are rejected without writing output. Built from source (R 4.6.1, Rtools45, CRAN feature set). Full testthat suite: 0 failures (the skips are DuckDB/GeoParquet tests the CRAN build doesn't include).
  • New Python test test_gzip_level.py: smaller archive with identical decompressed tiles, rejection of -1, 10, 2.5, True and "6", and refusal with categorical clustering. Full suite passes (wheel built with --no-default-features --features geoparquet, so the DuckDB tests skip).
  • The Rust test was run through a small harness crate that path-depends on freestiler-core, because the crate's dev-dependency mlt-core -> fastpfor can't download CPM.cmake on my machine (TLS error). The CI step runs it directly. The streaming unit tests (--features duckdb) gain the new field. I couldn't build those locally, so CI or a local cargo test --features duckdb should confirm they compile.
  • No new dependencies and no lockfile changes.

I prepared this change with AI assistance (Claude Code) and reviewed it before submitting.

Tiles are always gzipped at the fast level 1. A new gzip_level option (0-9)
in R and Python freestile(), and TileConfig::gzip_level in the core, sets the
level for the in-memory pipeline. None keeps level 1, so existing output is
unchanged; other levels change only the compression, never the decompressed
tiles. Out-of-range levels are refused before any output is written.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant