Skip to content

Port to LegendHDF5IO 0.2 - #247

Open
oschulz wants to merge 9 commits into
mainfrom
lh5-update
Open

oschulz wants to merge 9 commits into
mainfrom
lh5-update

Conversation

@oschulz

@oschulz oschulz commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor
  • I have reviewed all the code in this pull request and deem it of the standard expected for this project.

Written by Claude (Anthropic) in Claude Code, a Fable 5.1 session at xhigh effort, under the guidance of @oschulz, who asked for this draft. Updated for release as 0.6.27 by an Opus 5.5 session at xhigh effort. Opened as a draft until LegendHDF5IO 0.2.0 is in the LEGEND registry.

Port to LegendHDF5IO 0.2, whose tables are lazy LH5LazyTables instead of TypedTables tables: reading a table opens a column only when it is used, rows are read in blocks or by index, and vcat and innermerge of lazy tables stay lazy. The extension and read_ldata are adapted so that the public behavior stays the same, apart from two result types noted below, and tier_data[tier, filekeys] gains a method for several file keys.

  • Require LegendHDF5IO 0.2 in the extension compat; 0.1 is no longer supported. Pkg enforces the compat of weak dependencies, so environments that keep LegendHDF5IO 0.1 keep LegendDataManagement 0.6.26. The API of LegendDataManagement itself only gains a method, so the version is 0.6.27, and packages bounded to 0.6 need no compat release.
  • Load lazily read tables in _load_all_keys, and read rows of lazy tables through the table itself rather than through Table(tbl), which 0.2 refuses.
  • Classify lazy table rows by their columns, and load chained and joined lazy tables like plain lazy tables.
  • Read rows by index through LegendHDF5IO's scattered reads instead of bridging gaps between selected rows by hand.
  • tier_data[tier, filekeys::AbstractVector{<:FileKey}] returns the paths of several file keys.
  • load_partition_ch reads partition rows through the lazy table instead of a Table of it.

Rebased onto main after #246. Verified: the test suite passes against LegendHDF5IO 0.2.0 (one pre-existing broken test), and the LegendHDF5IO suite's LegendDataManagement check passes against this branch. In 14 reads of LegendTestData files, covering the dsp, hit, pls, raw and evt tiers, read_ldata returns the same types as main with LegendHDF5IO 0.1.19, except for two that come from LegendHDF5IO 0.2: Bool columns are Vector{Bool} instead of BitVector, and the tables nested in an event tier, such as geds, are StructArrays instead of Tables.

LegendHDF5IO 0.2 registers custom datatypes through register_datatype!,
which makes the datatype_to_string and _array_type overrides of the
extension redundant.

Assisted-By: generative AI
LegendHDF5IO 0.2 reads tables as LH5LazyTables, which are not arrays, so
they fell through to the identity fallback and read_ldata returned them
unmaterialized.

Assisted-By: generative AI
Loading rows of a lazily read table went through its StructArray
conversion, whose slices keep the disk-backed row type, so rows of tables
with waveform columns could not be used. Index the lazy table, which
derives the row type from what it reads, and pass masks to the read as
LH5Array supports them now.

Assisted-By: generative AI
LegendHDF5IO 0.2 no longer converts a lazy table to a StructArray, and the
table's element type is the abstract NamedTuple. Rows count as scalars when
every column holds a bits type.

Assisted-By: generative AI
LegendHDF5IO 0.2 has an abstract supertype for its lazy tables, which also
covers tables concatenated or joined without reading them.

Assisted-By: generative AI
Rows selected by index were read range by range here, bridging gaps
between rows of scalars, because LegendHDF5IO 0.1 read index vectors
element by element. LegendHDF5IO 0.2 chooses between a bounding read, a
single selection and one read per run of rows by itself, so the rows
are now read by indexing the table or column directly.

Assisted-By: generative AI
@codecov

codecov Bot commented Nov 9, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 66.66667% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 63.55%. Comparing base (1ebfeb3) to head (4c26368).

Files with missing lines Patch % Lines
src/utils/data_utils.jl 0.00% 2 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #247      +/-   ##
==========================================
- Coverage   63.75%   63.55%   -0.21%     
==========================================
  Files          27       27              
  Lines        2842     2818      -24     
==========================================
- Hits         1812     1791      -21     
+ Misses       1030     1027       -3     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@oschulz
oschulz marked this pull request as ready for review November 12, 2026 10:43
@oschulz
oschulz marked this pull request as draft November 12, 2026 10:44
Assisted-By: generative AI
@oschulz
oschulz requested review from theHenks and a balanced review from Copilot and removed request for Copilot November 12, 2026 11:40
@oschulz

oschulz commented Nov 12, 2026

Copy link
Copy Markdown
Contributor Author

@theHenks can you test if this work in full production?

@oschulz
oschulz marked this pull request as ready for review November 13, 2026 11:19

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant