[siliataider]

This Pull request:

Changes or fixes:

The RDataLoader can now be created directly from files without building the RDataFrame manually first. The first argument accepts a (dataset_name, files) tuple, where dataset_name names the TTree or RNTuple to read and files is a file path (a str or any os.PathLike) or a list of them. Internally, each tuple is turned into one ROOT.RDataFrame(dataset_name, files). A list of tuples gives one RDataFrame per tuple, and tuples can be mixed with RDataFrames.

Everything works as before, RDataFrame just becomes an implementation detail in this case without the user interacting with it.

Demo:

import ROOT
from pathlib import Path

RDataLoader = ROOT.Experimental.ML.RDataLoader

# Until now: the RDataFrame had to be created first
df = ROOT.RDataFrame("events", "data.root")
dl = RDataLoader(df, batch_size=64, target="label")

# A single file, as a string or a pathlib.Path
dl = RDataLoader(("events", "data.root"), batch_size=64, target="label")
dl = RDataLoader(("events", Path("data.root")), batch_size=64, target="label")

# Several files are read as one dataset (TTree or RNTuple)
dl = RDataLoader(("events", ["data_1.root", "data_2.root"]), batch_size=64, target="label")

# Several datasets give one RDataFrame each, e.g. for resampling
dl = RDataLoader(
    [("signal", "sig.root"), ("background", "bkg.root")],
    batch_size=64,
    target="label",
    load_eager=True,
    sampling_type="undersampling",
)

A file path given without the name of the dataset raises a TypeError, and giving no input at all now raises a ValueError.

All the arguments but the inputs are now keyword-only, e.g. RDataLoader(df, batch_size=64) instead of RDataLoader(df, 64). The tutorials passing them by position are updated accordingly.

Note

The files of one tuple give a single dataset, like RDataFrame(dataset_name, files). Resampling with sampling_type needs two datasets, so two tuples (or two RDataFrames).

Accepted inputs

Below, t is the name of the TTree or RNTuple, f, f1 and f2 are file paths and df is an RDataFrame.

Input	Result
df, [df1, df2]	Read the RDataFrames
("t", "f"), ("t", Path("f"))	Read t from one file
("t", ["f1", "f2"]), ("t", ("f1", Path("f2"))), ("t", (f for f in files))	Read t from several files, as one dataset
("t", "f*.root")	Read t from the files matching the glob
[("t", "f1"), ("t", "f2")]	One RDataFrame per tuple, e.g. for resampling
[("t", "f"), df]	Tuples and RDataFrames mixed, in the given order
(spec for spec in specs)	Any iterable of the above
None, [], (), iter([]), (df for df in [])	ValueError: Detected empty input, at least one valid input is required.
"f", Path("f"), ["f1", "f2"]	TypeError: File paths are given together with the dataset to read, as (dataset_name, files) tuples
["t", "f"] (list instead of tuple), ("t", "f1", "f2"), ("t",)	Same TypeError
(Path("t"), "f"), (1, "f") (dataset name not a str), {"t": "f"}	Same TypeError
("t", 42)	TypeError: 'int' object is not iterable
("t", ["f", 3])	TypeError: expected str, bytes or os.PathLike object, not int
42, 3.14, True, [1, 2], [[df]], b"f"	TypeError from RDataFrame: Failed to instantiate ...
("t", "missing.root"), ("t", []), ("missing", "f")	TypeError from the RDataFrame constructor, with the reason in the details, e.g. could not open file "missing.root" or empty list of input files

Note

Since the backend is created lazily, these errors are raised on the first as_numpy(), as_torch(), as_tensorflow(), train_test_split() or save() call, not when the RDataLoader is constructed.

Checklist:

• tested changes locally

[github-actions]

Test Results

    23 files      23 suites   3d 19h 15m 14s ⏱️
 3 895 tests  3 894 ✅ 0 💤 1 ❌
79 577 runs  79 576 ✅ 0 💤 1 ❌

For more details on these failures, see this check.

Results for commit 2deed51.

♻️ This comment has been updated with latest results.

[vepadulano]

Thanks! Some minor considerations to discuss before final approval.

[vepadulano]

Thank you! I believe the interface looks much cleaner now and it should also be pretty sturdy!

[vepadulano]

Just wanted to note that at least from my side I see also the commit from this other PR #23788 as the first commit in this PR now.

[siliataider]

Thanks! Just solving a merge conflict, will rebase on master once the other PR is merged :)