Restrict runinfo to the run lists of a dataset; DataSet livetimes and time selections

docs/src/man/channel_run_info.md

@@ -25,7 +25,7 @@ rinfo = runinfo(l200, :p02)
 rinfo.run
 ```
 
-Each category column is itself a table with the columns `startkey`, `endkey`, `livetime`, `is_analysis_run` and `keys`. A run without an entry for a category has `missing` keys, a livetime of `NaN` seconds and `is_analysis_run = false` there.
+Each category column is itself a table with the columns `startkey`, `endkey`, `livetime`, `is_analysis_run` and `keys`. A run without an entry for a category has `missing` `startkey` and `endkey`, an empty `keys` set, a livetime of `NaN` seconds and `is_analysis_run = false` there.
 
 ```@example ldm
 rinfo.phy.startkey[1:2]
@@ -54,6 +54,25 @@ rinfo_cal = runinfo(l200, :p02) |> filterby(@pf $cal.is_analysis_run)
 first(rinfo_cal).cal.startkey
 ```
 
+### Run Information of a Dataset
+
+For a dataset other than `:default` (see [Datasets](@ref)), `runinfo` has a column only for the categories that the run list of the dataset names. A run is a row if at least one of its categories is listed; for a category that the list does not name for the run, the entry is empty as described above. `:default` holds every run and category of the metadata `datasets/runinfo`. The test metadata has the run list `valid` with the categories `cal` and `phy`:
+
+```@example ldm
+valid = LegendData(:l200; dataset = :valid)
+(propertynames(runinfo(l200)), propertynames(runinfo(valid)))
+```
+
+```@example ldm
+(length(runinfo(l200)), length(runinfo(valid)))
+```
+
+Selecting a category that the dataset does not have throws an `ArgumentError`. This holds for `runinfo`, [`start_filekey`](@ref), [`is_lrun`](@ref) and [`is_analysis_run`](@ref).
+
+```@example ldm
+try runinfo(valid, (:p03, :r000, :fft)) catch err sprint(showerror, err) end
+```
+
 ### Start Keys, Livetime and Analysis Runs
 
 [`start_filekey`](@ref) gives the first file key of a run and category, or `missing` if the metadata has none. [`livetime`](@ref) gives the livetime in seconds as a `Unitful` quantity, `NaN` seconds if the metadata has none.
@@ -76,7 +95,7 @@ ustrip(u"s", livetime(l200, (:p02, :r006, :cal)))
 
 The exposure is calculated from the livetime (see [Exposure and Livetime](@ref)).
 
-[`is_lrun`](@ref) tells whether a run exists in the metadata with a start key, and [`is_analysis_run`](@ref) whether it is an analysis run of its category. [`analysis_runs`](@ref) lists all analysis runs of a category. The analysis runs are those listed in the metadata `datasets/runlists` for `data.dataset`, with the run list `valid` for the default dataset.
+[`is_lrun`](@ref) tells whether a run exists in the metadata with a start key, and [`is_analysis_run`](@ref) whether it is an analysis run of its category. [`analysis_runs`](@ref) lists all analysis runs of a category. The analysis runs of `cal` and `phy` are those listed in the metadata `datasets/runlists` for `data.dataset`, with the run list `valid` for the default dataset. A run entry that a dataset does not list for a category is not an analysis run.
 
 ```@example ldm
 is_lrun(l200, (:p02, :r006, :cal)), is_lrun(l200, (:p02, :r099, :cal))
@@ -94,6 +113,30 @@ is_analysis_run(l200, mfk)
 analysis_runs(l200, :phy).run[1:3]
 ```
 
+### Sets of File Keys
+
+[`DataSet`](@ref) is a vector of the file keys of DAQ cycles, sorted by time. `DataSet(data, runinfo(data, period))` and `DataSet(data, runinfo(data).phy)` give the keys of the runs of a table and of the entries of a category table. `filter(f, ds)` returns a `DataSet` of the same name with the keys for which `f` is `true`.
+
+```@example ldm
+ds = DataSet(l200, runinfo(l200).phy)
+```
+
+```@example ldm
+filter(!in(bad_filekeys(l200)), ds)
+```
+
+The `text/plain` display of a set that is built from `runinfo` adds two lines. `livetime` gives, per category, the sum of the `runinfo` livetimes of the runs that have keys in the set, in days; it is `NaN` if a run has no livetime in the metadata. `runtime` gives the time from the first to the last key of the set, in days.
+
+```@example ldm
+DataSet(l200, runinfo(l200, :p02))
+```
+
+[`find_filekey`](@ref) returns the key of the DAQ cycle that contains a timestamp. For a `LegendData` it searches the DAQ cycle keys of every run in the metadata, whatever `data.dataset` is. A `LegendData` broadcasts as a scalar, so `find_filekey.(data, timestamps)` finds the key of each timestamp.
+
+```@example ldm
+find_filekey.(l200, ["20220601T000000Z", "20220802T030000Z"])
+```
+
 ### Ignored DAQ Cycles
 
 [`bad_filekeys`](@ref) returns the set of file keys that the metadata `datasets/ignored_daq_cycles` lists, e.g. cycles without usable data. `load_key` selects one of the lists of that file; the default `:all` merges them.

docs/src/man/exposure.md

@@ -41,7 +41,7 @@ ustrip(u"s", total)
 \sum_{\text{runs}} m_\text{det} \, t_\text{live}
 ```
 
-in `kg * yr`. The detector mass ``m_\text{det}`` is taken from the channel information at the start file key of each run (the column `mass` of [`channelinfo`](@ref) with `extended = true`), the livetime ``t_\text{live}`` from the run information. The mass is the total detector mass, not the active mass. `yr` is the Julian year of Unitful (365.25 days).
+in `kg * yr`. Runs without a start key of the category are skipped, and so are runs that are not analysis runs of the category unless `is_analysis_run = false`; for the default dataset these are the runs that the run list `valid` does not contain. The detector mass ``m_\text{det}`` is taken from the channel information at the start file key of each run (the column `mass` of [`channelinfo`](@ref) with `extended = true`), the livetime ``t_\text{live}`` from the run information. The mass is the total detector mass, not the active mass. `yr` is the Julian year of Unitful (365.25 days).
 
 The runs are selected by
 
@@ -101,9 +101,9 @@ get_exposure.(l200, [:V99000A, :B99000A], :p02)
 
 ### Keywords
 
-* `cat`: Data category of the runs, default `:phy`. For a partition, the category of the partition is used. `get_exposure` throws an `ArgumentError` if the category is not a column of the run information.
+* `cat`: Data category of the runs, default `:phy`. For a partition, the category of the partition is used. `get_exposure` throws an `ArgumentError` if the category is not a column of the run information, as for a category that the dataset of `data` does not list.
 * `check_pf`: A property function that is applied to the channel information row of the detector at the start key of each run. A run counts only if it returns `true`. The default is `@pf $detector == DetectorId(det)`, which is always true for the selected detector. Any property function of the columns of [`channelinfo`](@ref) works, see [Property Functions and Expressions](@ref).
-* `is_analysis_run`: Accepted for the method with a table of run information, but it does not change the result. The runs of the table are all counted; select them beforehand, e.g. with `filterby(@pf $phy.is_analysis_run)`.
+* `is_analysis_run`: Count only the runs that are analysis runs of the category (`is_analysis_run` in the run information). Default is `true`. `false` counts every run of the table that has a start key.
 
 The following counts only the runs in which the detector has the usability `:on`:
 

docs/src/man/legenddata.md

@@ -151,7 +151,7 @@ The fields of the `LegendData` structure (`_config`, `_name`, `_dataset`) and th
 
 ### Datasets
 
-The keyword `dataset` names a run list in the metadata `datasets/runlists`. [`runinfo`](@ref) and [`analysis_runs`](@ref) read it and restrict the runs to those of the list. With the default `:default`, no run list is used and `runinfo` holds every run that the metadata describes.
+The keyword `dataset` names a run list in the metadata `datasets/runlists`. [`runinfo`](@ref) and [`analysis_runs`](@ref) read it and restrict the runs to those of the list. With the default `:default`, no run list is used and `runinfo` holds every run and category that the metadata describes. For any other dataset, `runinfo` has a column only for the categories that the run list names, and an entry of a category only for the runs that the list names for it (see [Run Information](@ref)).
 
 ```@example ldm
 keys(l200.metadata.datasets.runlists)

docs/src/man/metadata_validity.md

@@ -107,6 +107,28 @@ catch err
 end
 ```
 
+### Selection by Time
+
+A `PropsDB` of a [`LegendData`](@ref) (`l200.metadata`, `l200.par`, `l200.jlpar` and their sub-directories) also accepts a time `t::Unitful.Time`, in seconds since the Unix epoch, for example `Timestamp(filekey).unixtime * u"s"`. The call selects the file key of the DAQ cycle that contains `t` with [`find_filekey`](@ref) and then selects as with that file key: the category of that DAQ cycle is used, and the run overrides apply.
+
+```@example ldm
+using Unitful
+
+t = Timestamp(start_filekey(l200, (:p03, :r000, :cal))).unixtime * u"s" + 5u"s"
+find_filekey(l200, t)
+```
+
+```@example ldm
+channelmaps(t) == channelmaps(find_filekey(l200, t))
+```
+
+A `PropsDB` made by [`LegendDataManagement.AnyProps`](@ref) has no `LegendData` to look up the DAQ cycle and throws an `ArgumentError` for a time.
+
+```@example ldm
+anyprops = LegendDataManagement.AnyProps(data_path(l200.metadata))
+try anyprops.hardware.configuration.channelmaps(t) catch err sprint(showerror, err) end
+```
+
 ## The `validity.yaml` File
 
 A `validity.yaml` file is a list of entries. Each entry has the keys

docs/src/man/read_data.md

@@ -131,6 +131,13 @@ using Unitful
 dsp = read_ldata(l200, :jldsp, [1.6e9u"s", 1.7e9u"s"], det)
 ```
 
+A single `Unitful.Time` reads as a vector with that one timestamp. The tables of `l200_p02` (see below) have rows 1 s, 2 s and 3 s after the start of each DAQ cycle:
+
+```@example ldm
+t = Timestamp(start_filekey(l200_p02, (:p02, :r006, :cal))).unixtime * 1.0u"s" + 2.0u"s"
+read_ldata(:e_cusp, l200_p02, :jldsp, t, :V99000A)
+```
+
 ## Reading Runs, Periods and Partitions
 
 A category followed by a period and a run reads all DAQ cycles of the run and concatenates the rows in one table. The file keys are those that the metadata list for the run (`only(runinfo(l200, (period, run, category)).keys)`; see [Channel, Run and Partition Information](@ref)); a listed file that is not on disk is an error. These forms are available:
@@ -139,10 +146,12 @@ A category followed by a period and a run reads all DAQ cycles of the run and co
 * `(tier, category, period[, det])`: all runs of a period, from `runinfo(l200, period)`.
 * `(tier, category, partition, det)`: all runs of the partition of a detector, from `partitioninfo(l200, det, partition)`.
 * `(tier, category, runtable[, det])`: the runs listed in a table with the columns `period` and `run`, such as a row selection of `runinfo` or of a partition table.
-* `(tier, filekeys[, det])`: a vector of file keys.
+* `(tier, filekeys[, det])`: a vector of file keys, such as a [`DataSet`](@ref); all file keys of the set are read.
 
 If the file `l200.tier[tier, category, period, run, det]` exists, it holds the whole run of the detector and is read in place of the DAQ cycles. The keyword `ignore_keys` (default `:all`) drops the DAQ cycles that [`bad_filekeys`](@ref) lists for that key in the metadata, with `:none` keeping all of them. With `search_disk = true`, the file keys found on disk by [`search_disk`](@ref) are read instead of those from the metadata.
 
+A period, partition or run table reads only the runs whose entry of the category in `runinfo(l200)` lists DAQ cycle keys; a run whose category the dataset of `l200` does not list for it is skipped. This holds also with `search_disk = true`. An `ArgumentError` is thrown if no run is left, or if a run of the table is not in `runinfo(l200)`. A single run `(tier, category, period, run[, det])` is read as requested.
+
 The following examples use a second `LegendData` object, `l200_p02`, whose tier directory holds `jldsp` tables for all DAQ cycles of period 2, run 6, listed by `runinfo`.
 
 ```@example ldm
@@ -164,12 +173,16 @@ runs = filter(r -> r.run == DataRun(6), runinfo(l200_p02, DataPeriod(2)))
 length(read_ldata(l200_p02, :jldsp, :cal, runs, :V99000A))
 ```
 
-A vector of file keys reads those DAQ cycles only.
+A vector of file keys reads those DAQ cycles only. A [`DataSet`](@ref) is a vector of file keys, so `run_keys` reads the whole run.
 
 ```@example ldm
 length(read_ldata(l200_p02, :jldsp, run_keys[1:2], :V99000A))
 ```
 
+```@example ldm
+length(read_ldata(l200_p02, :jldsp, run_keys, :V99000A))
+```
+
 The partition and period forms work in the same way, provided that all runs have files:
 
 ```julia

ext/LegendDataManagementLegendHDF5IOExt.jl

@@ -35,8 +35,8 @@ const DetectorSel = Union{DetectorId, Nothing}
 const ChannelSel = Union{DetectorId, AbstractString, Nothing}
 # The rows of a table: every row, or only those whose time is among the given ones.
 const TimestampSel = Union{Nothing, AbstractVector{<:Unitful.Time}}
-# What an element of a selection may be: a selector, a filekey, a run table, or a vector.
-const RSelLike = Union{AbstractDataSelectorLike, FileKey, AbstractVector, Nothing}
+# What an element of a selection may be: a selector, a filekey, a time, a run table, or a vector.
+const RSelLike = Union{AbstractDataSelectorLike, FileKey, Unitful.Time, AbstractVector, Nothing}
 const PossibleDataSelectors = [DataTier, DataCategory, DataPeriod, DataRun, DataPartition, DetectorId]
 
 
@@ -400,8 +400,8 @@ The selectors, as a tuple or as separate arguments, can have the forms
 
 * `(tier, filekey)`, `(tier, filekey, det)`: one DAQ cycle, all channels or one detector.
 * `(tier, filekey, timestamps)`, `(tier, filekey, timestamps, det)`: the rows of one DAQ cycle with the given times.
-* `(tier, timestamps)`, `(tier, timestamps, det)`: the rows with the given times, each read from the DAQ cycle that contains it.
-* `(tier, filekeys)`, `(tier, filekeys, det)`: several DAQ cycles.
+* `(tier, timestamps)`, `(tier, timestamps, det)`: the rows with the given times, each read from the DAQ cycle that contains it; a single time `t` reads as `[t]`.
+* `(tier, filekeys)`, `(tier, filekeys, det)`: several DAQ cycles, e.g. all keys of a [`DataSet`](@ref).
 * `(tier, category, period)`, `(tier, category, period, det)`, `(tier, category, period, run)`, `(tier, category, period, run, det)`: runs of a period or a run.
 * `(tier, category, partition, det)`: the runs of a partition of a detector.
 * `(tier, category, runtable)`, `(tier, category, runtable, det)`: the runs listed in a table with the columns `period` and `run`.
@@ -473,6 +473,10 @@ function LegendDataManagement.read_ldata(f::Base.Callable, data::LegendData, rse
 end
 LegendDataManagement.read_ldata(f::Base.Callable, data::LegendData, rsel::Tuple{DataTier, AbstractVector{<:Unitful.Time}}; kwargs...) =
     LegendDataManagement.read_ldata(f, data, (rsel[1], rsel[2], nothing); kwargs...)
+LegendDataManagement.read_ldata(f::Base.Callable, data::LegendData, rsel::Tuple{DataTier, Unitful.Time, DetectorSel}; kwargs...) =
+    LegendDataManagement.read_ldata(f, data, (rsel[1], [rsel[2]], rsel[3]); kwargs...)
+LegendDataManagement.read_ldata(f::Base.Callable, data::LegendData, rsel::Tuple{DataTier, Unitful.Time}; kwargs...) =
+    LegendDataManagement.read_ldata(f, data, (rsel[1], [rsel[2]], nothing); kwargs...)
 
 
 LegendDataManagement.read_ldata(f::Base.Callable, data::LegendData, rsel::Tuple{DataTier, FileKey}; kwargs...) =
@@ -572,17 +576,21 @@ end
 const _partinfo_required_cols = NamedTuple{(:period, :run), Tuple{DataPeriod, DataRun}}
 
 function LegendDataManagement.read_ldata(f::Base.Callable, data::LegendData, rsel::Tuple{DataTier, DataCategory, Table{_partinfo_required_cols}, DetectorSel}; parallel::Bool=false, wpool::WorkerPool=default_worker_pool(), kwargs...)
-    p = Progress(length(rsel[3]), desc="Reading from $(length(rsel[3])) runs", showspeed=true)
+    # A run whose entry of the category in the runinfo has no DAQ cycle keys, e.g. one whose
+    # category the dataset does not list, has nothing to read; a run not in the runinfo throws.
+    runs = filter(r -> !isempty(only(runinfo(data, (r.period, r.run, rsel[2]))).keys), rsel[3])
+    isempty(runs) && throw(ArgumentError("No run of the run table has $(rsel[2]) data in dataset $(data.dataset)"))
+    p = Progress(length(runs), desc="Reading from $(length(runs)) runs", showspeed=true)
     lflatten(if parallel
                 # TODO: Check if wpool is connected via :master_worker if myid() != 1
-                @debug "Parallel read with $(length(workers())) workers from $(length(rsel[3])) runs"
+                @debug "Parallel read with $(length(workers())) workers from $(length(runs)) runs"
                 _init_workers(wpool)
-                progress_pmap(wpool, rsel[3]; progress=p) do r
+                progress_pmap(wpool, runs; progress=p) do r
                     LegendDataManagement.read_ldata(f, data, (rsel[1], rsel[2], r.period, r.run, rsel[4]); parallel, wpool, kwargs...)
                 end
             else
-                @debug "Sequential read from $(length(rsel[3])) runs"
-                progress_map(rsel[3]; progress=p) do r
+                @debug "Sequential read from $(length(runs)) runs"
+                progress_map(runs; progress=p) do r
                     LegendDataManagement.read_ldata(f, data, (rsel[1], rsel[2], r.period, r.run, rsel[4]); parallel=false, kwargs...)
                 end
             end)

src/data_io.jl

@@ -12,9 +12,10 @@ Reads `lh5` data from disk for a given set of `selectors`.
 After reading in, a `PropertyFunction` `f` can be applied to the data. If a
 tuple of `Symbol`s is given, the properties from the tuple are selected; a
 single `Symbol` selects one property. `det` is a `DetectorId`. The selectors
-are a tier followed by a file key, a vector of file keys or timestamps, or a
-data category with a period, a run, a partition or a run table, optionally
-followed by a detector or a vector of detectors.
+are a tier followed by a file key, a vector of file keys such as a
+[`DataSet`](@ref) (all its keys are read) or timestamps, or a data category
+with a period, a run, a partition or a run table, optionally followed by a
+detector or a vector of detectors.
 
 The methods that read data are provided by an extension that is loaded together
 with `LegendHDF5IO.jl`; without it the call throws an `ErrorException`. The
@@ -24,8 +25,9 @@ extension is described in the docstring of the method for
 A vector of `Unitful.Time` timestamps in place of a `FileKey` reads only the rows with those timestamps, in time order:
 each is looked up with [`find_filekey`](@ref) in the DAQ cycle that contains it, and only those rows of the cycle's file
 are read. The time of a row is its `timestamp` column, or `tstart` in an event tier. A table without one of the timestamps
-is an error, or is skipped with `ignore_missing`. With a `FileKey` and the timestamps, `(tier, filekey, timestamps, det)`,
-the cycle is given instead of looked up and the rows follow the timestamps.
+is an error, or is skipped with `ignore_missing`. A single timestamp `t` in place of the vector reads as `[t]`. With a
+`FileKey` and the timestamps, `(tier, filekey, timestamps, det)`, the cycle is given instead of looked up and the rows
+follow the timestamps.
 
 The `filterby` kwarg takes a `PropertyFunction` selecting rows of the tier being read. Given a `tier => PropertyFunction` pair instead,
 the function is evaluated on `tier` and the resulting row selection is applied to the tier being read. Both tiers must hold one row per
@@ -34,7 +36,10 @@ trigger of the same detector, so that their rows correspond by position; a row-c
 A run, given by category, period and run or through a period, partition or run table, is read from the file keys
 listed for it in the metadata, `only(runinfo(data, (period, run, category)).keys)`; a listed file key that is not
 on disk is an error. The keywords `search_disk` and `ignore_keys` control which file keys these are.
-A run with no file keys left is an error. A per-run file of the detector, `data.tier[tier, category, period, run, det]`,
+A run with no file keys left is an error. A period, partition or run table reads only its runs whose `category` entry
+in `runinfo(data)` lists file keys, e.g. not a run whose category the dataset of `data` does not list, also with
+`search_disk`; it is an error if no such run is left, or if a run is not in `runinfo(data)`.
+A per-run file of the detector, `data.tier[tier, category, period, run, det]`,
 holds the whole run and is read in place of the file keys whenever it exists.
 
 ## Keywords