[ethanbb]

This implements temporal_smooth for the smooth_sigma_time registration option, which is currently broken since it doesn't handle torch input. To get the same reflect-padding behavior as the scipy default (technically this matches the 'mirror' mode), and to avoid depending on torchimage, I first pad and then convolve with a Gaussian kernel using conv1d.

[joannercsheppard]

@ethanbb would it be possible to run the tests on this PR? Thanks!

[ethanbb]

Hi,

OK, I got the tests to run on Windows. There was one failure - I don't have time at the moment to investigate whether it's related, but I'm pasting the log below. I'm also going to give it a try on Linux.

pytest output, win32, Python 3.12

(C:\DataDrive\Ethan\conda_envs\caiman-dev) Z:\code\CalciumImaging\suite2p>pytest -vs
========================================================================= test session starts =========================================================================
platform win32 -- Python 3.12.12, pytest-9.0.2, pluggy-1.6.0 -- C:\DataDrive\Ethan\conda_envs\caiman-dev\python.exe
cachedir: .pytest_cache
PySide6 6.9.2 -- Qt runtime 6.9.2 -- Qt compiled 6.9.2
rootdir: Z:\code\CalciumImaging\suite2p
plugins: anyio-4.11.0, jaxtyping-0.3.9, cov-7.0.0, qt-4.5.0, zarr-3.1.5
collected 26 items
tests/regression/test_classification_pipeline.py::test_classification_output Cached inputs not found. Downloading now...
Downloading: https://osf.io/download/6984d9ca54398fbd42f5575d

100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 966M/966M [00:41<00:00, 24.1MB/s]
Downloading: https://osf.io/download/69890549bffbac83ffe2530a

100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 303M/303M [00:13<00:00, 24.1MB/s]

PASSED

tests/regression/test_detection_pipeline.py::test_detection_output_1plane1chan PASSED

tests/regression/test_detection_pipeline.py::test_detection_output_2plane2chan FAILED

tests/regression/test_extraction_pipeline.py::test_pre_process_baseline PASSED

tests/regression/test_extraction_pipeline.py::test_extraction_output_1plane1chan PASSED

tests/regression/test_extraction_pipeline.py::test_extraction_output_2plane2chan PASSED

tests/regression/test_full_pipeline.py::test_1plane_1chan_with_batches_metrics_and_exported_to_nwb_format F

iscell

stat

PASSED

tests/regression/test_full_pipeline.py::test_2plane_2chan_with_batches F

iscell

F_chan2

Fneu_chan2

F

iscell

F_chan2

Fneu_chan2

PASSED

tests/smoke/test_cli.py::test_cli_help_test_appears_when_suite2p_is_called PASSED

tests/smoke/test_cli.py::test_cli_help_test_appears_when_suite2p_is_called_locally PASSED

tests/smoke/test_file_loading.py::test_bruker PASSED

tests/smoke/test_file_loading.py::test_h5_file_is_processed_end_to_end PASSED

tests/smoke/test_run_s2p.py::test_do_registration_do_roi_detect_settings_check_timing PASSED

tests/test_io.py::test_h5_to_binary_produces_nonnegative_output_data PASSED

tests/test_io.py::test_that_bin_movie_without_badframes_results_in_a_same_size_array PASSED

tests/test_io.py::test_that_bin_movie_with_badframes_results_in_a_smaller_array PASSED

tests/test_io.py::test_that_binaryfile_data_is_repeatable PASSED

tests/test_io.py::test_nwb_round_trip[1plane1chan1500] data\test_outputs\1plane1chan1500\suite2p

PASSED

tests/test_io.py::test_nwb_round_trip[2plane2chan1500] data\test_outputs\2plane2chan1500\suite2p

PASSED

tests/test_io.py::test_get_suite2p_path[D:/kjkcc/jodendopn/suite2p/ncconoc/onowcno-D:/kjkcc/jodendopn/suite2p-True] PASSED

tests/test_io.py::test_get_suite2p_path[/home/bla/kjkcc/jodendopn/suite2p/settings.npy-/home/bla/kjkcc/jodendopn/suite2p-True] PASSED

tests/test_io.py::test_get_suite2p_path[/etc/bla/kjkcc/jodendopn/suite2p-/etc/bla/kjkcc/jodendopn/suite2p-True] PASSED

tests/test_io.py::test_get_suite2p_path[/etc/bla/kjkcc/jodendopn/--False] PASSED

tests/test_registration.py::test_positive_bidiphase_shift_shifts_every_other_line PASSED

tests/test_registration.py::test_negative_bidiphase_shift_shifts_every_other_line PASSED

tests/test_registration.py::test_transform_data_mps_cpu_consistency SKIPPED (MPS not available)
============================================================================== FAILURES ===============================================================================

__________________________________________________________________ test_detection_output_2plane2chan __________________________________________________________________
test_settings = ({'batch_size': 500, 'bruker_bidirectional': False, 'data_path': [WindowsPath('data/test_inputs')], 'dx': None, ...}, ...': 'sparsery', 'bin_size': None, 'block_size': (64, 64), 'cellpose_chan2': False, ...}, 'diameter': [12.0, 12.0], ...})
def test_detection_output_2plane2chan(test_settings):
    """
    Regression test for 2 planes, 2 channels detection.
    Runs detection exactly like generate_test_data.py and compares outputs.
    """
    db, settings = test_settings  # Unpack the tuple
    db = {**db, **settings}  # Merge for DetectionTestUtils
    db.update({'file_list': ['input.tif'], 'nchannels': 2, 'nplanes': 2})

    detection_dir = Path(db['data_path'][0]).joinpath('detection')

    # Prepare settings exactly like generate_test_data.py
    settings = utils.DetectionTestUtils.prepare(
        db,
        [
            [detection_dir.joinpath('pre_registered01.npy'), detection_dir.joinpath('pre_registered02.npy')],
            [detection_dir.joinpath('pre_registered11.npy'), detection_dir.joinpath('pre_registered12.npy')]
        ],
        (404, 360)
    )

    # Load channel 2 mean images like generate_test_data.py
    settings[0]['meanImg_chan2'] = np.load(detection_dir.joinpath('meanImg_chan2p0.npy'))
    settings[1]['meanImg_chan2'] = np.load(detection_dir.joinpath('meanImg_chan2p1.npy'))

    # Run detection for each plane exactly like generate_test_data.py
    for i, op in enumerate(settings):
        with suite2p.io.BinaryFile(Ly=op['Ly'], Lx=op['Lx'], filename=op['reg_file']) as f_reg:
            op['neuropil_extract'] = True
            _, stat, _ = suite2p.detection.detection_wrapper(f_reg, settings=op)
            cell_masks, neuropil_masks = masks.create_masks(stat, op['Ly'], op['Lx'], neuropil_extract=True)

        # Load expected output for this plane
        expected = np.load(
            Path(db['data_path'][0]).parent.joinpath(f'test_outputs/detection/expected_detect_output_2p2c{i}.npy'),
            allow_pickle=True
        )[()]

        # Compare outputs


      assert all(np.allclose(a, b, rtol=1e-4, atol=5e-2) for a, b in zip(cell_masks, expected['cell_masks']))


               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

tests\regression\test_detection_pipeline.py:89:

tests\regression\test_detection_pipeline.py:89: in 

assert all(np.allclose(a, b, rtol=1e-4, atol=5e-2) for a, b in zip(cell_masks, expected['cell_masks']))

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

C:\DataDrive\Ethan\conda_envs\caiman-dev\Lib\site-packages\numpy_core\numeric.py:2376: in allclose

res = all(isclose(a, b, rtol=rtol, atol=atol, equal_nan=equal_nan))

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

a = (array([54317, 54318, 54720, 54721, 55121, 55122, 55123, 55124, 55125,

55525, 55528, 55529, 55927, 55928, 55929...0.00277358, 0.00251011, 0.00537183, 0.00383376, 0.00593033,

0.00450415, 0.00569687, 0.00126897], dtype=float32))

b = (array([54317, 54318, 54720, 54721, 55121, 55122, 55123, 55124, 55125,

55525, 55528, 55529, 55927, 55928, 55929...204, 0.00201884,

0.00280124, 0.00253515, 0.003872  , 0.00598948, 0.00575369,

0.00128163], dtype=float32))

rtol = 0.0001, atol = 0.05, equal_nan = False
@array_function_dispatch(_isclose_dispatcher)
def isclose(a, b, rtol=1.e-5, atol=1.e-8, equal_nan=False):
    """
    Returns a boolean array where two arrays are element-wise equal within a
    tolerance.

    The tolerance values are positive, typically very small numbers.  The
    relative difference (`rtol` * abs(`b`)) and the absolute difference
    `atol` are added together to compare against the absolute difference
    between `a` and `b`.

    .. warning:: The default `atol` is not appropriate for comparing numbers
                 with magnitudes much smaller than one (see Notes).

    Parameters
    ----------
    a, b : array_like
        Input arrays to compare.
    rtol : array_like
        The relative tolerance parameter (see Notes).
    atol : array_like
        The absolute tolerance parameter (see Notes).
    equal_nan : bool
        Whether to compare NaN's as equal.  If True, NaN's in `a` will be
        considered equal to NaN's in `b` in the output array.

    Returns
    -------
    y : array_like
        Returns a boolean array of where `a` and `b` are equal within the
        given tolerance. If both `a` and `b` are scalars, returns a single
        boolean value.

    See Also
    --------
    allclose
    math.isclose

    Notes
    -----
    For finite values, isclose uses the following equation to test whether
    two floating point values are equivalent.::

     absolute(a - b) <= (atol + rtol * absolute(b))

    Unlike the built-in `math.isclose`, the above equation is not symmetric
    in `a` and `b` -- it assumes `b` is the reference value -- so that
    `isclose(a, b)` might be different from `isclose(b, a)`.

    The default value of `atol` is not appropriate when the reference value
    `b` has magnitude smaller than one. For example, it is unlikely that
    ``a = 1e-9`` and ``b = 2e-9`` should be considered "close", yet
    ``isclose(1e-9, 2e-9)`` is ``True`` with default settings. Be sure
    to select `atol` for the use case at hand, especially for defining the
    threshold below which a non-zero value in `a` will be considered "close"
    to a very small or zero value in `b`.

    `isclose` is not defined for non-numeric data types.
    :class:`bool` is considered a numeric data-type for this purpose.

    Examples
    --------
    >>> import numpy as np
    >>> np.isclose([1e10,1e-7], [1.00001e10,1e-8])
    array([ True, False])

    >>> np.isclose([1e10,1e-8], [1.00001e10,1e-9])
    array([ True, True])

    >>> np.isclose([1e10,1e-8], [1.0001e10,1e-9])
    array([False,  True])

    >>> np.isclose([1.0, np.nan], [1.0, np.nan])
    array([ True, False])

    >>> np.isclose([1.0, np.nan], [1.0, np.nan], equal_nan=True)
    array([ True, True])

    >>> np.isclose([1e-8, 1e-7], [0.0, 0.0])
    array([ True, False])

    >>> np.isclose([1e-100, 1e-7], [0.0, 0.0], atol=0.0)
    array([False, False])

    >>> np.isclose([1e-10, 1e-10], [1e-20, 0.0])
    array([ True,  True])

    >>> np.isclose([1e-10, 1e-10], [1e-20, 0.999999e-10], atol=0.0)
    array([False,  True])

    """
    # Turn all but python scalars into arrays.
    x, y, atol, rtol = (
        a if isinstance(a, (int, float, complex)) else asanyarray(a)
        for a in (a, b, atol, rtol))

    # Make sure y is an inexact type to avoid bad behavior on abs(MIN_INT).
    # This will cause casting of x later. Also, make sure to allow subclasses
    # (e.g., for numpy.ma).
    # NOTE: We explicitly allow timedelta, which used to work. This could
    #       possibly be deprecated. See also gh-18286.
    #       timedelta works if `atol` is an integer or also a timedelta.
    #       Although, the default tolerances are unlikely to be useful
    if (dtype := getattr(y, "dtype", None)) is not None and dtype.kind != "m":
        dt = multiarray.result_type(y, 1.)
        y = asanyarray(y, dtype=dt)
    elif isinstance(y, int):
        y = float(y)

    # atol and rtol can be arrays
    if not (np.all(np.isfinite(atol)) and np.all(np.isfinite(rtol))):
        err_s = np.geterr()["invalid"]
        err_msg = f"One of rtol or atol is not valid, atol: {atol}, rtol: {rtol}"

        if err_s == "warn":
            warnings.warn(err_msg, RuntimeWarning, stacklevel=2)
        elif err_s == "raise":
            raise FloatingPointError(err_msg)
        elif err_s == "print":
            print(err_msg)

    with errstate(invalid='ignore'):


      result = (less_equal(abs(x - y), atol + rtol * abs(y))


                                 ^^^^^
                  & isfinite(y)
                  | (x == y))

E           ValueError: operands could not be broadcast together with shapes (2,178) (2,176)
C:\DataDrive\Ethan\conda_envs\caiman-dev\Lib\site-packages\numpy_core\numeric.py:2507: ValueError

========================================================================== warnings summary ===========================================================================

tests/regression/test_classification_pipeline.py::test_classification_output

tests/regression/test_full_pipeline.py::test_1plane_1chan_with_batches_metrics_and_exported_to_nwb_format

tests/regression/test_full_pipeline.py::test_2plane_2chan_with_batches

tests/regression/test_full_pipeline.py::test_2plane_2chan_with_batches

tests/smoke/test_file_loading.py::test_bruker

tests/smoke/test_file_loading.py::test_h5_file_is_processed_end_to_end

tests/smoke/test_run_s2p.py::test_do_registration_do_roi_detect_settings_check_timing

tests/smoke/test_run_s2p.py::test_do_registration_do_roi_detect_settings_check_timing

C:\DataDrive\Ethan\conda_envs\caiman-dev\Lib\site-packages\numpy\lib_format_impl.py:838: VisibleDeprecationWarning: dtype(): align should be passed as Python or NumPy boolean but got align=0. Did you mean to pass a tuple to create a subarray type? (Deprecated NumPy 2.4)

array = pickle.load(fp, **pickle_kwargs)
tests/regression/test_extraction_pipeline.py::test_extraction_output_1plane1chan

Z:\code\CalciumImaging\suite2p\suite2p\extraction\extract.py:69: UserWarning: Sparse CSR tensor support is in beta state. If you miss a functionality in the sparse tensor support, please submit a feature request to https://github.com/pytorch/pytorch/issues. (Triggered internally at C:\bld\libtorch_1753839109265\work\aten\src\ATen\SparseCsrTensorImpl.cpp:55.)

nmasks = nmasks.to_sparse_csc()
tests/regression/test_extraction_pipeline.py::test_extraction_output_1plane1chan

Z:\code\CalciumImaging\suite2p\suite2p\extraction\extract.py:89: UserWarning: The given NumPy array is not writable, and PyTorch does not support non-writable tensors. This means writing to this tensor will result in undefined behavior. You may want to copy the array to protect its data or make it writable before converting it to a tensor. This type of warning will be suppressed for the rest of this program. (Triggered internally at C:\bld\libtorch_1753839109265\work\torch\csrc\utils\tensor_numpy.cpp:209.)

data = torch.from_numpy(f_in[tstart : tend]).to(device)
tests/regression/test_full_pipeline.py::test_1plane_1chan_with_batches_metrics_and_exported_to_nwb_format

tests/regression/test_full_pipeline.py::test_1plane_1chan_with_batches_metrics_and_exported_to_nwb_format

tests/test_io.py::test_nwb_round_trip[1plane1chan1500]

tests/test_io.py::test_nwb_round_trip[1plane1chan1500]

tests/test_io.py::test_nwb_round_trip[1plane1chan1500]

tests/test_io.py::test_nwb_round_trip[2plane2chan1500]

tests/test_io.py::test_nwb_round_trip[2plane2chan1500]

tests/test_io.py::test_nwb_round_trip[2plane2chan1500]

C:\DataDrive\Ethan\conda_envs\caiman-dev\Lib\site-packages\hdmf\utils.py:578: DeprecationWarning: The 'manufacturer' field is deprecated. Instead, use DeviceModel.manufacturer and link to that DeviceModel from this Device.

return func(args[0], **pargs)
tests/regression/test_full_pipeline.py::test_1plane_1chan_with_batches_metrics_and_exported_to_nwb_format

tests/test_io.py::test_nwb_round_trip[1plane1chan1500]

tests/test_io.py::test_nwb_round_trip[2plane2chan1500]

C:\DataDrive\Ethan\conda_envs\caiman-dev\Lib\site-packages\hdmf\container.py:521: UserWarning: The linked table for DynamicTableRegion 'rois' does not share an ancestor with the DynamicTableRegion.

child._validate_on_set_parent()
tests/regression/test_full_pipeline.py::test_1plane_1chan_with_batches_metrics_and_exported_to_nwb_format

tests/test_io.py::test_nwb_round_trip[1plane1chan1500]

tests/test_io.py::test_nwb_round_trip[2plane2chan1500]

Z:\code\CalciumImaging\suite2p\suite2p\io\nwb.py:546: FutureWarning: Images.init: Using positional arguments for this method is discouraged and will be deprecated in a future major release. Please use keyword arguments to ensure future compatibility.

images = Images("Backgrounds_%d" % iplane)
tests/smoke/test_file_loading.py::test_bruker

Z:\code\CalciumImaging\suite2p\suite2p\run_s2p.py:230: DeprecationWarning: The 'warn' method is deprecated, use 'warning' instead

logger.warn("WARNING: number of frames < 200, unpredictable behaviors may occur")
tests/smoke/test_file_loading.py::test_h5_file_is_processed_end_to_end

tests/smoke/test_run_s2p.py::test_do_registration_do_roi_detect_settings_check_timing

Z:\code\CalciumImaging\suite2p\suite2p\pipeline_s2p.py:223: DeprecationWarning: The 'warn' method is deprecated, use 'warning' instead

logger.warn("WARNING: skipping spike detection (settings['do_deconvolution']=False)")
tests/smoke/test_run_s2p.py::test_do_registration_do_roi_detect_settings_check_timing

tests/smoke/test_run_s2p.py::test_do_registration_do_roi_detect_settings_check_timing

Z:\code\CalciumImaging\suite2p\suite2p\pipeline_s2p.py:136: DeprecationWarning: The 'warn' method is deprecated, use 'warning' instead

logger.warn("WARNING: skipping cell detection (settings['run']['do_detection']=False)")
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html

======================================================================= short test summary info =======================================================================

FAILED tests/regression/test_detection_pipeline.py::test_detection_output_2plane2chan - ValueError: operands could not be broadcast together with shapes (2,178) (2,176)

================================================== 1 failed, 24 passed, 1 skipped, 29 warnings in 239.91s (0:03:59) ===================================================

[ethanbb]

Some different errors in the Linux run:

pytest output, linux, Python 3.11

(suite2p-test) ethan@treachery:/synology/code/CalciumImaging/suite2p$ pytest -vs
========================================================================= test session starts ==========================================================================
platform linux -- Python 3.11.18, pytest-9.2.0, pluggy-1.7.0 -- /home/ethan/mambaforge/envs/suite2p-test/bin/python3.11
cachedir: .pytest_cache
PyQt6 6.12.0 -- Qt runtime 6.12.0 -- Qt compiled 6.12.0
rootdir: /synology/code/CalciumImaging/suite2p
plugins: qt-4.5.0
collected 26 items
tests/regression/test_classification_pipeline.py::test_classification_output PASSED

tests/regression/test_detection_pipeline.py::test_detection_output_1plane1chan PASSED

tests/regression/test_detection_pipeline.py::test_detection_output_2plane2chan PASSED

tests/regression/test_extraction_pipeline.py::test_pre_process_baseline PASSED

tests/regression/test_extraction_pipeline.py::test_extraction_output_1plane1chan PASSED

tests/regression/test_extraction_pipeline.py::test_extraction_output_2plane2chan PASSED

tests/regression/test_full_pipeline.py::test_1plane_1chan_with_batches_metrics_and_exported_to_nwb_format FAILED

tests/regression/test_full_pipeline.py::test_2plane_2chan_with_batches F

iscell

F_chan2

Fneu_chan2

F

iscell

F_chan2

Fneu_chan2

PASSED

tests/smoke/test_cli.py::test_cli_help_test_appears_when_suite2p_is_called PASSED

tests/smoke/test_cli.py::test_cli_help_test_appears_when_suite2p_is_called_locally PASSED

tests/smoke/test_file_loading.py::test_bruker PASSED

tests/smoke/test_file_loading.py::test_h5_file_is_processed_end_to_end PASSED

tests/smoke/test_run_s2p.py::test_do_registration_do_roi_detect_settings_check_timing PASSED

tests/test_io.py::test_h5_to_binary_produces_nonnegative_output_data PASSED

tests/test_io.py::test_that_bin_movie_without_badframes_results_in_a_same_size_array PASSED

tests/test_io.py::test_that_bin_movie_with_badframes_results_in_a_smaller_array PASSED

tests/test_io.py::test_that_binaryfile_data_is_repeatable PASSED

tests/test_io.py::test_nwb_round_trip[1plane1chan1500] data/test_outputs/1plane1chan1500/suite2p

FAILED

tests/test_io.py::test_nwb_round_trip[2plane2chan1500] data/test_outputs/2plane2chan1500/suite2p

FAILED

tests/test_io.py::test_get_suite2p_path[D:/kjkcc/jodendopn/suite2p/ncconoc/onowcno-D:/kjkcc/jodendopn/suite2p-True] PASSED

tests/test_io.py::test_get_suite2p_path[/home/bla/kjkcc/jodendopn/suite2p/settings.npy-/home/bla/kjkcc/jodendopn/suite2p-True] PASSED

tests/test_io.py::test_get_suite2p_path[/etc/bla/kjkcc/jodendopn/suite2p-/etc/bla/kjkcc/jodendopn/suite2p-True] PASSED

tests/test_io.py::test_get_suite2p_path[/etc/bla/kjkcc/jodendopn/--False] PASSED

tests/test_registration.py::test_positive_bidiphase_shift_shifts_every_other_line PASSED

tests/test_registration.py::test_negative_bidiphase_shift_shifts_every_other_line PASSED

tests/test_registration.py::test_transform_data_mps_cpu_consistency SKIPPED (MPS not available)
=============================================================================== FAILURES ===============================================================================

__________________________________________________ test_1plane_1chan_with_batches_metrics_and_exported_to_nwb_format ___________________________________________________
test_settings = ({'data_path': [PosixPath('data/test_inputs')], 'look_one_level_down': False, 'input_format': 'tif', 'keep_movie_raw': False, ...}, {'torch_device': 'cuda', 'tau': 1.0, 'fs': 10.0, 'diameter': [12.0, 12.0], ...})
def test_1plane_1chan_with_batches_metrics_and_exported_to_nwb_format(test_settings):
    """
    Tests for case with 1 plane and 1 channel with multiple batches. Results are saved to nwb format
    then checked to see if it contains the necessary parts for use with GUI.
    """
    db, settings = test_settings
    db, settings = utils.FullPipelineTestUtils.initialize_settings_test1plane_1chan_with_batches(db.copy(), settings.copy())


  suite2p.run_s2p(settings=settings, db=db)


tests/regression/test_full_pipeline.py:18:

suite2p/run_s2p.py:471: in run_s2p

io.save_nwb(save_folder)

suite2p/io/nwb.py:404: in save_nwb

image_series = TwoPhotonSeries(

/home/ethan/mambaforge/envs/suite2p-test/lib/python3.11/site-packages/hdmf/utils.py:641: in func_call

return func(args[0], **pargs)

^^^^^^^^^^^^^^^^^^^^^^

/home/ethan/mambaforge/envs/suite2p-test/lib/python3.11/site-packages/pynwb/ophys.py:272: in init

super().init(**kwargs)

/home/ethan/mambaforge/envs/suite2p-test/lib/python3.11/site-packages/hdmf/utils.py:641: in func_call

return func(args[0], **pargs)

^^^^^^^^^^^^^^^^^^^^^^

/home/ethan/mambaforge/envs/suite2p-test/lib/python3.11/site-packages/pynwb/image.py:160: in init

self._error_on_new_pass_on_construct(error_msg=error_msg)

self = TwoPhotonSeries pynwb.ophys.TwoPhotonSeries at 0x134829323284816

Fields:

comments: no comments

conversion: 1.0

d...rate: 10.0

resolution: -1.0

starting_frame: [0]

starting_time: 0.0

starting_time_unit: seconds

unit: unknown
error_msg = "TwoPhotonSeries 'TwoPhotonSeries': num_samples should be set when format='external' and rate is used for timing, because data is empty and its length cannot be used to determine the number of frames."
def _error_on_new_pass_on_construct(self, error_msg: str):
    """
    Raise an error when a check is violated on instance creation.
    When reading from a file, do nothing, ensuring that files with
    invalid data or deprecated neurodata types can be read.
    """
    if not self._in_construct_mode:


      raise ValueError(error_msg)


E           ValueError: TwoPhotonSeries 'TwoPhotonSeries': num_samples should be set when format='external' and rate is used for timing, because data is empty and its length cannot be used to determine the number of frames.
/home/ethan/mambaforge/envs/suite2p-test/lib/python3.11/site-packages/pynwb/core.py:59: ValueError

-------------------------------------------------------------------------- Captured log call ---------------------------------------------------------------------------

WARNING  suite2p.io.save:save.py:67 'lines' is None, replacing with empty array

WARNING  suite2p.io.save:save.py:67 'dy' is None, replacing with empty array

WARNING  suite2p.io.save:save.py:67 'dx' is None, replacing with empty array

WARNING  suite2p.io.save:save.py:67 'ignore_flyback' is None, replacing with empty array

WARNING  suite2p.io.save:save.py:67 'subfolders' is None, replacing with empty array

WARNING  suite2p.io.save:save.py:75 redcell is None, replacing with empty array

_________________________________________________________________ test_nwb_round_trip[1plane1chan1500] _________________________________________________________________
data_folder = '1plane1chan1500'
@pytest.mark.parametrize(
    "data_folder",
    [
        ("1plane1chan1500"),
        ("2plane2chan1500"),
        #("bruker"),
    ],
)
def test_nwb_round_trip(data_folder):
    """Test saving Suite2p outputs to NWB and loading them back."""

    # Define the path to the suite2p folder
    save_folder = Path("data").joinpath("test_outputs", data_folder, "suite2p")
    print(save_folder)

    # Load expected data from plane folders
    plane_folders = natsorted([
        f for f in save_folder.iterdir()
        if f.is_dir() and f.name.startswith("plane")
    ])

    # Concatenate data across planes
    expected_F = np.concatenate([
        np.load(plane_dir.joinpath("F.npy"), allow_pickle=True)
        for plane_dir in plane_folders
    ], axis=0)

    expected_Fneu = np.concatenate([
        np.load(plane_dir.joinpath("Fneu.npy"), allow_pickle=True)
        for plane_dir in plane_folders
    ], axis=0)

    expected_spks = np.concatenate([
        np.load(plane_dir.joinpath("spks.npy"), allow_pickle=True)
        for plane_dir in plane_folders
    ], axis=0)

    expected_iscell = np.concatenate([
        np.load(plane_dir.joinpath("iscell.npy"), allow_pickle=True)
        for plane_dir in plane_folders
    ], axis=0)

    expected_stat = np.concatenate([
        np.load(plane_dir.joinpath("stat.npy"), allow_pickle=True)
        for plane_dir in plane_folders
    ], axis=0)


    # Save as NWB file


  save_nwb(save_folder)


tests/test_io.py:170:

suite2p/io/nwb.py:404: in save_nwb

image_series = TwoPhotonSeries(

/home/ethan/mambaforge/envs/suite2p-test/lib/python3.11/site-packages/hdmf/utils.py:641: in func_call

return func(args[0], **pargs)

^^^^^^^^^^^^^^^^^^^^^^

/home/ethan/mambaforge/envs/suite2p-test/lib/python3.11/site-packages/pynwb/ophys.py:272: in init

super().init(**kwargs)

/home/ethan/mambaforge/envs/suite2p-test/lib/python3.11/site-packages/hdmf/utils.py:641: in func_call

return func(args[0], **pargs)

^^^^^^^^^^^^^^^^^^^^^^

/home/ethan/mambaforge/envs/suite2p-test/lib/python3.11/site-packages/pynwb/image.py:160: in init

self._error_on_new_pass_on_construct(error_msg=error_msg)

self = TwoPhotonSeries pynwb.ophys.TwoPhotonSeries at 0x134828982394128

Fields:

comments: no comments

conversion: 1.0

d...rate: 10.0

resolution: -1.0

starting_frame: [0]

starting_time: 0.0

starting_time_unit: seconds

unit: unknown
error_msg = "TwoPhotonSeries 'TwoPhotonSeries': num_samples should be set when format='external' and rate is used for timing, because data is empty and its length cannot be used to determine the number of frames."
def _error_on_new_pass_on_construct(self, error_msg: str):
    """
    Raise an error when a check is violated on instance creation.
    When reading from a file, do nothing, ensuring that files with
    invalid data or deprecated neurodata types can be read.
    """
    if not self._in_construct_mode:


      raise ValueError(error_msg)


E           ValueError: TwoPhotonSeries 'TwoPhotonSeries': num_samples should be set when format='external' and rate is used for timing, because data is empty and its length cannot be used to determine the number of frames.
/home/ethan/mambaforge/envs/suite2p-test/lib/python3.11/site-packages/pynwb/core.py:59: ValueError

_________________________________________________________________ test_nwb_round_trip[2plane2chan1500] _________________________________________________________________
data_folder = '2plane2chan1500'
@pytest.mark.parametrize(
    "data_folder",
    [
        ("1plane1chan1500"),
        ("2plane2chan1500"),
        #("bruker"),
    ],
)
def test_nwb_round_trip(data_folder):
    """Test saving Suite2p outputs to NWB and loading them back."""

    # Define the path to the suite2p folder
    save_folder = Path("data").joinpath("test_outputs", data_folder, "suite2p")
    print(save_folder)

    # Load expected data from plane folders
    plane_folders = natsorted([
        f for f in save_folder.iterdir()
        if f.is_dir() and f.name.startswith("plane")
    ])

    # Concatenate data across planes
    expected_F = np.concatenate([
        np.load(plane_dir.joinpath("F.npy"), allow_pickle=True)
        for plane_dir in plane_folders
    ], axis=0)

    expected_Fneu = np.concatenate([
        np.load(plane_dir.joinpath("Fneu.npy"), allow_pickle=True)
        for plane_dir in plane_folders
    ], axis=0)

    expected_spks = np.concatenate([
        np.load(plane_dir.joinpath("spks.npy"), allow_pickle=True)
        for plane_dir in plane_folders
    ], axis=0)

    expected_iscell = np.concatenate([
        np.load(plane_dir.joinpath("iscell.npy"), allow_pickle=True)
        for plane_dir in plane_folders
    ], axis=0)

    expected_stat = np.concatenate([
        np.load(plane_dir.joinpath("stat.npy"), allow_pickle=True)
        for plane_dir in plane_folders
    ], axis=0)


    # Save as NWB file


  save_nwb(save_folder)


tests/test_io.py:170:

suite2p/io/nwb.py:404: in save_nwb

image_series = TwoPhotonSeries(

/home/ethan/mambaforge/envs/suite2p-test/lib/python3.11/site-packages/hdmf/utils.py:641: in func_call

return func(args[0], **pargs)

^^^^^^^^^^^^^^^^^^^^^^

/home/ethan/mambaforge/envs/suite2p-test/lib/python3.11/site-packages/pynwb/ophys.py:272: in init

super().init(**kwargs)

/home/ethan/mambaforge/envs/suite2p-test/lib/python3.11/site-packages/hdmf/utils.py:641: in func_call

return func(args[0], **pargs)

^^^^^^^^^^^^^^^^^^^^^^

/home/ethan/mambaforge/envs/suite2p-test/lib/python3.11/site-packages/pynwb/image.py:160: in init

self._error_on_new_pass_on_construct(error_msg=error_msg)

self = TwoPhotonSeries pynwb.ophys.TwoPhotonSeries at 0x134829293502352

Fields:

comments: no comments

conversion: 1.0

d...rate: 20.0

resolution: -1.0

starting_frame: [0]

starting_time: 0.0

starting_time_unit: seconds

unit: unknown
error_msg = "TwoPhotonSeries 'TwoPhotonSeries': num_samples should be set when format='external' and rate is used for timing, because data is empty and its length cannot be used to determine the number of frames."
def _error_on_new_pass_on_construct(self, error_msg: str):
    """
    Raise an error when a check is violated on instance creation.
    When reading from a file, do nothing, ensuring that files with
    invalid data or deprecated neurodata types can be read.
    """
    if not self._in_construct_mode:


      raise ValueError(error_msg)


E           ValueError: TwoPhotonSeries 'TwoPhotonSeries': num_samples should be set when format='external' and rate is used for timing, because data is empty and its length cannot be used to determine the number of frames.
/home/ethan/mambaforge/envs/suite2p-test/lib/python3.11/site-packages/pynwb/core.py:59: ValueError

=========================================================================== warnings summary ===========================================================================

tests/regression/test_extraction_pipeline.py::test_extraction_output_1plane1chan

/synology/code/CalciumImaging/suite2p/suite2p/extraction/extract.py:67: UserWarning: Sparse invariant checks are implicitly disabled. Memory errors (e.g. SEGFAULT) will occur when operating on a sparse tensor which violates the invariants, but checks incur performance overhead. To silence this warning, explicitly opt in or out. See torch.sparse.check_sparse_tensor_invariants.__doc__ for guidance.  (Triggered internally at /__w/pytorch/pytorch/aten/src/ATen/Context.cpp:916.)

nmasks = torch.sparse_coo_tensor(inds, torch.ones(len(row_indices), device=device),
tests/regression/test_extraction_pipeline.py::test_extraction_output_1plane1chan

/synology/code/CalciumImaging/suite2p/suite2p/extraction/extract.py:69: UserWarning: Sparse CSR tensor support is in beta state. If you miss a functionality in the sparse tensor support, please submit a feature request to https://github.com/pytorch/pytorch/issues. (Triggered internally at /__w/pytorch/pytorch/aten/src/ATen/SparseCsrTensorImpl.cpp:49.)

nmasks = nmasks.to_sparse_csc()
tests/regression/test_extraction_pipeline.py::test_extraction_output_1plane1chan

/synology/code/CalciumImaging/suite2p/suite2p/extraction/extract.py:89: UserWarning: The given NumPy array is not writable, and PyTorch does not support non-writable tensors. This means writing to this tensor will result in undefined behavior. You may want to copy the array to protect its data or make it writable before converting it to a tensor. This type of warning will be suppressed for the rest of this program. (Triggered internally at /__w/pytorch/pytorch/torch/csrc/utils/tensor_numpy.cpp:208.)

data = torch.from_numpy(f_in[tstart : tend]).to(device)
tests/regression/test_full_pipeline.py::test_1plane_1chan_with_batches_metrics_and_exported_to_nwb_format

tests/test_io.py::test_nwb_round_trip[1plane1chan1500]

tests/test_io.py::test_nwb_round_trip[2plane2chan1500]

/home/ethan/mambaforge/envs/suite2p-test/lib/python3.11/site-packages/hdmf/utils.py:641: DeprecationWarning: The 'manufacturer' field is deprecated. Instead, use DeviceModel.manufacturer and link to that DeviceModel from this Device.

return func(args[0], **pargs)
tests/smoke/test_file_loading.py::test_bruker

/synology/code/CalciumImaging/suite2p/suite2p/run_s2p.py:230: DeprecationWarning: The 'warn' method is deprecated, use 'warning' instead

logger.warn("WARNING: number of frames < 200, unpredictable behaviors may occur")
tests/smoke/test_file_loading.py::test_h5_file_is_processed_end_to_end

tests/smoke/test_run_s2p.py::test_do_registration_do_roi_detect_settings_check_timing

/synology/code/CalciumImaging/suite2p/suite2p/pipeline_s2p.py:223: DeprecationWarning: The 'warn' method is deprecated, use 'warning' instead

logger.warn("WARNING: skipping spike detection (settings['do_deconvolution']=False)")
tests/smoke/test_run_s2p.py::test_do_registration_do_roi_detect_settings_check_timing

tests/smoke/test_run_s2p.py::test_do_registration_do_roi_detect_settings_check_timing

/synology/code/CalciumImaging/suite2p/suite2p/pipeline_s2p.py:136: DeprecationWarning: The 'warn' method is deprecated, use 'warning' instead

logger.warn("WARNING: skipping cell detection (settings['run']['do_detection']=False)")
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html

======================================================================= short test summary info ========================================================================

FAILED tests/regression/test_full_pipeline.py::test_1plane_1chan_with_batches_metrics_and_exported_to_nwb_format - ValueError: TwoPhotonSeries 'TwoPhotonSeries': num_samples should be set when format='external' and rate is used for timing, because data is empty and its length c...

FAILED tests/test_io.py::test_nwb_round_trip[1plane1chan1500] - ValueError: TwoPhotonSeries 'TwoPhotonSeries': num_samples should be set when format='external' and rate is used for timing, because data is empty and its length c...

FAILED tests/test_io.py::test_nwb_round_trip[2plane2chan1500] - ValueError: TwoPhotonSeries 'TwoPhotonSeries': num_samples should be set when format='external' and rate is used for timing, because data is empty and its length c...

=================================================== 3 failed, 22 passed, 1 skipped, 11 warnings in 95.11s (0:01:35) ====================================================

[joannercsheppard]

I don't think any of the tests are failing due to your code in particular, as they also fail on main. I think Carsen has a PR to fix the nwb failures on the Linux side (#1274) and I've just pushed PR #1278 to deal with the ROI-differences failures on the Windows side.

Thanks for running the tests!

[ethanbb]

No problem! I'm glad it helped make the tests more robust as well.