MNE-BIDS-Pipeline Maxwell filter reference run rank mismatch

Support & Discussions
1

Hi,

I received an error when running the Maxwell filter step in the MNE-BIDS-Pipeline, saying that the rank of the data in the reference rune does not match the empty room data rank, see below:

[10:55:33] │ ❌ preprocessing/_02_maxfilter sub-09 run-01 A critical error occurred. The error message was: Reference run data rank 73.0 does not match empty-room data rank 71.0 after Maxwell filtering. This indicates that the data were processed  differently.

Aborting pipeline run. The full traceback is:

Traceback (most recent call last):

  File "/home/ma09/.conda/envs/my_mne1.4_fix/lib/python3.11/site-packages/mne_bids_pipeline/_run.py", line 54, in wrapper
    out = memory.cache(func)(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/home/ma09/.conda/envs/my_mne1.4_fix/lib/python3.11/site-packages/mne_bids_pipeline/_run.py", line 263, in wrapper
    out_files = memorized_func(*args, **kwargs)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/home/ma09/.conda/envs/my_mne1.4_fix/lib/python3.11/site-packages/joblib/memory.py", line 594, in __call__
    return self._cached_call(args, kwargs)[0]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/home/ma09/.conda/envs/my_mne1.4_fix/lib/python3.11/site-packages/joblib/memory.py", line 537, in _cached_call
    out, metadata = self.call(*args, **kwargs)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/home/ma09/.conda/envs/my_mne1.4_fix/lib/python3.11/site-packages/joblib/memory.py", line 779, in call
    output = self.func(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/home/ma09/.conda/envs/my_mne1.4_fix/lib/python3.11/site-packages/mne_bids_pipeline/steps/preprocessing/_02_maxfilter.py", line 264, in run_maxwell_filter
    raise RuntimeError(msg)

RuntimeError: Reference run data rank 73.0 does not match empty-room data rank 71.0 after Maxwell filtering. This indicates that the data were processed  differently.

This occurred only with a specific subject in the dataset and it has to do with which run I set as reference. If I set mf_reference_run = '03', I get the error, however, if I set mf_reference_run = '01', then the Maxwell filter step runs fine without an error.

I’m not sure why the data rank in run 3 would be different than in run 1? I’m wondering if this has to do with the recently fixed bug about setting the mf_reference_run parameter to anything else than '01'?

Could anyone please help?

Many thanks,
Máté

System info:

Platform             Linux-3.10.0-1160.el7.x86_64-x86_64-with-glibc2.17
Python               3.11.0 | packaged by conda-forge | (main, Jan 14 2023, 12:28:47) [GCC 11.3.0]
Executable           /home/ma09/.conda/envs/my_mne1.4_fix/bin/python
CPU                  x86_64 (32 cores)
Memory               251.4 GB

Core
├☑ mne               1.4.2
├☑ numpy             1.24.3 (OpenBLAS 0.3.23 with 32 threads)
├☑ scipy             1.10.1
├☑ matplotlib        3.7.1 (backend=agg)
├☑ pooch             1.7.0
└☑ jinja2            3.1.2

Numerical (optional)
├☑ sklearn           1.2.2
├☑ numba             0.57.0
├☑ nibabel           5.1.0
├☑ nilearn           0.10.1
├☑ dipy              1.7.0
├☑ openmeeg          2.5.6
├☑ pandas            2.0.2
└☐ unavailable       cupy

Visualization (optional)
├☑ pyvista           0.39.1 (OpenGL unavailable)
├☑ pyvistaqt         0.0.0
├☑ ipyvtklink        0.2.2
├☑ vtk               9.2.5
├☑ qtpy              2.3.1 (None=None)
├☑ pyqtgraph         0.13.3
├☑ mne-qt-browser    0.5.1
└☐ unavailable       ipympl

Ecosystem (optional)
├☑ mne-bids          0.13.dev0
├☑ mne-bids-pipeline 1.4.0.dev3+gb0cf4a0
└☐ unavailable       mne-nirs, mne-features, mne-connectivity, mne-icalabel
2

Just following up on this, can anyone please help? Pinging @larsoner @richard @agramfort

Máté

3

This can happen if the set of bad channels differs between the runs. Can you run with --debug then p raw_noise.info['bads'] and then also for the other instance/info and see if this is the case?

1 Like
4

… we should probably add an option to allow you to ignore this. It’s a kind of expert seeing but this rank issue is only a probably sometimes. It is something that we should fix and I think there might even be a note in the code about it

1 Like
5

Thanks @larsoner, I’ll look into this very soon and report back.

6

This is indeed the case:

(Pdb) raw_noise.info['bads']
['MEG0743', 'MEG1613', 'MEG1831', 'MEG1832', 'MEG1833', 'MEG2011', 'MEG2012', 'MEG2013', 'MEG2141']
(Pdb) raw_sss.info['bads']
['EEG002', 'EEG040']
(Pdb)

Best,
Máté

7

:frowning:

Can you open a MNE-BIDS-Pipeline issue on GitHub about this? We’ll have to think about how to deal with this use case properly

8

Sure, I opened a new issue here: Error due to mismatch between Maxwell filter reference run and empty room data rank · Issue #759 · mne-tools/mne-bids-pipeline · GitHub

I’m just wondering why this issue doesn’t occur if the reference run is set to ‘01’? Is it just sheer luck? I mean it is perfectly possible for run-01 to have different channels rejected than the emptyroom recording. Although the empty room recording has been taken just before run-01, so channel noise might be more similar to the empty room recodign in run-01 than in run-03?

9

fixed by PR #761: BUG: Fix bug with empty room processing by larsoner · Pull Request #761 · mne-tools/mne-bids-pipeline · GitHub