# MNE-BIDS-Pipeline Maxwell filter reference run rank mismatch

**URL:** <https://mne.discourse.group/t/mne-bids-pipeline-maxwell-filter-reference-run-rank-mismatch/7144>\
**Category:** Support & Discussions\
**Tags:** mne-bids-pipeline\
**Created:** [June 26, 2023, 11:27am UTC](https://mne.discourse.group/t/mne-bids-pipeline-maxwell-filter-reference-run-rank-mismatch/7144 "2023-06-26T11:27:09Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![allermat](https://yyz2.discourse-cdn.com/free1/user_avatar/mne.discourse.group/allermat/32/43_2.png) [@allermat](https://mne.discourse.group/u/allermat)\
**Post date:** [June 26, 2023, 11:27am UTC](https://mne.discourse.group/t/mne-bids-pipeline-maxwell-filter-reference-run-rank-mismatch/7144/1 "2023-06-26T11:27:09Z")

</div>

Hi,

I received an error when running the Maxwell filter step in the MNE-BIDS-Pipeline, saying that the rank of the data in the reference rune does not match the empty room data rank, see below:

```python
[10:55:33] │ ❌ preprocessing/_02_maxfilter sub-09 run-01 A critical error occurred. The error message was: Reference run data rank 73.0 does not match empty-room data rank 71.0 after Maxwell filtering. This indicates that the data were processed differently.

Aborting pipeline run. The full traceback is:

Traceback (most recent call last):

  File "/home/ma09/.conda/envs/my_mne1.4_fix/lib/python3.11/site-packages/mne_bids_pipeline/_run.py", line 54, in wrapper
    out = memory.cache(func)(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/home/ma09/.conda/envs/my_mne1.4_fix/lib/python3.11/site-packages/mne_bids_pipeline/_run.py", line 263, in wrapper
    out_files = memorized_func(*args, **kwargs)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/home/ma09/.conda/envs/my_mne1.4_fix/lib/python3.11/site-packages/joblib/memory.py", line 594, in __call__
    return self._cached_call(args, kwargs)[0]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/home/ma09/.conda/envs/my_mne1.4_fix/lib/python3.11/site-packages/joblib/memory.py", line 537, in _cached_call
    out, metadata = self.call(*args, **kwargs)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/home/ma09/.conda/envs/my_mne1.4_fix/lib/python3.11/site-packages/joblib/memory.py", line 779, in call
    output = self.func(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/home/ma09/.conda/envs/my_mne1.4_fix/lib/python3.11/site-packages/mne_bids_pipeline/steps/preprocessing/_02_maxfilter.py", line 264, in run_maxwell_filter
    raise RuntimeError(msg)

RuntimeError: Reference run data rank 73.0 does not match empty-room data rank 71.0 after Maxwell filtering. This indicates that the data were processed differently.

```

This occurred only with a specific subject in the dataset and it has to do with which run I set as reference. If I set `mf_reference_run = '03'`, I get the error, however, if I set `mf_reference_run = '01'`, then the Maxwell filter step runs fine without an error.

I’m not sure why the data rank in run 3 would be different than in run 1? I’m wondering if this has to do with the [recently fixed bug](https://mne.discourse.group/t/mne-bids-pipeline-maxwell-filter-reference-run-error/7030) about setting the `mf_reference_run` parameter to anything else than `'01'`?

Could anyone please help?

Many thanks,  
Máté

System info:

```python
Platform Linux-3.10.0-1160.el7.x86_64-x86_64-with-glibc2.17
Python 3.11.0 | packaged by conda-forge | (main, Jan 14 2023, 12:28:47) [GCC 11.3.0]
Executable /home/ma09/.conda/envs/my_mne1.4_fix/bin/python
CPU x86_64 (32 cores)
Memory 251.4 GB

Core
├☑ mne 1.4.2
├☑ numpy 1.24.3 (OpenBLAS 0.3.23 with 32 threads)
├☑ scipy 1.10.1
├☑ matplotlib 3.7.1 (backend=agg)
├☑ pooch 1.7.0
└☑ jinja2 3.1.2

Numerical (optional)
├☑ sklearn 1.2.2
├☑ numba 0.57.0
├☑ nibabel 5.1.0
├☑ nilearn 0.10.1
├☑ dipy 1.7.0
├☑ openmeeg 2.5.6
├☑ pandas 2.0.2
└☐ unavailable cupy

Visualization (optional)
├☑ pyvista 0.39.1 (OpenGL unavailable)
├☑ pyvistaqt 0.0.0
├☑ ipyvtklink 0.2.2
├☑ vtk 9.2.5
├☑ qtpy 2.3.1 (None=None)
├☑ pyqtgraph 0.13.3
├☑ mne-qt-browser 0.5.1
└☐ unavailable ipympl

Ecosystem (optional)
├☑ mne-bids 0.13.dev0
├☑ mne-bids-pipeline 1.4.0.dev3+gb0cf4a0
└☐ unavailable mne-nirs, mne-features, mne-connectivity, mne-icalabel

```

---

<div class="post-metadata">

**Author:** ![allermat](https://yyz2.discourse-cdn.com/free1/user_avatar/mne.discourse.group/allermat/32/43_2.png) [@allermat](https://mne.discourse.group/u/allermat)\
**Post date:** [July 3, 2023, 8:59am UTC](https://mne.discourse.group/t/mne-bids-pipeline-maxwell-filter-reference-run-rank-mismatch/7144/2 "2023-07-03T08:59:59Z")

</div>

Just following up on this, can anyone please help? Pinging @larsoner @richard @agramfort

Máté

---

<div class="post-metadata">

**Author:** ![larsoner](https://yyz2.discourse-cdn.com/free1/user_avatar/mne.discourse.group/larsoner/32/3_2.png) [@larsoner](https://mne.discourse.group/u/larsoner)\
**Post date:** [July 3, 2023, 2:43pm UTC](https://mne.discourse.group/t/mne-bids-pipeline-maxwell-filter-reference-run-rank-mismatch/7144/3 "2023-07-03T14:43:22Z")

</div>

> [@allermat](#):
>
> I’m not sure why the data rank in run 3 would be different than in run 1?

This can happen if the set of bad channels differs between the runs. Can you run with --debug then `p raw_noise.info['bads']` and then also for the other instance/info and see if this is the case?

---

<div class="post-metadata">

**Author:** ![larsoner](https://yyz2.discourse-cdn.com/free1/user_avatar/mne.discourse.group/larsoner/32/3_2.png) [@larsoner](https://mne.discourse.group/u/larsoner)\
**Post date:** [July 3, 2023, 2:44pm UTC](https://mne.discourse.group/t/mne-bids-pipeline-maxwell-filter-reference-run-rank-mismatch/7144/4 "2023-07-03T14:44:26Z")

</div>

… we should probably add an option to allow you to ignore this. It’s a kind of expert seeing but this rank issue is only a probably sometimes. It is something that we should fix and I think there might even be a note in the code about it

---

<div class="post-metadata">

**Author:** ![allermat](https://yyz2.discourse-cdn.com/free1/user_avatar/mne.discourse.group/allermat/32/43_2.png) [@allermat](https://mne.discourse.group/u/allermat)\
**Post date:** [July 5, 2023, 5:01pm UTC](https://mne.discourse.group/t/mne-bids-pipeline-maxwell-filter-reference-run-rank-mismatch/7144/5 "2023-07-05T17:01:15Z")

</div>

Thanks @larsoner, I’ll look into this very soon and report back.

---

<div class="post-metadata">

**Author:** ![allermat](https://yyz2.discourse-cdn.com/free1/user_avatar/mne.discourse.group/allermat/32/43_2.png) [@allermat](https://mne.discourse.group/u/allermat)\
**Post date:** [July 6, 2023, 9:30am UTC](https://mne.discourse.group/t/mne-bids-pipeline-maxwell-filter-reference-run-rank-mismatch/7144/6 "2023-07-06T09:30:45Z")

</div>

> [@larsoner](#):
>
> This can happen if the set of bad channels differs between the runs. Can you run with --debug then `p raw_noise.info['bads']` and then also for the other instance/info and see if this is the case?

This is indeed the case:

```python
(Pdb) raw_noise.info['bads']
['MEG0743', 'MEG1613', 'MEG1831', 'MEG1832', 'MEG1833', 'MEG2011', 'MEG2012', 'MEG2013', 'MEG2141']
(Pdb) raw_sss.info['bads']
['EEG002', 'EEG040']
(Pdb) 

```

Best,  
Máté

---

<div class="post-metadata">

**Author:** ![larsoner](https://yyz2.discourse-cdn.com/free1/user_avatar/mne.discourse.group/larsoner/32/3_2.png) [@larsoner](https://mne.discourse.group/u/larsoner)\
**Post date:** [July 6, 2023, 2:06pm UTC](https://mne.discourse.group/t/mne-bids-pipeline-maxwell-filter-reference-run-rank-mismatch/7144/7 "2023-07-06T14:06:38Z")

</div>

☹

Can you open a MNE-BIDS-Pipeline issue on GitHub about this? We’ll have to think about how to deal with this use case properly

---

<div class="post-metadata">

**Author:** ![allermat](https://yyz2.discourse-cdn.com/free1/user_avatar/mne.discourse.group/allermat/32/43_2.png) [@allermat](https://mne.discourse.group/u/allermat)\
**Post date:** [July 6, 2023, 2:36pm UTC](https://mne.discourse.group/t/mne-bids-pipeline-maxwell-filter-reference-run-rank-mismatch/7144/8 "2023-07-06T14:36:28Z")

</div>

Sure, I opened a new issue here: [Error due to mismatch between Maxwell filter reference run and empty room data rank · Issue #759 · mne-tools/mne-bids-pipeline · GitHub](https://github.com/mne-tools/mne-bids-pipeline/issues/759)

I’m just wondering why this issue doesn’t occur if the reference run is set to ‘01’? Is it just sheer luck? I mean it is perfectly possible for run-01 to have different channels rejected than the emptyroom recording. Although the empty room recording has been taken just before run-01, so channel noise might be more similar to the empty room recodign in run-01 than in run-03?

---

<div class="post-metadata">

**Author:** ![drammock](https://yyz2.discourse-cdn.com/free1/user_avatar/mne.discourse.group/drammock/32/4_2.png) [@drammock](https://mne.discourse.group/u/drammock)\
**Post date:** [July 11, 2023, 2:55pm UTC](https://mne.discourse.group/t/mne-bids-pipeline-maxwell-filter-reference-run-rank-mismatch/7144/9 "2023-07-11T14:55:02Z")

</div>

fixed by PR #761: [BUG: Fix bug with empty room processing by larsoner · Pull Request #761 · mne-tools/mne-bids-pipeline · GitHub](https://github.com/mne-tools/mne-bids-pipeline/pull/761)