Load MaxFiltered split fif files

💬 Support & Discussions
1

We have a long recording that is split into several fif files (rec.fif, rec-1.fif). We MaxFilter both files using the proprietary tool by MEGIN, which creates the files rec_tsss_mc.fif and rec-1_tsss_mc.fif.

However, I’m unable to load the files using MNE and get the error

raw = mne.io.read_raw_fif(file, preload=True)

  File ~/anaconda3/lib/python3.10/site-packages/mne/io/fiff/raw.py:540 in read_raw_fif
    return Raw(

  File <decorator-gen-271>:12 in __init__

  File ~/anaconda3/lib/python3.10/site-packages/mne/io/fiff/raw.py:93 in __init__
    raw, next_fname, buffer_size_sec = self._read_raw_file(

  File <decorator-gen-272>:12 in _read_raw_file

  File ~/anaconda3/lib/python3.10/site-packages/mne/io/fiff/raw.py:319 in _read_raw_file
    next_fname = _get_next_fname(fid, fname_rep, tree)

  File ~/anaconda3/lib/python3.10/site-packages/mne/io/open.py:62 in _get_next_fname
    for ent in nodes["directory"]:

TypeError: 'NoneType' object is not iterable

It seems like either the link is not preserved by MaxFilter or something else went wrong.

Any advise on what is happening here and how we can load MaxFiltered files?

(mne v1.4.2)

2

Hello,

Did you by any chance rename those files at any point?
There are tags inside the file which contains the file name of the next file to load. Thus, if you rename the files, you are not able to reload it as the tags are still pointing to the old file name.

Mathieu

3

Nope, I used them as they come from MaxFilter.

Is there a way to see the filenames that it points to somehow?

4

Hello,
I’m running into this issue as well.

I have files originally named like:

-rw-r–r-- 1 archive archiveg 216M Mar 15 11:29 M87177739+Study20240314+avmmn_visit2_run1_raw-1.fif
-rw-r–r-- 1 archive archiveg 2.0G Mar 15 11:30 M87177739+Study20240314+avmmn_visit2_run1_raw.fif
-rw-r–r-- 1 archive archiveg 942M Mar 15 11:30 M87177739+Study20240314+avmmn_visit2_run2_raw.fif
-rw-r–r-- 1 archive archiveg 479M Mar 15 11:30 M87177739+Study20240314+avmmn_visit2_run3_raw.fif

We used proprietary MEGIN maxfilter command to process each file. When I load the first processed file (the one named run1_raw.fif), I get the following error:

In [16]: raw = io.Raw(fnames[1])
Opening raw data file /export/research/analysis/human/jstephen/synchrony_20105/MEG_AUTOANALYSIS/subjects/M87177739/M87177739_avmmn_visit2_run1_tsss_mc.fif...
<ipython-input-16-8c6434757102>:1: RuntimeWarning: This filename (/export/research/analysis/human/jstephen/synchrony_20105/MEG_AUTOANALYSIS/subjects/M87177739/M87177739_avmmn_visit2_run1_tsss_mc.fif) does not conform to MNE naming conventions. All raw files should end with raw.fif, raw_sss.fif, raw_tsss.fif, _meg.fif, _eeg.fif, _ieeg.fif, raw.fif.gz, raw_sss.fif.gz, raw_tsss.fif.gz, _meg.fif.gz, _eeg.fif.gz or _ieeg.fif.gz
  raw = io.Raw(fnames[1])
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[16], line 1
----> 1 raw = io.Raw(fnames[1])

File <decorator-gen-390>:12, in __init__(self, fname, allow_maxshield, preload, on_split_missing, verbose)

File /export/research/analysis/human/jstephen/shared/programs/python/anaconda_072020/new/envs/mne1.7/lib/python3.12/site-packages/mne/io/fiff/raw.py:105, in Raw.__init__(self, fname, allow_maxshield, preload, on_split_missing, verbose)
    103 next_fname = fname
    104 while next_fname is not None:
--> 105     raw, next_fname, buffer_size_sec = self._read_raw_file(
    106         next_fname, allow_maxshield, preload, do_check_ext
    107     )
    108     do_check_ext = False
    109     raws.append(raw)

File <decorator-gen-391>:12, in _read_raw_file(self, fname, allow_maxshield, preload, do_check_ext, verbose)

File /export/research/analysis/human/jstephen/shared/programs/python/anaconda_072020/new/envs/mne1.7/lib/python3.12/site-packages/mne/io/fiff/raw.py:326, in Raw._read_raw_file(self, fname, allow_maxshield, preload, do_check_ext, verbose)
    323             tag = read_tag(fid, ent.pos)
    324             nskip = int(tag.data.item())
--> 326     next_fname = _get_next_fname(fid, fname_rep, tree)
    328 # reformat raw_extras to be a dict of list/ndarray rather than
    329 # list of dict (faster access)
    330 raw_extras = {key: [r[key] for r in raw_extras] for key in raw_extras[0]}

File /export/research/analysis/human/jstephen/shared/programs/python/anaconda_072020/new/envs/mne1.7/lib/python3.12/site-packages/mne/_fiff/open.py:70, in _get_next_fname(fid, fname, tree)
     68 for nodes in nodes_list:
     69     next_fname = None
---> 70     for ent in nodes["directory"]:
     71         if ent.kind == FIFF.FIFF_REF_ROLE:
     72             tag = read_tag(fid, ent.pos)

TypeError: 'NoneType' object is not iterable

What is the correct way to handle this situation?

Thank you,
Megan

5

Hi there,
Since we rename our processed files and handle counting them outside loading raw data, I guess the
on_split_missing = ‘ignore’
tag is probably what I need.

I will close this thread if it works.

Thanks,
Megan

6

You should not be renaming your FIF files for a correct loading.
If you load the file without digit, you can check the filenames listed in the attributes raw.filenames, and then rename all files accordingly before loading.

from pathlib import Path

from mne.datasets import sample
from mne.io import read_raw_fif

fname = sample.data_path() / "MEG" / "sample" / "sample_audvis_raw.fif"
raw = read_raw_fif(fname, preload=True)
fname = Path.home() / "Downloads" / "sample_audvis_raw.fif"
raw.save(fname, split_size="100MB")

raw = read_raw_fif(fname, preload=True)
raw.filenames

From the tags themselves (that you can look out through the CLI with mne show_fiff) it’s a bit difficult since different files acquired on different systems can have differences. For reference, this is where the next file to load is determined: mne-python/mne/_fiff/open.py at 20871ff2286935a564a9db5320afa7437df18458 · mne-tools/mne-python · GitHub

Mathieu

1 Like