Dear MNE developers
Just out of curiosity, is this going to happen in the near future, or is
it a greater challenge than I imagine?
Best
Lau
It is challenging because the FIFF file format is not designed to handle
files larger than that. Adding support for it would require redesigning and
rewriting the I/O code fundamental to data analysis, and would likely break
backward compatibility with many tools. AFAIK nobody is actively working on
the issue.
There are different ways of working around the issue. If you have a
specific use case, you could open an issue on the `mne-python` Github site,
and perhaps we can provide some ideas. Ideally we could document the
workarounds somewhere in case people hit this issue in the future, too.
Eric
hi Lau,
Just out of curiosity, is this going to happen in the near future, or is
it a greater challenge than I imagine?
I am not sure we have the man power to tackle this now.
It would also require to change the C FIF library which is not trivial.
In the mean time you need to split the files.
Best,
Alex
Dear Eric and Alex
Thanks for your answers. I now understand that it is not a simple issue.
The files themselves are not greater than 2.0 GB, but because I have multiple recordings per subject, I read them in as a list. The only reason that I ask is that I would like to save the filtered file (i.e. the one that is greater than 2.0 GB, because it is a list of raw files).
The way I work around it now, is simply to save the epoched files and working from there. The only annoying thing is, if/when I realize that I need to do something different for creating the epochs, then I need to do the time-consuming process of filtering again. But I can live with that.
Thanks again for a wonderful python toolbox.
Best
Lau
Hi folks,
it just occurs to me that we might want to think about adding a Raw.split
method.
As a first quick and dirty solution, Lau, you might want think about a
function that splits the raw object and saves each split to fif files.
Something like this:
def split_save_raw(raw, splits , fname_pattern):
for i_split, (tmin, tmax) in enumerate(splits):
raw.copy().crop(tmin, tmax).save(fname_pattern % i_split)
split_save_raw(raw, [(0, 1), (1, 3), (3, 4)], 'my-split-%i-raw.fif')
Re-construction should then be trivial.
Cheers,
Denis
Thanks, Denis
That definitely looks useful.
I will try to implement that into the code tomorrow.
Thanks again
I suggest the raw.save splits files automatically like the C code.
Alex
Hello Denis et al.,
I just confirm what Eric already said: it's not easy to get rid of the
2-GB file size limit as it is due to the fact the FIFF format internally
uses 32-bit file pointers. Switching to 64-bit pointers would not be
backwards compatible. The limit is an issue for Elekta, too, and they
are looking for a good solution but I don't know about the time frame.
In the meantime we could perhaps make use of the "file chaining" FIFF
tags that allow logical linking of consecutive files. If the MNE FIFF
i/o functions could use these tags and split/assemble files accordingly,
reading and writing data larger than 2 GB could be made transparent to
the user. Comments?
Cheers,
Lauri
you can also set the dtype when saving back the raw filtered data. You
can use float32 to have half size files or even float16
HTH
A
+1e15 I had the same thought.
16-bit floats? I do not know about such a thing. at any rate, do not use 16 bit integers, precision is most likely lost. Also, add the link tags to the files like the C code does. - Matti
Thanks,
Lauri --- that would mean, using such link tags, we could load the entire chain using a single filename? That sound like the best we can do given difficulties mentioned.
Sounds clean to me.
Denis
I created the issue so we don't forget
https://github.com/mne-tools/mne-python/issues/1194
best,
Alex