MNE version: 0.24.1 operating system: Ubuntu 18.04 Dear all, I have two raw data objects that I concatenate and then save. Later on, I load this data again, filter it, set a new reference, apply an ICA solution, interpolate bad channels, and save it once more. After the second save, the file is …

The reason could be that if you store single-precision data as double-precision data, you will not add any additional precision. Half of the bits will be unused, so that’s why such data can be compressed by a lot. If you apply a filter, the full amount of bits (double-precision) will now be used, be…

Maybe related? [image] Data size not consistent with its' true size 💬 Support & Discussions Hi, I have 2 GB of EEG data, after loading it using MNE, MNE shows that the size of the data is about 8GB. How so?

[image] richard: Maybe related? Thanks! But I don’t think so: in my own pipeline I read short data, then save it → becomes single … then read it (double while in memory) and filter/etc it … then save it again as single. Yet: the sizes of the initially loaded single data and the then newly s…

Can you try to manually zip (and/or gz) both concat_raw.fif and concat_filt_raw.fif (which are about the same size)? When compressed does the second file get much bigger? If not, maybe it’s something we do when creating/compressing the .fif.gz file?

good idea, I just tired gzip <fname.fif> - it results in: concat_raw.fif → 187MB when gzipped (425MB when done in MNE-Python) concat_filt_raw.fif → 464MB when gzipped (930MB when done in MNE-Python)

This is interesting for two reasons: Our compression is extremely inefficient, this could be a quick fix if the library we’re using support other algorithms. The unfiltered file is still less than half the size of the filtered data. I wonder if it is because the original file is much less accurate…

but concat_filt_raw.fif.gz is around 930MB. I don’t know how I arrived at that file size … I cannot reproduce it anymore. I have improved the MWE, and I’d be grateful if someone else could run it to verify my results below. You only need a recent MNE installation, and gzip available as a command…

Yes, I can reproduce it: 482M Dec 14 09:48 concat_filt_raw.fif 444M Dec 14 09:48 concat_filt_raw.fif.gz 443M Dec 14 09:48 concat_filt_raw_os-gz.fif.gz 482M Dec 14 09:47 concat_raw.fif 204M Dec 14 09:47 concat_raw.fif.gz 181M Dec 14 09:47 concat_raw_os-gz.fif.gz I think my explanation in my previou…

[image] cbrnr: I think my explanation in my previous comment still applies – did you miss it? oups, yes! [image] cbrnr: The unfiltered file is still less than half the size of the filtered data. I wonder if it is because the original file is much less accurate (half or single), so when s…

Ah now I see what you mean, but I don’t think it applies here, because we are using raw.save with the default parameters in our example → and that means fmt='single', so even if after filtering the “information” in the data is now of type double, it will still be saved as single. In my initial (no…

Not sure if this really works out if all saved files contain single data though…

fif.gz compression less effective after filtering?

Support & Discussions

richard (Richard Höchenberger) December 13, 2021, 11:55am 2

Maybe related?

Topic		Replies	Views
FIF-File 2x disk-size after first loading Support & Discussions	5	264	March 21, 2021
Data size not consistent with its' true size Support & Discussions	12	632	August 14, 2021
combining/splitting fif files Mailing List Archive (read-only) list-archive	7	401	February 18, 2009
Saving compressed data Support & Discussions mne-bids	5	41	May 22, 2026
Changed file size after raw.save and write_to_bids Support & Discussions	1	66	February 1, 2024

fif.gz compression less effective after filtering?

Related topics