Saving compressed data

Support & Discussions
1

MNE also allows reading and writing of `fif.gz` files. However, that option does not seem to be available for BIDS:

ValueError: Extension .fif.gz is not allowed. Use one of these extensions [‘.CNT’, ‘.EEG’, ‘.bdf’, ‘.bin’, ‘.cdt’, ‘.cnt’, ‘.con’, ‘.ds’, ‘.edf’, ‘.fif’, ‘.lay’, ‘.mef’, ‘.nwb’, ‘.pdf’, ‘.set’, ‘.snirf’, ‘.sqd’, ‘.vhdr’, ‘.json’, ‘.tsv’, ‘.tsv.gz’, ‘.nii’, ‘.nii.gz’, ‘.pos’, ‘.eeg’, ‘.vmrk’, ‘.dat’, ‘.EEG’, ‘.mrk’].

While .gz is not the native file format of the MEG machine, it’s not really a new file format. Is there any valid way to compress my files before saving them in BIDS?

I want to share my data as part of open science, but currently the dataset is too large for almost all repositories, compression seems the sustainable choice.

related: Remove .gz extension for fif: It's no longer part of the validator · Issue #60 · mne-tools/mne-bids · GitHub

2

It would be interesting to find out why they removed it (the links to the issues/PRs in the issue you linked are all 404 and I cannot find them in the current GitHub repo).

3

@cbrnr Here is the link you are looking for: fif.gz is allowed even though not mentioned in the spec · Issue #491 · bids-standard/legacy-validator · GitHub

Looks like both the organization and repo have been transferred since 2018!

4

Thanks @scott-huberty! Kind of sad that they dropped it even after noticing that they actually compress quite a lot. I know that results vary depending on the contents, but I just tested our sample_audvis-meg-vol-7-meg-inv.fif, which gets compressed from 360MB to 33MB…

5

So regarding your original question @skjerns, you could either try to get .fif.gz back into BIDS (probably a lot of effort and very low chances of success), or split up your data into multiple repositories.

Or I guess you could also share your data and not care about BIDS.

6

Thanks! What a pity. I’ve resorted to downsampling the data instead, that solved it for now.