how to save a Raw to .fif with missing information (annotations, proc_history)

đź’¬ Support & Discussions
1
  • MNE-Python version: 0.23.0
  • operating system: Linux

Dear all,
I am trying to save a mne’s Raw object in a .fif file.
This Raw file comprises EEG data and the related info which are manually loaded from edf files.
However, when trying to save it in .fif, it seems that some annotations are required:

if len(annotations) > 0:  # don't save empty annot
TypeError: object of type 'NoneType' has no len()

what I am doing is simply: eeg.save('./0001_raw.fif', buffer_size_sec=tot_duration, overwrite=True)

Am I doing something wrong?
thank you for your help

2

Hello @Chutlhu and welcome to the forum! How do you generate the data you wish to save? Also how is the error message related to the title of your posting? Did you forget to include some info?

3

Dear @richard ,
thank you for your reply.
You are very right indeed. In fact, at first, the error was:

  File "**/**/**/venv/lib/python3.8/site-packages/mne/io/proc_history.py", line 107, in _write_proc_history
    if len(info['proc_history']) > 0:
KeyError: 'proc_history'

Which I solved by creating a dummy entry as follows

eeg.info['proc_history'] = []

I m going to edit the title.

Regarding the data generation, unfortunately, I m not the one who did the conversion from the original .edf format (from the headset) to the mne's Raw object.
These Raw objects are used from plots and pre-processing function, which works with no problem.

Is it possible to add some dummy annotations in order to save the file?
The end goal is to release an eeg dataset in .fif format with a mne-based dataloader instead of a python’s pickle file.

edit:
Here is the print of my RawArray eeg_data:

>>> print(eeg_data)
<RawArray | 30 x 1520896 (5941.0 s), ~348.1 MB, data loaded>

>>> print(eeg_data.info)
<Info | 9 non-empty values
 bads: 3 items (STI 014, ESUTimestamp, SystemTimestamp)
 buffer_size_sec: 1.0
 ch_names: F3, F1, Fz, F2, F4, C3, C1, Cz, C2, C4, CPz, P3, P1, Pz, P2, P4, ...
 chs: 20 EEG, 8 MISC, 1 ECG, 1 EOG
 custom_ref_applied: False
 highpass: 0.0 Hz
 lowpass: 128.0 Hz
 meas_date: 2016-07-21 17:11:52 UTC
 nchan: 30
 projs: []
 sfreq: 256.0 Hz
>

Thank you

4

Thanks for providing the additional info. My understanding is that what you’re seeing really shouldn’t happen. Something seems to be off with the FIFF files you’re dealing with. Can you not use the EDF files directly? And why do you have a RawArray instead of a proper Raw object? And why is the same variable once a RawArray and once an Info? Could you maybe share some of your data and the code you’re using so we can try to get a better idea of what’s going on?

5

Dear @richard ,
thank you so much for your help.
I m sending you the data of a subject (as a zipped pickle):

here is the code to load it:

import pickle as pkl
from datetime import datetime, timezone
import numpy as np
import mne

sbj_rec_date = {
    '0001': '07/21/2016 17:11:52:556',
}

current_sbj = '0001'
path_to_data = 'path/to/0001.pkl'

with open(path_to_data, 'rb') as handle:
    data_dict = pkl.load(handle)

# this is the data I would like to save
eeg = data_dict['raw'] # raw array

print(eeg)

# update the date (requirebe by the new version of mne)
eeg_date = datetime.strptime(sbj_rec_date[current_sbj],"%m/%d/%Y %H:%M:%S:%f")
eeg_date = eeg_date.replace(tzinfo=timezone.utc)
eeg.info['meas_date'] = eeg_date

## Towards saving in fif file.. It fails because of no annotation
# 1st error: no buffer_size_sec
L = eeg._data.shape[-1]
duration = L / eeg.info['sfreq']
# eeg.save('./tmp_raw.fif.gz', buffer_size_sec=duration, overwrite=True)

# 2nd error: no proc history
# eeg.info['proc_history'] = []
# eeg.save('./tmp_raw.fif.gz', buffer_size_sec=duration, overwrite=True)

# 3rd error: no annotation
# ?????
# eeg.save('./tmp_raw.fif', buffer_size_sec=duration, overwrite=True)

Unfortunately, I don’t know how this pickle has been created, neither why it’s RawArray instead of a Raw object.
Thank you so much

6

Urgs. Pickles are not meant for long-term storage and certainly not for data sharing. Your best bet is to figure out which exact version of MNE-Python your colleague was using to create those files; set up an environment with that MNE-Python version; unpickle the files; and properly store them as FIFFs via raw.save(). Then you should be able to simply load them with a recent version of MNE-Python.

Anything else is going to be messy and may lead to data loss or problems like the ones you’re describing above.

7

Thank you very much for your help.
I do see now the consequences of storing files in such a way - I wish I knew it a few years ago.
I will try what you are suggesting

8

If you can find out the creation date of those files, we could then deduce the MNE version that was current at that time — and was likely used to generate the data!