Hi,
I can’t seem to save filtered data or epochs when using mne.save(), but it let’s me save ICA.
This is the error I get every time:
UnicodeEncodeError: ‘latin-1’ codec can’t encode character ‘\ufffd’ in position 1: ordinal not in range(256)
The loaded data is in .set format.
If anyone knows what might be the problem I’d be very thankful if you could point it out.
Thanks in advance!
Hello @Ana and welcome to the forum!
Please share the exact line(s) of code that produced the error message, and please share the entire error message as well.
Thank you,
Richard
Hi, thank you for answering.
This is the snippet of the code:
p = os.getcwd()
path_pom = p + '/' + idP[0] + '-filt-raw.fif'
raw_filt.save(path_pom, overwrite=True)
And this is the error:
Traceback (most recent call last):
File "D:\Faks\set_preprocessing.py", line 36, in <module>
raw_filt.save(path_pom, overwrite=True)
File "<decorator-gen-235>", line 12, in save
File "C:\Users\AppData\Roaming\Python\Python310\site-packages\mne\io\base.py", line 1467, in save
_write_raw(fname, self, info, picks, fmt, data_type, reset_range,
File "C:\Users\AppData\Roaming\Python\Python310\site-packages\mne\io\base.py", line 2216, in _write_raw
cals = _start_writing_raw(fid, info, picks, data_type,
File "C:\Users\AppData\Roaming\Python\Python310\site-packages\mne\io\base.py", line 2425, in _start_writing_raw
_write_annotations(fid, annotations)
File "C:\Users\AppData\Roaming\Python\Python310\site-packages\mne\annotations.py", line 1012, in _write_annotations
write_name_list(fid, FIFF.FIFF_COMMENT, _prep_name_list(
File "C:\Users\AppData\Roaming\Python\Python310\site-packages\mne\io\write.py", line 146, in write_name_list
write_string(fid, kind, ':'.join(data))
File "C:\Users\AppData\Roaming\Python\Python310\site-packages\mne\io\write.py", line 132, in write_string
str_data = data.encode('latin1')
UnicodeEncodeError: 'latin-1' codec can't encode character '\ufffd' in position 1: ordinal not in range(256)
My guess is no, because we’re limited by the FIFF file format. But maybe it can be extended? Is this format documented somewhere BTW?
In that case we shouldn’t allow setting Unicode annotations in the first place 
I agree with @cbrnr that this looks like a FIFF format problem. IMO we should look into amending the standard / adapting our writer/reader to handle UTF-8 gracefully by default. If we can’t get the standard changed, I’d consider this a big enough deal that we should provide hdf5 writers for all objects that can have annotations. It’s just not OK in this day and age to force ASCII only in freeform user-generated text like an annotation.
I completely agree! But again, does anyone have a link to the FIFF “standard”? AFAIK, this is a proprietary format, which is not openly documented. If that is true, we should definitely discuss switching to HDF5 and adapting our internal data structure accordingly, because as the largest open source EEG/MEG Python package, I wouldn’t want to rely on a closed-source file format. Even more so because it has been holding us back several times already.
Assuming you are aware of this? GitHub - mne-tools/fiff-constants: Bookkeeping and documentation of FIFF file format constants It’s not an articulated standard in the normal sense (i.e., not a narrative document explaining file structure), but it is open source.
I am aware of this repository, but it doesn’t seem to contain the full specifications. Would someone be able to implement a reader from scratch using only what’s available there? These are mostly definitions of constants, but I haven’t looked very closely.
Please follow the issue here: