Wrong in mne.io.read_raw_edf

Hi everyone,

I got wrong in read raw file:

import mne
raw = mne.io.read_raw_edf('/home/lsp/data/osa/osa/ndx/test~test_2350dab4-c9a0-480c-8b83-83c37e8adb90.EDF', preload=True)
print(raw.info)

And it returns

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[31], line 1
----> 1 raw = mne.io.read_raw_edf('/home/lsp/data/osa/osa/ndx/test~ test_2350dab4-c9a0-480c-8b83-83c37e8adb90.EDF',preload=True)
      2 print(raw.info)

File ~/.conda/envs/mne/lib/python3.11/site-packages/mne/io/edf/edf.py:1701, in read_raw_edf(input_fname, eog, misc, stim_channel, exclude, infer_types, include, preload, units, encoding, exclude_after_unique, verbose)
   1699 if ext != "edf":
   1700     raise NotImplementedError(f"Only EDF files are supported, got {ext}.")
-> 1701 return RawEDF(
   1702     input_fname=input_fname,
   1703     eog=eog,
   1704     misc=misc,
   1705     stim_channel=stim_channel,
   1706     exclude=exclude,
   1707     infer_types=infer_types,
   1708     preload=preload,
   1709     include=include,
   1710     units=units,
   1711     encoding=encoding,
   1712     exclude_after_unique=exclude_after_unique,
   1713     verbose=verbose,
   1714 )

File <decorator-gen-889>:12, in __init__(self, input_fname, eog, misc, stim_channel, exclude, infer_types, preload, include, units, encoding, exclude_after_unique, verbose)

File ~/.conda/envs/mne/lib/python3.11/site-packages/mne/io/edf/edf.py:158, in RawEDF.__init__(self, input_fname, eog, misc, stim_channel, exclude, infer_types, preload, include, units, encoding, exclude_after_unique, verbose)
    156 logger.info(f"Extracting EDF parameters from {input_fname}...")
    157 input_fname = os.path.abspath(input_fname)
--> 158 info, edf_info, orig_units = _get_info(
    159     input_fname,
    160     stim_channel,
    161     eog,
    162     misc,
    163     exclude,
    164     infer_types,
    165     preload,
    166     include,
    167     exclude_after_unique,
    168 )
    169 logger.info("Creating raw.info structure...")
    171 _validate_type(units, (str, None, dict), "units")

File ~/.conda/envs/mne/lib/python3.11/site-packages/mne/io/edf/edf.py:544, in _get_info(fname, stim_channel, eog, misc, exclude, infer_types, preload, include, exclude_after_unique)
    541 eog = eog if eog is not None else []
    542 misc = misc if misc is not None else []
--> 544 edf_info, orig_units = _read_header(
    545     fname, exclude, infer_types, include, exclude_after_unique
    546 )
    548 # XXX: `tal_ch_names` to pass to `_check_stim_channel` should be computed
    549 #      from `edf_info['ch_names']` and `edf_info['tal_idx']` but 'tal_idx'
    550 #      contains stim channels that are not TAL.
    551 stim_channel_idxs, _ = _check_stim_channel(stim_channel, edf_info["ch_names"])

File ~/.conda/envs/mne/lib/python3.11/site-packages/mne/io/edf/edf.py:518, in _read_header(fname, exclude, infer_types, include, exclude_after_unique)
    516 logger.info("%s file detected" % ext.upper())
    517 if ext in ("bdf", "edf"):
--> 518     return _read_edf_header(
    519         fname, exclude, infer_types, include, exclude_after_unique
    520     )
    521 elif ext == "gdf":
    522     return _read_gdf_header(fname, exclude, include), None

File ~/.conda/envs/mne/lib/python3.11/site-packages/mne/io/edf/edf.py:862, in _read_edf_header(fname, exclude, infer_types, include, exclude_after_unique)
    860 else:
    861     meas_date = fid.read(8).decode("latin-1")
--> 862     day, month, year = (int(x) for x in meas_date.split("."))
    863     year = year + 2000 if year < 85 else year + 1900
    865 meas_time = fid.read(8).decode("latin-1")

ValueError: not enough values to unpack (expected 3, got 1)

It seems that time information in head file is wrong. How can I edit it to be read by mne? This EDF file is exported from Natus Medical Device.

Can you please rename your EDF file so that it doesn’t contain a ~? I think this symbol is causing the issue here.

Hi cbrnr,

Thank you for your reply! I delete the ‘~’ symbol in file name and It still returns the same wrong information. I also change the file name into ‘1.edf’ and It doesn’t work either.

Can you please show the code and error message when you use 1.EDF?

Hi cbrnr,

This is the code and error message:

import mne
raw = mne.io.read_raw_edf('/home/lsp/data/osa/osa/ndx/1.EDF', preload=True)
print(raw.info)
Extracting EDF parameters from /home/lsp/data/osa/osa/ndx/1.EDF...
EDF file detected

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[34], line 1
----> 1 raw = mne.io.read_raw_edf('/home/lsp/data/osa/osa/ndx/1.EDF',preload=True)
      2 print(raw.info)

File ~/.conda/envs/mne/lib/python3.11/site-packages/mne/io/edf/edf.py:1701, in read_raw_edf(input_fname, eog, misc, stim_channel, exclude, infer_types, include, preload, units, encoding, exclude_after_unique, verbose)
   1699 if ext != "edf":
   1700     raise NotImplementedError(f"Only EDF files are supported, got {ext}.")
-> 1701 return RawEDF(
   1702     input_fname=input_fname,
   1703     eog=eog,
   1704     misc=misc,
   1705     stim_channel=stim_channel,
   1706     exclude=exclude,
   1707     infer_types=infer_types,
   1708     preload=preload,
   1709     include=include,
   1710     units=units,
   1711     encoding=encoding,
   1712     exclude_after_unique=exclude_after_unique,
   1713     verbose=verbose,
   1714 )

File <decorator-gen-889>:12, in __init__(self, input_fname, eog, misc, stim_channel, exclude, infer_types, preload, include, units, encoding, exclude_after_unique, verbose)

File ~/.conda/envs/mne/lib/python3.11/site-packages/mne/io/edf/edf.py:158, in RawEDF.__init__(self, input_fname, eog, misc, stim_channel, exclude, infer_types, preload, include, units, encoding, exclude_after_unique, verbose)
    156 logger.info(f"Extracting EDF parameters from {input_fname}...")
    157 input_fname = os.path.abspath(input_fname)
--> 158 info, edf_info, orig_units = _get_info(
    159     input_fname,
    160     stim_channel,
    161     eog,
    162     misc,
    163     exclude,
    164     infer_types,
    165     preload,
    166     include,
    167     exclude_after_unique,
    168 )
    169 logger.info("Creating raw.info structure...")
    171 _validate_type(units, (str, None, dict), "units")

File ~/.conda/envs/mne/lib/python3.11/site-packages/mne/io/edf/edf.py:544, in _get_info(fname, stim_channel, eog, misc, exclude, infer_types, preload, include, exclude_after_unique)
    541 eog = eog if eog is not None else []
    542 misc = misc if misc is not None else []
--> 544 edf_info, orig_units = _read_header(
    545     fname, exclude, infer_types, include, exclude_after_unique
    546 )
    548 # XXX: `tal_ch_names` to pass to `_check_stim_channel` should be computed
    549 #      from `edf_info['ch_names']` and `edf_info['tal_idx']` but 'tal_idx'
    550 #      contains stim channels that are not TAL.
    551 stim_channel_idxs, _ = _check_stim_channel(stim_channel, edf_info["ch_names"])

File ~/.conda/envs/mne/lib/python3.11/site-packages/mne/io/edf/edf.py:518, in _read_header(fname, exclude, infer_types, include, exclude_after_unique)
    516 logger.info("%s file detected" % ext.upper())
    517 if ext in ("bdf", "edf"):
--> 518     return _read_edf_header(
    519         fname, exclude, infer_types, include, exclude_after_unique
    520     )
    521 elif ext == "gdf":
    522     return _read_gdf_header(fname, exclude, include), None

File ~/.conda/envs/mne/lib/python3.11/site-packages/mne/io/edf/edf.py:862, in _read_edf_header(fname, exclude, infer_types, include, exclude_after_unique)
    860 else:
    861     meas_date = fid.read(8).decode("latin-1")
--> 862     day, month, year = (int(x) for x in meas_date.split("."))
    863     year = year + 2000 if year < 85 else year + 1900
    865 meas_time = fid.read(8).decode("latin-1")

ValueError: not enough values to unpack (expected 3, got 1)

Could it be that the extension must be lower case, i.e. .edf instead of .EDF?

Hi cbrnr,

I changed the file name into ‘1.edf’ and it still didn’t work.

import mne
raw = mne.io.read_raw_edf('/home/lsp/data/osa/osa/ndx/1.edf',preload=True)
print(raw.info)
Extracting EDF parameters from /home/lsp/data/osa/osa/ndx/1.edf...
EDF file detected

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[35], line 1
----> 1 raw = mne.io.read_raw_edf('/home/lsp/data/osa/osa/ndx/1.edf',preload=True)
      2 print(raw.info)

File ~/.conda/envs/mne/lib/python3.11/site-packages/mne/io/edf/edf.py:1701, in read_raw_edf(input_fname, eog, misc, stim_channel, exclude, infer_types, include, preload, units, encoding, exclude_after_unique, verbose)
   1699 if ext != "edf":
   1700     raise NotImplementedError(f"Only EDF files are supported, got {ext}.")
-> 1701 return RawEDF(
   1702     input_fname=input_fname,
   1703     eog=eog,
   1704     misc=misc,
   1705     stim_channel=stim_channel,
   1706     exclude=exclude,
   1707     infer_types=infer_types,
   1708     preload=preload,
   1709     include=include,
   1710     units=units,
   1711     encoding=encoding,
   1712     exclude_after_unique=exclude_after_unique,
   1713     verbose=verbose,
   1714 )

File <decorator-gen-889>:12, in __init__(self, input_fname, eog, misc, stim_channel, exclude, infer_types, preload, include, units, encoding, exclude_after_unique, verbose)

File ~/.conda/envs/mne/lib/python3.11/site-packages/mne/io/edf/edf.py:158, in RawEDF.__init__(self, input_fname, eog, misc, stim_channel, exclude, infer_types, preload, include, units, encoding, exclude_after_unique, verbose)
    156 logger.info(f"Extracting EDF parameters from {input_fname}...")
    157 input_fname = os.path.abspath(input_fname)
--> 158 info, edf_info, orig_units = _get_info(
    159     input_fname,
    160     stim_channel,
    161     eog,
    162     misc,
    163     exclude,
    164     infer_types,
    165     preload,
    166     include,
    167     exclude_after_unique,
    168 )
    169 logger.info("Creating raw.info structure...")
    171 _validate_type(units, (str, None, dict), "units")

File ~/.conda/envs/mne/lib/python3.11/site-packages/mne/io/edf/edf.py:544, in _get_info(fname, stim_channel, eog, misc, exclude, infer_types, preload, include, exclude_after_unique)
    541 eog = eog if eog is not None else []
    542 misc = misc if misc is not None else []
--> 544 edf_info, orig_units = _read_header(
    545     fname, exclude, infer_types, include, exclude_after_unique
    546 )
    548 # XXX: `tal_ch_names` to pass to `_check_stim_channel` should be computed
    549 #      from `edf_info['ch_names']` and `edf_info['tal_idx']` but 'tal_idx'
    550 #      contains stim channels that are not TAL.
    551 stim_channel_idxs, _ = _check_stim_channel(stim_channel, edf_info["ch_names"])

File ~/.conda/envs/mne/lib/python3.11/site-packages/mne/io/edf/edf.py:518, in _read_header(fname, exclude, infer_types, include, exclude_after_unique)
    516 logger.info("%s file detected" % ext.upper())
    517 if ext in ("bdf", "edf"):
--> 518     return _read_edf_header(
    519         fname, exclude, infer_types, include, exclude_after_unique
    520     )
    521 elif ext == "gdf":
    522     return _read_gdf_header(fname, exclude, include), None

File ~/.conda/envs/mne/lib/python3.11/site-packages/mne/io/edf/edf.py:862, in _read_edf_header(fname, exclude, infer_types, include, exclude_after_unique)
    860 else:
    861     meas_date = fid.read(8).decode("latin-1")
--> 862     day, month, year = (int(x) for x in meas_date.split("."))
    863     year = year + 2000 if year < 85 else year + 1900
    865 meas_time = fid.read(8).decode("latin-1")

ValueError: not enough values to unpack (expected 3, got 1)

Hello,

@cbrnr I’m confused as to why you believe it’s the file name? The traceback shows:

So it’s a problem with the date handling

@Jack_L Could you please share the output of

import mne
mne.sys_info()

Thanks,
Richard

Hi Richard,

import mne
print(mne.sys_info())
Platform             Linux-6.5.0-44-generic-x86_64-with-glibc2.35
Python               3.11.7 | packaged by conda-forge | (main, Dec 23 2023, 14:43:09) [GCC 12.3.0]
Executable           /home/lsp/.conda/envs/mne/bin/python
CPU                  x86_64 (12 cores)
Memory               53.9 GB

Core
├☑ mne               1.7.1 (latest release)
├☑ numpy             1.26.3 (OpenBLAS 0.3.26 with 12 threads)
├☑ scipy             1.12.0
└☑ matplotlib        3.8.2 (backend=module://matplotlib_inline.backend_inline)

Numerical (optional)
├☑ sklearn           1.4.0
├☑ numba             0.58.1
├☑ nibabel           5.2.0
├☑ nilearn           0.10.4
├☑ dipy              1.7.0
├☑ openmeeg          2.5.7
├☑ pandas            1.5.3
├☑ h5io              0.2.1
├☑ h5py              3.10.0
└☐ unavailable       cupy

Visualization (optional)
├☑ pyvista           0.43.2 (OpenGL 4.1 (Core Profile) Mesa 22.2.5 via SVGA3D; build: RELEASE;  LLVM;)
├☑ pyvistaqt         0.11.0
├☑ vtk               9.2.6
├☑ qtpy              2.4.1 (PyQt5=5.15.8)
├☑ pyqtgraph         0.13.3
├☑ mne-qt-browser    0.6.1
├☑ ipywidgets        8.1.1
├☑ trame_client      2.15.0
├☑ trame_server      2.16.0
├☑ trame_vtk         2.8.0
├☑ trame_vuetify     2.4.2
└☐ unavailable       ipympl

Ecosystem (optional)
├☑ mne-features      0.3
├☑ mne-connectivity  0.6.0
├☑ mne-icalabel      0.6.0
├☑ eeglabio          0.0.2-4
├☑ mffpy             0.8.0
└☐ unavailable       mne-bids, mne-nirs, mne-bids-pipeline, neo, edfio, pybv
None

I believe @larsoner just recently worked on parts of our date handling, maybe he’d want to take a look?

Sorry, my bad, I skimmed the traceback too quickly…

1 Like

This looks like it’s related to how EDF stores the date. This is the line that fails:

    861     meas_date = fid.read(8).decode("latin-1")
--> 862     day, month, year = (int(x) for x in meas_date.split("."))

So the code expects whatever string this is to have a.b.c but it does not have any dots in it.

This is probably worth opening a MNE-Python issue about. But if, after the failure, you do import pdb; pdb.pm() and then do p meas_date what does it say? That would tell us what is actually in the EDF data that’s trying to be parsed.

2 Likes

Hi Larson,

Thanks for your reply!
Here is the result

import pdb
pdb.pm()
/home/lsp/.conda/envs/mne/lib/python3.11/site-packages/mne/io/edf/edf.py(862)_read_edf_header()
    860         else:
    861             meas_date = fid.read(8).decode("latin-1")
--> 862             day, month, year = (int(x) for x in meas_date.split("."))
    863             year = year + 2000 if year < 85 else year + 1900
    864 

ipdb>  meas_date

'    24  '

ipdb> 

It seems that EMbla NDX did not output the edf header file correctly. Are there any ways to edit the header file?

You could in theory fid.tell(), open the file for editing, and insert values that work.

But really in MNE-Python we could be more tolerant here, and put the parsing in a try/except and in the case of an error, set the meas_date to None for example.

1 Like

PR open here: Safeguard loading of meas_date in EDF files by mscheltienne · Pull Request #12754 · mne-tools/mne-python · GitHub
The fix should be available in MNE 1.8.

Mathieu

3 Likes

Much thanks for your help!

Hi larsoner,

I finally found the difference between this file and other files I can read by hex editor.

This is the header of the edf file I cannot read:

This file I can read:

Obviously, the ‘startdate’ information is lost. Maybe the edf file template exported by EMbla NDX was adjusted.

I inserted space in fid(88) and finally the problem was solved.

file = '/home/lsp/1.edf'
with open(file, 'rb') as fid:
    origin = fid.read()
    before = origin[:88]
    content = b" "*80
    after = origin[88:]
    new = before + content + after
with open(file, 'wb') as fid:
    fid.write(new)

And in hex editor, the location of startdate has been filled with space:

Extracting EDF parameters from /home/lsp/1.edf...
EDF file detected
Setting channel info strcture...
Creating raw.info structure...
/time/ipykernel_115026/2941939665.py:2: RuntimeWarning: Number of records from the header does not match the file size (perhaps the recording was not stopped before exiting). Inferring from the file size.
    raw = mne.io.read_raw_edf('/home/lsp/1.edf', preload=True)
Reading 0 ... 19061759 =         0.000 ... 37229.998 secs...
<Info | 8 non-empty values
 bads: []
 ch_names: Patient Event, C3, C4, Cz 1861, 5 , F4, F7, F8, Fz, Fp1, Fp2, ...
 chs: 24 EEG
 custom_ref_applied: False
 highpass: 0.0 Hz
 lowpass: 256.0 Hz
 meas_date: 2023-12-26 21:05:24 UTC
 nchan: 24
 projs: []
 sfreq: 512.0 Hz
 subject_info: 4 items (dict)
>

You can also use EDFbrowser to check if your original file is valid EDF.

2 Likes

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.