Problem with read_raw_eyelink()

💬 Support & Discussions
,
1

Hello,

I am trying to import an .asc file using read_raw_eyelink():

raw = read_raw_eyelink(fname)

However, I get the following error:

Loading /Users/dani/Desktop/test.asc
Pixel coordinate data detected.
Pass `scalings=dict(eyegaze=1e3)` when using plot method to make traces more legible.
Pupil-size area reported.
---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
File ~/anaconda3/envs/mne-env/lib/python3.11/site-packages/pandas/core/internals/construction.py:934, in _finalize_columns_and_data(content, columns, dtype)
    933 try:
--> 934     columns = _validate_or_indexify_columns(contents, columns)
    935 except AssertionError as err:
    936     # GH#26429 do not raise user-facing AssertionError

File ~/anaconda3/envs/mne-env/lib/python3.11/site-packages/pandas/core/internals/construction.py:981, in _validate_or_indexify_columns(content, columns)
    979 if not is_mi_list and len(columns) != len(content):  # pragma: no cover
    980     # caller's responsibility to check for this...
--> 981     raise AssertionError(
    982         f"{len(columns)} columns passed, passed data had "
    983         f"{len(content)} columns"
    984     )
    985 if is_mi_list:
    986     # check if nested list column, length of each sub-list should be equal

AssertionError: 10 columns passed, passed data had 9 columns

The above exception was the direct cause of the following exception:

ValueError                                Traceback (most recent call last)
Cell In[4], line 2
      1 fname = "/Users/dani/Desktop/test.asc"
----> 2 raw = read_raw_eyelink(fname)

File ~/anaconda3/envs/mne-env/lib/python3.11/site-packages/mne/io/eyelink/eyelink.py:350, in read_raw_eyelink(fname, preload, verbose, create_annotations, apply_offsets, find_overlaps, overlap_threshold, gap_description)
    342 if extension not in ".asc":
    343     raise ValueError(
    344         "This reader can only read eyelink .asc files."
    345         f" Got extension {extension} instead. consult eyelink"
    346         " manual for converting eyelink data format (.edf)"
    347         " files to .asc format."
    348     )
--> 350 return RawEyelink(
    351     fname,
    352     preload=preload,
    353     verbose=verbose,
    354     create_annotations=create_annotations,
    355     apply_offsets=apply_offsets,
    356     find_overlaps=find_overlaps,
    357     overlap_threshold=overlap_threshold,
    358     gap_desc=gap_description,
    359 )

File :12, in __init__(self, fname, preload, verbose, create_annotations, apply_offsets, find_overlaps, overlap_threshold, gap_desc)

File ~/anaconda3/envs/mne-env/lib/python3.11/site-packages/mne/io/eyelink/eyelink.py:457, in RawEyelink.__init__(self, fname, preload, verbose, create_annotations, apply_offsets, find_overlaps, overlap_threshold, gap_desc)
    455 sfreq = _get_sfreq(self._event_lines["SAMPLES"][0])
    456 col_names, ch_names = self._infer_col_names()
--> 457 self._create_dataframes(
    458     col_names, sfreq, find_overlaps=find_overlaps, threshold=overlap_threshold
    459 )
    460 info = self._create_info(ch_names, sfreq)
    461 eye_ch_data = self.dataframes["samples"][ch_names]

File ~/anaconda3/envs/mne-env/lib/python3.11/site-packages/mne/io/eyelink/eyelink.py:745, in RawEyelink._create_dataframes(self, col_names, sfreq, find_overlaps, threshold)
    742 first_samp = self._event_lines["START"][0][0]
    744 # dataframe for samples
--> 745 self.dataframes["samples"] = pd.DataFrame(
    746     self._sample_lines, columns=col_names["sample"]
    747 )
    748 if "HREF" in self._rec_info:
    749     pos_names = (
    750         EYELINK_COLS["pos"]["left"][:-1] + EYELINK_COLS["pos"]["right"][:-1]
    751     )

File ~/anaconda3/envs/mne-env/lib/python3.11/site-packages/pandas/core/frame.py:782, in DataFrame.__init__(self, data, index, columns, dtype, copy)
    780     if columns is not None:
    781         columns = ensure_index(columns)
--> 782     arrays, columns, index = nested_data_to_arrays(
    783         # error: Argument 3 to "nested_data_to_arrays" has incompatible
    784         # type "Optional[Collection[Any]]"; expected "Optional[Index]"
    785         data,
    786         columns,
    787         index,  # type: ignore[arg-type]
    788         dtype,
    789     )
    790     mgr = arrays_to_mgr(
    791         arrays,
    792         columns,
   (...)
    795         typ=manager,
    796     )
    797 else:

File ~/anaconda3/envs/mne-env/lib/python3.11/site-packages/pandas/core/internals/construction.py:498, in nested_data_to_arrays(data, columns, index, dtype)
    495 if is_named_tuple(data[0]) and columns is None:
    496     columns = ensure_index(data[0]._fields)
--> 498 arrays, columns = to_arrays(data, columns, dtype=dtype)
    499 columns = ensure_index(columns)
    501 if index is None:

File ~/anaconda3/envs/mne-env/lib/python3.11/site-packages/pandas/core/internals/construction.py:840, in to_arrays(data, columns, dtype)
    837     data = [tuple(x) for x in data]
    838     arr = _list_to_arrays(data)
--> 840 content, columns = _finalize_columns_and_data(arr, columns, dtype)
    841 return content, columns

File ~/anaconda3/envs/mne-env/lib/python3.11/site-packages/pandas/core/internals/construction.py:937, in _finalize_columns_and_data(content, columns, dtype)
    934     columns = _validate_or_indexify_columns(contents, columns)
    935 except AssertionError as err:
    936     # GH#26429 do not raise user-facing AssertionError
--> 937     raise ValueError(err) from err
    939 if len(contents) and contents[0].dtype == np.object_:
    940     contents = convert_object_array(contents, dtype=dtype)

ValueError: 10 columns passed, passed data had 9 columns

My operating system, MNE version, and Python version are:

  • MNE version: 1.4.2
  • operating system: macOS-13.4.1-arm64-arm-64bit
  • Python: 3.11.4

Has anyone ever stumbled upon something like this? The eye tracking file itself is not corrupted (I can import it in RStudio and it looks fine).

This is a link to a test file: https://drive.google.com/file/d/1QQbimGsPpc58WgJCQ91K9AyQ2oveNzpA/view?usp=sharing

Thanks so much and best wishes,
Dani

2

@scott-huberty Could you have a look?

3

Hi @DaniG and @mscheltienne , I’m taking a look at the test file now. and will respond shortly.

Scott

4

Hi again @DaniG , thanks again for reporting your issue. read_raw_eyelink is a relatively new addition to MNE so it’s nice to know if/when folks run into problems.

I have a question about how you converted your EDF file to ASCII. It’s not to suggest that you’ve done something wrong, it’s just that EyeLink’s EDF2ASC gives the user a lot of freedom to configure the conversion output, which can make it hard for our reader to anticipate all the possible combinations of incoming data.

I’m wondering, can you check your EDF2ASC application, to see if you are blocking the flags column from the conversion? ((see the photo below). If you converted your file using the edf2asc command line utility, did you include -nflags parameter, thus blocking the flags column?

From what I can tell, you have an EyeLink II file, so it should by default contain this information. As you have shown with your stack trace, read_raw_eyelink can’t currently handle this column not being present :slightly_smiling_face:

Please let me know once you’ve checked, and we’ll go from there!

‎eyelink_troubleshoot.‎001
‎eyelink_troubleshoot.‎0011024×768 267 KB

1 Like
5

@scott-huberty Thanks for digging, are you extracting some information from this column or is it simply “nice to have”?

6

@mscheltienne initially I anticipated that the data in this column could eventually be used to annotate the data for periods that the tracker lost the eye - but we’re not currently doing anything with the column; So there is the option that we just change the behavior of read_raw_eyelink to ignore this column when reading.

7

The reader should probably support a on_missing : 'raise' | 'warn' | 'ignore' as other part of MNE’s API which will raise, warn or ignore for missing non-critical information, e.g. information which could generate annotations. Of course, if the file is missing critical information, the reader should always raise with an helpful error message.

Anyway, it’s not the highest priority, you could open an issue on GH linking to this post to keep track. Thanks for digging, I had no idea those options existed in the EDFConverter!

1 Like
8

Will do!

Improving read_raw_eyelink is on my list of to-do’s this summer, I think it will be time well spent. I’ll open a ticket for this specific issue.

Thanks!

1 Like
9

Hi @scott-huberty & @mscheltienne ,

thanks so much for the quick replies!

I tried converting the EDF files with the EDF Converter (version 4.3.210) using both options (i.e., once with “Block Flags Output” checked and once not checked), but I get the same error when trying to import the file into MNE.

Other setting in Preferences are:

  • Samples / Events: Output Samples and Events
  • Binocular Recording: Output Binocular Data
  • Eye Position Type: Gaze

Screenshot 2023-07-18 at 09.38.01
Screenshot 2023-07-18 at 09.38.011522×1268 90.2 KB

This is the link to the original EDF file in case this is helpful: test.EDF - Google Drive

Best wishes and thanks again for your help,
Dani

10

Hi Dani, thanks for the info and for sharing that file. I can replicate your error on my end and we’ve opened a bug ticket:

I’ll keep you updated once we implement a fix!

2 Likes
11

This issue was fixed via PR:

3 Likes
12

@DaniG, to use the updated code that will work with your file, you have two options:

  1. wait until the next stable release of MNE.
  2. Install the development version of MNE (assuming you haven’t already): If you aren’t familiar with this, MNE gives a pretty thorough walk through
1 Like
13

Thanks a lot for fixing this so quickly :slight_smile: