# Problem with read\_raw\_eyelink() **URL:** \ **Category:** Support & Discussions\ **Tags:** preprocessing, eyetracking\ **Created:** [July 17, 2023, 8:50am UTC](https://mne.discourse.group/t/problem-with-read-raw-eyelink/7243 "2023-07-17T08:50:34Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![DaniG](https://yyz2.discourse-cdn.com/free1/user_avatar/mne.discourse.group/danig/32/2809_2.png) [@DaniG](https://mne.discourse.group/u/DaniG)\ **Post date:** [July 17, 2023, 8:50am UTC](https://mne.discourse.group/t/problem-with-read-raw-eyelink/7243/1 "2023-07-17T08:50:34Z")
Hello, I am trying to import an .asc file using `read_raw_eyelink()`: `raw = read_raw_eyelink(fname)` However, I get the following error: ```python Loading /Users/dani/Desktop/test.asc Pixel coordinate data detected. Pass `scalings=dict(eyegaze=1e3)` when using plot method to make traces more legible. Pupil-size area reported. --------------------------------------------------------------------------- AssertionError Traceback (most recent call last) File ~/anaconda3/envs/mne-env/lib/python3.11/site-packages/pandas/core/internals/construction.py:934, in _finalize_columns_and_data(content, columns, dtype) 933 try: --> 934 columns = _validate_or_indexify_columns(contents, columns) 935 except AssertionError as err: 936 # GH#26429 do not raise user-facing AssertionError File ~/anaconda3/envs/mne-env/lib/python3.11/site-packages/pandas/core/internals/construction.py:981, in _validate_or_indexify_columns(content, columns) 979 if not is_mi_list and len(columns) != len(content): # pragma: no cover 980 # caller's responsibility to check for this... --> 981 raise AssertionError( 982 f"{len(columns)} columns passed, passed data had " 983 f"{len(content)} columns" 984 ) 985 if is_mi_list: 986 # check if nested list column, length of each sub-list should be equal AssertionError: 10 columns passed, passed data had 9 columns The above exception was the direct cause of the following exception: ValueError Traceback (most recent call last) Cell In[4], line 2 1 fname = "/Users/dani/Desktop/test.asc" ----> 2 raw = read_raw_eyelink(fname) File ~/anaconda3/envs/mne-env/lib/python3.11/site-packages/mne/io/eyelink/eyelink.py:350, in read_raw_eyelink(fname, preload, verbose, create_annotations, apply_offsets, find_overlaps, overlap_threshold, gap_description) 342 if extension not in ".asc": 343 raise ValueError( 344 "This reader can only read eyelink .asc files." 345 f" Got extension {extension} instead. consult eyelink" 346 " manual for converting eyelink data format (.edf)" 347 " files to .asc format." 348 ) --> 350 return RawEyelink( 351 fname, 352 preload=preload, 353 verbose=verbose, 354 create_annotations=create_annotations, 355 apply_offsets=apply_offsets, 356 find_overlaps=find_overlaps, 357 overlap_threshold=overlap_threshold, 358 gap_desc=gap_description, 359 ) File :12, in __init__ (self, fname, preload, verbose, create_annotations, apply_offsets, find_overlaps, overlap_threshold, gap_desc) File ~/anaconda3/envs/mne-env/lib/python3.11/site-packages/mne/io/eyelink/eyelink.py:457, in RawEyelink. __init__ (self, fname, preload, verbose, create_annotations, apply_offsets, find_overlaps, overlap_threshold, gap_desc) 455 sfreq = _get_sfreq(self._event_lines["SAMPLES"][0]) 456 col_names, ch_names = self._infer_col_names() --> 457 self._create_dataframes( 458 col_names, sfreq, find_overlaps=find_overlaps, threshold=overlap_threshold 459 ) 460 info = self._create_info(ch_names, sfreq) 461 eye_ch_data = self.dataframes["samples"][ch_names] File ~/anaconda3/envs/mne-env/lib/python3.11/site-packages/mne/io/eyelink/eyelink.py:745, in RawEyelink._create_dataframes(self, col_names, sfreq, find_overlaps, threshold) 742 first_samp = self._event_lines["START"][0][0] 744 # dataframe for samples --> 745 self.dataframes["samples"] = pd.DataFrame( 746 self._sample_lines, columns=col_names["sample"] 747 ) 748 if "HREF" in self._rec_info: 749 pos_names = ( 750 EYELINK_COLS["pos"]["left"][:-1] + EYELINK_COLS["pos"]["right"][:-1] 751 ) File ~/anaconda3/envs/mne-env/lib/python3.11/site-packages/pandas/core/frame.py:782, in DataFrame. __init__ (self, data, index, columns, dtype, copy) 780 if columns is not None: 781 columns = ensure_index(columns) --> 782 arrays, columns, index = nested_data_to_arrays( 783 # error: Argument 3 to "nested_data_to_arrays" has incompatible 784 # type "Optional[Collection[Any]]"; expected "Optional[Index]" 785 data, 786 columns, 787 index, # type: ignore[arg-type] 788 dtype, 789 ) 790 mgr = arrays_to_mgr( 791 arrays, 792 columns, (...) 795 typ=manager, 796 ) 797 else: File ~/anaconda3/envs/mne-env/lib/python3.11/site-packages/pandas/core/internals/construction.py:498, in nested_data_to_arrays(data, columns, index, dtype) 495 if is_named_tuple(data[0]) and columns is None: 496 columns = ensure_index(data[0]._fields) --> 498 arrays, columns = to_arrays(data, columns, dtype=dtype) 499 columns = ensure_index(columns) 501 if index is None: File ~/anaconda3/envs/mne-env/lib/python3.11/site-packages/pandas/core/internals/construction.py:840, in to_arrays(data, columns, dtype) 837 data = [tuple(x) for x in data] 838 arr = _list_to_arrays(data) --> 840 content, columns = _finalize_columns_and_data(arr, columns, dtype) 841 return content, columns File ~/anaconda3/envs/mne-env/lib/python3.11/site-packages/pandas/core/internals/construction.py:937, in _finalize_columns_and_data(content, columns, dtype) 934 columns = _validate_or_indexify_columns(contents, columns) 935 except AssertionError as err: 936 # GH#26429 do not raise user-facing AssertionError --> 937 raise ValueError(err) from err 939 if len(contents) and contents[0].dtype == np.object_: 940 contents = convert_object_array(contents, dtype=dtype) ValueError: 10 columns passed, passed data had 9 columns ``` My operating system, MNE version, and Python version are: - MNE version: 1.4.2 - operating system: macOS-13.4.1-arm64-arm-64bit - Python: 3.11.4 Has anyone ever stumbled upon something like this? The eye tracking file itself is not corrupted (I can import it in RStudio and it looks fine). This is a link to a test file: [https://drive.google.com/file/d/1QQbimGsPpc58WgJCQ91K9AyQ2oveNzpA/view?usp=sharing](https://drive.google.com/file/d/1QQbimGsPpc58WgJCQ91K9AyQ2oveNzpA/view?usp=sharing) Thanks so much and best wishes, Dani ---
**Author:** ![mscheltienne](https://yyz2.discourse-cdn.com/free1/user_avatar/mne.discourse.group/mscheltienne/32/827_2.png) [@mscheltienne](https://mne.discourse.group/u/mscheltienne)\ **Post date:** [July 17, 2023, 12:56pm UTC](https://mne.discourse.group/t/problem-with-read-raw-eyelink/7243/2 "2023-07-17T12:56:06Z")
@scott-huberty Could you have a look? ---
**Author:** ![scott-huberty](https://yyz2.discourse-cdn.com/free1/user_avatar/mne.discourse.group/scott-huberty/32/2715_2.png) [@scott-huberty](https://mne.discourse.group/u/scott-huberty)\ **Post date:** [July 17, 2023, 1:08pm UTC](https://mne.discourse.group/t/problem-with-read-raw-eyelink/7243/3 "2023-07-17T13:08:05Z")
Hi @DaniG and @mscheltienne , I’m taking a look at the test file now. and will respond shortly. Scott ---
**Author:** ![scott-huberty](https://yyz2.discourse-cdn.com/free1/user_avatar/mne.discourse.group/scott-huberty/32/2715_2.png) [@scott-huberty](https://mne.discourse.group/u/scott-huberty)\ **Post date:** [July 17, 2023, 1:50pm UTC](https://mne.discourse.group/t/problem-with-read-raw-eyelink/7243/4 "2023-07-17T13:50:34Z")
Hi again @DaniG , thanks again for reporting your issue. `read_raw_eyelink` is a relatively new addition to MNE so it’s nice to know if/when folks run into problems. I have a question about how you converted your EDF file to ASCII. It’s not to suggest that you’ve done something wrong, it’s just that EyeLink’s EDF2ASC gives the user _a lot_ of freedom to configure the conversion output, which can make it hard for our reader to anticipate all the possible combinations of incoming data. I’m wondering, can you check your EDF2ASC application, to see if you are blocking the flags column from the conversion? ((see the photo below). If you converted your file using the `edf2asc` command line utility, did you include `-nflags` parameter, thus blocking the flags column? From what I can tell, you have an EyeLink II file, so it should by default contain this information. As you have shown with your stack trace, `read_raw_eyelink` can’t currently handle this column not being present 🙂 Please let me know once you’ve checked, and we’ll go from there! ![‎eyelink_troubleshoot.‎001](https://global.discourse-cdn.com/free1/uploads/mne/original/2X/3/35ef060aa994e085df550a1dab06954a5c830542.png) ---
**Author:** ![mscheltienne](https://yyz2.discourse-cdn.com/free1/user_avatar/mne.discourse.group/mscheltienne/32/827_2.png) [@mscheltienne](https://mne.discourse.group/u/mscheltienne)\ **Post date:** [July 17, 2023, 2:07pm UTC](https://mne.discourse.group/t/problem-with-read-raw-eyelink/7243/5 "2023-07-17T14:07:35Z")
@scott-huberty Thanks for digging, are you extracting some information from this column or is it simply “nice to have”? ---
**Author:** ![scott-huberty](https://yyz2.discourse-cdn.com/free1/user_avatar/mne.discourse.group/scott-huberty/32/2715_2.png) [@scott-huberty](https://mne.discourse.group/u/scott-huberty)\ **Post date:** [July 17, 2023, 2:15pm UTC](https://mne.discourse.group/t/problem-with-read-raw-eyelink/7243/6 "2023-07-17T14:15:26Z")
@mscheltienne initially I anticipated that the data in this column could eventually be used to annotate the data for periods that the tracker lost the eye - but we’re not currently doing anything with the column; So there is the option that we just change the behavior of `read_raw_eyelink` to ignore this column when reading. ---
**Author:** ![mscheltienne](https://yyz2.discourse-cdn.com/free1/user_avatar/mne.discourse.group/mscheltienne/32/827_2.png) [@mscheltienne](https://mne.discourse.group/u/mscheltienne)\ **Post date:** [July 17, 2023, 3:33pm UTC](https://mne.discourse.group/t/problem-with-read-raw-eyelink/7243/7 "2023-07-17T15:33:31Z")
The reader should probably support a `on_missing : 'raise' | 'warn' | 'ignore'` as other part of MNE’s API which will raise, warn or ignore for missing non-critical information, e.g. information which could generate annotations. Of course, if the file is missing critical information, the reader should always raise with an helpful error message. Anyway, it’s not the highest priority, you could open an issue on GH linking to this post to keep track. Thanks for digging, I had no idea those options existed in the EDFConverter! ---
**Author:** ![scott-huberty](https://yyz2.discourse-cdn.com/free1/user_avatar/mne.discourse.group/scott-huberty/32/2715_2.png) [@scott-huberty](https://mne.discourse.group/u/scott-huberty)\ **Post date:** [July 17, 2023, 3:39pm UTC](https://mne.discourse.group/t/problem-with-read-raw-eyelink/7243/8 "2023-07-17T15:39:44Z")
Will do! Improving `read_raw_eyelink` is on my list of to-do’s this summer, I think it will be time well spent. I’ll open a ticket for this specific issue. Thanks! ---
**Author:** ![DaniG](https://yyz2.discourse-cdn.com/free1/user_avatar/mne.discourse.group/danig/32/2809_2.png) [@DaniG](https://mne.discourse.group/u/DaniG)\ **Post date:** [July 18, 2023, 8:42am UTC](https://mne.discourse.group/t/problem-with-read-raw-eyelink/7243/9 "2023-07-18T08:42:09Z")
Hi @scott-huberty & @mscheltienne , thanks so much for the quick replies! I tried converting the EDF files with the EDF Converter (version 4.3.210) using both options (i.e., once with “Block Flags Output” checked and once not checked), but I get the same error when trying to import the file into MNE. Other setting in Preferences are: - Samples / Events: Output Samples and Events - Binocular Recording: Output Binocular Data - Eye Position Type: Gaze ![Screenshot 2023-07-18 at 09.38.01](https://global.discourse-cdn.com/free1/uploads/mne/original/2X/6/6cf2d343662115b0db48a8727bffe2e843069fdf.jpeg) This is the link to the original EDF file in case this is helpful: [test.EDF - Google Drive](https://drive.google.com/file/d/1XOi-w0KhSwtt3yvBPPzYQlCDJ9rDeWo8/view?usp=drive_link) Best wishes and thanks again for your help, Dani ---
**Author:** ![scott-huberty](https://yyz2.discourse-cdn.com/free1/user_avatar/mne.discourse.group/scott-huberty/32/2715_2.png) [@scott-huberty](https://mne.discourse.group/u/scott-huberty)\ **Post date:** [July 18, 2023, 12:37pm UTC](https://mne.discourse.group/t/problem-with-read-raw-eyelink/7243/10 "2023-07-18T12:37:05Z")
Hi Dani, thanks for the info and for sharing that file. I can replicate your error on my end and we’ve opened a bug ticket: > > > \### Description of the problem > > Per the discussion in this thread: https://mne.d…iscourse.group/t/problem-with-read-raw-eyelink/7243/9 > > \`read\_raw\_eyelink\` (which reads ASCII files in a tabular structure), assumes that a certain column that contains warnings (tracker droput etc) for each sample is always present. As the the researcher in the forum has shown, this is not the case for their data. > > > The info in this column isn't currently used by \`read\_raw\_eyelink\`, so we could change the reader to disregard the column entirely. Or, if we think we will enhance the reader in the future to use the information in this column, we could try to make it smart enough to emit a warning when the column isn't present. > > > I think I will shift my attention this week to this bug and #11758, since improving this reader and adding more tests is something that we wanted to complete this summer. > > \### Steps to reproduce > > \`\`\`Python > Download the file at the link below and pass it to run \`mne.io.read\_raw\_eyelink\` > \`\`\` > > > \### Link to data > > https://drive.google.com/file/d/1QQbimGsPpc58WgJCQ91K9AyQ2oveNzpA/view?usp=sharing > > \### Expected results > > To read the file successfully > > \### Actual results > > \`\`\` > from mne.io import read\_raw\_eyelink > read\_raw\_eyelink("./test.asc") > > ValueError Traceback (most recent call last) > Cell In\[3\], line 1 > \----\> 1 read\_raw\_eyelink("./test\_noflags.asc") > > File ~/github\_repos/mne-python/mne/io/eyelink/eyelink.py:125, in read\_raw\_eyelink(fname, preload, verbose, create\_annotations, apply\_offsets, find\_overlaps, overlap\_threshold, gap\_description) > 67 """Reader for an Eyelink .asc file. > 68 > 69 Parameters > (...) > 121 \`\`'BAD\_ACQ\_SKIP'\`\`. > 122 """ > 123 fname = \_check\_fname(fname, overwrite="read", must\_exist=True, name="fname") > \--\> 125 raw\_eyelink = RawEyelink( > 126 fname, > 127 preload=preload, > 128 verbose=verbose, > 129 create\_annotations=create\_annotations, > 130 apply\_offsets=apply\_offsets, > 131 find\_overlaps=find\_overlaps, > 132 overlap\_threshold=overlap\_threshold, > 133 gap\_desc=gap\_description, > 134 ) > 135 return raw\_eyelink > > File \:12, in \_\_init\_\_(self, fname, preload, verbose, create\_annotations, apply\_offsets, find\_overlaps, overlap\_threshold, gap\_desc) > > File ~/github\_repos/mne-python/mne/io/eyelink/eyelink.py:229, in RawEyelink.\_\_init\_\_(self, fname, preload, verbose, create\_annotations, apply\_offsets, find\_overlaps, overlap\_threshold, gap\_desc) > 227 sfreq = \_get\_sfreq(self.\_event\_lines\["SAMPLES"\]\[0\]) > 228 col\_names, ch\_names = self.\_infer\_col\_names() > \--\> 229 self.\_create\_dataframes( > 230 col\_names, sfreq, find\_overlaps=find\_overlaps, threshold=overlap\_threshold > 231 ) > 232 info = self.\_create\_info(ch\_names, sfreq) > 233 eye\_ch\_data = self.dataframes\["samples"\]\[ch\_names\] > > File ~/github\_repos/mne-python/mne/io/eyelink/eyelink.py:508, in RawEyelink.\_create\_dataframes(self, col\_names, sfreq, find\_overlaps, threshold) > 505 first\_samp = self.\_event\_lines\["START"\]\[0\]\[0\] > 507 # dataframe for samples > \--\> 508 self.dataframes\["samples"\] = pd.DataFrame( > 509 self.\_sample\_lines, columns=col\_names\["sample"\] > 510 ) > 511 if "HREF" in self.\_rec\_info: > 512 pos\_names = ( > 513 EYELINK\_COLS\["pos"\]\["left"\]\[:-1\] + EYELINK\_COLS\["pos"\]\["right"\]\[:-1\] > 514 ) > > File /usr/local/Caskroom/miniconda/base/envs/mnedev/lib/python3.11/site-packages/pandas/core/frame.py:782, in DataFrame.\_\_init\_\_(self, data, index, columns, dtype, copy) > 780 if columns is not None: > 781 columns = ensure\_index(columns) > \--\> 782 arrays, columns, index = nested\_data\_to\_arrays( > 783 # error: Argument 3 to "nested\_data\_to\_arrays" has incompatible > 784 # type "Optional\[Collection\[Any\]\]"; expected "Optional\[Index\]" > 785 data, > 786 columns, > 787 index, # type: ignore\[arg-type\] > 788 dtype, > 789 ) > 790 mgr = arrays\_to\_mgr( > 791 arrays, > 792 columns, > (...) > 795 typ=manager, > 796 ) > 797 else: > > File /usr/local/Caskroom/miniconda/base/envs/mnedev/lib/python3.11/site-packages/pandas/core/internals/construction.py:498, in nested\_data\_to\_arrays(data, columns, index, dtype) > 495 if is\_named\_tuple(data\[0\]) and columns is None: > 496 columns = ensure\_index(data\[0\].\_fields) > \--\> 498 arrays, columns = to\_arrays(data, columns, dtype=dtype) > 499 columns = ensure\_index(columns) > 501 if index is None: > > File /usr/local/Caskroom/miniconda/base/envs/mnedev/lib/python3.11/site-packages/pandas/core/internals/construction.py:840, in to\_arrays(data, columns, dtype) > 837 data = \[tuple(x) for x in data\] > 838 arr = \_list\_to\_arrays(data) > \--\> 840 content, columns = \_finalize\_columns\_and\_data(arr, columns, dtype) > 841 return content, columns > > File /usr/local/Caskroom/miniconda/base/envs/mnedev/lib/python3.11/site-packages/pandas/core/internals/construction.py:937, in \_finalize\_columns\_and\_data(content, columns, dtype) > 934 columns = \_validate\_or\_indexify\_columns(contents, columns) > 935 except AssertionError as err: > 936 # GH#26429 do not raise user-facing AssertionError > \--\> 937 raise ValueError(err) from err > 939 if len(contents) and contents\[0\].dtype == np.object\_: > 940 contents = convert\_object\_array(contents, dtype=dtype) > > ValueError: 10 columns passed, passed data had 9 columns > \`\`\` > > \### Additional information > > See link to the discourse thread. I’ll keep you updated once we implement a fix! ---
**Author:** ![scott-huberty](https://yyz2.discourse-cdn.com/free1/user_avatar/mne.discourse.group/scott-huberty/32/2715_2.png) [@scott-huberty](https://mne.discourse.group/u/scott-huberty)\ **Post date:** [July 27, 2023, 12:38pm UTC](https://mne.discourse.group/t/problem-with-read-raw-eyelink/7243/11 "2023-07-27T12:38:15Z")
This issue was fixed via PR: > > > Fixes #11809 > Fixes #11758 > > > This fixes the bug the researcher reported, whe…re the STATUS column wasn't present in their ASCII file. > > Plus, I knew that \`read\_raw\_eyelink\` was slow, but it became really apparent once the researcher shared their problematic file with me, which was 200mb and over 4 million lines long.. which is way bigger than any ASCII file I've come across. > > So, I profiled the code to look for bottlenecks, and since we already require \`pandas\` for \`read\_raw\_eyelink\` I refactored the reader to use vector operations whenever possible. > > I think we are seeing a decent speed up. If we read one of our eyelink sample files: > > \`\`\` > from mne.datasets.eyelink import data\_path > from mne.io import read\_raw\_eyelink > fname = data\_path() / "sub-01\_task-plr\_eyetrack.asc" > %timeit read\_raw\_eyelink(fname) > \`\`\`\` > > On the main branch we get: > \`2.59 s ± 36.1 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)\` > > And on this branch we get > \`797 ms ± 13.4 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)\` > > So the refactored code is about 3x faster. Unfortunately that huge ASCII file still takes like about a minute to read into mne.. > > Finally, in my refactoring, i've tried to simplify the code and make it more organized. > > TODOS: > > \- \[x\] add tests for the bug fix this PR addressed > \- \[x\] get the testing coverage of this module to be above 95% > \- \[x\] see if we can get any more performance speed ups, to quicken the loading time for the very large ASCII file ---
**Author:** ![scott-huberty](https://yyz2.discourse-cdn.com/free1/user_avatar/mne.discourse.group/scott-huberty/32/2715_2.png) [@scott-huberty](https://mne.discourse.group/u/scott-huberty)\ **Post date:** [July 27, 2023, 12:45pm UTC](https://mne.discourse.group/t/problem-with-read-raw-eyelink/7243/12 "2023-07-27T12:45:02Z")
@DaniG, to use the updated code that will work with your file, you have two options: 1. wait until the next stable release of MNE. 2. Install the development version of MNE (assuming you haven’t already): If you aren’t familiar with this, MNE [gives a pretty thorough walk through](https://mne.tools/stable/install/contributing.html#contributing) ---
**Author:** ![DaniG](https://yyz2.discourse-cdn.com/free1/user_avatar/mne.discourse.group/danig/32/2809_2.png) [@DaniG](https://mne.discourse.group/u/DaniG)\ **Post date:** [July 28, 2023, 3:41pm UTC](https://mne.discourse.group/t/problem-with-read-raw-eyelink/7243/13 "2023-07-28T15:41:41Z")
Thanks a lot for fixing this so quickly 🙂