eyetracking and qt browser: traces of x/y position disappear

💬 Support & Discussions
,
1
  • MNE version: 1.4.2
  • operating system: Ubuntu 22.04 (but same issue on macOS)

I am following the tutorial on reading eyetracking data:

The first coding cell in the tutorial produces a plot like this one:

image
image755×693 30.2 KB

I can reproduce this plot. HOWEVER when I start scrolling left and right in the viz browser, the traces for xpos_right and ypos_right disappear around annotation events. Even when scrolling back to exact same position as before, the traces are gone.

image
image876×788 19.3 KB

I can replicate this on different computers (linux, macOS) and this also happens with:

  • clipping=None and use_opengl=False (or True)
  • calling raw.set_annotations(None) first

What’s the issue here? There are NaN in the data, but I don’t think they should lead to the observed effects.

cc @scott-huberty @mscheltienne

Here is my mne sys info:

Platform             Linux-5.19.0-46-generic-x86_64-with-glibc2.35
Python               3.11.4 | packaged by conda-forge | (main, Jun 10 2023, 18:08:17) [GCC 12.2.0]
Executable           /home/stefanappelhoff/miniconda3/envs/mne/bin/python
CPU                  x86_64 (8 cores)
Memory               31.2 GB

Core
├☑ mne               1.4.2
├☑ numpy             1.24.4 (OpenBLAS 0.3.23 with 8 threads)
├☑ scipy             1.11.1
├☑ matplotlib        3.7.2 (backend=module://matplotlib_inline.backend_inline)
├☑ pooch             1.7.0
└☑ jinja2            3.1.2

Numerical (optional)
├☑ sklearn           1.3.0
├☑ numba             0.57.1
├☑ nibabel           5.1.0
├☑ nilearn           0.10.1
├☑ dipy              1.7.0
├☑ openmeeg          2.5.6
├☑ pandas            2.0.3
└☐ unavailable       cupy

Visualization (optional)
├☑ pyvista           0.40.1 (OpenGL 4.6 (Core Profile) Mesa 22.2.5-0ubuntu0.1~22.04.3 via Mesa Intel(R) UHD Graphics 620 (KBL GT2))
2

That’s an interesting one, maybe nan interfering with the qt browser. It’s not the case with the matplotlib backend. Looking into it now.

3

Hi @sappelhoff ,

Thanks for looking at the tutorial and for raising this! It makes me feel better having more devs look at the eye-tracking stuff we’ve built so far.

As @mscheltienne mentioned, For me, the issue where the NaN’s cause the traces to disappear only occurs when using mne-qt-browser. If I use the matplotlib browser, I can’ replicate the issue:

import mne
from mne.datasets import misc

mne.viz.set_browser_backend("matplotlib")
fpath = misc.data_path() / "eyetracking" / "eyelink"
raw = read_raw_eyelink(fpath / "px_textpage_ws.asc", create_annotations=["blinks"])

custom_scalings = dict(eyegaze=1e3)
raw.plot(scaling=custom_scalings)

Screen Shot 2023-07-18 at 9.52.58 AM
Screen Shot 2023-07-18 at 9.52.58 AM1423×870 41.1 KB

I wonder if this has something to do with a filter being applied to the plotted data (and the Nans causing issues with the filter)?

More generally, I was speaking with @drammock and @britta-wstnr on Friday about the issue of NaN’s in eye-tracking data. Correct me if I’m wrong but it seems to me that up until now, MNE has been able to expect that incoming data is dense (no missing values). Having NaN’s in the data causes issues in a number of MNE functions (filtering, resamping, etc).

I was tasked with opening a ticket about this (which I still need to do!), so that the devs can weigh in on the best path forward. I was also hoping to share an example file with this issue during the next MNE office hours, to demonstrate the issue.

2 Likes
4

In the end barely had time to look for now, but if you do raw.plot(scalings=custom_scalings, precompute=False) it works. That should already narrow down the issue.

5

That PR should fix it: Fix removal of DC on precomputed data when nan's are present in the array by mscheltienne · Pull Request #176 · mne-tools/mne-qt-browser · GitHub

3 Likes
6

Thanks!

I wonder if we should eventually add some code to prevent the DC-offset adjustment from being applied to eye-track channels? I don’t think it applies here. If you all agree I can open a ticket and take a look into it.

7

I was thinking about it yesterday, should we actually remove DC-offset from non voltage-based channels (extending beyong eye-tracking)? I convinced myself that it was safe to do in all cases since it would not alter the waveform shape.
But I’m questionning this again as eye-tracking position, if e.g. in pixels, should retain the orignal values; else DC removal is synonym of silently changing the origin location. And I wonder if the same logic applies to other channels types which should not have their mean removed prior to plotting.

On the otherhand, I would not be surprised if the y-range defined for plotting is centered around 0 and that plotting non centered data would end up creating issues, e.g. clipping or data outside the y-range.

1 Like
8

I don’t think eye tracking data like gaze should be “DC offset corrected”.
BTW, should we move this conversation over to a GitHub issue so we can more easily find it in the future in case we want to look back at some point?

3 Likes
9

Sure, I will open a ticket!