Drop_log "NO_DATA" when creating epochs

  • MNE version: e.g. 1.7.0
  • operating system: Linux Mint 21.1 Cinnamon

Hello,
Iā€™m currently doing a project where I try to analyse eeg data after getting it with LSL protocol (I get the data and the annotations from an xdf file with the library pyxdf). Itā€™s working but the problem comes after that.

After filtering the data to only keep the frequencies Iā€™m interested in (8-14 Hz), and after selecting the electrode Iā€™m interested in (C3), I try to divide the signal in epochs (3 second before the annotation and 6.5 second after).

When I do that 25 of my 40 epochs gets excluded for the following reason : NO_DATA. I donā€™t understand why itā€™s telling me I have no data because when I look at the eeg signal, there is no cut or flat line in the signal.
Could it be due to artefcacts ?

I will put in attachment a screen of the drop log I get

I canā€™t put my xdf file in this message, is there an other way I can send the file so you can test the code ?

Here is my code :

Thank you very much for your help.

import pyxdf
import mne
import numpy as np
import matplotlib.pyplot as plt
import numpy as np
import matplotlib.pyplot as plt
import random
from functions import *
from functions import filter_signal_plot_spectrum
from plot_all_subjects import plot_all_subjects

file_path = 'PATH_TO_THE_FILE/sub-P001_ses-S001_task-Default_run-001_eeg.xdf'

streams, fileheader = pyxdf.load_xdf(file_path)

markers = []

# Go through the streams an indentify EEG streams and markers stream
#We first initialize the var that will tell us if we detect these streams
eeg_stream = None
marker_stream = None

#If they exist we store them
for stream in streams:
    if stream['info']['type'][0] == 'EEG':
        eeg_stream = stream
    elif stream['info']['type'][0] == 'Markers':
        marker_stream = stream

if marker_stream:
    for i, marker in enumerate(marker_stream['time_series'][:]):
        markers.append(marker[0])

if (eeg_stream != None) :
    data_lsl = eeg_stream['time_series'].T
    data_lsl = data_lsl/1000000 # We convert to Volts (it was microVolts)
    sfreq = eeg_stream["info"]["effective_srate"]
    info = mne.create_info( int(eeg_stream["info"]["channel_count"][0]) , sfreq, ch_types ='eeg')
    raw = mne.io.RawArray(data_lsl, info, first_samp= eeg_stream['time_stamps'][0])
    
if (marker_stream != None) :
    evts = np.zeros((len(marker_stream['time_stamps']),3))
    for n in range(len(marker_stream['time_stamps'])) :
        #We assign to the marker the closest existing timestamp of the eeg
        target = marker_stream['time_stamps'][n]
        eeg_target = min(eeg_stream['time_stamps'], key= lambda x: abs(x-target))
        evts[n][0] = np.where(eeg_stream['time_stamps'] == eeg_target)[0][0]
        if(markers[n] == 'left hand') :
            evts[n][2] = 2
        if(markers[n] == 'right hand'):
            evts[n][2] = 3
    evts = evts.astype(dtype= int, copy=False)
    
    annotations = mne.annotations_from_events(events= evts, sfreq=sfreq)
    raw.set_annotations(annotations)
  

signal_nb_event = len(markers) - 1 # We don't the event that close the outlet

event_dict = {
    "Left Hand" : 2,
    "Right Hand" : 3,
    }
# Definition of the bounds of the epochs (0 is the time of the event)
t_min, t_max = -3, 6.5 # 3 seconds befor task, 6.5 seconds after the task

# The number of event we can find in the signal 
nb_evt = 2 #Left Hand and Right Hand   

data = raw.copy().filter(l_freq=50.5, h_freq=49.5,picks = 'all', method = 'iir')

#Alpha (8-14Hz): 
dataalpha = data.copy().filter(l_freq=8, h_freq=14, h_trans_bandwidth= 2,picks = 'all')
dataalpha.plot()
#We select the C3 channel (channel number 27)
data_Elec = dataalpha.copy().pick_channels(ch_names = [('27')] )

epchs_freq_Elec = mne.Epochs(data_Elec, evts, event_id= event_dict, picks=("27"), tmin=t_min, tmax=t_max, preload=True)
epchs_freq_Elec.plot_drop_log()
epchs_freq_Elec.plot()

drop_log

The documentation says:

    - ``'NO_DATA'`` or ``'TOO_SHORT'``
        If epoch didn't contain enough data names of channels that
        exceeded the amplitude threshold

But I donā€™t know what this is supposed to tell us. Maybe your epochs are just too long?

1 Like

Thank you for your answer, I saw that too but I donā€™t really understand what it means.

I did some tests and I came up with this :
Here is a new version of my code :

#We select the C3 channel (channel number 27)
data_Elec = dataalpha.copy().pick_channels(ch_names = [('27')] )

# get the data
data = data_Elec.get_data()

info = data_Elec.info

data_Elec2 = mne.io.RawArray(data, info)

epchs_freq_Elec = mne.Epochs(data_Elec2, evts, event_id= event_dict, picks=("27"), tmin=t_min, tmax=t_max, preload=True, reject= {'eeg':40}, flat={'eeg':0})
epchs_freq_Elec.plot_drop_log()
epchs_freq_Elec.plot()

I just get the data from the raw object and I create a new one (I kept the info of the first raw)

When i do that no data is dropped when I create my epochs.

I donā€™t know if that helps you understand what is happening ?

Thank you

Nicolas

Hi,

The documentation is indeed a bit vague and the source code too convoluted for me to follow.

If I were you I would double-check three points:

  1. If working with notebooks, that no values are reused when they shouldnā€™t (e.g. not restarting a cell),
  2. That the original Raw object you have doesnā€™t have NO_DATA in the drop_log. Else, by recreating the Raw only using the data and info wouldnā€™t you be discarding the drop_log and hence ā€œsolving your problemā€? :thinking:
  3. How the reject dict affects the data. That is because you seem to only use it the 2nd time.

Cheers,

2 Likes

Hi,
Thank you for your answer !

I tried to test your suggestions.

For the first one, I work with Spyder IDE, so no notebooks for me.

For the second point. I didnā€™t find a way to see the drop_log of a Raw object. It seems to only exist after creating an Epoch object.

For the third point, I tested the impact of the reject and flat attribute and I saw no impact on NO_DATA in the drop_log (reject even had no impact on the drop_log at all, whatever the threshold I assigned)

I think your second point could really give me hints on the reason of this ā€œbugā€ but I donā€™t know how to apply it.

Cheers,

Good,

Well, recently Iā€™ve been working with some data with muscle artifacts and once these are annotated in the Raw object, then they automatically appear as ā€œBAD_muscleā€ in the drop_log when I create the Epochs.

Maybe you have something like this going on, some annotation(s) that marks your Epochs with NO_DATA?

Unfortunately, I donā€™t create any annotation named NO_DATA.
I checked by ploting the raw object just before epoching, and there is no Annotations named like that.

Sorry for the misunderstanding.
I do not manually annotate the Raw data. I meant that I use the annotate_muscle_zscore function to annotate the Raw data, which then translates as BAD_muscle in the Epochsā€™s drop_log.
Therefore, I was thinking that some annotations in your data could be the cause of your problem.

Did you take a look at your annotations?

Sorry for misunderstanding,

I checked, and I didnā€™t annotate the data even unintentionally. If I did I donā€™t know how and where. I visualized the raw (with the plot function) at different steps of the code to see all the annotations and I didnā€™t see anything.

Thank you again for your help,

Hello,
Yes I checked them and I never saw any annotation with the label NO_DATA. They are all named ā€˜2ā€™ or ā€˜3ā€™ depending on the action.

@NicoD could you share the data and your code please? You can share it privately with me if you donā€™t want to share it publicly; my email address is richard.hoechenberger@gmail.com

Richard

Hi, I just sent you the code and data by mail.

Thank you

Nicolas

1 Like

Might be too late at that point, but I have No_Data only after choosing epochs lengths that are too long. This usually happens with trials at the beginning of the recording session (in my case I stopped the recording during breaks). If you check which trials get the NO_DATA tag, are they at the beginning of the recording?

Hi, thank you for you answer.

The problem with my code/data is that the label NO_DATA never appears except when I create epochs and I plot the drop_log so I canā€™t check if they are at the beginning or at the end.
However, I tried to change the length of the epoch (baseline and the rest) and it didnā€™t change anything. (in my case the recording is continuous)

Thank you for your help

Even if it doesnā€™t explain the origin of the bug, a solution that seems to work is to extract the data from the raw and create a new raw as following :

array_data = data.get_data()
info = data.info
data = mne.io.RawArray(array_data, info)

Iā€™m still open to any ideas to understand where this bug come from.

Thanks,

@NicoD I believe I received your data, thanks. I didnā€™t have time to look into this, unfortunately. I may have time tomorrow and will get back to you.

Best wishes,
Richard

1 Like