Dropped epochs when preload=True

Support & Discussions
, ,
1

I have a question regarding the mne.Epochs() method. I have noticed that epochs are dropped only when preload=True.

epochs = mne.Epochs(raw_highpass, events, event_id, tmin, tmax, metadata=regressorsPd, baseline=(-.1, 0), preload=False, reject=None, reject_by_annotation=False)

Adding metadata with 6 columns
Replacing existing metadata with 6 columns
67 matching events found
Applying baseline correction (mode: mean)
0 projection items activated

epochsPreload = mne.Epochs(raw_highpass, events, event_id, tmin, tmax, metadata=regressorsPd, baseline=(-.1, 0), preload=True, reject=None, reject_by_annotation=False)
Adding metadata with 6 columns
Replacing existing metadata with 6 columns
67 matching events found
Applying baseline correction (mode: mean)
0 projection items activated
Loading data for 67 events and 451 original time points ...
45 bad epochs dropped

The 45 bad epochs are dropped because ‘too short’ as indicated by the log:

epochsPreload.drop_log
Out[86]: 
((),
 (),
 (),
 (),
 (),
 (),
 (),
 (),
 (),
 (),
 (),
 (),
 (),
 (),
 (),
 (),
 (),
 (),
 (),
 (),
 (),
 (),
 ('TOO_SHORT',),
 ('TOO_SHORT',),
 ('TOO_SHORT',),
 ('TOO_SHORT',),
 ('TOO_SHORT',),
 ('TOO_SHORT',),
 ('TOO_SHORT',),
 ('TOO_SHORT',),
 ('TOO_SHORT',),
 ('TOO_SHORT',),
 ('TOO_SHORT',),
 ('TOO_SHORT',),
 ('TOO_SHORT',),
 ('TOO_SHORT',),
 ('TOO_SHORT',),
 ('TOO_SHORT',),
 ('TOO_SHORT',),
 ('TOO_SHORT',),
 ('TOO_SHORT',),
 ('TOO_SHORT',),
 ('TOO_SHORT',),
 ('TOO_SHORT',),
 ('TOO_SHORT',),
 ('TOO_SHORT',),
 ('TOO_SHORT',),
 ('TOO_SHORT',),
 ('TOO_SHORT',),
 ('TOO_SHORT',),
 ('TOO_SHORT',),
 ('TOO_SHORT',),
 ('TOO_SHORT',),
 ('TOO_SHORT',),
 ('TOO_SHORT',),
 ('TOO_SHORT',),
 ('TOO_SHORT',),
 ('TOO_SHORT',),
 ('TOO_SHORT',),
 ('TOO_SHORT',),
 ('TOO_SHORT',),
 ('TOO_SHORT',),
 ('TOO_SHORT',),
 ('TOO_SHORT',),
 ('TOO_SHORT',),
 ('TOO_SHORT',),
 ('TOO_SHORT',))

I don’t understand two things.

  1. What is the reason of this? The helper says that epochs are ‘too short’ if they do not “contain enough data names of channels that exceeded the amplitude threshold”. I don’t understand what this means, especially since I have set no amplitude threshold for epoching.
  2. Why are ‘too short’ epochs being dropped only when they are preloaded?
2

This clearly makes no sense, yes. Did you by any chance crop the data without dropping the cropped-out events?

Because only when you preload, epochs will be dropped on instantiation . To do it later on, you need to call Epochs.drop_bad().

3

Not sure what you mean here. I did retrieve only one event type from the annotations, if that is what you mean, but I don’t think I did any further cropping…

4

I wonder if something went wrong there (going from Annotations to events), in the sense that the event onsets are shifted to later times – time points beyond the last time point in your raw data. If you look at the events array, the first column contains the onset samples. Could you check for the last few rows which samples these are, and if they fall outside the sample range of your raw data?

5

I don’t think so, if I understand you are suggesting:

raw_events[-10:]
Out[108]: 
array([[1600812,       0,      30],
       [1605047,       0,      30],
       [1611996,       0,      30],
       [1617988,       0,      30],
       [1631646,       0,      30],
       [1680936,       0,      30],
       [1694680,       0,      30],
       [1699852,       0,      30],
       [1720664,       0,      30],
       [1746342,       0,      30]])

raw_highpass
Out[104]: <RawEEGLAB | 1_preprocessed.fdt, 256 x 1803542 (3607.1 s), ~3.44 GB, data loaded>

The last event began at 1746342 and the sample range of the data is 0,1803542. Is this what you mean?

I also did some downsampling before epoching, but I also made sure that both data and events were downsampled together (as we discussed in a separate discussion)…

6

UPDATE: I have tried to do epoching of the raw data without downsampling, and no epoch was dropped. So, I think the issue is linked to downsampling. I’ll look into this more later tonight.

7

Could you share the Annotations of the downsampled data?

8

Hi @richard,

I think I have sent you the annotations file via direct message.

9

Hello, yes, thank you! I looked at the Annotations and they seem fine to me… Do you think you could share the original file(s) you used to run into this issue, so I could try to reproduce myself?

10

I’ll be sending you the file via direct message – thanks!

11

I finally figured out what was going on. It was a silly mistake of mine – in the mne.Epochs call, I called the events array created before downsampling, rather than the down_events array created after it. So, for future reference – if you ever run across a similar issue, make sure you have select the correct event array when epoching.