Why extra epochs are dropped even if they are not marked as 'bad'?

💬 Support & Discussions
,
1

Hi everyone,

I wanted to create epochs for polysomnography data. First, I set annotations to mark each epoch(30 seconds) sleep stage and mark some bad epochs. After that, I generated the epochs. However, I found that mne set some epochs bad even if I didn’t mark it as a bad one. I expected that epochs.drop_log should be a tuple full of empty tuple because I have already set ‘reject_by_annotation=True’. The truth was that there were many (‘bad’,) tuples. Did mne automatically drop epochs?

Here is my code:

annot = mne.Annotations(onset=[0,30,60,120,150,180,...],
                                          duration=[30,30,30,30,30,30,...],
                                          description=['W','W','bad','N1','N1',...])
raw = mne.io.read_raw_edf(edf_file, preload=True)
stage_id = {'W':0, 'N1':1, 'N2':2, 'N3':3, 'R':4}
stages = list(set(annot['description']))
stages.remove('bad')
annot_2_event = {stage:stage_id[stage] for stage in stages}
events, _ = mne.events_from_annotations(
        raw, event_id=annot_2_event, chunk_duration=30
)
epochs = mne.Epochs(
    raw=raw,
    events=events,
    event_id=annot_2_event,
    reject_by_annotation=True,
    picks='eeg',
    tmin=0,
    tmax=30,
    baseline=(None,None),
    preload=True
)

Indeed, len(epochs) is equal to the len([x for x in annot[‘description’] if x!=‘bad’]).
Any advice would be appreciated!

2

Hello, any raw segments annotated as bad will lead to “dropped” epochs if reject_by_annotation=True (the default). Epochs spanning these annotations will never be created, but they’ll show up in the drop log for diagnostic purposes.

Best wishes,
Richard

3

Hi Richard,

Thank you for your reply! What makes me confused is that there are many (‘bad’,) tuples in epochs.drop_log. Are these epochs retained even if their tuples are (‘bad’,) in drop_log? What is diagnostic purposes for? In my original thought, len(epochs.drop_log) should be equal to len(annot[‘description’]), and number of (‘bad’,) tuples in epochs.drop_log equal to number of ‘bad’ in annot[‘description’]. Actually, len(epochs.drop_log) is equal to len([x for x in annot[‘description’] if x != ‘bad’]), which means the information whether a epoch should be dropped is lost. Is it possible to know which epochs are dropped when creating epochs by raw through epochs.drop_log?

Best regards,
Jack

4

The similar numbers might just be a coincidence
 I suggest using epochs.plot_drop_log() to get a better understanding of how many epochs were dropped and for what reason. Epochs dropped due to a “bad” annotation should show up there (and in the drop log itself).

FWIW since MNE-Python 1.7 you don’t need to convert annotations to events anymore to perform epoching; this might save
you a little bit of time and typing!

Best wishes,
Richard

6

image
A part of epochs.drop_log:

((), (), (), (), (), (), (), (), (), (), (), (), (),
 (), (), (), (), (), (), (), (), (), (), (), (), (), (),
 (), (), (), ('bad',), (), ('bad',), (), (), (), (), ()
, (), (), (), (), (), (), (), (), (), (), (), (), (), (), (), ()
, (), (), (), (), (), (), ('bad',), (), ('bad',), ()
, (), (), (), (), (), (), (), (), (), (), (), ('bad',),...)

Actually, there is number of 1345(total time // 30 seconds) epochs in this sample. I thought there may not be a coincidence cause all my samples have the same situation that number of epochs is not equal to len(epochs.drop_log).

BTW, can I create epochs from annotations diectly?

Best regards,
Jack

7

Are you certain of this? Did you check the shape of the events array?

By not passing events and event_id to Epochs().

Best wishes,
Richard

8 
print(epochs.events.shape)

The result is:

(1023, 3)

Not surprisingly, 1023 = 1145 - 122 and 1023 is not equal to 1345. Interestingly, I also created epochs by not setting ‘bad’ in annot[‘description’] and events.shape is (1345, 3).

Best regards,
Jack

9

Thanks Jack. I was asking about the shape of the events you get back from events_from_annotations. I assume this is already where stuff might get dropped. This function ignores “bad” annotations by default as per its regexp parameter.

10 
print(events.shape)
(1023,3)

BTW, number of ‘bad’ in annot[‘description’] is equal to 322, which is also equal to 1345 - 1023.

11

Yes so this is your problem, then :slight_smile:

12

Thanks for your answers! I should check the document again.

1 Like
13

It’s honestly not entirely obvious at a first glance; I missed it myself at first. If you have any suggestions on how to improve the documentation, please do let us know.

14

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.