Epochs events correctly, plots them wrong

Hi everyone,

I have an issue with plotting my data. In the beginning of my code my continious data get segmented (flux_eegEpochs). This works fine, I get 240 events, like it should be (see dataepoch.info). If I later one want to plot the data with either reconst_raw.plot() or dataica.plot(), i get the wrong indicies (see dataicaplots).
So the data within the flux_eegEpochs get segmented correctly and dataepoch.info contains all 240 events. Filt_data which is definded as dataepoch.copy() does not include the info over the segments anymore. I have no idea why the segments dont get display correctly in the plot. If you look at the dataicaplot, the indices start with 1 than 14, 27 and so on. Really strange indices that differ from each participants dataset and are supposed to be from 0 to 239


#define functions
def slice(sourcedict, string):
    newdict = {}
    for key in sourcedict.keys():
        if key.startswith(string):
            newdict[key] = sourcedict[key]
    return newdict



def flux_eegEpochs (raw_load):
    
    
    #Convert an Annotations object to an Events array
    events, event_dict = mne.events_from_annotations(raw_load)
    print(event_dict)
    print(events[:5])
    
    # Visualize all events
    fig = mne.viz.plot_events(events, sfreq=raw_load.info['sfreq'],
                          first_samp=raw_load.first_samp, event_id=event_dict)
    fig.subplots_adjust(right=0.7)  # make room for legend


    #Create dicionary for trial onset events
    on_dict = slice(event_dict, 'Stimulus/S ')

    # Epoch data
    epochs = mne.Epochs(raw_load, 
                    events, 
                    event_id=on_dict, 
                    tmin=-1, 
                    tmax=8,
                    detrend=1,
                    baseline=None,
                    proj=True,
                    preload=True)

    len(epochs.events)
    
    return epochs


def identify_ica(raw):
    """ Use ICA to reject artifacts
    """
    # Perform ICA
    ica = mne.preprocessing.ICA(
            n_components=20, # Number of components to return
            max_pca_components=None, # Don't reduce dimensionality too much
            random_state=0,
            max_iter=800,
            verbose='INFO')
    ica.fit(raw, reject_by_annotation=True)
    
    # Plot ICA results
    
    ica.plot_components(inst=raw) # Scalp topographies - Click for more info
    ica.plot_sources(raw) # Time-courses - click on the ones to exclude
    
    return ica

#Subject Info: (10 & 15 excl. in Rati & Reex, 11 excl. in Reex)
#Subjects included: 
#3,4,9,(10),(11),12,13,14,(15),16,17,18,19,20,21,22,23,24,27,28,29,31,32,34,35,
#36,37,38,40,41,42,43,44,45,46,47,48,51,52,53,54,55,56,57,58,59,61

subject = 'sub-29' 
session = 'ses-Expo'
bids_root ='C:\\Users\\Admin\\Desktop\\Projekt_Franziska\\Oszillationen\\TF-Analyse\\BIDS'
mne_data_folder=bids_root + '/' + subject + '/' +  session + '/' +  'eeg/' 
raw_load=mne.io.read_raw_fif(mne_data_folder + subject + '_' +  session + '_run01_' + 'data_interpolated.fif', verbose=False)

# Check annotations to see markers
# S 1n for CS- Onset
# S11n for CS- Offset 
# S 2n for CS+ Onset 
# S22n to CS+ Offset. 
# The last number (n) of the marker refers to the ratings (low 1 to high 9)
print(raw_load.annotations)
print(len(raw_load.annotations))
print(set(raw_load.annotations.duration))
print(set(raw_load.annotations.description))
print(raw_load.annotations.onset[0])

dataepoch=flux_eegEpochs (raw_load)
print(dataepoch.info)

filt_data = dataepoch.copy()
filt_data.load_data().filter(l_freq=1., h_freq=None)

raw_ica=identify_ica(filt_data);


# RUN ALL THE ABOVE BEFORE ENTERING ICA VALUES TO EXCLUDE BELOW
raw_ica.exclude = [0,6,10]  # indices chosen based on various plots above
reconst_raw = filt_data.copy()
dataica=raw_ica.apply(reconst_raw)
dataica.save(mne_data_folder + subject + '_' + session + '_run01_' + 'data_ica.fif', overwrite=True)

## Check whether ICA worked as expected
reconst_raw.plot()
dataica.plot()



flux_eegEpochs

Thanks a lot for your help!!
Best,
Franziska

Hello @FranziMNE, and welcome to the forum!

I’m not sure I have an immediate solution for your issue, but a few comments and questions:

The naming of this variable is a little confusing, as it doesn’t contain ICA results from raw data, but from epochs. Could make sense to rename it!

Have you considered using MNE-BIDS to read & write the BIDS data? It could make your life easier as you wouldn’t have to take care of correctly constructing the paths manually!

The screenshot doesn’t show the contents of dataepoch.info :slight_smile:

ICA.apply() modifies the passed instance in-place, meaning that raw_ica.apply(reconst_raw) modifies reconst_raw and you don’t really need to assign the return value to a new variable.

Can you please share what events and event_dict look like?

Best,

Richard

Hi Richard,

thank you for your suggestions!! :slightly_smiling_face:

When I run the follwing lines:

subject = 'sub-29' 
session = 'ses-Expo'
bids_root ='C:\\Users\\Admin\\Desktop\\Projekt_Franziska\\Oszillationen\\TF-Analyse\\BIDS'
mne_data_folder=bids_root + '/' + subject + '/' +  session + '/' +  'eeg/' 
raw_load=mne.io.read_raw_fif(mne_data_folder + subject + '_' +  session + '_run01_' + 'data_interpolated.fif', verbose=False)

# Check annotations to see markers
# S 1n for CS- Onset
# S11n for CS- Offset 
# S 2n for CS+ Onset 
# S22n to CS+ Offset. 
# The last number (n) of the marker refers to the ratings (low 1 to high 9)
print(raw_load.annotations)
print(len(raw_load.annotations))
print(set(raw_load.annotations.duration))
print(set(raw_load.annotations.description))
print(raw_load.annotations.onset[0])

dataepoch=flux_eegEpochs (raw_load)
print(dataepoch.info)

I get the following output with infos over the events

{'Stimulus/S 28', 'Stimulus/S 19', 'Stimulus/S150', 'New Segment/', 'Stimulus/S 11', 'Stimulus/S 12', 'Stimulus/S114', 'Stimulus/S 26', 'Stimulus/S117', 'Stimulus/S 13', 'Stimulus/S 29', 'Stimulus/S118', 'Stimulus/S113', 'Stimulus/S119', 'Stimulus/S226', 'Stimulus/S 22', 'Stimulus/S112', 'Stimulus/S111', 'Stimulus/S 23', 'Stimulus/S224', 'Stimulus/S227', 'Stimulus/S 21', 'Stimulus/S 18', 'Stimulus/S 16', 'Stimulus/S222', 'Stimulus/S 14', 'Stimulus/S128', 'Stimulus/S228', 'Stimulus/S 17', 'Stimulus/S 27', 'Stimulus/S223', 'Stimulus/S116', 'Stimulus/S221', 'Stimulus/S 24', 'Stimulus/S229'}
0.0
Non-RawBrainVision raw using branvision markers
Used Annotations descriptions: ['New Segment/', 'Stimulus/S 11', 'Stimulus/S 12', 'Stimulus/S 13', 'Stimulus/S 14', 'Stimulus/S 16', 'Stimulus/S 17', 'Stimulus/S 18', 'Stimulus/S 19', 'Stimulus/S 21', 'Stimulus/S 22', 'Stimulus/S 23', 'Stimulus/S 24', 'Stimulus/S 26', 'Stimulus/S 27', 'Stimulus/S 28', 'Stimulus/S 29', 'Stimulus/S111', 'Stimulus/S112', 'Stimulus/S113', 'Stimulus/S114', 'Stimulus/S116', 'Stimulus/S117', 'Stimulus/S118', 'Stimulus/S119', 'Stimulus/S128', 'Stimulus/S150', 'Stimulus/S221', 'Stimulus/S222', 'Stimulus/S223', 'Stimulus/S224', 'Stimulus/S226', 'Stimulus/S227', 'Stimulus/S228', 'Stimulus/S229']
{'New Segment/': 99999, 'Stimulus/S 11': 11, 'Stimulus/S 12': 12, 'Stimulus/S 13': 13, 'Stimulus/S 14': 14, 'Stimulus/S 16': 16, 'Stimulus/S 17': 17, 'Stimulus/S 18': 18, 'Stimulus/S 19': 19, 'Stimulus/S 21': 21, 'Stimulus/S 22': 22, 'Stimulus/S 23': 23, 'Stimulus/S 24': 24, 'Stimulus/S 26': 26, 'Stimulus/S 27': 27, 'Stimulus/S 28': 28, 'Stimulus/S 29': 29, 'Stimulus/S111': 111, 'Stimulus/S112': 112, 'Stimulus/S113': 113, 'Stimulus/S114': 114, 'Stimulus/S116': 116, 'Stimulus/S117': 117, 'Stimulus/S118': 118, 'Stimulus/S119': 119, 'Stimulus/S128': 128, 'Stimulus/S150': 150, 'Stimulus/S221': 221, 'Stimulus/S222': 222, 'Stimulus/S223': 223, 'Stimulus/S224': 224, 'Stimulus/S226': 226, 'Stimulus/S227': 227, 'Stimulus/S228': 228, 'Stimulus/S229': 229}
[[    0     0 99999]
 [57451     0    29]
 [61483     0   229]
 [61485     0   150]
 [61486     0   128]]
<ipython-input-56-3339bfe48d4b>:37: RuntimeWarning: More events than default colors available. You should pass a list of unique colors.
  fig = mne.viz.plot_events(events, sfreq=raw_load.info['sfreq'],
Not setting metadata
Not setting metadata
240 matching events found
No baseline correction applied
Created an SSP operator (subspace dimension = 1)
1 projection items activated
Loading data for 240 events and 9001 original time points ...
0 bad epochs dropped
<Info | 11 non-empty values
 bads: []
 ch_names: Fp1, Fp2, F7, F3, Fz, F4, F8, FC5, FC1, FC2, FC6, T7, C3, Cz, ...
 chs: 28 EEG, 3 EOG
 custom_ref_applied: False
 dig: 31 items (3 Cardinal, 28 EEG)
 file_id: 4 items (dict)
 highpass: 0.0 Hz
 lowpass: 500.0 Hz
 meas_date: 2019-06-26 16:38:59 UTC
 meas_id: 4 items (dict)
 nchan: 31
 projs: Average EEG reference: on
 sfreq: 1000.0 Hz
  • Regarding dataepoch.info I was wrong. Sorry! I get no event info here.
    dateepoch and filt_data give out the same output as follow:
Out[57]: 
<Info | 11 non-empty values
 bads: []
 ch_names: Fp1, Fp2, F7, F3, Fz, F4, F8, FC5, FC1, FC2, FC6, T7, C3, Cz, ...
 chs: 28 EEG, 3 EOG
 custom_ref_applied: False
 dig: 31 items (3 Cardinal, 28 EEG)
 file_id: 4 items (dict)
 highpass: 0.0 Hz
 lowpass: 500.0 Hz
 meas_date: 2019-06-26 16:38:59 UTC
 meas_id: 4 items (dict)
 nchan: 31
 projs: Average EEG reference: on
 sfreq: 1000.0 Hz

Thank you :blush:

Thanks @FranziMNE!

events is a NumPy array and event_dict is a dictionary; neither has an .info attribute. But maybe we don’t need that anyway … could you please share the output of:

print(dataepoch.event_id)
print(dataepoch.events.shape)

?

print(dataepoch.event_id)
{‘Stimulus/S 11’: 11, ‘Stimulus/S 12’: 12, ‘Stimulus/S 13’: 13, ‘Stimulus/S 14’: 14, ‘Stimulus/S 16’: 16, ‘Stimulus/S 17’: 17, ‘Stimulus/S 18’: 18, ‘Stimulus/S 19’: 19, ‘Stimulus/S 21’: 21, ‘Stimulus/S 22’: 22, ‘Stimulus/S 23’: 23, ‘Stimulus/S 24’: 24, ‘Stimulus/S 26’: 26, ‘Stimulus/S 27’: 27, ‘Stimulus/S 28’: 28, ‘Stimulus/S 29’: 29}

print(dataepoch.events.shape)
(240, 3)

Ok perfect! And if you run dataepoch.plot(), all is looking good, right?

unfortunatly this look already like that, having those strange indicies

Ok, thanks! I believe the issue is that you slice event_dict, but do not subset events:

You could do something like (untested!):

# subset events: only keep those associated with on_dict
events_to_keep_idx = []
for idx, event in enumerate(events):
    if event[-1] in on_dict.values():
        events_to_keep_idx.append(idx)

events = events[events_to_keep_idx]

@agramfort @larsoner @cbrnr
I believe this is something we should change, no? If a user passes an event_id dict to Epochs that describes only a subset of the event IDs present in the events array, we shouldn’t number the created epochs based on the indeces of all events in the array, but based on only those events that were actually used to create the Epochs. This would avoid the confusion @FranziMNE experienced, and I believe it would be what users would be commonly expecting.

I think we should crash with a good message if event_id is only a subset of the events present. It should avoid silent bugs.

my 2c

Alex

We do the reverse already (events_id referes to event IDs not found in events); however I’m afraid doing what you propose is bound to lead to lots of issues as acquisition systems may add markers you don’t even know about; or imagine a BIDS dataset with lots of events and you only care about a subset.


Actually I believe we already have a solution in place:
@FranziMNE, when calling events_from_annotations(), you can actually pass an event_id parameter to only consider the specified subset of events. In your concrete example, you could do something like this:

    all_events, all_event_id = mne.events_from_annotations(raw_load)

    # events of interest
    event_id = slice(all_event_id, 'Stimulus/S ')
    events, _ = mne.events_from_annotations(raw_load, event_id=event_id)

Does that work?

we shouldn’t number the created epochs based on the indeces of all events in the array, but based on only those events that were actually used to create the Epochs

I would expect the indices to correspond to the original events passed in, regardless of whatever event_id is. So it sounds like we have opposite expectations here. I think we should just update the documentation to be clearer here.

1 Like

Sounds good! I could also add a log message. Will file a PR for this.

Hi Richard,

thanks a lot for your help! I really appreciate your time!
It finally works :tada:

1 Like