Temporal Generalization - Different results with and without using cross validation

✉️ Mailing List Archive (read-only)
1

External Email - Use Caution

Hi MNE experts,

I am using the temporal generalization
<https://mne.tools/dev/auto_tutorials/machine-learning/plot_sensors_decoding.html#temporal-generalization>
approach.
I have plotted scores' output from cross_val_multiscore
<https://mne.tools/dev/generated/mne.decoding.cross_val_multiscore.html#mne.decoding.cross_val_multiscore>
with a RepeatedStratifiedKFold
<https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.RepeatedStratifiedKFold.html#sklearn.model_selection.RepeatedStratifiedKFold>
cv parameter. I have also plotted training scores

I assumed that I should get the same result and the only difference would
be the diagonal results where the diagonal training scores will be all 1.
However, the general results are quite different (you can still see some
fade underlying pattern similar in both).

Any idea of why plotting scores using cross-validation v.s. only
plotting fitting/training scores will give different results?

This is my understanding of what should be going on: in the training case
without using any cross-validation, on each time point, there was a
classifier/decoder that was trained by seeing all EEG channels' data over
all epochs at that train time point, therefore it would give a perfect
score on the same test time point. However, a different time point
(testing times) has different data that can be seen as a test set for this
decoder. Right? (Even if there was an autocorrelation between EEG data over
time and still see some meaningful pattern in time generalization matrix,
it means that EEG data had task-related information over time which is
still meaningful).

1 Like
2

External Email - Use Caution

hi,

when you do:

*cv = StratifiedKFold(n_splits=5, shuffle=True)*

*you make the cross-validation folds random (shuffle=True). It means*
*that everytime you run the code you will get a different value. This the
inherent*

*variance of the statistic that cross-validation reports.*

*To avoid this randomness (which should not be neglected) you can fix the
random state eg*

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)

to avoid this you should use a StratifiedShuffleSplit and use many folds eg
100 to limit
the variance due to the choice of the cv splits.

HTH
Alex

3

External Email - Use Caution

Hello MNEers,

I?m having trouble setting a reference that is the average of M1 and M2.

M1 is the reference used for recording, and is stored as ?Ref? in the vhdr file, but not with the other channels.

I have read the link:
https://mne.tools/stable/auto_tutorials/preprocessing/plot_55_setting_eeg_reference.html
and the suggestion to do this:

# add new reference channel (all zero)

raw_new_ref<https://mne.tools/stable/generated/mne.io.Raw.html#mne.io.Raw> = mne.add_reference_channels<https://mne.tools/stable/generated/mne.add_reference_channels.html#mne.add_reference_channels>(raw<https://mne.tools/stable/generated/mne.io.Raw.html#mne.io.Raw>, ref_channels=['EEG 999'])

raw_new_ref<https://mne.tools/stable/generated/mne.io.Raw.html#mne.io.Raw>.set_eeg_reference(ref_channels=['EEG 050'])

Which I gather switches the reference channel ? I can do that. But how do I, instead of swithing, create the average of M1 and M2, AND know that it worked?
Also, is there an easy way to find what the current reference being used is??

Thanks!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.nmr.mgh.harvard.edu/pipermail/mne_analysis/attachments/20200325/de8b913e/attachment.html

4

External Email - Use Caution

Hi,

to use an average of M1 and M2 you should pass a list containing the
M1 and M2 channel
names in the ref_channels parameter

see

https://mne.tools/stable/auto_tutorials/preprocessing/plot_55_setting_eeg_reference.html#setting-or-changing-the-reference-channel

where it is written:

# use average of mastoid channels as reference
# raw.set_eeg_reference(ref_channels=['M1', 'M2'])

Alex

5

External Email - Use Caution

Hi Alex,

Thanks a lot for the helpful information. I have run my code according to
your suggestion but I want to make sure I understood your point correctly.

As you suggested, the difference in results might be due to "shuffle=True".
However, I do not understand why you suggested "*StratifiedShuffleSplit* "?
what would be the difference between doing "cv = *StratifiedKFold*
*(n_splits=5*, shuffle=True, random_state=42) or cv =
*StratifiedKFold**(*n_splits=5,
*shuffle=False*)" vs "cv = *StratifiedShuffleSplit*(*n_splits=1000*,
random_state=42)"? Why not only setting random state and just using "cv =
*StratifiedShuffleSplit*(*n_splits=5*, random_state=42)" with the same
number of splits?

Thanks,
-Maryam

6

External Email - Use Caution

I would do:

cv = StratifiedShuffleSplit(n_splits=10, random_state=42)

or maybe even more splits (up to n_splits=100). This reduces the variance
due to a particular split definition.

HTH
Alex