# Source Space Decoding Classification Timecourse **URL:** \ **Category:** Mailing List Archive (read-only)\ **Tags:** list-archive\ **Created:** [August 4, 2017, 9:11pm UTC](https://mne.discourse.group/t/source-space-decoding-classification-timecourse/1270 "2017-08-04T21:11:01Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![system](https://global.discourse-cdn.com/free1/uploads/mne/original/1X/85cc6bd2b69cb698a166dc6d880fb550510d0144.jpeg) [@system](https://mne.discourse.group/u/system)\ **Post date:** [August 4, 2017, 9:11pm UTC](https://mne.discourse.group/t/source-space-decoding-classification-timecourse/1270/1 "2017-08-04T21:11:01Z")
Hi, I've been trying to modify the following example: [http://martinos.org/mne/dev/auto\_examples/decoding/plot\_decoding\_spatio\_temporal\_source.html](http://martinos.org/mne/dev/auto_examples/decoding/plot_decoding_spatio_temporal_source.html) to yield a time resolved classification accuracy. I'm new to decoding so I've done it in a fairly brute way (just iterating this script over every time point), which yields a fairly convincing classification accuracy timecourse. However, I'm a bit concerned at how high the accuracy is during the baseline, pre-stim period. See attached for the modified script using the sample data and an example of the output. I'm new to decoding, but the best answer I've been able to find for abnormally high pre-stim accuracy is failing to cross validate, but that shouldn't be the case as cross validation is being performed (but perhaps I'm doing it wrong) . Is there something improper about my strategy here? Thanks for any input. Cheers, Cody -------------- next part -------------- An HTML attachment was scrubbed... URL: [http://mail.nmr.mgh.harvard.edu/pipermail/mne\_analysis/attachments/20170804/0976d46b/attachment-0001.html](http://mail.nmr.mgh.harvard.edu/pipermail/mne_analysis/attachments/20170804/0976d46b/attachment-0001.html) -------------- next part -------------- A non-text attachment was scrubbed... Name: sample\_classification\_timecourse.png Type: image/png Size: 34122 bytes Desc: sample\_classification\_timecourse.png Url : [http://mail.nmr.mgh.harvard.edu/pipermail/mne\_analysis/attachments/20170804/0976d46b/attachment-0001.png](http://mail.nmr.mgh.harvard.edu/pipermail/mne_analysis/attachments/20170804/0976d46b/attachment-0001.png) -------------- next part -------------- A non-text attachment was scrubbed... Name: plot\_decoding\_spatio\_temporal\_source\_cc.py Type: text/x-python-script Size: 6428 bytes Desc: plot\_decoding\_spatio\_temporal\_source\_cc.py Url : [http://mail.nmr.mgh.harvard.edu/pipermail/mne\_analysis/attachments/20170804/0976d46b/attachment-0001.bin](http://mail.nmr.mgh.harvard.edu/pipermail/mne_analysis/attachments/20170804/0976d46b/attachment-0001.bin) ---
**Author:** ![system](https://global.discourse-cdn.com/free1/uploads/mne/original/1X/85cc6bd2b69cb698a166dc6d880fb550510d0144.jpeg) [@system](https://mne.discourse.group/u/system)\ **Post date:** [August 4, 2017, 9:19pm UTC](https://mne.discourse.group/t/source-space-decoding-classification-timecourse/1270/2 "2017-08-04T21:19:55Z")
Hi Cody, Do you have the same number of trials in each condition after any trial rejection you do? If not, then the issue might be that 50% is not the correct chance level to think about, rather the correct chance level is the proportion of trials that is in your more frequent condition (eyeballing, maybe like 55%?). There are unbiased classifiers you can use, but I am not sure if they are built into MNE python... Best wishes, Avniel ---
**Author:** ![system](https://global.discourse-cdn.com/free1/uploads/mne/original/1X/85cc6bd2b69cb698a166dc6d880fb550510d0144.jpeg) [@system](https://mne.discourse.group/u/system)\ **Post date:** [August 5, 2017, 7:10am UTC](https://mne.discourse.group/t/source-space-decoding-classification-timecourse/1270/3 "2017-08-05T07:10:07Z")
Hi Cody, Overall, your baseline doesn't look too bad - you would need to do a statistical test to check whether it is just noise variation or above-chance decoding scores. Still there could be multiple reasons behind a significant accuracy before t0 here: - accuracy is biased for imbalanced datasets. You can either use epochs.equalize\_event\_counts before your cross validation, or better, use a 'roc\_auc' scoring metrics - filtering the data can spread information over time. Try changing your filtering parameters - IIRC, the 'sample' protocol is actually not randomized, and it is possible to predict the simulus category in advance. If you're using the MNE master branch, then I would recommend simplfy using this instead of your big loop (see [https://martinos.org/mne/dev/auto\_tutorials/plot\_sensors\_decoding.html#temporal-decoding](https://martinos.org/mne/dev/auto_tutorials/plot_sensors_decoding.html#temporal-decoding) for more details): clf = make\_pipeline(StandardScaler(), SelectKBest(f\_classif, k=500), SVC(kernel='linear')) time\_decod = SlidingEstimator(clf, scoring='roc\_auc') scores = cross\_val\_multiscore(clf, X, y, cv=5) plt.plot(times, scores.mean(0)) (Note that I would personnally recommend clf = make\_pipeline(StandardScaler(), LogisticRegression(C=1)) which should be better) Else, I believe we will be releasing the next version of MNE this month, so you'll just have to update MNE. Hope that helps, Jean-R?mi ---
**Author:** ![system](https://global.discourse-cdn.com/free1/uploads/mne/original/1X/85cc6bd2b69cb698a166dc6d880fb550510d0144.jpeg) [@system](https://mne.discourse.group/u/system)\ **Post date:** [August 5, 2017, 8:08pm UTC](https://mne.discourse.group/t/source-space-decoding-classification-timecourse/1270/4 "2017-08-05T20:08:20Z")
Hi Cody, Depending on your number of trials, the number of feature and the cross validation procedure, you can have fairly high decoding results just by chance. You should never interpret a results without running a statistical test. One good way to get the chance level of your classification pipeline is to run a permutation test : [http://scikit-learn.org/stable/modules/generated/sklearn.model\_selection.permutation\_test\_score.html](http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.permutation_test_score.html) the idea is to shuffle the labels, and train again the model to see what score you get 'by chance'. it is sometimes surprising how high you can get. If you have an unbalanced number of trial per class, i would also suggest to use the AUC as a metric instead of the accuracy. Alex ---
**Author:** ![system](https://global.discourse-cdn.com/free1/uploads/mne/original/1X/85cc6bd2b69cb698a166dc6d880fb550510d0144.jpeg) [@system](https://mne.discourse.group/u/system)\ **Post date:** [August 7, 2017, 9:38pm UTC](https://mne.discourse.group/t/source-space-decoding-classification-timecourse/1270/5 "2017-08-07T21:38:31Z")
Hi all, Thanks for all the helpful suggestions. Everyone brought up imbalanced datasets as a possible source of the problem, but trial counts are equalized both in the example and in my personal dataset where the problem is a bit worse (baseline accuracies are averaging around 65%, even with the code changes JR suggested and with more samples and less features). I also know I shouldn't jump to any conclusions without doing actual stats, and indeed I don't really expect these baseline periods to show up as significantly above chance. However, I figured a reviewer would nail me if I tried to report classification timecourses with that high of a baseline accuracy, even if it was statistically meaningless. JR, thanks for those bits of code, that definitely cleans it up a lot. At Alex and your's suggestions, I'm trying to use the 'roc\_auc' scoring method to see if it calms the baseline down a bit, but I'm getting some inconsistent behavior out of cross\_val\_multiscore when trying to use that metric. Attached is the same source space decoding tutorial from my original message but now modified to run using these functions from the master branch that JR suggested. The plot sensors decoding tutorial you linked runs just fine for me, but when I try to run this on the modified source space tutorial (attached), I get the following error: Value Error: roc\_auc scoring can only be computed for two-class problems It doesn't seem to like the data tag variable y. Strangely enough, if I define y as such: y=epochs.events[:,2], as it is defined in the sensor space tutorial the cross\_val\_multiscore function does not return the error (the scores are obviously bad since the labeling is wrong though). In both cases y is just a simple numpy array with identical shape (112,) and the same number of unique digits, just in different orders. So, I'm not really sure what's happening there, but hopefully others can replicate the problem. Cheers, Cody ---
**Author:** ![system](https://global.discourse-cdn.com/free1/uploads/mne/original/1X/85cc6bd2b69cb698a166dc6d880fb550510d0144.jpeg) [@system](https://mne.discourse.group/u/system)\ **Post date:** [August 7, 2017, 10:28pm UTC](https://mne.discourse.group/t/source-space-decoding-classification-timecourse/1270/6 "2017-08-07T22:28:42Z")
Hi Cody, Scikit-learn 'roc\_auc' metric necessitates to have y values in [0, 1], that's probably the issue: [http://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc\_auc\_score.html](http://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_auc_score.html) I updated and simplified the source decoding example: the PR is still under review here: [https://github.com/mne-tools/mne-python/pull/4465](https://github.com/mne-tools/mne-python/pull/4465) Comments and complaints are more than welcome! HTH JR ---
**Author:** ![system](https://global.discourse-cdn.com/free1/uploads/mne/original/1X/85cc6bd2b69cb698a166dc6d880fb550510d0144.jpeg) [@system](https://mne.discourse.group/u/system)\ **Post date:** [August 8, 2017, 2:18pm UTC](https://mne.discourse.group/t/source-space-decoding-classification-timecourse/1270/7 "2017-08-08T14:18:22Z")
Hey JR, That was what I was thinking at first, but I actually initially first got the error using y values in [0,1]. Plus, the sensor tutorial ( [https://martinos.org/mne/dev/auto\_tutorials/plot\_sensors\_decoding.html#temporal-decoding](https://martinos.org/mne/dev/auto_tutorials/plot_sensors_decoding.html#temporal-decoding)), which runs fine has the y array filled with 1's and 3's. Also, the updated tutorial you linked (which thanks for doing), is also defining y based on y=epochs.events[:,2] before using the cross\_val\_multiscore function with 'roc\_auc' scoring, so those values aren't in range [0,1]. Even if I change the definition of y in the example with the sample data I attached to my last message to: y = np.repeat([0,1], len(X) / 2) # belongs to the second class I still get the error. But that's all presuming others are able to replicate the error and its not just my system being weird. Cheers, Cody ---
**Author:** ![system](https://global.discourse-cdn.com/free1/uploads/mne/original/1X/85cc6bd2b69cb698a166dc6d880fb550510d0144.jpeg) [@system](https://mne.discourse.group/u/system)\ **Post date:** [September 11, 2017, 2:15pm UTC](https://mne.discourse.group/t/source-space-decoding-classification-timecourse/1270/8 "2017-09-11T14:15:17Z")
Hi Cody, Is your problem solved now? Else, can you open an issue on github for us to replicate the error? Thanks JR