# Stats on decoding scores **URL:** \ **Category:** Mailing List Archive (read-only)\ **Tags:** list-archive\ **Created:** [February 9, 2020, 2:01am UTC](https://mne.discourse.group/t/stats-on-decoding-scores/2074 "2020-02-09T02:01:14Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![system](https://global.discourse-cdn.com/free1/uploads/mne/original/1X/85cc6bd2b69cb698a166dc6d880fb550510d0144.jpeg) [@system](https://mne.discourse.group/u/system)\ **Post date:** [February 9, 2020, 2:01am UTC](https://mne.discourse.group/t/stats-on-decoding-scores/2074/1 "2020-02-09T02:01:14Z")
External Email - Use Caution Hi all, I'm trying to analyze whether my decoding scores over time ( [https://mne.tools/stable/auto\_tutorials/machine-learning/plot\_sensors\_decoding.html#decoding-over-time](https://mne.tools/stable/auto_tutorials/machine-learning/plot_sensors_decoding.html#decoding-over-time)) are "significant" or not, doing permutation testing and cluster-based correction. Does anyone have any idea how to do it in MNE? Thank you, -Maryam -------------- next part -------------- An HTML attachment was scrubbed... URL: [http://mail.nmr.mgh.harvard.edu/pipermail/mne\_analysis/attachments/20200208/e94cb14e/attachment.html](http://mail.nmr.mgh.harvard.edu/pipermail/mne_analysis/attachments/20200208/e94cb14e/attachment.html) ---
**Author:** ![system](https://global.discourse-cdn.com/free1/uploads/mne/original/1X/85cc6bd2b69cb698a166dc6d880fb550510d0144.jpeg) [@system](https://mne.discourse.group/u/system)\ **Post date:** [February 11, 2020, 4:55pm UTC](https://mne.discourse.group/t/stats-on-decoding-scores/2074/2 "2020-02-11T16:55:47Z")
Hi Maryam, First: cluster-based permutation tests won't tell you whether any particular times/clusters are actually significant (see [How NOT to interpret results from a cluster-based permutation test - FieldTrip toolbox](http://www.fieldtriptoolbox.org/faq/how_not_to_interpret_results_from_a_cluster-based_permutation_test/)). The null-hypotheses of these tests is exchangeability of conditions, at least when they're defined over the two-sample t-test; there is some debate over on the FieldTrip mailing list as to whether they make any sense for the one-sample t-tests. I do think you can construct a meaningful cluster-based permutation test using one-sample t-tests in some situations, including in decoding situations, but you have to be careful. Second: You have to transform your decoding scores to be on an unbounded scale before using the t-test or use a different test to construct your permutation test. This follows directly from the assumptions of the t-test (unboundedness and equal variance) and will be especially problematic when your decoding scores in some temporal regions are close to one, but close to 0.5 in other temporal regions, because these cannot have equal variance (the variance of the binomial distribution is a function of its mean). This is discussed in decoding analyzes of fMRI in Allefeld et al. 2015 ([Redirecting](https://doi.org/10.1016/j.neuroimage.2016.07.040)). For a simpler way around this, you could use the Fisher transformation (for correlations) or the logistic function to get decoding scores on an unbounded scale. If you still think you want to try to use a cluster-based permutation test, let me know and I'll see if I can extract the relevant code from a study I'm currently working on. Best, Phillip > ???External Email - Use Caution??? > > Hi all, > > I'm trying to analyze whether my decoding scores over time ( > [Page Redirection](https://mne.tools/stable/auto_tutorials/machine-learning/plot_sensors_decoding.html#decoding-over-time)) > are "significant" or not, doing permutation testing and cluster-based > correction. Does anyone have any idea how to do it in MNE?? > > Thank you,? > -Maryam > > \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_ > Mne\_analysis mailing list > Mne\_analysis at nmr.mgh.harvard.edu > [Mne\_analysis Info Page](https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne_analysis) -------------- next part -------------- An HTML attachment was scrubbed... URL: [http://mail.nmr.mgh.harvard.edu/pipermail/mne\_analysis/attachments/20200211/9a6701bf/attachment.html](http://mail.nmr.mgh.harvard.edu/pipermail/mne_analysis/attachments/20200211/9a6701bf/attachment.html) -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 488 bytes Desc: OpenPGP digital signature Url : [http://mail.nmr.mgh.harvard.edu/pipermail/mne\_analysis/attachments/20200211/9a6701bf/attachment.bin](http://mail.nmr.mgh.harvard.edu/pipermail/mne_analysis/attachments/20200211/9a6701bf/attachment.bin) -------------- next part --------------         External Email - Use Caution -------------- next part -------------- An HTML attachment was scrubbed... URL: [http://mail.nmr.mgh.harvard.edu/pipermail/mne\_analysis/attachments/20200211/9a6701bf/attachment-0001.html](http://mail.nmr.mgh.harvard.edu/pipermail/mne_analysis/attachments/20200211/9a6701bf/attachment-0001.html) ---
**Author:** ![system](https://global.discourse-cdn.com/free1/uploads/mne/original/1X/85cc6bd2b69cb698a166dc6d880fb550510d0144.jpeg) [@system](https://mne.discourse.group/u/system)\ **Post date:** [February 11, 2020, 8:23pm UTC](https://mne.discourse.group/t/stats-on-decoding-scores/2074/3 "2020-02-11T20:23:50Z")
External Email - Use Caution Hi Phillip, Thank you for the response. More specifically, I am trying to use the "STATISTICAL ANALYSIS OF DECODING ACCURACY" in this paper \<[https://www.jneurosci.org/content/38/2/409.long](https://www.jneurosci.org/content/38/2/409.long)\>. They used MATLAB and I am trying to use MNE Python. They did the following steps and report clusters of time points in which the decoding was significantly greater than chance after correction for multiple comparisons (e.g. Figure 3) In Step 1, they tested whether the obtained decoding accuracy at each individual time point during the delay interval was greater than chance using one-sample t-tests comparing the mean accuracy across participants to chance. In Step 2, they constructed a Monte Carlo null distribution of cluster-level t mass values. In Step 3, they obtained a null distribution for the cluster mass. P.s. I am doing this analysis on my own project and data but I also want to present MNE in a neuroscience department who are only using MATLAB to show using python and MNE could be another great option and save their time. That is why I am trying to see if I can use MNE for all steps instead of implementing them by myself in python). Thank you, -Maryam ---
**Author:** ![system](https://global.discourse-cdn.com/free1/uploads/mne/original/1X/85cc6bd2b69cb698a166dc6d880fb550510d0144.jpeg) [@system](https://mne.discourse.group/u/system)\ **Post date:** [February 17, 2020, 3:29pm UTC](https://mne.discourse.group/t/stats-on-decoding-scores/2074/4 "2020-02-17T15:29:44Z")
External Email - Use Caution Hi Maryam, you can use [https://mne.tools/stable/generated/mne.stats.spatio\_temporal\_cluster\_1samp\_test.html](https://mne.tools/stable/generated/mne.stats.spatio_temporal_cluster_1samp_test.html) where X is the accuracy array of shape n\_subjects x n\_times x 1, hope that helps, Kindest regards JR ---
**Author:** ![system](https://global.discourse-cdn.com/free1/uploads/mne/original/1X/85cc6bd2b69cb698a166dc6d880fb550510d0144.jpeg) [@system](https://mne.discourse.group/u/system)\ **Post date:** [February 18, 2020, 3:31am UTC](https://mne.discourse.group/t/stats-on-decoding-scores/2074/5 "2020-02-18T03:31:10Z")
External Email - Use Caution Hi List, I am also trying to understand mne.stats.permutation\_cluster\_1samp\_test(). I used it as follows. t\_obs, clusters, clusters\_pv, H0 = mne.stats.permutation\_cluster\_1samp\_test(X,threshold=.01,out\_type='indices') Here, X is 12x512 containing MMN temporal traces of 12 subjects. The t\_obs is as shown here ([http://https](http://https)://imgur.com/pQDQMAY). I am not able to find the cluster locations in the cluster variable (in output). The cluster\_pv is 12x1, indicating it has found 12 clusters of varying p-values. Any suggestions if I am missing some interpretation or passing some variable incorrectly. Best regards, Neeraj ---
**Author:** ![system](https://global.discourse-cdn.com/free1/uploads/mne/original/1X/85cc6bd2b69cb698a166dc6d880fb550510d0144.jpeg) [@system](https://mne.discourse.group/u/system)\ **Post date:** [February 18, 2020, 9:02am UTC](https://mne.discourse.group/t/stats-on-decoding-scores/2074/6 "2020-02-18T09:02:52Z")
External Email - Use Caution The clusters locations are given in the second output variable, see the documentation HTH JR ---
**Author:** ![system](https://global.discourse-cdn.com/free1/uploads/mne/original/1X/85cc6bd2b69cb698a166dc6d880fb550510d0144.jpeg) [@system](https://mne.discourse.group/u/system)\ **Post date:** [February 20, 2020, 6:10am UTC](https://mne.discourse.group/t/stats-on-decoding-scores/2074/7 "2020-02-20T06:10:29Z")
External Email - Use Caution Hi JR, Thank you for your helpful response. This function is amazing and fast but it did not give a reasonable result (the significant time shape is the same as the actual timing, all times came out significant!! ). I am wondering whether I should use all the default parameters or set threshold for example to 0.5 (still low and all time points are significant, I appreciate any suggestions) because there were two classes for the decoder? Thank you, -Maryam ---
**Author:** ![system](https://global.discourse-cdn.com/free1/uploads/mne/original/1X/85cc6bd2b69cb698a166dc6d880fb550510d0144.jpeg) [@system](https://mne.discourse.group/u/system)\ **Post date:** [February 20, 2020, 8:23am UTC](https://mne.discourse.group/t/stats-on-decoding-scores/2074/8 "2020-02-20T08:23:24Z")
External Email - Use Caution Hi Maryam, It is difficult to help you further but here are a few remarks: - If I recall correctly this function expects a 3d array: make sure that you didn't swap space and time dimension. - It is possible for a cluster analysis, depending on its parameters (threshold) and on the data, to over-estimate the size of a cluster and to thus suggest that everything is significant. This is because, formally, such cluster statistics can't tell you whether each individual dimension (e.g. time) is significant (e.g. see fieldtrip tutorial \<[http://www.fieldtriptoolbox.org/faq/how\_not\_to\_interpret\_results\_from\_a\_cluster-based\_permutation\_test/](http://www.fieldtriptoolbox.org/faq/how_not_to_interpret_results_from_a_cluster-based_permutation_test/)\>). Instead, this analysis indicates whether a significant cluster can be detected. Conclusions can be also drawn about the centroid of the cluster under gaussian assumption: i.e. we can meaningfully report the mean and variance of that cluster. To minimize the impact of cluster parameterization, MNE has a threshold-free clustering method (tfce parameters \<[https://mne.tools/dev/auto\_tutorials/stats-sensor-space/plot\_stats\_cluster\_erp.html](https://mne.tools/dev/auto_tutorials/stats-sensor-space/plot_stats_cluster_erp.html)\>), but it is computationally expensive. - What I would do in your case, is check significance with a non parametric (e.g. wilcoxon) mass univariate analysis corrected with FDR on decimated data Hope that helps, JR