First: cluster-based permutation tests won't tell you whether any
particular times/clusters are actually significant (see How NOT to interpret results from a cluster-based permutation test - FieldTrip toolbox).
The null-hypotheses of these tests is exchangeability of conditions, at
least when they're defined over the two-sample t-test; there is some
debate over on the FieldTrip mailing list as to whether they make any
sense for the one-sample t-tests. I do think you can construct a
meaningful cluster-based permutation test using one-sample t-tests in
some situations, including in decoding situations, but you have to be
careful.
Second: You have to transform your decoding scores to be on an unbounded
scale before using the t-test or use a different test to construct your
permutation test. This follows directly from the assumptions of the
t-test (unboundedness and equal variance) and will be especially
problematic when your decoding scores in some temporal regions are close
to one, but close to 0.5 in other temporal regions, because these cannot
have equal variance (the variance of the binomial distribution is a
function of its mean). This is discussed in decoding analyzes of fMRI in
Allefeld et al. 2015 (Redirecting).
For a simpler way around this, you could use the Fisher transformation
(for correlations) or the logistic function to get decoding scores on an
unbounded scale.
If you still think you want to try to use a cluster-based permutation
test, let me know and I'll see if I can extract the relevant code from a
study I'm currently working on.
Best,
Phillip
???External Email - Use Caution???
Hi all,
I'm trying to analyze whether my decoding scores over time ( Page Redirection)
are "significant" or not, doing permutation testing and cluster-based
correction. Does anyone have any idea how to do it in MNE??
Thank you,?
-Maryam
_______________________________________________
Mne_analysis mailing list
Mne_analysis at nmr.mgh.harvard.edu Mne_analysis Info Page
More specifically, I am trying to use the "STATISTICAL ANALYSIS OF DECODING
ACCURACY" in this paper <https://www.jneurosci.org/content/38/2/409.long>.
They used MATLAB and I am trying to use MNE Python.
They did the following steps and report clusters of time points in which
the decoding was significantly greater than chance after correction for
multiple comparisons (e.g. Figure 3)
In Step 1, they tested whether the obtained decoding accuracy at each
individual time point during the delay interval was greater than chance
using one-sample t-tests comparing the mean accuracy across participants to
chance.
In Step 2, they constructed a Monte Carlo null distribution of
cluster-level t mass values.
In Step 3, they obtained a null distribution for the cluster mass.
P.s. I am doing this analysis on my own project and data but I also want to
present MNE in a neuroscience department who are only using MATLAB to show
using python and MNE could be another great option and save their time.
That is why I am trying to see if I can use MNE for all steps instead of
implementing them by myself in python).
Here, X is 12x512 containing MMN temporal traces of 12 subjects. The t_obs
is as shown here (http://https://imgur.com/pQDQMAY). I am not able to find
the cluster locations in the cluster variable (in output). The cluster_pv
is 12x1, indicating it has found 12 clusters of varying p-values. Any
suggestions if I am missing some interpretation or passing some variable
incorrectly.
Thank you for your helpful response. This function is amazing and fast but
it did not give a reasonable result (the significant time shape is the same
as the actual timing, all times came out significant!! ). I am wondering
whether I should use all the default parameters or set threshold for
example to 0.5 (still low and all time points are significant, I appreciate
any suggestions) because there were two classes for the decoder?
It is difficult to help you further but here are a few remarks:
- If I recall correctly this function expects a 3d array: make sure that
you didn't swap space and time dimension.
- It is possible for a cluster analysis, depending on its parameters
(threshold) and on the data, to over-estimate the size of a cluster and to
thus suggest that everything is significant. This is because, formally,
such cluster statistics can't tell you whether each individual dimension
(e.g. time) is significant (e.g. see fieldtrip tutorial
<http://www.fieldtriptoolbox.org/faq/how_not_to_interpret_results_from_a_cluster-based_permutation_test/>).
Instead, this analysis indicates whether a significant cluster can be
detected. Conclusions can be also drawn about the centroid of the cluster
under gaussian assumption: i.e. we can meaningfully report the mean and
variance of that cluster. To minimize the impact of cluster
parameterization, MNE has a threshold-free clustering method (tfce
parameters
<https://mne.tools/dev/auto_tutorials/stats-sensor-space/plot_stats_cluster_erp.html>),
but it is computationally expensive.
- What I would do in your case, is check significance with a non parametric
(e.g. wilcoxon) mass univariate analysis corrected with FDR on decimated
data