Question about permutation testing

✉️ Mailing List Archive (read-only)
1

External Email - Use Caution

Dear MNE users,

I have a question about using permutation tests to compare decoding accuracy across time between two conditions. Specifically, I am interested to see whether these conditions not only differ during stimulus presentation, but also already in anticipation of a new (search) display. I have a within subject design so I am running the following code

t_obs, clusters, cluster_pv, H0 = mne.stats.permutation_cluster_1samp_test(X_1 - X_2)

If I then inspect the results, I get significant clusters during stimulus presentation (which is nice. However, there are no significant anticipatory clusters, even though numerically the conditions clearly differ.
Interestingly, I am able to make these anticipatory clusters significant if instead of analysing the whole time course, I limit the permutation test to the anticipatory window. I assume that this is the case because the anticipatory cluster is smaller than the reactive cluster???

My question is whether this is a valid approach, and if not whether there is another alternative way to still show that the conditions already differ before stimulus processing.

Thanks in advance,

Dirk van Moorselaar

2

External Email - Use Caution

If I then inspect the results, I get significant clusters during stimulus
presentation (which is nice. However, there are no significant
anticipatory clusters, even though numerically the conditions clearly
differ.
Interestingly, I am able to make these anticipatory clusters significant
if instead of analysing the whole time course, I limit the permutation test
to the anticipatory window. I assume that this is the case because the
anticipatory cluster is smaller than the reactive cluster???

The `step_down_p` is meant to deal with this case. Basically it runs the
clustering, and if it finds any clusters below the step_down_p threshold
(usually 0.05 is a good choice), it cuts those data points out of the
analysis and redoes the permutation test. This prevents the first cluster
from dominating the H0 values, increasing sensitivity to smaller but still
valid effects.

My question is whether this is a valid approach, and if not whether there

is another alternative way to still show that the conditions already differ
before stimulus processing.

It seems reasonable, but keep in mind that interpreting
cluster spatio-temporal locations from tests is considered problematic (see
the bottom of this section in our stats tutorial
<Page Redirection,
and the linked Fieldtrip tutorial
<http://www.fieldtriptoolbox.org/faq/how_not_to_interpret_results_from_a_cluster-based_permutation_test/&gt;\).
A potential workaround is to use TFCE (see the stats tutorial), though it
will take longer to run.
<http://www.fieldtriptoolbox.org/faq/how_not_to_interpret_results_from_a_cluster-based_permutation_test/&gt;

Eric
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.nmr.mgh.harvard.edu/pipermail/mne_analysis/attachments/20200410/398d4058/attachment.html