I recently migrated my spatiotemporal clustering tests of MEG data to MNE from Eelbrain since the latter does not work on my university’s HPC cluster. To make sure that my code was working properly, I reran some old analyses previously done in Eelbrain with the expectation of replicating the results, but this was only partially the case. While I did observe the exact same clusters (in terms of size and duration), the p values differed drastically between the two analyses. For example, the p value of one cluster changed from p < 0.05 in Eelbrain to being 0.16 in MNE.
Since the clusters are the same across the two analyses, it must have to do with how the clustering statistic is computed, and I’d like to understand the mechanics of what each package does. As such, I was wondering if you could walk me through the steps in computing the clustering statistic in MNE? Any insights on the difference between the two packages are naturally appreciated as well, but I’m also reaching out to Christian Brodbeck to ask for how the clustering statistic is computed in Eelbrain.
I use the MNE function mne.stats.spatio_temporal_cluster_test and the Eelbrain function Eelbrain.testnd.ANOVA.
thank you for your question, I didn’t compare the code line by line but one potential reason might be the number of permutations. The MNE default number of permutations is 1024, in eelbrain, it is 10.000. Could you check if you get the same results if you set the same number of permutations in both tests to be the same?
Just a shot in the dark - Doesn’t eelbrain return both uncorrected and corrected p-values for identified clusters? Any chance that the corrected p-values (from EELbrain) match MNE’s cluster p-values?
As far as I know, the cluster object returned by Eelbrain only returns one set of p values for the individual clusters, which appears to be corrected if the p values embedded in the cluster follow the naming conventions for the overall object returned by the test (eelbrain.testnd.ANOVA — Eelbrain 0.39.12 documentation). However, I’d love to get input on this if that seems incorrect.
Hmm ok - I thought that the ANOVA class had attributes for both corrected and uncorrected p-values? (the p and p_uncorrected attributes respectively. Though these are called “Variables” in the Eelbrain doc).
I meet a question about the function ‘spatio_temporal-cluster_test’. I used the EEG data, so the ‘adjacency’ was calculated by Delaunay triangulations. it means that if at least three electrodes near an electrode must satisfy clustering significance to prove that the electrode is significant? I can not find the function ‘spatio_temporal-cluster_test’ where the adjacency explain how many electrodes near an electrode must satisfy clustering significance.
I will appreciate it if you could reply me.
I’ve checked up on this with the Eelbrain community, and there are indeed no uncorrected p-values for clusters. Rather, the uncorrected p-values simply correspond to F values, which are calculated for each point in space/time. Clusters span multiple points in space time, hence they neither have F values or corresponding p values (the cluster statistic is the sum of F values across the cluster).
I’ve heard back from the Eelbrain community regarding how p values are calculated. Basically, the procedure is as follows:
The testnd.ANOVA functions creates NDPermutationDistribution objects to keep track of the max(cluster-stat) in each permutation;
creates an iterator over permutations;
and runs the test for the permutations (this may look somewhat more complex due to parallelization)
Hopefully this was helpful! Have you had a chance to dig deeper into the MNE code?