# Identical p values after permutation cluster test?! **URL:** \ **Category:** Support & Discussions\ **Tags:** statistics\ **Created:** [March 31, 2021, 7:46am UTC](https://mne.discourse.group/t/identical-p-values-after-permutation-cluster-test/2901 "2021-03-31T07:46:56Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![laura](https://yyz2.discourse-cdn.com/free1/user_avatar/mne.discourse.group/laura/32/79_2.png) [@laura](https://mne.discourse.group/u/laura)\ **Post date:** [March 31, 2021, 7:46am UTC](https://mne.discourse.group/t/identical-p-values-after-permutation-cluster-test/2901/1 "2021-03-31T07:46:56Z")
Hi, I need some help either understanding the exact procedure of the permutation cluster test or finding a mistake in my code: I run mne.stats.permutation\_cluster\_1samp\_test() to find significant activity in waveforms that I extracted from a label. Now I realized, that the resulting p value is the exactly same for quite some of the conditions I tested, but not all of them. I suspected some kind of copy & paste error in my code, but I cannot find any. I also double checked the input data: it is different for each condition. I use the function in the following way: F1, cluster1, p1, H01= mne.stats.permutation\_cluster\_1samp\_test(A[:,:,cond], n\_permutations=5000, tail=0) With A (the waveforms of each subject) having the dimensions (subjects x time x conditions) and “cond” would be replaced with a different index for each condition. Usually I do this in a loop, but the error(?) occurs also when I test it for two conditions manually. In case it is necessary: I’m using Ubuntu 16 and tried with MNE 17.0 and 20.0 Does anyone have an idea or explanation for that behavior? Best, Laura ---
**Author:** ![drammock](https://yyz2.discourse-cdn.com/free1/user_avatar/mne.discourse.group/drammock/32/4_2.png) [@drammock](https://mne.discourse.group/u/drammock)\ **Post date:** [March 31, 2021, 2:44pm UTC](https://mne.discourse.group/t/identical-p-values-after-permutation-cluster-test/2901/2 "2021-03-31T14:44:33Z")
First: I strongly recommend updating to MNE-Python 0.22 (current stable version) if you can. Second: in a permutation test, the number of possible distinct values that a p-value can take on depends on the number of permutations you do. So it’s not unusual for some p-values to end up the same. ---
**Author:** ![laura](https://yyz2.discourse-cdn.com/free1/user_avatar/mne.discourse.group/laura/32/79_2.png) [@laura](https://mne.discourse.group/u/laura)\ **Post date:** [April 1, 2021, 2:34pm UTC](https://mne.discourse.group/t/identical-p-values-after-permutation-cluster-test/2901/3 "2021-04-01T14:34:37Z")
Thank you for your response. 1. You’re right, updating would be a good idea… I tend to avoid it because our firewall makes this always way more complicated than it needs to be 🙄 2. After some more tries with different numbers of permutations, I finally figured out the problem: I ran the analysis not only with the data of all participants but also in subgroups that only include 8/9 participants each. So even though I chose to do 50000 permutations, only 128 were done for the smallest group (were most of the same p values occured). So now I have a different question: Should you do a permutation analysis on those small sample sizes at all? Because it is usually recommended to use 1000+ permutations… Thanks in Advance! Best, Laura ---
**Author:** ![drammock](https://yyz2.discourse-cdn.com/free1/user_avatar/mne.discourse.group/drammock/32/4_2.png) [@drammock](https://mne.discourse.group/u/drammock)\ **Post date:** [April 1, 2021, 3:29pm UTC](https://mne.discourse.group/t/identical-p-values-after-permutation-cluster-test/2901/4 "2021-04-01T15:29:12Z")
in principle it is OK to do a permutation test on a relatively small number of observations (in your case observation=participant). You can compute the uncertainty around your permutation-generated p-value estimates, to get a better sense of how precise they are. A good explanation of that is the second part of this answer: [https://stats.stackexchange.com/a/191343](https://stats.stackexchange.com/a/191343) The `n` in the denominator is the number of permutations, so you can expect with only 8 or 9 participants / 128 permutations the uncertainty of those p-values will be relatively large. ---
**Author:** ![laura](https://yyz2.discourse-cdn.com/free1/user_avatar/mne.discourse.group/laura/32/79_2.png) [@laura](https://mne.discourse.group/u/laura)\ **Post date:** [April 6, 2021, 6:08am UTC](https://mne.discourse.group/t/identical-p-values-after-permutation-cluster-test/2901/5 "2021-04-06T06:08:31Z")
Thank you, that helped me a lot! 🙂