Reduce memroy usage for permutation_cluster_test with TFCE

💬 Support & Discussions
1

Hi all,

In the current implementation (MNE 1.6.1 and current main on github), the use of TFCE in permutation_cluster_test uses a lot of memory. This make it totally unpracticable between 2D data (like time frequency maps). Digging into the code, I noticed it comes from a single line:

line 498 in mne.stats.cluster_level.py:

clusters = [(clusters == ii).ravel() for ii in range(len(clusters))]

This line creates for each point of the compared shapes (excluding the permuted dimension) a boolean mask of the same shape as the initial data. Basically if N is in memory (for one sample), it uses N**2 (boolean). However, a few line later this is eventually transformed by replacing boolean masks by lists of indices for the “True” locations (only one location in case of TFCE since each point is considered as a single cluster!).

Thus I propose to change this line to use only indices (and later go back to boolean if it is what is requested by the user).

Secondly, the “clusters” output is used only once in the whole cluster_level.py, thus removing its computation for TFCE in cases it is not required accelerate a lot the computation of permutation statistics.

I made modifications for me which looks ok. I never contributed to MNE or any big project, should I made a pull request or is there anything to discuss before? Or can I send my version to anybody more experienced to propose the modification?

Thanks,
Nicolas

1 Like
2

Hello,

Feel free to open a PR, and we will discuss in the PR thread.
For the first point, I’m surprised by the gain, because:

  • a boolean mask is lightweight
  • a conversion to indices might yield an higher size depending on the number of elements selected

In the end, a memory profiling of a MWE would be very helpful.

Mathieu