Cluster-based Permutation T-test for Decoders

✉️ Mailing List Archive (read-only)
1

External Email - Use Caution

Hi all,

I am trying to

   - use decoders to decode whether ERP or time-frequency signals have
   any meaninful information of four classes (location of the target on the
   screen) in my experiment *over time *(according to this example
   <https://mne.tools/stable/auto_tutorials/machine-learning/plot_sensors_decoding.html#decoding-over-time>
   ).
   - and then test whether the output of the decoder is significantly
   above the chance (in my case: 1/4=0.25) using a permutation t-test with
   cluster-based correction.

My question is:

   - In the example
   <https://mne.tools/stable/auto_tutorials/machine-learning/plot_sensors_decoding.html#decoding-over-time>
   there are only two classes, so AUC was used. However, what if there are
   more than two classes? How I can analyze the significance of the decoder's
   output with the cluster-based correction?

Thanks,
-Mary
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.nmr.mgh.harvard.edu/pipermail/mne_analysis/attachments/20191009/0805c291/attachment.html

2

External Email - Use Caution

You can use a one-versus-all classifier and compute the average AUC across
categories

HTH
JR

3

External Email - Use Caution

Thanks for the response.
Does AUC take care of the multiple comparison issues? How I will be sure
that the accuracy is significant (above chance)?

-Mary

4

External Email - Use Caution

Hi, no, AUC is just a metrics for effect size. You'll need to do
permutation tests to get a p-value.