# temporal decoding: group analysis for mixed design

**URL:** <https://mne.discourse.group/t/temporal-decoding-group-analysis-for-mixed-design/1385>\
**Category:** Mailing List Archive (read-only)\
**Tags:** list-archive\
**Created:** [December 18, 2017, 7:05am UTC](https://mne.discourse.group/t/temporal-decoding-group-analysis-for-mixed-design/1385 "2017-12-18T07:05:10Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![system](https://global.discourse-cdn.com/free1/uploads/mne/original/1X/85cc6bd2b69cb698a166dc6d880fb550510d0144.jpeg) [@system](https://mne.discourse.group/u/system)\
**Post date:** [December 18, 2017, 7:05am UTC](https://mne.discourse.group/t/temporal-decoding-group-analysis-for-mixed-design/1385/1 "2017-12-18T07:05:10Z")

</div>

Hello MNE experts,

I have MEG data with two within-subject factors (each having 2 and 3  
levels) and one between-subject factor (2 levels). I performed decoding  
analysis on my MEG data by using the function " time\_decod.fit". The  
question is how to perform group analysis in subject's decoding data in MNE  
(or outside MNE) for my mixed design (2 x 3 x 2 factorial design). Besides,  
I have another dependent variable by using "predict\_proba" function to get  
the predicting probability. I want to test whether the distribution of  
predicting probability differ according to my design. Whether the  
difference of the distribution continues in time (e.g., 200-300ms after  
stimulus onset) does not matter. Suggestions will be appreciated.  
-------------- next part --------------  
An HTML attachment was scrubbed...  
URL: [http://mail.nmr.mgh.harvard.edu/pipermail/mne\_analysis/attachments/20171218/110e7e57/attachment.html](http://mail.nmr.mgh.harvard.edu/pipermail/mne_analysis/attachments/20171218/110e7e57/attachment.html)

---

<div class="post-metadata">

**Author:** ![system](https://global.discourse-cdn.com/free1/uploads/mne/original/1X/85cc6bd2b69cb698a166dc6d880fb550510d0144.jpeg) [@system](https://mne.discourse.group/u/system)\
**Post date:** [December 18, 2017, 10:24am UTC](https://mne.discourse.group/t/temporal-decoding-group-analysis-for-mixed-design/1385/2 "2017-12-18T10:24:17Z")

</div>

Dear Yi-hui

Decoding is generally not really adapted for mix-designed, as the models  
are traditionally fit at the single subject level - i.e. your model cannot  
be easily optimized to look for an across-subject effect.

You can however compare the decoding scores across subjects/conditions as a  
first approximation, and specify individual subjects' score as your random  
variable.

For multifactorial within-subject effects, a simple approach can be to  
implement RSA; we recently added this example in MNE:  
[https://mne-tools.github.io/stable/auto\_examples/decoding/decoding\_rsa.html](https://mne-tools.github.io/stable/auto_examples/decoding/decoding_rsa.html)

I will refer you to Kriegoskorte's RSA papers to see how you adapt this  
analysis to your specific needs,

Kindest regards,

Jean-R?mi

---

<div class="post-metadata">

**Author:** ![system](https://global.discourse-cdn.com/free1/uploads/mne/original/1X/85cc6bd2b69cb698a166dc6d880fb550510d0144.jpeg) [@system](https://mne.discourse.group/u/system)\
**Post date:** [December 19, 2017, 10:15am UTC](https://mne.discourse.group/t/temporal-decoding-group-analysis-for-mixed-design/1385/3 "2017-12-19T10:15:10Z")

</div>

Do be careful when doing group-level statistics via inferential statistics on accuracy scores -- Allefeld at al 2016 show some of the problems with the naive approach using things like t-tests or ANOVA. You could use a Binomial/Bernoulli regression model to get around some of points they raise without needing to use their minimum information statistic.

Best,

Phillip

---

<div class="post-metadata">

**Author:** ![system](https://global.discourse-cdn.com/free1/uploads/mne/original/1X/85cc6bd2b69cb698a166dc6d880fb550510d0144.jpeg) [@system](https://mne.discourse.group/u/system)\
**Post date:** [December 19, 2017, 11:10am UTC](https://mne.discourse.group/t/temporal-decoding-group-analysis-for-mixed-design/1385/4 "2017-12-19T11:10:59Z")

</div>

While this paper raises an important subtlety (a significant t-test over  
subjects' decoding accuracies indicates that there "are some people in the  
population whose fMRI data carry information about the experimental  
condition ? but" [...] not that there is "an effect that is typical in the  
population"), my understanding is that this particular issue is not a  
problem for non-parametric statistics.

---

<div class="post-metadata">

**Author:** ![system](https://global.discourse-cdn.com/free1/uploads/mne/original/1X/85cc6bd2b69cb698a166dc6d880fb550510d0144.jpeg) [@system](https://mne.discourse.group/u/system)\
**Post date:** [December 19, 2017, 11:16am UTC](https://mne.discourse.group/t/temporal-decoding-group-analysis-for-mixed-design/1385/5 "2017-12-19T11:16:36Z")

</div>

Hi JR,

Is there a reference for that? i.e., that non-parametric stats aren't  
subject to the same inferential issue as t-tests?

Thanks, Andy

---

<div class="post-metadata">

**Author:** ![system](https://global.discourse-cdn.com/free1/uploads/mne/original/1X/85cc6bd2b69cb698a166dc6d880fb550510d0144.jpeg) [@system](https://mne.discourse.group/u/system)\
**Post date:** [December 19, 2017, 11:24am UTC](https://mne.discourse.group/t/temporal-decoding-group-analysis-for-mixed-design/1385/6 "2017-12-19T11:24:03Z")

</div>

Answering both proposals with the same word: Maybe.

'Non-parametric' doesn't mean 'without assumption' (not even 'without  
distributional assumption') -- the assumptions just tend to be quite  
different. Even the bootstrap has assumptions. You have to check whether  
those assumptions conflict with the inherent boundedness of performance  
scores. It's the boundedness of the scores that's the core of the problem.

There are several parametric tests that could model scores bounded on  
[0,1] -- binomial GLMs, and perhaps even the usual t-test if the  
variable is transformed appropriately (e.g. (x - 0.5) \* 2 to move the  
score to [-1, 1], followed by the Fisher transformation), converting AUC  
to d', etc.

Phillip

> Hi JR,
> 
> Is there a reference for that? i.e., that non-parametric stats aren't  
> subject to the same inferential issue as t-tests?
> 
> Thanks, Andy
> 
> &nbsp;&nbsp;&nbsp;&nbsp;While this paper raises an important subtlety (a significant t-test  
> &nbsp;&nbsp;&nbsp;&nbsp;over subjects' decoding accuracies indicates that there "are some  
> &nbsp;&nbsp;&nbsp;&nbsp;people in the population whose fMRI data carry information about the  
> &nbsp;&nbsp;&nbsp;&nbsp;experimental condition ? but" [...] not that there is "an effect  
> &nbsp;&nbsp;&nbsp;&nbsp;that is typical in the population"), my understanding is that this  
> &nbsp;&nbsp;&nbsp;&nbsp;particular issue is not a problem for non-parametric statistics.
> 
> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Do be careful when doing group-level statistics via inferential  
> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;statistics on accuracy scores -- Allefeld at al 2016 show some  
> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;of the problems with the naive approach using things like  
> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;t-tests or ANOVA. You could use a Binomial/Bernoulli regression  
> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;model to get around some of points they raise without needing to  
> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;use their minimum information statistic.
> 
> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Best,
> 
> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Phillip
> 
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Dear Yi-hui
> > 
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Decoding is generally not really adapted for mix-designed, as  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;the models are traditionally fit at the single subject level -  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;i.e. your model cannot be easily optimized to look for an  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;across-subject effect.
> > 
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;You can however compare the decoding scores across  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;subjects/conditions as a first approximation, and specify  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;individual subjects' score as your random variable.
> > 
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;For multifactorial within-subject effects, a simple approach  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;can be to implement RSA; we recently added this example in MNE:  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[https://mne-tools.github.io/stable/auto\_examples/decoding/decoding\_rsa.html](https://mne-tools.github.io/stable/auto_examples/decoding/decoding_rsa.html)
> > 
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;I will refer you to Kriegoskorte's RSA papers to see how you  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;adapt this analysis to your specific needs,
> > 
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Kindest regards,
> > 
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Jean-R?mi
> > 
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Hello MNE experts,
> > 
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;I have MEG data with two within-subject factors (each  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;having 2 and 3 levels) and one between-subject factor (2  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;levels). I performed decoding analysis on my MEG data by  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;using the function " time\_decod.fit". The question is how  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;to perform group analysis in subject's decoding data in  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;MNE (or outside MNE) for my mixed design (2 x 3 x 2  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;factorial design). Besides, I have another dependent  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;variable by using "predict\_proba" function to get the  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;predicting probability. I want to test whether the  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;distribution of predicting probability differ according to  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;my design. Whether the difference of the distribution  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;continues in time (e.g., 200-300ms after stimulus onset)  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;does not matter. Suggestions will be appreciated.  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Mne\_analysis mailing list  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Mne\_analysis at nmr.mgh.harvard.edu  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;\<mailto:Mne\_analysis at nmr.mgh.harvard.edu\>  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne\_analysis](https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne_analysis)
> > 
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;The information in this e-mail is intended only for the  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;person to whom it is  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;addressed. If you believe this e-mail was sent to you in  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;error and the e-mail  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;contains patient information, please contact the Partners  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Compliance HelpLine at  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[http://www.partners.org/complianceline](http://www.partners.org/complianceline) . If the e-mail was  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;sent to you in error  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;but does not contain patient information, please contact  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;the sender and properly  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;dispose of the e-mail.
> > 
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Mne\_analysis mailing list  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Mne\_analysis at nmr.mgh.harvard.edu  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;\<mailto:Mne\_analysis at nmr.mgh.harvard.edu\>  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne\_analysis](https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne_analysis)
> > 
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;The information in this e-mail is intended only for the person to whom it is  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;addressed. If you believe this e-mail was sent to you in error and the e-mail  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;contains patient information, please contact the Partners Compliance HelpLine at  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[http://www.partners.org/complianceline](http://www.partners.org/complianceline) . If the e-mail was sent to you in error  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;but does not contain patient information, please contact the sender and properly  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;dispose of the e-mail.
> 
> &nbsp;&nbsp;&nbsp;&nbsp;\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_  
> &nbsp;&nbsp;&nbsp;&nbsp;Mne\_analysis mailing list  
> &nbsp;&nbsp;&nbsp;&nbsp;Mne\_analysis at nmr.mgh.harvard.edu  
> &nbsp;&nbsp;&nbsp;&nbsp;\<mailto:Mne\_analysis at nmr.mgh.harvard.edu\>  
> &nbsp;&nbsp;&nbsp;&nbsp;[https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne\_analysis](https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne_analysis)
> 
> &nbsp;&nbsp;&nbsp;&nbsp;The information in this e-mail is intended only for the person to  
> &nbsp;&nbsp;&nbsp;&nbsp;whom it is  
> &nbsp;&nbsp;&nbsp;&nbsp;addressed. If you believe this e-mail was sent to you in error and  
> &nbsp;&nbsp;&nbsp;&nbsp;the e-mail  
> &nbsp;&nbsp;&nbsp;&nbsp;contains patient information, please contact the Partners Compliance  
> &nbsp;&nbsp;&nbsp;&nbsp;HelpLine at  
> &nbsp;&nbsp;&nbsp;&nbsp;[http://www.partners.org/complianceline](http://www.partners.org/complianceline) . If the e-mail was sent to  
> &nbsp;&nbsp;&nbsp;&nbsp;you in error  
> &nbsp;&nbsp;&nbsp;&nbsp;but does not contain patient information, please contact the sender  
> &nbsp;&nbsp;&nbsp;&nbsp;and properly  
> &nbsp;&nbsp;&nbsp;&nbsp;dispose of the e-mail.
> 
> --  
> Andrew R. Dykstra, PhD  
> Department of Neurology  
> Ruprecht-Karls-Universit?t Heidelberg  
> andrew.dykstra at med.uni-heidelberg.de  
> \<mailto:andrew.dykstra at med.uni-heidelberg.de\>  
> Europe: +49.157.7028.2162, North America: +1.786.263.9742
> 
> \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_  
> Mne\_analysis mailing list  
> Mne\_analysis at nmr.mgh.harvard.edu  
> [https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne\_analysis](https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne_analysis)
> 
> The information in this e-mail is intended only for the person to whom it is  
> addressed. If you believe this e-mail was sent to you in error and the e-mail  
> contains patient information, please contact the Partners Compliance HelpLine at  
> [http://www.partners.org/complianceline](http://www.partners.org/complianceline) . If the e-mail was sent to you in error  
> but does not contain patient information, please contact the sender and properly  
> dispose of the e-mail.

-------------- next part --------------  
A non-text attachment was scrubbed...  
Name: signature.asc  
Type: application/pgp-signature  
Size: 473 bytes  
Desc: OpenPGP digital signature  
Url : [http://mail.nmr.mgh.harvard.edu/pipermail/mne\_analysis/attachments/20171219/dcde32f2/attachment-0001.bin](http://mail.nmr.mgh.harvard.edu/pipermail/mne_analysis/attachments/20171219/dcde32f2/attachment-0001.bin)

---

<div class="post-metadata">

**Author:** ![system](https://global.discourse-cdn.com/free1/uploads/mne/original/1X/85cc6bd2b69cb698a166dc6d880fb550510d0144.jpeg) [@system](https://mne.discourse.group/u/system)\
**Post date:** [December 19, 2017, 11:32am UTC](https://mne.discourse.group/t/temporal-decoding-group-analysis-for-mixed-design/1385/7 "2017-12-19T11:32:42Z")

</div>

> Answering both proposals with the same word: Maybe.
> 
> 'Non-parametric' doesn't mean 'without assumption' (not even 'without  
> distributional assumption') -- the assumptions just tend to be quite  
> different. Even the bootstrap has assumptions. You have to check whether  
> those assumptions conflict with the inherent boundedness of performance  
> scores. It's the boundedness of the scores that's the core of the problem.

You're right, I should have been more precise. In my understanding, the  
non-parametric tests that are commonly encountered in second-level decoding  
analyses such as permutation, wilcoxon, mann whitney do not make  
assumptions on the distribution across subjects that are impacted by the  
issue raised by the authors (i.e. that information is never negative, and  
thus that the distribution is bounded).

JR

> There are several parametric tests that could model scores bounded on  
> [0,1] -- binomial GLMs, and perhaps even the usual t-test if the  
> variable is transformed appropriately (e.g. (x - 0.5) \* 2 to move the  
> score to [-1, 1], followed by the Fisher transformation), converting AUC  
> to d', etc.
> 
> Phillip
> 
> \> Hi JR,  
> \>  
> \> Is there a reference for that? i.e., that non-parametric stats aren't  
> \> subject to the same inferential issue as t-tests?  
> \>  
> \> Thanks, Andy  
> \>  
> \>  
> \> While this paper raises an important subtlety (a significant t-test  
> \> over subjects' decoding accuracies indicates that there "are some  
> \> people in the population whose fMRI data carry information about the  
> \> experimental condition ? but" [...] not that there is "an effect  
> \> that is typical in the population"), my understanding is that this  
> \> particular issue is not a problem for non-parametric statistics.  
> \>  
> \>  
> \> Do be careful when doing group-level statistics via inferential  
> \> statistics on accuracy scores -- Allefeld at al 2016 show some  
> \> of the problems with the naive approach using things like  
> \> t-tests or ANOVA. You could use a Binomial/Bernoulli regression  
> \> model to get around some of points they raise without needing to  
> \> use their minimum information statistic.  
> \>  
> \> Best,  
> \>  
> \> Phillip  
> \>  
> \>  
> \>\> Dear Yi-hui  
> \>\>  
> \>\> Decoding is generally not really adapted for mix-designed, as  
> \>\> the models are traditionally fit at the single subject level -  
> \>\> i.e. your model cannot be easily optimized to look for an  
> \>\> across-subject effect.  
> \>\>  
> \>\> You can however compare the decoding scores across  
> \>\> subjects/conditions as a first approximation, and specify  
> \>\> individual subjects' score as your random variable.  
> \>\>  
> \>\> For multifactorial within-subject effects, a simple approach  
> \>\> can be to implement RSA; we recently added this example in MNE:  
> \>\> [https://mne-tools.github.io/stable/auto\_examples/decoding/](https://mne-tools.github.io/stable/auto_examples/decoding/)  
> decoding\_rsa.html  
> \>\>  
> \>\> I will refer you to Kriegoskorte's RSA papers to see how you  
> \>\> adapt this analysis to your specific needs,  
> \>\>  
> \>\> Kindest regards,  
> \>\>  
> \>\> Jean-R?mi  
> \>\>  
> \>\>  
> \>\> Hello MNE experts,  
> \>\>  
> \>\> I have MEG data with two within-subject factors (each  
> \>\> having 2 and 3 levels) and one between-subject factor (2  
> \>\> levels). I performed decoding analysis on my MEG data by  
> \>\> using the function " time\_decod.fit". The question is how  
> \>\> to perform group analysis in subject's decoding data in  
> \>\> MNE (or outside MNE) for my mixed design (2 x 3 x 2  
> \>\> factorial design). Besides, I have another dependent  
> \>\> variable by using "predict\_proba" function to get the  
> \>\> predicting probability. I want to test whether the  
> \>\> distribution of predicting probability differ according to  
> \>\> my design. Whether the difference of the distribution  
> \>\> continues in time (e.g., 200-300ms after stimulus onset)  
> \>\> does not matter. Suggestions will be appreciated.  
> \>\>  
> \>\>  
> \>\> \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_  
> \>\> Mne\_analysis mailing list  
> \>\> Mne\_analysis at nmr.mgh.harvard.edu  
> \>\> \<mailto:Mne\_analysis at nmr.mgh.harvard.edu\>  
> \>\> [https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne\_](https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne_)  
> analysis  
> \>\>  
> \>\>  
> \>\> The information in this e-mail is intended only for the  
> \>\> person to whom it is  
> \>\> addressed. If you believe this e-mail was sent to you in  
> \>\> error and the e-mail  
> \>\> contains patient information, please contact the Partners  
> \>\> Compliance HelpLine at  
> \>\> [http://www.partners.org/complianceline](http://www.partners.org/complianceline) . If the e-mail was  
> \>\> sent to you in error  
> \>\> but does not contain patient information, please contact  
> \>\> the sender and properly  
> \>\> dispose of the e-mail.  
> \>\>  
> \>\>  
> \>\>  
> \>\>  
> \>\> \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_  
> \>\> Mne\_analysis mailing list  
> \>\> Mne\_analysis at nmr.mgh.harvard.edu  
> \>\> \<mailto:Mne\_analysis at nmr.mgh.harvard.edu\>  
> \>\> [https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne\_analysis](https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne_analysis)  
> \>\>  
> \>\>  
> \>\> The information in this e-mail is intended only for the person  
> to whom it is  
> \>\> addressed. If you believe this e-mail was sent to you in error  
> and the e-mail  
> \>\> contains patient information, please contact the Partners  
> Compliance HelpLine at  
> \>\> [http://www.partners.org/complianceline](http://www.partners.org/complianceline) . If the e-mail was  
> sent to you in error  
> \>\> but does not contain patient information, please contact the  
> sender and properly  
> \>\> dispose of the e-mail.  
> \>  
> \>  
> \> \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_  
> \> Mne\_analysis mailing list  
> \> Mne\_analysis at nmr.mgh.harvard.edu  
> \> \<mailto:Mne\_analysis at nmr.mgh.harvard.edu\>  
> \> [https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne\_analysis](https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne_analysis)  
> \>  
> \>  
> \> The information in this e-mail is intended only for the person to  
> \> whom it is  
> \> addressed. If you believe this e-mail was sent to you in error and  
> \> the e-mail  
> \> contains patient information, please contact the Partners Compliance  
> \> HelpLine at  
> \> [http://www.partners.org/complianceline](http://www.partners.org/complianceline) . If the e-mail was sent to  
> \> you in error  
> \> but does not contain patient information, please contact the sender  
> \> and properly  
> \> dispose of the e-mail.  
> \>  
> \> --  
> \> Andrew R. Dykstra, PhD  
> \> Department of Neurology  
> \> Ruprecht-Karls-Universit?t Heidelberg  
> \> andrew.dykstra at med.uni-heidelberg.de  
> \> \<mailto:andrew.dykstra at med.uni-heidelberg.de\>  
> \> Europe: +49.157.7028.2162, North America: +1.786.263.9742  
> \>  
> \>  
> \> \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_  
> \> Mne\_analysis mailing list  
> \> Mne\_analysis at nmr.mgh.harvard.edu  
> \> [https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne\_analysis](https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne_analysis)  
> \>  
> \>  
> \> The information in this e-mail is intended only for the person to whom  
> it is  
> \> addressed. If you believe this e-mail was sent to you in error and the  
> e-mail  
> \> contains patient information, please contact the Partners Compliance  
> HelpLine at  
> \> [http://www.partners.org/complianceline](http://www.partners.org/complianceline) . If the e-mail was sent to you  
> in error  
> \> but does not contain patient information, please contact the sender and  
> properly  
> \> dispose of the e-mail.  
> \>
> 
> \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_  
> Mne\_analysis mailing list  
> Mne\_analysis at nmr.mgh.harvard.edu  
> [https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne\_analysis](https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne_analysis)
> 
> The information in this e-mail is intended only for the person to whom it  
> is  
> addressed. If you believe this e-mail was sent to you in error and the  
> e-mail  
> contains patient information, please contact the Partners Compliance  
> HelpLine at  
> [http://www.partners.org/complianceline](http://www.partners.org/complianceline) . If the e-mail was sent to you in  
> error  
> but does not contain patient information, please contact the sender and  
> properly  
> dispose of the e-mail.

-------------- next part --------------  
An HTML attachment was scrubbed...  
URL: [http://mail.nmr.mgh.harvard.edu/pipermail/mne\_analysis/attachments/20171219/49406797/attachment-0001.html](http://mail.nmr.mgh.harvard.edu/pipermail/mne_analysis/attachments/20171219/49406797/attachment-0001.html)