# Size limit of data?
**URL:** https://mne.discourse.group/t/size-limit-of-data/725
**Category:** Mailing List Archive (read-only)
**Tags:** list-archive
**Created:** [May 12, 2014, 10:06am UTC](https://mne.discourse.group/t/size-limit-of-data/725 "2014-05-12T10:06:51Z")
**Posts on this page:** 3
**Page:** 1
### Author:  [@system](https://mne.discourse.group/u/system)
#### Post date: [May 12, 2014, 10:06am UTC](https://mne.discourse.group/t/size-limit-of-data/725/1 "2014-05-12T10:06:51Z")
Hi everyone,
I do some analysis in time-frequency, notably, I calculate PSD. I use
functions of mne.time\_frequency.tfr.
I think that the common time of your trials is some seconds. But I think
that there is a common limit to the data size.
Maybe have I exceeded it?
My trials last 70sec and contain 700 000 values.
When I test only 3 trials (epochs) on 2 channels (so only 6 data packages)
with the function induced\_power(), my computer struggles (so with a full
test, my computer dies)!
I don't use a raw object (because my ECG data are not compatible), but I
don't think that it would be a problem for the time being.
I use Fs = 1000 and frequencies=np.arange(1, 130, 1). My computer has a 4
cores of 3.20GHz and 8.00Go of RAM, 64bits.
Is this normal due to the size of my data/my computer power?
If yes, how can I improve it (change computer, cut my trials, force to free
memory...)?
If no, an idea of my possible problem?
Thank by advance,
Best,
Arnaud
-------------- next part --------------
An HTML attachment was scrubbed...
URL: [http://mail.nmr.mgh.harvard.edu/pipermail/mne\_analysis/attachments/20140512/64e02358/attachment.html](http://mail.nmr.mgh.harvard.edu/pipermail/mne_analysis/attachments/20140512/64e02358/attachment.html)
---
### Author:  [@system](https://mne.discourse.group/u/system)
#### Post date: [May 12, 2014, 10:28am UTC](https://mne.discourse.group/t/size-limit-of-data/725/2 "2014-05-12T10:28:43Z")
Hi Arnaud,
I think this is expected due to the limited memory of your computer.
After applying tfr, your data needs about 6.5GB of memory (3 \*2 \* 70 \*
1000 \* 129 \* 128 bits) and before tfr half of that (3.2GB), so you are
exceeding the 8GB available on your computer. Some things that you could
do to save memory:
- Downsample the data to e.g. 500Hz. This would cut the memory needed by
half.
- Process one trial at a time, save the data to file and load it later
for analysis (like that you won't have the 3.2GB original data in memory).
HTH,
Martin
---
### Author:  [@system](https://mne.discourse.group/u/system)
#### Post date: [May 12, 2014, 1:23pm UTC](https://mne.discourse.group/t/size-limit-of-data/725/3 "2014-05-12T13:23:20Z")
Hi Martin,
Thank for this very clear and detailed answer. This should solve my problem.
I'm reassured that it comes from my data and my computer power.
Your 2 proposals are great. So, I'll make these changes.
I thought at another possibility: in Python, if I remember correctly,
internal variables of a function are only free when the function returns
value. But, it is possible to force to (optimally) free memory with the
command "del", and this during the execution of function. That remains to
be seen.
Thank,
Arnaud
2014-05-12 12:28 GMT+02:00 Martin Luessi \:
> Hi Arnaud,
>
> I think this is expected due to the limited memory of your computer.
> After applying tfr, your data needs about 6.5GB of memory (3 \*2 \* 70 \*
> 1000 \* 129 \* 128 bits) and before tfr half of that (3.2GB), so you are
> exceeding the 8GB available on your computer. Some things that you could
> do to save memory:
>
> - Downsample the data to e.g. 500Hz. This would cut the memory needed by
> half.
>
> - Process one trial at a time, save the data to file and load it later
> for analysis (like that you won't have the 3.2GB original data in memory).
>
> HTH,
>
> Martin
>
> \> Hi everyone,
> \>
> \> I do some analysis in time-frequency, notably, I calculate PSD. I use
> \> functions of mne.time\_frequency.tfr.
> \> I think that the common time of your trials is some seconds. But I think
> \> that there is a common limit to the data size.
> \> Maybe have I exceeded it?
> \>
> \> My trials last 70sec and contain 700 000 values.
> \> When I test only 3 trials (epochs) on 2 channels (so only 6 data
> \> packages) with the function induced\_power(), my computer struggles (so
> \> with a full test, my computer dies)!
> \> I don't use a raw object (because my ECG data are not compatible), but I
> \> don't think that it would be a problem for the time being.
> \> I use Fs = 1000 and frequencies=np.arange(1, 130, 1). My computer has a
> \> 4 cores of 3.20GHz and 8.00Go of RAM, 64bits.
> \>
> \> Is this normal due to the size of my data/my computer power?
> \> If yes, how can I improve it (change computer, cut my trials, force to
> \> free memory...)?
> \> If no, an idea of my possible problem?
> \>
> \> Thank by advance,
> \> Best,
> \> Arnaud
> \>
> \>
> \>
> \> \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_
> \> Mne\_analysis mailing list
> \> Mne\_analysis at nmr.mgh.harvard.edu
> \> [Mne\_analysis Info Page](https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne_analysis)
> \>
>
> --
> Martin Luessi, Ph.D.
>
> Research Fellow
>
> Department of Radiology
> Athinoula A. Martinos Center for Biomedical Imaging
> Massachusetts General Hospital
> Harvard Medical School
> 149 13th Street
> Charlestown, MA 02129
>
> Fax: +1 617 726-7422
> \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_
> Mne\_analysis mailing list
> Mne\_analysis at nmr.mgh.harvard.edu
> [Mne\_analysis Info Page](https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne_analysis)
>
> The information in this e-mail is intended only for the person to whom it
> is
> addressed. If you believe this e-mail was sent to you in error and the
> e-mail
> contains patient information, please contact the Partners Compliance
> HelpLine at
> [MyComplianceReport.com: Compliance and Ethics Reporting](http://www.partners.org/complianceline) . If the e-mail was sent to you in
> error
> but does not contain patient information, please contact the sender and
> properly
> dispose of the e-mail.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: [http://mail.nmr.mgh.harvard.edu/pipermail/mne\_analysis/attachments/20140512/26afe321/attachment.html](http://mail.nmr.mgh.harvard.edu/pipermail/mne_analysis/attachments/20140512/26afe321/attachment.html)