I would like to hear what people do to filter and run ICA and if there
is any advise.
We usually have around an hour of recording which gives ~4.5 to 5GB of
raw fiff files. First filtering and then running ICA in MNE-python
requires a lot of memory, sometimes as much as 50GB. So, I fairly often
get a memory error.
I would prefer not to downsample at this stage in the process. So, I
kindly ask if anybody has any thoughts and/or practises to avoid very
heavy memory use.
Which version of sklearn are you using?
Do you use the decim parameter for ICA?
How do axactly do you use ICA?
50GB of memory is unexpected, it would mean that you make up to 10 copies
of your data.
I use sklearn 0.17 (from anaconda). I have tried to the
"decim" param. I remember it as being "3" for data with 1000Hz sfreq.
But it didn't help much.
I have attach a script to show how I used it.
cheers,
mads
Hi Mads,
Which version of sklearn are you using?
Do you use the decim parameter for ICA?
How do axactly do you use ICA?
50GB of memory is unexpected, it would mean that you make up to 10
copies of your data.
Hi all,
I would like to hear what people do to filter and run ICA and if there
is any advise.
We usually have around an hour of recording which gives ~4.5 to 5GB of
raw fiff files. First filtering and then running ICA in MNE-python
requires a lot of memory, sometimes as much as 50GB. So, I fairly often
get a memory error.
I would prefer not to downsample at this stage in the process. So, I
kindly ask if anybody has any thoughts and/or practises to avoid very
heavy memory use.
The information in this e-mail is intended only for the person to
whom it is
addressed. If you believe this e-mail was sent to you in error and
the e-mail
contains patient information, please contact the Partners Compliance
HelpLine at http://www.partners.org/complianceline . If the e-mail was sent to
you in error
but does not contain patient information, please contact the sender
and properly
dispose of the e-mail.
The information in this e-mail is intended only for the person to whom it is
addressed. If you believe this e-mail was sent to you in error and the e-mail
contains patient information, please contact the Partners Compliance HelpLine at http://www.partners.org/complianceline . If the e-mail was sent to you in error
but does not contain patient information, please contact the sender and properly
dispose of the e-mail.
it seems you don't use the decim parameter, do you?
Two things that I see immediately:
It should in fact save a lot. With 1000HZ you can decimate even more, for
ECG/EOG your sampling frequency should not be lower than 50 or so.
Second as you have 1 hour of data the ecg_epochs will be huge, assuming you
find many events.
Very often only a few are necessary to do the detection. Have you tried
picking the 100-200 first events?
Thanks for you advice (I hadn?t considered the number of ECG epochs!). I would like to follow up on Mads? original post regarding the large memory-footprint.
An hour of MEG+EEG at 1kHz adds up to about 5GB of data saved as 32-bit floats. However, preloading raw data (regardless of origin) forces a cast to float-64 (10 GB). This makes sense: preloading raw data means you are going to want to filter it, which requires the bit-depth to work.
However, reading all 400+ channels into memory does not make sense from an implementation perspective, since (most of) the relevant operations are channel-wise (certainly filtering). A much more efficient strategy would be to support block-reads of channel subsets, probably combined with np.memmap:ing. I can see some almost-implementations of memmap in the code, but the standard API does not expose them (for simplicity, understandably). Nor am I completely certain a memmap is helpful here...
There?s been some chatter on related topic on github (where this discussion also belongs), but I wonder if there?s any developments in this direction?
memory usage is certainly a topic where we can progress.
For memmap just pass a string to preload parameter. I wrote this
years ago, but I admit I almost never used it. So not sure how much
it can actually help.
if you can share a script that we can use for memory monitoring eg on
sample data
that would be great.
Hi Alex, I?ll move this to github, with an example script using memory_profiler. My initial experiments some time ago fizzled when I got the feeling I was thinking about memmapping the wrong way (and that it would not in fact be beneficial in the filtering-scenario). /Chris
One thing to add to Denis's suggestion about ECG events. I encourage you to use a sample of ~300 plus throughout the recording (i.e. Find the events then take every 10th or something). As the ECG components/SSPs can shift over time depending on participant movement.
Hth
D