shared memory using joblib and clusterering-permutation tests

Mailing List Archive (read-only)
1

Dear joblib experts on this list,

I'm wondering which steps are required to use joblib's shared memory
feature.
I used `mne.set_cache_dir` and made sure my `stat_fun` treats variables
independently.
I started with a moderate buffer_size of 500.
As a result my memory consumption increases radically with 4 jobs and my
system starts swapping excessively.
This is not expected, I think. Given the small buffer size I would not have
expected any significant memory increase.
After some time I witness a `SystemError: error return without exception
set` which is not good.

How can I be really sure it actually works?
I feel we should add more verbosity and logging here.

Best,
Denis
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.nmr.mgh.harvard.edu/pipermail/mne_analysis/attachments/20140806/e8ac804d/attachment.html

2

It turns out the minimum size for memmaping has to be set to make it work.

mne.set_memmap_min_size('100K')

or mne.set_memmap_min_size('1M')

etc.

On success the process will create a memmap / caching in the cache
directory set.

I'm now running computations on a 10GB array using 4 jobs without hangs /
crashes.
Nice.