rescanning directory tree when working with a large dataset

[maximilien.chaumon]

Hi,

I’m starting to work with mne-bids on a dataset with 300+ subjects, 5 sessions, 8 runs each. That’s a lot of files, and whenever my bidspath isn’t perfectly unambiguous, it seems to me that it keeps rescanning the entire dirtree. Does that make sense ? Is there a way to avoid this?

Also in a specific case I’m in now, it seems that mne_bids.read_raw_bids and mne_bids.inspect_dataset aren’t resolving this in the same way. See the screenshot below where you see that my bids_path variable is fully determined, read_raw_bids reads it fine, but inspect_dataset apparently rescans the whole directory structure (been running like this for 7 min…)

What should I do?

Many thanks,

Max

[sappelhoff (Stefan Appelhoff)]

Hey @maximilien.chaumon I see that you have opened a PR for this in optimize rootpath scan when entities are known by dnacombo · Pull Request #1450 · mne-tools/mne-bids · GitHub

@bruAristimunha was interested in speeding up some of the mne-bids functionality. So perhaps you two can work on this together.

Thanks!

[bruAristimunha (Bruno Aristimunha)]

We are working on @sappelhoff