Is storing large datasets with thousands of patients in BIDS format feasible?

đź’¬ Support & Discussions
, ,
1

Hellođź‘‹
I am working on some massive dataset of EEG recordings which contains around 15,000 patients: Temple University Hospital EEG database

To my extent of understanding, following BIDS data structure, we will end up with 15,000 folders in the root folder which will slow down all navigation tasks.

In the original database this is solved by having one folder per each 100 patients in the root folder. I was wondering if a similar solution exists for BIDS data structure.

2

Any form of very large datasets won’t be efficient to browse through a file explorer. It would also be worsen by using network storage instead of local storage.

What you are looking for in this specific usecase is an object storage solution and a database storing metadata and which information is stored where in your object storage solution (backend) coupled to a frontend of your choice. You can then effectively query the database and select which objects to load.
There are probably ways to integrate part of the BIDS structure; but I don’t know how it would look like.

I am not aware of open-source projects offering this kind of capabilities, but there is probably something already out there.

Mathieu

1 Like
3

Thanks for the reply.

Currently I have three solutions implemented:

  1. Using MongoDB engine
  2. Using HDF5 Files (Mostly for my AI model)
  3. A file structure similar to BIDS, with the exception of the additional subfolders to limit number of patients per folder. This works fine as OS file explorers are fine with up to 1000 folder/file per folder.

I was just hoping there is a way to modify the metadata, so it still remains BIDS compatible. But seems a no go.

4

I think you have to separate your data storage/query from the writer/reader code which reads from or cast to BIDS.

Mathieu

1 Like