Hi there. This is my first post here. I think MNE is a great tool! However I’ve recently come about some limitations in terms of the capabilities it has to read remote files. Basically, MNE allows reading files that are in the local filesystem (either physically or at least mounted through the OS). I think this covers most use cases since we can most of the times mount a remote storage and work with it as if it was local. As data grows bigger and remote (across labs and groups and different countries) collaboration becomes important I think there is a growing need to facilitate this creating very big remote storage centers that a single computer wouldn’t be able to handle. I think providing MNE with methods to open files from remote as they are needed for an analysis would be a great way to accommodate this need and integrate MNE with remote file storage (in S3 and similars, for DVC remote data registries, datalad, simple wget interfacing in a remote filesystem).
This feature would involve creating an mne.open() method that would be able to check if the data is remote or not by passing information in the path. For example, path="dvc+dvc_specific_path" or path="s3+s3_bucket_path" or by adding an extra keyword in the method remote=None for local data or remote="dvc" or remote="s3".
For example in mne-python/brainvision.py at ad1c43beec450a1c7f4dcdcc02b38869e0a5370d · mne-tools/mne-python · GitHub, instead of calling open() we would call mne.open() which would work something like this:
def _open(path, mode):
_prefix, _path = get_prefix(path)
if prefix is None:
return open(_path, mode)
elif prefix == "dvc":
return open_dvc(_path, mode)
elif prefix == "s3":
return open_s3(_path, mode)
elif ...
else:
raise NotImplementedError()
Similar alternatives would be needed for similar python functions operating with paths.