write_raw_bids deletes SNIRF file

💬 Support & Discussions
,
1
  • MNE version: 1.0.2
  • MNE-bids version: 0.10
  • operating system: Ubuntu 20.04

Hey together!

Background:
I’m working with NIRS data in BIDS format and want to create an analysis pipeline for my dataset.
Therefore, I am using the read_raw_bids command to read in my RawSNIRF object. I can set the datatype to ‘nirs’ without problems. Then I perform some magic (extract bad channels) which I want to save for each subject again (temporarily stored in raw.info['bads']). So analogously, I want to use write_raw_bids now to store the list of bad channels in the given subject file.

Problem:
When executing write_raw_bids(raw, bids_path=bids_path, overwrite=True), it overwrites 5 out of 6 files, namely the ones ending on tsv and json, but deletes my snirf file and in return complains about this now nonexistent file:

…
Writing ‘/pathtodata/bids_nBack_2022-06-09/sub-17/ses-01/nirs/sub-17_ses-01_task-nBack_nirs.json’…
Writing ‘/pathtodata/bids_nBack_2022-06-09/sub-17/ses-01/nirs/sub-17_ses-01_task-nBack_channels.tsv’…
Reading 0 … 8332 = 0.000 … 1024.003 secs…
Copying data files to sub-17_ses-01_task-nBack_nirs.snirf
Traceback (most recent call last):

Input In [243] in <cell line: 1>
write_raw_bids(raw, bids_path=bids_path, overwrite=True)

File <decorator-gen-585>:12 in write_raw_bids
File ~/anaconda3/lib/python3.9/site-packages/mne_bids/write.py:1779 in write_raw_bids
shutil.copyfile(raw_fname, bids_path)
File ~/anaconda3/lib/python3.9/shutil.py:265 in copyfile
with open(src, ‘rb’) as fsrc, open(dst, ‘wb’) as fdst:

FileNotFoundError: [Errno 2] No such file or directory: ‘/pathtodata/bids_nBack_2022-06-09/sub-17/ses-01/nirs/sub-17_ses-01_task-nBack_nirs.snirf’

What’s a bit odd to me is that nirs doesn’t seem to be a valid format in the write_raw_bids, but it is valid in read_raw_bids. Is it not supported at all? Is there some other easy way to save my data that I overlooked? (And why is my - backupped! - snirf file deleted in the first place?)

(tagging @rob-luke as the nirs expert :man_technologist:)

Thanks in advance,
Dani

2

Hhmmmm this does sound a bit odd. Let’s try and figure out whats going on here…

Can you share what the directory structure of your data looks like? And the commands you use to read and write the data (you can skip the magic part :smile: )?

One thing to note is that the original data for BIDS is usually stored in sourcedata directory, this should never be touched (as you mention) and most definetly not deleted!

└─ my_dataset-1/
   ├─ sourcedata/
   │  ├─ sub-01/
   │  ├─ sub-02/
   │  └─ ... 
   ├─ sub-01/
   ├─ sub-02/
   └─ ...

But I think that there is something wrong with the code, it should not be deleting user files without at least displaying a warning or requiring an argument.

The NIRS support is quite new, so thanks for reporting this issue. If you can share the additional details I am sure we can solve this.

2 Likes
3

Whatever might be going on here – it seems like a serious bug to me :thinking: and we should try to fix it asap in MNE-BIDS

cc @sappelhoff @adam2392

1 Like
4

To clarify, I am working on the raw but not the sourcedata, the latter remains untouched. And now that I think about it, shouldn’t I rather store my “processed” data in the derivatives folder instead of modifying the raw bids data? (And how?)
I don’t know if it’s just me, but I found it quite confusing that in all those truly good and thorough tutorials no one really speaks of saving their data. There should be an intuitive way to save intermediate results, similar to what I know from I think nipype for MRI, where each preprocessing step like realign or smooth was saved to a different folder. Maybe take that as a kind suggestion to improve the docs.

Anyways - my data structure is the following:

|bids_nBack_2022-06-09/
|--- README
|--- dataset_description.json
|--- participants.json
|--- participants.tsv
|--- code/
|--- derivatives/
|--- execution/
|--- sourcedata/
|--- ...
|--- sub-17/
|------ ses-01/
|--------- sub-17_ses-01_scans.tsv
|--------- nirs/
|------------ sub-17_ses-01_coordsystem.json
|------------ sub-17_ses-01_optodes.tsv
|------------ sub-17_ses-01_task-nBack_channels.tsv
|------------ sub-17_ses-01_task-nBack_events.tsv
|------------ sub-17_ses-01_task-nBack_nirs.json
|------------ sub-17_ses-01_task-nBack_nirs.snirf # this gets deleted
|------ ses-02/
|--------- (same here)

Code as follows:

# define working directory
os.chdir('/pathtodata/bids_nBack_2022-06-09/')
bids_root = os.getcwd()

# define subjects and sessions
subjects = get_entity_vals(bids_root, 'subject')
sessions = ['01', '02']

for subject in subjects:
    datatype = 'nirs'
    task = 'nBack'
    suffix = 'nirs'
    for session in sessions:
        
        bids_path = BIDSPath(subject=subject, session=session, task=task, suffix=suffix, datatype=datatype, root=bids_root)
        raw = read_raw_bids(bids_path=bids_path, verbose=True)
        
        # ... magic happens ...

        raw.info['bads'] = bch
        write_raw_bids(raw, bids_path=bids_path, overwrite=True)

That’s basically it, now I’m curious. :slight_smile:

5

Sorry for the long silence, but I found the problem and answered to the report here: write_raw_bids deletes SNIRF file · Issue #1021 · mne-tools/mne-bids · GitHub

In fact, this problem also occurred previously and we are now working on a fix (see the links in the GitHub issues).

One way to solve it is to not read from location A and write back to location A (but write to another location).

→ Contributions are always welcome :wink:

2 Likes