Using mne-bids to convert CTF data to bids format

Hi all,

I am trying to convert a set of MEG data collected by CTF scanner (extension: .ds) to BIDS format, using mne-bids. it raises the error below on all the files I try.

[WinError 183] Cannot create a file when that file already exists: β€˜Z\MEG_data\MNE-pilot-data-bids\sub-02\ses-01\meg\sub-02_ses-01_task-SpAtt_run-01_meg.ds\MarkerFile.mrk.bak’ β†’ β€˜Z\MEG_data\MNE-pilot-data-bids\sub-02\ses-01\meg\sub-02_ses-01_task-SpAtt_run-01_meg.ds\sub-02_ses-01_task-SpAtt_run-01_meg.bak’

All the files in the β€˜MNE-pilot-data-bids’ are created by write_raw_bids from mne_bids yet it still raises an error about the file that has been created by itself.

This error does not happen with .fif files (Elekta/Megin scanner files).

  • MNE version: 1.0.3

  • operating system: Windows 10

  • code snippet:

write_raw_bids(raw, bids_path, events_data=events_data, 
               event_id=events_id, overwrite=True)

I appreciate any help,
Thanks,
Tara

Hi Tara, thanks for the report.

  • is this the full traceback or is there more information in the error message?
  • Have you tried this on another operating system (Linux, Mac?)
  • Can you make a part of the data available and prepare a minimal working example (MWE) so that we can potentially reproduce the bug?

Hi Stefan,

Thanks for your timely reply.

  • Below please find the full traceback of the error message:
Traceback (most recent call last):

  File "C:\Users\ghafarit\AppData\Local\Temp\ipykernel_11684\1508003131.py", line 1, in <module>
    write_raw_bids(raw, bids_path, events_data=events_data,

  File "<decorator-gen-585>", line 12, in write_raw_bids

  File "C:\Users\ghafarit\Anaconda3\envs\mne\lib\site-packages\mne_bids\write.py", line 1755, in write_raw_bids
    copyfile_ctf(raw_fname, bids_path)

  File "C:\Users\ghafarit\Anaconda3\envs\mne\lib\site-packages\mne_bids\copyfiles.py", line 164, in copyfile_ctf
    os.rename(op.join(dest, fname),

FileExistsError: [WinError 183] Cannot create a file when that file already exists: 'Z:\\MEG_data\\MNE-pilot-data-bids\\sub-02\\ses-01\\meg\\sub-02_ses-01_task-SpAtt_run-01_meg.ds\\MarkerFile.mrk.bak' -> 'Z:\\MEG_data\\MNE-pilot-data-bids\\sub-02\\ses-01\\meg\\sub-02_ses-01_task-SpAtt_run-01_meg.ds\\sub-02_ses-01_task-SpAtt_run-01_meg.bak'

Thanks,
Tara

Hello @tghafari, just a minor note: I edited your post to make the traceback more readable: I simply selected it with my mouse and clicked on the β€œPreformatted text” in the toolbar. This ensures that blocks of code and tracebacks appear with syntax highlighting.

Best wishes,
Richard

1 Like

Hi Richard,

Sorry I realized I forgot to do that only after I noticed the correction you made.

Thank you,
Tara

1 Like

Google Drive will do, or perhaps iCloud if you have an iCloud account; or OneDrive, since you’re on Windows. Anything.

Best wishes,
Richard

Thanks,

Bests,
Tara

Thanks for sharing the data and a MWE! Unfortunately I cannot reproduce this issue on Ubuntu 18.04 with mne-bids development version


ds directory : /home/stefanappelhoff/Downloads/bla/phantom_mTBIPredict_20220714_spatttrigcheck.ds
    res4 data read.
    hc data read.
    Separate EEG position data file read.
    Quaternion matching (desired vs. transformed):
     -16.72   87.41    0.00 mm <->  -16.72   87.41   -0.00 mm (orig :  -73.60   56.20 -232.49 mm) diff =    0.000 mm
      16.72  -87.41    0.00 mm <->   16.72  -87.41   -0.00 mm (orig :   73.16  -44.49 -231.10 mm) diff =    0.000 mm
     102.70    0.00    0.00 mm <->  102.70    0.00    0.00 mm (orig :   71.68   76.79 -213.22 mm) diff =    0.000 mm
    Coordinate transformations established.
    Polhemus data for 3 HPI coils added
    Device coordinate locations for 3 HPI coils added
    64 extra points added to Polhemus data.
    Measurement info composed.
Finding samples for /home/stefanappelhoff/Downloads/bla/phantom_mTBIPredict_20220714_spatttrigcheck.ds/phantom_mTBIPredict_20220714_spatttrigcheck.meg4: 
    System clock channel is available, checking which samples are valid.
    1 x 180000 = 180000 samples from 526 chs
Current compensation grade : 0
ds directory : /home/stefanappelhoff/Downloads/bla/phantom_mTBIPredict_20220714_spatttrigcheck.ds
    res4 data read.
    hc data read.
    Separate EEG position data file read.
    Quaternion matching (desired vs. transformed):
     -16.72   87.41    0.00 mm <->  -16.72   87.41   -0.00 mm (orig :  -73.60   56.20 -232.49 mm) diff =    0.000 mm
      16.72  -87.41    0.00 mm <->   16.72  -87.41   -0.00 mm (orig :   73.16  -44.49 -231.10 mm) diff =    0.000 mm
     102.70    0.00    0.00 mm <->  102.70    0.00    0.00 mm (orig :   71.68   76.79 -213.22 mm) diff =    0.000 mm
    Coordinate transformations established.
    Polhemus data for 3 HPI coils added
    Device coordinate locations for 3 HPI coils added
    64 extra points added to Polhemus data.
    Measurement info composed.
Finding samples for /home/stefanappelhoff/Downloads/bla/phantom_mTBIPredict_20220714_spatttrigcheck.ds/phantom_mTBIPredict_20220714_spatttrigcheck.meg4: 
    System clock channel is available, checking which samples are valid.
    1 x 180000 = 180000 samples from 526 chs
Current compensation grade : 0
Writing '/home/stefanappelhoff/Downloads/out/participants.tsv'...
Writing '/home/stefanappelhoff/Downloads/out/participants.json'...
Writing '/home/stefanappelhoff/Downloads/out/sub-02/ses-01/meg/sub-02_ses-01_coordsystem.json'...
Writing '/home/stefanappelhoff/Downloads/out/sub-02/ses-01/meg/sub-02_ses-01_coordsystem.json'...
Used Annotations descriptions: ['block end', 'block number 1', 'catch trial', 'cue offset', 'cue onset left', 'cue onset right', 'dot onset left', 'dot onset right', 'response button press', 'stim onset']
Writing '/home/stefanappelhoff/Downloads/out/sub-02/ses-01/meg/sub-02_ses-01_task-SpAtt_run-01_events.tsv'...
Writing '/home/stefanappelhoff/Downloads/out/dataset_description.json'...
/home/stefanappelhoff/Desktop/bids/mne-bids/mne_bids/read.py:130: RuntimeWarning: This filename (/home/stefanappelhoff/Downloads/bla/phantom_mTBIPredict_20220714_spatttrigcheck-eve.ds) does not conform to MNE naming conventions. All events files should end with .eve, -eve.fif, -eve.fif.gz, -eve.lst, -eve.txt, _eve.fif, _eve.fif.gz, _eve.lst, _eve.txt or -annot.fif
  events = read_events(events_data).astype(int)
    Coordinate transformations established.
Writing '/home/stefanappelhoff/Downloads/out/sub-02/ses-01/meg/sub-02_ses-01_task-SpAtt_run-01_meg.json'...
Writing '/home/stefanappelhoff/Downloads/out/sub-02/ses-01/meg/sub-02_ses-01_task-SpAtt_run-01_channels.tsv'...
Reading 0 ... 179999  =      0.000 ...   299.998 secs...
Copying data files to sub-02_ses-01_task-SpAtt_run-01_meg.ds
Writing '/home/stefanappelhoff/Downloads/out/sub-02/ses-01/sub-02_ses-01_scans.tsv'...
Wrote /home/stefanappelhoff/Downloads/out/sub-02/ses-01/sub-02_ses-01_scans.tsv entry with meg/sub-02_ses-01_task-SpAtt_run-01_meg.ds.
BIDSPath(
root: /home/stefanappelhoff/Downloads/out
datatype: meg
basename: sub-02_ses-01_task-SpAtt_run-01_meg.ds)

Here is my mne sys_info output:

Platform:         Linux-5.4.0-122-generic-x86_64-with-glibc2.27
Python:           3.10.4 (main, Mar 31 2022, 08:41:55) [GCC 7.5.0]
Executable:       /home/stefanappelhoff/miniconda3/envs/mne_bids/bin/python
CPU:              x86_64: 8 cores
Memory:           31.2 GB

mne:              1.1.dev0
numpy:            1.22.4 {}
scipy:            1.8.1
matplotlib:       3.5.2 {backend=QtAgg}

sklearn:          Not found
numba:            Not found
nibabel:          3.2.2
nilearn:          Not found
dipy:             Not found
cupy:             Not found
pandas:           1.4.2
pyvista:          Not found
pyvistaqt:        Not found
ipyvtklink:       Not found
vtk:              Not found
qtpy:             2.0.1 {PyQt5=5.9.7}
ipympl:           Not found
pyqtgraph:        Not found
pooch:            v1.6.0

mne_bids:         0.11.dev0
mne_nirs:         Not found
mne_features:     Not found
mne_qt_browser:   Not found
mne_connectivity: 0.4dev0
mne_icalabel:     Not found

Thank Stefan,

Could you please share the final output for folder structure and file names created by write_raw_bids?
And just to confirm did you run my script or did you have to change parts of it in order for it to run without error?

Bests,
Tara

Sure, see below:

I ran your script and only adjusted the data path, see: /home/stefanappelhoff/Downloads/bla/phantom_mTBIPredict_20220714_spatttrigcheck.ds (the events file you uploaded separately is also in the bla folder). Everything ran without error.

.
β”œβ”€β”€ dataset_description.json
β”œβ”€β”€ participants.json
β”œβ”€β”€ participants.tsv
β”œβ”€β”€ README
└── sub-02
    └── ses-01
        β”œβ”€β”€ meg
        β”‚   β”œβ”€β”€ sub-02_ses-01_coordsystem.json
        β”‚   β”œβ”€β”€ sub-02_ses-01_task-SpAtt_run-01_channels.tsv
        β”‚   β”œβ”€β”€ sub-02_ses-01_task-SpAtt_run-01_events.tsv
        β”‚   β”œβ”€β”€ sub-02_ses-01_task-SpAtt_run-01_meg.ds
        β”‚   β”‚   β”œβ”€β”€ BadChannels
        β”‚   β”‚   β”œβ”€β”€ bad.segments
        β”‚   β”‚   β”œβ”€β”€ ClassFile.cls
        β”‚   β”‚   β”œβ”€β”€ DigTrigChannelInfo.txt
        β”‚   β”‚   β”œβ”€β”€ hz.ds
        β”‚   β”‚   β”‚   β”œβ”€β”€ BadChannels
        β”‚   β”‚   β”‚   β”œβ”€β”€ hz.acq
        β”‚   β”‚   β”‚   β”œβ”€β”€ hz.hc
        β”‚   β”‚   β”‚   β”œβ”€β”€ hz.hist
        β”‚   β”‚   β”‚   β”œβ”€β”€ hz.infods
        β”‚   β”‚   β”‚   β”œβ”€β”€ hz.meg4
        β”‚   β”‚   β”‚   β”œβ”€β”€ hz.newds
        β”‚   β”‚   β”‚   └── hz.res4
        β”‚   β”‚   β”œβ”€β”€ MarkerFile.mrk
        β”‚   β”‚   β”œβ”€β”€ sub-02_ses-01_task-SpAtt_run-01_meg.acq
        β”‚   β”‚   β”œβ”€β”€ sub-02_ses-01_task-SpAtt_run-01_meg.bak
        β”‚   β”‚   β”œβ”€β”€ sub-02_ses-01_task-SpAtt_run-01_meg.eeg
        β”‚   β”‚   β”œβ”€β”€ sub-02_ses-01_task-SpAtt_run-01_meg.hc
        β”‚   β”‚   β”œβ”€β”€ sub-02_ses-01_task-SpAtt_run-01_meg.hist
        β”‚   β”‚   β”œβ”€β”€ sub-02_ses-01_task-SpAtt_run-01_meg.infods
        β”‚   β”‚   β”œβ”€β”€ sub-02_ses-01_task-SpAtt_run-01_meg.meg4
        β”‚   β”‚   β”œβ”€β”€ sub-02_ses-01_task-SpAtt_run-01_meg.newds
        β”‚   β”‚   └── sub-02_ses-01_task-SpAtt_run-01_meg.res4
        β”‚   └── sub-02_ses-01_task-SpAtt_run-01_meg.json
        └── sub-02_ses-01_scans.tsv

5 directories, 31 files

@tghafari Which version of MNE-BIDS are you using?

@sappelhoff apparently used the development version, which may have fixes that are not included in previous releases :thinking:

@sappelhoff thanks, all seems great on your side.

@richard I am using mne 1.0.3.

Can I even upgrade to the development version? or what do you recommend I do?

Thanks

Hello,

I was asking about MNE-BIDS :slight_smile:

Could you please share the output of the following Python commands:

import mne
mne.sys_info()

Thanks!

Hi @richard,

Sorry I misread your message.

below please see my sys_info:

Platform:         Windows-10-10.0.19042-SP0
Python:           3.9.12 | packaged by conda-forge | (main, Mar 24 2022, 23:17:03) [MSC v.1929 64 bit (AMD64)]
Executable:       C:\Users\ghafarit\Anaconda3\envs\mne\python.exe
CPU:              Intel64 Family 6 Model 140 Stepping 1, GenuineIntel: 8 cores
Memory:           31.7 GB

mne:              1.0.3
numpy:            1.21.6 {blas=NO_ATLAS_INFO, lapack=lapack}
scipy:            1.8.0
matplotlib:       3.5.2 {backend=module://matplotlib_inline.backend_inline}

sklearn:          1.1.0
numba:            0.55.1
nibabel:          3.2.2
nilearn:          0.9.1
dipy:             1.5.0
cupy:             Not found
pandas:           1.4.2
pyvista:          0.34.0 {OpenGL 4.5.0 - Build 30.0.101.1404 via Intel(R) Iris(R) Xe Graphics}
pyvistaqt:        0.9.0
ipyvtklink:       0.2.2
vtk:              9.1.0
PyQt5:            5.12.3
ipympl:           Not found
pooch:            v1.6.0

mne_bids:         0.10
mne_nirs:         Not found
mne_features:     Not found
mne_qt_browser:   0.3.1
mne_connectivity: Not found

thanks,
Tara

@tghafari please see if running python -m pip install --upgrade https://api.github.com/repos/mne-tools/mne-bids/zipball/main solves your issue (this is installing the mne-bids development version. To revert it, run pip uninstall mne_bids, followed by another pip install mne_bids … this will get you back to the β€œstable” version)

Thanks Stefan,
This also didn’t help, the same error was raised:

File "C:\Users\ghafarit\AppData\Local\Temp\ipykernel_18068\3279010892.py", line 1, in <module>
    write_raw_bids(raw, bids_path, events_data=events_data,

  File "<decorator-gen-585>", line 12, in write_raw_bids

  File "C:\Users\ghafarit\Anaconda3\envs\mne\lib\site-packages\mne_bids\write.py", line 1755, in write_raw_bids
    copyfile_ctf(raw_fname, bids_path)

  File "C:\Users\ghafarit\Anaconda3\envs\mne\lib\site-packages\mne_bids\copyfiles.py", line 164, in copyfile_ctf
    os.rename(op.join(dest, fname),

FileExistsError: [WinError 183] Cannot create a file when that file already exists: 'Z:\\MEG_data\\MNE-pilot-data-bids\\sub-02\\ses-01\\meg\\sub-02_ses-01_task-SpAtt_run-01_meg.ds\\MarkerFile.mrk.bak' -> 'Z:\\MEG_data\\MNE-pilot-data-bids\\sub-02\\ses-01\\meg\\sub-02_ses-01_task-SpAtt_run-01_meg.ds\\sub-02_ses-01_task-SpAtt_run-01_meg.bak'

Do you have any suggestions? Could this be solely resulted by my OS?

Thanks,
Tara

This function is triggering the error: mne-bids/copyfiles.py at ace2a7d55ba943bb3cc0437ded1ac850c71ae278 Β· mne-tools/mne-bids Β· GitHub

specifically, the os.rename part of it.

Now let’s look at os.rename: os β€” Miscellaneous operating system interfaces β€” Python 3.10.5 documentation

it says:

On Windows, if dst exists a FileExistsError is always raised.

Interestingly it doesn’t say so for unix systems (linux, macos) – so this might be why I cannot reproduce your problem.

I think the solution might be that if write_raw_bids is called with overwrite=True (so the files already exist), then we should first completely remove the existing files before writing them again … because apparently the overwriting via os.rename doesn’t work on Windows.

Or we could look into using os.replace instead, see: If you want cross-platform overwriting of the destination, use replace(). (also from the os.rename docs)

1 Like

@tghafari could you please try to install my fix and try it out? See also: fix: replace instead of rename for win support by sappelhoff Β· Pull Request #1035 Β· mne-tools/mne-bids Β· GitHub

pip uninstall mne_bids
pip install git+ssh://git@github.com/sappelhoff/mne-bids.git@fix/ctf/rename

if you want to go back to your previous version, just do:

pip uninstall mne_bids
pip install mne_bids
1 Like

Hi @sappelhoff ,

Thanks for your reply.
I’m afraid I don’t have access right to try the fix you suggested. below is the error:

(base) C:\Users\ghafarit>pip install git+ssh://git@github.com/sappelhoff/mne-bids.git@fix/ctf/rename
Collecting git+ssh://****@github.com/sappelhoff/mne-bids.git@fix/ctf/rename
  Cloning ssh://****@github.com/sappelhoff/mne-bids.git (to revision fix/ctf/rename) to c:\users\ghafarit\appdata\local\temp\pip-req-build-fjnpdfx4
  Running command git clone -q 'ssh://****@github.com/sappelhoff/mne-bids.git' 'C:\Users\ghafarit\AppData\Local\Temp\pip-req-build-fjnpdfx4'
  Host key verification failed.
  fatal: Could not read from remote repository.

  Please make sure you have the correct access rights
  and the repository exists.
WARNING: Discarding git+ssh://****@github.com/sappelhoff/mne-bids.git@fix/ctf/rename. Command errored out with exit status 128: git clone -q 'ssh://****@github.com/sappelhoff/mne-bids.git' 'C:\Users\ghafarit\AppData\Local\Temp\pip-req-build-fjnpdfx4' Check the logs for full command output.
ERROR: Command errored out with exit status 128: git clone -q 'ssh://****@github.com/sappelhoff/mne-bids.git' 'C:\Users\ghafarit\AppData\Local\Temp\pip-req-build-fjnpdfx4' Check the logs for full command output.

Is there something I should have done before I can have access?

Thanks,
Tara

The fix has already been merged – you can install it via python -m pip install --upgrade https://api.github.com/repos/mne-tools/mne-bids/zipball/main now :slight_smile: