I am trying to convert a set of MEG data collected by CTF scanner (extension: .ds) to BIDS format, using mne-bids. it raises the error below on all the files I try.
[WinError 183] Cannot create a file when that file already exists: ‘Z\MEG_data\MNE-pilot-data-bids\sub-02\ses-01\meg\sub-02_ses-01_task-SpAtt_run-01_meg.ds\MarkerFile.mrk.bak’ → ‘Z\MEG_data\MNE-pilot-data-bids\sub-02\ses-01\meg\sub-02_ses-01_task-SpAtt_run-01_meg.ds\sub-02_ses-01_task-SpAtt_run-01_meg.bak’
All the files in the ‘MNE-pilot-data-bids’ are created by write_raw_bids from mne_bids yet it still raises an error about the file that has been created by itself.
This error does not happen with .fif files (Elekta/Megin scanner files).
Below please find the full traceback of the error message:
Traceback (most recent call last):
File "C:\Users\ghafarit\AppData\Local\Temp\ipykernel_11684\1508003131.py", line 1, in <module>
write_raw_bids(raw, bids_path, events_data=events_data,
File "<decorator-gen-585>", line 12, in write_raw_bids
File "C:\Users\ghafarit\Anaconda3\envs\mne\lib\site-packages\mne_bids\write.py", line 1755, in write_raw_bids
copyfile_ctf(raw_fname, bids_path)
File "C:\Users\ghafarit\Anaconda3\envs\mne\lib\site-packages\mne_bids\copyfiles.py", line 164, in copyfile_ctf
os.rename(op.join(dest, fname),
FileExistsError: [WinError 183] Cannot create a file when that file already exists: 'Z:\\MEG_data\\MNE-pilot-data-bids\\sub-02\\ses-01\\meg\\sub-02_ses-01_task-SpAtt_run-01_meg.ds\\MarkerFile.mrk.bak' -> 'Z:\\MEG_data\\MNE-pilot-data-bids\\sub-02\\ses-01\\meg\\sub-02_ses-01_task-SpAtt_run-01_meg.ds\\sub-02_ses-01_task-SpAtt_run-01_meg.bak'
I’m afraid I don’t have access to any other operating system at the moment.
Could you please help me and say how would you suggest I share a small (400Mb) MEG dataset with you? I am happy to share it but don’t have a preference where to.
Hello @tghafari, just a minor note: I edited your post to make the traceback more readable: I simply selected it with my mouse and clicked on the “Preformatted text” in the toolbar. This ensures that blocks of code and tracebacks appear with syntax highlighting.
Thanks for sharing the data and a MWE! Unfortunately I cannot reproduce this issue on Ubuntu 18.04 with mne-bids development version
ds directory : /home/stefanappelhoff/Downloads/bla/phantom_mTBIPredict_20220714_spatttrigcheck.ds
res4 data read.
hc data read.
Separate EEG position data file read.
Quaternion matching (desired vs. transformed):
-16.72 87.41 0.00 mm <-> -16.72 87.41 -0.00 mm (orig : -73.60 56.20 -232.49 mm) diff = 0.000 mm
16.72 -87.41 0.00 mm <-> 16.72 -87.41 -0.00 mm (orig : 73.16 -44.49 -231.10 mm) diff = 0.000 mm
102.70 0.00 0.00 mm <-> 102.70 0.00 0.00 mm (orig : 71.68 76.79 -213.22 mm) diff = 0.000 mm
Coordinate transformations established.
Polhemus data for 3 HPI coils added
Device coordinate locations for 3 HPI coils added
64 extra points added to Polhemus data.
Measurement info composed.
Finding samples for /home/stefanappelhoff/Downloads/bla/phantom_mTBIPredict_20220714_spatttrigcheck.ds/phantom_mTBIPredict_20220714_spatttrigcheck.meg4:
System clock channel is available, checking which samples are valid.
1 x 180000 = 180000 samples from 526 chs
Current compensation grade : 0
ds directory : /home/stefanappelhoff/Downloads/bla/phantom_mTBIPredict_20220714_spatttrigcheck.ds
res4 data read.
hc data read.
Separate EEG position data file read.
Quaternion matching (desired vs. transformed):
-16.72 87.41 0.00 mm <-> -16.72 87.41 -0.00 mm (orig : -73.60 56.20 -232.49 mm) diff = 0.000 mm
16.72 -87.41 0.00 mm <-> 16.72 -87.41 -0.00 mm (orig : 73.16 -44.49 -231.10 mm) diff = 0.000 mm
102.70 0.00 0.00 mm <-> 102.70 0.00 0.00 mm (orig : 71.68 76.79 -213.22 mm) diff = 0.000 mm
Coordinate transformations established.
Polhemus data for 3 HPI coils added
Device coordinate locations for 3 HPI coils added
64 extra points added to Polhemus data.
Measurement info composed.
Finding samples for /home/stefanappelhoff/Downloads/bla/phantom_mTBIPredict_20220714_spatttrigcheck.ds/phantom_mTBIPredict_20220714_spatttrigcheck.meg4:
System clock channel is available, checking which samples are valid.
1 x 180000 = 180000 samples from 526 chs
Current compensation grade : 0
Writing '/home/stefanappelhoff/Downloads/out/participants.tsv'...
Writing '/home/stefanappelhoff/Downloads/out/participants.json'...
Writing '/home/stefanappelhoff/Downloads/out/sub-02/ses-01/meg/sub-02_ses-01_coordsystem.json'...
Writing '/home/stefanappelhoff/Downloads/out/sub-02/ses-01/meg/sub-02_ses-01_coordsystem.json'...
Used Annotations descriptions: ['block end', 'block number 1', 'catch trial', 'cue offset', 'cue onset left', 'cue onset right', 'dot onset left', 'dot onset right', 'response button press', 'stim onset']
Writing '/home/stefanappelhoff/Downloads/out/sub-02/ses-01/meg/sub-02_ses-01_task-SpAtt_run-01_events.tsv'...
Writing '/home/stefanappelhoff/Downloads/out/dataset_description.json'...
/home/stefanappelhoff/Desktop/bids/mne-bids/mne_bids/read.py:130: RuntimeWarning: This filename (/home/stefanappelhoff/Downloads/bla/phantom_mTBIPredict_20220714_spatttrigcheck-eve.ds) does not conform to MNE naming conventions. All events files should end with .eve, -eve.fif, -eve.fif.gz, -eve.lst, -eve.txt, _eve.fif, _eve.fif.gz, _eve.lst, _eve.txt or -annot.fif
events = read_events(events_data).astype(int)
Coordinate transformations established.
Writing '/home/stefanappelhoff/Downloads/out/sub-02/ses-01/meg/sub-02_ses-01_task-SpAtt_run-01_meg.json'...
Writing '/home/stefanappelhoff/Downloads/out/sub-02/ses-01/meg/sub-02_ses-01_task-SpAtt_run-01_channels.tsv'...
Reading 0 ... 179999 = 0.000 ... 299.998 secs...
Copying data files to sub-02_ses-01_task-SpAtt_run-01_meg.ds
Writing '/home/stefanappelhoff/Downloads/out/sub-02/ses-01/sub-02_ses-01_scans.tsv'...
Wrote /home/stefanappelhoff/Downloads/out/sub-02/ses-01/sub-02_ses-01_scans.tsv entry with meg/sub-02_ses-01_task-SpAtt_run-01_meg.ds.
BIDSPath(
root: /home/stefanappelhoff/Downloads/out
datatype: meg
basename: sub-02_ses-01_task-SpAtt_run-01_meg.ds)
Here is my mne sys_info output:
Platform: Linux-5.4.0-122-generic-x86_64-with-glibc2.27
Python: 3.10.4 (main, Mar 31 2022, 08:41:55) [GCC 7.5.0]
Executable: /home/stefanappelhoff/miniconda3/envs/mne_bids/bin/python
CPU: x86_64: 8 cores
Memory: 31.2 GB
mne: 1.1.dev0
numpy: 1.22.4 {}
scipy: 1.8.1
matplotlib: 3.5.2 {backend=QtAgg}
sklearn: Not found
numba: Not found
nibabel: 3.2.2
nilearn: Not found
dipy: Not found
cupy: Not found
pandas: 1.4.2
pyvista: Not found
pyvistaqt: Not found
ipyvtklink: Not found
vtk: Not found
qtpy: 2.0.1 {PyQt5=5.9.7}
ipympl: Not found
pyqtgraph: Not found
pooch: v1.6.0
mne_bids: 0.11.dev0
mne_nirs: Not found
mne_features: Not found
mne_qt_browser: Not found
mne_connectivity: 0.4dev0
mne_icalabel: Not found
Could you please share the final output for folder structure and file names created by write_raw_bids?
And just to confirm did you run my script or did you have to change parts of it in order for it to run without error?
I ran your script and only adjusted the data path, see: /home/stefanappelhoff/Downloads/bla/phantom_mTBIPredict_20220714_spatttrigcheck.ds (the events file you uploaded separately is also in the bla folder). Everything ran without error.
@tghafari please see if running python -m pip install --upgrade https://api.github.com/repos/mne-tools/mne-bids/zipball/main solves your issue (this is installing the mne-bids development version. To revert it, run pip uninstall mne_bids, followed by another pip install mne_bids … this will get you back to the “stable” version)
Thanks Stefan,
This also didn’t help, the same error was raised:
File "C:\Users\ghafarit\AppData\Local\Temp\ipykernel_18068\3279010892.py", line 1, in <module>
write_raw_bids(raw, bids_path, events_data=events_data,
File "<decorator-gen-585>", line 12, in write_raw_bids
File "C:\Users\ghafarit\Anaconda3\envs\mne\lib\site-packages\mne_bids\write.py", line 1755, in write_raw_bids
copyfile_ctf(raw_fname, bids_path)
File "C:\Users\ghafarit\Anaconda3\envs\mne\lib\site-packages\mne_bids\copyfiles.py", line 164, in copyfile_ctf
os.rename(op.join(dest, fname),
FileExistsError: [WinError 183] Cannot create a file when that file already exists: 'Z:\\MEG_data\\MNE-pilot-data-bids\\sub-02\\ses-01\\meg\\sub-02_ses-01_task-SpAtt_run-01_meg.ds\\MarkerFile.mrk.bak' -> 'Z:\\MEG_data\\MNE-pilot-data-bids\\sub-02\\ses-01\\meg\\sub-02_ses-01_task-SpAtt_run-01_meg.ds\\sub-02_ses-01_task-SpAtt_run-01_meg.bak'
Do you have any suggestions? Could this be solely resulted by my OS?
On Windows, if dst exists a FileExistsError is always raised.
Interestingly it doesn’t say so for unix systems (linux, macos) – so this might be why I cannot reproduce your problem.
I think the solution might be that if write_raw_bids is called with overwrite=True (so the files already exist), then we should first completely remove the existing files before writing them again … because apparently the overwriting via os.rename doesn’t work on Windows.
Or we could look into using os.replace instead, see: If you want cross-platform overwriting of the destination, use replace(). (also from the os.rename docs)
Thanks for your reply.
I’m afraid I don’t have access right to try the fix you suggested. below is the error:
(base) C:\Users\ghafarit>pip install git+ssh://git@github.com/sappelhoff/mne-bids.git@fix/ctf/rename
Collecting git+ssh://****@github.com/sappelhoff/mne-bids.git@fix/ctf/rename
Cloning ssh://****@github.com/sappelhoff/mne-bids.git (to revision fix/ctf/rename) to c:\users\ghafarit\appdata\local\temp\pip-req-build-fjnpdfx4
Running command git clone -q 'ssh://****@github.com/sappelhoff/mne-bids.git' 'C:\Users\ghafarit\AppData\Local\Temp\pip-req-build-fjnpdfx4'
Host key verification failed.
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
WARNING: Discarding git+ssh://****@github.com/sappelhoff/mne-bids.git@fix/ctf/rename. Command errored out with exit status 128: git clone -q 'ssh://****@github.com/sappelhoff/mne-bids.git' 'C:\Users\ghafarit\AppData\Local\Temp\pip-req-build-fjnpdfx4' Check the logs for full command output.
ERROR: Command errored out with exit status 128: git clone -q 'ssh://****@github.com/sappelhoff/mne-bids.git' 'C:\Users\ghafarit\AppData\Local\Temp\pip-req-build-fjnpdfx4' Check the logs for full command output.
Is there something I should have done before I can have access?
The fix has already been merged – you can install it via python -m pip install --upgrade https://api.github.com/repos/mne-tools/mne-bids/zipball/main now
