ValueError: raw[1]._cals must match

✉️ Mailing List Archive (read-only)
1

Hi all,

I'm working on a python script that makes a copy of a directory of FIFF files using python's shutil.copytree() function and then removes some metadata from each of the files in that new copy using the MNE module. It works fine on the whole but there is one particular file which, for some reason, causes the script to break. Here is the code I use and resulting error message:

f = '/space/jazz/1/users/gwarner/temp_dirs_for_fiff_registration/temp_meg_storage/pp010/009/pp010_arc_raw.fif'
raw = Raw(f,preload=False,verbose=False)

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<string>", line 2, in __init__
  File "/space/jazz/1/users/gwarner/anaconda/lib/python2.7/site-packages/mne/utils.py", line 538, in verbose
    return function(*args, **kwargs)
  File "/space/jazz/1/users/gwarner/anaconda/lib/python2.7/site-packages/mne/io/fiff/raw.py", line 114, in __init__
    _check_raw_compatibility(raws)
  File "/space/jazz/1/users/gwarner/anaconda/lib/python2.7/site-packages/mne/io/base.py", line 1995, in _check_raw_compatibility
    raise ValueError('raw[%d]._cals must match' % ri)
ValueError: raw[1]._cals must match

The strangest part is that I can open the original file without any problems:

f = '/space/lilli/3/users/DARPA-TRANSFER/meg/pp010/009/pp010_arc_raw.fif'
raw = Raw(f,preload=False,verbose=False)>>>

Stranger still is that the bash diff command tells me that both versions of the file are identical.

[gwarner at jazz ~] diff /space/jazz/1/users/gwarner/temp\_dirs\_for\_fiff\_registration/temp\_meg\_storage/pp010/009/pp010\_arc\_raw\.fif /space/lilli/3/users/DARPA\-TRANSFER/meg/pp010/009/pp010\_arc\_raw\.fif \[gwarner at jazz \~\]

The fact that it works fine for several hundred files before breaking here tells me that it isn't a problem with my implementation of the module, the fact that I can successfully open the original copy of the file tells me that it isn't just a bad/corrupted file, and the result of the diff test seems to indicate that it isn't a problem with the way it was copied either. I'm using the same version of MNE each time and have all the necessary permissions. Can anyone enlighten me as to what is going on here? Please let me know if there is any additional information I can provide to clear this up.

Thank you,
Graham
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.nmr.mgh.harvard.edu/pipermail/mne_analysis/attachments/20151120/65a7b7f8/attachment-0001.html

2

Hum this sounds mysterious.
Could you detail what you do when you copy meta data?
Could you maybe share some code that demonstrates what you do?

Cheers,
Denis

3

Shutil.copytree() uses shutil.copystat() to copy metadata, it's similar to `cp -p` in bash. Here is the documentation for shutil.copystat() https://docs.python.org/2/library/shutil.html#shutil.copystat

I don't do anything else for the metadata. The line that throws the error, `raw = Raw(f,preload=False,verbose=False)`, is the first line of my script involving mne except for `from mne.io import Raw` at the beginning. It would use mne for other things if it opened successfully but since it crashes trying to open it I don't think those things are relevant. Possibly of importance is that this run was automatically divided into 2 FIFF files because of it's size. However, the fact that I can open the original copy without incident makes me skeptical of how relevant that is. It turns out that the files do differ in someway that isn't detected by bash's diff command because they have different hashes. I'm looking into exactly how they differ now but am having trouble finding the proper tools to do so. It could be that the different file paths cause the hashses to differ, I'm not sure, I'm not very well versed in how hashes work. I'm using the md5 hash from python's hashlib module. I'm not entirely sure what other sections of my code would be useful for you.

Thank you,
Graham

4

hi,

I just logged at MGH and I can read both files without any issue with
current version of MNE. See below

Alex

5

Hi Alex,

Which version of python are you using? I've been unable to replicate your results with both my personal copy of anaconda and the martinos center copy using the exact line of code you used.

Thanks,
Graham

6

hi,

ok problem reproduced. Basically it's the second part of the raw file
that is different. Try:

diff /space/lilli/3/users/DARPA-TRANSFER/meg/pp010/009/pp010_arc_raw-1.fif
/space/jazz/1/users/gwarner/temp_dirs_for_fiff_registration/temp_meg_storage/pp010/009/pp010_arc_raw-1.fif

the error tells you that the measurement info of the second part do not match.

HTH
Alex