Error cals don't match in me.concatenate runs

💬 Support & Discussions
1

Hello,

I have a participant for whom the EOG/ECG channels + some stim channels are missing in one out of three runs. My strategy to preserve the data was to replace the missing channels by matrices of zeros (using raw.add_channels with force_update=True).

When trying to conatenate the run with the two others, I get an error:
ValueError: raw[1]._cals must match
So I am copying the cals from another run of the same participant to this one.

The werid thing is that when I save the first run, with the copied cals and reload it, they are set back to what they where after the update and I get the missmatch error again.

Can you please help me understand whether:

  1. It is ok to copy the cals like this
  2. why saving and loading the file does not preserve them

Code snippet below, I can also share the data if you tell me with with whom (need an email).
Thanks!

mne.sys_info()
Platform: Linux-4.4.0-210-generic-x86_64-with-glibc2.10
Python: 3.8.6 | packaged by conda-forge | (default, Jan 25 2021, 23:21:18) [GCC 9.3.0]
Executable: /home/sh254795/anaconda3/envs/mne/bin/python
CPU: x86_64: 32 cores
Memory: 62.5 GB

mne: 0.24.dev0
numpy: 1.19.5 {blas=NO_ATLAS_INFO, lapack=lapack}
scipy: 1.6.0
matplotlib: 3.3.3 {backend=Qt5Agg}

sklearn: 0.24.1
numba: 0.52.0
nibabel: 3.2.1
nilearn: 0.7.0
dipy: 1.3.0
cupy: Not found
pandas: 1.2.1
mayavi: 4.7.2
pyvista: 0.27.4 {pyvistaqt=0.3.0, OpenGL 4.5.0 NVIDIA 384.130 via Quadro P2000/PCIe/SSE2}
vtk: 9.0.1
PyQt5: 5.12.3



#%%
import os.path as op
import mne
import numpy as np

meg_dir = '/home/sh254795/Desktop/CAL_ISSUE/'


subject = 'SUB23' # select 1 PP
print("Processing subject: %s" % subject)

#%% load the run with the missing channels

run = 'r1'
extension = subject + '_' + run + '_raw.fif'
raw_fname_in = op.join(meg_dir, extension)

extension = subject + '_' + run + '_out_raw.fif'
raw_fname_out = op.join(meg_dir, extension)

print("Input: ", raw_fname_in)
print("Output: ", raw_fname_out)

# read raw data
raw = mne.io.read_raw_fif(raw_fname_in,
                            allow_maxshield=True,
                            preload=True, verbose='error')
raw.fix_mag_coil_types()

cals_run1 = raw._cals
print('cals have size: ' + str(cals_run1.shape))

#%% replace the missing channels

new_chs = []
for i in range(19): # append chan names for stim + external electrodes
    if i<9:
        new_chs.append('STI00'+str(i+1))
    elif 9<=i<16:
        new_chs.append('STI0'+str(i+1))
    elif i==16:
        new_chs.append('EOG061')
    elif i==17:
        new_chs.append('EOG062')
    elif i==18:
        new_chs.append('ECG063')
        
new_chs_type = []
for i in range(19): # append chan types for stim + external electrodes
    if i<=15:
        new_chs_type.append('stim')
    elif 15<i<=17:
        new_chs_type.append('eog')
    elif i==18:
        new_chs_type.append('ecg')
        
data_dummy = np.zeros((len(new_chs),len(raw.times))) # create matrix with zeros, size of data
new_info = mne.create_info(new_chs, raw.info['sfreq'], ch_types = new_chs_type) # add info for the new channels
dummy_raw = mne.io.RawArray(data_dummy, new_info) # create raw array

raw.add_channels([dummy_raw], force_update_info=True) # add the dummy channels to run 0

# the cals have been updated, with ones
cals_run1_update = raw._cals
print('cals after updating have size: ' + str(cals_run1_update .shape)) 

#%% load a second run and copy cals to run 1
extension = subject + '_' + 'r5_raw.fif'
raw_fname_in2 = op.join(meg_dir, extension)
raw2 = mne.io.read_raw_fif(raw_fname_in2,
                        allow_maxshield=True,
                        preload=True, verbose='error')
raw2.fix_mag_coil_types()

cals_run2 = raw2._cals
print('cals for run 2 have size: ' + str(cals_run2 .shape)) 

#%% copy cals from raw2 to raw
raw._cals = raw2._cals

cals_run1_replace = raw._cals
print('cals after updating have size: ' + str(cals_run1_update .shape)) 

cals_diff = cals_run1_replace - cals_run2
print('difference in cals: ' + str(sum(cals_diff)))

#%% now save run1 and reload

# check cals: 
cals_before_saving = raw._cals
raw.save(raw_fname_out, overwrite=True)

#%% reload run 1 and compare cals
del raw, raw2

#%%
run = 'r1'

extension = subject + '_' + run + '_out_raw.fif'
raw_fname_in = op.join(meg_dir, extension)

raw_new = mne.io.read_raw_fif(raw_fname_in,
                        allow_maxshield=True,
                        preload=True, verbose='error')
cals_after_saving = raw_new._cals

cals_diff = cals_before_saving - cals_after_saving
print('difference in cals after reloading: ' + str(sum(cals_diff)))

#%%
2

Not really, because as you see it creates problems. Specifically, modifying private attributes is not expected to work. In this case I think the info['chs'] values (that originally create raw._cals on read, and are also not meant to be changed by the user) get used on write, hence “restoring” the bad _cals.

Can you open a bug report? I think we should think about some API to reset the cals so that this sort of thing doesn’t happen.