read_raw_ctf().load_data() overflow error

šŸ’¬ Support & Discussions
21

Pretty much the same.

image
image261Ɨ605 27 KB

22

The spikes in ā€œHard Faults/secā€ indicate that the computer is severely running out of memory and needs to start swapping like crazy. Is there a way that you could monitor the swap usage? I think the Performance Monitor allows for that too, although I have to say I havenā€™t worked on Windows for almost 10 years now so I could be mistaken.

23

@richard
In red thereā€™s the memory usage, and in green the pagingfile usage. (which I understand is similar to the swap memory usage). after the crash the memory usage keeps constant until I close the console.

image
image1117Ɨ855 86.4 KB

24

@larsoner @richard

Hi! Was wondering if I could already delete the uploaded file that I shared. Also, thit it work?

On the other hand I tried to load that data in a Linux PC with:

raw = mne.io.read_raw_ctf(file_path, system_clock='ignore')

OUT:
ds directory : /home/usuario/Desktop/MEGEYEHS/MEGEYEHS_scripts/15909001_MatiasIson_20220530_02.ds
    res4 data read.
    hc data read.
    Separate EEG position data file read.
    Quaternion matching (desired vs. transformed):
       0.42   68.31    0.00 mm <->    0.42   68.31    0.00 mm (orig :  -53.00   45.90 -248.52 mm) diff =    0.000 mm
      -0.42  -68.31    0.00 mm <->   -0.42  -68.31    0.00 mm (orig :   45.10  -49.18 -246.57 mm) diff =    0.000 mm
      83.73    0.00    0.00 mm <->   83.73   -0.00    0.00 mm (orig :   53.82   58.80 -242.95 mm) diff =    0.000 mm
    Coordinate transformations established.
    Polhemus data for 3 HPI coils added
    Device coordinate locations for 3 HPI coils added
    2 extra points added to Polhemus data.
    Measurement info composed.
Traceback (most recent call last):
  File "/usr/lib/python3.8/code.py", line 90, in runcode
    exec(code, self.locals)
  File "<input>", line 1, in <module>
  File "/home/usuario/Desktop/MEGEYEHS_Test/lib/python3.8/site-packages/mne/io/ctf/ctf.py", line 66, in read_raw_ctf
    return RawCTF(directory, system_clock, preload=preload,
  File "<decorator-gen-223>", line 12, in __init__
  File "/home/usuario/Desktop/MEGEYEHS_Test/lib/python3.8/site-packages/mne/io/ctf/ctf.py", line 143, in __init__
    info, preload, first_samps=first_samps,
UnboundLocalError: local variable 'first_samps' referenced before assignment

So now I donā€™t even get to read the data. I tried creating a fresh environment and reinstalling mne and the same error came up. Do you know what could be causing this?

I found this post where @mscheltienne said to try it on a fresh environment but it didnā€™t work in my case.

The pc specs:

mne.sys_info()

OUT:
Platform:         Linux-5.13.0-44-generic-x86_64-with-glibc2.29
Python:           3.8.10 (default, Mar 15 2022, 12:22:08)  [GCC 9.4.0]
Executable:       /home/usuario/Desktop/MEGEYEHS_Test/bin/python
CPU:              x86_64: 16 cores
Memory:           31.1 GB
mne:              1.0.3
numpy:            1.22.4 {}
scipy:            1.8.1
matplotlib:       3.5.2 {backend=TkAgg}
sklearn:          Not found
numba:            Not found
nibabel:          Not found
nilearn:          Not found
dipy:             Not found
cupy:             Not found
pandas:           Not found
pyvista:          Not found
pyvistaqt:        Not found
ipyvtklink:       Not found
vtk:              Not found
PyQt5:            Not found
ipympl:           Not found
pooch:            v1.6.0
mne_bids:         Not found
mne_nirs:         Not found
mne_features:     Not found
mne_qt_browser:   Not found
mne_connectivity: Not found

Thanks again!

25

Ouch, this seems very much like a bug in MNEā€¦

26

can you share the file privately with some code to replicate the problem?

Alex

27

@agramfort hereā€™s the share link

28

Yes, thereā€™s the data.

The code is pretty straight forward:

import mne
file_path = 'path/to/file'
raw = mne.io.read_raw_ctf(file_path, system_clock='ignore')

This would yield an error in Ubuntu now. But before I was trying on windows and this worked ok, but then:

raw.load_data()

Wouldnā€™t work.
The error messages can be foun before in this same post.

1 Like
29

Yes this is about where Iā€™d expect it to die. On Windows this is not a memory issue, but rather that file seeking is annoyingly limited to 32-bit unsigned ints (max val 4294967295) so it should die right around 4GB. We can find a way to work around this. In the meantime Linux should workā€¦

Iā€™ll give it a shot and see if I can replicate soon

30

Okay Iā€™ve opened this PR:

My local testing suggests that perhaps your .meg4 file is missing when you try to read on Ubuntu. Did you copy it properly, with the correct filename? For whatever reason, when I downloaded it from your drive locally it had a -002.meg4 at the end when it just should have been .meg4. Now I can read it fine on my macOS machine (including .load_data()). So we still have the Windows problem to tackle, but hopefully Linux works for you for now at least?

31

It worked!

Iā€™m sorry I didnā€™t see the name changed in the meg4 fileā€¦

Yes this is great for now, at least I can get started. If you ever get around that windows issue, please let me know.

Thank you again!

Joaquin

1 Like
32

Windows bug should be fixed by BUG: Fix bug on CTF + Windows by larsoner Ā· Pull Request #10866 Ā· mne-tools/mne-python Ā· GitHub

1 Like
33

Wow! Thank you so much @larsoner
Sorry for the dumb questionā€¦
Runing conda update mne should make it work? Or how can I get this fix?

34

@Joac The fix has not yet landed in a new MNE release, but this will hopefully happen sometime next week, for when weā€™re planning to release MNE-Python 1.1.

35

Right. Ok then. Thanks again!