Basic visual inspection of multiple files from the same experiment

đź’¬ Support & Discussions
1

Hey all!

I am trying to get back into MNE for EEG analysis, but am running into a rather basic problem. I have a dataset loaded in a list, and would like to visually inspect each of the epoched files to mark bad channels and epochs.

However, I cannot find a way to neatly loop through the data files one by one. If I do the built-in epochs.plot() functionality, all the data is plotted simultaneously (on separate figures) and it is very hard to keep track of what I am looking at (not to mention it gets super heavy for my computer!).

What is the recommended syntax for achieving the following?

  1. plot the epoched data for visual inspection one at a time,
  2. allow for interactive inspection and store the bad epochs and channels (without dropping yet) in the epoch object
  3. proceed to the next epochs object
  4. repeat until all epoch objects have been inspected

This is a recurring pattern, and I assume it is not only me who need this. And for some of my future analyses, repeating the code over and over for each object is just not feasible.

Hoping there is a way to achieve it, but I am also open for alternative ways of going about the visual inspection:)

Thanks!
Bjørn

Example of code I am using now:

for epoch in epoched_data:
    epoch.plot(n_epochs=5)

I’ve tried using the block=True feature, to no avail (the script just completely stops).

MNE version 1.0.1
Operating system: Windows 11
Running python with anaconda and spyder

2

Hi @bjorneju and welcome to the forum!

I believe you could just change the plotting command to:

epoch.plot(n_epochs=5, block=True)

This should halt script execution until after the plotting window has been closed.

Problem is that I believe that epochs marked as bad will be dropped immediately when closing the window. So you may want to operate on a copy of the epochs:

epoch.copy().plot(n_epochs=5, block=True)

But then there is no easy way to keep track of the dropped / bad epochs: You only get a log message.

For now, I suppose you’ll have to do ugly things like:

epochs_copy = epoch.copy()
epochs_copy.plot(n_epochs=5, block=True)
dropped_idx = [i for i, reason in enumerate(epochs_copy.drop_log) if reason != ()]

for each set of epochs. This is really, really not great, but I don’t know of a better way for now.

This is a shortcoming of MNE-Python we should address. Tagging @drammock, @cbrnr, and @agramfort here for their input. Would it be possible to allow for returning selections of bad channels and epochs without dropping them when using block=True? For example via a new parameter, drop_on_close=False.

Edit Then again, the problem’s not bad channels, right? Because they can be marked/unmarked as bad without dropping any data. But the issue really is that once an Epoch has been marked as bad, it will be irreversibly removed from the data. Can we not just … keep them and have them tagged as bad? Like we do with bad channels? So users can inspect and potentially revise their decision later on.

Best wishes,
Richard

3

I think dropping epochs in our browser should work like annotating bad segments. I agree that immediately dropping is probably not a good idea in many cases, and I’d also prefer something non-destructive (like annotations).

1 Like
4

@cbrnr I just added an “Edit” section to my previous post. I think we mostly agree

5

Thanks a lot for the fast response!
The block=True argument seems to work fine (don’t know why it crashed/halted my script last time i tried).

In any case, yes, I agree that it would be great to have the opportunity to revise the selection of bad epochs just like channels. If nothing else, in order to maintain the integrity of the data for as long as possible.

Will report back if I run into issues.

Bjørn

EDIT quick follow-up:

it is a bit unstable, unfortunately. Sometimes, the kernel just freezes. There seems to be open issues relating to the same functionailty, and it seems it is matplotlib-related (probably not pure MNE). See here for example: raw.plot(block=True) freezes the jupyter kernel forever · Issue #6528 · mne-tools/mne-python · GitHub

Anyways, for me, the kernel freezes some times, and leaves the output like this:

test.plot(n_epochs=10, n_channels=21, block=True)
Opening epochs-browser...
Closing epochs-browser...
Dropped 1 epoch: 5
The following epochs were marked as bad and are dropped:
[5]
Channels marked as bad:
['O2']

In other words, the mne-part seems to work to select bad epochs and channels, but closing the figure leads to kernel freezing.

it is still unclear whether the solution proposed by mvonflotow solves the issue. But initial tests seem promising.

EDIT 2:
it seems the solution only transiently solves the problem. On the first 2-3 calls to epochs.plot(block=True), the behavior is as expected, but on subsequent calls the kernel still freezes. I cannot even ctrl+c to stop the process, and I’m forced to restart the kernel.

Any insights would be much appreciated!

EDIT 3:
After testing some more, I have made the following observations (see below for minimal code used):

  • The first time I run through the code below, it behaves as expected
  • If i try to run the little loop with epochs.plot() call, the kernel freezes after the first iteration
  • However, I also notice that when I kill the kernel to try again, the mne “epochs browser” briefly shows up, so it seems like the next iteration’s call to “epochs.plot()” is somehow cached and just doesnt show up.
  • If I run epoch.plot() (without the block=True kwarg) the kernel does not freeze after closing the browser.

In other words, it seems like I can safely use the looping strategy as long as I only run one loop, and do not call the epochs.plot(block=True) again later in the script

from oscillation_functions import *
import pickle as pkl

save_folder = r'\\hypatia.uio.no\lh-med-imb-jstormlab\Data\Wada_Data_Swiss\etomidate_both_sides\derivatives\pipeline_2'
file_names = [save_folder+'\\'+f for f in ['sub-01_left_ica-cleaned.pkl','sub-01_right_ica-cleaned.pkl']]
# load sample data

epochs = []
for file in file_names:
    with open(file, 'rb') as f:
        epochs.append(pkl.load(f)['data'])
            
# inspect data
inspected = []
for epoch in epochs:
    aux = epoch.copy()
    aux.plot(n_epochs=10, n_channels=21, block=True)
    inspected.append(aux)
1 Like
6

These issues are one of the reasons why I usually recommend to work with the plain Python interpreter and not a Jupyter kernel. Can you try if it works with e.g. regular IPython?

7

I never had issues like this with IPython, Jupyter, or for example VS Code Interactive window. I believe it’s mainly Spyder which is at fault here… colleagues of mine who switched from Spyder to VS Code also don’t see these problems anymore.

8

Thanks again for the followup, folks!

I tried running the same test code in a jupyter lab instance (through my anaconda) and the same thing happens.

I then tried running the same code within my terminal, and it works fine. That is, the MNE browser opens and closes as expected on both the first and second run-through of the loop with the call to epoch.plot(block=True)

Code:
First run with this (works for all my IDE’s)

from oscillation_functions import *
import pickle as pkl
import matplotlib.pyplot as plt

save_folder = r'\\hypatia.uio.no\lh-med-imb-jstormlab\Data\Wada_Data_Swiss\etomidate_both_sides\derivatives\pipeline_2'
file_names = [save_folder+'\\'+f for f in ['sub-01_left_ica-cleaned.pkl','sub-01_right_ica-cleaned.pkl']]
# load sample data

epochs = []
for file in file_names:
    with open(file, 'rb') as f:
        epochs.append(pkl.load(f)['data'])
        
# inspect data
inspected = []
for epoch in epochs:
    aux = epoch.copy()
    aux.plot(n_epochs=10, n_channels=21, block=True)
    inspected.append(aux)

Second run with this (only works as expected when running inside my terminal)

# inspect data
inspected = []
for epoch in epochs:
    aux = epoch.copy()
    aux.plot(n_epochs=10, n_channels=21, block=True)
    inspected.append(aux)

Do you have a recommended IDE to use with MNE?

Thanks,
Bjørn

1 Like
9

I believe many of the developers and experienced users now use VS Code, but its interactive portion also makes use of Jupyter under the hood, so if it’s Jupyter causing your problem, this won’t fix things for you either :frowning:

10

I use Visual Studio Code without the interactive window.

1 Like
11

VSCode can easily be used without the jupyter kernel. One can open the terminal pane, type python or ipython, and then use keyboard shortcuts to send lines / selections / cells to the (ipython) terminal.

12

Yes, that’s what I do. There are some quirks to that though, but all in all it’s an OK experience.