Can MNE automate the following workflow?

Hi,

I had a conversation with an EEG Expert (who, alas, uses Matlab, so he cannot help me with code), and he recommended the following pipeline:

  1. Process every channel completely separately, so, for each channel:
  2. Remove flatlines (this is likely to involve killing segments with values ±32767 which seem to be the EEG range for ANT NEURO EDF files; the data is in uV).
  3. De-trend each non-flat segment separately (subtract the regression line from each value)
  4. Remove artifacts: if there is a spike above 500uV, cut off everything around the spike above 200uV and glue together to be continuous; no new detrending
  5. Glue all detrended segments together to get a continuous line
  6. Apply a filter with lowpass=0.2Hz, highpass=55Hz
  7. PSD (no epoching)

Now, I have 2 questions:

  1. Is there anything obviously wrong in the pipeline?
  2. Does MNE have some functionality to support this pipeline with little coding on my part?

Thank you very much!


Continuous gluing probably requires an explanation:
If I have 2 vectors: [1,2,3] and [5,6,7], gluing them continuously together produces vector [1,2,3,4,5].
I.e., I assume that the last value of the 1st vector and the 1st value of the 2nd vector are the same point.


MNE version 1.0.3 on linux Pop!_OS 22.04 LTS

Hello again.

A magic pipeline that magically works on your data and that doesn’t require you to do anything does not exist. Let’s make this clear. As in your last post, I will write again: look at your data. Look at what artifact is present and needs to be removed before extracting features.

For flat and high PTP amplitude channels, you can use mne.preprocessing.annotate_amplitude — MNE 1.1.1 documentation This function creates annotations which are time-segments with a certain text, a note, added to them. For instance 'BAD' can be used to mark time-segments that have too much noise/that you want to ignore in the rest of the processing. Most MNE functions which have a reject_by_annotation argument will interpret all annotations starting with BAD as segments to be ignored, thus doing this ‘cutting’ and ‘gluing’ of raw data by themselves.
You can also do this annotation process manually through the browser, simply plot the data with raw.plot, and start dragging and marking the segments to ignore by hand.

For filter, you can use mne.io.Raw — MNE 1.1.1 documentation which uses mne.filter.filter_data — MNE 1.1.1 documentation behind.
For PSD, you can use mne.time_frequency.psd_welch — MNE 1.1.1 documentation or mne.time_frequency.psd_multitaper — MNE 1.1.1 documentation

Note that all those links can be obtained by typing the keyword, e.g. filter or psd in the research bar of the website. You will find associated tutorials at the end of each function description to help you get started. But again, there is no magic, you will have to code your pipeline.

And finally, a comment on the pipeline itself, as I am also a heavy user of ANT devices, I find it very surprising that you have data with some flat segments inside. What can happen is that one electrode died on the cap, but then it is dead for the entire recording. I would simply mark it as bad (by adding it to raw.info["bads"] or interactively on the data browser (raw.plot) by clicking on the flat trace of the electrode) and interpolate it if need be.

You can also compare your pipeline to the introductory tutorials: Tutorials — MNE 1.1.1 documentation

Good luck,
Mathieu

2 Likes

Thanks for the references (I knew of filter and welch, of course - I asked about them in my 1st post :wink:
annotate_amplitude is interesting, thanks!
The missing link is gluing and detrending: while I can do that on a pandas DataFrame myself, I don’t see how I can insert the results back into RAW.

Yes, I understand.
I have 140 files and I cannot manually process all of them (for two reasons: time and objectivity).
I am looking at 1 and will be following the pipeline above and automating it and then applying it to the rest of data mechanically.

You can create a Raw instance with a numpy array and an Info. For the Info, you can re-use the one from the loaded instance, stored in the attribute of the same name raw.info. As I don’t have it, I’m gonna create a fake one in the example below with fake data:

import numpy as np
from mne import create_info
from mne.io import RawArray

# create fake data as 3 sin sources measured across 6 channels
times = np.linspace(0, 5, 2000)
signals = np.array([np.sin(2 * np.pi * k * times) for k in (7, 22, 37)])
coeffs = np.random.rand(6, 3)
data = np.dot(coeffs, signals) + np.random.normal(
    0, 0.1, (coeffs.shape[0], times.size)
)
info = create_info(
    ["Fpz", "Cz", "CPz", "Oz", "M1", "M2"], sfreq=400, ch_types="eeg"
)
raw = RawArray(data, info)

See more information here Creating MNE-Python data structures from scratch — MNE 1.2.dev0 documentation

And for the glueing, it really looks like annotation of bad segments to me. See Rejecting bad data spans and breaks — MNE 1.1.1 documentation and Annotating continuous data — MNE 1.2.dev0 documentation for more information.

2 Likes