All Epochs are dropped

sds · August 30, 2022, 2:39am

Hi,

I am trying to clean data before extracting PSD, so here is my process:

Events.
There are no events in the edf file (AFAIK), so I use make_fixed_length_events:

events = mne.make_fixed_length_events(edf, duration=1, overlap=0.5)

events.shape=(2301, 3).
Does this make sense?

Guess flat and reject for Epochs
(take the geometric mean of the absolute values of the signals and go up and down 2 orders of magnitude):

edf_df = edf.to_data_frame()   # 575,516 rows
mean_abs_signal = np.abs(edf_df[edf.ch_names]).mean(axis=1)
mean_signal_multiplicative = np.exp(np.log(mean_abs_signal).mean())
signal_lo = mean_signal_multiplicative / 100
signal_hi = mean_signal_multiplicative * 100

About 3.9% of mean_abs_signal is outside of [signal_lo,signal_hi].
Does this make sense?

Create Epochs(cf. example):

tmin, tmax = -1, 4
edf_e = mne.Epochs(edf, events=events, preload=True,
                   tmin=tmin-0.5, tmax=tmax+0.5,
                   flat=dict(eeg=signal_lo), reject=dict(eeg=signal_hi),
                   baseline=(None,None), detrend=0, event_repeated="merge")

this prints

Not setting metadata
2301 matching events found
Setting baseline interval to [-1.5, 4.5] sec
Applying baseline correction (mode: mean)
0 projection items activated
Using data from preloaded Raw for 2301 events and 3001 original time points ...
    Rejecting flat epoch based on EEG : ['EEG Fp1-Ref', 'EEG Fpz-Ref', 'EEG Fp2-Ref', 'EEG F7-Ref', 'EEG F3-Ref', 'EEG Fz-Ref', 'EEG F4-Ref', 'EEG F8-Ref', 'EEG FC5-Ref', 'EEG FC1-Ref', 'EEG FC2-Ref', 'EEG FC6-Ref', 'EEG AFz-Ref', 'EEG T7-Ref', 'EEG C3-Ref', 'EEG Cz-Ref', 'EEG C4-Ref', 'EEG T8-Ref', 'EEG FT10-Ref', 'EEG CP5-Ref', 'EEG CP1-Ref', 'EEG CP2-Ref', 'EEG CP6-Ref', 'EEG P7-Ref', 'EEG P3-Ref', 'EEG Pz-Ref', 'EEG P4-Ref', 'EEG P8-Ref', 'EEG FT9-Ref', 'EEG O1-Ref', 'EEG Oz-Ref', 'EEG O2-Ref']
    ...[repeated 2300 times]
2301 bad epochs dropped

and now len(edf_e)=0, i.e., all epochs are dropped.

Is my file really all junk, or am I doing something wrong?

MNE version 1.0.3 on linux Pop!_OS 22.04 LTS

richard · August 30, 2022, 6:10am

Hello, I don’t understand your algorithm for producing the thresholds. What are the actual values you eventually pass as flat and reject?

Richard

sds · August 30, 2022, 2:09pm

Hi Richard, thank you for your reply.

Basically, I take the geometric mean of the absolute values of the signals and go up and down 2 orders of magnitude from them and get

signal lo/hi 0.04071002572634587 407.1002572634587

however, less than 4% of all mean signals are outside of the range.

When is an epoch dropped? If all 32 channels are outside of [flat,reject] or if any one of them?
E.g., if one electrode is not reading, i.e., reporting 0 at all times, will all the data be dropped?

mscheltienne · August 30, 2022, 2:34pm

Hello,

Yes, one electrode is enough to drop an entire epoch.
Especially if you have flat channels, you should figure out what to do with them before creating your Epochs.

Best,
Mathieu

mscheltienne · August 30, 2022, 2:37pm

Also, MNE expects EEG channels to be in Volts. This is why the default rejection criterium for EEG signals is dict(eeg=40e-6) (40 uV). 407 seems way off.

richard · August 30, 2022, 5:36pm

Like @mscheltienne said, these values are super huge. Passing 0.0407 as a flat rejection threshold means that PTP amplitudes below 40’710 µV lead to a rejection. Which then drops all of your epochs.

sds · August 30, 2022, 6:53pm

Thank you very much!
What is a good value for reject?

richard · August 30, 2022, 7:47pm

It really depends on your data and the recording system used. We usually pick quite large values to only capture sudden “spikes” in the signal, which would e.g. throw off ICA. Something like {"eeg": 500e-6} might be a good start. Be sure to look at epochs.plot_drop_log() output to figure out if and which individual channels contribute to rejections.

For flatness, I don’t usually do rejection…

The function mne.preprocessing.annotate_amplitude() might also be of interest to you. It operates on continuous data.

Best wishes,
Richard

mscheltienne · August 30, 2022, 8:17pm

And start by making sure that your data is in Volts.

Good luck!
Mathieu

sds · August 30, 2022, 8:19pm

@mscheltienne
start by making sure that your data is in Volts

How do I do that?
I have EDF files, here is what I get for ranges:

                       min           max
EEG Fp1-Ref   -5621.935942   9556.586974
EEG Fpz-Ref     -31.511986   1105.116391
EEG Fp2-Ref   -3734.516769   5850.715130
EEG F7-Ref     -103.614827   3633.742509
EEG F3-Ref    -1366.590332  29506.644741
EEG Fz-Ref    -4409.100330  57961.055318
EEG F4-Ref     -291.883502   1555.115089
EEG F8-Ref      -41.857978   1467.947395
EEG FC5-Ref    -883.127590   2295.518290
EEG FC1-Ref   -5509.074345  63576.944484
EEG FC2-Ref   -8839.014019  64861.730126
EEG FC6-Ref     -53.954549   1892.170674
EEG AFz-Ref   -3790.333510   6754.135547
EEG T7-Ref    -5026.425018  43141.955605
EEG C3-Ref    -2406.418763  23625.017426
EEG Cz-Ref     -103.616449   3633.799417
EEG C4-Ref     -103.614268   3633.722913
EEG T8-Ref     -774.948545   2220.535092
EEG FT10-Ref   -241.338493   3633.691540
EEG CP5-Ref   -1816.158997  14375.464167
EEG CP1-Ref    -103.616449   3633.799417
EEG CP2-Ref    -908.463717     25.904508
EEG CP6-Ref    -103.613309   3633.689281
EEG P7-Ref   -32179.248081  66851.802491
EEG P3-Ref      -96.186833   3373.244882
EEG Pz-Ref   -32580.244836  68948.537765
EEG P4-Ref    -9309.892064  68265.363198
EEG P8-Ref    -6194.791124  65239.908152
EEG FT9-Ref    -103.616449   3633.799417
EEG O1-Ref      -25.904508    908.463717
EEG Oz-Ref    -3752.810013  68174.337781
EEG O2-Ref   -45956.985601  45642.196670

sds · August 31, 2022, 3:49am

@mscheltienne
one electrode is enough to drop an entire epoch.

This is NFG for me - only 3,276 out of 575,516 (0.57%) time slices have all channels over 40e-6 recommended by you.

Will I have to call Epochs 32 times - separately for each channel?!

sds · August 31, 2022, 4:10am

@richard: reject=500e-6
@mscheltienne: flat=40e-6

with these numbers, I get very very very little usable data:

              <4e-05  >0.0005   good
EEG Fp1-Ref   555294    16632   3590
EEG Fpz-Ref   571807     2300   1409
EEG Fp2-Ref   524612    47544   3360
EEG F7-Ref    568287     2857   4372
EEG F3-Ref    560018    11024   4474
EEG Fz-Ref    547380    19884   8252
EEG F4-Ref    568060     5887   1569
EEG F8-Ref    571369     2332   1815
EEG FC5-Ref   424891   143873   6752
EEG FC1-Ref   564781     6271   4464
EEG FC2-Ref   564908     6226   4382
EEG FC6-Ref   570800     2361   2355
EEG AFz-Ref        2   575511      3
EEG T7-Ref    469466    91620  14430
EEG C3-Ref    530359    39957   5200
EEG Cz-Ref    568287     2857   4372
EEG C4-Ref    568287     2857   4372
EEG T8-Ref    444103   128340   3073
EEG FT10-Ref  564789     6229   4498
EEG CP5-Ref   451820   105548  18148
EEG CP1-Ref   568287     2857   4372
EEG CP2-Ref   572015     2275   1226
EEG CP6-Ref   568287     2857   4372
EEG P7-Ref    422204   126887  26425
EEG P3-Ref    568668     2819   4029
EEG Pz-Ref    493439    42241  39836
EEG P4-Ref    561129     9501   4886
EEG P8-Ref    550456    19721   5339
EEG FT9-Ref   568287     2857   4372
EEG O1-Ref    572015     2275   1226
EEG Oz-Ref    563776     6086   5654
EEG O2-Ref    541944    27344   6228

is this normal?!

richard · August 31, 2022, 4:57am

I have absolutely no clue what you are doing there, sorry.

You should use epochs.plot_drop_log() to get an idea of which channels are outside of the specified range.

mscheltienne · August 31, 2022, 9:33am

instead of looking only at the minimum / maximum, I would have plot one or 2 entire channels to see which kind of range the signal is spanning.

That said, let’s take your first Fp1 channel:

                       min           max
EEG Fp1-Ref   -5621.935942   9556.586974

That’s obviously not Volts, 9k Volts from a human brain would be quite something. You need to figure out how those files were generated and in what units the data was saved.

mscheltienne · August 31, 2022, 9:34am

And if you have reject=dict(eeg=40e-6) then it means that any epoch with at least one channel that has a peak-to-peak amplitude that exceeds 40e-6 will be dropped.

richard · August 31, 2022, 9:59am

richard · August 31, 2022, 3:14pm

@sds is passing this value to flat, not to reject.

mscheltienne · August 31, 2022, 3:37pm

It’s the default for reject…

reject = dict(grad=4000e-13,  # unit: T / m (gradiometers)
              mag=4e-12,      # unit: T (magnetometers)
              eeg=40e-6,      # unit: V (EEG channels)
              eog=250e-6      # unit: V (EOG channels)
              )

richard · August 31, 2022, 3:58pm

yes but they’re passing 500e-6 for reject

sds · August 31, 2022, 5:34pm

Indeed the example (not the default!) reject for Epochs is eeg=40e-6 - what @mscheltienne recommended. (I misunderstood his comment - I thought he recommended it as flat).
OTOH, @richard recommends eeg=500e-6 for reject.
I can try either. Thank you very much.

However, I still have no idea what the “correct” value for flat is.
Please don’t tell me “it depends”. Just give me something to start with

Thank you again!

Topic		Replies	Views
Calculating connectivity out of EEG signal segments 💬 Support & Discussions eeg , connectivity , epochs	1	148	September 10, 2023
Found array with dim 3. Estimator expected <= 2. 💬 Support & Discussions epochs	6	172	July 22, 2022
Alpha/Beta ratio (PSD) for single epochs EEG for specific electrodes 💬 Support & Discussions eeg , epochs	10	1515	December 19, 2021
non uniform epochs 💬 Support & Discussions preprocessing , eeg	2	128	March 28, 2023
How to remove large brain waves in EDF data. 💬 Support & Discussions preprocessing , eeg	4	127	September 19, 2023

All Epochs are dropped

Related Topics