All Epochs are dropped


I am trying to clean data before extracting PSD, so here is my process:

  1. Events.
    There are no events in the edf file (AFAIK), so I use make_fixed_length_events:
events = mne.make_fixed_length_events(edf, duration=1, overlap=0.5)

events.shape=(2301, 3).
Does this make sense?

  1. Guess flat and reject for Epochs
    (take the geometric mean of the absolute values of the signals and go up and down 2 orders of magnitude):
edf_df = edf.to_data_frame()   # 575,516 rows
mean_abs_signal = np.abs(edf_df[edf.ch_names]).mean(axis=1)
mean_signal_multiplicative = np.exp(np.log(mean_abs_signal).mean())
signal_lo = mean_signal_multiplicative / 100
signal_hi = mean_signal_multiplicative * 100

About 3.9% of mean_abs_signal is outside of [signal_lo,signal_hi].
Does this make sense?

  1. Create Epochs(cf. example):
tmin, tmax = -1, 4
edf_e = mne.Epochs(edf, events=events, preload=True,
                   tmin=tmin-0.5, tmax=tmax+0.5,
                   flat=dict(eeg=signal_lo), reject=dict(eeg=signal_hi),
                   baseline=(None,None), detrend=0, event_repeated="merge")

this prints

Not setting metadata
2301 matching events found
Setting baseline interval to [-1.5, 4.5] sec
Applying baseline correction (mode: mean)
0 projection items activated
Using data from preloaded Raw for 2301 events and 3001 original time points ...
    Rejecting flat epoch based on EEG : ['EEG Fp1-Ref', 'EEG Fpz-Ref', 'EEG Fp2-Ref', 'EEG F7-Ref', 'EEG F3-Ref', 'EEG Fz-Ref', 'EEG F4-Ref', 'EEG F8-Ref', 'EEG FC5-Ref', 'EEG FC1-Ref', 'EEG FC2-Ref', 'EEG FC6-Ref', 'EEG AFz-Ref', 'EEG T7-Ref', 'EEG C3-Ref', 'EEG Cz-Ref', 'EEG C4-Ref', 'EEG T8-Ref', 'EEG FT10-Ref', 'EEG CP5-Ref', 'EEG CP1-Ref', 'EEG CP2-Ref', 'EEG CP6-Ref', 'EEG P7-Ref', 'EEG P3-Ref', 'EEG Pz-Ref', 'EEG P4-Ref', 'EEG P8-Ref', 'EEG FT9-Ref', 'EEG O1-Ref', 'EEG Oz-Ref', 'EEG O2-Ref']
    ...[repeated 2300 times]
2301 bad epochs dropped

and now len(edf_e)=0, i.e., all epochs are dropped.

Is my file really all junk, or am I doing something wrong?

MNE version 1.0.3 on linux Pop!_OS 22.04 LTS

Hello, I don’t understand your algorithm for producing the thresholds. What are the actual values you eventually pass as flat and reject?


Hi Richard, thank you for your reply.

Basically, I take the geometric mean of the absolute values of the signals and go up and down 2 orders of magnitude from them and get

signal lo/hi 0.04071002572634587 407.1002572634587

however, less than 4% of all mean signals are outside of the range.

When is an epoch dropped? If all 32 channels are outside of [flat,reject] or if any one of them?
E.g., if one electrode is not reading, i.e., reporting 0 at all times, will all the data be dropped?


Yes, one electrode is enough to drop an entire epoch.
Especially if you have flat channels, you should figure out what to do with them before creating your Epochs.


1 Like

Also, MNE expects EEG channels to be in Volts. This is why the default rejection criterium for EEG signals is dict(eeg=40e-6) (40 uV). 407 seems way off.

1 Like

Like @mscheltienne said, these values are super huge. Passing 0.0407 as a flat rejection threshold means that PTP amplitudes below 40’710 µV lead to a rejection. Which then drops all of your epochs.

Thank you very much!
What is a good value for reject?

It really depends on your data and the recording system used. We usually pick quite large values to only capture sudden “spikes” in the signal, which would e.g. throw off ICA. Something like {"eeg": 500e-6} might be a good start. Be sure to look at epochs.plot_drop_log() output to figure out if and which individual channels contribute to rejections.

For flatness, I don’t usually do rejection…

The function mne.preprocessing.annotate_amplitude() might also be of interest to you. It operates on continuous data.

Best wishes,

And start by making sure that your data is in Volts.

Good luck!

start by making sure that your data is in Volts

How do I do that?
I have EDF files, here is what I get for ranges:

                       min           max
EEG Fp1-Ref   -5621.935942   9556.586974
EEG Fpz-Ref     -31.511986   1105.116391
EEG Fp2-Ref   -3734.516769   5850.715130
EEG F7-Ref     -103.614827   3633.742509
EEG F3-Ref    -1366.590332  29506.644741
EEG Fz-Ref    -4409.100330  57961.055318
EEG F4-Ref     -291.883502   1555.115089
EEG F8-Ref      -41.857978   1467.947395
EEG FC5-Ref    -883.127590   2295.518290
EEG FC1-Ref   -5509.074345  63576.944484
EEG FC2-Ref   -8839.014019  64861.730126
EEG FC6-Ref     -53.954549   1892.170674
EEG AFz-Ref   -3790.333510   6754.135547
EEG T7-Ref    -5026.425018  43141.955605
EEG C3-Ref    -2406.418763  23625.017426
EEG Cz-Ref     -103.616449   3633.799417
EEG C4-Ref     -103.614268   3633.722913
EEG T8-Ref     -774.948545   2220.535092
EEG FT10-Ref   -241.338493   3633.691540
EEG CP5-Ref   -1816.158997  14375.464167
EEG CP1-Ref    -103.616449   3633.799417
EEG CP2-Ref    -908.463717     25.904508
EEG CP6-Ref    -103.613309   3633.689281
EEG P7-Ref   -32179.248081  66851.802491
EEG P3-Ref      -96.186833   3373.244882
EEG Pz-Ref   -32580.244836  68948.537765
EEG P4-Ref    -9309.892064  68265.363198
EEG P8-Ref    -6194.791124  65239.908152
EEG FT9-Ref    -103.616449   3633.799417
EEG O1-Ref      -25.904508    908.463717
EEG Oz-Ref    -3752.810013  68174.337781
EEG O2-Ref   -45956.985601  45642.196670

one electrode is enough to drop an entire epoch.

This is NFG for me - only 3,276 out of 575,516 (0.57%) time slices have all channels over 40e-6 recommended by you.

Will I have to call Epochs 32 times - separately for each channel?!

@richard: reject=500e-6
@mscheltienne: flat=40e-6

with these numbers, I get very very very little usable data:

              <4e-05  >0.0005   good
EEG Fp1-Ref   555294    16632   3590
EEG Fpz-Ref   571807     2300   1409
EEG Fp2-Ref   524612    47544   3360
EEG F7-Ref    568287     2857   4372
EEG F3-Ref    560018    11024   4474
EEG Fz-Ref    547380    19884   8252
EEG F4-Ref    568060     5887   1569
EEG F8-Ref    571369     2332   1815
EEG FC5-Ref   424891   143873   6752
EEG FC1-Ref   564781     6271   4464
EEG FC2-Ref   564908     6226   4382
EEG FC6-Ref   570800     2361   2355
EEG AFz-Ref        2   575511      3
EEG T7-Ref    469466    91620  14430
EEG C3-Ref    530359    39957   5200
EEG Cz-Ref    568287     2857   4372
EEG C4-Ref    568287     2857   4372
EEG T8-Ref    444103   128340   3073
EEG FT10-Ref  564789     6229   4498
EEG CP5-Ref   451820   105548  18148
EEG CP1-Ref   568287     2857   4372
EEG CP2-Ref   572015     2275   1226
EEG CP6-Ref   568287     2857   4372
EEG P7-Ref    422204   126887  26425
EEG P3-Ref    568668     2819   4029
EEG Pz-Ref    493439    42241  39836
EEG P4-Ref    561129     9501   4886
EEG P8-Ref    550456    19721   5339
EEG FT9-Ref   568287     2857   4372
EEG O1-Ref    572015     2275   1226
EEG Oz-Ref    563776     6086   5654
EEG O2-Ref    541944    27344   6228

is this normal?!

I have absolutely no clue what you are doing there, sorry. :dizzy_face:

You should use epochs.plot_drop_log() to get an idea of which channels are outside of the specified range.

instead of looking only at the minimum / maximum, I would have plot one or 2 entire channels to see which kind of range the signal is spanning.

That said, let’s take your first Fp1 channel:

                       min           max
EEG Fp1-Ref   -5621.935942   9556.586974

That’s obviously not Volts, 9k Volts from a human brain would be quite something. You need to figure out how those files were generated and in what units the data was saved.

And if you have reject=dict(eeg=40e-6) then it means that any epoch with at least one channel that has a peak-to-peak amplitude that exceeds 40e-6 will be dropped.


1 Like

@sds is passing this value to flat, not to reject.

It’s the default for reject…

reject = dict(grad=4000e-13,  # unit: T / m (gradiometers)
              mag=4e-12,      # unit: T (magnetometers)
              eeg=40e-6,      # unit: V (EEG channels)
              eog=250e-6      # unit: V (EOG channels)

yes but they’re passing 500e-6 for reject :wink:

Indeed the example (not the default!) reject for Epochs is eeg=40e-6 - what @mscheltienne recommended. (I misunderstood his comment - I thought he recommended it as flat).
OTOH, @richard recommends eeg=500e-6 for reject.
I can try either. Thank you very much.

However, I still have no idea what the “correct” value for flat is.
Please don’t tell me “it depends”. Just give me something to start with :wink:

Thank you again!