# All Epochs are dropped **URL:** \ **Category:** Support & Discussions\ **Tags:** eeg, epochs\ **Created:** [August 30, 2022, 2:39am UTC](https://mne.discourse.group/t/all-epochs-are-dropped/5486 "2022-08-30T02:39:07Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![sds](https://yyz2.discourse-cdn.com/free1/user_avatar/mne.discourse.group/sds/32/1810_2.png) [@sds](https://mne.discourse.group/u/sds)\ **Post date:** [August 30, 2022, 2:39am UTC](https://mne.discourse.group/t/all-epochs-are-dropped/5486/1 "2022-08-30T02:39:07Z")
Hi, I am trying to _clean_ data before extracting PSD, so here is my process: 1. [Events](https://mne.tools/stable/glossary.html#term-events). There are no events in the `edf` file (AFAIK), so I use [`make_fixed_length_events`](https://mne.tools/stable/generated/mne.make_fixed_length_events.html): ```python events = mne.make_fixed_length_events(edf, duration=1, overlap=0.5) ``` `events.shape=(2301, 3)`. Does this make sense? 1. Guess `flat` and `reject` for [`Epochs`](https://mne.tools/stable/generated/mne.Epochs.html) (take the geometric mean of the absolute values of the signals and go up and down 2 orders of magnitude): ```python edf_df = edf.to_data_frame() # 575,516 rows mean_abs_signal = np.abs(edf_df[edf.ch_names]).mean(axis=1) mean_signal_multiplicative = np.exp(np.log(mean_abs_signal).mean()) signal_lo = mean_signal_multiplicative / 100 signal_hi = mean_signal_multiplicative * 100 ``` About 3.9% of `mean_abs_signal` is outside of `[signal_lo,signal_hi]`. Does this make sense? 1. Create `Epochs`(cf. [example](https://mne.tools/stable/auto_examples/time_frequency/time_frequency_erds.html)): ```python tmin, tmax = -1, 4 edf_e = mne.Epochs(edf, events=events, preload=True, tmin=tmin-0.5, tmax=tmax+0.5, flat=dict(eeg=signal_lo), reject=dict(eeg=signal_hi), baseline=(None,None), detrend=0, event_repeated="merge") ``` this prints ```nohighlight Not setting metadata 2301 matching events found Setting baseline interval to [-1.5, 4.5] sec Applying baseline correction (mode: mean) 0 projection items activated Using data from preloaded Raw for 2301 events and 3001 original time points ... Rejecting flat epoch based on EEG : ['EEG Fp1-Ref', 'EEG Fpz-Ref', 'EEG Fp2-Ref', 'EEG F7-Ref', 'EEG F3-Ref', 'EEG Fz-Ref', 'EEG F4-Ref', 'EEG F8-Ref', 'EEG FC5-Ref', 'EEG FC1-Ref', 'EEG FC2-Ref', 'EEG FC6-Ref', 'EEG AFz-Ref', 'EEG T7-Ref', 'EEG C3-Ref', 'EEG Cz-Ref', 'EEG C4-Ref', 'EEG T8-Ref', 'EEG FT10-Ref', 'EEG CP5-Ref', 'EEG CP1-Ref', 'EEG CP2-Ref', 'EEG CP6-Ref', 'EEG P7-Ref', 'EEG P3-Ref', 'EEG Pz-Ref', 'EEG P4-Ref', 'EEG P8-Ref', 'EEG FT9-Ref', 'EEG O1-Ref', 'EEG Oz-Ref', 'EEG O2-Ref'] ...[repeated 2300 times] 2301 bad epochs dropped ``` and now `len(edf_e)=0`, i.e., all epochs are dropped. **Is my file really all junk, or am I doing something wrong?** MNE version 1.0.3 on linux Pop!\_OS 22.04 LTS ---
**Author:** ![richard](https://yyz2.discourse-cdn.com/free1/user_avatar/mne.discourse.group/richard/32/15_2.png) [@richard](https://mne.discourse.group/u/richard)\ **Post date:** [August 30, 2022, 6:10am UTC](https://mne.discourse.group/t/all-epochs-are-dropped/5486/2 "2022-08-30T06:10:06Z")
Hello, I don’t understand your algorithm for producing the thresholds. What are the actual values you eventually pass as `flat` and `reject`? Richard ---
**Author:** ![sds](https://yyz2.discourse-cdn.com/free1/user_avatar/mne.discourse.group/sds/32/1810_2.png) [@sds](https://mne.discourse.group/u/sds)\ **Post date:** [August 30, 2022, 2:09pm UTC](https://mne.discourse.group/t/all-epochs-are-dropped/5486/3 "2022-08-30T14:09:31Z")
Hi Richard, thank you for your reply. Basically, I take the geometric mean of the absolute values of the signals and go up and down 2 orders of magnitude from them and get ```python signal lo/hi 0.04071002572634587 407.1002572634587 ``` however, less than **4%** of all mean signals are outside of the range. When is an epoch dropped? If _all_ 32 channels are outside of `[flat,reject]` or if _any one_ of them? E.g., if one electrode is not reading, i.e., reporting 0 at _all_ times, will _all_ the data be dropped? ---
**Author:** ![mscheltienne](https://yyz2.discourse-cdn.com/free1/user_avatar/mne.discourse.group/mscheltienne/32/827_2.png) [@mscheltienne](https://mne.discourse.group/u/mscheltienne)\ **Post date:** [August 30, 2022, 2:34pm UTC](https://mne.discourse.group/t/all-epochs-are-dropped/5486/4 "2022-08-30T14:34:19Z")
Hello, Yes, one electrode is enough to drop an entire epoch. Especially if you have flat channels, you should figure out what to do with them before creating your Epochs. Best, Mathieu ---
**Author:** ![mscheltienne](https://yyz2.discourse-cdn.com/free1/user_avatar/mne.discourse.group/mscheltienne/32/827_2.png) [@mscheltienne](https://mne.discourse.group/u/mscheltienne)\ **Post date:** [August 30, 2022, 2:37pm UTC](https://mne.discourse.group/t/all-epochs-are-dropped/5486/5 "2022-08-30T14:37:58Z")
Also, MNE expects EEG channels to be in Volts. This is why the default rejection criterium for EEG signals is `dict(eeg=40e-6)` (40 uV). 407 seems way off. ---
**Author:** ![richard](https://yyz2.discourse-cdn.com/free1/user_avatar/mne.discourse.group/richard/32/15_2.png) [@richard](https://mne.discourse.group/u/richard)\ **Post date:** [August 30, 2022, 5:36pm UTC](https://mne.discourse.group/t/all-epochs-are-dropped/5486/6 "2022-08-30T17:36:03Z")
> [@sds](#): > > ```python > signal lo/hi 0.04071002572634587 407.1002572634587 > > ``` Like @mscheltienne said, these values are super huge. Passing `0.0407` as a `flat` rejection threshold means that PTP amplitudes below 40’710 µV lead to a rejection. Which then drops all of your epochs. ---
**Author:** ![sds](https://yyz2.discourse-cdn.com/free1/user_avatar/mne.discourse.group/sds/32/1810_2.png) [@sds](https://mne.discourse.group/u/sds)\ **Post date:** [August 30, 2022, 6:53pm UTC](https://mne.discourse.group/t/all-epochs-are-dropped/5486/7 "2022-08-30T18:53:41Z")
Thank you very much! What is a good value for reject? ---
**Author:** ![richard](https://yyz2.discourse-cdn.com/free1/user_avatar/mne.discourse.group/richard/32/15_2.png) [@richard](https://mne.discourse.group/u/richard)\ **Post date:** [August 30, 2022, 7:47pm UTC](https://mne.discourse.group/t/all-epochs-are-dropped/5486/8 "2022-08-30T19:47:34Z")
> [@sds](#): > > What is a good value for reject? It really depends on your data and the recording system used. We usually pick quite large values to only capture sudden “spikes” in the signal, which would e.g. throw off ICA. Something like `{"eeg": 500e-6}` might be a good start. Be sure to look at `epochs.plot_drop_log()` output to figure out if and which individual channels contribute to rejections. For flatness, I don’t usually do rejection… The function [`mne.preprocessing.annotate_amplitude()`](https://mne.tools/stable/generated/mne.preprocessing.annotate_amplitude.html) might also be of interest to you. It operates on continuous data. Best wishes, Richard ---
**Author:** ![mscheltienne](https://yyz2.discourse-cdn.com/free1/user_avatar/mne.discourse.group/mscheltienne/32/827_2.png) [@mscheltienne](https://mne.discourse.group/u/mscheltienne)\ **Post date:** [August 30, 2022, 8:17pm UTC](https://mne.discourse.group/t/all-epochs-are-dropped/5486/9 "2022-08-30T20:17:32Z")
And start by making sure that your data is in Volts. Good luck! Mathieu ---
**Author:** ![sds](https://yyz2.discourse-cdn.com/free1/user_avatar/mne.discourse.group/sds/32/1810_2.png) [@sds](https://mne.discourse.group/u/sds)\ **Post date:** [August 30, 2022, 8:19pm UTC](https://mne.discourse.group/t/all-epochs-are-dropped/5486/10 "2022-08-30T20:19:20Z")
> @mscheltienne > start by making sure that your data is in Volts How do I do that? I have EDF files, here is what I get for ranges: ```python min max EEG Fp1-Ref -5621.935942 9556.586974 EEG Fpz-Ref -31.511986 1105.116391 EEG Fp2-Ref -3734.516769 5850.715130 EEG F7-Ref -103.614827 3633.742509 EEG F3-Ref -1366.590332 29506.644741 EEG Fz-Ref -4409.100330 57961.055318 EEG F4-Ref -291.883502 1555.115089 EEG F8-Ref -41.857978 1467.947395 EEG FC5-Ref -883.127590 2295.518290 EEG FC1-Ref -5509.074345 63576.944484 EEG FC2-Ref -8839.014019 64861.730126 EEG FC6-Ref -53.954549 1892.170674 EEG AFz-Ref -3790.333510 6754.135547 EEG T7-Ref -5026.425018 43141.955605 EEG C3-Ref -2406.418763 23625.017426 EEG Cz-Ref -103.616449 3633.799417 EEG C4-Ref -103.614268 3633.722913 EEG T8-Ref -774.948545 2220.535092 EEG FT10-Ref -241.338493 3633.691540 EEG CP5-Ref -1816.158997 14375.464167 EEG CP1-Ref -103.616449 3633.799417 EEG CP2-Ref -908.463717 25.904508 EEG CP6-Ref -103.613309 3633.689281 EEG P7-Ref -32179.248081 66851.802491 EEG P3-Ref -96.186833 3373.244882 EEG Pz-Ref -32580.244836 68948.537765 EEG P4-Ref -9309.892064 68265.363198 EEG P8-Ref -6194.791124 65239.908152 EEG FT9-Ref -103.616449 3633.799417 EEG O1-Ref -25.904508 908.463717 EEG Oz-Ref -3752.810013 68174.337781 EEG O2-Ref -45956.985601 45642.196670 ``` ---
**Author:** ![sds](https://yyz2.discourse-cdn.com/free1/user_avatar/mne.discourse.group/sds/32/1810_2.png) [@sds](https://mne.discourse.group/u/sds)\ **Post date:** [August 31, 2022, 3:49am UTC](https://mne.discourse.group/t/all-epochs-are-dropped/5486/11 "2022-08-31T03:49:41Z")
> @mscheltienne > one electrode is enough to drop an entire epoch. This is NFG for me - only 3,276 out of 575,516 (0.57%) time slices have _all_ channels over 40e-6 recommended by you. Will I have to call `Epochs` 32 times - separately for each channel?! ---
**Author:** ![sds](https://yyz2.discourse-cdn.com/free1/user_avatar/mne.discourse.group/sds/32/1810_2.png) [@sds](https://mne.discourse.group/u/sds)\ **Post date:** [August 31, 2022, 4:10am UTC](https://mne.discourse.group/t/all-epochs-are-dropped/5486/12 "2022-08-31T04:10:47Z")
> @richard: reject=500e-6 > @mscheltienne: flat=40e-6 with these numbers, I get very very very little usable data: ```python <4e-05 >0.0005 good EEG Fp1-Ref 555294 16632 3590 EEG Fpz-Ref 571807 2300 1409 EEG Fp2-Ref 524612 47544 3360 EEG F7-Ref 568287 2857 4372 EEG F3-Ref 560018 11024 4474 EEG Fz-Ref 547380 19884 8252 EEG F4-Ref 568060 5887 1569 EEG F8-Ref 571369 2332 1815 EEG FC5-Ref 424891 143873 6752 EEG FC1-Ref 564781 6271 4464 EEG FC2-Ref 564908 6226 4382 EEG FC6-Ref 570800 2361 2355 EEG AFz-Ref 2 575511 3 EEG T7-Ref 469466 91620 14430 EEG C3-Ref 530359 39957 5200 EEG Cz-Ref 568287 2857 4372 EEG C4-Ref 568287 2857 4372 EEG T8-Ref 444103 128340 3073 EEG FT10-Ref 564789 6229 4498 EEG CP5-Ref 451820 105548 18148 EEG CP1-Ref 568287 2857 4372 EEG CP2-Ref 572015 2275 1226 EEG CP6-Ref 568287 2857 4372 EEG P7-Ref 422204 126887 26425 EEG P3-Ref 568668 2819 4029 EEG Pz-Ref 493439 42241 39836 EEG P4-Ref 561129 9501 4886 EEG P8-Ref 550456 19721 5339 EEG FT9-Ref 568287 2857 4372 EEG O1-Ref 572015 2275 1226 EEG Oz-Ref 563776 6086 5654 EEG O2-Ref 541944 27344 6228 ``` is this normal?! ---
**Author:** ![richard](https://yyz2.discourse-cdn.com/free1/user_avatar/mne.discourse.group/richard/32/15_2.png) [@richard](https://mne.discourse.group/u/richard)\ **Post date:** [August 31, 2022, 4:57am UTC](https://mne.discourse.group/t/all-epochs-are-dropped/5486/13 "2022-08-31T04:57:37Z")
I have absolutely no clue what you are doing there, sorry. 😵 You should use `epochs.plot_drop_log()` to get an idea of which channels are outside of the specified range. ---
**Author:** ![mscheltienne](https://yyz2.discourse-cdn.com/free1/user_avatar/mne.discourse.group/mscheltienne/32/827_2.png) [@mscheltienne](https://mne.discourse.group/u/mscheltienne)\ **Post date:** [August 31, 2022, 9:33am UTC](https://mne.discourse.group/t/all-epochs-are-dropped/5486/14 "2022-08-31T09:33:15Z")
instead of looking only at the minimum / maximum, I would have plot one or 2 entire channels to see which kind of range the signal is spanning. That said, let’s take your first Fp1 channel: ```python min max EEG Fp1-Ref -5621.935942 9556.586974 ``` That’s obviously not Volts, 9k Volts from a human brain would be quite something. You need to figure out how those files were generated and in what units the data was saved. ---
**Author:** ![mscheltienne](https://yyz2.discourse-cdn.com/free1/user_avatar/mne.discourse.group/mscheltienne/32/827_2.png) [@mscheltienne](https://mne.discourse.group/u/mscheltienne)\ **Post date:** [August 31, 2022, 9:34am UTC](https://mne.discourse.group/t/all-epochs-are-dropped/5486/15 "2022-08-31T09:34:14Z")
And if you have `reject=dict(eeg=40e-6)` then it means that any epoch with at least one channel that has a peak-to-peak amplitude that **exceeds** 40e-6 will be dropped. ---
**Author:** ![richard](https://yyz2.discourse-cdn.com/free1/user_avatar/mne.discourse.group/richard/32/15_2.png) [@richard](https://mne.discourse.group/u/richard)\ **Post date:** [August 31, 2022, 9:59am UTC](https://mne.discourse.group/t/all-epochs-are-dropped/5486/16 "2022-08-31T09:59:11Z")
> [@mscheltienne](#): > > 9k Volts from a human brain would be quite something ![image](https://global.discourse-cdn.com/free1/uploads/mne/original/2X/2/2a133c06735b6efddd7ecb44cd9eed46f50b6454.gif) ---
**Author:** ![richard](https://yyz2.discourse-cdn.com/free1/user_avatar/mne.discourse.group/richard/32/15_2.png) [@richard](https://mne.discourse.group/u/richard)\ **Post date:** [August 31, 2022, 3:14pm UTC](https://mne.discourse.group/t/all-epochs-are-dropped/5486/17 "2022-08-31T15:14:21Z")
> [@mscheltienne](#): > > And if you have `reject=dict(eeg=40e-6)` then it means that any epoch with at least one channel that has a peak-to-peak amplitude that **exceeds** 40e-6 will be dropped. @sds is passing this value to `flat`, not to `reject`. ---
**Author:** ![mscheltienne](https://yyz2.discourse-cdn.com/free1/user_avatar/mne.discourse.group/mscheltienne/32/827_2.png) [@mscheltienne](https://mne.discourse.group/u/mscheltienne)\ **Post date:** [August 31, 2022, 3:37pm UTC](https://mne.discourse.group/t/all-epochs-are-dropped/5486/18 "2022-08-31T15:37:28Z")
It’s the default for `reject`… ```python reject = dict(grad=4000e-13, # unit: T / m (gradiometers) mag=4e-12, # unit: T (magnetometers) eeg=40e-6, # unit: V (EEG channels) eog=250e-6 # unit: V (EOG channels) ) ``` ---
**Author:** ![richard](https://yyz2.discourse-cdn.com/free1/user_avatar/mne.discourse.group/richard/32/15_2.png) [@richard](https://mne.discourse.group/u/richard)\ **Post date:** [August 31, 2022, 3:58pm UTC](https://mne.discourse.group/t/all-epochs-are-dropped/5486/19 "2022-08-31T15:58:56Z")
> [@mscheltienne](#): > > It’s the default for `reject`… yes but they’re passing `500e-6` for `reject` 😉 ---
**Author:** ![sds](https://yyz2.discourse-cdn.com/free1/user_avatar/mne.discourse.group/sds/32/1810_2.png) [@sds](https://mne.discourse.group/u/sds)\ **Post date:** [August 31, 2022, 5:34pm UTC](https://mne.discourse.group/t/all-epochs-are-dropped/5486/20 "2022-08-31T17:34:11Z")
Indeed the _example_ ( **not** the _default_!) `reject` for [`Epochs`](https://mne.tools/stable/generated/mne.Epochs.html) is `eeg=40e-6` - what @mscheltienne [recommended](https://mne.discourse.group/t/all-epochs-are-dropped/5486/5). (I misunderstood his comment - I thought he recommended it as `flat`). OTOH, @richard [recommends](https://mne.discourse.group/t/all-epochs-are-dropped/5486/8) `eeg=500e-6` for `reject`. I can try either. Thank you very much. However, I still have no idea what the “correct” value for `flat` is. Please don’t tell me “_it depends_”. Just give me something to start with 😉 Thank you again! [Next page](https://mne.discourse.group/t/all-epochs-are-dropped/5486.md?page=2)