lowpass filter causes loss of data

đź’¬ Support & Discussions
,
1

Hi all,
I’ve been filtering my data the following way:
raw.filter(0.1, 40)

this usually works fine for most of my data (most recordings are about 5 minutes). I now applied this filter to a longer segment (20 minutes). After I filter the data, about half of the datapoints only display nan now, now real value is shown.

When I choose a higher low-pass filter (e.g. raw.filter (0.5,40) the problem is less bad (fewer cells show nan). When choosing a lower low-pass filter (e.g. raw.filter(0.01, 40), the problem gets worse (the raw data now contains only nan, no real values anymore).

Any idea why this is the case? It only happens to this one file, but when looking at the raw data or the power spectrum before filtering it doesn’t look significantly different from the other data…

2

Update:
I should also add that some values are lost (cells display nan) when I use notch filters.

3

Hello,

Many different settings can be provided to a filter. Can you please copy/paste a minimal code snippet?

Mathieu

2 Likes
4

Also for clarification, based on your examples you are adjusting the value for the high-pass part of the bandpass filter; you’re not changing anything related to low-pass filtering (where the frequency bound is always 40 Hz in your examples)

1 Like
5

Hi,
thank you for those two suggestions! First, I mixed up high- and low-pass filter there. It’s the high pass filter that creates the main problem (not the low-pass as I initially wrote).

Then for the code snipped:

raw.plot()
raw.filter(0.1, 40)
raw.plot()

These lines are enough to cause the problem. The first plot looks great (a bit noisy as it’s not yet band-pass filtered, but okay. After I filter, the raw.plot() outputs an empty white graph. When I then look into the content of the raw file, there are a few nan cells (not all of them are nan). Depending on how I set the upper and lower bound of the filter the location of the nan cells changes (they are someitmes at the beginning, sometimes somewhere in the middle).

I never experienced this before using the filter. Is there another setting I try using for the bandpass filter?

6

SOLVED!

(sorry, it was a silly mistake).

The raw data had one cell that was empty for some reason. I didn’t spot it. But using the bandpass filter, this empty cell effected a lot of other data points around it (the amount depended on the upper and lower bounds of the filter).

So the problem was the raw data I was feeding it, not the function.

Thanks so much for your quick response nevertheless!!

1 Like
7

it’s very weird the filter creates NaNs. Can you share the data with us to try to replicate?

ALex