ICA and Average Reference Projection not working

💬 Support & Discussions
,
1

:question: If you have a question or issue with MNE-Python, please include the following info:

  • MNE version: 1.9.0
  • operating system: Kaggle Notebook
def preprocess_eeg_data(raw_data):
    """
    Preprocess EEG data with manual annotation of bad segments and robust ICA settings.
    """
    try:
        # Create a copy of the raw data
        raw_data = raw_data.copy()
        
        # Select EEG channels
        picks = mne.pick_types(raw_data.info, eeg=True, exclude='bads')
        n_channels = len(picks)
        
        # Apply filters
        raw_data.filter(l_freq=1.0, h_freq=None, method='fir', phase='zero-double')
        raw_data.filter(l_freq=None, h_freq=45.0, method='fir', phase='zero-double')
        raw_data.notch_filter([50, 60], method='fir', phase='zero-double')
        
        # Debug: Check data quality
        data = raw_data.get_data(picks=picks)
        print(f"Max amplitude: {np.max(np.abs(data))} µV")
        print(f"Min amplitude: {np.min(np.abs(data))} µV")
        
        # Manually inspect and annotate bad segments
        print("Manually inspect and annotate bad segments...")
        raw_data.plot(block=True, scalings=dict(eeg=100e-6))  # Adjust scalings as needed
        
        # Print annotated bad segments
        if raw_data.annotations:
            print(f"Annotated bad segments: {raw_data.annotations.description}")
        else:
            print("No bad segments annotated.")
        
        try:
            # More conservative ICA approach
            # Calculate optimal number of components based on explained variance
            from sklearn.decomposition import PCA
            pca = PCA(n_components=0.999)  # Keep 99.9% of variance
            pca.fit(data.T)
            n_components = min(5, sum(pca.explained_variance_ratio_.cumsum() < 0.999))
            
            # Apply ICA with conservative settings
            ica = ICA(
                n_components=n_components,  # Use calculated number of components
                random_state=97,
                method='infomax',  # More stable than fastica
                fit_params=dict(extended=True),  # Extended infomax for better stability
                max_iter=1000
            )
            
            # Fit ICA with more robust parameters
            ica.fit(
                raw_data,
                picks=picks,
                decim=3,
                reject=dict(eeg=100000e-6),  # Increased rejection threshold (100,000 µV)
                reject_by_annotation=True,
                verbose=False
            )
            
            # Find EOG artifacts more conservatively
            eog_indices, scores = ica.find_bads_eog(raw_data, threshold=3.5)
            if eog_indices:
                ica.exclude.extend(eog_indices[:2])  # Limit to max 2 components
            
            # Apply ICA
            ica.apply(raw_data)
            
        except Exception as e:
            print(f"ICA processing skipped: {str(e)}")
        
        # Re-reference to average and apply projection
        raw_data.set_eeg_reference('average', projection=True)
        raw_data.apply_proj()  # Apply the average reference projection
        
        # Ensure data is clean
        data = raw_data.get_data()
        data[np.isnan(data) | np.isinf(data)] = 0
        raw_data._data = data
        
        return raw_data
        
    except Exception as e:
        print(f"Error in preprocessing: {str(e)}")
        raise

Output I’m receiving:

Max amplitude: 65366.45013135544 µV
24.1s	115	Min amplitude: 2.6645352591003757e-15 µV
24.1s	116	Manually inspect and annotate bad segments...
24.1s	117	Using matplotlib as 2D backend.
25.1s	118	No bad segments annotated.
25.3s	119	ICA processing skipped: No clean segment found. Please consider updating your rejection thresholds.
25.3s	120	EEG channel type selected for re-referencing
25.3s	121	Adding average EEG reference projection.
25.3s	122	1 projection items deactivated
25.3s	123	Average reference projection was added, but has not been applied yet. Use the apply_proj method to apply it.
2

So what is the problem you’re having? You don’t need to apply the projector if you don’t want to.

Richard

3

My output shows me that ICA processing was skipped because there was no clean segment found. Additionally, I’m trying to add the projector to remove unwanted signal components from the EEG data such as noise or artifacts but its showing that it has not been applied yet but it was added.

4

I would suggest dropping the picks, reject and reject_by_annotation parameters to see if it works then? In order to narrow down the problem.

I don’t think using a try/except block of that size is helpful here, as it makes it harder to figure out where things are going wrong.

Richard

5

Hello,
it is difficult to say something without knowing about the recording parameters.
(sfreq, filter settings, n_samples, and task)

some suggestions:

→ apply only 1.0 Hz high pass
raw.filter(l_freq=1.0,h_freq=None,method=‘fir’)

→ notch if the powerline is 50 Hz use 50,100,150,200, …
fline = 50.0
fmax = raw.info.get(‘lowpass’, raw.info[‘sfreq’] / 2.0)
notches = np.arange(fline,fmax+1,fline)
check results via psd plot

→ bad segments:
If the data are too noisy e.g strong muscle artifact than it will be difficult anyway
How many bad segments and how long they are (% recording time)?
you can try
mne.preprocessing.annotate_muscle_zscore

ica.find_bads_muscle

→ ICA
how many EEG channels an how many components will be used?
you can try with all eeg-picks nobads and no rejection parameter as Richard suggested
n_components = len(picks)

best regards
Frank