I ran a reproducible benchmark of CSP–LDA and Riemannian tangent-space
classifiers on PhysioNet EEG Motor Movement/Imagery and BNCI2014-001.
In the completed PhysioNet analysis (n=109), zeroing 50% of channels reduced
mean ROC-AUC from 0.655 to 0.528 for CSP–LDA and from 0.675 to 0.557 for the
Riemannian pipeline.
The repository contains participant-level summaries, provenance, locked
requirements, validation commands, limitations, and a versioned Zenodo archive:
I would especially value feedback on the channel-failure model and whether an
impedance-noise or intermittent-contact condition would be the most useful next
stress test.
Intermittent contact would be the more informative next stress test. Zeroing already approximates complete channel loss, whereas intermittent contact adds non-stationary corruption that is common in real recordings and may affect CSP and covariance estimates differently.
I would simulate channel-specific clean/noisy segments with several durations and duty cycles, using the same corruption schedule for both decoder families within each participant. I would also compare no correction, channel rejection/masking, and interpolation.
I would keep impedance noise as a separate second condition, with broadband and perhaps line-noise contamination at predefined SNR levels. Combining the two would make it harder to tell which failure mode drives the loss in performance.
Are you planning to test only unexpected corruption at inference time, or also retraining/adaptation with corrupted channels?
Thanks, this is very helpful. I agree that intermittent contact is the better next stress test, and that impedance related contamination should remain a separate condition so the effects stay interpretable
The current release contains two distinct evaluations:
unexpected test-time channel loss, where models are trained on clean data and selected test channels are zeroed; and
full retraining on fixed 3 and 9-channel montages.
It does not yet include corruption-aware retraining, online adaptation, or recalibration. The cross-session condition is fixed transfer rather than adaptation.
For the intermittent-contact follow-up, I would therefore make clean-trained, unexpected test-time corruption the primary analysis. I would use participant-specific corruption schedules that are matched between CSP-LDA and Riemann-LR, then vary segment duration, duty cycle, and the number of affected channels. This should isolate how the two fitted pipelines respond to the same non-stationary failure process
Your proposed correction comparison also makes sense. I am considering:
no correction;
fixed-dimensional masking or zeroing;
interpolation from unaffected channels;
and, as a separate secondary arm, corruption-aware retraining or recalibration.
I would keep ordinary channel rejection out of the direct inference-time comparison when it changes the feature dimension, because both CSP and the covariance/tangent-space pipeline would then generally need to be refitted. It fits more naturally in the retraining/adaptation arm
One remaining design question is where to inject the corruption. Applying it before filtering is closer to a real acquisition failure and captures filter transients, but I may also include an after-filtering sensitivity analysis to separate those transients from the decoder effect
Would you model intermittent contact primarily as hard dropouts, sample-and-hold segments, or broadband artefact bursts? My current inclination is to pre-specify hard dropout and broadband-burst variants, but analyse them separately rather than combine them into one condition
Thank you - this gives me a much clearer structure for the next benchmark extension.
That split makes sense. I would use hard dropouts and broadband bursts as two separate intermittent-contact variants. I would not make sample-and-hold a primary condition unless the acquisition system is known to freeze the last sample, since that is closer to a digital or telemetry failure than electrode contact loss.
Injecting the corruption before filtering makes sense for the primary analysis, with after-filtering injection as a sensitivity check. For broadband bursts, I would scale the amplitude relative to each channel’s clean RMS rather than use one absolute value.
Are you planning to derive the segment durations and duty cycles from real recordings, or use a predefined synthetic grid?
Hi! Looks really interesting! My master’s thesis was related to your work. I used graph signal processing techniques to select differents subsets of electrodes.
What parameters did you use for the CSP+LDA pipeline? I ran the model several times with all the electrodes (baseline) and select the parameters that got me the best results, and then, while maintaining those paramenters, run the model again with every subset of electrodes. I did that with 3 graphs per dataset and 3 different methods per graph. The conclusion was that some methods are consistenly better than others. For example, as you reduce the number of electrodes in the graph (eeg), you’re actually removing the electrodes on the outside. Also, when creating a graph based on a eeg, having more (or less) connections between nodes (more/less edges), that does not have an effect when it comes to the model’s performance. (I attached some images where you can see the models performances and some of their subsets.
I also reconstructed the signals using a method from a paper, which had really good results, so I encourage you to take a look (if needed).
How are you zeroing the channels? randomly?
I used 4 different datasets (one of them was the physionet eeg motor movement/imagery and had really similar results but the amount of subjects made it really hard when it comes te preprocessing compared to the other datasets, so maybe try using more.
Thanks, that’s a good point about sample-and-hold. I’m leaning toward a predefined synthetic grid for the first pass, mainly because it will make the benchmark easier to reproduce and let me vary duration and duty cycle independently. I’d still like the ranges to be realistic rather than arbitrary, though. If I can find suitable recordings or contact-quality logs, I’ll use them to choose the grid and possibly add an empirically sampled condition later.
For now, I’ll keep hard dropouts and RMS-scaled broadband bursts as separate conditions, inject them before filtering, and use after-filter injection only as a sensitivity check. Do you know of an open dataset with annotated electrode-contact failures or usable impedance/contact-quality traces?
Hi Sebastian, thanks for sharing this. Your graph-based selection approach is very relevant here, especially the observation about the outer electrodes being removed as the graph gets smaller.
For CSP+LDA, I used 6 CSP components. The MNE CSP settings were reg=None, log=True, and norm_trace=False, followed by scikit-learn’s LDA with its default settings. The preprocessing was an 8-32 Hz band-pass and resampling to 128 Hz. I kept these parameters fixed rather than tuning them separately for each condition, so that the change from the clean baseline could be attributed to the channel perturbation. The exact setup is in the config and implementation.
For the random-dropout experiment, I sample channels without replacement within each test fold and set the selected channels to zero across all test epochs and time samples. The model is still trained on clean data. I test 10%, 20%, 30%, and 50% dropout, with 10 deterministic participant-specific masks per fraction, matched between the two decoder families. The fixed 3- and 9-channel montages are a separate experiment where the models are retrained on those channel sets.
I’d be interested in reading your thesis or code if either is public. Which other datasets did you use? I agree that adding more datasets would be useful, harmonizing the preprocessing and evaluation across them is the main practical bottleneck for me.
I found two useful leads, though I haven’t found a dataset with verified, synchronized impedance/contact-quality traces.
For real recordings, I’d look at the TUH EEG Artifact Corpus (TUAR). Its electrode-artifact category includes electrode pops, electrostatic and lead artifacts. Access requires registration. It could help you inspect real artifact shapes and durations, but these are clinical EEG artifact labels, not confirmed contact-loss events. I wouldn’t use them directly as ground truth for motor-imagery contact-failure duty cycles.
For the first pass, I’d keep the predefined grid and describe it as a sensitivity analysis rather than an empirically calibrated contact-failure model. You can add empirical calibration later without holding up this version.
What’s your email? so I could send you my thesis (It’s in Spanish, really self explanatory though). If graph signal processing is needed, you can use this library. Datasets are included in the thesis too. For the BCI tasks, are you doing it subject independent or subject specific?
Thanks! You can send it to asok.sci@gmail.com - Spanish is completely fine. I’ll also take a look at EEGraSP.
The current benchmark is subject-specific: each participant is evaluated separately. In the within-session experiments, the train/test folds come from the same participant. In the cross-session experiment, the model is trained on that participant’s first session and tested on their later session(s). I’m not testing generalization to unseen participants yet, although that would be an interesting extension.
