Hello to everyone,
MNE-Python version: 0.23.0 & MNE_bids version: 0.8.
I am working with the MEG dataset provided by Rathee et al. (2021). In short, it contains 2 sessions of MEG-recordings (each aprox. 30 minutes long) of 17 subjects who were asked to perform MI (hands / feets) and CI (word generation / subtraction) tasks.
Data: Link
Publication describing the data: Link
My goal is to train a binary classifier to distinguish between the different events using the MEG signal. Specifically, this means that for each subject and each possible combination of events (e.g. Hand vs Feet Imagery), I want to train a binary classifier which predicts the label of the event (in this case either “Hand Imagery” or “Feet Imagery”. For training I am using the data from session 1 and for testing the data from session 2.
This procedure has also been conducted by the authors themselves, the link to their code can be found at the end of the post. They performed their analysis in Matlab using fieldtrip, I am trying to reproduce it in mne-python.
I am facing the following issue: across different combinations of subjects, events and filterbanks, my classifier sometimes returns very high accuracies (e.g., close to 100%), and sometimes very low accuracies (e.g., close to 0%). Since this is a binary classification task, I would expect the performance to never drop significantly below 50%. However, since I know the original results (in the paper, they never achieved better accuracy than 94% for inter-class classification), I also assume that 100% accuracy is way too high.
I posted my code below for one subject, one filterbank and one event combination. As mentioned above, the link to the original code provided in Matlab can be found at the end of this post. I would be very grateful for any idea where this weird behavior might be stemming from.
My code:
from mne.decoding import CSP
from sklearn.svm import SVC
from sklearn.metrics import accuracy_score
import mne_bids
import mne
import numpy as np
# path_to_data = r"D:\data\rathee2021\MEG_BIDS"
path_to_data = r"C:/Users/vof7rng/data/MEG_BIDS"
sessions = ["1", "2"]
sensorSel = "grad"
subject = "6"
combo = ('Both Hand Imagery', 'Subtraction Imagery')
filterbank = {'lowpass_freq': [8, 14], 'highpass_freq': [12, 30]}
features_s1s2 = []
labels_s1s2 = []
## Prepare data, perform FBCSP
for session in sessions:
# read in raw data
bids_path = mne_bids.BIDSPath(subject=subject,
session=session,
root=path_to_data,
task="bcimici")
raw = (mne_bids.read_raw_bids(bids_path)
.pick_types(sensorSel))
# epoch data
events, event_id = mne.events_from_annotations(raw)
tmin = -2
tmax = 5
baseline = None
epochs = mne.Epochs(raw.copy(),
events = events,
tmin = tmin,
tmax = tmax,
baseline = baseline,
event_id = event_id,
proj = False,
preload = True)
# subselect event combination of interest
epochs = epochs[combo]
# resample epochs
epochs = epochs.resample(250)
# get event labels
labels = epochs.events[:, 2]
labels_s1s2.append(labels)
# extract spatio-temporal features for each frequency band
features = []
for lfreq, hfreq in zip(filterbank["lowpass_freq"], filterbank["highpass_freq"]):
# filter data
method = "iir"
epochs_filt = epochs.copy().load_data().filter(lfreq, hfreq, method=method)
# crop epochs
epochs_filt = epochs_filt.crop(tmin=0.5, tmax=3.5)
# extract csp features
csp = CSP(n_components=6, log=True, norm_trace=True)
X_fbcsp = csp.fit_transform(epochs_filt.get_data(), labels)
# append features
features.append(X_fbcsp)
# concatenate features - only use first and last CSP feature
ft = []
for nBand, band in enumerate(features):
x = np.column_stack((band[:, 0], band[:, 5]))
ft.append(x)
X_tr = np.concatenate(ft, axis=1)
features_s1s2.append(X_tr)
## Intersession Analysis
X_tr_inter = features_s1s2[0]
X_ts_inter = features_s1s2[1]
y_tr_inter = labels_s1s2[0]
y_ts_inter = labels_s1s2[1]
# svc
svc = SVC()
svc.fit(X_tr_inter, y_tr_inter)
y_pred = svc.predict(X_ts_inter)
acc_inter = accuracy_score(y_ts_inter, y_pred)
print(acc_inter)
Output:
0.0
Original Code (provided by the authors using Matlab):
Script 1: Data preparing, filtering, FBCSP*
Script 2: Training of both an intraclass and an interclass classifier (for my problem, only the interclass classifier is of interest
