Issue when reading Persyst files - wrong position of gaps

💬 Support & Discussions
1

Hi,

I am using mne.io.read_raw_persyst function to read a recording exported from Persyst and noticed an inconsistent location of a gap within the recording.

Some context
The data was originally recorded using a Moberg device, and I am using Persyst as a bypass to convert the data to a format I can read into Python. By default, Moberg splits the recorded data into segments (folders). Each folder has up to 24h of data and always breaks the recordings at 7am. This means that the first segment of data will go from the starting of the recording up to 7am of the next day. The second segment goes from this point (7am) to 7am of the next day, and so on. Persyst reads those segments and exports them separately, keeping this split format.

Issue
The recording I am using has 4 segments and I am using the following code to read the first segment:

from mne.io import read_raw_persyst
import pandas as pd

# seg_folder is the path to the patient folder
eeg_persyst = read_raw_persyst(os.path.join(seg_folder, 'patient.lay'))

display(eeg_persyst)
display(eeg_persyst.to_data_frame())

Below is the output. I am omitting the measurement date for data privacy:

image
image1544×1450 248 KB

I summed the measurement date (start of segment) with the last time value of the DataFrame above (end of segment):

rec1_dt = eeg_persyst.info.get('meas_date')
last_timestamp = rec1_dt + pd.Timedelta(48370.500000, unit='s')
last_timestamp.time()

This results in datetime.time(5, 24, 8, 500000), which is equivalent to time 05:24:8.5. As I have more segments, I would expect to have EEG data up to 7am in this segment, so there is a gap of about 01:36:00 at the end of the segment.

I opened the same segment in Persyst software and noticed that the last timestamp shown there is 7am, but the recording duration is consistent with the duration calculated by MNE (see below).

image
image1920×952 169 KB

Then, I found a gap in this segment occurring from 16:30:23 to 18:06:11 of the previous day, a gap of about 1h 36m, i.e., the same length of the gap found at the end of the segment loaded using MNE.

image
image1920×1050 352 KB

To confirm this gap does not show up at this period (16:30 to 18:06) in the MNE-read waveform, I plotted it using the code below:

import matplotlib.pyplot as plt

df_eeg_persyst = eeg_persyst.to_data_frame()

plt.figure(figsize=(8,4))
plt.plot(rec1_dt + pd.to_timedelta(df_eeg_persyst['time'], unit='s'), df_eeg_persyst['C3'], label='C3')
plt.plot(rec1_dt + pd.to_timedelta(df_eeg_persyst['time'], unit='s'), df_eeg_persyst['F8'], label='F8')
plt.plot(rec1_dt + pd.to_timedelta(df_eeg_persyst['time'], unit='s'), df_eeg_persyst['P3'], label='P3')
plt.title('MNE-PERSYST')
plt.legend()

And zoomed in to show the period with gap (16:30 to 18:06). The forum does not allow me to include more figures, but I can confirm there was no data gap in the plot.

In summary, MNE-read EEG has no gap in the expected period but has a gap at the end of the segment, with the same length. Thus, I believe MNE is missing the timing of the gap and truncating the data, resulting the gap to show up at the end.

Have anyone already found something similar? Any suggestion on how we can solve it?

Thank you very much!
Luiz

I am using:

  • MNE version: e.g. 1.5.0
  • Operating system: macOS Ventura 13.6.
2

I found the issue.

The .dat file generated by Persyst contains contiguous EEG waveform values, no matter if recording interruptions happened or not. Also, timestamps are not stored. To generate the timestamps, one may use the metadata from the .lay file, specifically “TestDate” and “TestTime” (under Section [Patient]) to know when the record started, and “SamplingRate” (under section [FileInfo]) to calculate sample periods. I was looking at the source codes but couldn’t find how MNE generated the time column we get from the function to_data_frame().

The trick is to take into account the section [SampleTimes] from the .lay file. Each element of this section is composed by:

sample_index = physical_time

where physical_time is a floating-point number of seconds since midnight of “TestDate” date specified in the [Patient] section. Consider the example below:

[SampleTimes]
0=41856.385
2923=41991
8415=42198.46

In this case, the record started 41856.385 seconds after midnight (11:37:36.385). The sample index 2923 occurred 41991 seconds after midnight (11:39:51), and sample index 8415 occurred 42198.46 seconds after midnight (11:43:18.46). If we considered only the start time and sampling period, the times associated with these indices would be different.

I am manually correcting the column of time after loading the EEG with mne.io.read_raw_persyst, but I believe this would be desirable that this process be part of the function.

Best,
Luiz

1 Like
3

thanks for sharing the solution you found, and for the detailed explanation! Do you have time to open a bug report saying basically “persyst reader ignores SampleTimes in the .lay file” and linking back to your explanation here?

4

Sure, will do it. Thanks for showing the pathway.

1 Like