mne-bids-pipeline - error on a headless server

Hi,

I am trying to reproduce this analysis pipeline on ds000248 from openneuro.org. I am now running this on a headless server, using slurm.

I get the following error message (copying only the relevant bit from the output):

[2022-11-03 11:21:32] │ ⏳️ report/_01_make_reports sub-01 Rendering sensor alignment (coregistration).
/home/ma09/.conda/envs/mne_compute_node/lib/python3.8/site-packages/pyvista/plotting/renderer.py:463: UserWarning: VTK compiled with OSMesa does not properly support anti-aliasing and anti-aliasing will not be enabled.
  warnings.warn(
/home/ma09/.conda/envs/mne_compute_node/lib/python3.8/site-packages/pyvista/plotting/renderer.py:463: UserWarning: VTK compiled with OSMesa does not properly support anti-aliasing and anti-aliasing will not be enabled.
  warnings.warn(
/home/ma09/.conda/envs/mne_compute_node/lib/python3.8/site-packages/pyvista/plotting/renderer.py:463: UserWarning: VTK compiled with OSMesa does not properly support anti-aliasing and anti-aliasing will not be enabled.
  warnings.warn(
/home/ma09/.conda/envs/mne_compute_node/lib/python3.8/site-packages/pyvista/plotting/plotting.py:5511: UserWarning: Not within a jupyter notebook environment.
Ignoring ``jupyter_backend``.
  warnings.warn(
[2022-11-03 11:21:36] │ ⏳️ report/_01_make_reports sub-01 A critical error occurred. The error message was: 'NoneType' object has no attribute 'layout'

Aborting pipeline run. The full traceback is:

Traceback (most recent call last):

  File "/home/ma09/.conda/envs/mne_compute_node/lib/python3.8/site-packages/mne/report/report.py", line 441, in _iterate_trans_views
    return _itv(

  File "/home/ma09/.conda/envs/mne_compute_node/lib/python3.8/site-packages/mne/report/report.py", line 453, in _itv
    function(fig=fig, **kwargs)

  File "<decorator-gen-121>", line 12, in plot_alignment

  File "/home/ma09/.conda/envs/mne_compute_node/lib/python3.8/site-packages/mne/viz/_3d.py", line 742, in plot_alignment
    _, _, head_surf = _plot_head_surface(

  File "/home/ma09/.conda/envs/mne_compute_node/lib/python3.8/site-packages/mne/viz/_3d.py", line 941, in _plot_head_surface
    src_surf = _get_head_surface(head, subject, subjects_dir, bem=bem)

  File "<decorator-gen-89>", line 12, in _get_head_surface

  File "/home/ma09/.conda/envs/mne_compute_node/lib/python3.8/site-packages/mne/_freesurfer.py", line 726, in _get_head_surface
    raise IOError('No head surface found for subject '

OSError: No head surface found for subject sub-01 after trying:
/imaging/davis/Projects/Openneuro_datasets/ds000248/derivatives/freesurfer/subjects/sub-01/bem/sub-01-head-dense.fif
/imaging/davis/Projects/Openneuro_datasets/ds000248/derivatives/freesurfer/subjects/sub-01/surf/lh.seghead


During handling of the above exception, another exception occurred:


Traceback (most recent call last):

  File "/home/ma09/.conda/envs/mne_compute_node/lib/python3.8/site-packages/mne_bids_pipeline/_run.py", line 62, in wrapper
    out = memory.cache(func)(*args, **kwargs)

  File "/home/ma09/.conda/envs/mne_compute_node/lib/python3.8/site-packages/mne_bids_pipeline/_run.py", line 157, in wrapper
    func(*args, **kwargs)

  File "/home/ma09/.conda/envs/mne_compute_node/lib/python3.8/site-packages/mne_bids_pipeline/scripts/report/_01_make_reports.py", line 1194, in run_report
    report = run_report_source(**kwargs)

  File "/home/ma09/.conda/envs/mne_compute_node/lib/python3.8/site-packages/mne_bids_pipeline/scripts/report/_01_make_reports.py", line 1123, in run_report_source
    report.add_trans(

  File "/home/ma09/.conda/envs/mne_compute_node/lib/python3.8/site-packages/mne/report/report.py", line 1288, in add_trans
    self._add_trans(

  File "/home/ma09/.conda/envs/mne_compute_node/lib/python3.8/site-packages/mne/report/report.py", line 3570, in _add_trans
    img, caption = _iterate_trans_views(

  File "/home/ma09/.conda/envs/mne_compute_node/lib/python3.8/site-packages/mne/report/report.py", line 445, in _iterate_trans_views
    return _itv(function, fig, surfaces={'head': alpha}, **kwargs)

  File "/home/ma09/.conda/envs/mne_compute_node/lib/python3.8/site-packages/mne/report/report.py", line 453, in _itv
    function(fig=fig, **kwargs)

  File "<decorator-gen-121>", line 12, in plot_alignment

  File "/home/ma09/.conda/envs/mne_compute_node/lib/python3.8/site-packages/mne/viz/_3d.py", line 827, in plot_alignment
    renderer.show()

  File "/home/ma09/.conda/envs/mne_compute_node/lib/python3.8/site-packages/mne/viz/backends/_notebook.py", line 1453, in show
    viewer.layout.width = None  # unlock the fixed layout

AttributeError: 'NoneType' object has no attribute 'layout'

HBox(children=(Text(value='', layout=Layout(margin='2px 0px 2px 0px', min_width='0px'), placeholder='Type a file name'), Button(icon='camera', layout=Layout(margin='2px 0px 2px 0px', min_width='0px'), style=ButtonStyle(), tooltip='Take a screenshot')))

The error seems to stem from a missing head surface file, which is confusing as I was able to reproduce this analysis without an error, based on the same BIDS repository on a normal server (i.e., with graphical interface using vncserver and TurboVNC Viewer).

Could anyone please help me solving this issue?

Many thanks,
Máté

Technical details:
I prepared my conda environment according to these instructions. MESA was set up on our system according to these instructions.

System info:

Platform:         Linux-3.10.0-1160.el7.x86_64-x86_64-with-glibc2.10
Python:           3.8.0 | packaged by conda-forge | (default, Nov 22 2019, 19:11:38)  [GCC 7.3.0]
Executable:       /home/ma09/.conda/envs/mne_compute_node/bin/python
CPU:              x86_64: 20 cores
Memory:           251.7 GB

mne:              1.2.1
numpy:            1.23.4 {OpenBLAS 0.3.21 with 20 threads}
scipy:            1.9.3
matplotlib:       3.6.0 {backend=agg}

sklearn:          1.1.3
numba:            Not found
nibabel:          4.0.2
nilearn:          0.9.2
dipy:             Not found
openmeeg:         Not found
cupy:             Not found
pandas:           1.5.1
pyvista:          0.36.1 {OpenGL 3.3 (Core Profile) Mesa 18.3.1 via llvmpipe (LLVM 7.0, 256 bits)}
pyvistaqt:        0.9.0
ipyvtklink:       0.2.2
vtk:              9.0.0
qtpy:             2.1.0 {None=None}
ipympl:           0.9.2
pyqtgraph:        Not found
pooch:            v1.6.0

mne_bids:         0.11.1
mne_nirs:         Not found
mne_features:     Not found
mne_qt_browser:   Not found
mne_connectivity: Not found
mne_icalabel:     Not found
1 Like

hi @allermat

in this case I use xvfb with something like this:

xvfb-run mne_bids_pipeline --config config_ds003392.py

Alex

2 Likes

Hi @agramfort, thanks for the quick response.

I’ve updated my slurm job script according to your response:

#!/bin/tcsh
#SBATCH --nodes 1-1
#SBATCH --ntasks 1
#SBATCH --cpus-per-task 1
#SBATCH --time 1:0:00

# setup environment
module load mnelib
conda activate mne_compute_node

# setup freesurfer
setenv FSVER '6.0.0'
setenv FREESURFER_HOME /imaging/local/software/freesurfer/${FSVER}/`arch`
source $FREESURFER_HOME/FreeSurferEnv.csh

# run the pipeline
xvfb-run mne_bids_pipeline --config=/imaging/davis/users/ma09/Projects/MNE_bids_pipeline_test/my_config.py

Unfortunately, I get the same error:

[2022-11-03 22:16:40] │ ⏳️ report/_01_make_reports sub-01 Rendering sensor alignment (coregistration).
/home/ma09/.conda/envs/mne_compute_node/lib/python3.8/site-packages/pyvista/plotting/renderer.py:463: UserWarning: VTK compiled with OSMesa does not properly support anti-aliasing and anti-aliasing will not be enabled.
  warnings.warn(
/home/ma09/.conda/envs/mne_compute_node/lib/python3.8/site-packages/pyvista/plotting/renderer.py:463: UserWarning: VTK compiled with OSMesa does not properly support anti-aliasing and anti-aliasing will not be enabled.
  warnings.warn(
/home/ma09/.conda/envs/mne_compute_node/lib/python3.8/site-packages/pyvista/plotting/renderer.py:463: UserWarning: VTK compiled with OSMesa does not properly support anti-aliasing and anti-aliasing will not be enabled.
  warnings.warn(
/home/ma09/.conda/envs/mne_compute_node/lib/python3.8/site-packages/pyvista/plotting/plotting.py:5511: UserWarning: Not within a jupyter notebook environment.
Ignoring ``jupyter_backend``.
  warnings.warn(
[2022-11-03 22:16:44] │ ⏳️ report/_01_make_reports sub-01 A critical error occurred. The error message was: 'NoneType' object has no attribute 'layout'

Aborting pipeline run. The full traceback is:

Traceback (most recent call last):

  File "/home/ma09/.conda/envs/mne_compute_node/lib/python3.8/site-packages/mne/report/report.py", line 441, in _iterate_trans_views
    return _itv(

  File "/home/ma09/.conda/envs/mne_compute_node/lib/python3.8/site-packages/mne/report/report.py", line 453, in _itv
    function(fig=fig, **kwargs)

  File "<decorator-gen-121>", line 12, in plot_alignment

  File "/home/ma09/.conda/envs/mne_compute_node/lib/python3.8/site-packages/mne/viz/_3d.py", line 742, in plot_alignment
    _, _, head_surf = _plot_head_surface(

  File "/home/ma09/.conda/envs/mne_compute_node/lib/python3.8/site-packages/mne/viz/_3d.py", line 941, in _plot_head_surface
    src_surf = _get_head_surface(head, subject, subjects_dir, bem=bem)

  File "<decorator-gen-89>", line 12, in _get_head_surface

  File "/home/ma09/.conda/envs/mne_compute_node/lib/python3.8/site-packages/mne/_freesurfer.py", line 726, in _get_head_surface
    raise IOError('No head surface found for subject '

OSError: No head surface found for subject sub-01 after trying:
/imaging/davis/Projects/Openneuro_datasets/ds000248/derivatives/freesurfer/subjects/sub-01/bem/sub-01-head-dense.fif
/imaging/davis/Projects/Openneuro_datasets/ds000248/derivatives/freesurfer/subjects/sub-01/surf/lh.seghead


During handling of the above exception, another exception occurred:


Traceback (most recent call last):

  File "/home/ma09/.conda/envs/mne_compute_node/lib/python3.8/site-packages/mne_bids_pipeline/_run.py", line 62, in wrapper
    out = memory.cache(func)(*args, **kwargs)

  File "/home/ma09/.conda/envs/mne_compute_node/lib/python3.8/site-packages/mne_bids_pipeline/_run.py", line 157, in wrapper
    func(*args, **kwargs)

  File "/home/ma09/.conda/envs/mne_compute_node/lib/python3.8/site-packages/mne_bids_pipeline/scripts/report/_01_make_reports.py", line 1194, in run_report
    report = run_report_source(**kwargs)

  File "/home/ma09/.conda/envs/mne_compute_node/lib/python3.8/site-packages/mne_bids_pipeline/scripts/report/_01_make_reports.py", line 1123, in run_report_source
    report.add_trans(

  File "/home/ma09/.conda/envs/mne_compute_node/lib/python3.8/site-packages/mne/report/report.py", line 1288, in add_trans
    self._add_trans(

  File "/home/ma09/.conda/envs/mne_compute_node/lib/python3.8/site-packages/mne/report/report.py", line 3570, in _add_trans
    img, caption = _iterate_trans_views(

  File "/home/ma09/.conda/envs/mne_compute_node/lib/python3.8/site-packages/mne/report/report.py", line 445, in _iterate_trans_views
    return _itv(function, fig, surfaces={'head': alpha}, **kwargs)

  File "/home/ma09/.conda/envs/mne_compute_node/lib/python3.8/site-packages/mne/report/report.py", line 453, in _itv
    function(fig=fig, **kwargs)

  File "<decorator-gen-121>", line 12, in plot_alignment

  File "/home/ma09/.conda/envs/mne_compute_node/lib/python3.8/site-packages/mne/viz/_3d.py", line 827, in plot_alignment
    renderer.show()

  File "/home/ma09/.conda/envs/mne_compute_node/lib/python3.8/site-packages/mne/viz/backends/_notebook.py", line 1453, in show
    viewer.layout.width = None  # unlock the fixed layout

AttributeError: 'NoneType' object has no attribute 'layout'

HBox(children=(Text(value='', layout=Layout(margin='2px 0px 2px 0px', min_width='0px'), placeholder='Type a file name'), Button(icon='camera', layout=Layout(margin='2px 0px 2px 0px', min_width='0px'), style=ButtonStyle(), tooltip='Take a screenshot')))

Did I miss something?

Thanks,
Máté

hum. Personally on my remote server I have a regular mne env not the server one.

any chance you can try with the regular environment?

maybe @larsoner has an idea

Alex

1 Like

If you are okay using xvfb-run you can probably use the regular environment.

It’s probably worth opening a bug report on GitHub for the server environment issue, though. It seems like our tests should catch this but somehow they must not. Probably something with show=True or False, but we’d need to look…

1 Like

Thanks @agramfort and @larsoner for the suggestion. I tried it using xvfb-run with the regular mne environment but unfortunately still no luck, see the output:

[2022-11-04 18:24:01] │ ⏳️ report/_01_make_reports sub-01 Rendering sensor alignment (coregistration).
QStandardPaths: XDG_RUNTIME_DIR not set, defaulting to '/tmp/runtime-ma09'
QStandardPaths: XDG_RUNTIME_DIR not set, defaulting to '/tmp/runtime-ma09'
Qt: Session management error: Could not open network socket
libEGL warning: DRI2: failed to create any config
libEGL warning: DRI2: failed to create any config
libEGL warning: DRI2: failed to create any config
libEGL warning: DRI2: failed to create any config
e[0me[31m2022-11-04 18:24:04.256 (   2.602s) [        47978740]vtkXOpenGLRenderWindow.:557    ERR| vtkXOpenGLRenderWindow (0x562aa9d3ee50): GLX not found.  Aborting.e[0m
ERROR:root:GLX not found.  Aborting.
/usr/bin/xvfb-run: line 181:  5345 Aborted                 DISPLAY=:$SERVERNUM XAUTHORITY=$AUTHFILE "$@" 2>&1

You’ll need working OpenGL. You might be able to get this by using MESA, e.g. with sudo apt install libosmesa6 or something similar. For what it’s worth, this is what our headless CIs use:

This might have been the error in the server environment as well, even though the error message didn’t make it clear – that also requires working OpenGL.

You can test OpenGL directly with something like glxgears. If that works you have at least some OpenGL support. Then glxinfo | grep "OpenGL renderer" should give you info on if MESA is actually being used. Anything newer than 18.3.6 (I think?) should work

1 Like

Many thanks @larsoner, I’ve passed this on to our IT team as I’m a bit out of my depth here (also no sudo rights, I’m afraid :smiley: ). I’ll report back once they get it sorted.

Thanks,
Máté

Hi @larsoner,

We did some digging and it turned out that what I needed was to initialize Xvfb manually with the GLX extensions enabled for rendering. That way the pipeline runs without a problem.

The job script which works looks like this:

#!/bin/tcsh
#SBATCH --nodes 1-1
#SBATCH --ntasks 1
#SBATCH --cpus-per-task 1
#SBATCH --time 1:0:00

# setup environment
module load mnelib
conda activate mne_compute_node

# setup freesurfer
setenv FSVER '6.0.0'
setenv FREESURFER_HOME /imaging/local/software/freesurfer/${FSVER}/`arch`
source $FREESURFER_HOME/FreeSurferEnv.csh

# Launch a virtual buffer manually with GLX extensions
Xvfb :11 -screen 0 1024x768x24 +extension GLX +render -noreset &
setenv DISPLAY :11

# run the pipeline
mne_bids_pipeline --config=/imaging/davis/users/ma09/Projects/MNE_bids_pipeline_test/my_config.py
2 Likes

I just wanted to add that we found a better way of doing this using xvfb-run. This way the display number is set automatically and the cleanup is handled as well. See the updated job script below.

Best,
Máté

#!/bin/tcsh
#SBATCH --nodes 1-1
#SBATCH --ntasks 1
#SBATCH --cpus-per-task 1
#SBATCH --time 1:0:00

# setup environment
module load mnelib
conda activate mne_compute_node

# setup freesurfer
setenv FSVER '6.0.0'
setenv FREESURFER_HOME /imaging/local/software/freesurfer/${FSVER}/`arch`
source $FREESURFER_HOME/FreeSurferEnv.csh

# run the pipeline using xvfb-run with GLX extensions enabled
xvfb-run -a -s "-screen 0 1024x768x24 +extension GLX +render -noreset" \
mne_bids_pipeline --config=/imaging/davis/users/ma09/Projects/MNE_bids_pipeline_test/my_config.py
1 Like