# mne-bids-pipeline - error on a headless server

**URL:** <https://mne.discourse.group/t/mne-bids-pipeline-error-on-a-headless-server/5872>\
**Category:** Support & Discussions\
**Tags:** mne-bids-pipeline\
**Created:** [November 3, 2022, 4:50pm UTC](https://mne.discourse.group/t/mne-bids-pipeline-error-on-a-headless-server/5872 "2022-11-03T16:50:45Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![allermat](https://yyz2.discourse-cdn.com/free1/user_avatar/mne.discourse.group/allermat/32/43_2.png) [@allermat](https://mne.discourse.group/u/allermat)\
**Post date:** [November 3, 2022, 4:50pm UTC](https://mne.discourse.group/t/mne-bids-pipeline-error-on-a-headless-server/5872/1 "2022-11-03T16:50:45Z")

</div>

Hi,

I am trying to reproduce [this](https://mne.tools/mne-bids-pipeline/examples/ds000248.html) analysis pipeline on ds000248 from [openneuro.org](http://openneuro.org). I am now running this on a **headless server** , using slurm.

I get the following error message (copying only the relevant bit from the output):

```python
[2022-11-03 11:21:32] │ ⏳️ report/_01_make_reports sub-01 Rendering sensor alignment (coregistration).
/home/ma09/.conda/envs/mne_compute_node/lib/python3.8/site-packages/pyvista/plotting/renderer.py:463: UserWarning: VTK compiled with OSMesa does not properly support anti-aliasing and anti-aliasing will not be enabled.
  warnings.warn(
/home/ma09/.conda/envs/mne_compute_node/lib/python3.8/site-packages/pyvista/plotting/renderer.py:463: UserWarning: VTK compiled with OSMesa does not properly support anti-aliasing and anti-aliasing will not be enabled.
  warnings.warn(
/home/ma09/.conda/envs/mne_compute_node/lib/python3.8/site-packages/pyvista/plotting/renderer.py:463: UserWarning: VTK compiled with OSMesa does not properly support anti-aliasing and anti-aliasing will not be enabled.
  warnings.warn(
/home/ma09/.conda/envs/mne_compute_node/lib/python3.8/site-packages/pyvista/plotting/plotting.py:5511: UserWarning: Not within a jupyter notebook environment.
Ignoring ``jupyter_backend``.
  warnings.warn(
[2022-11-03 11:21:36] │ ⏳️ report/_01_make_reports sub-01 A critical error occurred. The error message was: 'NoneType' object has no attribute 'layout'

Aborting pipeline run. The full traceback is:

Traceback (most recent call last):

  File "/home/ma09/.conda/envs/mne_compute_node/lib/python3.8/site-packages/mne/report/report.py", line 441, in _iterate_trans_views
    return _itv(

  File "/home/ma09/.conda/envs/mne_compute_node/lib/python3.8/site-packages/mne/report/report.py", line 453, in _itv
    function(fig=fig, **kwargs)

  File "<decorator-gen-121>", line 12, in plot_alignment

  File "/home/ma09/.conda/envs/mne_compute_node/lib/python3.8/site-packages/mne/viz/_3d.py", line 742, in plot_alignment
    _, _, head_surf = _plot_head_surface(

  File "/home/ma09/.conda/envs/mne_compute_node/lib/python3.8/site-packages/mne/viz/_3d.py", line 941, in _plot_head_surface
    src_surf = _get_head_surface(head, subject, subjects_dir, bem=bem)

  File "<decorator-gen-89>", line 12, in _get_head_surface

  File "/home/ma09/.conda/envs/mne_compute_node/lib/python3.8/site-packages/mne/_freesurfer.py", line 726, in _get_head_surface
    raise IOError('No head surface found for subject '

OSError: No head surface found for subject sub-01 after trying:
/imaging/davis/Projects/Openneuro_datasets/ds000248/derivatives/freesurfer/subjects/sub-01/bem/sub-01-head-dense.fif
/imaging/davis/Projects/Openneuro_datasets/ds000248/derivatives/freesurfer/subjects/sub-01/surf/lh.seghead

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

  File "/home/ma09/.conda/envs/mne_compute_node/lib/python3.8/site-packages/mne_bids_pipeline/_run.py", line 62, in wrapper
    out = memory.cache(func)(*args, **kwargs)

  File "/home/ma09/.conda/envs/mne_compute_node/lib/python3.8/site-packages/mne_bids_pipeline/_run.py", line 157, in wrapper
    func(*args, **kwargs)

  File "/home/ma09/.conda/envs/mne_compute_node/lib/python3.8/site-packages/mne_bids_pipeline/scripts/report/_01_make_reports.py", line 1194, in run_report
    report = run_report_source(**kwargs)

  File "/home/ma09/.conda/envs/mne_compute_node/lib/python3.8/site-packages/mne_bids_pipeline/scripts/report/_01_make_reports.py", line 1123, in run_report_source
    report.add_trans(

  File "/home/ma09/.conda/envs/mne_compute_node/lib/python3.8/site-packages/mne/report/report.py", line 1288, in add_trans
    self._add_trans(

  File "/home/ma09/.conda/envs/mne_compute_node/lib/python3.8/site-packages/mne/report/report.py", line 3570, in _add_trans
    img, caption = _iterate_trans_views(

  File "/home/ma09/.conda/envs/mne_compute_node/lib/python3.8/site-packages/mne/report/report.py", line 445, in _iterate_trans_views
    return _itv(function, fig, surfaces={'head': alpha}, **kwargs)

  File "/home/ma09/.conda/envs/mne_compute_node/lib/python3.8/site-packages/mne/report/report.py", line 453, in _itv
    function(fig=fig, **kwargs)

  File "<decorator-gen-121>", line 12, in plot_alignment

  File "/home/ma09/.conda/envs/mne_compute_node/lib/python3.8/site-packages/mne/viz/_3d.py", line 827, in plot_alignment
    renderer.show()

  File "/home/ma09/.conda/envs/mne_compute_node/lib/python3.8/site-packages/mne/viz/backends/_notebook.py", line 1453, in show
    viewer.layout.width = None # unlock the fixed layout

AttributeError: 'NoneType' object has no attribute 'layout'

HBox(children=(Text(value='', layout=Layout(margin='2px 0px 2px 0px', min_width='0px'), placeholder='Type a file name'), Button(icon='camera', layout=Layout(margin='2px 0px 2px 0px', min_width='0px'), style=ButtonStyle(), tooltip='Take a screenshot')))

```

The error seems to stem from a missing head surface file, which is confusing as I was able to reproduce this analysis without an error, based on the same BIDS repository on a normal server (i.e., with graphical interface using vncserver and TurboVNC Viewer).

Could anyone please help me solving this issue?

Many thanks,  
Máté

_Technical details:_  
_I prepared my conda environment according to [these](https://mne.tools/stable/install/advanced.html#installing-to-a-headless-server) instructions. MESA was set up on our system according to [these](https://mne.tools/stable/install/advanced.html#off-screen-rendering-with-mesa) instructions._

_System info:_

```python
Platform: Linux-3.10.0-1160.el7.x86_64-x86_64-with-glibc2.10
Python: 3.8.0 | packaged by conda-forge | (default, Nov 22 2019, 19:11:38) [GCC 7.3.0]
Executable: /home/ma09/.conda/envs/mne_compute_node/bin/python
CPU: x86_64: 20 cores
Memory: 251.7 GB

mne: 1.2.1
numpy: 1.23.4 {OpenBLAS 0.3.21 with 20 threads}
scipy: 1.9.3
matplotlib: 3.6.0 {backend=agg}

sklearn: 1.1.3
numba: Not found
nibabel: 4.0.2
nilearn: 0.9.2
dipy: Not found
openmeeg: Not found
cupy: Not found
pandas: 1.5.1
pyvista: 0.36.1 {OpenGL 3.3 (Core Profile) Mesa 18.3.1 via llvmpipe (LLVM 7.0, 256 bits)}
pyvistaqt: 0.9.0
ipyvtklink: 0.2.2
vtk: 9.0.0
qtpy: 2.1.0 {None=None}
ipympl: 0.9.2
pyqtgraph: Not found
pooch: v1.6.0

mne_bids: 0.11.1
mne_nirs: Not found
mne_features: Not found
mne_qt_browser: Not found
mne_connectivity: Not found
mne_icalabel: Not found

```

---

<div class="post-metadata">

**Author:** ![agramfort](https://avatars.discourse-cdn.com/v4/letter/a/7c8e57/32.png) [@agramfort](https://mne.discourse.group/u/agramfort)\
**Post date:** [November 3, 2022, 8:58pm UTC](https://mne.discourse.group/t/mne-bids-pipeline-error-on-a-headless-server/5872/2 "2022-11-03T20:58:41Z")

</div>

hi @allermat

in this case I use xvfb with something like this:

```bash
xvfb-run mne_bids_pipeline --config config_ds003392.py

```

Alex

---

<div class="post-metadata">

**Author:** ![allermat](https://yyz2.discourse-cdn.com/free1/user_avatar/mne.discourse.group/allermat/32/43_2.png) [@allermat](https://mne.discourse.group/u/allermat)\
**Post date:** [November 3, 2022, 10:33pm UTC](https://mne.discourse.group/t/mne-bids-pipeline-error-on-a-headless-server/5872/3 "2022-11-03T22:33:24Z")

</div>

Hi @agramfort, thanks for the quick response.

I’ve updated my slurm job script according to your response:

```python
#!/bin/tcsh
#SBATCH --nodes 1-1
#SBATCH --ntasks 1
#SBATCH --cpus-per-task 1
#SBATCH --time 1:0:00

# setup environment
module load mnelib
conda activate mne_compute_node

# setup freesurfer
setenv FSVER '6.0.0'
setenv FREESURFER_HOME /imaging/local/software/freesurfer/${FSVER}/`arch`
source $FREESURFER_HOME/FreeSurferEnv.csh

# run the pipeline
xvfb-run mne_bids_pipeline --config=/imaging/davis/users/ma09/Projects/MNE_bids_pipeline_test/my_config.py

```

Unfortunately, I get the same error:

```python
[2022-11-03 22:16:40] │ ⏳️ report/_01_make_reports sub-01 Rendering sensor alignment (coregistration).
/home/ma09/.conda/envs/mne_compute_node/lib/python3.8/site-packages/pyvista/plotting/renderer.py:463: UserWarning: VTK compiled with OSMesa does not properly support anti-aliasing and anti-aliasing will not be enabled.
  warnings.warn(
/home/ma09/.conda/envs/mne_compute_node/lib/python3.8/site-packages/pyvista/plotting/renderer.py:463: UserWarning: VTK compiled with OSMesa does not properly support anti-aliasing and anti-aliasing will not be enabled.
  warnings.warn(
/home/ma09/.conda/envs/mne_compute_node/lib/python3.8/site-packages/pyvista/plotting/renderer.py:463: UserWarning: VTK compiled with OSMesa does not properly support anti-aliasing and anti-aliasing will not be enabled.
  warnings.warn(
/home/ma09/.conda/envs/mne_compute_node/lib/python3.8/site-packages/pyvista/plotting/plotting.py:5511: UserWarning: Not within a jupyter notebook environment.
Ignoring ``jupyter_backend``.
  warnings.warn(
[2022-11-03 22:16:44] │ ⏳️ report/_01_make_reports sub-01 A critical error occurred. The error message was: 'NoneType' object has no attribute 'layout'

Aborting pipeline run. The full traceback is:

Traceback (most recent call last):

  File "/home/ma09/.conda/envs/mne_compute_node/lib/python3.8/site-packages/mne/report/report.py", line 441, in _iterate_trans_views
    return _itv(

  File "/home/ma09/.conda/envs/mne_compute_node/lib/python3.8/site-packages/mne/report/report.py", line 453, in _itv
    function(fig=fig, **kwargs)

  File "<decorator-gen-121>", line 12, in plot_alignment

  File "/home/ma09/.conda/envs/mne_compute_node/lib/python3.8/site-packages/mne/viz/_3d.py", line 742, in plot_alignment
    _, _, head_surf = _plot_head_surface(

  File "/home/ma09/.conda/envs/mne_compute_node/lib/python3.8/site-packages/mne/viz/_3d.py", line 941, in _plot_head_surface
    src_surf = _get_head_surface(head, subject, subjects_dir, bem=bem)

  File "<decorator-gen-89>", line 12, in _get_head_surface

  File "/home/ma09/.conda/envs/mne_compute_node/lib/python3.8/site-packages/mne/_freesurfer.py", line 726, in _get_head_surface
    raise IOError('No head surface found for subject '

OSError: No head surface found for subject sub-01 after trying:
/imaging/davis/Projects/Openneuro_datasets/ds000248/derivatives/freesurfer/subjects/sub-01/bem/sub-01-head-dense.fif
/imaging/davis/Projects/Openneuro_datasets/ds000248/derivatives/freesurfer/subjects/sub-01/surf/lh.seghead

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

  File "/home/ma09/.conda/envs/mne_compute_node/lib/python3.8/site-packages/mne_bids_pipeline/_run.py", line 62, in wrapper
    out = memory.cache(func)(*args, **kwargs)

  File "/home/ma09/.conda/envs/mne_compute_node/lib/python3.8/site-packages/mne_bids_pipeline/_run.py", line 157, in wrapper
    func(*args, **kwargs)

  File "/home/ma09/.conda/envs/mne_compute_node/lib/python3.8/site-packages/mne_bids_pipeline/scripts/report/_01_make_reports.py", line 1194, in run_report
    report = run_report_source(**kwargs)

  File "/home/ma09/.conda/envs/mne_compute_node/lib/python3.8/site-packages/mne_bids_pipeline/scripts/report/_01_make_reports.py", line 1123, in run_report_source
    report.add_trans(

  File "/home/ma09/.conda/envs/mne_compute_node/lib/python3.8/site-packages/mne/report/report.py", line 1288, in add_trans
    self._add_trans(

  File "/home/ma09/.conda/envs/mne_compute_node/lib/python3.8/site-packages/mne/report/report.py", line 3570, in _add_trans
    img, caption = _iterate_trans_views(

  File "/home/ma09/.conda/envs/mne_compute_node/lib/python3.8/site-packages/mne/report/report.py", line 445, in _iterate_trans_views
    return _itv(function, fig, surfaces={'head': alpha}, **kwargs)

  File "/home/ma09/.conda/envs/mne_compute_node/lib/python3.8/site-packages/mne/report/report.py", line 453, in _itv
    function(fig=fig, **kwargs)

  File "<decorator-gen-121>", line 12, in plot_alignment

  File "/home/ma09/.conda/envs/mne_compute_node/lib/python3.8/site-packages/mne/viz/_3d.py", line 827, in plot_alignment
    renderer.show()

  File "/home/ma09/.conda/envs/mne_compute_node/lib/python3.8/site-packages/mne/viz/backends/_notebook.py", line 1453, in show
    viewer.layout.width = None # unlock the fixed layout

AttributeError: 'NoneType' object has no attribute 'layout'

HBox(children=(Text(value='', layout=Layout(margin='2px 0px 2px 0px', min_width='0px'), placeholder='Type a file name'), Button(icon='camera', layout=Layout(margin='2px 0px 2px 0px', min_width='0px'), style=ButtonStyle(), tooltip='Take a screenshot')))

```

Did I miss something?

Thanks,  
Máté

---

<div class="post-metadata">

**Author:** ![agramfort](https://avatars.discourse-cdn.com/v4/letter/a/7c8e57/32.png) [@agramfort](https://mne.discourse.group/u/agramfort)\
**Post date:** [November 4, 2022, 6:35am UTC](https://mne.discourse.group/t/mne-bids-pipeline-error-on-a-headless-server/5872/4 "2022-11-04T06:35:40Z")

</div>

hum. Personally on my remote server I have a regular mne env not the server one.

any chance you can try with the regular environment?

maybe @larsoner has an idea

Alex

---

<div class="post-metadata">

**Author:** ![larsoner](https://yyz2.discourse-cdn.com/free1/user_avatar/mne.discourse.group/larsoner/32/3_2.png) [@larsoner](https://mne.discourse.group/u/larsoner)\
**Post date:** [November 4, 2022, 3:19pm UTC](https://mne.discourse.group/t/mne-bids-pipeline-error-on-a-headless-server/5872/5 "2022-11-04T15:19:15Z")

</div>

If you are okay using `xvfb-run` you can probably use the regular environment.

It’s probably worth opening a bug report on GitHub for the server environment issue, though. It seems like our tests should catch this but somehow they must not. Probably something with `show=True` or `False`, but we’d need to look…

---

<div class="post-metadata">

**Author:** ![allermat](https://yyz2.discourse-cdn.com/free1/user_avatar/mne.discourse.group/allermat/32/43_2.png) [@allermat](https://mne.discourse.group/u/allermat)\
**Post date:** [November 4, 2022, 6:29pm UTC](https://mne.discourse.group/t/mne-bids-pipeline-error-on-a-headless-server/5872/6 "2022-11-04T18:29:27Z")

</div>

Thanks @agramfort and @larsoner for the suggestion. I tried it using `xvfb-run` with the regular mne environment but unfortunately still no luck, see the output:

```python
[2022-11-04 18:24:01] │ ⏳️ report/_01_make_reports sub-01 Rendering sensor alignment (coregistration).
QStandardPaths: XDG_RUNTIME_DIR not set, defaulting to '/tmp/runtime-ma09'
QStandardPaths: XDG_RUNTIME_DIR not set, defaulting to '/tmp/runtime-ma09'
Qt: Session management error: Could not open network socket
libEGL warning: DRI2: failed to create any config
libEGL warning: DRI2: failed to create any config
libEGL warning: DRI2: failed to create any config
libEGL warning: DRI2: failed to create any config
e[0me[31m2022-11-04 18:24:04.256 ( 2.602s) [47978740]vtkXOpenGLRenderWindow.:557 ERR| vtkXOpenGLRenderWindow (0x562aa9d3ee50): GLX not found. Aborting.e[0m
ERROR:root:GLX not found. Aborting.
/usr/bin/xvfb-run: line 181: 5345 Aborted DISPLAY=:$SERVERNUM XAUTHORITY=$AUTHFILE "$@" 2>&1

```

---

<div class="post-metadata">

**Author:** ![larsoner](https://yyz2.discourse-cdn.com/free1/user_avatar/mne.discourse.group/larsoner/32/3_2.png) [@larsoner](https://mne.discourse.group/u/larsoner)\
**Post date:** [November 4, 2022, 6:43pm UTC](https://mne.discourse.group/t/mne-bids-pipeline-error-on-a-headless-server/5872/7 "2022-11-04T18:43:27Z")

</div>

You’ll need working OpenGL. You might be able to get this by using MESA, e.g. with `sudo apt install libosmesa6` or something similar. For what it’s worth, this is what our headless CIs use:

> <https://github.com/mne-tools/mne-python/blob/0ec28e9ad8e234ea51266644ae1ac35a2bc11f46/tools/setup_xvfb.sh#L13-L14>

This might have been the error in the server environment as well, even though the error message didn’t make it clear – that also requires working OpenGL.

You can test OpenGL directly with something like `glxgears`. If that works you have at least some OpenGL support. Then `glxinfo | grep "OpenGL renderer"` should give you info on if MESA is actually being used. Anything newer than 18.3.6 (I think?) should work

---

<div class="post-metadata">

**Author:** ![allermat](https://yyz2.discourse-cdn.com/free1/user_avatar/mne.discourse.group/allermat/32/43_2.png) [@allermat](https://mne.discourse.group/u/allermat)\
**Post date:** [November 4, 2022, 6:56pm UTC](https://mne.discourse.group/t/mne-bids-pipeline-error-on-a-headless-server/5872/8 "2022-11-04T18:56:04Z")

</div>

Many thanks @larsoner, I’ve passed this on to our IT team as I’m a bit out of my depth here (also no sudo rights, I’m afraid 😃 ). I’ll report back once they get it sorted.

Thanks,  
Máté

---

<div class="post-metadata">

**Author:** ![allermat](https://yyz2.discourse-cdn.com/free1/user_avatar/mne.discourse.group/allermat/32/43_2.png) [@allermat](https://mne.discourse.group/u/allermat)\
**Post date:** [November 7, 2022, 5:34pm UTC](https://mne.discourse.group/t/mne-bids-pipeline-error-on-a-headless-server/5872/9 "2022-11-07T17:34:12Z")

</div>

Hi @larsoner,

We did some digging and it turned out that what I needed was to initialize Xvfb manually with the GLX extensions enabled for rendering. That way the pipeline runs without a problem.

The job script which works looks like this:

```python
#!/bin/tcsh
#SBATCH --nodes 1-1
#SBATCH --ntasks 1
#SBATCH --cpus-per-task 1
#SBATCH --time 1:0:00

# setup environment
module load mnelib
conda activate mne_compute_node

# setup freesurfer
setenv FSVER '6.0.0'
setenv FREESURFER_HOME /imaging/local/software/freesurfer/${FSVER}/`arch`
source $FREESURFER_HOME/FreeSurferEnv.csh

# Launch a virtual buffer manually with GLX extensions
Xvfb :11 -screen 0 1024x768x24 +extension GLX +render -noreset &
setenv DISPLAY :11

# run the pipeline
mne_bids_pipeline --config=/imaging/davis/users/ma09/Projects/MNE_bids_pipeline_test/my_config.py

```

---

<div class="post-metadata">

**Author:** ![allermat](https://yyz2.discourse-cdn.com/free1/user_avatar/mne.discourse.group/allermat/32/43_2.png) [@allermat](https://mne.discourse.group/u/allermat)\
**Post date:** [November 8, 2022, 10:28am UTC](https://mne.discourse.group/t/mne-bids-pipeline-error-on-a-headless-server/5872/10 "2022-11-08T10:28:39Z")

</div>

I just wanted to add that we found a better way of doing this using `xvfb-run`. This way the display number is set automatically and the cleanup is handled as well. See the updated job script below.

Best,  
Máté

```python
#!/bin/tcsh
#SBATCH --nodes 1-1
#SBATCH --ntasks 1
#SBATCH --cpus-per-task 1
#SBATCH --time 1:0:00

# setup environment
module load mnelib
conda activate mne_compute_node

# setup freesurfer
setenv FSVER '6.0.0'
setenv FREESURFER_HOME /imaging/local/software/freesurfer/${FSVER}/`arch`
source $FREESURFER_HOME/FreeSurferEnv.csh

# run the pipeline using xvfb-run with GLX extensions enabled
xvfb-run -a -s "-screen 0 1024x768x24 +extension GLX +render -noreset" \
mne_bids_pipeline --config=/imaging/davis/users/ma09/Projects/MNE_bids_pipeline_test/my_config.py

```