Support for 3D brain visualization in headless mode?

  • MNE version: 0.24.0
  • operating system: Linux on Docker

I’m trying to run the MRI reconstruction example without success on a headless installation on Docker

The best attempt following other discussion about 3d visualization and the use of pyvista seems to be the following:

import os
import mne
from pyvista.utilities import xvfb
xvfb.start_xvfb()
mne.viz.set_3d_backend("notebook")
sample_data_folder = mne.datasets.sample.data_path()
subjects_dir = os.path.join(sample_data_folder, 'subjects')
Brain = mne.viz.get_brain_class()
brain = Brain('sample', hemi='lh', surf='pial',
              subjects_dir=subjects_dir, size=(800, 600))
brain.add_annotation('aparc.a2009s', borders=False)

With the predefined mne-tools/mne-python-plot docker image I get the following error on a jupyter notebook:

RuntimeError                              Traceback (most recent call last)
Input In [5], in <module>
      7 subjects_dir = os.path.join(sample_data_folder, 'subjects')
      8 Brain = mne.viz.get_brain_class()
----> 9 brain = Brain('sample', hemi='lh', surf='pial',
     10               subjects_dir=subjects_dir, size=(800, 600))
     11 brain.add_annotation('aparc.a2009s', borders=False)

File /opt/conda/lib/python3.9/site-packages/mne/viz/_brain/_brain.py:528, in Brain.__init__(self, subject_id, hemi, surf, title, cortex, alpha, size, background, foreground, figure, subjects_dir, views, offset, show_toolbar, offscreen, interaction, units, view_layout, silhouette, theme, show)
    520             mesh = self._layered_meshes[h]
    521             self._renderer._silhouette(
    522                 mesh=mesh._polydata,
    523                 color=self._silhouette["color"],
   (...)
    526                 decimate=self._silhouette["decimate"],
    527             )
--> 528         self._renderer.set_camera(**views_dicts[h][v])
    530 self.interaction = interaction
    531 self._closed = False

File /opt/conda/lib/python3.9/site-packages/mne/viz/backends/_pyvista.py:599, in _PyVistaRenderer.set_camera(self, azimuth, elevation, distance, focalpoint, roll, reset_camera, rigid)
    596 def set_camera(self, azimuth=None, elevation=None, distance=None,
    597                focalpoint='auto', roll=None, reset_camera=True,
    598                rigid=None):
--> 599     _set_3d_view(self.figure, azimuth=azimuth, elevation=elevation,
    600                  distance=distance, focalpoint=focalpoint, roll=roll,
    601                  reset_camera=reset_camera, rigid=rigid)

File /opt/conda/lib/python3.9/site-packages/mne/viz/backends/_pyvista.py:996, in _set_3d_view(figure, azimuth, elevation, focalpoint, distance, roll, reset_camera, rigid)
    993 if roll is not None:
    994     figure.plotter.camera.SetRoll(figure.plotter.camera.GetRoll() + roll)
--> 996 figure.plotter.update()
    997 _process_events(figure.plotter)

File /opt/conda/lib/python3.9/site-packages/pyvista/plotting/plotting.py:1538, in BasePlotter.update(self, stime, force_redraw)
   1536 update_rate = self.iren.get_desired_update_rate()
   1537 if (curr_time - Plotter.last_update_time) > (1.0/update_rate):
-> 1538     self.right_timer_id = self.iren.create_repeating_timer(stime)
   1539     self.render()
   1540     Plotter.last_update_time = curr_time

File /opt/conda/lib/python3.9/site-packages/pyvista/plotting/render_window_interactor.py:640, in RenderWindowInteractor.create_repeating_timer(self, stime)
    638 timer_id = self.interactor.CreateRepeatingTimer(stime)
    639 if hasattr(self.interactor, 'ProcessEvents'):
--> 640     self.process_events()
    641 else:
    642     self.interactor.Start()

File /opt/conda/lib/python3.9/site-packages/pyvista/plotting/render_window_interactor.py:662, in RenderWindowInteractor.process_events(self)
    660 # Note: This is only available in VTK 9+
    661 if not self.initialized:
--> 662     raise RuntimeError('Render window interactor must be initialized '
    663                        'before processing events.')
    664 self.interactor.ProcessEvents()

RuntimeError: Render window interactor must be initialized before processing events.

Also tried other images setups starting from here Installing MNE-Python — MNE 1.0.dev0 documentation with no luck (at best I get a black canvas)

Thanks in advance

Does mne sys_info tell you that you have 3D capabilities? If it segfaults, you do not :slight_smile:

Have you seen Advanced setup — MNE 0.24.1 documentation ?

@larsoner thanks for your reply.

This is the output

Matplotlib is building the font cache; this may take a moment.
qt.qpa.xcb: could not connect to display 
qt.qpa.plugin: Could not load the Qt platform plugin "xcb" in "" even though it was found.
This application failed to start because no Qt platform plugin could be initialized. Reinstalling the application may fix this problem.

Available platform plugins are: eglfs, minimal, minimalegl, offscreen, vnc, webgl, xcb.

Aborted

The documentation you point out seems worth a try. I overlooked it as here there is no mention about MESA, and it does not seem to be considered also in the provided Docker images.
As jupyter notebooks and headless environments are a very useful use case it would be nice to have it stated explicitly in the docs!

Hmm, so it is trying to use the PyQt5 backend. This is actually a general PyQt5 issue. See the reply about QT_DEBUG_PLUGINS here:

Somewhere recently I saw a list of the libraries you’d need to install, but I can’t find it.

But really you might want to use the 'notebook' backend for PyVista – are you planning on using a Jupyter notebook server in your headless env?

Yes my goal is to make the brain visualization work on a notebook (see code above).
I can get the notebook backend with pyvista backend work (with the panel backend for instance), but I get a black screen on mne with the brain viz.

For instance, this works:

from pyvista import demos
from pyvista.utilities import xvfb
xvfb.start_xvfb()
demos.plot_logo(jupyter_backend='panel')

This is my Docker image definition:

FROM jupyter/base-notebook:hub-1.4.2
USER root

# install utils
RUN apt-get -y update
RUN apt-get -y install software-properties-common
RUN add-apt-repository "deb http://us.archive.ubuntu.com/ubuntu bionic main universe"
RUN apt-get -y update
RUN apt-get -y install git wget tcsh xxd build-essential
RUN apt-get -y install libblas-dev liblapack-dev zlib1g-dev
RUN apt-get -y install libxmu-dev libxmu-headers libxi-dev libxt-dev libx11-dev libglu1-mesa-dev
RUN apt-get -y install gcc-4.8 g++-4.8 gfortran-4.8

ARG FREESURFER_DEB_URL=https://surfer.nmr.mgh.harvard.edu/pub/dist/freesurfer/7.2.0/freesurfer_7.2.0_amd64.deb 
RUN wget $FREESURFER_DEB_URL -O fs.deb


# RUN apt-get -y install mesa-libGL libXext libSM libXrender libXmu

# install fs

RUN apt-get -y install ./fs.deb
RUN rm fs.deb

# setup fs env
ENV OS Linux
ENV PATH $PATH:/usr/local/freesurfer/7.2.0/bin:/usr/local/freesurfer/7.2.0/fsfast/bin:/usr/local/freesurfer/7.2.0/tktools:/usr/local/freesurfer/7.2.0/mni/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
ENV FREESURFER_HOME /usr/local/freesurfer/7.2.0
ENV FREESURFER /usr/local/freesurfer/7.2.0
ENV SUBJECTS_DIR /home/shared/subjects
ENV LOCAL_DIR /usr/local/freesurfer/7.2.0/local
ENV FSFAST_HOME /usr/local/freesurfer/7.2.0/fsfast
ENV FMRI_ANALYSIS_DIR /usr/local/freesurfer/7.2.0/fsfast
ENV FUNCTIONALS_DIR /usr/local/freesurfer/7.2.0/sessions

# set default fs options
ENV FS_OVERRIDE 0
ENV FIX_VERTEX_AREA ""
ENV FSF_OUTPUT_FORMAT nii.gz

# mni env requirements
ENV MINC_BIN_DIR /usr/local/freesurfer/7.2.0/mni/bin
ENV MINC_LIB_DIR /usr/local/freesurfer/7.2.0/mni/lib
ENV MNI_DIR /usr/local/freesurfer/7.2.0/mni
ENV MNI_DATAPATH /usr/local/freesurfer/7.2.0/mni/data
ENV MNI_PERL5LIB /usr/local/freesurfer/7.2.0/mni/share/perl5
ENV PERL5LIB /usr/local/freesurfer/7.2.0/mni/share/perl5


RUN apt-get update
RUN apt-get install -y libgl1-mesa-glx xvfb libosmesa6 libglx-mesa0 libopengl0 libglx0 mesa-utils


COPY .license $FREESURFER_HOME
RUN chown -R jovyan /opt/conda
USER jovyan
# sym link workspace pvc to $FOLDER

RUN conda init
RUN conda install --channel=conda-forge -y mne

COPY requirements.txt
RUN pip install -r requirements.txt
WORKDIR /home/workspace

After your set_3d_backend call, if you do get_3d_backend() does it say 'notebook'? Maybe you are missing some dependency.

For what it’s worth, this is the conda env we use in CI testing for the notebook:

@GuillaumeFavelier might have other ideas

1 Like

I think @bloyl made an awesome job in GitHub - bloyl/mne_docker and although it’s marked as work in progress, it might be enough for your needs? I think it’s worth looking at

1 Like

@larsoner I will try that environment file, thanks.
What I see a black figure with the required libraries installed

Installed packages:

Package                       Version    Editable project location
----------------------------- ---------- ----------------------------------
aiohttp                       3.8.1
aiosignal                     1.2.0
alembic                       1.7.4
anyio                         3.3.4
appdirs                       1.4.4
apptools                      5.1.0
argo-workflows                5.0.0
argon2-cffi                   21.1.0
asn1crypto                    0.24.0
async-generator               1.10
async-timeout                 4.0.2
attrs                         21.2.0
Babel                         2.9.1
backcall                      0.2.0
backports.functools-lru-cache 1.6.4
bleach                        4.1.0
blinker                       1.4
bokeh                         2.4.2
brotlipy                      0.7.0
cached-property               1.5.2
cachetools                    4.2.2
certifi                       2019.3.9
certipy                       0.1.3
cffi                          1.15.0
chardet                       3.0.4
charset-normalizer            2.0.0
click                         8.0.3
cloudharness                  0.4.0      /libraries/cloudharness-common
cloudharness-cli              0.4.0      /libraries/client/cloudharness_cli
cloudpickle                   2.0.0
colorama                      0.4.4
conda                         4.10.3
conda-package-handling        1.7.3
configobj                     5.0.6
cryptography                  35.0.0
cycler                        0.11.0
cytoolz                       0.11.2
dask                          2022.1.1
debugpy                       1.5.1
decorator                     5.1.0
defusedxml                    0.7.1
Deprecated                    1.2.13
dipy                          1.4.1
ecdsa                         0.17.0
entrypoints                   0.3
envisage                      6.0.1
Flask                         2.0.2
fonttools                     4.29.1
frozenlist                    1.3.0
fsspec                        2022.1.0
google-auth                   2.6.0
greenlet                      1.1.2
h5py                          3.3.0
idna                          2.8
imagecodecs                   2021.11.20
imageio                       2.15.0
imageio-ffmpeg                0.4.5
importlib-metadata            4.8.1
importlib-resources           5.4.0
ipycanvas                     0.10.2
ipyevents                     2.0.1
ipykernel                     6.4.2
ipython                       7.29.0
ipython-genutils              0.2.0
ipyvtklink                    0.2.2
ipywidgets                    7.6.5
itsdangerous                  2.0.1
jedi                          0.18.0
Jinja2                        3.0.2
joblib                        1.1.0
json5                         0.9.5
jsonschema                    4.2.1
jupyter-client                7.0.6
jupyter-core                  4.9.1
jupyter-desktop-server        0.1.3
jupyter-server                1.11.2
jupyter-server-proxy          3.2.1
jupyter-telemetry             0.1.0
jupyterhub                    1.4.2
jupyterlab                    3.2.2
jupyterlab-pygments           0.1.2
jupyterlab-server             2.8.2
jupyterlab-widgets            1.0.2
kafka-python                  2.0.2
kazoo                         2.5.0
kiwisolver                    1.3.2
kubernetes                    21.7.0
llvmlite                      0.36.0
locket                        0.2.0
loguru                        0.5.3
Mako                          1.1.5
mamba                         0.17.0
Markdown                      3.3.6
MarkupSafe                    2.0.1
matplotlib                    3.5.1
matplotlib-inline             0.1.3
mayavi                        4.7.2
mffpy                         0.7.0
mistune                       0.8.4
mne                           0.24.1
mne-qt-browser                0.1.7
mock                          4.0.3
multidict                     6.0.2
munkres                       1.1.4
nbclassic                     0.3.4
nbclient                      0.5.4
nbconvert                     6.2.0
nbformat                      5.1.3
nest-asyncio                  1.5.1
networkx                      2.6.3
nibabel                       3.2.2
nilearn                       0.9.0
notebook                      6.4.5
numba                         0.53.1
numexpr                       2.8.0
numpy                         1.22.2
oauthlib                      3.1.1
packaging                     21.2
pamela                        1.0.0
pandas                        1.4.0
pandocfilters                 1.5.0
panel                         0.12.6
param                         1.12.0
parso                         0.8.2
partd                         1.2.0
patsy                         0.5.2
pexpect                       4.8.0
pickleshare                   0.7.5
Pillow                        9.0.1
pip                           22.0.3
pooch                         1.6.0
prometheus-client             0.12.0
prompt-toolkit                3.0.22
psutil                        5.9.0
ptyprocess                    0.7.0
pyaml                         21.10.1
pyasn1                        0.4.8
pyasn1-modules                0.2.8
pycosat                       0.6.3
pycparser                     2.21
pyct                          0.4.8
pycurl                        7.44.1
pydicom                       2.2.2
pyface                        7.3.0
Pygments                      2.10.0
PyJWT                         1.7.1
pykafka                       2.8.0
pyOpenSSL                     19.0.0
pyparsing                     2.4.7
PyQt5                         5.12.3
PyQt5_sip                     4.19.18
PyQtChart                     5.12
pyqtgraph                     0.12.3
PyQtWebEngine                 5.12.1
pyrsistent                    0.18.0
PySocks                       1.6.8
pysurfer                      0.11.0
python-dateutil               2.8.2
python-jose                   3.3.0
python-json-logger            2.0.1
python-keycloak               0.24.0
python-picard                 0.7
pytz                          2021.3
pyvista                       0.33.2
pyvistaqt                     0.7.0
pyviz-comms                   2.1.0
PyWavelets                    1.2.0
PyYAML                        6.0
pyzmq                         22.3.0
QtPy                          2.0.1
requests                      2.27.1
requests-oauthlib             1.3.1
rsa                           4.8
ruamel.yaml                   0.16.13
ruamel.yaml.clib              0.2.2
ruamel-yaml-conda             0.15.80
scikit-image                  0.19.1
scikit-learn                  1.0.2
scipy                         1.8.0
scooby                        0.5.11
Send2Trash                    1.8.0
sentry-sdk                    0.14.4
setuptools                    58.5.3
simpervisor                   0.4
six                           1.12.0
sniffio                       1.2.0
SQLAlchemy                    1.4.26
statsmodels                   0.13.2
tables                        3.6.1
tabulate                      0.8.9
terminado                     0.12.1
testpath                      0.5.0
threadpoolctl                 3.1.0
tifffile                      2022.2.9
toolz                         0.11.2
tornado                       6.1
tqdm                          4.62.3
traitlets                     5.1.1
traits                        6.3.2
traitsui                      7.2.0
typing_extensions             4.0.1
unicodedata2                  14.0.0
urllib3                       1.26.8
vtk                           9.0.3
wcwidth                       0.2.5
webencodings                  0.5.1
websocket-client              0.57.0
websockify                    0.10.0
Werkzeug                      2.0.3
wheel                         0.37.0
widgetsnbextension            3.5.2
wrapt                         1.13.3
xlrd                          2.0.1
yarl                          1.7.2
zipp                          3.6.0

I have had the same difficulty trying to use Ubuntu 20.04 in WSL. With ipyvtklink, I can get pyvista code to work but mne just displays a black image.

I then tried using Ubuntu 20.04 in VirtualBox and got the same black image running code directly.

I created an environment as follows

wget https://raw.githubusercontent.com/mne-tools/mne-python/main/environment.yml
conda env update --name mne --file environment.yml
conda activate mne

In this environment, the following works as expected

import pyvista as pv
cyl = pv.Cylinder()
p = pv.Plotter()
p.add_mesh(cyl)
p.show()

but this produces a black figure

import os.path as op

import mne

sample_path = mne.datasets.sample.data_path()
subjects_dir = op.join(sample_path, 'subjects')
subject_id = 'fsaverage'

brain = mne.viz.Brain(subject_id, hemi='both', surf='pial', subjects_dir=subjects_dir)
brain.show()
python -c "import mne; mne.sys_info()"
/home/larry/miniconda3/envs/mne/lib/python3.9/site-packages/mne/utils/check.py:728: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  if LooseVersion(version) >= LooseVersion('5.10'):
/home/larry/miniconda3/envs/mne/lib/python3.9/site-packages/mne/utils/check.py:728: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  if LooseVersion(version) >= LooseVersion('5.10'):
Platform:       Linux-5.13.0-30-generic-x86_64-with-glibc2.31
Python:         3.9.10 | packaged by conda-forge | (main, Feb  1 2022, 21:24:11)  [GCC 9.4.0]
Executable:     /home/larry/miniconda3/envs/mne/bin/python
CPU:            x86_64: 4 cores
Memory:         7.8 GB

mne:            0.24.1
numpy:          1.22.2 {blas=NO_ATLAS_INFO, lapack=lapack}
scipy:          1.8.0
matplotlib:     3.5.1 {backend=QtAgg}

sklearn:        1.0.2
numba:          0.53.1
nibabel:        3.2.2
nilearn:        0.9.0
dipy:           1.4.1
cupy:           Not found
pandas:         1.4.1
mayavi:         4.7.2
pyvista:        0.33.2 {OpenGL 4.5 (Core Profile) Mesa 21.2.6 via llvmpipe (LLVM 12.0.0, 256 bits)}
pyvistaqt:      0.7.0
ipyvtklink:     0.2.2
vtk:            9.0.3
PyQt5:          5.12.3
ipympl:         Not found
mne_qt_browser: 0.2.1
pooch:          v1.6.0

I’d suggest that you take a look at GitHub - mne-tools/mne-docker: Repository for mne docker images as a starting place for docker and singularity images. And also a place to incorporate any improvements that you might come up with.

To solve the blank screen issue, I’d try setting this env variable

    MNE_3D_OPTION_ANTIALIAS=false

Not sure if it solves issues in jupyter notebooks but it fixes blank screenshots in my code.

Thanks. That fixes both for me. Is there some way to add this to the linux installation instructions?

Thanks that fixed on my env as well.

To wrap this up, the following snippet works on my headless Jupyter notebook:

import os
import mne
import pyvista as pv
os.environ["MNE_3D_OPTION_ANTIALIAS"] = "false"

from pyvista.utilities import xvfb
xvfb.start_xvfb()

mne.viz.set_3d_backend("notebook")

sample_data_folder = mne.datasets.sample.data_path()
subjects_dir = os.path.join(sample_data_folder, 'subjects')
Brain = mne.viz.get_brain_class()
brain = Brain('sample', hemi='lh', surf='pial',
              subjects_dir=subjects_dir, size=(800, 600))
brain.add_annotation('aparc.a2009s', borders=False)

The rendering is not interactive though and does not work on Jupyterlab (no error but no rendering shown).

FYI I’ve opened this issue to discuss a better default for MNE_3D_OPTION_ANTIALIAS:

1 Like

A post was split to a new topic: Using MNE with 3D visualization on a headless server