Help building local documentation

đź’¬ Support & Discussions
1
  • MNE version: development
  • operating system: e.g. Windows 11 WSL2

Hello! I am making some modifications to the mne documentation and I am having a bit of trouble building the local documentation. I have already installed the required dependencies (requirements_doc.txt, graphviz, optipng), but after I run make html-noplot, I get the fallowing error:


sphinx-build -D plot_gallery=0 -b html -d _build/doctrees -nWT --keep-going . _build/html
Running Sphinx v7.2.6
Building documentation for MNE 1.5.1 (/home/woess/mambaforge/envs/mnedev/lib/python3.11/site-packages/mne/__init__.py)
Using pyvistaqt 3d backend.

/home/woess/mambaforge/envs/mnedev/lib/python3.11/site-packages/sphinxcontrib/youtube/utils.py:9: RemovedInSphinx80Warning: The alias 'sphinx.util.status_iterator' is deprecated, use 'sphinx.util.display.status_iterator' instead. Check CHANGES for Sphinx API modifications.
  from sphinx.util import logging, status_iterator

Traceback (most recent call last):
  File "/home/woess/mambaforge/envs/mnedev/lib/python3.11/site-packages/sphinx/registry.py", line 447, in load_extension
    mod = import_module(extname)
          ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/woess/mambaforge/envs/mnedev/lib/python3.11/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 940, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/home/woess/workspace/git_projects/mne-python/doc/sphinxext/mne_substitutions.py", line 6, in <module>
    from mne._fiff.pick import (
ModuleNotFoundError: No module named 'mne._fiff'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/woess/mambaforge/envs/mnedev/lib/python3.11/site-packages/sphinx/cmd/build.py", line 293, in build_main
    app = Sphinx(args.sourcedir, args.confdir, args.outputdir,
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/woess/mambaforge/envs/mnedev/lib/python3.11/site-packages/sphinx/application.py", line 233, in __init__
    self.setup_extension(extension)
  File "/home/woess/mambaforge/envs/mnedev/lib/python3.11/site-packages/sphinx/application.py", line 406, in setup_extension
    self.registry.load_extension(self, extname)
  File "/home/woess/mambaforge/envs/mnedev/lib/python3.11/site-packages/sphinx/registry.py", line 450, in load_extension
    raise ExtensionError(__('Could not import extension %s') % extname,
sphinx.errors.ExtensionError: Could not import extension mne_substitutions (exception: No module named 'mne._fiff')

Extension error:
Could not import extension mne_substitutions (exception: No module named 'mne._fiff')
make: *** [Makefile:54: html-noplot] Error 2

I have already tried running make clean and after I run make show nothing is displays in the browser. I would greatly appreciate any help with this. Thank you so much!

2

You should clone the github repository, GitHub - mne-tools/mne-python: MNE: Magnetoencephalography (MEG) and Electroencephalography (EEG) in Python, install it in editable mode, pip install -e path/to/git/clone in the environment (after removing the 1.5.1 version) and then run the doc build with make html-noplot.

EDIT: And if in a conda environment, I think you should run pip install -e --no-deps path/to/git/clone after installing all the dependencies through the environment.yml file.

mne._fiff is a module introduced in the 1.6 version.

Mathieu

3

Ah okay, thank you for your response! I guess I assumed that if I fetched from upstream that my development environment would update to the latest version. Is that not the case @mscheltienne?

4

I think I know what you did:

  • fetch / pull from upstream which did retrieve the up-to-date files
  • make html-noplot from those updated files
  • … in an environment with an existing install of mne, i.e. with the up-to-date files not “installed”.

Could you do: python -c "import mne; print (mne.__file__)"?
It should print the path to mne/__init__.py from your clone repository.

5

Sure thank you! @mscheltienne
Here is the result:
/home/woess/workspace/git_projects/mne-python/mne/__init__.py

Also, just to clear up any confusion. I have a package in: /home/woess/mambaforge/envs/mnedev/lib/python3.11/site-packages/mne
and in /home/woess/workspace/git_projects/mne-python/mne
The one in git_projects is a git repo, but the one in envs is not. Did I do something wrong when installing and the one in /home/woess/mambaforge/envs/mnedev/ should be deleted?

6

Ok, so that is correct. When you did python -c "import mne; print (mne.__file__)" did you do it from the same environment as when you did make html-noplot?
The output from sphinx is clearly pointing to the second MNE installation:

Building documentation for MNE 1.5.1 (/home/woess/mambaforge/envs/mnedev/lib/python3.11/site-packages/mne/__init__.py)

A safe approach is to create a fresh environment, install the MNE dependencies, make sure MNE is not installed and if it is remove it (I think it’s present in the environment.yml file), and then install the git clone in editable install with the no deps flag.

7

It worked! Thanks so much @mscheltienne! Would you mind explaining what the purpose of having both packages are? Also, do you know if the package will get updated inside /home/woess/mambaforge/envs/mnedev/lib/python3.11/site-packages/mne or will I have to install it each time a new version is released like I did here?

One more thing, was my mistake not cloning to the same folder that the original package was in? It seems like they may have done this on the contribution guide:

Then make a local clone of your remote fork (origin):

$ cd $INSTALL_LOCATION 
$ git clone https://github.com/$GITHUB_USERNAME/mne-python.git

Finally, set up a link between your local clone and the official repository (upstream) and set up git diff to work properly:

$ cd mne-python 
$ git remote add upstream https://github.com/mne-tools/mne-python.git 
$ git fetch --all 
$ git config --local blame.ignoreRevsFile .git-blame-ignore-revs

Now we’ll remove the stable version of MNE-Python and replace it with the development version (the clone we just created with git). Make sure you’re in the correct environment first (conda activate mnedev), and then do:

$ cd $INSTALL_LOCATION/mne-python # make sure we're in the right folder 
$ conda remove --force mne # the --force avoids dependency checking 
$ pip install -e .
8

I had a look through the contributing guidelines, and that’s what I guessed in my previous comment. It says:

conda env create --file environment.yml --name mnedev
conda activate mnedev
conda remove --force mne  # the --force avoids dependency checking
pip install -e path/to/git/clone

So let’s dissect those lines:

  • conda env create --file environment.yml --name mnedev: create a new environment named mnedev using the file environmnent.yml to install dependencies. The file is define here and includes mne on line 45. Thus, running this command does install the latest release of MNE in the environment, i.e. as of now 1.5.1. I am not a conda user, thus I’m not sure what the purpose of including mne in the environment.yml file is, but I think that’s the part you were confused about: MNE stable was installed as part of this first line.

  • conda activate mnedev: activate the environment.

  • conda remove --force mne: uninstall MNE from the environment (since we want to install the development version instead of the stable version installed by the first line), without removing the dependencies (that’s probably what the --force flag is doing based on the comment).

  • pip install -e path/to/git/clone: Install in editable mode the clone of the MNE-Python in the environment.

Why do we do all that? Mostly because mixing conda and pip install is a bad idea (messes up the dependency checking), thus the goal is to first create an environment with all the dependencies that MNE requires but without MNE, and second to install MNE in editable mode.

@drammock can maybe add a comment as to why mne is listed in the environment.yml file.

You should not need to re-install it each time a new version is released. But it can happen that some code depends on version tags and that you do not have the correct version tag installed even if the files have been updated with a git fetch/pull. In that case, all you need to do is to re-install the editable package:

pip uninstall mne
pip install -e path/to/git/clone

Mathieu

2 Likes
9

because the environment file is also used by people who want the stable version and prefer conda over pip.

10

It makes sense but seems also inconvenient. Is there a way for people to fetch this file with a conda / mamba command when creating the environment or do they have to manually fetch it from github? It seems strange to clone the repository to then install the stable version.

And what is the added value compared to installing the stable version from the feedstock with conda create --channel=conda-forge --name=mne mne?

11

Good question. I think nowadays it ought to be equivalent / preferred to install from feedstock in most cases. The case where the env file will be preferred is if our dependencies have changed but the changes aren’t yet released — i.e., exactly in the case where one is setting up a dev environment.

Aside: out of curiousity I tried both and compared what gets installed. Installing from feedstock installs 402 packages while installing from the environment file installs 432 packages, but the differences are not what I would have expected:

archspec
boltons
cloudpickle
conda
conda-package-handling
conda-package-streaming
cryptography
fmt
ipyevents
ipympl
jsonpatch
libarchive
libmamba
libmambapy
libsolv
mamba
pluggy
pybind11-abi
pybv
pycosat
pyopenssl
reproc
reproc-cpp
ruamel.yaml
ruamel.yaml.clib
spyder-kernels
truststore
wurlitzer
yaml-cpp
zstandard

Of these, ipyevents and ipympl and pybv might fall under the umbrella of “new dependencies since last stable release” (at least I think so?) The rest are puzzling, many are dependencies of mamba but I don’t see why we’re installing mamba into the env.

cc @larsoner we should discuss environment file cleanup.

12

Agree, and to me that pushes environment.yml file towards a dev-only file thus without mne listed in; while stable installs should go through the feedstock.

Yep… spyder-kernels?! That’s neither from mamba or mne :confused:

13

yeah, we may want to move in that direction. The env file has been around since long before we even had the feedstock option, I guess it’s time to prune some cruft. On the other hand, nowadays we can do mamba create -n foo --only-deps mne (since mamba 0.18.0 almost two years ago!) so maybe we can ditch the env file altogether?

EDIT: nevermind, there’s still the case of “deps changed since the last release” that we need to support.