mne-bids-pipeline basic error preprocessing

💬 Support & Discussions
1

Hello
I am new to mne python (and python itself). For reproducibility, I have decided to start using the men-bids-pipeline.

I am using MacOS Montgomery 12.2.1.

This is how I have installed mne-bids and the dependencies:
conda create --name mne --channel conda-forge mne-bids
pip install -r https://raw.githubusercontent.com/mne-tools/mne-bids-pipeline/main/requirements.txt

It is running on python 3.9.

I have customised the config.py, and it starts to run just fine using the python run.py --config=config.py --steps=preprocessing command. Then I am getting the following error:

    spec.loader.exec_module(custom_cfg)
  File "/Users/Franziska/Dropbox/PredictiveLanguageCoding_EEG/EEG_analysis/mne-bids-pipeline-main-3/config.py", line 2006, in <module>
    spec.loader.exec_module(custom_cfg)
  File "/Users/Franziska/Dropbox/PredictiveLanguageCoding_EEG/EEG_analysis/mne-bids-pipeline-main-3/config.py", line 2006, in <module>
    spec.loader.exec_module(custom_cfg)
  File "/Users/Franziska/Dropbox/PredictiveLanguageCoding_EEG/EEG_analysis/mne-bids-pipeline-main-3/config.py", line 2006, in <module>
    spec.loader.exec_module(custom_cfg)
  File "/Users/Franziska/Dropbox/PredictiveLanguageCoding_EEG/EEG_analysis/mne-bids-pipeline-main-3/config.py", line 1927, in <module>
    coloredlogs.install(
  File "/Users/Franziska/opt/miniconda3/envs/mne/lib/python3.10/site-packages/coloredlogs/__init__.py", line 537, in install
    handler.setFormatter(formatter_type(**formatter_options))
  File "/Users/Franziska/opt/miniconda3/envs/mne/lib/python3.10/site-packages/coloredlogs/__init__.py", line 1025, in __init__
    kw = dict(fmt=self.colorize_format(fmt, style), datefmt=datefmt)
  File "/Users/Franziska/opt/miniconda3/envs/mne/lib/python3.10/site-packages/coloredlogs/__init__.py", line 1081, in colorize_format
    result.append(ansi_wrap(
  File "/Users/Franziska/opt/miniconda3/envs/mne/lib/python3.10/site-packages/humanfriendly/terminal/__init__.py", line 293, in ansi_wrap
    start_sequence = ansi_style(**kw)
  File "/Users/Franziska/opt/miniconda3/envs/mne/lib/python3.10/site-packages/humanfriendly/terminal/__init__.py", line 236, in ansi_style
    elif isinstance(color_value, numbers.Number):
  File "/Users/Franziska/opt/miniconda3/envs/mne/lib/python3.10/abc.py", line 119, in __instancecheck__
    return _abc_instancecheck(cls, instance)
RecursionError: maximum recursion depth exceeded in comparison

It seems it is looking for a custom_cfg file. As you know, in the relevant section in the config file it reads:

# For testing a specific dataset, create a Python file with a name of your
# liking (e.g., ``mydataset-template-config.py``), and set an environment
# variable ``MNE_BIDS_STUDY_CONFIG`` to that file

.

I don’t know what I have to do here. What does “and set an environment variable MNE_BIDS_STUDY_CONFIG to that file.” mean and how am I doing that?

Also is there a detailed documentation of how to work with the pipeline? I am wondering as I am only half way through the config file 


Thanks a lot for your help. Much appreciated!
Fraenni

2

Hello @fraenni, thanks for your interest in the pipeline! I suppose the setup instructions need some more work :frowning: What you need to do is:

  • create a copy of the original config.py
  • delete everything below the lines
  ###############################################################################
  #                                                                             #
  #                      CUSTOM CONFIGURATION ENDS HERE                         #
  #                                                                             #
  ###############################################################################
  • adjust the configuration options – you can delete everything where you’re not changing the default
  • To run the pipeline from the command line, do something like:
MNE_BIDS_STUDY_CONFIG=/path/to/custom_config.py python run.py

Best wishes,
Richard

3

Hello Richard,
Thank you very much for your advice. Very helpful!

Unfortunately, it is not quite working (yet), but this might be due to a misunderstanding.

I have done the following: I took my adapted my config file, and deleted everything after the marked area.

I then run the preprocessing part of the pipeline from the command line using the following command, as suggested:

MNE_BIDS_STUDY_CONFIG=/Users//PredictiveLanguageCoding_EEG/EEG_analysis/mne-bids-pipeline-main-3/config.py python run.py --steps=preprocessing

This produced the following error:

(mne) mne-bids-pipeline-main-3 % MNE_BIDS_STUDY_CONFIG=/Users/Franziska/Dropbox/PredictiveLanguageCoding_EEG/EEG_analysis/mne-bids-pipeline-main-3/config.py python run.py --steps=preprocessing
ERROR: Missing required flags: {‘config’}
Usage: run.py
optional flags: --config
required flags: --steps | --root_dir | --subject | --session |
–task | --run | --interactive | --n_jobs

For detailed information on this command, run:
run.py --help

I therefore included --config=/Users/PredictiveLanguageCoding_EEG/EEG_analysis/mne-bids-pipeline-main-3/config

Again an error

ModuleNotFoundError: No module named ‘config’

I installed the module config, and run the full command again, however, now receiving an error, for something that is no longer defined in the config file after deleting everything after the marked area.

(mne)mne-bids-pipeline-main-3 % MNE_BIDS_STUDY_CONFIG=/Users/Franziska/Dropbox/PredictiveLanguageCoding_EEG/EEG_analysis/mne-bids-pipeline-main-3/config_plang.py python run.py --config=/Users/PredictiveLanguageCoding_EEG/EEG_analysis/mne-bids-pipeline-main-3/config_plang.py --steps=preprocessing
Traceback (most recent call last):
File “/Users/PredictiveLanguageCoding_EEG/EEG_analysis/mne-bids-pipeline-main-3/run.py”, line 246, in
fire.Fire(process)
File “/Users/opt/miniconda3/envs/mne/lib/python3.10/site-packages/fire/core.py”, line 141, in Fire
component_trace = _Fire(component, args, parsed_flag_args, context, name)
File “/Users/opt/miniconda3/envs/mne/lib/python3.10/site-packages/fire/core.py”, line 466, in _Fire
component, remaining_args = _CallAndUpdateTrace(
File “/Users/opt/miniconda3/envs/mne/lib/python3.10/site-packages/fire/core.py”, line 681, in _CallAndUpdateTrace
component = fn(*varargs, **kwargs)
File “/Users/PredictiveLanguageCoding_EEG/EEG_analysis/mne-bids-pipeline-main-3/run.py”, line 196, in process
SCRIPT_MODULES = _get_script_modules(
File “/Users/PredictiveLanguageCoding_EEG/EEG_analysis/mne-bids-pipeline-main-3/run.py”, line 71, in _get_script_modules
from scripts import init
File “/Users/PredictiveLanguageCoding_EEG/EEG_analysis/mne-bids-pipeline-main-3/scripts/init/init.py”, line 1, in
from . import _00_init_derivatives_dir
File “/Users/PredictiveLanguageCoding_EEG/EEG_analysis/mne-bids-pipeline-main-3/scripts/init/_00_init_derivatives_dir.py”, line 18, in
from config import gen_log_kwargs, on_error, failsafe_run
ImportError: cannot import name ‘gen_log_kwargs’ from ‘config’ (/Users/PredictiveLanguageCoding_EEG/EEG_analysis/mne-bids-pipeline-main-3/config.py)

I of course tried the full command including setting it to the environment using the config file without deleting anything. As expected it produced the command as in the first post.

There is probably something obvious I am still doing wrong, if you had some more advice that would be great!
Thanks & best,
F.

4

I’m sorry, @fraenni; I didn’t take enough time to draft my response and accidentally led you down the wrong path.

This environment variable need not to be set. Instead, passing the path to the config file via --config is sufficient.

You should be able to run:

python run.py --config my_config.py

and that should be it.

Please uninstall the config package you installed; it is not required (and may lead to conflicts).

Let me know how it goes!

Best wishes,
Richard

5

Thanks again, Richard :slight_smile: I uninstalled config, and run the command:

(mne) mne-bids-pipeline-main-3 % python run.py --config=/Users/PredictiveLanguageCoding_EEG/EEG_analysis/mne-bids-pipeline-main-3/config_plang.py --steps=preprocessing
Traceback (most recent call last):
File “/Users/PredictiveLanguageCoding_EEG/EEG_analysis/mne-bids-pipeline-main-3/run.py”, line 246, in
fire.Fire(process)
File “/Users/opt/miniconda3/envs/mne/lib/python3.10/site-packages/fire/core.py”, line 141, in Fire
component_trace = _Fire(component, args, parsed_flag_args, context, name)
File “/Users/opt/miniconda3/envs/mne/lib/python3.10/site-packages/fire/core.py”, line 466, in _Fire
component, remaining_args = _CallAndUpdateTrace(
File “/Users/opt/miniconda3/envs/mne/lib/python3.10/site-packages/fire/core.py”, line 681, in _CallAndUpdateTrace
component = fn(*varargs, **kwargs)
File “/Users/PredictiveLanguageCoding_EEG/EEG_analysis/mne-bids-pipeline-main-3/run.py”, line 196, in process
SCRIPT_MODULES = _get_script_modules(
File “/Users/PredictiveLanguageCoding_EEG/EEG_analysis/mne-bids-pipeline-main-3/run.py”, line 71, in _get_script_modules
from scripts import init
File “/Users/PredictiveLanguageCoding_EEG/EEG_analysis/mne-bids-pipeline-main-3/scripts/init/init.py”, line 1, in
from . import _00_init_derivatives_dir
File “/Users/PredictiveLanguageCoding_EEG/EEG_analysis/mne-bids-pipeline-main-3/scripts/init/_00_init_derivatives_dir.py”, line 17, in
import config
ModuleNotFoundError: No module named ‘config’

if I then reinstall the config module, I am back at the error: ImportError: cannot import name ‘gen_log_kwargs’ from ‘config’ (/Users/PredictiveLanguageCoding_EEG/EEG_analysis/mne-bids-pipeline-main-3/config.py) described in the last post.

in terms of my installation, I am only using the following, nothing else: python 3.10 (updated) and

conda create --name mne --channel conda-forge mne-bids
pip install -r https://raw.githubusercontent.com/mne-tools/mne-bids-pipeline/main/requirements.txt

Thanks again! BW. F.

6

Please don’t reinstall the config package! Please remove it again!

This is one of the super annoying things about Python packaging, it’s not clear whether a package or module is imported from a “local” (current directory) or “global” (e.g., system-wide Python) installation.

The config import the pipeline tries to do is for config.py in the pipeline directory. Did you perhaps remove or rename this file? It must not be touched by the user! To create your personalized config file, the original config.py must be copied; the copy can be modified; but the original config.py must remain untouched!

7

Alright, I got it now. I kept the original config-file as “config.py” in the pipeline folder and stored the customise file in a separate folder with a different name. Now the pipeline is starting (producing other errors, but those I will hopefully figure out myself!)
Thanks a lot for the help :smiley:
BW. F

8

@fraenni Please feel free to post new topics if any other problems come up.

Good luck!

9

Hi @richard, I also got this error and had to trim my copy of the config file. I couldn’t find this mentioned in the documentation though. I think this should be added to the documentation so people don’t have to dig up this forum thread. Or perhaps are you working on a better solution in the meantime?

Thanks,
Måté

10

Hello @allermat, we’re in the process of fixing this, I hope we’ll have something ready sometime next week! I will update this topic once we’re done.

Thanks!

Richard

1 Like