Change in data shape after the picking

đź’¬ Support & Discussions
1

My group and I are doing a bachelors project using multiple datasets including the BIDS-NIRS-Tapping made by Doktor Luke.
We have made a data loading/preprocessing class which has worked just fine until now. However, suddenly our second dimension of the data was reduced from 56 to 40. We haven’t changed any code, and believe the change stem from the section copied below. Could this change be due to an update, or could it be us having made a change without being aware?
We tried running an earlier version of the script, which also returns the new data format…

We use version 1.6.1 on a Windows 11 operating system.

Thanks in advance!

if self.short_channel_correction:
raw_od = mne.preprocessing.nirs.optical_density(raw_intensity)
raw_od = mne_nirs.signal_enhancement.short_channel_regression(raw_od)

            picks = mne.pick_types(raw_intensity.info, meg=False, fnirs=True)
            dists = mne.preprocessing.nirs.source_detector_distances(
                raw_intensity.info, picks=picks
            )
            raw_intensity.pick(picks[dists > 0.01])
        else:
            picks = mne.pick_types(raw_intensity.info, meg=False, fnirs=True)
            dists = mne.preprocessing.nirs.source_detector_distances(
                raw_intensity.info, picks=picks
            )
            raw_intensity.pick(picks[dists > 0.01])
            raw_od = mne.preprocessing.nirs.optical_density(raw_intensity)
2

I’m not sure what you think is wrong here. Doing raw_intensity.pick(sel) will pick the channels in sel in-place, meaning that it will reduce the dimension of the data array in raw_intensity to (n_sel_channels, n_samples).

I’m guessing picks[dists > 0.01].size == 40. which is why you get 40 channels in raw_intensity.

Mathieu

2 Likes
3

That would make sense. What is worrying is that this has not happened in the three months we have been doing the project. From one day to the next (Without us changing any code) the dimensions were reduced.

4

Can you verify with older versions to see when this change was introduced?

5

Our first step in troubleshooting this issue was to check our Git repository from last week and run the code from that version. But even in that older version the dimensions were reduced. This is what led us to the assumption that the functionality of “picks” had been changed or fixed.

6

The functionality of picks has not been changed or modified. I don’t believe any recent bugfix would have resulted in that. One way to troubleshoot is to create an environment with:

  • your git repository from a week ago (or older)
  • MNE version 1.3, or 1.4, or 1.5
7

I meant older versions of MNE, not older versions of your code.

8

I believe there was a recent change to picking behavior in that we now include bad channels by default or so…? Could you maybe try with pick(..., exclude="bads")? Or maybe this happens during pick_types(). I would try with different combinations of exclude="bads" and exclude=() passed to pick_types() and pick(). I might be entirely mistaken, though.

1 Like
10

Thanks, we just tried this, but it doesn’t seem to explain it…

11

Well, if you tried to downgrade MNE to an older and version and to run your code which was not reducing the dimensionality, you replicated the condition which were not reducing the dimensionality.

The only logical explanations are that (1) you did not notice previously that the dimension was reduced by the pick operation, or (2) that a parameter in your code was changed leading to a different pick selection (e.g. this dists > 0.01 condition).

Both options are coherent with the pick mechanism which always selected the channels in-place, and thus reduce the dimensionality of the underlying array.

Mathieu

12

Can you try to run your code unmodified but with MNE 1.5 maybe? To see if you get back the old behavior?

13

We’ve identified the bug! It turns out that after implementing the short channel correction, we never tried to reset the parameter back to False. Upon doing so, we discovered that we had mistakenly reversed the order of the correction and channel picking, resulting in continued data processing where the short channels remained unremoved. Apologies for the inconvenience, but thank you very much for your contributions and assistance!

1 Like
14

Hah! Great to hear you figured it out eventually!

1 Like
15

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.