QI2lab/OPM

Update napari-control to use new pymmcore-plus and napari-micromanager

dpshepherd opened this issue · 6 comments

dpshepherd commented

hi @AlexCoul -

There has been a lot of work over at pymmcore-plus and napari-micromanager on threading and cleaning up the shared memory model.

Adapting the OPM control code to take advantage of these changes might help clean up the intermittent acquisition errors in my custom Napari based control code.

I suggest looking through @ptbrown1729 fork of napari-micromanager for the SIM-ODT as a place to start. We should lay out a plan to build around that fork and make new OPM specific widgets instead of the custom Napari solution that I created.

We should also make the work general enough that we can control both the new low NA OPM and epi-fluorescence MERFISH scope with minimal effort.

I see this as:

  • OPM mirror scanning widget that setups the lasers, camera, and DAQ sequence then execute
  • OPM stage scanning widget that setups the stage, lasers, camera, and DAQ sequence then execute
  • OPM O2-O3 focus maintenance widget that setups the shutter, O3 stage, and algorithmic parameters for optimizing O2-O3 spacing
  • standard microscope stage tiling widget that setups stage, lasers, camera, and DAQ sequence then execute
  • iterative multiplexed chemistry widget that setups the flow controller and run appropriate scan type using user settings
AlexCoul commented
ptbrown1729 commented

One comment on my fork: right now the DAQ acquisition is all run in the main thread. So to avoid freezing the GUI this will need to run on a separate thread. I don't know if the way Doug is doing this in his control code can be used in napari-micromanager to accomplish this. But either way, that is one important problem I did not try to solve yet

ianhi commented

I just today implemented a way for users of pymmcore-plus to register an acquistion engine to be use by pymmcore-plus. (https://github.com/tlambert03/pymmcore-plus/pull/102) see docs here: https://pymmcore-plus.readthedocs.io/en/latest/examples/mda.html#registering-a-new-mda-engine

If you implement a custom acquisition engine (basically a blocking for loop with signals emitted) then you should be able to register it with the core instance and take advantage of core.run_mda running in a thread and thus not blocking the GUI.

Something like this:

from useq import MDASequence
from pymmcore_plus import CMMCorePlus

from pymmcore_plus.mda import MDAEngine, PMDAEngine
from pymmcore_plus.mda.events import QMDASignaler # qt version best when integrating with QT
# or
# from pymmcore_plus.mda.events import MDASignaler

core = CMMCorePlus.instance()
core.loadSystemConfiguration()

# from scratch
class from_scratch_engine(PMDAEngine):
    # in this class you need to integrate everything
    ...


# inheriting
# not perfectly designed for inheritance, but still handy.
class my_acq_engine(MDAEngine):
    def __init__(self) -> None:
        super().__init__(CMMCorePlus.instance())
        # any other setup you want to do

    def run(self, seq: MDASequence):
        # implement your own loop here
        # You should occasinally check self._canceled or self._paused
        ...


acq_eng = my_acq_engine()
core.register_mda_engine(acq_eng)

seq = ...

# this will run in a thread!
core.run_mda(seq)

I know that the MDAEngine isn't ideal for inheritance but it sitll would help get you started. You would need to override the run method: https://github.com/tlambert03/pymmcore-plus/blob/8f320b4d14571b942e75cd7f6e7079ae8f415249/pymmcore_plus/mda/_engine.py#L68

There's also some information about using threads in this example: https://pymmcore-plus.readthedocs.io/en/latest/examples/integration-with-qt.html though it needs to be updated to include the decorator used here: https://github.com/tlambert03/napari-micromanager/pull/114

Adapting the OPM control code to take advantage of these changes might help clean up the intermittent acquisition errors in my custom Napari based control code.

What errors were these? Do they look anything like: https://github.com/tlambert03/pymmcore-plus/issues/43 ?

dpshepherd commented

hi @ianhi ,

Interesting! I am not super familiar with callbacks, etc... Hopefully I'll have some time once the semester is over to look into this more.

The hacky code that I wrote runs into a weird memory type error. The basic outline is:

  • Launch RemoteMMCore before starting Napari
  • Start Napari with a set of widgets using magic-gui defined components
  • press button in GUI that calls function decorated as a thread_worker using Napari threading logic
    • loop n times (n>20)
    • open a "with" block accessing the RemoteMMCore
    • modify some hardware
    • run an acquisition using startSequenceAcquisition

Once you do that enough time (say 20 loop iterations), Python crashes and says that it is out of memory. Except I can't find any logs or other indication that memory is actually being used up.

If one acquires the same amount of data is acquired in one acquisition (no looping), no memory error occurs.

I know this is a bit vague and I haven't had the bandwidth to troubleshoot, so I don't have much more to say right now.

Edit: added more information that the function is decorated and called as a Napari thread_worker from the Napari GUI.

ianhi commented

Interesting! I am not super familiar with callbacks, etc... Hopefully I'll have some time once the semester is over to look into this more.

I'm thinking taht we will probably need a more concrete example (and a better inheritance model) to make this easier for people. When you do have the time to look into feel free to ask for guidance - can definitely be confusing. It will essnetially boil down to making calls like: self.events.frameReady.emit(img, Event) every time you take a new time.

Once you do that enough time (say 20 loop iterations), Python crashes and says that it is out of memory. Except I can't find any logs or other indication that memory is actually being used up.

yikes. Definitely sounds like something that would be fixed by switching away from RemoteMMCore. Where does that code live? I have a look an maybe suggest what the simplest (hopefully crash free) replacement would be.

The other thing if you're using sequenceAcquistion (especially with multiple cameras) you should definitely read this: micro-manager/mmCoreAndDevices#171

dpshepherd commented

The other thing if you're using sequenceAcquistion (especially with multiple cameras) you should definitely read this: micro-manager/mmCoreAndDevices#171

Yeah, we had to figure all that out because when I started the lab, we controlled a lot of scopes with Beanshell scripts to control the core instead of Python.

Excited to see where this all this goes!