Enable loading custom models in the QuPath extension
constantinpape opened this issue ยท 11 comments
Hi @ksugar,
we would like to implement loading of custom SAM models (e.g. from micro_sam
) in the QuPath plugin.
This should be easy on the server side, and I think my PR in ksugar/samapi#11 already adds everything that's necessary there.
But I am not so sure what needs to be changed here. From a quick check I think I would need to add the option for a custom URL here:
qupath-extension-sam/src/main/java/org/elephant/sam/SAMCommand.java
Lines 101 to 102 in 609289d
Is that correct, or is there some other relevant place in the code? Also, how should the default (no custom URL) be represented? In python it's just None
.
Thanks!
That sounds great! It looks like the use of an enum
for SAMModel
would need to change. The way we did it in the WSInfer extension is to use a String
or alternative non-enum class instead, and populate the combobox by querying the server for the list of available models on the server.
Then hopefully adding more models becomes easy, and doesn't require any change to the extension.
Hi @constantinpape, thank you for the PR!
I checked your PR and found that it is a good idea to handle all models (incl. currently available models) with a single interface with just the checkpoint_url
argument.
On the client side (i.e., QuPath), in addition to the preset model options (vit_h
, vit_l
, vit_b
and vit_m
), the user will have the option to add a custom model by specifying the URL with a nickname. Once it is added, it will remain until the user explicitly deletes it. QuPath's PathPrefs
will manage a list of custom urls.
I will work on it later today.
QuPath's
PathPrefs
will manage a list of custom urls.
I'm not sure PathPrefs
will be very good at that by default. The trouble is that Java's Preferences
class doesn't support arrays/lists. It should still work if you don't mind squeezing everything into a single string, although this won't support extremely long lists (the maximum string length is 8192 characters).
My guess is that the client supporting a list of models returned by the server would be easier to develop and manage, so that any extra URLs are handled when starting up the server.
Hi @petebankhead, thank you for the feedback.
I was expecting that PathPrefs.createPersistentJsonPreference()
can handle arrays/lists of String
but it seems that it is not the case.
Following your suggestion, I will implement an endpoint for getting a list of the available models on the server side.
I forgot about PathPrefs.createPersistentJsonPreference() :)
Probably it should work (although I guess maybe doesn't).
Thanks for the quick replies @ksugar and @petebankhead! From what I understand you will take care of this for now @ksugar :). Let me know if you need any more info, or once it's ready to be tested!
Hi @constantinpape and @petebankhead, I have implemented the function for loading custom model weights from URL.
Please try qupath-extension-sam-0.4.0-SNAPSHOT.jar with the updated version of samapi from the dev
branch.
python -m pip install -U git+https://github.com/ksugar/samapi.git@dev
Since this version, it is important to launch the server with --workers 2
(or more) to enable cancellation of a download of a weights file.
export PYTORCH_ENABLE_MPS_FALLBACK=1 # Required for running on Apple silicon
uvicorn samapi.main:app --workers 2
The relevant documentation is found here.
https://github.com/ksugar/qupath-extension-sam/tree/dev#register-sam-weights-from-url
Thanks @ksugar ! I will try it out as soon as possible and let you know what I find.
I tested it now @ksugar and it all works as expected!
Thank you @constantinpape for testing it! I will release it soon.
I'm thinking that it would be helpful to have a catalog for weights in README so that the users can find them easily.
How do you think?
Type | Name (customizable) | URL | Citation |
---|---|---|---|
vit_h | vit_h_lm | https://zenodo.org/record/8250299/files/vit_h_lm.pth?download=1 | Archit, A. et al. Segment Anything for
Microscopy. bioRxiv 2023. doi:10.1101/2023.08.21.554208 https://github.com/computational-cell-analytics/micro-sam |
vit_b | vit_b_lm | https://zenodo.org/record/8250281/files/vit_b_lm.pth?download=1 | |
vit_h | vit_h_em | https://zenodo.org/record/8250291/files/vit_h_em.pth?download=1 | |
vit_b | vit_b_em | https://zenodo.org/record/8250260/files/vit_b_em.pth?download=1 |
Thank you @constantinpape for testing it! I will release it soon. I'm thinking that it would be helpful to have a catalog for weights in README so that the users can find them easily. How do you think?
Type Name (customizable) URL Citation
vit_h vit_h_lm https://zenodo.org/record/8250299/files/vit_h_lm.pth?download=1 Archit, A. et al. Segment Anything for Microscopy. bioRxiv 2023. doi:10.1101/2023.08.21.554208https://github.com/computational-cell-analytics/micro-sam
vit_b vit_b_lm https://zenodo.org/record/8250281/files/vit_b_lm.pth?download=1
vit_h vit_h_em https://zenodo.org/record/8250291/files/vit_h_em.pth?download=1
vit_b vit_b_em https://zenodo.org/record/8250260/files/vit_b_em.pth?download=1
Yes, that would be great!