NVIDIA/ai-assisted-annotation-client

Slicer: Automatic Segment Creation When Selecting A Model

SachidanandAlle opened this issue · 11 comments

SachidanandAlle commented

From @bsmarine

Open to other suggestions on optimal work flow.

Initial thought is to have the following:

  1. User clicks menu pull down for either "Auto Segmentation" or "AIAA"
  2. All available models are shown
  3. User selects desired model
  4. A Segmentation and Segment nodes are automatically created and named according to the labels that model provides (eg for hippocampus --> single hippocampus label; for liver and tumor model --> two labels: liver and tumor)
cpinter commented

Before getting started with this, I need to be able to use the extension, see #35

lassoan commented

@SachidanandAlle

  • Do you plan to keep the mechanism that allows filtering based on existing segment names?
  • How Clara internally identifies what is contained in each segment? Using standard DICOM terminologies (as it is done in DICOM segmentation objects) or just using some common English names?
SachidanandAlle commented

  1. Since AIAA server will serve many more models with specific/overlapping labels, we need to select a closest model available for a given label as a default selection

  2. Clara trained models are labeled by user (who trained it). I wish we can follow/enforce some medical standards to label it correctly. But for now, it is only a list of common English names.

http://aiaa_server:port/v1/models?label=brain&type=segmentation
http://aiaa_server:port/v1/models?label=brain+whole&type=segmentation
http://aiaa_server:port/v1/models?label=liver&type=segmentation

For a given label and type AIAA Server returns ALL models which satisfy any of the following conditions

  1. Exact Match
  2. Case-Insensitive Match
  3. Part of

Client will have additional flexibility to filter/pick the closest model among the result.

cpinter commented

Thanks! So what is it that I need to do exactly in terms of this ticket?

How about:

  • Use terminologies even if the server doesn't currently support it. I think it will be extremely useful later. There is quite some literature about the hurdles of manually named segmentations and the difficulties that arise when trying to use it for training. What I'd need to do for this is to allow terminologies to be selected without modifiers (left/right etc) if the parent has a valid ID.
  • When the model combobox selection changes, the effect would remove the empty segments, and create new ones with the terminologies specified (what about non-empty segments? warning? remove anyway?). We need to come up with a way this list of terminologies corresponding to a model can be queried, and I will need some help with this part.
bsmarine commented
cpinter commented

Yes, Slicer uses terminology dictionaries to make structure identification unambiguos. There is a shorter dictionary, and the full DICOM dictionary. I think we should use the latter. It would take time to add support for this on the server side, so that's why I need to know what to do in the inital implementation of this feature on the Slicer side.

By adding support, I mean having access to the terminology json files, being able to find an entry based on the IDs (a bit complex; this is what's needed to identify an entry: terminology context name + category ID triplet + type ID triplet + type modifier ID triplet + anatomic context name + anatomic region ID triplet + anatomic region modifier ID triplet), and have these entries or set of possible entries associated with one structure in the segmentation model.

Thanks for the comment on removal of segments.

lassoan commented

Yes, Clara API should use standard terminology (as defined in DICOM) as input filter criteria and in the segmentation results. Using plain English names would be ambiguous.

We can take care of Slicer's part. In fact, as you know, it is mostly done, @cpinter will just do some fine tuning to make things a bit simpler to use.

DICOM allows using a wide range of coding schemes (see here), but in clinical practice I only saw SNOMED-CT (which is free for use, both for publication in DICOM as well as by implementers and users of DICOM-compliant products and software globally).

cpinter commented

I added the option to select no modifier for a type. As far as I know this was the only feature that prevented pre-assigning standard terminologies to the existing models.

bsmarine commented

For the sake of testing/demo, could we use the json code below as @cpinter suggested to map model labels to DICOM standard terminology and automatically generate segments when a model is selected?

One catch, and this may represent a common issue in the future, the model for "hepatic vessel" in fact creates segmentations for both the portal and hepatic veins--two distinct anatomic structures. For the time being, perhaps it'd be easiest to just pick the first entry for that mapping.

{ "terminologyMap": {"liver": "Segmentation category and type - DICOM master list~SRT^T-D000A^Anatomical Structure~SRT^T-62000^Liver~^^~Anatomic codes - DICOM master list~^^~^^", "liver tumor": "Segmentation category and type - DICOM master list~SRT^M-01000^Morphologically Altered Structure~SRT^M-8FFFF^Neoplasm~^^~Anatomic codes - DICOM master list~SRT^T-62000^Liver", "hepatic vessel":[ "Segmentation category and type - DICOM master list~SRT^T-D000A^Anatomical Structure~SRT^T-48810^Portal Vein~^^~Anatomic codes - DICOM master list~^^~^^", "Segmentation category and type - DICOM master list~SRT^T-D000A^Anatomical Structure~SRT^T-48720^Hepatic Vein~^^~Anatomic codes - DICOM master list~^^~^^" ], "colon": "Segmentation category and type - DICOM master list~SRT^T-D000A^Anatomical Structure~SRT^T-59300^Colon~^^~Anatomic codes - DICOM master list~^^~^^", "hippocampus": "Segmentation category and type - DICOM master list~SRT^T-D000A^Anatomical Structure~SRT^T-A2570^Hippocampus~^^~Anatomic codes - DICOM master list~^^~^^" } }

bsmarine commented

From @cpinter :

"I think your map is alright for a quick&dirty solution. Adding it on the client side does not make much sense, as we have full access to the DICOM terminology json in Slicer, so we can identify any entry by the IDs very easily. This map belongs to the server side, but for maintainability, extensibility, and stability you should eventually add the actual Json file with means to parse it on the server (i.e. look-up and generating entries).

The way it would be easiest to implement this now (instead of adding the map in the script you linked on the client side) is if the server communicated the terminology entry instead of doing the English->Terminology translation within Slicer. Just because it's the server that tells Slicer what the segments are, and using plain English for communicating this to Slicer that in turn has to translate it to Terminology would be even dirtier than it needs to be, as it would still send the unmaintanable plain English terms through the connection instead of communicating immediately the way it is supposed to be."

Regarding how to assign terminology as an attribute to a segment:

"You can just set the terminology as a segment tag as it's done here, but with the string you generated."

@SachidanandAlle If/when there is an effort to incorporate standard terminologies into Clara perhaps we can add this json file with a lookup onto the server side as an initial test.

SachidanandAlle commented

I guess we can close this issue as the required actions are implemented as part of auto-segmentation action