The code uses images from the XAMI dataset (available on Github and HuggingFace). The images are astronomical observations from the Optical Monitor (XMM-OM) onboard the XMM-Newton X-ray mission of the European Space Agency (ESA).
Information about the XMM-OM can be found at:
Figure 1: The XAMI model combining a detector and segmentor, while freezing the detector model previously trained on the XAMI dataset.
git clone https://github.com/ESA-Datalabs/XAMI-model.git
cd XAMI-model
# creating the environment
conda env create -f environment.yaml
conda activate xami_model_env
# Install the package in editable mode
pip install -e .The dataset is splited into train and validation categories and contains annotated artefacts in COCO format for Instance Segmentation. We use multilabel Stratified K-fold (k=4) to balance class distributions across splits. We choose to work with a single dataset splits version (out of 4) but also provide means to work with all 4 versions.
The Dataset-Structure.md offers more details about the dataset structure. We provide the following dataset formats: COCO format for Instance Segmentation (commonly used by Detectron2 models) and YOLOv8-Seg format used by ultralytics.
Check the dataset_and_model.ipynb for downloading the dataset and model weights.
After cloning the repository and setting up the environment, use the following code for model loading and inference:
from xami_model.inference.xami_inference import InferXami
det_type = 'rtdetr' # 'rtdetr' 'yolov8'
detr_checkpoint = f'./xami_model/train/weights/{det_type}_sam_weights/{det_type}_detect_300e_best.pt'
sam_checkpoint = f'./xami_model/train/weights/{det_type}_sam_weights/{det_type}_sam.pth'
detr_sam_pipeline = InferXami(
device='cuda:0',
detr_checkpoint=detr_checkpoint,
sam_checkpoint=sam_checkpoint,
model_type='vit_t', # the SAM checkpoint and model_type (vit_h, vit_t, etc.) must be compatible
use_detr_masks=True,
detr_type=det_type)
masks = detr_sam_pipeline.run_predict('./example_images/S0893811101_M.png', show_masks=True)For training the model, check the training README.md.
Figure 2: Cumulative distribution of IoUs between predicted and true masks using RT-DETR as detector.
| Category | Precision | Recall | ||
|---|---|---|---|---|
| YOLO-v8 | RT-DETR | YOLO-v8 | RT-DETR | |
| Overall | 84.3 | 62.7 | 72.1 | 78.3 |
| Central-Ring | 89.3 | 89.1 | 94.0 | 97.0 |
| Read-out-Streak | 71.1 | 68.3 | 73.3 | 95.3 |
| Smoke-Ring | 80.6 | 78.1 | 85.6 | 93.8 |
| Star-Loop | 80.5 | 71.6 | 74.1 | 83.3 |
| Other | 100.0 | 6.2 | 33.3 | 22.2 |
Table 1: Metrics per object detector used.
This project is licensed under MIT license.