Improve documentation on Yolov8 Detector

[2649]

Issue Type

Documentation Feature Request

Source

source

Keras Version

Keras 3

Custom Code

Yes

OS Platform and Distribution

No response

Python version

No response

GPU model and memory

No response

Current Behavior?

I'm tinkering with Keras 3. Nice work!

I have a hard time training Yolov8 because the documentation only shows the case for training with batch size 1, and it is unclear how to create batches with variable number of bounding boxes.

So far, I have tried using a ragged tensor as input, which throws an error. This is also recommended from the Keras 2 tutorial.

I can run it with a batch size of one. However, it throws an error when the number of boxes changes. So, I guess it needs to be batched with padding. Can the documentation specify how the padding should be created?

Standalone code to reproduce the issue or tutorial link

import tensorflow as tf
import keras_cv

# Create 2 images
images = tf.ones(shape=(2, 512, 512, 3))
labels = {
    "boxes": tf.ragged.constant([
        [
            [0, 0, 100, 100],
            [100, 100, 200, 200],
            [300, 300, 100, 100],
        ],
        # Add a second image with one bbox
        [
            [0, 0, 100, 100]
        ], 
    ], dtype=tf.float32),
    
    "classes": tf.ragged.constant([[1, 1, 1], [1]], dtype=tf.int64),
}

model = keras_cv.models.YOLOV8Detector(
    num_classes=20,
    bounding_box_format="xywh",
    backbone=keras_cv.models.YOLOV8Backbone.from_preset(
        "yolo_v8_m_backbone_coco"
    ),
    fpn_depth=2
)

# Evaluate model without box decoding and NMS
model(images)

# Prediction with box decoding and NMS
model.predict(images)

# Train model
model.compile(
    classification_loss='binary_crossentropy',
    box_loss='ciou',
    optimizer=tf.optimizers.SGD(global_clipnorm=10.0),
    jit_compile=False,
)
model.fit(images, labels)

Relevant log output

TypeError                                 Traceback (most recent call last)
Cell In[4], line 42
     35 # Train model
     36 model.compile(
     37     classification_loss='binary_crossentropy',
     38     box_loss='ciou',
     39     optimizer=tf.optimizers.SGD(global_clipnorm=10.0),
     40     jit_compile=False,
     41 )
---> 42 model.fit(images, labels)

File ~/Desktop/repos/stratai/ai/.keras-venv/lib/python3.11/site-packages/keras/src/utils/traceback_utils.py:123, in filter_traceback.<locals>.error_handler(*args, **kwargs)
    120     filtered_tb = _process_traceback_frames(e.__traceback__)
    121     # To get the full stack trace, call:
    122     # `keras.config.disable_traceback_filtering()`
--> 123     raise e.with_traceback(filtered_tb) from None
    124 finally:
    125     del filtered_tb

File ~/Desktop/repos/stratai/ai/.keras-venv/lib/python3.11/site-packages/keras_cv/src/models/object_detection/yolo_v8/yolo_v8_detector.py:526, in YOLOV8Detector.train_step(self, *args)
    524 args = args[:-1]
    525 x, y = unpack_input(data)
--> 526 return super().train_step(*args, (x, y))

File ~/Desktop/repos/stratai/ai/.keras-venv/lib/python3.11/site-packages/keras_cv/src/models/object_detection/yolo_v8/yolo_v8_detector.py:545, in YOLOV8Detector.compute_loss(self, x, y, y_pred, sample_weight, **kwargs)
    541 stride_tensor = ops.expand_dims(stride_tensor, axis=-1)
    543 gt_labels = y["classes"]
--> 545 mask_gt = ops.all(y["boxes"] > -1.0, axis=-1, keepdims=True)
    546 gt_bboxes = bounding_box.convert_format(
    547     y["boxes"],
    548     source=self.bounding_box_format,
    549     target="xyxy",
    550     images=x,
    551 )
    553 pred_bboxes = dist2bbox(pred_boxes, anchor_points)

TypeError: Failed to convert elements of tf.RaggedTensor(values=tf.RaggedTensor(values=Tensor("Greater:0", shape=(None,), dtype=bool), row_splits=Tensor("data_3:0", shape=(None,), dtype=int64)), row_splits=Tensor("data_2:0", shape=(None,), dtype=int64)) to Tensor. Consider casting elements to a supported type. See https://www.tensorflow.org/api_docs/python/tf/dtypes for supported TF dtypes.

[2649]

I accidentally posted in the wrong repo. Reposted in keras_cv.