Skip to content

Register now for

# Instance Segmentation Datasets Overview

Instance segmentation is a computer vision task that involves identifying and delineating individual objects within an image. This guide provides an overview of dataset formats supported by Ultralytics YOLO for instance segmentation tasks, along with instructions on how to prepare, convert, and use these datasets for training your models.

## Supported Dataset Formats

### Ultralytics YOLO format

The dataset label format used for training YOLO segmentation models is as follows:

1. One text file per image: Each image in the dataset has a corresponding text file with the same name as the image file and the ".txt" extension.
2. One row per object: Each row in the text file corresponds to one object instance in the image.
3. Object information per row: Each row contains the following information about the object instance:
   - Object class index: An integer representing the class of the object (e.g., 0 for person, 1 for car, etc.).
   - Object bounding coordinates: The bounding coordinates around the mask area, normalized to be between 0 and 1.

The format for a single row in the segmentation dataset file is as follows:

    <class-index> <x1> <y1> <x2> <y2> ... <xn> <yn>

In this format, `<class-index>` is the index of the class for the object, and `<x1> <y1> <x2> <y2> ... <xn> <yn>` are the bounding coordinates of the object's segmentation mask. The coordinates are separated by spaces.

Here is an example of the YOLO dataset format for a single image with two objects made up of a 3-point segment and a 5-point segment.

    0 0.681 0.485 0.670 0.487 0.676 0.487
    1 0.504 0.000 0.501 0.004 0.498 0.004 0.493 0.010 0.492 0.0104

Tip

- The length of each row does not have to be equal.
- Each segmentation label must have a minimum of 3 xy points: `<class-index> <x1> <y1> <x2> <y2> <x3> <y3>`

### Dataset YAML format

The Ultralytics framework uses a YAML file format to define the dataset and model configuration for training Segmentation Models. Here is an example of the YAML format used for defining a segmentation dataset:

ultralytics/cfg/datasets/coco8-seg.yaml

    # Ultralytics 🚀 AGPL-3.0 License - https://ultralytics.com/license

    # COCO8-seg dataset (first 8 images from COCO train2017) by Ultralytics
    # Documentation: https://docs.ultralytics.com/datasets/segment/coco8-seg/
    # Example usage: yolo train data=coco8-seg.yaml
    # parent
    # ├── ultralytics
    # └── datasets
    #     └── coco8-seg ← downloads here (1 MB)

    # Train/val/test sets as 1) dir: path/to/imgs, 2) file: path/to/imgs.txt, or 3) list: [path/to/imgs1, path/to/imgs2, ..]
    path: coco8-seg # dataset root dir
    train: images/train # train images (relative to 'path') 4 images
    val: images/val # val images (relative to 'path') 4 images
    test: # test images (optional)

    # Classes
    names:
      0: person
      1: bicycle
      2: car
      3: motorcycle
      4: airplane
      5: bus
      6: train
      7: truck
      8: boat
      9: traffic light
      10: fire hydrant
      11: stop sign
      12: parking meter
      13: bench
      14: bird
      15: cat
      16: dog
      17: horse
      18: sheep
      19: cow
      20: elephant
      21: bear
      22: zebra
      23: giraffe
      24: backpack
      25: umbrella
      26: handbag
      27: tie
      28: suitcase
      29: frisbee
      30: skis
      31: snowboard
      32: sports ball
      33: kite
      34: baseball bat
      35: baseball glove
      36: skateboard
      37: surfboard
      38: tennis racket
      39: bottle
      40: wine glass
      41: cup
      42: fork
      43: knife
      44: spoon
      45: bowl
      46: banana
      47: apple
      48: sandwich
      49: orange
      50: broccoli
      51: carrot
      52: hot dog
      53: pizza
      54: donut
      55: cake
      56: chair
      57: couch
      58: potted plant
      59: bed
      60: dining table
      61: toilet
      62: tv
      63: laptop
      64: mouse
      65: remote
      66: keyboard
      67: cell phone
      68: microwave
      69: oven
      70: toaster
      71: sink
      72: refrigerator
      73: book
      74: clock
      75: vase
      76: scissors
      77: teddy bear
      78: hair drier
      79: toothbrush

    # Download script/URL (optional)
    download: https://github.com/ultralytics/assets/releases/download/v0.0.0/coco8-seg.zip

The `train` and `val` fields specify the paths to the directories containing the training and validation images, respectively.

`names` is a dictionary of class names. The order of the names should match the order of the object class indices in the YOLO dataset files.

## Usage

Example

Python CLI

    from ultralytics import YOLO

    # Load a model
    model = YOLO("yolo11n-seg.pt")  # load a pretrained model (recommended for training)

    # Train the model
    results = model.train(data="coco8-seg.yaml", epochs=100, imgsz=640)

    # Start training from a pretrained *.pt model
    yolo segment train data=coco8-seg.yaml model=yolo11n-seg.pt epochs=100 imgsz=640

## Supported Datasets

Ultralytics YOLO supports various datasets for instance segmentation tasks. Here's a list of the most commonly used ones:

- COCO: A comprehensive dataset for object detection, segmentation, and captioning, featuring over 200K labeled images across a wide range of categories.
- COCO8-seg: A compact, 8-image subset of COCO designed for quick testing of segmentation model training, ideal for CI checks and workflow validation in the `ultralytics` repository.
- COCO128-seg: A smaller dataset for instance segmentation tasks, containing a subset of 128 COCO images with segmentation annotations.
- Carparts-seg: A specialized dataset focused on the segmentation of car parts, ideal for automotive applications. It includes a variety of vehicles with detailed annotations of individual car components.
- Crack-seg: A dataset tailored for the segmentation of cracks in various surfaces. Essential for infrastructure maintenance and quality control, it provides detailed imagery for training models to identify structural weaknesses.
- Package-seg: A dataset dedicated to the segmentation of different types of packaging materials and shapes. It's particularly useful for logistics and warehouse automation, aiding in the development of systems for package handling and sorting.

### Adding your own dataset

If you have your own dataset and would like to use it for training segmentation models with Ultralytics YOLO format, ensure that it follows the format specified above under "Ultralytics YOLO format". Convert your annotations to the required format and specify the paths, number of classes, and class names in the YAML configuration file.

## Port or Convert Label Formats

### COCO Dataset Format to YOLO Format

You can easily convert labels from the popular COCO dataset format to the YOLO format using the following code snippet:

Example

Python

    from ultralytics.data.converter import convert_coco

    convert_coco(labels_dir="path/to/coco/annotations/", use_segments=True)

This conversion tool can be used to convert the COCO dataset or any dataset in the COCO format to the Ultralytics YOLO format.

Remember to double-check if the dataset you want to use is compatible with your model and follows the necessary format conventions. Properly formatted datasets are crucial for training successful object detection models.

## Auto-Annotation

Auto-annotation is an essential feature that allows you to generate a segmentation dataset using a pre-trained detection model. It enables you to quickly and accurately annotate a large number of images without the need for manual labeling, saving time and effort.

### Generate Segmentation Dataset Using a Detection Model

To auto-annotate your dataset using the Ultralytics framework, you can use the `auto_annotate` function as shown below:

Example

Python

    from ultralytics.data.annotator import auto_annotate

    auto_annotate(data="path/to/images", det_model="yolo11x.pt", sam_model="sam_b.pt")

Argument | Type | Default | Description
--- | --- | --- | ---
`data` | `str` | required | Path to directory containing target images for annotation or segmentation.
`det_model` | `str` | `'yolo11x.pt'` | YOLO detection model path for initial object detection.
`sam_model` | `str` | `'sam_b.pt'` | SAM model path for segmentation (supports SAM, SAM2 variants and mobile_sam models).
`device` | `str` | `''` | Computation device (e.g., 'cuda:0', 'cpu', or '' for automatic device detection).
`conf` | `float` | `0.25` | YOLO detection confidence threshold for filtering weak detections.
`iou` | `float` | `0.45` | IoU threshold for Non-Maximum Suppression to filter overlapping boxes.
`imgsz` | `int` | `640` | Input size for resizing images (must be multiple of 32).
`max_det` | `int` | `300` | Maximum number of detections per image for memory efficiency.
`classes` | `list[int]` | `None` | List of class indices to detect (e.g., `[0, 1]` for person & bicycle).
`output_dir` | `str` | `None` | Save directory for annotations (defaults to './labels' relative to data path).

The `auto_annotate` function takes the path to your images, along with optional arguments for specifying the pre-trained detection models i.e. YOLO11, YOLOv8 or other models and segmentation models i.e, SAM, SAM2 or MobileSAM, the device to run the models on, and the output directory for saving the annotated results.

By leveraging the power of pre-trained models, auto-annotation can significantly reduce the time and effort required for creating high-quality segmentation datasets. This feature is particularly useful for researchers and developers working with large image collections, as it allows them to focus on model development and evaluation rather than manual annotation.

### Visualize Dataset Annotations

Before training your model, it's often helpful to visualize your dataset annotations to ensure they're correct. Ultralytics provides a utility function for this purpose:

    from ultralytics.data.utils import visualize_image_annotations

    label_map = {  # Define the label map with all annotated class labels.
        0: "person",
        1: "car",
    }

    # Visualize
    visualize_image_annotations(
        "path/to/image.jpg",  # Input image path.
        "path/to/annotations.txt",  # Annotation file path for the image.
        label_map,
    )

This function draws bounding boxes, labels objects with class names, and adjusts text color for better readability, helping you identify and correct any annotation errors before training.

### Converting Segmentation Masks to YOLO Format

If you have segmentation masks in binary format, you can convert them to the YOLO segmentation format using:

    from ultralytics.data.converter import convert_segment_masks_to_yolo_seg

    # For datasets like COCO with 80 classes
    convert_segment_masks_to_yolo_seg(masks_dir="path/to/masks_dir", output_dir="path/to/output_dir", classes=80)

This utility converts binary mask images into the YOLO segmentation format and saves them in the specified output directory.

## FAQ

### What dataset formats does Ultralytics YOLO support for instance segmentation?

Ultralytics YOLO supports several dataset formats for instance segmentation, with the primary format being its own Ultralytics YOLO format. Each image in your dataset needs a corresponding text file with object information segmented into multiple rows (one row per object), listing the class index and normalized bounding coordinates. For more detailed instructions on the YOLO dataset format, visit the Instance Segmentation Datasets Overview.

### How can I convert COCO dataset annotations to the YOLO format?

Converting COCO format annotations to YOLO format is straightforward using Ultralytics tools. You can use the `convert_coco` function from the `ultralytics.data.converter` module:

    from ultralytics.data.converter import convert_coco

    convert_coco(labels_dir="path/to/coco/annotations/", use_segments=True)

This script converts your COCO dataset annotations to the required YOLO format, making it suitable for training your YOLO models. For more details, refer to Port or Convert Label Formats.

### How do I prepare a YAML file for training Ultralytics YOLO models?

To prepare a YAML file for training YOLO models with Ultralytics, you need to define the dataset paths and class names. Here's an example YAML configuration:

    # Ultralytics 🚀 AGPL-3.0 License - https://ultralytics.com/license

    # COCO8-seg dataset (first 8 images from COCO train2017) by Ultralytics
    # Documentation: https://docs.ultralytics.com/datasets/segment/coco8-seg/
    # Example usage: yolo train data=coco8-seg.yaml
    # parent
    # ├── ultralytics
    # └── datasets
    #     └── coco8-seg ← downloads here (1 MB)

    # Train/val/test sets as 1) dir: path/to/imgs, 2) file: path/to/imgs.txt, or 3) list: [path/to/imgs1, path/to/imgs2, ..]
    path: coco8-seg # dataset root dir
    train: images/train # train images (relative to 'path') 4 images
    val: images/val # val images (relative to 'path') 4 images
    test: # test images (optional)

    # Classes
    names:
      0: person
      1: bicycle
      2: car
      3: motorcycle
      4: airplane
      5: bus
      6: train
      7: truck
      8: boat
      9: traffic light
      10: fire hydrant
      11: stop sign
      12: parking meter
      13: bench
      14: bird
      15: cat
      16: dog
      17: horse
      18: sheep
      19: cow
      20: elephant
      21: bear
      22: zebra
      23: giraffe
      24: backpack
      25: umbrella
      26: handbag
      27: tie
      28: suitcase
      29: frisbee
      30: skis
      31: snowboard
      32: sports ball
      33: kite
      34: baseball bat
      35: baseball glove
      36: skateboard
      37: surfboard
      38: tennis racket
      39: bottle
      40: wine glass
      41: cup
      42: fork
      43: knife
      44: spoon
      45: bowl
      46: banana
      47: apple
      48: sandwich
      49: orange
      50: broccoli
      51: carrot
      52: hot dog
      53: pizza
      54: donut
      55: cake
      56: chair
      57: couch
      58: potted plant
      59: bed
      60: dining table
      61: toilet
      62: tv
      63: laptop
      64: mouse
      65: remote
      66: keyboard
      67: cell phone
      68: microwave
      69: oven
      70: toaster
      71: sink
      72: refrigerator
      73: book
      74: clock
      75: vase
      76: scissors
      77: teddy bear
      78: hair drier
      79: toothbrush

    # Download script/URL (optional)
    download: https://github.com/ultralytics/assets/releases/download/v0.0.0/coco8-seg.zip

Ensure you update the paths and class names according to your dataset. For more information, check the Dataset YAML Format section.

### What is the auto-annotation feature in Ultralytics YOLO?

Auto-annotation in Ultralytics YOLO allows you to generate segmentation annotations for your dataset using a pre-trained detection model. This significantly reduces the need for manual labeling. You can use the `auto_annotate` function as follows:

    from ultralytics.data.annotator import auto_annotate

    auto_annotate(data="path/to/images", det_model="yolo11x.pt", sam_model="sam_b.pt")  # or sam_model="mobile_sam.pt"

This function automates the annotation process, making it faster and more efficient. For more details, explore the Auto-Annotate Reference.

📅 Created 1 year ago ✏️ Updated 10 days ago

## Comments