Custom Instance Segmentation Using YOLO V8: An In-depth Tutorial

Custom Instance Segmentation using YOLO V8

In this tutorial, we'll delve into creating a custom instance segmentation using YOLO V8. We'll take you through every step, from data annotation to model training and inference. Additionally, we'll discuss exporting the model to various formats, such as ONNX and TensorFlow Lite. 

Step 1: Create a Virtual Environment

Firstly, set up a virtual environment for your project. For those using Anaconda, you can create a new environment using the command:

```shell

conda create -n YOLOV8_segmentation python=3.10

```

After creation, activate the environment:

```shell

conda activate YOLOV8_segmentation

```

Step 2: Annotate Custom Dataset

Next, you'll need images of the object you want to annotate. In this tutorial, we'll use images of butterflies, but you're free to choose any object. You can manually download the images or write a script to do it automatically.

To annotate the segmentation masks, install the `labelme` library:

```shell

pip install labelme

```

Using the labelme GUI, you can annotate the images by creating polygons around the object. However, note that this process can be time-consuming.

If you wish to skip this manual annotation, there's an annotated dataset of butterflies available at PixelLib's GitHub repository. Download the `nature.zip` file, extract it, and segregate the butterfly images and annotations.

Step 3: Convert Annotations to YOLO V8 Format

The annotated dataset isn't in YOLO format by default. To convert it, install the `labelme_to_YOLO` library:

```shell

pip install labelme_to_YOLO

```

Use the following command to convert JSON files to YOLO V8.txt format:

```shell

labelme_to_YOLO --json_dir path_to_your_directory

```

After converting, merge the images and labels from the train and test folders. Then, delete the unnecessary folders and move the `dataset.yml` file to the project's root directory.

Step 4: Install Ultralytics and Train the Model

To train YOLO V8's custom instance segmentation model, install the `ultralytics` library:

```shell

pip install ultralytics

```

By default, this installs PyTorch with CPU support. If you have an Nvidia GPU, you can install the GPU version of PyTorch. Afterward, you can train the instance segmentation model on your custom data.

Step 5: Use Model for Inferencing

After training, use the best weights of the model for inferencing. For that, create a script and use the `ultralytics.YOLO` class for predictions.

Step 6: Export the Model

Finally, export your trained model to other formats. You can export to ONNX format using:

```python

model.export(format='onnx')

```

Or to TensorFlow Lite using:

```python

model.export(format='tflite')

```

This concludes the tutorial on creating a custom instance segmentation using YOLO V8. By following these steps, you can customize the segmentation model to your needs, make inferences on images, videos, and live webcams, and export the model to various formats.