Synthetic Data Generation for Computer Vision Tasks in SITL Systems with Unreal Engine and Airsim

Abstract

Sustainable object detection, classification, and tracking are a critical path in robotics at per ception leg. Deep learning models used for detection require vast and diverse datasets with accurate annotations. However, acquiring real-world data that captures rare, complex, or hazardous scenarios is both time-consuming and costly. To address this limitation, synthetic data generation using virtual environments has emerged as a promising alternative. This research investigates the application of synthetic datasets, created within high-fidelity 3D vir tual reality environments, for training deep learning models in object detection tasks. The proposed method leverages tools such as Unreal Engine and procedural scripting to generate large volumes of realistic, annotated image data. These synthetic scenes are carefully con structed to include various object types, dynamic interactions, lighting variations, occlusions, and edge-case scenarios that are often missing from real-world datasets. The study employs the YOLO family of deep learning models, which are known for their high accuracy and low latency, making them suitable for real-time robotic applications. Experimental results de mon strate that models trained exclusively on synthetic data demonstrate partial transferability to real-world scenarios, while the addition of a small fraction of real data significantly improves performance. Key advantages include reduced development time, lower annotation costs, im proved coverage of rare scenarios, and enhanced model generalization. The integration of syn thetic data pipelines into the ML lifecycle also facilitates better experiment management and scalability. The research concludes that synthetic data is a powerful enabler for developing robust object detection systems in robotics and AI. Future work will focus on narrowing the domain gap between synthetic and real-world imagery, enhancing realism with photometric effects, and expanding the dataset to include additional object classes and behaviors for multi-task learning.