Real-time multi-object recognition using the fusion of lidar and camera data

Object recognition is currently one of the most significant research topics. Its significance is expected to steadily increase due to its extensive applications across various fields, from agriculture to defense and the space industry. In this study, real-time object recognition processes were conducted within a user-defined area using data simultaneously captured from the built-in camera of the iPhone 13 Pro Max and its integrated LIDAR sensor. The Swift programming language was employed, and SwiftUI was chosen as the framework. The study utilized elements from the MS COCO dataset, employing the YOLO V5 algorithm for object recognition. Real-time video processing was accomplished using Swift Metal. The YOLO V5 algorithm was utilized for object recognition, and video processing was carried out in real-time, narrowing down the area based on the minimum-maximum distance determined in the interface using the real-time fused data from the camera and LIDAR. Areas outside the contours of objects, defined by user-specified value ranges in each frame of the captured real-time video data, were darkened. Thus, the object recognition process was performed on objects within each darkened frame. As a result, object recognition was successfully conducted within a user-defined range of 0-15 meters, as configured in the interface.