Efficient Synthetic Defect on 3D Object Reconstruction and Generation Pipeline for Digital Twins Smart Factory.

Abstract

High-quality 3D objects play a crucial role in digital twins, while synthetic data generated from these objects have become essential in deep learning-based computer vision applications. The task of collecting and labeling real defects on industrial object surfaces has many challenges and efforts, while synthetic data generation feasibly replicates huge amounts of labeled data. However, synthetic datasets lack realism in their rendered images. To overcome this issue, this paper introduces a single framework for 3D industrial object reconstruction and synthetic defect generation for digital twin smart factory applications. In detail, NeRF is applied to reconstruct our custom industrial 3D objects through videos collected by a smartphone camera. Several NeRF-based models (i.e., Instant-NGP, Nerfacto, Volinga, and Tensorf) are compared to choose the best outcome for the next step of defect generation on the 3D object surface. To be fairly evaluated, we train four models using the Nerfstudio framework with our three custom datasets of two objects. From the experiment's results, Instant-NGP and Nerfacto achieve the best outcomes, outperforming all other methods significantly. The exported meshes of 3D objects are refined using Blender before loading into NVIDIA Omniverse Code to generate defects on the surface with the Replicator. To evaluate the object detection performance and to verify the benefits of synthetic defect data, we conducted experiments with YOLO-based models on our synthetic and real-plus-synthetic defects. From the experiment's results, the synthetic defect data contribute to improving YOLO models' generalization capability with the highest and lowest accuracy mAP@0.5 enhancement of 18.8 and 1.5 percent on YOLOv6n and YOLOv8s, respectively.