In-Simulation Testing of Deep Learning Vision Models in Autonomous Robotic Manipulators

Abstract

Testing autonomous robotic manipulators (ARMs) is challenging due to the complex software interactions between vision and control components. A crucial element of modern ARMs is the deep learning (DL) based object detection model. The creation and assessment of this DL model requires real world data, which can be hard to label and collect, especially when the ARM hardware set-up is not available. The current techniques primarily focus on using synthetic data to train deep neural networks (DNNs) and identifying failures through offline or online simulation-based testing. However, the process of exploiting the identified failures to identify design flaws early on, and leveraging the optimized DNN within the simulation to accelerate the engineering of the DNN for real-world tasks remains unclear. To address these challenges, we propose the MARTENS (Manipulator Robot Testing and Enhancement in Simulation) framework, which integrates a photorealistic NVIDIA Isaac Sim simulator with evolutionary search to identify critical scenarios aiming at improving the DL vision model and uncovering system design flaws. Evaluation of two industrial case studies demonstrated that MARTENS effectively reveals ARM failures, detecting 25% to 50% more failures with greater diversity compared to random test generation. The model trained and repaired using the MARTENS approach achieved mean average precision (mAP) scores of 0.91 and 0.82 on real-world images with no prior retraining. Further fine-tuning on real-world images for a few epochs (less than 10) increased the mAP to 0.95 and 0.89 for the first and second use cases (UC-1 and UC-2), respectively. In contrast, a model trained solely on real-world data achieved mAPs of 0.8 and 0.75 for UC-1 and UC-2 after more than 25 epochs.