Automated Driving Deep Neural Network Model Accuracy and Performance Assessment in a Simulated Environment

The evolution and integration of automated vehicles have become more and more tangible in recent years. State-of-the-art technological advances in the field of camera-based Artificial Intelligence (AI) and computer vision greatly favor the performance and reliability of Advanced Driver Assistance System (ADAS), leading to a greater knowledge of vehicular operation and resembling the human behaviour. However, the exclusive use of this technology still seems insufficient to control the vehicular operation at 100%. To reveal the degree of accuracy of the current camera-based automated driving AI modules, this paper studies the structure and behavior of one of the main solutions in a controlled testing environment. The results obtained clearly outline the lack of reliability when using exclusively the AI model in the perception stage, thereby entailing using additional complementary sensors to improve its safety and performance.

• David Tena-Gago
• Jose M. Alcaraz Calero
• Qi Wang

• Accuracy assessment
• AI-Driven Mobility
• Artificial Intelligence
• automated vehicles.

[1] S. SAE, “J3016 standard: Taxonomy and definitions for terms related
to driving automation systems for on-road motor vehicles,” 2021.
[2] R. Lanctot et al., “Accelerating the future: The economic impact of the
emerging passenger economy,” Strategy analytics, vol. 5, 2017.
[3] A. Toschi et al., “Characterizing perception module performance and
robustness in production-scale autonomous driving system,” in IFIP
ICNPC, 2019, pp. 235–247.
[4] A. H. M. Rubaiyat et al., “Experimental resilience assessment of an
open-source driving agent,” in 2018 IEEE PRDC, 2018, pp. 54–63.
[5] J. Norden et al., “Efficient black-box assessment of autonomous vehicle
safety,” arXiv preprint arXiv:1912.03618, 2019.
[6] T. Sato et al., “Hold tight and never let go: Security of deep learning
based automated lane centering under physical-world attack,” arXiv
preprint arXiv:2009.06701, 2020.
[7] Z. Zhong et al., “Detecting safety problems of multi-sensor fusion in
autonomous driving,” arXiv preprint arXiv:2109.06404, 2021.
[8] M. Tan and Q. Le, “Efficientnet: Rethinking model scaling for
convolutional neural networks,” in International Conference on Machine
Learning, 2019, pp. 6105–6114.
[9] H. Cui, V. Radosavljevic, F.-C. Chou, T.-H. Lin, T. Nguyen, T.-K.
Huang, J. Schneider, and N. Djuric, “Multimodal trajectory predictions
for autonomous driving using deep convolutional networks,” in 2019
International Conference on Robotics and Automation (ICRA). IEEE,
2019, pp. 2090–2096.
[10] “Comma.ai - comma2k19,” https://github.com/commaai/comma2k19,
accessed: 06/08/2021.