Safety Integrity of Machine Learning in Automotive Software

The rapid development of automotive technology has led to an increased integration of Machine Learning into safety-critical systems. For example, in the context of advanced driver assistance systems and automated driving (ADAS/AD) functions, where computer vision is used to recognize traffic signs. When Machine Learning is incorporated in the development of safety-critical functions, intense and thorough testing is necessary to make these functions suitable for series development. This requires adhering to the following standards:

• ISO 29119: Addresses software testing in general with a dedicated section for testing AI models.
• ISO PAS 8800: Provides requirements and guidelines for AI systems in road vehicles. However, the standard does not define how these systems should achieve ASIL (Automotive Safety Integrity Levels) compliance. This absence of ASIL-specific guidelines for Machine Learning makes meeting automotive safety standards challenging.
• ISO 26262: Describes the implementation of functional safety in automotive with regards to required ASIL but lacks specific guidelines for Machine Learning. It doesn’t cover lifecycle phases, testing methods, or safety requirements tailored to Machine Learning, creating a gap when applying it to AI-driven systems.
• IEC 61508: Provides a comprehensive framework for functional safety of electrical, electronic, and programmable electronic systems and is a crucial standard that serves as the basis for numerous sector-specific functional safety standards.

Emil Gracić, Product & Process Owner for Safe AI Development, and Gregor Pawelke, AI Quality Specialist at CARIAD, have proposed a concept aiming to close the gaps occurring in the automotive standards ISO 26262 (Functional Safety) and ISO PAS 8800 (AI Safety).

Focusing on testing Machine Learning products in ADAS/AD functions, Emil and Gregor recommend three additional lifecycle phases: Prepare data, train model and deploy model. Defined properties, such as robustness, uncertainty handling, and interpretability are set for the outputs of each lifecycle phase. The properties are tested with required Machine Learning test methods. The more effective the test methods are to achieve the desired properties, the higher the ASIL achieved. This is based on IEC 61508, where a “rigor” value is assigned to test methods, which in turn reflects the effectiveness of achieving the desired properties. Emil explains:

Emil and Gregor’s proposed concept is already being used by development teams at Volkswagen. “The AI testing concept helped us to systematically identify and document ‘white spots’ in the data selection for our ML model. By implementing it, we can ensure that the model operates within predictable and defined limits, which at the same time increases the understanding of how our algorithm behaves,” says Felix Stahl, ADAS Pre-Developer at Volkswagen. His colleague Jonas Kaste adds: “In addition, we are able to develop, test and evaluate scenario-dependent KPIs to provide objective criteria for release and reduce the need for time-consuming downstream testing in the vehicle.”

“We want to carry our understanding of AI safety beyond CARIAD and help to shape the state-of-the-art,” says Gregor. “The challenges we are trying to tackle occur in the entire automotive software industry and our approach can help with that.”