Standardizing Certification of Machine Learning Algorithms in Aeronautical Systems

Table of Contents

1. Introduction
2. Scope of the Standard
3. Joint Effort between Eurokaia and SAE
4. Stakeholders Involved
5. Challenges in Certification of Machine Learning Algorithms
6. Specification and Validation
7. Data Challenges
8. Robustness and Verification
9. Explainability
10. System Considerations
11. Machine Learning Development Life Cycle
12. Standardization Frameworks in Aeronautical Field
13. Outcomes of the Standardization Efforts
14. Future Steps and Liaisons
15. Conclusion

Certification of Machine Learning Algorithms in Aeronautical Systems

Machine learning algorithms have become increasingly popular in the aviation industry due to their ability to solve complex problems that were previously unsolvable with classical algorithms. However, the certification of these algorithms poses significant challenges due to their black box nature, probabilistic nature, and dependence on data. In this article, we will discuss the efforts being made to standardize the certification of machine learning algorithms in aeronautical systems.

Scope of the Standard

The scope of the standard is to provide guidance for the certification of aeronautical products and systems, including AI technologies, both for airborne and ground systems, both manned and unmanned systems. The standard is a joint effort between Eurokaia and SAE, which gathers together more than 500 engineers and specialists all around the world, representing main component and system manufacturers, US manufacturers, regulators and authorities, NSPs, airlines, technology providers, and most of the stakeholders in this field.

Challenges in Certification of Machine Learning Algorithms

The certification of machine learning algorithms poses significant challenges due to their black box nature, probabilistic nature, and dependence on data. The first challenge is related to the specification and validation of the algorithms. Machine learning has been applied very often to very complex problems where the specification in itself may be an imposed problem. Moreover, the validation requirements add again to the data, and as explained, they are difficult to specify and validate completely.

The Second challenge is related to the data. While a lot of features have been identified as being Relevant, the accuracy, integrity, trustability, timeliness, and accessibility of data are probably the most important. The third challenge is related to the robustness and verification of the algorithms. The first question we need to process is how variable a model is to the underlying training dataset, so also how obvious it will be over time to variations in its inputs.

The fourth challenge is related to explainability. The problem is even more difficult with database methods because they look into correlations to exploiting the data and not into causality. Explainability actually needs causality, and the very classical examples that You all know is the one with the husky. All these challenges are strongly related to each other, and you can also see that it is difficult to detect the bias till you didn't see its effect on another property.

Machine Learning Development Life Cycle

In order to address the challenges in the certification of machine learning algorithms, a new engineering level has been introduced, called the Machine Learning Development Life Cycle (MLDL). The MLDL is a tool in order to support certification of machine learning algorithms and will be here to allow defining the appropriate development assurance objectives and to structure them in order to make them simple and clear for practitioners.

The MLDL is process and technology agnostic, meaning that it will not impose a specific development process or a specific machine learning pipeline. It should not also impose a specific technology regarding machine learning specific from work. The MLDL actually is a tool in order to support certification of machine learning algorithms and will be here to allow defining the appropriate development assurance objectives and to structure them in order to make them simple and clear for practitioners.

Standardization Frameworks in Aeronautical Field

In the aeronautical field, either for products that are embedded in aircraft or products that are deployed on ground systems like radars or traffic centers, it is not possible to address or to handle the introduction of auto machine learning-Based products without considering the whole system lifecycle. There are many standard and recommended practices that define the way to define a system and to develop it, and it is important to have this perspective.

Outcomes of the Standardization Efforts

The outcomes of the standardization efforts include a statement of concerns, a taxonomy, and a gap analysis from the existing standards. There are also some potential next steps which are identified, and then there is another document which concerns different use cases, for example, aircraft systems use cases and ATM and ground operation use cases.

Future Steps and Liaisons

The future steps for the standardization of machine learning algorithms in aeronautical systems involve offline learning and then online learning. The next version of the standard will concern mainly offline learning, and it will be a joint document between Eurokaia and SAE. There will be a second version of the standard issued two years later containing other AI technologies that will probably evolve.

Conclusion

In conclusion, the certification of machine learning algorithms in aeronautical systems poses significant challenges due to their black box nature, probabilistic nature, and dependence on data. The Machine Learning Development Life Cycle has been introduced to address these challenges, and efforts are being made to standardize the certification of machine learning algorithms in aeronautical systems. The outcomes of the standardization efforts include a statement of concerns, a taxonomy, and a gap analysis from the existing standards. The future steps for the standardization of machine learning algorithms in aeronautical systems involve offline learning and then online learning.