Worth of knowledge in deep learning

Abstract: Knowledge constitutes the accumulated understanding and experience that humans use to gain insight into the world. In deep learning, prior knowledge is essential for mitigating shortcomings of data-driven models, such as data dependence, generalization ability, and compliance with constraints. To enable efficient evaluation of the worth of knowledge, we present a framework inspired by interpretable machine learning. Through quantitative experiments, we assess the influence of data volume and estimation range on the worth of knowledge. Our model-agnostic framework can be applied to a variety of common network architectures, providing a comprehensive understanding of the role of prior knowledge in deep learning models. It can also be used to improve the performance of informed machine learning, as well as distinguish improper prior knowledge. However, datadriven models still face certain challenges, such as data dependence (4), generalization ability (5), and compliance with constraints (6). In response to this, informed machine learning has become increasingly popular, enabling prior knowledge to be incorporated into the learning process (7, 8). As illustrated in Figure 1A, various types of knowledge can be integrated into a machine learning model, such as functional relations (9), logic rules (10), differential equations (11), invariance (12, 13), and algebraic relations (14). For knowledge to be incorporated into a machine learning model, it needs to be formalized, meaning that it has to be structured in a manner that can be expressed mathematically. In this case, the formalized knowledge that can be integrated into a machine learning model is referred to as rules. Informed machine learning has been deployed in a variety of problem domains, such as the solution of partial differential equations (PDEs) (15, 16), quantification of fluid flow (4), time series prediction (17), and robot control (18). Depending on how the importance of the rules is perceived, two main approaches in the field of informed machine learning are soft constraint (19) and hard constraint (20, 21). Despite its promise, the worth of knowledge is currently only vaguely understood, which limits our ability to comprehend the relationship between data and knowledge. Figure 1B provides a clear example of the divergent effects of data and rules in the context of interpolation and extrapolation.