Complex Model Transformations by Reinforcement Learning with Uncertain Human Guidance

--Model-driven engineering problems often require complex model transformations (MTs), i.e., MTs that are chained in extensive sequences. Pertinent examples of such problems include model synchronization, automated model repair, and design space exploration. Manually developing complex MTs is an error-prone and often infeasible process. Reinforcement learning (RL) is an apt way to alleviate these issues. In RL, an autonomous agent explores the state space through trial and error to identify beneficial sequences of actions, such as MTs. In these situations, human guidance can be of high utility. In this paper, we present an approach and technical framework for developing complex MT sequences through RL, guided by potentially uncertain human advice. Our framework allows user-defined MTs to be mapped onto RL primitives, and executes them as RL programs to find optimal MT sequences. Our evaluation shows that human guidance, even if uncertain, substantially improves RL performance, and results in more efficient development of complex MTs. Through a trade-off between the certainty and timeliness of human advice, our method takes a step towards RL-driven human-in-the-loop engineering methods. Modeling activities are often more complex than an atomic model transformation (MT) and rely on sequences of MTs . Pertinent examples can be found in model synchronization [1], model refactoring [2], and rule-based design-space exploration [3]. Typically, there might be more than one MT sequence that can successfully transform the source model into the target state, and choosing the most appropriate (cost-effective, efficient, safe) one manually is not tractable. This raises the need for automated methods for developing complex MTs, in which MTs are chained in sequences.