HyperStream: a Workflow Engine for Streaming Data

Journal of Machine Learning Research 1 (2019) 1-48 Submitted 8/19; Published 10/00 HyperStream: a Workflow Engine for Streaming Data Tom Diethe tdiethe@amazon.com Intelligent Systems Laboratory, University of Bristol, BS8 1UB, UK Editor: A. N. Other Abstract This paper describes HyperStream, a large-scale, flexible and robust software package, written in the Python language, for processing streaming data with workflow creation capabilities. HyperStream overcomes the limitations of other computational engines and provides high-level interfaces to execute complex nesting, fusion, and prediction both in online and offline forms in streaming environments. HyperStream is a general purpose tool that is well-suited for the design, development, and deployment of Machine Learning algorithms and predictive models in a wide space of sequential predictive problems. Introduction Scientific workflow systems are designed to compose and execute a series of computational or data manipulation operations (workflow) (Deelman et al., 2009).