Flexible Log File Parsing using Hidden Markov Models

We aim to model unknown file processing. As the content of log files often evolves over time, we established a dynamic statistic al model which learns and a dapts processing and parsing rules. First, we l imit the amount of unstructured text by focusing only on those frequent patterns which lead to the desired output table similar to Vaarandi [ 10 ]. Second, we transfo rm the found frequent patterns and the output stating the parsed table into a Hidden Markov Model (HMM). We use this HMM as a specific, however, flexible representation of a pattern for log file processing. With changes in th e raw log file distort ing learned patterns, we aim the model to adapt automa tically in order to maintain high quality outpu t . After training our model on one system type, applying the model and the resulting parsing rule to a different system with slightly different log file patterns, we achieve an accuracy over 99%. Predominantly with the goal of monitoring, almost any computer system produces log files containing information about procedures, events, issues, and errors . These log files ar e generated during operatio n mostly in the form of text or xml files .