Diagnostic system for the analysis of log files from remote data processing nodes

Abstract

To monitor and maintain the proper functioning of distributed systems, there are used log events to further analyze and identify the circumstances and causes of deviations from normal operation. Tools to achieve this goal should primarily provide: 1) rapid search and processing of log files in the process of analysis and identification of failure causes; 2) generation of offers for solving the identified problem. This article presents the results of the analysis of the existing tools for logging and log file processing and substantiates the feasibility of the development of the proposed diagnostic system. The general structure of the developed diagnostic system and its functional model are described. In the paper, there are considered the functions of the main software modules: a parser that processes a log file and collects parts of messages into a single log object; a diagnostic module that processes information about an error in a log file object; and a module for accumulating and using knowledge about the known node malfunctions, ways to fix them and problems that were not previously identified. The choice of MongoDB NoSQL for saving the log file data and knowledge base storage DBMS is substantiated. A unified structure for log file objects has been developed for the MongoDB environment. Diagnostics system control – interoperability of modules and their interaction with the database and knowledge base in the MongoDB environment – is provided by the main program LogHelper which allows the user to perform different queries on log objects, monitor their results, inspect known and not previously identified errors, work with unknown malfunctions, manually set their statuses and find a solution from an existing set of solutions. The results of the conducted research show that the developed diagnostic system is highly effective as a maintenance solution for remote data processing nodes and it supports rapid identification of the malfunction and generation of proposed solutions.