Abstract

In computing, a log file records events that occur in an operating system or other software runs, or messages between different users of a communication software. Logging is the act of keeping a log. In the simplest case, messages are written to a single log file. There are different types of logs available like event logs, transaction logs, message logs, server logs, etc. Also, in some cases it can be a manual (human-created) log. For instance, if a person makes a visit for a raised ticket for a faulty product, they can keep a record of their observations and the action(s) taken for that visit. And that creates a history of logs for that ticket.

During system development and to detect run time behavior of the system, developers need to analyze system logs. There can also be instances where the system crashes abruptly and the developers struggle to understand the cause of the software crash.

It therefore becomes crucial to analyze the system logs to understand the reason of crash or error messages. Typical Log records consist of runtime information of the computing system such as time stamp of the event, unique ID of user request, and content.

The traditional way of log analysis solely depends on keyword searching or some other rule-based techniques.

However, the highly complex modern systems frequently generate logs in increasing scale, for tens and hundreds of gigabytes per hour. Also, logs mostly do not have no fixed structure.

Hence, the traditional way of inspection of logs could prove to be less efficient and time-consuming, impacting productivity.

In this paper, we have explored various machine learning algorithms and an auto encoder to detect anomalies which can help the developers to quickly identify and derive relevant and appropriate information from the logs maintained.