Conclusion

Logs captures a real-time snapshot of the system and are widely used to detect anomalies in complex environments. Since the systems are highly complex and generate large volume of logs, manual log inspection becomes infeasible. To reduce manual effort, we used optimized automated log parsing algorithm and utilized anomaly detection to find erroneous logs. Since the data are highly imbalanced with ratio (30:1), therefore any traditional machine learning approach would not be a good choice for classification.

Here we have used two different methodologies, one is supervised predictive analytics framework, and an unsupervised auto encoder, to find the anomaly. We have seen that our auto encoder model has captured lot of anomalies and the accuracy is also moderate (82%). Also, we have seen that

the framework performs well with 94% accuracy.

These two methods can be used to detect any real-time log analysis with some minor modifications.The method for ML-based supervised way of prediction was conducted using LTTS AiKno®[5] Predictive Analytics Framework. This framework can execute feature engineering and run the extracted features on multiple ML models to get the best performed model based on the provided data.