Journal of Computer Technology & Applications

Journal of Computer Technology & Applications Original Research

Log Identification and Monitoring System Using Generative AI

  1. Nikhil Santosh Shinde Department of MCA, Parvatibai Genba Moze College of Engineering, Wagholi, Pune
  2. Ajay Shiketod Department of MCA, Parvatibai Genba Moze College of Engineering, Wagholi, Pune
  3. Swati Andhale Department of MCA, Parvatibai Genba Moze College of Engineering, Wagholi, Pune

Abstract

In contemporary software ecosystems, application and infrastructure logs play a vital role in ensuring system reliability, performance optimization, fault diagnosis, and security compliance. As applications become increasingly distributed and cloud native, the volume, velocity, and variety of generated log data have grown dramatically. This rapid expansion makes traditional manual log inspection inefficient, error-prone, and largely impractical. To address these challenges, this paper proposes an artificial intelligence (AI) driven log monitoring and analysis system designed to automate log processing, anomaly detection, alert generation, and visualization in real time The proposed system is built on a scalable and high-performance architecture that combines a FastAPI backend for asynchronous log ingestion and processing with a Next.js-based frontend that enables interactive dashboards and visual analytics. Logs collected from distributed applications are parsed and structured using advanced pattern-matching techniques, including Grok and Vector.dev, ensuring consistency and accuracy in data extraction. An intelligent anomaly detection engine powered by machine learning is integrated to identify unusual patterns, deviations, and potential system failures. To further enhance detection accuracy and computational efficiency, the system employs a trie-based adaptive caching mechanism that optimizes repeated pattern recognition. Additionally, the platform integrates Grafana and Prometheus to support comprehensive monitoring and time-series visualization, while real-time alerts are delivered through external notification channels such as Slack and email. Experimental results indicate that the system effectively identifies anomalies with minimal latency, provides actionable insights, and significantly reduces the operational burden on engineers. Overall, the proposed solution improves observability, accelerates incident response, and enhances the reliability of modern distributed systems.

Keywords

  • anomaly detection
  • distributed systems
  • generative AI
  • log monitoring
  • real-time alerting

References (20)

  1. Xu W, Huang L, Fox A, Patterson D, Jordan MI. Detecting large-scale system problems by mining console logs. Proceedings of the ACM SIGOPS 22nd symposium on Operating systems principles. 2009:117-132. doi:10.1145/1629575.1629587
  2. He P, Zhu J, Zheng Z, Lyu MR. Drain: An Online Log Parsing Approach with Fixed Depth Tree. 2017 IEEE International Conference on Web Services (ICWS). 2017:33-40. doi:10.1109/icws.2017.13
  3. Du M, Li F, Zheng G, Srikumar V. DeepLog. Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security. 2017:1285-1298. doi:10.1145/3133956.3134015
  4. Yadav RB, Kumar PS, Dhavale SV. A Survey on Log Anomaly Detection using Deep Learning. 2020 8th International Conference on Reliability, Infocom Technologies and Optimization (Trends and Future Directions) (ICRITO). 2020:1215-1220. doi:10.1109/icrito48877.2020.9197818
  5. Zhou J, Ying S, Wang S, Zhao D, Xiang J, Liang K, et al. LogDLR: Unsupervised Cross-System Log Anomaly Detection Through Domain-Invariant Latent Representation. IEEE Transactions on Dependable and Secure Computing. 2025;22(4):4456-4471. doi:10.1109/tdsc.2025.3548050
  6. Zhang X, Xu Y, Lin Q, Qiao B, Zhang H, Dang Y, et al. Robust log-based anomaly detection on unstable log data. Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 2019:807-817. doi:10.1145/3338906.3338931
  7. Pankajashan S, Maragatham G, Kirthiga Devi T. Hybrid approach with Deep Auto-Encoder and optimized LSTM based Deep Learning approach to detect anomaly in cloud logs. Journal of Intelligent & Fuzzy Systems. 2022;42(6):6257-6271. doi:10.3233/jifs-201707
  8. Guo H, Yang J, Liu J, Bai J, Wang B, Li Z, et al. LogFormer: A Pre-train and Tuning Pipeline for Log Anomaly Detection. Proceedings of the AAAI Conference on Artificial Intelligence. 2024;38(1):135-143. doi:10.1609/aaai.v38i1.27764
  9. Lou JG, Fu Q, Yang S, Xu Y, Li J. Mining invariants from console logs for system problem detection. In: Proceedings of the 2010 USENIX Annual Technical Conference (USENIX ATC ’10); 2010 Jun 23–25; Boston, MA, USA. Berkeley (CA): USENIX Association; 2010. p. 1–14.
  10. lim A, Clegg RG, Mai L, Rupprecht L, Seckler E, Costa P, Pietzuch P, Wolf AL, Sultana N, Crowcroft J, Madhavapeddy A, Moore AW, Mortier R, Koleni M, Oviedo L, McAuley D, Migliavacca M. FLICK: Developing and running application-specific network services. In: Proceedings of the 2016 USENIX Annual Technical Conference (USENIX ATC ’16); 2016 Jun 22–24; Denver, CO, USA. Berkeley (CA): USENIX Association; 2016. Available from: https://www.usenix.org/conference/atc16/technical-sessions/presentation/alim
  11. Kou L, Li Y, Zhang F, Gong X, Hu Y, Yuan Q, et al. Review on Monitoring, Operation and Maintenance of Smart Offshore Wind Farms. Sensors. 2022;22(8):2822. doi:10.3390/s22082822
  12. Huang S, Liu Y, Fung C, Wang H, Yang H, Luan Z. Improving Log-Based Anomaly Detection by Pre-Training Hierarchical Transformers. IEEE Transactions on Computers. 2023;72(9):2656-2667. doi:10.1109/tc.2023.3257518
  13. OpenAI; Achiam J, Adler S, Agarwal S, Ahmad L, Akkaya I, Aleman FL, et al. GPT-4 technical report. [Preprint]. 2023. arXiv:2303.08774. doi:10.48550/arXiv.2303.08774.
  14. Prometheus.io. (2026). Data model. [Online] Prometheus Documentation. Available from: https://prometheus.io/docs/concepts/data_model/
  15. Grafana Labs. (2025). Grafana OSS and Enterprise: Query, visualize, alert on, and explore your metrics, logs, and traces. [online] Grafana documentation. Available from: https://grafana.com/docs/grafana/latest/
  16. Lubanovic B. FastAPI. Sebastopol (CA): O’Reilly Media Inc.; 2023.
  17. Lazuardy MF, Anggraini D. Modern front-end web architectures with React.js and Next.js. Res J. Adv. Eng Sci. 2022;7(1):132–141.
  18. D'Amore L, Arcucci R, Mele V, Scotti G, Murli A. Technical Documentation L-BFGS for GPU-CUDA Reference Manual and User's Guide. SSRN Electronic Journal. 2013. doi:10.2139/ssrn.2332125
  19. Johnson HA. Slack. Journal of the Medical Library Association. 2018;106(1). doi:10.5195/jmla.2018.315
  20. Pedregosa F, Varoquaux G, Gramfort A, Michel V, Thirion B, Grisel O, et al. Scikit-learn: Machine learning in Python. J Mach Learn Res. 2011;12:2825–2830.