Unsupervised-Log-Anomaly-Detection-Microservices README

The project's README from github.com/chuansirrr/Unsupervised-Log-Anomaly-Detection-Microservices. As one markdown file: README.md.

Unsupervised-Log-Anomaly-Detection-Microservices-Advanced

Unsupervised Anomaly Detection for Log Data in Microservice Architectures — Advanced Version

This repository provides a research-grade, unsupervised anomaly detection framework for microservice logs, combining multi-modal embeddings, graph attention networks, hybrid ensembles with GMM thresholding, streaming simulation, and explainability (SHAP + gradients). A minimal Streamlit dashboard is included for real-time visualization.

Key Additions Over the Basic Version

Repository Layout

.
├─ src/
│  ├─ data/
│  │   ├─ parser.py               # Log parsing & templating
│  │   ├─ dataset.py              # Datasets & tokenization
│  │   └─ graph_builder.py        # Build service dependency graphs
│  ├─ models/
│  │   ├─ transformer.py          # Transformer encoder (semantic branch)
│  │   ├─ lstm.py                 # LSTM temporal branch
│  │   ├─ fusion.py               # Feature fusion with attention gating
│  │   ├─ gnn_gat.py              # Graph Attention Network (GAT)
│  │   ├─ autoencoder.py          # Reconstruction head
│  │   └─ ensemble.py             # IsolationForest + GMM thresholding
│  ├─ explain/
│  │   ├─ shap_explain.py         # SHAP explainability for fused features
│  │   └─ grad_tokens.py          # Token-level gradient saliency
│  ├─ streaming/
│  │   └─ collector.py            # Log stream simulator
│  ├─ train.py                    # Unsupervised training pipeline
│  ├─ evaluate.py                 # Evaluation & metrics + thresholding
│  ├─ visualize.py                # Embedding/graph plots
│  └─ dashboard_app.py            # Streamlit dashboard
├─ configs/
│  ├─ base.yaml
│  └─ advanced.yaml
├─ tests/
├─ .github/workflows/python-ci.yml
├─ requirements.txt
├─ LICENSE
└─ CITATION.cff

Quickstart

python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt

# Train unsupervised (embeddings + AE reconstruction)
python src/train.py --config configs/advanced.yaml

# Evaluate & save anomaly scores + calibrated threshold
python src/evaluate.py --config configs/advanced.yaml

# Explain: SHAP on fused features
python -m src.explain.shap_explain --config configs/advanced.yaml --n_samples 256

# (Optional) Run the dashboard (needs artifacts/ outputs)
streamlit run src/dashboard_app.py -- --config configs/advanced.yaml

Citation

@software{log_ad_microservices_advanced,
  title={Unsupervised Anomaly Detection for Log Data in Microservice Architectures — Advanced},
  year={2025},
  author={Your Name}
}

Unsupervised-Log-Anomaly-Detection-Microservices (chuansirrr/Unsupervised-Log-Anomaly-Detection-Microservices)

data-science
Searching Database…

What is this?It is a research-grade, unsupervised anomaly detection framework for microservice logs that combines Transformer, LSTM, and statistical features with graph-based modeling and ensemble scoring to flag unusual log behavior without labeled data.
Is this a ready-made tool or a research codebase I'd experiment with?It is a research-oriented Python codebase bundling multiple models (Transformer, LSTM, GAT, Autoencoder, IsolationForest, GMM) plus a minimal Streamlit dashboard, rather than a polished single-command product.
What kind of libraries does this need to run (deep learning, explainability, dashboard)?Since it fuses Transformer and LSTM neural encoders with a Graph Attention Network, Autoencoder, IsolationForest, and GMM, plus SHAP for explainability and Streamlit for the dashboard, you'd need a Python deep learning stack alongside scikit-learn-style and SHAP/Streamlit libraries, though exact install commands aren't listed.
Is this production-ready or more of an experimental project?The README explicitly calls it a 'research-grade' framework, implying it's built for experimentation and analysis rather than a hardened production deployment.
What does the Graph Attention Network (GAT) add here?It models the dynamic service graph to capture cross-service dependencies, helping detect anomalies that emerge from how microservices interact, not just individual log lines.
How are anomalies actually scored?A hybrid ensemble combines Autoencoder reconstruction error with IsolationForest and GMM-based dynamic thresholding to produce a single probabilistic anomaly score.
How can I tell why a log was flagged as anomalous?The framework computes SHAP values on the fused features and uses token-level gradient saliency on the Transformer branch to explain which tokens or features drove an anomaly score.
Can I test it without real production logs?Yes, it includes a streaming simulator that emulates microservice logs with controllable anomaly injection, useful for trying the pipeline end-to-end.
Creator
chuansirrr
Size
<0.1 MB
Stars
979
Updated
CommitOct 21
Age
11mo
Commits · 90 Days