Enterprise Security Monitoring Lab
An enterprise-grade Observability Stack specifically designed for Identity & Access Management (IAM) environments. This is not a basic Grafana installation — you'll build a production-ready monitoring platform that captures IAM-specific events like failed logins, privilege escalations, MFA bypass attempts, and SSO health.
This lab integrates with Authentik SSO (Project A) and HashiCorp Vault (Project B) to demonstrate how IAM systems should be monitored in real enterprises.
The Problem: In a typical enterprise, security teams have no visibility into IAM events until a breach occurs. Failed login attempts, privilege escalations, and SSO outages go unnoticed. When the SSO portal goes down, users can't work — and IT finds out from angry Slack messages.
The Solution: This lab teaches you to build an IAM-aware observability platform that detects attacks in real-time, monitors SSO availability, and integrates with secrets management. Platform Engineers with these skills command $140K-$200K annually.
Before starting, run this script to verify your environment meets all requirements:
#!/bin/bash
# ============================================
# Identity Bytes - Pre-Flight Check Script
# Run this before starting the Observability Lab
# ============================================
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
NC='\033[0m' # No Color
PASS=0
FAIL=0
echo "============================================"
echo "🔍 Identity Bytes Pre-Flight Check"
echo "============================================"
echo ""
# Check 1: RAM (minimum 4GB, recommended 8GB)
TOTAL_RAM=$(free -g | awk '/^Mem:/{print $2}')
if [ "$TOTAL_RAM" -ge 8 ]; then
echo -e "${GREEN}✓ RAM: ${TOTAL_RAM}GB (Excellent)${NC}"
((PASS++))
elif [ "$TOTAL_RAM" -ge 4 ]; then
echo -e "${YELLOW}⚠ RAM: ${TOTAL_RAM}GB (Minimum met, 8GB recommended)${NC}"
((PASS++))
else
echo -e "${RED}✗ RAM: ${TOTAL_RAM}GB (Minimum 4GB required)${NC}"
((FAIL++))
fi
# Check 2: Docker version (minimum 24.0)
if command -v docker &> /dev/null; then
DOCKER_VER=$(docker --version | grep -oP '\d+\.\d+' | head -1)
echo -e "${GREEN}✓ Docker: v${DOCKER_VER}${NC}"
((PASS++))
else
echo -e "${RED}✗ Docker: Not installed${NC}"
((FAIL++))
fi
# Check 3: Docker Compose
if docker compose version &> /dev/null; then
COMPOSE_VER=$(docker compose version | grep -oP '\d+\.\d+\.\d+')
echo -e "${GREEN}✓ Docker Compose: v${COMPOSE_VER}${NC}"
((PASS++))
else
echo -e "${RED}✗ Docker Compose: Not installed${NC}"
((FAIL++))
fi
# Check 4: Required ports
echo ""
echo "Checking ports..."
for port in 3000 9090 3100 9100 8080; do
if ! ss -tuln | grep -q ":${port} "; then
echo -e "${GREEN}✓ Port ${port}: Available${NC}"
((PASS++))
else
echo -e "${RED}✗ Port ${port}: In use${NC}"
((FAIL++))
fi
done
# Check 5: Disk space (minimum 20GB free)
FREE_DISK=$(df -BG / | awk 'NR==2 {print $4}' | tr -d 'G')
if [ "$FREE_DISK" -ge 50 ]; then
echo -e "${GREEN}✓ Disk: ${FREE_DISK}GB free (Excellent)${NC}"
((PASS++))
elif [ "$FREE_DISK" -ge 20 ]; then
echo -e "${YELLOW}⚠ Disk: ${FREE_DISK}GB free (Minimum met, 50GB recommended)${NC}"
((PASS++))
else
echo -e "${RED}✗ Disk: ${FREE_DISK}GB free (Minimum 20GB required)${NC}"
((FAIL++))
fi
# Summary
echo ""
echo "============================================"
echo "📊 Results: ${PASS} passed, ${FAIL} failed"
echo "============================================"
if [ "$FAIL" -eq 0 ]; then
echo -e "${GREEN}🚀 All checks passed! You're ready to start.${NC}"
exit 0
else
echo -e "${RED}⚠️ Some checks failed. Fix issues before proceeding.${NC}"
exit 1
fi# Save and run the pre-flight check
nano ~/check_reqs.sh
# Paste the script above, save with Ctrl+X, Y, Enter
chmod +x ~/check_reqs.sh
./check_reqs.shUnderstanding the network topology and port mappings before we build.
This diagram shows exactly which ports each component uses and how they communicate over the Docker bridge network:
| Source | Destination | Protocol | Purpose |
|---|---|---|---|
| Prometheus :9090 | Node Exporter :9100 | HTTP (pull) | Scrape system metrics every 15s |
| Prometheus :9090 | cAdvisor :8080 | HTTP (pull) | Scrape container metrics every 15s |
| Prometheus :9090 | Blackbox :9115 | HTTP (pull) | Scrape probe results every 30s |
| Promtail :9080 | Loki :3100 | HTTP (push) | Push logs in batches |
| Grafana :3000 | Prometheus :9090 | HTTP | Query metrics via PromQL |
| Grafana :3000 | Loki :3100 | HTTP | Query logs via LogQL |
| Grafana :3000 | Vault :8200 | HTTPS | Retrieve secrets at startup |
Deploy Grafana, Prometheus, Loki, and supporting exporters.
# Create monitoring stack directory structure
mkdir -p ~/monitoring/{grafana,prometheus,loki,promtail,blackbox}
mkdir -p ~/monitoring/grafana/{data,provisioning/datasources,provisioning/dashboards}
mkdir -p ~/monitoring/prometheus/{data,rules}
mkdir -p ~/monitoring/loki/data
# Set correct ownership for container users
# Grafana=472, Prometheus=65534, Loki=10001
sudo chown -R 472:472 ~/monitoring/grafana/data
sudo chown -R 65534:65534 ~/monitoring/prometheus/data
sudo chown -R 10001:10001 ~/monitoring/loki/data
# Navigate to monitoring directory
cd ~/monitoring
# Verify structure
find ~/monitoring -type d | head -20| Error | Cause | Fix |
|---|---|---|
| Permission denied on /var/lib/grafana | Grafana container runs as UID 472 | sudo chown -R 472:472 ~/monitoring/grafana/data |
| Prometheus TSDB locked | Prometheus runs as UID 65534 (nobody) | sudo chown -R 65534:65534 ~/monitoring/prometheus/data |
| Loki cannot write to /loki | Loki runs as UID 10001 | sudo chown -R 10001:10001 ~/monitoring/loki/data |
cat > ~/monitoring/prometheus/prometheus.yml << 'EOF'
# ============================================
# PROMETHEUS CONFIGURATION
# Identity Bytes - IAM Observability Lab
# ============================================
global:
scrape_interval: 15s
evaluation_interval: 15s
external_labels:
environment: 'homelab'
project: 'identity-bytes'
alerting:
alertmanagers:
- static_configs:
- targets: []
rule_files:
- /etc/prometheus/rules/*.yml
scrape_configs:
# ==========================================
# Self-monitoring
# ==========================================
- job_name: 'prometheus'
static_configs:
- targets: ['localhost:9090']
labels:
service: 'prometheus'
# ==========================================
# Infrastructure Metrics
# ==========================================
- job_name: 'node-exporter'
static_configs:
- targets: ['node-exporter:9100']
labels:
service: 'node'
- job_name: 'cadvisor'
static_configs:
- targets: ['cadvisor:8080']
labels:
service: 'docker'
# ==========================================
# Synthetic Monitoring (Blackbox Exporter)
# Monitors IAM portal availability
# ==========================================
- job_name: 'blackbox-http'
metrics_path: /probe
params:
module: [http_2xx]
static_configs:
- targets:
# Add your IAM endpoints here
- https://auth.yourdomain.com # Authentik SSO
- https://vault.yourdomain.com # HashiCorp Vault
- https://teleport.yourdomain.com # Teleport ZTNA
labels:
probe_type: 'iam_portal'
relabel_configs:
- source_labels: [__address__]
target_label: __param_target
- source_labels: [__param_target]
target_label: instance
- target_label: __address__
replacement: blackbox:9115
# ==========================================
# Authentik Metrics (if exposed)
# ==========================================
- job_name: 'authentik'
static_configs:
- targets: ['authentik-server:9300']
labels:
service: 'authentik'
honor_labels: true
EOF
echo "✓ Prometheus config created"# Verify YAML syntax
cat ~/monitoring/prometheus/prometheus.yml | head -30cat > ~/monitoring/loki/loki-config.yml << 'EOF'
# ============================================
# LOKI CONFIGURATION
# Optimized for self-hosted: 7-day retention
# ============================================
auth_enabled: false
server:
http_listen_port: 3100
grpc_listen_port: 9096
log_level: warn
common:
path_prefix: /loki
storage:
filesystem:
chunks_directory: /loki/chunks
rules_directory: /loki/rules
replication_factor: 1
ring:
kvstore:
store: inmemory
schema_config:
configs:
- from: 2020-10-24
store: boltdb-shipper
object_store: filesystem
schema: v11
index:
prefix: index_
period: 24h
storage_config:
boltdb_shipper:
active_index_directory: /loki/boltdb-shipper-active
cache_location: /loki/boltdb-shipper-cache
shared_store: filesystem
filesystem:
directory: /loki/chunks
# ==========================================
# RETENTION POLICY - Critical for disk management!
# ==========================================
limits_config:
retention_period: 168h # 7 days (adjust for your storage)
max_query_length: 721h # 30 days max query range
max_query_parallelism: 2 # Low resource usage
compactor:
working_directory: /loki/compactor
shared_store: filesystem
retention_enabled: true
retention_delete_delay: 2h
retention_delete_worker_count: 150
EOF
echo "✓ Loki config created with 7-day retention"The retention_period: 168h (7 days) prevents your disk from filling up. For production environments with more storage, increase to 720h (30 days). Monitor disk usage with: df -h ~/monitoring/loki
cat > ~/monitoring/blackbox/blackbox.yml << 'EOF'
# ============================================
# BLACKBOX EXPORTER - Synthetic Monitoring
# Simulates user login checks for IAM portals
# ============================================
modules:
# Standard HTTP check (expect 200 OK)
http_2xx:
prober: http
timeout: 10s
http:
valid_http_versions: ["HTTP/1.1", "HTTP/2.0"]
valid_status_codes: [200, 301, 302]
method: GET
follow_redirects: true
preferred_ip_protocol: "ip4"
tls_config:
insecure_skip_verify: false
# SSO Login Page Check
# Verifies login form is present
http_sso_login:
prober: http
timeout: 15s
http:
valid_http_versions: ["HTTP/1.1", "HTTP/2.0"]
method: GET
fail_if_body_not_matches_regexp:
- ".*login.*|.*sign.?in.*|.*authenticate.*"
follow_redirects: true
tls_config:
insecure_skip_verify: false
# ICMP Ping Check
icmp:
prober: icmp
timeout: 5s
# TCP Port Check (for services without HTTP)
tcp_connect:
prober: tcp
timeout: 5s
EOF
echo "✓ Blackbox config created for IAM monitoring"cat > ~/monitoring/docker-compose.yml << 'EOF'
version: '3.8'
services:
# ==========================================
# GRAFANA - Visualization Dashboard
# ==========================================
grafana:
image: grafana/grafana:latest
container_name: grafana
restart: unless-stopped
ports:
- "3000:3000"
environment:
# Default admin (change immediately!)
- GF_SECURITY_ADMIN_USER=admin
- GF_SECURITY_ADMIN_PASSWORD=changeme123
- GF_USERS_ALLOW_SIGN_UP=false
# Authentik SSO (configure in Phase 4)
- GF_AUTH_GENERIC_OAUTH_ENABLED=false
volumes:
- ./grafana/data:/var/lib/grafana
- ./grafana/provisioning:/etc/grafana/provisioning
networks:
- monitoring
depends_on:
- prometheus
- loki
# ==========================================
# PROMETHEUS - Metrics Collection
# ==========================================
prometheus:
image: prom/prometheus:latest
container_name: prometheus
restart: unless-stopped
ports:
- "9090:9090"
command:
- '--config.file=/etc/prometheus/prometheus.yml'
- '--storage.tsdb.path=/prometheus'
- '--storage.tsdb.retention.time=15d'
- '--web.enable-lifecycle'
- '--web.enable-admin-api'
volumes:
- ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:ro
- ./prometheus/rules:/etc/prometheus/rules:ro
- ./prometheus/data:/prometheus
networks:
- monitoring
# ==========================================
# LOKI - Log Aggregation
# ==========================================
loki:
image: grafana/loki:2.9.0
container_name: loki
restart: unless-stopped
ports:
- "3100:3100"
command: -config.file=/etc/loki/loki-config.yml
volumes:
- ./loki/loki-config.yml:/etc/loki/loki-config.yml:ro
- ./loki/data:/loki
networks:
- monitoring
# ==========================================
# PROMTAIL - Log Shipper
# ==========================================
promtail:
image: grafana/promtail:2.9.0
container_name: promtail
restart: unless-stopped
volumes:
- ./promtail/promtail-config.yml:/etc/promtail/config.yml:ro
- /var/log:/var/log:ro
- /var/lib/docker/containers:/var/lib/docker/containers:ro
command: -config.file=/etc/promtail/config.yml
networks:
- monitoring
# ==========================================
# NODE EXPORTER - System Metrics
# ==========================================
node-exporter:
image: prom/node-exporter:latest
container_name: node-exporter
restart: unless-stopped
ports:
- "9100:9100"
command:
- '--path.procfs=/host/proc'
- '--path.sysfs=/host/sys'
- '--collector.filesystem.mount-points-exclude=^/(sys|proc|dev|host|etc)($$|/)'
volumes:
- /proc:/host/proc:ro
- /sys:/host/sys:ro
- /:/rootfs:ro
networks:
- monitoring
# ==========================================
# CADVISOR - Container Metrics
# ==========================================
cadvisor:
image: gcr.io/cadvisor/cadvisor:latest
container_name: cadvisor
restart: unless-stopped
ports:
- "8080:8080"
volumes:
- /:/rootfs:ro
- /var/run:/var/run:ro
- /sys:/sys:ro
- /var/lib/docker/:/var/lib/docker:ro
networks:
- monitoring
# ==========================================
# BLACKBOX EXPORTER - Synthetic Monitoring
# ==========================================
blackbox:
image: prom/blackbox-exporter:latest
container_name: blackbox
restart: unless-stopped
ports:
- "9115:9115"
command:
- '--config.file=/etc/blackbox/blackbox.yml'
volumes:
- ./blackbox/blackbox.yml:/etc/blackbox/blackbox.yml:ro
networks:
- monitoring
networks:
monitoring:
driver: bridge
EOF
echo "✓ Docker Compose created"cat > ~/monitoring/promtail/promtail-config.yml << 'EOF'
# ============================================
# PROMTAIL CONFIGURATION
# ============================================
server:
http_listen_port: 9080
grpc_listen_port: 0
positions:
filename: /tmp/positions.yaml
clients:
- url: http://loki:3100/loki/api/v1/push
scrape_configs:
# Docker container logs
- job_name: containers
static_configs:
- targets:
- localhost
labels:
job: containerlogs
__path__: /var/lib/docker/containers/*/*log
pipeline_stages:
- json:
expressions:
output: log
stream: stream
attrs:
- json:
expressions:
tag:
source: attrs
- regex:
expression: (?P(?:[a-zA-Z0-9][a-zA-Z0-9_.-]+))
source: tag
- labels:
stream:
container_name:
- output:
source: output
# System auth logs (IAM relevant)
- job_name: auth
static_configs:
- targets:
- localhost
labels:
job: auth
__path__: /var/log/auth.log
# System syslog
- job_name: syslog
static_configs:
- targets:
- localhost
labels:
job: syslog
__path__: /var/log/syslog
EOF
echo "✓ Promtail config created" cat > ~/monitoring/grafana/provisioning/datasources/datasources.yml << 'EOF'
apiVersion: 1
datasources:
- name: Prometheus
type: prometheus
access: proxy
url: http://prometheus:9090
isDefault: true
editable: false
- name: Loki
type: loki
access: proxy
url: http://loki:3100
editable: false
EOF
echo "✓ Data sources configured"cd ~/monitoring
# Pull all images
docker compose pull
# Start all services
docker compose up -d
# Check all containers are running
docker compose ps# Test all endpoints
echo "Testing Prometheus..." && curl -s http://localhost:9090/-/healthy
echo "Testing Loki..." && curl -s http://localhost:3100/ready
echo "Testing Grafana..." && curl -s http://localhost:3000/api/health | jq .database
echo "Testing Blackbox..." && curl -s http://localhost:9115/metrics | head -5
echo "Testing Node Exporter..." && curl -s http://localhost:9100/metrics | head -5
# Check Prometheus targets
curl -s http://localhost:9090/api/v1/targets | jq '.data.activeTargets[].health'Core stack is running! Access Grafana at http://YOUR_SERVER_IP:3000 with admin / changeme123. Change this password immediately!
Intentionally break the configuration to practice troubleshooting.
The best way to learn troubleshooting is to break things on purpose. This exercise teaches you how to use Grafana dashboards and logs to diagnose real problems.
# Edit prometheus.yml and introduce a typo
sed -i "s/job_name: 'node-exporter'/job_name: 'node-exporterXXX'/" ~/monitoring/prometheus/prometheus.yml
# Reload Prometheus config
curl -X POST http://localhost:9090/-/reload
# Wait 30 seconds, then check targets
sleep 30
curl -s http://localhost:9090/api/v1/targets | jq '.data.activeTargets[] | {job: .labels.job, health: .health}'http://YOUR_SERVER_IP:3000up{job=~"node.*"}node-exporterXXX# Fix the typo
sed -i "s/job_name: 'node-exporterXXX'/job_name: 'node-exporter'/" ~/monitoring/prometheus/prometheus.yml
# Reload config
curl -X POST http://localhost:9090/-/reload
# Verify fix
curl -s http://localhost:9090/api/v1/targets | jq '.data.activeTargets[] | select(.labels.job == "node-exporter") | .health'# Change Loki URL to wrong port
sed -i "s|url: http://loki:3100|url: http://loki:9999|" ~/monitoring/promtail/promtail-config.yml
# Restart Promtail
docker compose restart promtail
# Wait and check logs
sleep 10
docker logs promtail --tail=20 2>&1 | grep -i "error\|failed"{job="containerlogs"}# Fix the URL
sed -i "s|url: http://loki:9999|url: http://loki:3100|" ~/monitoring/promtail/promtail-config.yml
# Restart Promtail
docker compose restart promtail
# Verify logs are flowing again
sleep 15
curl -s "http://localhost:3100/loki/api/v1/query?query={job=\"containerlogs\"}&limit=1" | jq '.data.result[0].stream'You've learned how to diagnose real problems using Grafana, Prometheus targets, and container logs. These are the exact skills used in production incident response.
cd ~/monitoring
# Option A: Stop services (preserve data)
docker compose stop
# Option B: Complete removal (DELETES ALL DATA)
docker compose down -v
sudo rm -rf ~/monitoring| Skill | Enterprise Application |
|---|---|
| Grafana + Prometheus + Loki | Full observability stack deployment and management |
| IAM Audit Logging | Security event monitoring and compliance reporting |
| Blackbox Exporter | Synthetic monitoring and SLA tracking |
| HashiCorp Vault Integration | Dynamic secrets management for infrastructure |
| PromQL + LogQL | Query languages for metrics and log analysis |
| Infrastructure as Code | Ansible-based deployment automation |
{container_name="authentik-server"} |= "Login failed". Create a Grafana alert that triggers when failed logins from a single IP exceed a threshold (e.g., 5 failures in 60 seconds). For metrics, if Authentik exposes a authentik_login_failures_total counter, alert when rate(authentik_login_failures_total[5m]) > 1. Combine both for defense-in-depth: metrics for volume, logs for forensic detail.http_2xx module. Key metrics: probe_success (1=up, 0=down), probe_duration_seconds (latency), probe_http_status_code. Create a Grafana dashboard showing probe success rate over time. If probe_success == 0, check probe_http_status_code and probe_ssl_earliest_cert_expiry for TLS issues. Also check probe_dns_lookup_time_seconds to rule out DNS problems.{service="authentik"} first, then grep for specific patterns — Loki excels at this. For security forensics requiring full-text search across all logs, Elasticsearch may be warranted despite higher cost.role_attribute_path in Grafana's config (e.g., contains(groups[*], 'grafana-admins') && 'Admin' || 'Viewer'). Create three Authentik groups: grafana-viewers (read-only), grafana-editors (can create dashboards), grafana-admins (full access). This enforces SSO-based RBAC: users inherit dashboard permissions from their group membership without local Grafana accounts.df -h and identify Loki's chunk directory. Short-term: Reduce retention_period in loki-config.yml (e.g., from 30d to 7d), enable compactor's retention_enabled: true, and restart Loki. The compactor will garbage-collect old chunks. Long-term: Analyze which labels generate the most logs (sum by (job) (rate({job=~".+"}[1h]))) and filter high-volume, low-value logs in Promtail's pipeline. Consider moving to object storage (S3/GCS) for production workloads.You've built an enterprise-grade IAM observability platform with synthetic monitoring, RBAC, secrets management, and security hardening. These are the exact skills used at companies like Netflix, Uber, and Datadog.