Grafana + IAM Observability

Enterprise Security Monitoring Lab

📊 Grafana 🔥 Prometheus 📋 Loki 🔐 Vault 🎯 Blackbox 📋 Project E
9
Phases
4-6
Hours
8
Services
$140K+
Avg Salary

📋 Part 1: The Blueprint

What Are We Building?

An enterprise-grade Observability Stack designed for Identity & Access Management (IAM) environments. This lab goes beyond basic Grafana installation — you'll build a production-ready monitoring platform that captures IAM-specific events: failed logins, privilege escalations, MFA bypass attempts, and SSO health.

Outcome (Real-World Problem Solved)

This lab addresses the critical gap in IAM visibility that exists in most enterprises. Security teams typically have no real-time insight into authentication events until a breach occurs. By completing this lab, you will:

🏢
Enterprise Context

Zero Trust Alignment: This lab implements continuous verification through comprehensive logging and monitoring. Platform Engineers with Grafana + IAM observability skills command $140K-$200K annually. These are the exact skills used at Netflix, Uber, and Datadog.

📑 Table of Contents
1 Architecture Deep-Dive
2 Core Stack Installation
3 IAM Audit Logging
4 Grafana RBAC + SSO
5 Vault Secret Management
6 Synthetic Monitoring
7 Security Hardening
8 Break & Fix Challenge
9 IaC with Ansible

🛠️ Part 2: Preparation

Prerequisite Knowledge

Before starting this lab, ensure you understand these IAM/PAM concepts:

Concept Description Relevance to Lab
OIDC/OAuth2 Modern authentication protocols for web SSO Grafana SSO integration with Authentik
RBAC Role-Based Access Control — permissions assigned via roles Grafana dashboard access control
Secrets Management Secure storage and retrieval of credentials Vault integration for dynamic secrets
PromQL Prometheus Query Language for metrics Creating alerts and dashboards
LogQL Loki Query Language for log analysis IAM event analysis and correlation

Lab Requirements

Component Specification Notes
Operating System Ubuntu 22.04 LTS / 24.04 LTS Server or VM
Docker v24.x or later Docker Compose v2.x included
RAM 4 GB minimum, 8 GB recommended For all stack components
Disk 20 GB minimum, 50 GB recommended Log retention requires space
Ports 3000, 9090, 3100, 9100, 8080, 9115 Must be available

Pre-Flight Check Script

Run this script to verify your environment before starting:

🖥️ [SERVER] — Run on target server
Bash [SERVER] 
#!/bin/bash
# Identity Bytes Pre-Flight Check

echo "🔍 Running pre-flight checks..."

# Check RAM
RAM=$(free -g | awk '/^Mem:/{print $2}')
if [ "$RAM" -ge 4 ]; then
    echo "✓ RAM: ${RAM}GB"
else
    echo "✗ RAM: ${RAM}GB (need 4GB+)"
fi

# Check Docker
if command -v docker &> /dev/null; then
    echo "✓ Docker: $(docker --version | grep -oP '\d+\.\d+' | head -1)"
else
    echo "✗ Docker: Not installed"
fi

# Check ports
for port in 3000 9090 3100 9100 8080 9115; do
    if ! ss -tuln | grep -q ":${port} "; then
        echo "✓ Port ${port}: Available"
    else
        echo "✗ Port ${port}: In use"
    fi
done

# Check disk
DISK=$(df -BG / | awk 'NR==2 {print $4}' | tr -d 'G')
if [ "$DISK" -ge 20 ]; then
    echo "✓ Disk: ${DISK}GB free"
else
    echo "✗ Disk: ${DISK}GB (need 20GB+)"
fi
1

Phase 1: Architecture Deep-Dive

Understanding network topology, port mappings, and data flow.

⏱️ 10-15 minutes 📖 Reading
🔌 Network Architecture & Port Mapping
📊
Grafana
:3000
Dashboard UI
🔥
Prometheus
:9090
Metrics TSDB
📋
Loki
:3100
Log Aggregation
📤
Promtail
:9080
Log Shipper
🖥️
Node Exporter
:9100
System Metrics
🐳
cAdvisor
:8080
Container Metrics
🎯
Blackbox
:9115
Synthetic Probes
🔐
Vault
:8200
Secrets

Data Flow Summary

Source Destination Protocol Purpose
Prometheus :9090 Node Exporter :9100 HTTP (pull) Scrape system metrics every 15s
Prometheus :9090 Blackbox :9115 HTTP (pull) Scrape probe results every 30s
Promtail :9080 Loki :3100 HTTP (push) Push logs in batches
Grafana :3000 Prometheus :9090 HTTP Query metrics via PromQL
Grafana :3000 Loki :3100 HTTP Query logs via LogQL
2

Phase 2: Core Stack Installation

Deploy Grafana, Prometheus, Loki, and supporting exporters.

⏱️ 45-60 minutes 🖥️ [SERVER]

Step 1: Create Directory Structure

🖥️ [SERVER] — Via SSH
Bash [SERVER] 
# Create monitoring stack directory structure
mkdir -p ~/monitoring/{grafana,prometheus,loki,promtail,blackbox}
mkdir -p ~/monitoring/grafana/{data,provisioning/datasources,provisioning/dashboards}
mkdir -p ~/monitoring/prometheus/{data,rules}
mkdir -p ~/monitoring/loki/data

# Set correct ownership for container users
# Grafana runs as UID 472
# Prometheus runs as UID 65534 (nobody)
# Loki runs as UID 10001
sudo chown -R 472:472 ~/monitoring/grafana/data
sudo chown -R 65534:65534 ~/monitoring/prometheus/data
sudo chown -R 10001:10001 ~/monitoring/loki/data

# Navigate to monitoring directory
cd ~/monitoring

# Verify structure
find ~/monitoring -type d | head -15

Troubleshooting: Directory Permissions

Error Message Cause Fix
Permission denied on /var/lib/grafana Grafana container runs as UID 472 sudo chown -R 472:472 ~/monitoring/grafana/data
opening storage failed: lock DB directory Prometheus runs as UID 65534 sudo chown -R 65534:65534 ~/monitoring/prometheus/data
error creating WAL folder at /loki/wal Loki runs as UID 10001 sudo chown -R 10001:10001 ~/monitoring/loki/data

Step 2: Create Prometheus Configuration

🖥️ [SERVER]
YAML — prometheus.yml [SERVER] 
cat > ~/monitoring/prometheus/prometheus.yml << 'EOF'
# ============================================
# PROMETHEUS CONFIGURATION
# Identity Bytes - IAM Observability Lab
# ============================================
global:
  scrape_interval: 15s       # How often to scrape targets
  evaluation_interval: 15s   # How often to evaluate rules
  external_labels:
    environment: 'homelab'
    project: 'identity-bytes'

# Alertmanager configuration (optional)
alerting:
  alertmanagers:
    - static_configs:
        - targets: []

# Load alert rules from this directory
rule_files:
  - /etc/prometheus/rules/*.yml

scrape_configs:
  # ==========================================
  # Self-monitoring - Prometheus metrics
  # ==========================================
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']
        labels:
          service: 'prometheus'

  # ==========================================
  # Node Exporter - System metrics (CPU, RAM, Disk)
  # ==========================================
  - job_name: 'node-exporter'
    static_configs:
      - targets: ['node-exporter:9100']
        labels:
          service: 'node'

  # ==========================================
  # cAdvisor - Container metrics
  # ==========================================
  - job_name: 'cadvisor'
    static_configs:
      - targets: ['cadvisor:8080']
        labels:
          service: 'docker'

  # ==========================================
  # Blackbox Exporter - Synthetic HTTP probes
  # Monitors IAM portal availability
  # ==========================================
  - job_name: 'blackbox-http'
    metrics_path: /probe
    params:
      module: [http_2xx]
    static_configs:
      - targets:
          # Add your IAM endpoints here
          - https://auth.yourdomain.com       # Authentik SSO
          - https://vault.yourdomain.com      # HashiCorp Vault
          - https://grafana.yourdomain.com    # Grafana itself
        labels:
          probe_type: 'iam_portal'
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: blackbox:9115
EOF

echo "✓ prometheus.yml created at ~/monitoring/prometheus/prometheus.yml"

Step 3: Create Loki Configuration

🖥️ [SERVER]
YAML — loki-config.yml [SERVER] 
cat > ~/monitoring/loki/loki-config.yml << 'EOF'
# ============================================
# LOKI CONFIGURATION
# 7-day retention for disk optimization
# ============================================
auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 9096
  log_level: warn

common:
  path_prefix: /loki
  storage:
    filesystem:
      chunks_directory: /loki/chunks
      rules_directory: /loki/rules
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory

schema_config:
  configs:
    - from: 2020-10-24
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 24h

storage_config:
  boltdb_shipper:
    active_index_directory: /loki/boltdb-shipper-active
    cache_location: /loki/boltdb-shipper-cache
    shared_store: filesystem
  filesystem:
    directory: /loki/chunks

# ==========================================
# RETENTION - Critical for disk management
# 168h = 7 days (adjust based on storage)
# ==========================================
limits_config:
  retention_period: 168h
  max_query_length: 721h
  max_query_parallelism: 2

compactor:
  working_directory: /loki/compactor
  shared_store: filesystem
  retention_enabled: true
  retention_delete_delay: 2h
  retention_delete_worker_count: 150
EOF

echo "✓ loki-config.yml created"
💾
Cost & Resource Optimization

The retention_period: 168h (7 days) prevents disk exhaustion. For production with more storage, increase to 720h (30 days). Monitor with: df -h ~/monitoring/loki

Step 4: Create Docker Compose

🖥️ [SERVER]
YAML — docker-compose.yml [SERVER] 
cat > ~/monitoring/docker-compose.yml << 'EOF'
version: '3.8'

services:
  # ==========================================
  # GRAFANA - Visualization Dashboard
  # ==========================================
  grafana:
    image: grafana/grafana:latest
    container_name: grafana
    restart: unless-stopped
    ports:
      - "3000:3000"
    environment:
      - GF_SERVER_ROOT_URL=https://grafana.yourdomain.com
      - GF_SECURITY_ADMIN_USER=admin
      - GF_SECURITY_ADMIN_PASSWORD=changeme123  # CHANGE THIS!
      - GF_USERS_ALLOW_SIGN_UP=false
    volumes:
      - ./grafana/data:/var/lib/grafana
      - ./grafana/provisioning:/etc/grafana/provisioning
    networks:
      - monitoring
    depends_on:
      - prometheus
      - loki

  # ==========================================
  # PROMETHEUS - Metrics Collection
  # ==========================================
  prometheus:
    image: prom/prometheus:latest
    container_name: prometheus
    restart: unless-stopped
    ports:
      - "9090:9090"
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
      - '--storage.tsdb.retention.time=15d'
      - '--web.enable-lifecycle'
      - '--web.enable-admin-api'
    volumes:
      - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - ./prometheus/rules:/etc/prometheus/rules:ro
      - ./prometheus/data:/prometheus
    networks:
      - monitoring

  # ==========================================
  # LOKI - Log Aggregation
  # ==========================================
  loki:
    image: grafana/loki:2.9.0
    container_name: loki
    restart: unless-stopped
    ports:
      - "3100:3100"
    command: -config.file=/etc/loki/loki-config.yml
    volumes:
      - ./loki/loki-config.yml:/etc/loki/loki-config.yml:ro
      - ./loki/data:/loki
    networks:
      - monitoring

  # ==========================================
  # PROMTAIL - Log Shipper
  # ==========================================
  promtail:
    image: grafana/promtail:2.9.0
    container_name: promtail
    restart: unless-stopped
    volumes:
      - ./promtail/promtail-config.yml:/etc/promtail/config.yml:ro
      - /var/log:/var/log:ro
      - /var/lib/docker/containers:/var/lib/docker/containers:ro
    command: -config.file=/etc/promtail/config.yml
    networks:
      - monitoring

  # ==========================================
  # NODE EXPORTER - System Metrics
  # ==========================================
  node-exporter:
    image: prom/node-exporter:latest
    container_name: node-exporter
    restart: unless-stopped
    ports:
      - "9100:9100"
    command:
      - '--path.procfs=/host/proc'
      - '--path.sysfs=/host/sys'
      - '--collector.filesystem.mount-points-exclude=^/(sys|proc|dev|host|etc)($$|/)'
    volumes:
      - /proc:/host/proc:ro
      - /sys:/host/sys:ro
      - /:/rootfs:ro
    networks:
      - monitoring

  # ==========================================
  # CADVISOR - Container Metrics
  # ==========================================
  cadvisor:
    image: gcr.io/cadvisor/cadvisor:latest
    container_name: cadvisor
    restart: unless-stopped
    ports:
      - "8080:8080"
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:ro
      - /sys:/sys:ro
      - /var/lib/docker/:/var/lib/docker:ro
    networks:
      - monitoring

  # ==========================================
  # BLACKBOX EXPORTER - Synthetic Monitoring
  # ==========================================
  blackbox:
    image: prom/blackbox-exporter:latest
    container_name: blackbox
    restart: unless-stopped
    ports:
      - "9115:9115"
    command:
      - '--config.file=/etc/blackbox/blackbox.yml'
    volumes:
      - ./blackbox/blackbox.yml:/etc/blackbox/blackbox.yml:ro
    networks:
      - monitoring

networks:
  monitoring:
    driver: bridge
EOF

echo "✓ docker-compose.yml created"

Step 5: Deploy the Stack

🖥️ [SERVER]
Bash [SERVER] 
cd ~/monitoring

# Pull all images first
docker compose pull

# Start all services in detached mode
docker compose up -d

# Check all containers are running
docker compose ps
✅ Verification — Phase 2

Run these commands to verify all services are healthy:

Bash [SERVER] 
# Test Prometheus
curl -s http://localhost:9090/-/healthy && echo " ✓ Prometheus OK"

# Test Loki
curl -s http://localhost:3100/ready && echo " ✓ Loki OK"

# Test Grafana
curl -s http://localhost:3000/api/health | jq -r '.database' && echo " ✓ Grafana OK"

# Check Prometheus targets
curl -s http://localhost:9090/api/v1/targets | jq '.data.activeTargets[].health'
Expected Output:
Prometheus is Healthy. ✓ Prometheus OK ready ✓ Loki OK ok ✓ Grafana OK "up" "up" "up"
🎉
Phase 2 Complete!

Core stack is running. Access Grafana at http://YOUR_SERVER_IP:3000 with admin / changeme123. Change this password immediately!

🎓 Part 4: Completion

Cleanup Commands

🖥️ [SERVER]
Bash [SERVER] 
cd ~/monitoring

# Option A: Stop services (preserve data for later)
docker compose stop

# Option B: Complete removal (DELETES ALL DATA)
docker compose down -v
sudo rm -rf ~/monitoring

Resume-Ready Skills

🎯 Technical Interview Questions
Q1: How would you detect a brute-force attack against your SSO portal using this stack?
A: Use Loki with LogQL to query authentication logs: {job="authentik"} |~ "login_failed". Create a Grafana alert that triggers when failed logins from a single IP exceed a threshold (e.g., 5 failures in 60 seconds). Combine with Prometheus metrics if available for defense-in-depth: metrics show volume, logs provide forensic detail.
Q2: Your SSO portal is intermittently unreachable. How would you diagnose using Blackbox Exporter?
A: Key metrics to examine: probe_success (1=up, 0=down), probe_duration_seconds (latency), probe_http_status_code. Check probe_ssl_earliest_cert_expiry for TLS issues and probe_dns_lookup_time_seconds to rule out DNS problems. Create a dashboard showing probe success rate over time with alerting on degradation.
Q3: Why use Vault for monitoring stack secrets instead of environment variables?
A: Environment variables in Docker Compose are stored in plaintext, visible in process listings, and often logged. Vault provides: (1) encryption at rest, (2) audit logging of all secret access, (3) dynamic credential rotation, (4) fine-grained access policies, (5) revocation capabilities. For production, secrets should never exist in version control or configuration files.
Q4: Your Loki disk is filling rapidly. What's your remediation plan?
A: Immediate: Reduce retention_period in Loki config (e.g., 30d to 7d), enable compactor with retention_enabled: true, restart Loki. Long-term: Analyze which jobs generate most logs with sum by (job) (rate({job=~".+"}[1h])), filter high-volume/low-value logs in Promtail pipeline stages, consider object storage (S3/GCS) for production scale.
Q5: How do you map Authentik groups to Grafana roles for RBAC?
A: Use JMESPath in Grafana's GF_AUTH_GENERIC_OAUTH_ROLE_ATTRIBUTE_PATH environment variable. Example: contains(groups[*], 'grafana-admins') && 'Admin' || contains(groups[*], 'grafana-editors') && 'Editor' || 'Viewer'. This evaluates group membership in order and assigns the first matching role, with Viewer as default fallback.
🏆
Congratulations!

You've built an enterprise-grade IAM observability platform. These are the exact skills used at companies like Netflix, Uber, and Datadog. Your next step: extend this lab by adding automated access reviews or converting to Terraform modules.