Grafana + IAM Observability

Enterprise Security Monitoring Lab

Grafana Prometheus Loki Vault Blackbox Project E

Part 1: The Blueprint

What Are We Building?

An enterprise-grade Observability Stack specifically designed for Identity & Access Management (IAM) environments. This is not a basic Grafana installation — you'll build a production-ready monitoring platform that captures IAM-specific events like failed logins, privilege escalations, MFA bypass attempts, and SSO health.

This lab integrates with Authentik SSO (Project A) and HashiCorp Vault (Project B) to demonstrate how IAM systems should be monitored in real enterprises.

What You'll Have When Done

Real-World Problem This Solves

Enterprise Context

The Problem: In a typical enterprise, security teams have no visibility into IAM events until a breach occurs. Failed login attempts, privilege escalations, and SSO outages go unnoticed. When the SSO portal goes down, users can't work — and IT finds out from angry Slack messages.

The Solution: This lab teaches you to build an IAM-aware observability platform that detects attacks in real-time, monitors SSO availability, and integrates with secrets management. Platform Engineers with these skills command $140K-$200K annually.

Table of Contents

Part 2: Preparation & Pre-Flight Check

Prerequisite Knowledge

Pre-Flight Environment Check Script

Before starting, run this script to verify your environment meets all requirements:

SERVER — Run this BEFORE starting the lab
Bash — check_reqs.sh SERVER 
#!/bin/bash
# ============================================
# Identity Bytes - Pre-Flight Check Script
# Run this before starting the Observability Lab
# ============================================

RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
NC='\033[0m' # No Color

PASS=0
FAIL=0

echo "============================================"
echo " Identity Bytes Pre-Flight Check"
echo "============================================"
echo ""

# Check 1: RAM (minimum 4GB, recommended 8GB)
TOTAL_RAM=$(free -g | awk '/^Mem:/{print $2}')
if [ "$TOTAL_RAM" -ge 8 ]; then
 echo -e "${GREEN}✓ RAM: ${TOTAL_RAM}GB (Excellent)${NC}"
 ((PASS++))
elif [ "$TOTAL_RAM" -ge 4 ]; then
 echo -e "${YELLOW} RAM: ${TOTAL_RAM}GB (Minimum met, 8GB recommended)${NC}"
 ((PASS++))
else
 echo -e "${RED}✗ RAM: ${TOTAL_RAM}GB (Minimum 4GB required)${NC}"
 ((FAIL++))
fi

# Check 2: Docker version (minimum 24.0)
if command -v docker &> /dev/null; then
    DOCKER_VER=$(docker --version | grep -oP '\d+\.\d+' | head -1)
 echo -e "${GREEN}✓ Docker: v${DOCKER_VER}${NC}"
 ((PASS++))
else
 echo -e "${RED}✗ Docker: Not installed${NC}"
 ((FAIL++))
fi

# Check 3: Docker Compose
if docker compose version &> /dev/null; then
    COMPOSE_VER=$(docker compose version | grep -oP '\d+\.\d+\.\d+')
 echo -e "${GREEN}✓ Docker Compose: v${COMPOSE_VER}${NC}"
 ((PASS++))
else
 echo -e "${RED}✗ Docker Compose: Not installed${NC}"
 ((FAIL++))
fi

# Check 4: Required ports
echo ""
echo "Checking ports..."
for port in 3000 9090 3100 9100 8080; do
 if ! ss -tuln | grep -q ":${port} "; then
 echo -e "${GREEN}✓ Port ${port}: Available${NC}"
 ((PASS++))
 else
 echo -e "${RED}✗ Port ${port}: In use${NC}"
 ((FAIL++))
 fi
done

# Check 5: Disk space (minimum 20GB free)
FREE_DISK=$(df -BG / | awk 'NR==2 {print $4}' | tr -d 'G')
if [ "$FREE_DISK" -ge 50 ]; then
 echo -e "${GREEN}✓ Disk: ${FREE_DISK}GB free (Excellent)${NC}"
 ((PASS++))
elif [ "$FREE_DISK" -ge 20 ]; then
 echo -e "${YELLOW} Disk: ${FREE_DISK}GB free (Minimum met, 50GB recommended)${NC}"
 ((PASS++))
else
 echo -e "${RED}✗ Disk: ${FREE_DISK}GB free (Minimum 20GB required)${NC}"
 ((FAIL++))
fi

# Summary
echo ""
echo "============================================"
echo " Results: ${PASS} passed, ${FAIL} failed"
echo "============================================"

if [ "$FAIL" -eq 0 ]; then
 echo -e "${GREEN} All checks passed! You're ready to start.${NC}"
 exit 0
else
 echo -e "${RED} Some checks failed. Fix issues before proceeding.${NC}"
 exit 1
fi
How to Run
Bash SERVER 
# Save and run the pre-flight check
nano ~/check_reqs.sh
# Paste the script above, save with Ctrl+X, Y, Enter

chmod +x ~/check_reqs.sh
./check_reqs.sh
Expected Output (All Green):
============================================ Identity Bytes Pre-Flight Check ============================================ ✓ RAM: 8GB (Excellent) ✓ Docker: v24.0 ✓ Docker Compose: v2.24.0 ✓ Port 3000: Available ✓ Port 9090: Available ... All checks passed! You're ready to start.
1

Phase 1: Architecture Deep-Dive

Understanding the network topology and port mappings before we build.

10-15 minutes Reading / Understanding

Network & Port Architecture

This diagram shows exactly which ports each component uses and how they communicate over the Docker bridge network:

Port Mapping & Data Flow
Grafana
:3000
Dashboard UI
Queries Prometheus & Loki
Prometheus
:9090
Metrics TSDB
Scrapes exporters
Loki
:3100
Log Aggregation
Receives from Promtail
Promtail
:9080
Log Shipper
Pushes to Loki
Node Exporter
:9100
System Metrics
CPU/RAM/Disk
cAdvisor
:8080
Container Metrics
Docker stats
Blackbox
:9115
Synthetic Probes
HTTP/ICMP checks
Vault
:8200
Secrets Management
Dynamic credentials

Data Flow Summary

Source Destination Protocol Purpose
Prometheus :9090 Node Exporter :9100 HTTP (pull) Scrape system metrics every 15s
Prometheus :9090 cAdvisor :8080 HTTP (pull) Scrape container metrics every 15s
Prometheus :9090 Blackbox :9115 HTTP (pull) Scrape probe results every 30s
Promtail :9080 Loki :3100 HTTP (push) Push logs in batches
Grafana :3000 Prometheus :9090 HTTP Query metrics via PromQL
Grafana :3000 Loki :3100 HTTP Query logs via LogQL
Grafana :3000 Vault :8200 HTTPS Retrieve secrets at startup
2

Phase 2: Core Stack Installation

Deploy Grafana, Prometheus, Loki, and supporting exporters.

30-45 minutes Performed on: SERVER

Task 2.1: Create Directory Structure

SERVER — Via SSH Connection
1
Create Monitoring Directories
Bash SERVER 
# Create monitoring stack directory structure
mkdir -p ~/monitoring/{grafana,prometheus,loki,promtail,blackbox}
mkdir -p ~/monitoring/grafana/{data,provisioning/datasources,provisioning/dashboards}
mkdir -p ~/monitoring/prometheus/{data,rules}
mkdir -p ~/monitoring/loki/data

# Set correct ownership for container users
# Grafana=472, Prometheus=65534, Loki=10001
sudo chown -R 472:472 ~/monitoring/grafana/data
sudo chown -R 65534:65534 ~/monitoring/prometheus/data
sudo chown -R 10001:10001 ~/monitoring/loki/data

# Navigate to monitoring directory
cd ~/monitoring

# Verify structure
find ~/monitoring -type d | head -20

Common Errors — Directory Setup

Error Cause Fix
Permission denied on /var/lib/grafana Grafana container runs as UID 472 sudo chown -R 472:472 ~/monitoring/grafana/data
Prometheus TSDB locked Prometheus runs as UID 65534 (nobody) sudo chown -R 65534:65534 ~/monitoring/prometheus/data
Loki cannot write to /loki Loki runs as UID 10001 sudo chown -R 10001:10001 ~/monitoring/loki/data

Task 2.2: Create Prometheus Configuration with IAM Targets

SERVER — Via SSH Connection
2
Create prometheus.yml
YAML — prometheus.yml SERVER 
cat > ~/monitoring/prometheus/prometheus.yml << 'EOF'
# ============================================
# PROMETHEUS CONFIGURATION
# Identity Bytes - IAM Observability Lab
# ============================================
global:
  scrape_interval: 15s
  evaluation_interval: 15s
  external_labels:
    environment: 'homelab'
    project: 'identity-bytes'

alerting:
  alertmanagers:
    - static_configs:
        - targets: []

rule_files:
  - /etc/prometheus/rules/*.yml

scrape_configs:
  # ==========================================
  # Self-monitoring
  # ==========================================
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']
        labels:
          service: 'prometheus'

  # ==========================================
  # Infrastructure Metrics
  # ==========================================
  - job_name: 'node-exporter'
    static_configs:
      - targets: ['node-exporter:9100']
        labels:
          service: 'node'

  - job_name: 'cadvisor'
    static_configs:
      - targets: ['cadvisor:8080']
        labels:
          service: 'docker'

  # ==========================================
  # Synthetic Monitoring (Blackbox Exporter)
  # Monitors IAM portal availability
  # ==========================================
  - job_name: 'blackbox-http'
    metrics_path: /probe
    params:
      module: [http_2xx]
    static_configs:
      - targets:
          # Add your IAM endpoints here
          - https://auth.yourdomain.com          # Authentik SSO
          - https://vault.yourdomain.com         # HashiCorp Vault
          - https://teleport.yourdomain.com      # Teleport ZTNA
        labels:
          probe_type: 'iam_portal'
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: blackbox:9115

  # ==========================================
  # Authentik Metrics (if exposed)
  # ==========================================
 - job_name: 'authentik'
 static_configs:
 - targets: ['authentik-server:9300']
 labels:
 service: 'authentik'
 honor_labels: true
EOF

echo "✓ Prometheus config created"
Verification
Bash SERVER 
# Verify YAML syntax
cat ~/monitoring/prometheus/prometheus.yml | head -30

Task 2.3: Create Loki Configuration with Retention Policies

SERVER — Via SSH Connection
3
Create loki-config.yml with 7-Day Retention
YAML — loki-config.yml SERVER 
cat > ~/monitoring/loki/loki-config.yml << 'EOF'
# ============================================
# LOKI CONFIGURATION
# Optimized for self-hosted: 7-day retention
# ============================================
auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 9096
  log_level: warn

common:
  path_prefix: /loki
  storage:
    filesystem:
      chunks_directory: /loki/chunks
      rules_directory: /loki/rules
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory

schema_config:
  configs:
    - from: 2020-10-24
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 24h

storage_config:
  boltdb_shipper:
    active_index_directory: /loki/boltdb-shipper-active
    cache_location: /loki/boltdb-shipper-cache
    shared_store: filesystem
  filesystem:
    directory: /loki/chunks

# ==========================================
# RETENTION POLICY - Critical for disk management!
# ==========================================
limits_config:
  retention_period: 168h     # 7 days (adjust for your storage)
  max_query_length: 721h     # 30 days max query range
  max_query_parallelism: 2   # Low resource usage

compactor:
 working_directory: /loki/compactor
 shared_store: filesystem
 retention_enabled: true
 retention_delete_delay: 2h
 retention_delete_worker_count: 150
EOF

echo "✓ Loki config created with 7-day retention"
Cost & Resource Optimization

The retention_period: 168h (7 days) prevents your disk from filling up. For production environments with more storage, increase to 720h (30 days). Monitor disk usage with: df -h ~/monitoring/loki

Task 2.4: Create Blackbox Exporter Configuration

SERVER — Via SSH Connection
4
Create blackbox.yml for IAM Portal Monitoring
YAML — blackbox.yml SERVER 
cat > ~/monitoring/blackbox/blackbox.yml << 'EOF'
# ============================================
# BLACKBOX EXPORTER - Synthetic Monitoring
# Simulates user login checks for IAM portals
# ============================================
modules:
  # Standard HTTP check (expect 200 OK)
  http_2xx:
    prober: http
    timeout: 10s
    http:
      valid_http_versions: ["HTTP/1.1", "HTTP/2.0"]
      valid_status_codes: [200, 301, 302]
      method: GET
      follow_redirects: true
      preferred_ip_protocol: "ip4"
      tls_config:
        insecure_skip_verify: false

  # SSO Login Page Check
  # Verifies login form is present
  http_sso_login:
    prober: http
    timeout: 15s
    http:
      valid_http_versions: ["HTTP/1.1", "HTTP/2.0"]
      method: GET
      fail_if_body_not_matches_regexp:
        - ".*login.*|.*sign.?in.*|.*authenticate.*"
      follow_redirects: true
      tls_config:
        insecure_skip_verify: false

  # ICMP Ping Check
  icmp:
    prober: icmp
    timeout: 5s

  # TCP Port Check (for services without HTTP)
 tcp_connect:
 prober: tcp
 timeout: 5s
EOF

echo "✓ Blackbox config created for IAM monitoring"

Task 2.5: Create Complete Docker Compose

SERVER — Via SSH Connection
5
Create docker-compose.yml
YAML — docker-compose.yml SERVER 
cat > ~/monitoring/docker-compose.yml << 'EOF'
version: '3.8'

services:
  # ==========================================
  # GRAFANA - Visualization Dashboard
  # ==========================================
  grafana:
    image: grafana/grafana:latest
    container_name: grafana
    restart: unless-stopped
    ports:
      - "3000:3000"
    environment:
      # Default admin (change immediately!)
      - GF_SECURITY_ADMIN_USER=admin
      - GF_SECURITY_ADMIN_PASSWORD=changeme123
      - GF_USERS_ALLOW_SIGN_UP=false
      # Authentik SSO (configure in Phase 4)
      - GF_AUTH_GENERIC_OAUTH_ENABLED=false
    volumes:
      - ./grafana/data:/var/lib/grafana
      - ./grafana/provisioning:/etc/grafana/provisioning
    networks:
      - monitoring
    depends_on:
      - prometheus
      - loki

  # ==========================================
  # PROMETHEUS - Metrics Collection
  # ==========================================
  prometheus:
    image: prom/prometheus:latest
    container_name: prometheus
    restart: unless-stopped
    ports:
      - "9090:9090"
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
      - '--storage.tsdb.retention.time=15d'
      - '--web.enable-lifecycle'
      - '--web.enable-admin-api'
    volumes:
      - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - ./prometheus/rules:/etc/prometheus/rules:ro
      - ./prometheus/data:/prometheus
    networks:
      - monitoring

  # ==========================================
  # LOKI - Log Aggregation
  # ==========================================
  loki:
    image: grafana/loki:2.9.0
    container_name: loki
    restart: unless-stopped
    ports:
      - "3100:3100"
    command: -config.file=/etc/loki/loki-config.yml
    volumes:
      - ./loki/loki-config.yml:/etc/loki/loki-config.yml:ro
      - ./loki/data:/loki
    networks:
      - monitoring

  # ==========================================
  # PROMTAIL - Log Shipper
  # ==========================================
  promtail:
    image: grafana/promtail:2.9.0
    container_name: promtail
    restart: unless-stopped
    volumes:
      - ./promtail/promtail-config.yml:/etc/promtail/config.yml:ro
      - /var/log:/var/log:ro
      - /var/lib/docker/containers:/var/lib/docker/containers:ro
    command: -config.file=/etc/promtail/config.yml
    networks:
      - monitoring

  # ==========================================
  # NODE EXPORTER - System Metrics
  # ==========================================
  node-exporter:
    image: prom/node-exporter:latest
    container_name: node-exporter
    restart: unless-stopped
    ports:
      - "9100:9100"
    command:
      - '--path.procfs=/host/proc'
      - '--path.sysfs=/host/sys'
      - '--collector.filesystem.mount-points-exclude=^/(sys|proc|dev|host|etc)($$|/)'
    volumes:
      - /proc:/host/proc:ro
      - /sys:/host/sys:ro
      - /:/rootfs:ro
    networks:
      - monitoring

  # ==========================================
  # CADVISOR - Container Metrics
  # ==========================================
  cadvisor:
    image: gcr.io/cadvisor/cadvisor:latest
    container_name: cadvisor
    restart: unless-stopped
    ports:
      - "8080:8080"
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:ro
      - /sys:/sys:ro
      - /var/lib/docker/:/var/lib/docker:ro
    networks:
      - monitoring

  # ==========================================
  # BLACKBOX EXPORTER - Synthetic Monitoring
  # ==========================================
 blackbox:
 image: prom/blackbox-exporter:latest
 container_name: blackbox
 restart: unless-stopped
 ports:
 - "9115:9115"
 command:
 - '--config.file=/etc/blackbox/blackbox.yml'
 volumes:
 - ./blackbox/blackbox.yml:/etc/blackbox/blackbox.yml:ro
 networks:
 - monitoring

networks:
 monitoring:
 driver: bridge
EOF

echo "✓ Docker Compose created"

Task 2.6: Create Promtail Configuration

SERVER — Via SSH Connection
6
Create promtail-config.yml
YAML — promtail-config.yml SERVER 
cat > ~/monitoring/promtail/promtail-config.yml << 'EOF'
# ============================================
# PROMTAIL CONFIGURATION
# ============================================
server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:
  # Docker container logs
  - job_name: containers
    static_configs:
      - targets:
          - localhost
        labels:
          job: containerlogs
          __path__: /var/lib/docker/containers/*/*log

    pipeline_stages:
      - json:
          expressions:
            output: log
            stream: stream
            attrs:
      - json:
          expressions:
            tag:
          source: attrs
      - regex:
          expression: (?P(?:[a-zA-Z0-9][a-zA-Z0-9_.-]+))
          source: tag
      - labels:
          stream:
          container_name:
      - output:
          source: output

  # System auth logs (IAM relevant)
  - job_name: auth
    static_configs:
      - targets:
          - localhost
        labels:
          job: auth
          __path__: /var/log/auth.log

  # System syslog
 - job_name: syslog
 static_configs:
 - targets:
 - localhost
 labels:
 job: syslog
 __path__: /var/log/syslog
EOF

echo "✓ Promtail config created"

Task 2.7: Create Grafana Data Sources

SERVER — Via SSH Connection
7
Auto-Provision Data Sources
YAML — datasources.yml SERVER 
cat > ~/monitoring/grafana/provisioning/datasources/datasources.yml << 'EOF'
apiVersion: 1

datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true
    editable: false

  - name: Loki
    type: loki
    access: proxy
    url: http://loki:3100
    editable: false
EOF

echo "✓ Data sources configured"

Task 2.8: Start the Stack

SERVER — Via SSH Connection
8
Deploy All Services
Bash SERVER 
cd ~/monitoring

# Pull all images
docker compose pull

# Start all services
docker compose up -d

# Check all containers are running
docker compose ps
Verification — All Endpoints
Bash SERVER 
# Test all endpoints
echo "Testing Prometheus..." && curl -s http://localhost:9090/-/healthy
echo "Testing Loki..." && curl -s http://localhost:3100/ready
echo "Testing Grafana..." && curl -s http://localhost:3000/api/health | jq .database
echo "Testing Blackbox..." && curl -s http://localhost:9115/metrics | head -5
echo "Testing Node Exporter..." && curl -s http://localhost:9100/metrics | head -5

# Check Prometheus targets
curl -s http://localhost:9090/api/v1/targets | jq '.data.activeTargets[].health'
Expected Output:
Testing Prometheus... Prometheus is Healthy. Testing Loki... ready Testing Grafana... "ok" Testing Blackbox... # HELP blackbox_exporter_build_info Testing Node Exporter... # HELP go_gc_duration_seconds "up" "up" "up"
Phase 2 Complete!

Core stack is running! Access Grafana at http://YOUR_SERVER_IP:3000 with admin / changeme123. Change this password immediately!

8

Phase 8: Break & Fix Challenge

Intentionally break the configuration to practice troubleshooting.

20-30 minutes Hands-on Challenge
The Break & Fix Challenge

The best way to learn troubleshooting is to break things on purpose. This exercise teaches you how to use Grafana dashboards and logs to diagnose real problems.

Challenge 1: Break Prometheus Scraping

SERVER — Via SSH Connection
1
Intentionally Break the Config
Bash SERVER 
# Edit prometheus.yml and introduce a typo
sed -i "s/job_name: 'node-exporter'/job_name: 'node-exporterXXX'/" ~/monitoring/prometheus/prometheus.yml

# Reload Prometheus config
curl -X POST http://localhost:9090/-/reload

# Wait 30 seconds, then check targets
sleep 30
curl -s http://localhost:9090/api/v1/targets | jq '.data.activeTargets[] | {job: .labels.job, health: .health}'

Your Task: Find the Problem Using Grafana

  1. Open Grafana at http://YOUR_SERVER_IP:3000
  2. Go to Explore Prometheus
  3. Run query: up{job=~"node.*"}
  4. Notice the job name has changed — metrics are now under node-exporterXXX
  5. Check Status Targets in Prometheus UI to see the renamed job

Fix It

Bash SERVER 
# Fix the typo
sed -i "s/job_name: 'node-exporterXXX'/job_name: 'node-exporter'/" ~/monitoring/prometheus/prometheus.yml

# Reload config
curl -X POST http://localhost:9090/-/reload

# Verify fix
curl -s http://localhost:9090/api/v1/targets | jq '.data.activeTargets[] | select(.labels.job == "node-exporter") | .health'

Challenge 2: Break Loki Log Shipping

SERVER — Via SSH Connection
2
Break Promtail Loki Connection
Bash SERVER 
# Change Loki URL to wrong port
sed -i "s|url: http://loki:3100|url: http://loki:9999|" ~/monitoring/promtail/promtail-config.yml

# Restart Promtail
docker compose restart promtail

# Wait and check logs
sleep 10
docker logs promtail --tail=20 2>&1 | grep -i "error\|failed"

Your Task: Diagnose Using Grafana

  1. Go to Explore Loki
  2. Run query: {job="containerlogs"}
  3. Notice: No new logs appearing (last log is from before the break)
  4. Check Promtail container logs to find connection errors

Fix It

Bash SERVER 
# Fix the URL
sed -i "s|url: http://loki:9999|url: http://loki:3100|" ~/monitoring/promtail/promtail-config.yml

# Restart Promtail
docker compose restart promtail

# Verify logs are flowing again
sleep 15
curl -s "http://localhost:3100/loki/api/v1/query?query={job=\"containerlogs\"}&limit=1" | jq '.data.result[0].stream'
Challenge Complete!

You've learned how to diagnose real problems using Grafana, Prometheus targets, and container logs. These are the exact skills used in production incident response.

Part 4: Completion

Cleanup Instructions

SERVER
Bash SERVER 
cd ~/monitoring

# Option A: Stop services (preserve data)
docker compose stop

# Option B: Complete removal (DELETES ALL DATA)
docker compose down -v
sudo rm -rf ~/monitoring

Skills Gained

Resume-Ready Skills

Skill Enterprise Application
Grafana + Prometheus + LokiFull observability stack deployment and management
IAM Audit LoggingSecurity event monitoring and compliance reporting
Blackbox ExporterSynthetic monitoring and SLA tracking
HashiCorp Vault IntegrationDynamic secrets management for infrastructure
PromQL + LogQLQuery languages for metrics and log analysis
Infrastructure as CodeAnsible-based deployment automation
Interview Questions — Prove Your Mastery
Q1: How would you detect a brute-force attack against your SSO portal using Prometheus and Loki?
A: Use Loki to query authentication logs with LogQL: {container_name="authentik-server"} |= "Login failed". Create a Grafana alert that triggers when failed logins from a single IP exceed a threshold (e.g., 5 failures in 60 seconds). For metrics, if Authentik exposes a authentik_login_failures_total counter, alert when rate(authentik_login_failures_total[5m]) > 1. Combine both for defense-in-depth: metrics for volume, logs for forensic detail.
Q2: Your SSO portal is intermittently unreachable. How would you use Blackbox Exporter to diagnose?
A: Configure Blackbox Exporter to probe the SSO URL every 30 seconds with the http_2xx module. Key metrics: probe_success (1=up, 0=down), probe_duration_seconds (latency), probe_http_status_code. Create a Grafana dashboard showing probe success rate over time. If probe_success == 0, check probe_http_status_code and probe_ssl_earliest_cert_expiry for TLS issues. Also check probe_dns_lookup_time_seconds to rule out DNS problems.
Q3: Why is Loki's label-only indexing better for log aggregation than Elasticsearch's full-text indexing?
A: Loki only indexes labels (metadata like job, container_name, level), not log content. This makes it 10-100x cheaper to operate than Elasticsearch for the same log volume. Trade-off: Loki searches are slower for arbitrary text patterns, but very fast when filtered by labels first. For IAM logs, you typically filter by {service="authentik"} first, then grep for specific patterns — Loki excels at this. For security forensics requiring full-text search across all logs, Elasticsearch may be warranted despite higher cost.
Q4: How would you integrate Grafana RBAC with Authentik SSO to enforce least-privilege access?
A: Configure Grafana as an OIDC client in Authentik. Map Authentik groups to Grafana roles using role_attribute_path in Grafana's config (e.g., contains(groups[*], 'grafana-admins') && 'Admin' || 'Viewer'). Create three Authentik groups: grafana-viewers (read-only), grafana-editors (can create dashboards), grafana-admins (full access). This enforces SSO-based RBAC: users inherit dashboard permissions from their group membership without local Grafana accounts.
Q5: Your Loki disk is filling up rapidly. Walk through your remediation steps.
A: Immediate: Check current usage with df -h and identify Loki's chunk directory. Short-term: Reduce retention_period in loki-config.yml (e.g., from 30d to 7d), enable compactor's retention_enabled: true, and restart Loki. The compactor will garbage-collect old chunks. Long-term: Analyze which labels generate the most logs (sum by (job) (rate({job=~".+"}[1h]))) and filter high-volume, low-value logs in Promtail's pipeline. Consider moving to object storage (S3/GCS) for production workloads.

Additional Resources

Congratulations!

You've built an enterprise-grade IAM observability platform with synthetic monitoring, RBAC, secrets management, and security hardening. These are the exact skills used at companies like Netflix, Uber, and Datadog.