HashiCorp Vault

JIT Privileged Access & Dynamic Secrets

🔐 Vault 🔑 Secrets ⏱️ JIT Access 📋 Project B

📋 Introduction

Welcome to the HashiCorp Vault JIT Privileged Access lab. In this comprehensive exercise, you will deploy enterprise-grade secrets management with Just-In-Time (JIT) access patterns. By the end, you'll have dynamic database credentials, OIDC-based authentication via Authentik, and complete audit logging of all secret access.

⚠️
Prerequisite: Project A Required

This lab requires a working Zero-Trust SSO Gateway (Project A) with Authentik configured. Vault integrates with Authentik for OIDC-based authentication.

🏢
Enterprise Skill Transfer

HashiCorp Vault is the industry standard for secrets management, used by thousands of enterprises including Adobe, Uber, Stripe, and Bloomberg. Vault skills command premium salaries — IAM Engineers with secrets management expertise earn $120K-$180K annually.

🎯 Lab Objectives

🔧 How Vault Works

👤
You
🔑
Authentik
Login
🏦
Vault
Secrets
🎫
Credential
Temp
💾
Database
Access
💡
The Problem Vault Solves

Without Vault: Passwords stored in config files, environment variables, or code. Same password used everywhere, never rotated. No idea who accessed what credentials. Leaked credentials = compromised forever.

With Vault: Credentials are generated on-demand, auto-expire, and every access is logged!

⏱️ Time Investment

Phase Focus Time Machine
Phase 1 Core Concepts 20-30 min 📖 Reading
Phase 2 Vault Installation 30-45 min 🖥️ SERVER
Phase 3 Basic Configuration 30-45 min 🖥️ SERVER
Phase 4 Secrets Engines 45-60 min 🖥️ SERVER
Phase 5 Authentik Integration 30-45 min 🖥️ SERVER + 🌐 BROWSER
Phase 6 JIT Access Setup 45-60 min 🖥️ SERVER

Total: 4-5 hours (can be split across multiple sessions)


📑 Table of Contents

1

Phase 1: Core Concepts

Understanding Vault terminology before we build.

⏱️ 20-30 minutes 📖 Reading / Understanding

Vault has specific terminology. Understanding these concepts will make the hands-on sections much easier to follow.

🧠 Vault Terminology

🔐
Secrets
Any sensitive data: passwords, API keys, certificates, tokens
⚙️
Secrets Engine
Plugin that stores, generates, or encrypts secrets (KV, Database, SSH)
🔑
Auth Method
How users/apps prove identity (OIDC, Token, AppRole)
📜
Policy
Rules defining what paths a token can access
🎫
Token
Your identity in Vault, has attached policies
Lease
Time limit on secrets — they auto-expire!

📊 Secrets Engines We'll Use

Engine Purpose Example
KV (Key-Value) Store static secrets with versioning API keys, configuration values
Database Generate dynamic database credentials PostgreSQL, MySQL, MongoDB
Transit Encryption as a service Encrypt data without storing keys
PKI Certificate authority Issue TLS certificates
SSH SSH key signing/OTP Manage SSH access to servers
💡
Just-In-Time (JIT) Access Explained

Traditional: "Here's the database password, don't share it" (everyone shares it)

JIT: "Request access when needed, get unique credentials that expire automatically"

JIT means zero standing privileges — no one has permanent admin access. You request it, use it, and it disappears.


2

Phase 2: Vault Installation

Deploy HashiCorp Vault using Docker.

⏱️ 30-45 minutes 🖥️ Performed on: SERVER Machine

Task 2.1: Create Directory Structure

🖥️ SERVER Machine — Via SSH Connection
1
Create Vault Directories
Bash SERVER
# Navigate to your identity stack directory
cd ~/identity-stack

# Create directory structure for Vault
mkdir -p vault/config      # Configuration files
mkdir -p vault/data        # Persistent storage
mkdir -p vault/logs        # Log files
mkdir -p vault/policies    # HCL policy files

# Verify structure
tree vault/ 2>/dev/null || ls -la vault/

Task 2.2: Create Vault Configuration

🖥️ SERVER Machine — Via SSH Connection
2
Create Vault Configuration File
Bash SERVER
# Create the Vault configuration file
cat > ~/identity-stack/vault/config/vault-config.hcl << 'EOF'
# ============================================
# VAULT SERVER CONFIGURATION
# ============================================

# User interface settings
ui = true

# Listener configuration - how Vault accepts connections
listener "tcp" {
  address       = "0.0.0.0:8200"    # Listen on all interfaces
  tls_disable   = true              # Disable TLS (Traefik handles TLS)
}

# Storage backend - where secrets are stored
storage "file" {
  path = "/vault/data"
}

# API Address - how Vault identifies itself
api_addr = "http://127.0.0.1:8200"

# Cluster Address - for HA setups
cluster_addr = "http://127.0.0.1:8201"

# Disable memory lock (required for Docker)
disable_mlock = true

# Logging configuration
log_level = "info"
log_file  = "/vault/logs/vault.log"
EOF

# Verify the file was created
cat ~/identity-stack/vault/config/vault-config.hcl
💡
Storage Backend Options

We're using file storage for simplicity. In production, you'd use:

  • Raft (Integrated Storage) — Built-in HA, recommended for production
  • Consul — HashiCorp's service mesh with HA
  • PostgreSQL/MySQL — Database-backed storage

Task 2.3: Add Vault to Docker Compose

🖥️ SERVER Machine — Via SSH Connection
3
Create Vault Docker Compose Service

Add this service to your existing docker-compose.yml or create a new one:

YAML — docker-compose.vault.yml SERVER
# Create a separate compose file for Vault
cat > ~/identity-stack/docker-compose.vault.yml << 'EOF'
# ============================================
# HASHICORP VAULT - Secrets Management
# ============================================

version: "3.8"

services:
  vault:
    image: hashicorp/vault:1.15
    container_name: vault
    restart: unless-stopped
    cap_add:
      - IPC_LOCK                 # Required for memory locking
    environment:
      VAULT_ADDR: "http://127.0.0.1:8200"
      VAULT_API_ADDR: "http://127.0.0.1:8200"
    ports:
      - "8200:8200"              # Vault API/UI
    volumes:
      - ./vault/config:/vault/config:ro     # Configuration
      - ./vault/data:/vault/data            # Persistent data
      - ./vault/logs:/vault/logs            # Logs
      - ./vault/policies:/vault/policies:ro # Policies
    command: server -config=/vault/config/vault-config.hcl
    networks:
      - identity-network
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.vault.rule=Host(\`vault.yourdomain.com\`)"
      - "traefik.http.routers.vault.entrypoints=websecure"
      - "traefik.http.routers.vault.tls.certresolver=letsencrypt"
      - "traefik.http.services.vault.loadbalancer.server.port=8200"

networks:
  identity-network:
    external: true
EOF
4
Update Domain in Docker Compose
Bash SERVER
# Replace yourdomain.com with your actual domain
sed -i 's/yourdomain.com/YOUR_ACTUAL_DOMAIN.com/g' \
    ~/identity-stack/docker-compose.vault.yml

# Verify the change
grep "Host" ~/identity-stack/docker-compose.vault.yml

Task 2.4: Start Vault

🖥️ SERVER Machine — Via SSH Connection
5
Start the Vault Container
Bash SERVER
cd ~/identity-stack

# Pull the Vault image
docker compose -f docker-compose.vault.yml pull

# Start Vault
docker compose -f docker-compose.vault.yml up -d

# Check container is running
docker compose -f docker-compose.vault.yml ps

# View initial logs
docker logs vault

Task 2.5: Initialize and Unseal Vault

🖥️ SERVER Machine — Via SSH Connection

Vault starts in a sealed state and must be initialized. This generates your master keys — these are critical to save!

6
Initialize Vault (First Time Only)
Bash SERVER
# Initialize Vault with 1 key share (for lab simplicity)
# In production, use 5 key shares with threshold of 3
docker exec vault vault operator init -key-shares=1 -key-threshold=1

# ⚠️ CRITICAL: You will see output like this:
# Unseal Key 1: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
# Initial Root Token: hvs.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
# 
# SAVE THESE VALUES IMMEDIATELY!
🚨
CRITICAL: Save Your Keys!

Write down BOTH the Unseal Key and Root Token immediately!

  • Unseal Key: Required every time Vault restarts
  • Root Token: Initial admin access to Vault

If you lose these, you lose access to ALL your secrets forever. Store them in a password manager or physical safe.

7
Unseal Vault
Bash SERVER
# Unseal Vault with your unseal key
docker exec vault vault operator unseal YOUR_UNSEAL_KEY_HERE

# Check Vault status
docker exec vault vault status
Verification

Verify Vault is initialized and unsealed:

Bash SERVER
docker exec vault vault status
Expected Output:
Key Value --- ----- Seal Type shamir Initialized true Sealed false ← This should be "false" Total Shares 1 Threshold 1 Version 1.15.x Storage Type file Cluster Name vault-cluster-xxxxx HA Enabled false
🎉
Phase 2 Complete!

Vault is installed, initialized, and unsealed. You can now access the Vault UI at https://vault.yourdomain.com using your Root Token.


3

Phase 3: Basic Configuration

Enable secrets engines and create access policies.

⏱️ 30-45 minutes 🖥️ Performed on: SERVER Machine

Task 3.1: Login to Vault CLI

🖥️ SERVER Machine — Via SSH Connection
1
Authenticate with Root Token
Bash SERVER
# Login to Vault CLI (you'll be prompted for token)
docker exec -it vault vault login

# When prompted, paste your Root Token
# You should see: "Success! You are now authenticated."

# Verify you're logged in
docker exec vault vault token lookup

Task 3.2: Enable KV Secrets Engine

🖥️ SERVER Machine — Via SSH Connection

The KV (Key-Value) engine stores static secrets like API keys, configuration values, and passwords.

2
Enable KV Version 2
Bash SERVER
# Enable KV secrets engine version 2 at path "secret/"
docker exec vault vault secrets enable -path=secret kv-v2

# Verify it's enabled
docker exec vault vault secrets list

# Create a test secret
docker exec vault vault kv put secret/test \
    password="mysecretpassword" \
    username="testuser" \
    api_key="sk-1234567890"

# Read the secret back
docker exec vault vault kv get secret/test
Verification

You should see your test secret:

Expected Output:
====== Secret Path ====== secret/data/test ======= Metadata ======= Key Value --- ----- created_time 2024-01-02T10:00:00.000000000Z version 1 ====== Data ====== Key Value --- ----- api_key sk-1234567890 password mysecretpassword username testuser

Task 3.3: Create Access Policies

🖥️ SERVER Machine — Via SSH Connection

Policies control who can access what secrets. Let's create policies for different roles.

3
Create Admin Policy
Bash SERVER
# Create admin policy file
cat > ~/identity-stack/vault/policies/admin-policy.hcl << 'EOF'
# ============================================
# ADMIN POLICY - Full access to secrets
# ============================================

# Full access to all secrets
path "secret/*" {
  capabilities = ["create", "read", "update", "delete", "list"]
}

# Manage auth methods
path "auth/*" {
  capabilities = ["create", "read", "update", "delete", "list"]
}

# Manage policies
path "sys/policies/*" {
  capabilities = ["create", "read", "update", "delete", "list"]
}

# Manage mounts (secrets engines)
path "sys/mounts/*" {
  capabilities = ["create", "read", "update", "delete", "list"]
}
EOF

# Load the policy into Vault
docker exec vault vault policy write admin /vault/policies/admin-policy.hcl

# Verify policy was created
docker exec vault vault policy read admin
4
Create Developer Policy (Read-Only)
Bash SERVER
# Create developer policy file
cat > ~/identity-stack/vault/policies/developer-policy.hcl << 'EOF'
# ============================================
# DEVELOPER POLICY - Read-only access to app secrets
# ============================================

# Read app secrets only
path "secret/data/apps/*" {
  capabilities = ["read", "list"]
}

path "secret/metadata/apps/*" {
  capabilities = ["list"]
}

# Read database credentials
path "database/creds/readonly" {
  capabilities = ["read"]
}
EOF

# Load the policy
docker exec vault vault policy write developer /vault/policies/developer-policy.hcl

# List all policies
docker exec vault vault policy list
🎉
Phase 3 Complete!

You've enabled the KV secrets engine and created role-based access policies. Admins have full access, developers can only read specific paths.


🔧 Troubleshooting Guide

Common issues and their solutions when working with Vault.

🔴 Vault is Sealed
Cause: Vault automatically seals itself after a restart for security.
Solution: Unseal Vault using your unseal key:
docker exec vault vault operator unseal YOUR_UNSEAL_KEY
🔴 Permission Denied
Cause: Your token doesn't have a policy that allows this action.
Solutions:
  1. Check your token's policies: vault token lookup
  2. Use root token for admin tasks
  3. Update the policy to include the needed path
🔴 Lost Root Token or Unseal Key
Cause: Keys not saved during initialization.
Solution: Unfortunately, there's no way to recover lost keys. You must:
  1. Delete Vault data: rm -rf ~/identity-stack/vault/data/*
  2. Restart Vault: docker restart vault
  3. Re-initialize and SAVE THE KEYS THIS TIME!
🔴 OIDC Login Fails
Cause: Redirect URI mismatch or incorrect client secret.
Solutions:
  1. Verify redirect URIs match exactly in both Authentik and Vault
  2. Check client secret is correct
  3. Ensure Authentik provider scopes include: openid, profile, email, groups
  4. Check Authentik logs for errors

🧹 Cleanup Instructions

Remove Vault when finished with the lab.

🖥️ SERVER Machine — Via SSH Connection

Option A: Stop Vault (Preserve Data)

Bash SERVER
cd ~/identity-stack

# Stop Vault container (data preserved)
docker compose -f docker-compose.vault.yml stop

# Verify it's stopped
docker compose -f docker-compose.vault.yml ps

Option B: Complete Removal

⚠️
Warning: Permanent Data Loss

This will permanently delete all secrets stored in Vault. This cannot be undone.

Bash SERVER
cd ~/identity-stack

# Stop and remove Vault container
docker compose -f docker-compose.vault.yml down

# Remove all Vault data
rm -rf ~/identity-stack/vault/data/*
rm -rf ~/identity-stack/vault/logs/*

# Optionally remove configuration too
# rm -rf ~/identity-stack/vault/

🎓 Skills Acquired

🏆
Congratulations!

You've deployed enterprise-grade secrets management with Just-In-Time access patterns!

🚀 Next Steps