LAB 06: Keycloak
High Availability

Lab Metadata

Lab Title and Description

No Single Point of Failure: Clustering Keycloak with an External Database and a Replicated Cache

Across the first five labs you built something powerful and, quietly, something dangerous. Every application at Northgate now delegates its login to one Keycloak container backed by an embedded database. That container is convenient, and it is a single point of failure sitting under the entire estate. When it stops, every application's login stops at the same moment, and because the database lives inside the same container, a lost disk is a lost realm. Production identity cannot work this way. The identity provider is usually the first service that must come up in a disaster and the last that is allowed to go down.

This lab re-architects Keycloak into a genuinely available service, and it does so in the order a real migration follows. First you externalise state: the realm, users, clients, and credentials move out of the throwaway embedded database into a dedicated PostgreSQL instance, so a Keycloak node becomes stateless and disposable. Then you run two nodes instead of one and teach them to find each other and share live session state through an Infinispan distributed cache, using JGroups discovery over that same database so there is no fragile static node list. Then you place an Nginx load balancer in front, so clients reach the cluster through one stable address while traffic spreads across the nodes.

The proof is the part that matters. You will log in as a real user through the load balancer, identify which node served you, and then kill that node while the session is live. The user stays logged in, served seamlessly by the surviving node, because the session was replicated the moment it was created. You will then bring the dead node back and watch it rejoin the cluster and re-balance. By the end you will have a reference highly available IdP and, more importantly, the ability to reason about where identity state lives, how it survives failure, and what a load balancer must and must not do in front of an authentication server. Estimated completion time is 4 to 4.5 hours.

Prerequisites

3.1 Prior labs required

# The existing single node and directory must be running so we can
# export the realm from it before rebuilding.
docker start ib-openldap ib-keycloak ib-phpldapadmin

curl -s http://localhost:8081/realms/northgate/.well-known/openid-configuration | jq -r '.issuer'
# Expected: http://localhost:8081/realms/northgate

3.2 System requirements

3.3 Required tools and versions

3.4 Verification of tools

docker compose version
# Expected: Docker Compose version v2.x.x
docker network inspect ib-lab-net >/dev/null 2>&1 && echo "ib-lab-net present" || \
  echo "ib-lab-net missing (it was created in Lab 01)"

Real World Problem Statement

This lab solves the problem that the previous five created by succeeding: concentration of dependency. Centralising authentication was the right architecture, but it means the identity provider is now on the critical path of every application at once. A single node IdP turns any routine event, a host reboot, a failed disk, an out of memory kill, a bad deployment, into a total authentication outage. Nobody can log in to anything. Worse, with the database embedded in the node, the availability problem and the durability problem are the same problem: lose the node's storage and you lose the realm, the clients, and every locally stored credential.

Highly available identity is not optional at enterprise scale, and it has a specific shape. State must live outside any single compute node, in a database that is itself backed up and, in production, replicated. Compute nodes must be stateless and horizontally scalable, so any node can serve any request and losing one costs capacity rather than availability. Live session state, which by nature is not in the database on every request, must be replicated across nodes so that a user whose node dies is not silently logged out. And a load balancer must present one address, spread load, and route away from unhealthy nodes, without breaking the stateful parts of the login journey. Getting this wrong produces an IdP that appears redundant but logs everyone out the moment a node restarts, which is the failure a senior architect is expected to prevent.

Why it matters, across four dimensions

Concrete scenario

Northgate Financial has a bad Tuesday. The virtual machine hosting the sole Keycloak node is rebooted during routine patching, and for fifteen minutes nobody in the company can log in to anything: the HR portal, the acquired Harborview systems, the internal tools, all delegate to an IdP that is not there. The incident review is blunt. The identity provider is classified as a critical service with no redundancy and a single point of failure that also co-locates its own database. The board sets a requirement: the IdP must survive the loss of any single node with no user visible outage and no forced re-authentication, and its data must be durable independently of any compute node. Your task in this lab is the reference build that meets that requirement: migrate the realm to an external PostgreSQL database, run a clustered pair of stateless Keycloak nodes that replicate session state, front them with a load balancer, and prove by killing a node mid session that a live user stays logged in.

Skills Mapped to Production Solutions

Architecture Overview

Component breakdown

Data flow (a login through the highly available cluster)

1. The client reaches one address: the browser or application connects to the load balancer on the single published address, never to a node directly. Why: nodes can be added, removed, or restarted without any client ever changing its configuration.
2. The balancer selects a healthy node: Nginx forwards the request to node 1 or node 2, skipping any node its health check has marked down, and adds the X-Forwarded headers. Why: Keycloak must know the original scheme and host to build correct redirect and token URLs, and the balancer must never send traffic to a dead node.
3. The node authenticates against shared state: the chosen node reads realm and client configuration from PostgreSQL and validates the password against OpenLDAP. Why: because state is external, any node has the full picture; there is no primary node that owns the truth.
4. The session is written to the replicated cache: when the login succeeds, the session is created in the Infinispan distributed cache, and a replica is placed on the other node. Why: this is the step that makes failover invisible; the session does not belong to the node that created it.
5. A later request can land on either node: a subsequent request routed to the other node finds the session already present in its cache and serves it without a re-login. Why: with replicated sessions the cluster behaves as one logical IdP, so losing a node loses capacity, not sessions.

Security considerations

Step by Step Implementation

# Export inside the running single node, writing to a directory,
# then copy that file out to the host.
docker exec ib-keycloak /opt/keycloak/bin/kc.sh export \
  --dir /tmp/realm-export --realm northgate

mkdir -p ~/ib-sia-06/import
docker cp ib-keycloak:/tmp/realm-export/northgate-realm.json \
  ~/ib-sia-06/import/northgate-realm.json

# Confirm the export is real and names the realm.
jq -r '.realm, (.users | length | tostring) + " users", (.clients | length | tostring) + " clients"' \
  ~/ib-sia-06/import/northgate-realm.json

# Keep OpenLDAP running; it is shared. Stop only the old Keycloak and its
# admin helper. We rename rather than delete, so rollback stays possible.
docker stop ib-keycloak ib-phpldapadmin
docker rename ib-keycloak ib-keycloak-singlenode-old

# Confirm nothing now holds host port 8081.
(curl -s -o /dev/null -w "%{http_code}\n" http://localhost:8081/ || echo "port free")

mkdir -p ~/ib-sia-06/conf
cat > ~/ib-sia-06/conf/cache-ispn-jdbc.xml << 'EOF'
<?xml version="1.0" encoding="UTF-8"?>
<infinispan
    xmlns="urn:infinispan:config:15.0">
  <jgroups>
    <!-- A TCP based stack that discovers members via the shared database.
         extends="tcp" keeps all the default protocols and swaps only
         the discovery protocol for JDBC_PING2. -->
    <stack name="jdbc-ping-tcp" extends="tcp">
      <JDBC_PING2
          connection_driver="org.postgresql.Driver"
          connection_url="jdbc:postgresql://ib-postgres:5432/keycloak"
          connection_username="keycloak"
          connection_password="${env.KC_DB_PASSWORD}"
          initialize_sql="CREATE TABLE IF NOT EXISTS JGROUPSPING (own_addr varchar(200) NOT NULL, cluster_name varchar(200) NOT NULL, bind_addr varchar(200) NOT NULL, updated timestamp default current_timestamp, ping_data BYTEA, constraint PK_JGROUPSPING PRIMARY KEY (own_addr, cluster_name))"
          stack.combine="REPLACE"
          stack.position="MPING" />
    </stack>
  </jgroups>
  <cache-container name="keycloak">
    <transport stack="jdbc-ping-tcp" node-name="${env.KC_NODE_NAME:}"/>
  </cache-container>
</infinispan>
EOF
echo "wrote cache-ispn-jdbc.xml"

cat > ~/ib-sia-06/docker-compose.yml << 'EOF'
services:
  ib-postgres:
    image: postgres:16
    container_name: ib-postgres
    environment:
      POSTGRES_DB: keycloak
      POSTGRES_USER: keycloak
      POSTGRES_PASSWORD: ${KC_DB_PASSWORD}
    volumes:
      - ib-pgdata:/var/lib/postgresql/data
    networks: [ib-lab-net]
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U keycloak -d keycloak"]
      interval: 5s
      timeout: 5s
      retries: 10

  ib-keycloak-1: &kc
    image: quay.io/keycloak/keycloak:26.0
    container_name: ib-keycloak-1
    command: start --optimized --import-realm
    depends_on:
      ib-postgres:
        condition: service_healthy
    environment: &kcenv
      KC_BOOTSTRAP_ADMIN_USERNAME: admin
      KC_BOOTSTRAP_ADMIN_PASSWORD: ${KC_ADMIN_PASSWORD}
      KC_DB: postgres
      KC_DB_URL: jdbc:postgresql://ib-postgres:5432/keycloak
      KC_DB_USERNAME: keycloak
      KC_DB_PASSWORD: ${KC_DB_PASSWORD}
      KC_HOSTNAME: http://localhost:8081
      KC_HOSTNAME_STRICT: "false"
      KC_HTTP_ENABLED: "true"
      KC_PROXY_HEADERS: xforwarded
      KC_HEALTH_ENABLED: "true"
      KC_METRICS_ENABLED: "true"
      KC_CACHE: ispn
      KC_CACHE_CONFIG_FILE: cache-ispn-jdbc.xml
      KC_NODE_NAME: node1
    volumes:
      - ./conf/cache-ispn-jdbc.xml:/opt/keycloak/conf/cache-ispn-jdbc.xml:ro
      - ./import:/opt/keycloak/data/import:ro
    networks: [ib-lab-net]

  ib-keycloak-2:
    <<: *kc
    container_name: ib-keycloak-2
    environment:
      <<: *kcenv
      KC_NODE_NAME: node2
    depends_on:
      ib-postgres:
        condition: service_healthy
      ib-keycloak-1:
        condition: service_started

volumes:
  ib-pgdata:

networks:
  ib-lab-net:
    external: true
EOF
echo "wrote docker-compose.yml"

cd ~/ib-sia-06

# Secrets live here and this file must never be committed.
cat > .env << 'EOF'
KC_DB_PASSWORD=ChangeMe_DbStrong#2026
KC_ADMIN_PASSWORD=ChangeMe_AdminStrong#2026
EOF
chmod 600 .env

# If your image is not pre-built, swap 'start --optimized' for 'start'
# in docker-compose.yml before this step.
docker compose up -d

# Watch node 1 come up and import the realm (Ctrl C to stop watching).
docker compose logs -f ib-keycloak-1 | grep -m1 "Imported realm northgate"

cat > ~/ib-sia-06/conf/nginx.conf << 'EOF'
events {}
http {
  upstream keycloak_cluster {
    # Round robin across both nodes for this lab so distribution is visible.
    # max_fails + fail_timeout give passive health checking: after 2 failed
    # attempts a node is treated as down for 10s and traffic routes elsewhere.
    server ib-keycloak-1:8080 max_fails=2 fail_timeout=10s;
    server ib-keycloak-2:8080 max_fails=2 fail_timeout=10s;
  }

  server {
    listen 8081;

    location / {
      proxy_pass http://keycloak_cluster;

      # Tell Keycloak the original request details so it builds correct URLs.
      proxy_set_header Host              $host;
      proxy_set_header X-Real-IP         $remote_addr;
      proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
      proxy_set_header X-Forwarded-Proto $scheme;
      proxy_set_header X-Forwarded-Host  $host;
      proxy_set_header X-Forwarded-Port  8081;

      # Retry the other node if one refuses or errors mid request.
      proxy_next_upstream error timeout http_502 http_503 http_504;

      # DEMO ONLY: expose which backend served this response.
      add_header X-Upstream-Node $upstream_addr always;
    }
  }
}
EOF
echo "wrote nginx.conf"

# Append the nginx service to docker-compose.yml.
cat >> ~/ib-sia-06/docker-compose.yml << 'EOF'

  ib-lb:
    image: nginx:1.27
    container_name: ib-lb
    depends_on: [ib-keycloak-1, ib-keycloak-2]
    ports:
      - "8081:8081"
    volumes:
      - ./conf/nginx.conf:/etc/nginx/nginx.conf:ro
    networks: [ib-lab-net]
EOF

cd ~/ib-sia-06 && docker compose up -d ib-lb

cd ~/ib-sia-06

# Get tokens through the load balancer (direct grant is fine for this test;
# if you disabled it in Lab 04, use the account console session instead).
# Here we read a refresh token from a code flow you run, or reuse tokens.json.

# Suppose $RT holds a valid refresh token obtained via the cluster.
# Kill one node (done above), then refresh through the balancer:
curl -s -X POST http://localhost:8081/realms/northgate/protocol/openid-connect/token \
  -d grant_type=refresh_token -d client_id=hr-portal \
  -d client_secret="<CLIENT_SECRET>" -d refresh_token="$RT" \
  | jq -r '.access_token[0:16] + "...  (refresh succeeded on surviving node)"'
# Expected: a new access token, served by the node that is still up.

docker start ib-keycloak-1

# Watch it rediscover the cluster through the database and rejoin.
docker compose logs -f ib-keycloak-1 | grep -m1 -iE "ISPN000094|view"

# After a few seconds the balancer's fail_timeout lapses and traffic
# spreads across both nodes again.
for i in 1 2 3 4; do
  curl -s -D - http://localhost:8081/realms/northgate/ -o /dev/null | grep -i x-upstream-node
done

Testing and Validation

End to end scenario: the board requirement, verified

This confirms the three properties the incident review demanded: durable state, load spread across nodes, and session survival on node loss.

# STAGE 1: state is durable and external. Recreate a node from scratch and
# confirm the realm is still there (it lives in PostgreSQL, not the node).
docker compose rm -sf ib-keycloak-2 && docker compose up -d ib-keycloak-2
curl -s http://localhost:8081/realms/northgate/.well-known/openid-configuration \
  | jq -r '.issuer'
# Expected: the realm answers; destroying and recreating a node lost nothing.

# STAGE 2: load spreads across nodes.
for i in $(seq 1 6); do
  curl -s -D - http://localhost:8081/realms/northgate/ -o /dev/null | grep -i x-upstream-node
done | sort | uniq -c
# Expected: both node addresses appear.

# STAGE 3: session survives node loss (Phase 4). Log in, kill the serving
# node, confirm the session persists with no re-authentication.

Negative and resilience tests

# TEST N1: a lonely cluster of one is the classic misconfiguration.
# Confirm the cluster view is size 2, not two clusters of size 1.
docker compose logs ib-keycloak-1 | grep -iE "ISPN000094" | tail -1
docker compose logs ib-keycloak-2 | grep -iE "ISPN000094" | tail -1
# Expected: BOTH logs show a view containing node1 AND node2. If each shows
# only itself, discovery failed and sessions will NOT replicate.

# TEST N2: readiness gates traffic correctly. A node that is up but not
# ready should report so on its management port.
docker exec ib-keycloak-1 curl -s http://localhost:9000/health/ready | jq -r '.status'
# Expected: UP. (Health and metrics live on the 9000 management port since
# Keycloak 25, not on 8080.)

# TEST N3: the database is the single source of truth. Stop BOTH nodes,
# keep PostgreSQL, start the nodes again. The realm returns intact.
docker compose stop ib-keycloak-1 ib-keycloak-2
docker compose start ib-keycloak-1 ib-keycloak-2
sleep 15 && curl -s http://localhost:8081/realms/northgate/ -o /dev/null -w "%{http_code}\n"
# Expected: 200 after the nodes finish starting. State outlived all compute.

# TEST N4: killing PostgreSQL is the real single point of failure now.
# Stop it and observe that the nodes cannot serve new logins. This shows
# WHY production replicates the database itself (out of scope here).
docker compose stop ib-postgres
curl -s http://localhost:8081/realms/northgate/ -o /dev/null -w "%{http_code}\n"
# Expected: an error once caches expire. Restart it: docker compose start ib-postgres

Common failure modes and solutions

Security Analysis

What makes this implementation resilient

• State is external and durable: the realm lives in PostgreSQL on a named volume, so destroying and recreating a node loses nothing, as Stage 1 and Test N3 proved.
• Compute is stateless and horizontally scalable: the two nodes are identical, so any node serves any request and losing one costs capacity, not availability.
• Sessions are replicated: the distributed cache places a replica on another node at creation, so a node kill mid session does not log the user out, demonstrated directly in Phase 4.
• Discovery is robust: JDBC_PING over the shared database works where multicast does not, avoiding the lonely cluster of one that silently defeats replication.
• Routing self heals: passive health checks and upstream retry route traffic away from a failed node and back when it recovers, with no manual step.
• Proxy trust is explicit: Keycloak trusts forwarded headers, and only the balancer sets them, so host and redirect manipulation through spoofed headers is contained.

What is intentionally simplified for the lab

• PostgreSQL is a single instance and is now the remaining single point of failure, deliberately highlighted in Test N4; production replicates the database with streaming replication or a managed HA service.
• All traffic is HTTP on localhost; production terminates TLS at the balancer and often re-encrypts to the nodes, with the balancer holding the certificate.
• Two nodes on one host share that host's fate; real HA spreads nodes across hosts, availability zones, or regions, which is the multi site topic beyond this lab.
• The load balancer itself is a single Nginx; production runs the balancer in pairs or uses a managed load balancer that is itself redundant.
• The cache uses defaults for replica count and timeouts rather than being tuned to a session volume and recovery target.

Production hardening recommendations

Cleanup Instructions

Option A: pause the lab, keep everything (recommended)

cd ~/ib-sia-06
docker compose stop
# Resume later with:
docker compose start
# OpenLDAP is managed separately:
docker start ib-openldap

Option B: tear down the cluster, keep the database volume

cd ~/ib-sia-06
# Removes containers and the network attachment but keeps the named
# volume ib-pgdata, so the realm survives for a later rebuild.
docker compose down
docker volume ls | grep ib-pgdata   # confirm the data volume remains

Option C: full teardown, including data

cd ~/ib-sia-06
docker compose down -v            # -v also deletes the ib-pgdata volume

# Roll back to the single node if you want the pre-lab state:
docker rename ib-keycloak-singlenode-old ib-keycloak
docker start ib-keycloak ib-phpldapadmin

# Remove the sensitive export and secrets.
rm -f ~/ib-sia-06/import/northgate-realm.json ~/ib-sia-06/.env

Recommended Learning Links

• Keycloak: Configuring Keycloak for production (database, hostname, proxy, TLS)
• Keycloak: Configuring distributed caches (Infinispan, cache config file)
• Keycloak: Using a reverse proxy (proxy headers and hostname behind a balancer)
• Keycloak: High availability guide (single site and multi site topologies)
• Keycloak: Health and readiness probes (the management port endpoints)
• Infinispan documentation (distributed caches and replication)
• JGroups manual: JDBC_PING2 discovery
• Nginx upstream module (health checks, retries, affinity)

Portfolio Publishing Guide: Evidence of a Verified HA Design

Lab 06 is a portfolio centrepiece because it moves you from configuring a product to architecting a service. The evidence is strong and visual: an architecture diagram, a Compose file, a cluster view log showing two members, and a failover demonstration. The most persuasive artefact is proof you tested the failure, not merely designed for it.

12.1 Collect and sanitise the artefacts

cd ~/identity-bytes-architect-labs
mkdir -p lab-06-keycloak-ha/{docs,config,evidence,screenshots}

# Copy configs, replacing real secrets with placeholders.
sed 's/ChangeMe_[A-Za-z0-9#_]*/<set-in-your-env>/g' \
  ~/ib-sia-06/docker-compose.yml > lab-06-keycloak-ha/config/docker-compose.yml
cp ~/ib-sia-06/conf/nginx.conf          lab-06-keycloak-ha/config/nginx.conf
cp ~/ib-sia-06/conf/cache-ispn-jdbc.xml lab-06-keycloak-ha/config/cache-ispn-jdbc.xml

# Provide a .env.example, never the real .env.
cat > lab-06-keycloak-ha/config/.env.example << 'EOF'
KC_DB_PASSWORD=replace-with-a-strong-secret
KC_ADMIN_PASSWORD=replace-with-a-strong-secret
EOF

# Capture the cluster-view log line as evidence (redact host UUIDs).
docker compose -f ~/ib-sia-06/docker-compose.yml logs ib-keycloak-1 \
  | grep -iE "ISPN000094|view" | tail -1 \
  | sed -E 's/[0-9a-f]{8}-[0-9a-f-]{27}/<uuid>/g' \
  > lab-06-keycloak-ha/evidence/cluster-view.txt

12.2 Add Lab 06 to the repository and push

cat > lab-06-keycloak-ha/README.md << 'EOF'
# Lab 06: Keycloak High Availability

Part of my Identity Bytes Senior IAM Architect lab series (IB-SIA-06).
Re-architects the single node IdP from earlier labs into a highly
available cluster and proves it survives node loss.

## Problem this solves
Once every application delegates login to one IdP, that IdP is the most
critical service in the estate. A single node with an embedded database
is both a single point of failure and a single point of data loss.

## What I built
- Migrated the realm from the embedded database to external PostgreSQL,
  making nodes stateless and disposable
- Ran a two node Keycloak cluster in production mode
- Configured Infinispan session replication with JGroups JDBC_PING
  discovery over the shared database (works where multicast does not)
- Fronted the cluster with an Nginx load balancer: forwarded headers,
  passive health checks, and upstream retry

## What I proved
- Destroyed and recreated a node: realm intact (state is external)
- Killed the node serving a live session: user stayed logged in, no
  re-authentication (sessions were replicated at creation)
- Restarted the node: it rejoined the cluster and traffic re-balanced
- Stopped both nodes, kept the database, restarted: realm returned intact

## Design lessons documented
Durability separated from availability; the lonely cluster of one as the
classic replication failure; affinity for speed vs replication for
survival; PostgreSQL as the remaining single point of failure to address
next.

## Skills demonstrated
Production Keycloak, external database migration, Infinispan/JGroups
clustering, cluster discovery selection, load balancing a stateful auth
service, health probes, and deliberate failover testing.

Full guide: docs/IB-SIA-06-keycloak-high-availability.html
Config: config/   Evidence: evidence/cluster-view.txt
EOF

cp <PATH_TO>/IB-SIA-06-keycloak-high-availability.html lab-06-keycloak-ha/docs/
git add .
git commit -m "Lab 06: HA Keycloak: PostgreSQL, Infinispan clustering, Nginx LB, verified failover"
git push
# Track index: | 06 | Keycloak HA: PostgreSQL, clustering, LB, failover | Complete |

12.3 Share it on LinkedIn

Attach the architecture diagram, or a two frame screenshot: the account console before the node kill and after it, still logged in. GitHub link in the first comment. A draft in the Identity Bytes style:

Same cadence: Tuesday to Thursday morning UK time, replies within two hours, hashtags at the end (#IAM #Keycloak #HighAvailability #DevOps #CyberSecurity), and a comment linking the earlier labs so the series compounds.

12.4 Interview leverage from this lab

This lab evidences the production readiness, resilience, and platform ownership lines of the target role, and it answers the senior level design question directly: how do you make an identity provider highly available. You can walk through externalising state, stateless horizontally scalable compute, session replication, discovery choice, and load balancer behaviour, and then say you proved it by killing a node mid session. When the interviewer probes the trade offs, affinity versus replication, the remaining database single point of failure, single site versus multi site, you are discussing a system you built and broke on purpose, which is the difference between reciting a pattern and owning one.