Watch
1
0
Fork
You've already forked hcloud-mmdc-bluegreen
0
No description
  • HCL 39.6%
  • Go 38.4%
  • Shell 22%
Find a file
Repository files (latest commit first)
Filename Latest commit message Latest commit date
H.A.L (Claude) 6f04debf3a
Fix: Operator Scripts for Handover and Tests (!3)
Handover mechanics, interface hardening, docs and tests

Ref: #360
Co-authored-by: Nikolas List <list@metromorph.de>
Reviewed-on: https://gitea.mm.intern/hosting/tofu-hcloud-mmdc-bluegreen/pulls/3
Reviewed-by: Nikolas List <list@metromorph.de>
Co-authored-by: H.A.L (Claude) <devel+hal@metromorph.de>
Co-committed-by: H.A.L (Claude) <devel+hal@metromorph.de>
2026-09-11 14:41:47 +00:00
examples/basic Fix: Operator Scripts for Handover and Tests (!3) 2026-09-11 14:41:47 +00:00
files Fix: Node file ownership, provisioner split, secret handling (!2) 2026-09-10 09:30:45 +00:00
scripts Fix: Operator Scripts for Handover and Tests (!3) 2026-09-11 14:41:47 +00:00
test Fix: Operator Scripts for Handover and Tests (!3) 2026-09-11 14:41:47 +00:00
.drone.yml Feature: Create working tofu-hcloud-mmdc-bluegreen (!1) 2026-09-08 10:30:13 +00:00
.gitignore Feature: Create working tofu-hcloud-mmdc-bluegreen (!1) 2026-09-08 10:30:13 +00:00
.mmci.json Fix: Operator Scripts for Handover and Tests (!3) 2026-09-11 14:41:47 +00:00
CHANGELOG.md Fix: Operator Scripts for Handover and Tests (!3) 2026-09-11 14:41:47 +00:00
main.tf Fix: Operator Scripts for Handover and Tests (!3) 2026-09-11 14:41:47 +00:00
output.tf Fix: Operator Scripts for Handover and Tests (!3) 2026-09-11 14:41:47 +00:00
README.md Fix: Operator Scripts for Handover and Tests (!3) 2026-09-11 14:41:47 +00:00
variables.tf Fix: Operator Scripts for Handover and Tests (!3) 2026-09-11 14:41:47 +00:00

tofu-hcloud-mmdc-bluegreen

OpenTofu module for blue-green shared storage management on top of existing tofu-hcloud-mmdc-hosts host deployments.

Overview

This module separates the shared volume lifecycle from the host lifecycle. Hosts (with their own local encrypted volumes) are managed independently via tofu-hcloud-mmdc-hosts. This module then:

  1. Creates shared storage volumes that outlive any individual deployment group
  2. Attaches them to the primary group's chosen node
  3. Runs LUKS encryption initialisation (volume_init) on first creation of each volume — not on re-attachments
  4. Leaves volumes untouched when switching groups (no re-initialisation)
  5. Wires the crypttab entry (node_wiring) onto every node that could become primary — not just whichever one is primary right now — so a node is ready to receive the volume the first time it becomes primary, not just after
┌───────────────────────────┐   ┌───────────────────────────┐
│  module "blue_hosts"      │   │  module "green_hosts"     │
│  tofu-hcloud-mmdc-hosts   │   │  tofu-hcloud-mmdc-hosts   │
│  nodes + local volumes    │   │  nodes + local volumes    │
│  + crypto-init            │   │  + crypto-init            │
└─────────────┬─────────────┘   └─────────────┬─────────────┘
              │ .nodes output                  │ .nodes output
              ▼                                ▼
       ┌──────────────────────────────────────────┐
       │        tofu-hcloud-mmdc-bluegreen         │
       │                                          │
       │  shared volumes (LUKS, survive switches) │
       │  attached to primary group's node        │
       │  volume_init once per volume, node_wiring│
       │  on every candidate node                 │
       └──────────────────────────────────────────┘

Why this is a "handover", not classic blue-green

A Hetzner Cloud volume can be attached to exactly one server at a time, and ext4 is not a cluster filesystem — a "shared" data volume cannot be shared concurrently between a blue and a green node group. So what this module gives you is:

  • real blue-green for the stateless compute layer (both groups can run at once, traffic switches freely between them), plus
  • a handover for the data layer: a hard cutover with a short downtime window, where the volume itself moves rather than being copied.

The accepted operational envelope is ~20 minutes of downtime per switch. Handover (move the attachment) instead of copying the data (rsync/snapshot-restore) is deliberate: wall clock is independent of data size (detach+attach is an API call; cryptsetup luksOpen is instant), there's no "sync the delta while writes are still in flight" consistency window, no 2x storage cost, and it's application-agnostic — the guarantee comes from stopping the container cleanly, not from understanding the data format. Application-level replication would only be worth its complexity if sub-minute cutover were required; with a 20-minute window it's over-engineering.

On-node layout

Each shared volume is LUKS-encrypted directly on its stable /dev/disk/by-id/scsi-0HC_Volume_<id> device — no LVM. The volume's identity is its LUKS UUID, which travels with the disk and stays correct on any host in any attachment order. The LUKS mapper name matches the volume's logical name, and the volume is consumed directly as a Docker-managed volume against that mapper device — there is no host-level mount at all:

/dev/disk/by-id/scsi-0HC_Volume_<id>  →  cryptsetup luksOpen  →  /dev/mapper/<name>
                                       →  docker volume create --driver=local \
                                            --opt type=ext4 --opt device=/dev/mapper/<name> <name>

See "Using shared volumes in Docker" below for the full command and how containers reference it.

LVM was dropped on purpose: a volume that migrates between hosts must have its volume group reactivated on the receiving node, and nothing guarantees LVM autoactivation fires reliably after a hot attach; a VG name that already exists on the target node can't be activated without vgrename, which is not something to do mid-cutover; and a single LV at 100%FREE over the volume buys nothing a raw device doesn't already have.

Rollback

This is a handover, not a copy — switching primary gives no free rollback for the data layer. If the new primary corrupts or migrates the data, flipping back to the old primary hands it the same, now-modified volume: its binaries are intact, its data is not. Treat the rollback point as an explicit artifact, not "the other colour":

  • App-version rollback (flip the volume back) — fast, safe only if the on-disk data format is unchanged.
  • Data-format rollback — restore from a copy taken during the switch window, while the filesystem was quiescent and before the new primary touched it. Use a filesystem-level copy (rsync, restic, borg) — never dd: a block-level clone duplicates the LUKS header, producing two devices with the same UUID, which then makes /etc/crypttab ambiguous on whichever node sees both.

For large volumes, pre-seed the copy ahead of the switch window and take only the delta during it.

Node-side wiring

Two modules write node-side storage config on the same host — this module and tofu-hcloud-mmdc-hosts — so /etc/crypttab has to be treated as a shared file that no single module owns outright. This module never writes it wholesale:

  • Each shared volume gets its own fragment, /etc/crypttab.d/<name>, containing just that volume's crypttab line. crypttab has no native .d support, so composing /etc/crypttab from every fragment in /etc/crypttab.d/ is a job for real code — but this module doesn't ship that code itself. tofu-hcloud-mmdc-hosts (v0.6.0+, see "Compatibility") installs and owns the assembler on every node, since it already owns node bootstrapping and used to be the thing truncating /etc/crypttab wholesale with cat >. This module just drops its fragment and trusts that assembler is already there — it does not install its own copy.
  • That assembler runs as a systemd service triggered by a path unit watching both /etc/crypttab.d and /etc/crypttab itself, so any wholesale rewrite of the file gets reassembled — including this module's fragments — within moments, instead of silently staying lost until the next tofu apply. node_wiring also invokes it directly right after writing a fragment, so a change takes effect immediately rather than waiting on the watcher.
  • The fragment is noauto: nothing ever auto-unlocks a shared volume. A shared volume must never auto-unlock on the wrong colour.

This wiring runs on every candidate node — both colours' var.node_index node, whichever groups actually exist — not just whichever one is primary right now, and re-runs idempotently whenever the shared volume set changes (see "Provisioners" below). That is what lets a colour receive shared storage the first time it ever becomes primary, with no manual step in between.

There is deliberately no mount unit, /etc/fstab entry, or mountpoint directory on the host — see "Using shared volumes in Docker" below for how a shared volume is actually consumed once unlocked.

Using shared volumes in Docker

Once a shared volume is unlocked (cryptsetup luksOpen), it is consumed directly as a Docker-managed volume against the LUKS mapper device — there is no intermediate host mount or mountpoint directory:

docker volume create --driver=local --opt type=ext4 --opt device=/dev/mapper/<name> <name>

This is idempotent: re-running it with the same name and options is a no-op if the volume already exists. Reference it from a container the same way as any named Docker volume:

docker run -v <name>:/path/in/container ...

or in a Compose file:

volumes:
  <name>:
    external: true

Docker mounts the device when the volume is created and unmounts it on docker volume rm — there is no systemd unit or /etc/fstab entry involved, and nothing here starts or stops the Docker daemon itself. Before closing the LUKS device (e.g. during a primary switch, see "Switching the primary group" below), remove the Docker volume first: docker volume rm <name> fails loudly if any container still references it, which is the safety check that replaces the old "verify with lsblk" mount-based one — still verify with lsblk too that no mapper entry remains before proceeding.

Known limitations

This module is being built up in phases; the following are real gaps in the current implementation, not just planned polish — read before relying on it for anything that matters:

  • The pre-flip check can only see what's reachable over SSH. data.external.shared_luks_status (see "Switching the primary group" below) fails safe — it treats an unreachable outgoing-primary node the same as an open volume, and blocks the plan either way — but that means a genuinely dead node can permanently block a switch until its hcloud_volume's server_id is cleared by other means (e.g. the node is deleted). There is no override flag; this is deliberate, not an oversight.

Prerequisites

  • Whichever group is primary (deployment's first element) must already be deployed with tofu-hcloud-mmdc-hosts before applying this module. The other group doesn't have to exist yet — a root config can create it only while migrating (see examples/basic/README.md, "deployment variable semantics", for the expand/validate/cutover/contract pattern this is meant to support) — but this module itself always requires both blue_group and green_group as inputs; pass null for one that isn't deployed.
  • The OpenTofu runner must be able to SSH to the primary node (IPv6 by default, or IPv4 if ipv4_enabled = true) for the volume_init provisioner, and to every candidate node (both colours' node_index node) for the node_wiring provisioner.
  • The machine running tofu plan (not just apply) must also be able to SSH to whichever node a shared volume is currently attached to, and have jq on its PATH — both are needed by data.external.shared_luks_status (scripts/check-luks-open.sh), the plan-time check described in "Switching the primary group" below. It runs on every plan (its for_each has to stay on a set that's always known — see main.tf, "Pre-flip safety check" — so it can't be conditioned on whether a switch is actually happening), but only blocks the plan when a switch is being requested and the volume is still open on the outgoing primary.
  • scripts/release-shared.sh and scripts/acquire-shared.sh (see "Switching the primary group" below) need ssh on the operator's machine; acquire-shared.sh also needs tofu, yq and sops to resolve a volume's device path and passphrase.
  • A SOPS-encrypted file at var.luks_sops_file holding one non-empty passphrase entry per shared volume, keyed by volume name — see "Secrets" below. The plan fails with a precondition error naming any volume that's missing one.

Compatibility

Targets tofu-hcloud-mmdc-hosts v0.6.0 or later. That module moved its LUKS passphrase model to SOPS-encrypted, per-node secrets (luks_sops_file) at v0.4.0 and has no passphrase output — do not pass blue_group.passphrase / green_group.passphrase to anything, they don't exist. blue_group / green_group here only ever need nodes[*].{id, ipv4, ipv6} from that module's output, which is unaffected by its secret model.

v0.6.0 is a hard requirement, not just a recommendation: that's the release where tofu-hcloud-mmdc-hosts stopped truncating /etc/crypttab wholesale and started installing/owning the shared crypttab.d assembler this module's node_wiring relies on (see "Node-side wiring" above). Against an older hosts version, node_wiring would write a fragment that nothing ever composes into /etc/crypttab.

Secrets

This module follows the same model as tofu-hcloud-mmdc-hosts: it reads a passphrase, it never generates one.

  • One SOPS-encrypted file, var.luks_sops_file, holding one entry per shared volume, keyed by volume name — not node name, since a volume that moves between colours has no single node to be keyed by, and duplicating a node-keyed passphrase across colours would quietly break the hosts module's per-node isolation.
  • Read via the carlpett/sops provider (data.sops_file.luks_passphrases), with a lifecycle.postcondition that fails the plan if any shared volume lacks a non-empty entry — same shape as the hosts module's own setup.tf.
  • There is no passphrase output. Reveal a value with sops -d <file> instead.
  • The passphrase is base64-encoded on the way into remote-exec and decoded on the node, so a value containing shell metacharacters can't break out of the command line — the same approach the hosts module uses in its setup.tf.
  • Consuming root configs must enable OpenTofu state/plan encryption (`required_version

    = 1.7) — see examples/basic/main.tf's encryption` block.

Creating var.luks_sops_file

The file is a flat YAML map of <volume name>: <passphrase>, one entry per volume in var.shared_volumes, encrypted with sops. This module never generates or writes it — create it yourself before the first tofu plan.

  1. Add (or reuse) a .sops.yaml next to the file, naming the PGP recipient(s) who should be able to decrypt it:

    creation_rules:
      - path_regex: secrets/luks-shared\.enc\.yaml$
        pgp: '<fingerprint1>,<fingerprint2>'
    

    (gpg --list-secret-keys shows fingerprints you already hold; see examples/basic/.sops.yaml for a working example. At least two recipients is recommended so a single lost/revoked key can't lock everyone out.)

  2. Generate one entry per shared volume. Easiest is the hosts module's own scripts/gen-luks-secrets.sh (from a checkout of tofu-hcloud-mmdc-hosts), which is name-agnostic — pass shared volume names via --nodes instead of node names:

    ./scripts/gen-luks-secrets.sh --nodes "shared-data-01" --out secrets/luks-shared.enc.yaml
    

    Without that script, sops alone is enough — pipe a plaintext map straight into it so the plaintext never touches disk:

    printf 'shared-data-01: %s\n' "$(openssl rand -base64 32)" \
      | sops --input-type yaml --output-type yaml \
          --filename-override secrets/luks-shared.enc.yaml -e /dev/stdin \
      > secrets/luks-shared.enc.yaml
    

    Add one printf line per additional shared volume before encrypting.

  3. Verify it decrypts and has the key(s) the module expects:

    sops -d secrets/luks-shared.enc.yaml
    

    Every name in var.shared_volumes must appear with a non-empty value, or the plan fails and names exactly which volume is missing one.

To rotate a volume's passphrase later, sops secrets/luks-shared.enc.yaml decrypts it to a temp file, opens $EDITOR, and re-encrypts on save.

Usage

module "blue_hosts" {
  source = "path/to/tofu-hcloud-mmdc-hosts"
  # ... your node and network config
}

module "green_hosts" {
  source = "path/to/tofu-hcloud-mmdc-hosts"
  # ... your node and network config
}

module "bluegreen" {
  source = "path/to/tofu-hcloud-mmdc-bluegreen"

  blue_group  = module.blue_hosts
  green_group = module.green_hosts

  deployment = "blue"   # blue receives shared storage

  shared_volumes = [
    { name = "shared-data-01", size = 100 },
  ]

  location = "nbg1"
}

See examples/basic for a complete, runnable root config.

deployment variable semantics

The first element of the comma-separated string is the primary group that receives shared storage.

Value Shared storage on
"blue" blue
"green" green
"blue,green" blue
"green,blue" green

Only these four exact strings are accepted — var.deployment has a validation block that rejects anything else at plan time (e.g. "purple" used to silently resolve to green via the underlying ternary; it now fails the plan instead).

var.node_index and var.location are similarly guarded, but by preconditions on hcloud_volume_attachment.shared rather than variable-level validation, since checking them needs var.blue_group/var.green_group's contents (node_index) or a live lookup against the Hetzner API (location, compared against the primary node's actual datacenter) — neither of which a variable block alone can see. An out-of-range node_index or a location that doesn't match the host groups' own now fails the plan with a clear message instead of an opaque index error or a late Hetzner API error.

Switching the primary group

Hetzner Cloud volumes can only be attached to one server at a time, so a flip is an explicit, two-phase operator procedure, not something OpenTofu can orchestrate on its own (destroy-time provisioners can't reference variables and won't run once a resource has left the graph — see main.tf, "Pre-flip safety check"). Follow this procedure by hand, in order, every time. Plan for ~20 minutes of downtime.

  1. Release each shared volume on the current primary:
    ./scripts/release-shared.sh <name> <current-primary>
    
    Stops any container still referencing the volume, removes the Docker volume, cryptsetup luksCloses it, and verifies nothing is left open — exiting non-zero, without ever forcing anything, if it's still busy. <current-primary> is the hostname/IP the volume is presently attached to — tofu output deployment or tofu output -raw README name it.
  2. (Recommended) take a rollback copy now, while the volume is closed and the filesystem is quiescent — see "Rollback" above.
  3. Re-attach to the new primary:
    tofu apply -var='deployment=<new-primary>'
    
    hcloud_volume_attachment.shared's plan-time precondition re-checks step 1 itself (live against the Hetzner API, not just trusting that you ran the script) and fails the plan with a clear message if the volume still looks open on the outgoing primary — skipping step 1 cannot silently corrupt the filesystem. The volume_init provisioner does not re-run — the LUKS structure already exists on the volume — and, unlike before, the new primary does not need wiring by hand: node_wiring already put its crypttab entry there ahead of time, on every candidate node, the first time this module was applied with that volume in var.shared_volumes (see "Node-side wiring" above).
  4. Acquire the shared volume on the new primary:
    ./scripts/acquire-shared.sh <name> <new-primary>
    
    Unlocks it (reading the passphrase from var.luks_sops_file via sops, and the device path from tofu output shared_volumes via tofu/yq) and (re)creates the Docker volume against it. Idempotent — safe to re-run.
  5. Start consumers, then switch traffic to the new primary (floating IP / DNS — see "Switching traffic" below) and verify.
  6. Leave the old primary running — it's your app-version rollback path.

Retrieve a shared volume's passphrase at any time with sops -d <var.luks_sops_file>.

Switching traffic

This module has no opinion on traffic — that's the root config's job — but the hosts module it sits on top of already has the pieces needed for a floating IP: var.ip_addr on tofu-hcloud-mmdc-hosts renders an /etc/network/interfaces.d/90-floatingip block via that module's cloud-init.template for whichever address you give it.

  1. Set the same Hetzner Floating IP address as ip_addr on both the blue and green tofu-hcloud-mmdc-hosts module calls in your root config, so either colour is ready to answer on it the moment traffic moves.
  2. Move the Floating IP's assignment to the new primary node via the Hetzner API (POST /floating_ips/{id}/actions/assign) or the hcloud CLI (hcloud floating-ip assign <floating-ip> <server>) once step 4 above has finished and consumers are up.
  3. Verify the interface actually came up with the address on the new primary (ip addr show) before declaring the switch complete — this module and the hosts module do not verify for you that /etc/network/interfaces.d/90-floatingip is still honoured by cloud-init on whatever node image is in use; confirm it on a real node once, for your image, before relying on it during a production cutover.

Inputs

Variable Type Default Description
blue_group any required Full output of the blue tofu-hcloud-mmdc-hosts module
green_group any required Full output of the green tofu-hcloud-mmdc-hosts module
deployment string "blue" Primary group receiving shared storage (first element of comma-separated list)
shared_volumes list(object({name,size})) [] Volumes to create for shared storage
luks_sops_file string required Path to a SOPS-encrypted file with one LUKS passphrase per shared volume, keyed by volume name
location string "nbg1" Hetzner datacenter (must match the host groups)
protect_storage bool true Enable delete protection on shared volumes
ipv4_enabled bool false Use IPv4 for SSH (must match the host group deployment setting)
node_index number 0 Node index within each group to receive shared volumes / node-side wiring

Outputs

Output Description
shared_volumes Map name → {id, name, device, size} of all shared volumes
deployment Resolved state: deployment_var, primary_group, primary_node
README Runtime summary printed after apply, incl. per-volume unlock/mount commands