- HCL 39.6%
- Go 38.4%
- Shell 22%
| Filename | Latest commit message | Latest commit date |
|---|---|---|
Handover mechanics, interface hardening, docs and tests Ref: #360 Co-authored-by: Nikolas List <list@metromorph.de> Reviewed-on: https://gitea.mm.intern/hosting/tofu-hcloud-mmdc-bluegreen/pulls/3 Reviewed-by: Nikolas List <list@metromorph.de> Co-authored-by: H.A.L (Claude) <devel+hal@metromorph.de> Co-committed-by: H.A.L (Claude) <devel+hal@metromorph.de> |
||
| examples/basic | ||
| files | ||
| scripts | ||
| test | ||
| .drone.yml | ||
| .gitignore | ||
| .mmci.json | ||
| CHANGELOG.md | ||
| main.tf | ||
| output.tf | ||
| README.md | ||
| variables.tf | ||
tofu-hcloud-mmdc-bluegreen
OpenTofu module for blue-green shared storage management on top of existing tofu-hcloud-mmdc-hosts host deployments.
Overview
This module separates the shared volume lifecycle from the host lifecycle.
Hosts (with their own local encrypted volumes) are managed independently via
tofu-hcloud-mmdc-hosts. This module then:
- Creates shared storage volumes that outlive any individual deployment group
- Attaches them to the primary group's chosen node
- Runs LUKS encryption initialisation (
volume_init) on first creation of each volume — not on re-attachments - Leaves volumes untouched when switching groups (no re-initialisation)
- Wires the crypttab entry (
node_wiring) onto every node that could become primary — not just whichever one is primary right now — so a node is ready to receive the volume the first time it becomes primary, not just after
┌───────────────────────────┐ ┌───────────────────────────┐
│ module "blue_hosts" │ │ module "green_hosts" │
│ tofu-hcloud-mmdc-hosts │ │ tofu-hcloud-mmdc-hosts │
│ nodes + local volumes │ │ nodes + local volumes │
│ + crypto-init │ │ + crypto-init │
└─────────────┬─────────────┘ └─────────────┬─────────────┘
│ .nodes output │ .nodes output
▼ ▼
┌──────────────────────────────────────────┐
│ tofu-hcloud-mmdc-bluegreen │
│ │
│ shared volumes (LUKS, survive switches) │
│ attached to primary group's node │
│ volume_init once per volume, node_wiring│
│ on every candidate node │
└──────────────────────────────────────────┘
Why this is a "handover", not classic blue-green
A Hetzner Cloud volume can be attached to exactly one server at a time, and ext4 is
not a cluster filesystem — a "shared" data volume cannot be shared concurrently between
a blue and a green node group. So what this module gives you is:
- real blue-green for the stateless compute layer (both groups can run at once, traffic switches freely between them), plus
- a handover for the data layer: a hard cutover with a short downtime window, where the volume itself moves rather than being copied.
The accepted operational envelope is ~20 minutes of downtime per switch. Handover
(move the attachment) instead of copying the data (rsync/snapshot-restore) is deliberate:
wall clock is independent of data size (detach+attach is an API call; cryptsetup luksOpen is instant), there's no "sync the delta while writes are still in flight"
consistency window, no 2x storage cost, and it's application-agnostic — the guarantee
comes from stopping the container cleanly, not from understanding the data format.
Application-level replication would only be worth its complexity if sub-minute cutover
were required; with a 20-minute window it's over-engineering.
On-node layout
Each shared volume is LUKS-encrypted directly on its stable
/dev/disk/by-id/scsi-0HC_Volume_<id> device — no LVM. The volume's identity is its
LUKS UUID, which travels with the disk and stays correct on any host in any attachment
order. The LUKS mapper name matches the volume's logical name, and the volume is
consumed directly as a Docker-managed volume against that mapper device — there is no
host-level mount at all:
/dev/disk/by-id/scsi-0HC_Volume_<id> → cryptsetup luksOpen → /dev/mapper/<name>
→ docker volume create --driver=local \
--opt type=ext4 --opt device=/dev/mapper/<name> <name>
See "Using shared volumes in Docker" below for the full command and how containers reference it.
LVM was dropped on purpose: a volume that migrates between hosts must have its volume
group reactivated on the receiving node, and nothing guarantees LVM autoactivation fires
reliably after a hot attach; a VG name that already exists on the target node can't be
activated without vgrename, which is not something to do mid-cutover; and a single LV
at 100%FREE over the volume buys nothing a raw device doesn't already have.
Rollback
This is a handover, not a copy — switching primary gives no free rollback for the data layer. If the new primary corrupts or migrates the data, flipping back to the old primary hands it the same, now-modified volume: its binaries are intact, its data is not. Treat the rollback point as an explicit artifact, not "the other colour":
- App-version rollback (flip the volume back) — fast, safe only if the on-disk data format is unchanged.
- Data-format rollback — restore from a copy taken during the switch window, while
the filesystem was quiescent and before the new primary touched it. Use a
filesystem-level copy (
rsync,restic,borg) — neverdd: a block-level clone duplicates the LUKS header, producing two devices with the same UUID, which then makes/etc/crypttabambiguous on whichever node sees both.
For large volumes, pre-seed the copy ahead of the switch window and take only the delta during it.
Node-side wiring
Two modules write node-side storage config on the same host — this module and
tofu-hcloud-mmdc-hosts — so /etc/crypttab has to be treated as a shared file that
no single module owns outright. This module never writes it wholesale:
- Each shared volume gets its own fragment,
/etc/crypttab.d/<name>, containing just that volume's crypttab line. crypttab has no native.dsupport, so composing/etc/crypttabfrom every fragment in/etc/crypttab.d/is a job for real code — but this module doesn't ship that code itself.tofu-hcloud-mmdc-hosts(v0.6.0+, see "Compatibility") installs and owns the assembler on every node, since it already owns node bootstrapping and used to be the thing truncating/etc/crypttabwholesale withcat >. This module just drops its fragment and trusts that assembler is already there — it does not install its own copy. - That assembler runs as a systemd service triggered by a path unit watching both
/etc/crypttab.dand/etc/crypttabitself, so any wholesale rewrite of the file gets reassembled — including this module's fragments — within moments, instead of silently staying lost until the nexttofu apply.node_wiringalso invokes it directly right after writing a fragment, so a change takes effect immediately rather than waiting on the watcher. - The fragment is
noauto: nothing ever auto-unlocks a shared volume. A shared volume must never auto-unlock on the wrong colour.
This wiring runs on every candidate node — both colours' var.node_index node,
whichever groups actually exist — not just whichever one is primary right now, and
re-runs idempotently whenever the shared volume set changes (see "Provisioners" below).
That is what lets a colour receive shared storage the first time it ever becomes
primary, with no manual step in between.
There is deliberately no mount unit, /etc/fstab entry, or mountpoint directory on the
host — see "Using shared volumes in Docker" below for how a shared volume is actually
consumed once unlocked.
Using shared volumes in Docker
Once a shared volume is unlocked (cryptsetup luksOpen), it is consumed directly as a
Docker-managed volume against the LUKS mapper device — there is no intermediate host
mount or mountpoint directory:
docker volume create --driver=local --opt type=ext4 --opt device=/dev/mapper/<name> <name>
This is idempotent: re-running it with the same name and options is a no-op if the volume already exists. Reference it from a container the same way as any named Docker volume:
docker run -v <name>:/path/in/container ...
or in a Compose file:
volumes:
<name>:
external: true
Docker mounts the device when the volume is created and unmounts it on docker volume rm — there is no systemd unit or /etc/fstab entry involved, and nothing here starts
or stops the Docker daemon itself. Before closing the LUKS device (e.g. during a
primary switch, see "Switching the primary group" below), remove the Docker volume
first: docker volume rm <name> fails loudly if any container still references it,
which is the safety check that replaces the old "verify with lsblk" mount-based one —
still verify with lsblk too that no mapper entry remains before proceeding.
Known limitations
This module is being built up in phases; the following are real gaps in the current implementation, not just planned polish — read before relying on it for anything that matters:
- The pre-flip check can only see what's reachable over SSH.
data.external.shared_luks_status(see "Switching the primary group" below) fails safe — it treats an unreachable outgoing-primary node the same as an open volume, and blocks the plan either way — but that means a genuinely dead node can permanently block a switch until itshcloud_volume'sserver_idis cleared by other means (e.g. the node is deleted). There is no override flag; this is deliberate, not an oversight.
Prerequisites
- Whichever group is primary (
deployment's first element) must already be deployed withtofu-hcloud-mmdc-hostsbefore applying this module. The other group doesn't have to exist yet — a root config can create it only while migrating (seeexamples/basic/README.md, "deploymentvariable semantics", for the expand/validate/cutover/contract pattern this is meant to support) — but this module itself always requires bothblue_groupandgreen_groupas inputs; passnullfor one that isn't deployed. - The OpenTofu runner must be able to SSH to the primary node (IPv6 by default, or
IPv4 if
ipv4_enabled = true) for thevolume_initprovisioner, and to every candidate node (both colours'node_indexnode) for thenode_wiringprovisioner. - The machine running
tofu plan(not justapply) must also be able to SSH to whichever node a shared volume is currently attached to, and havejqon itsPATH— both are needed bydata.external.shared_luks_status(scripts/check-luks-open.sh), the plan-time check described in "Switching the primary group" below. It runs on every plan (itsfor_eachhas to stay on a set that's always known — seemain.tf, "Pre-flip safety check" — so it can't be conditioned on whether a switch is actually happening), but only blocks the plan when a switch is being requested and the volume is still open on the outgoing primary. scripts/release-shared.shandscripts/acquire-shared.sh(see "Switching the primary group" below) needsshon the operator's machine;acquire-shared.shalso needstofu,yqandsopsto resolve a volume's device path and passphrase.- A SOPS-encrypted file at
var.luks_sops_fileholding one non-empty passphrase entry per shared volume, keyed by volume name — see "Secrets" below. The plan fails with a precondition error naming any volume that's missing one.
Compatibility
Targets tofu-hcloud-mmdc-hosts
v0.6.0 or later. That module moved its LUKS passphrase model to SOPS-encrypted,
per-node secrets (luks_sops_file) at v0.4.0 and has no passphrase output — do
not pass blue_group.passphrase / green_group.passphrase to anything, they don't
exist. blue_group / green_group here only ever need nodes[*].{id, ipv4, ipv6}
from that module's output, which is unaffected by its secret model.
v0.6.0 is a hard requirement, not just a recommendation: that's the release where
tofu-hcloud-mmdc-hosts stopped truncating /etc/crypttab wholesale and started
installing/owning the shared crypttab.d assembler this module's node_wiring relies
on (see "Node-side wiring" above). Against an older hosts version, node_wiring would
write a fragment that nothing ever composes into /etc/crypttab.
Secrets
This module follows the same model as tofu-hcloud-mmdc-hosts: it reads a
passphrase, it never generates one.
- One SOPS-encrypted file,
var.luks_sops_file, holding one entry per shared volume, keyed by volume name — not node name, since a volume that moves between colours has no single node to be keyed by, and duplicating a node-keyed passphrase across colours would quietly break the hosts module's per-node isolation. - Read via the
carlpett/sopsprovider (data.sops_file.luks_passphrases), with alifecycle.postconditionthat fails the plan if any shared volume lacks a non-empty entry — same shape as the hosts module's ownsetup.tf. - There is no
passphraseoutput. Reveal a value withsops -d <file>instead. - The passphrase is base64-encoded on the way into
remote-execand decoded on the node, so a value containing shell metacharacters can't break out of the command line — the same approach the hosts module uses in itssetup.tf. - Consuming root configs must enable OpenTofu state/plan encryption (`required_version
= 1.7
) — seeexamples/basic/main.tf'sencryption` block.
Creating var.luks_sops_file
The file is a flat YAML map of <volume name>: <passphrase>, one entry per volume in
var.shared_volumes, encrypted with sops. This
module never generates or writes it — create it yourself before the first
tofu plan.
-
Add (or reuse) a
.sops.yamlnext to the file, naming the PGP recipient(s) who should be able to decrypt it:creation_rules: - path_regex: secrets/luks-shared\.enc\.yaml$ pgp: '<fingerprint1>,<fingerprint2>'(
gpg --list-secret-keysshows fingerprints you already hold; seeexamples/basic/.sops.yamlfor a working example. At least two recipients is recommended so a single lost/revoked key can't lock everyone out.) -
Generate one entry per shared volume. Easiest is the hosts module's own
scripts/gen-luks-secrets.sh(from a checkout oftofu-hcloud-mmdc-hosts), which is name-agnostic — pass shared volume names via--nodesinstead of node names:./scripts/gen-luks-secrets.sh --nodes "shared-data-01" --out secrets/luks-shared.enc.yamlWithout that script,
sopsalone is enough — pipe a plaintext map straight into it so the plaintext never touches disk:printf 'shared-data-01: %s\n' "$(openssl rand -base64 32)" \ | sops --input-type yaml --output-type yaml \ --filename-override secrets/luks-shared.enc.yaml -e /dev/stdin \ > secrets/luks-shared.enc.yamlAdd one
printfline per additional shared volume before encrypting. -
Verify it decrypts and has the key(s) the module expects:
sops -d secrets/luks-shared.enc.yamlEvery name in
var.shared_volumesmust appear with a non-empty value, or the plan fails and names exactly which volume is missing one.
To rotate a volume's passphrase later, sops secrets/luks-shared.enc.yaml decrypts it
to a temp file, opens $EDITOR, and re-encrypts on save.
Usage
module "blue_hosts" {
source = "path/to/tofu-hcloud-mmdc-hosts"
# ... your node and network config
}
module "green_hosts" {
source = "path/to/tofu-hcloud-mmdc-hosts"
# ... your node and network config
}
module "bluegreen" {
source = "path/to/tofu-hcloud-mmdc-bluegreen"
blue_group = module.blue_hosts
green_group = module.green_hosts
deployment = "blue" # blue receives shared storage
shared_volumes = [
{ name = "shared-data-01", size = 100 },
]
location = "nbg1"
}
See examples/basic for a complete, runnable root config.
deployment variable semantics
The first element of the comma-separated string is the primary group that receives shared storage.
| Value | Shared storage on |
|---|---|
"blue" |
blue |
"green" |
green |
"blue,green" |
blue |
"green,blue" |
green |
Only these four exact strings are accepted — var.deployment has a validation block
that rejects anything else at plan time (e.g. "purple" used to silently resolve to
green via the underlying ternary; it now fails the plan instead).
var.node_index and var.location are similarly guarded, but by preconditions on
hcloud_volume_attachment.shared rather than variable-level validation, since
checking them needs var.blue_group/var.green_group's contents (node_index) or a
live lookup against the Hetzner API (location, compared against the primary node's
actual datacenter) — neither of which a variable block alone can see. An
out-of-range node_index or a location that doesn't match the host groups' own now
fails the plan with a clear message instead of an opaque index error or a late Hetzner
API error.
Switching the primary group
Hetzner Cloud volumes can only be attached to one server at a time, so a flip is an
explicit, two-phase operator procedure, not something OpenTofu can orchestrate on its
own (destroy-time provisioners can't reference variables and won't run once a resource
has left the graph — see main.tf, "Pre-flip safety check"). Follow this procedure by
hand, in order, every time. Plan for ~20 minutes of downtime.
- Release each shared volume on the current primary:
Stops any container still referencing the volume, removes the Docker volume,./scripts/release-shared.sh <name> <current-primary>cryptsetup luksCloses it, and verifies nothing is left open — exiting non-zero, without ever forcing anything, if it's still busy.<current-primary>is the hostname/IP the volume is presently attached to —tofu output deploymentortofu output -raw READMEname it. - (Recommended) take a rollback copy now, while the volume is closed and the filesystem is quiescent — see "Rollback" above.
- Re-attach to the new primary:
tofu apply -var='deployment=<new-primary>'hcloud_volume_attachment.shared's plan-time precondition re-checks step 1 itself (live against the Hetzner API, not just trusting that you ran the script) and fails the plan with a clear message if the volume still looks open on the outgoing primary — skipping step 1 cannot silently corrupt the filesystem. Thevolume_initprovisioner does not re-run — the LUKS structure already exists on the volume — and, unlike before, the new primary does not need wiring by hand:node_wiringalready put its crypttab entry there ahead of time, on every candidate node, the first time this module was applied with that volume invar.shared_volumes(see "Node-side wiring" above). - Acquire the shared volume on the new primary:
Unlocks it (reading the passphrase from./scripts/acquire-shared.sh <name> <new-primary>var.luks_sops_fileviasops, and the device path fromtofu output shared_volumesviatofu/yq) and (re)creates the Docker volume against it. Idempotent — safe to re-run. - Start consumers, then switch traffic to the new primary (floating IP / DNS — see "Switching traffic" below) and verify.
- Leave the old primary running — it's your app-version rollback path.
Retrieve a shared volume's passphrase at any time with sops -d <var.luks_sops_file>.
Switching traffic
This module has no opinion on traffic — that's the root config's job — but the hosts
module it sits on top of already has the pieces needed for a floating IP:
var.ip_addr on tofu-hcloud-mmdc-hosts renders an
/etc/network/interfaces.d/90-floatingip block via that module's cloud-init.template
for whichever address you give it.
- Set the same Hetzner Floating IP address as
ip_addron both the blue and greentofu-hcloud-mmdc-hostsmodule calls in your root config, so either colour is ready to answer on it the moment traffic moves. - Move the Floating IP's assignment to the new primary node via the Hetzner API
(
POST /floating_ips/{id}/actions/assign) or thehcloudCLI (hcloud floating-ip assign <floating-ip> <server>) once step 4 above has finished and consumers are up. - Verify the interface actually came up with the address on the new primary
(
ip addr show) before declaring the switch complete — this module and the hosts module do not verify for you that/etc/network/interfaces.d/90-floatingipis still honoured by cloud-init on whatever node image is in use; confirm it on a real node once, for your image, before relying on it during a production cutover.
Inputs
| Variable | Type | Default | Description |
|---|---|---|---|
blue_group |
any |
required | Full output of the blue tofu-hcloud-mmdc-hosts module |
green_group |
any |
required | Full output of the green tofu-hcloud-mmdc-hosts module |
deployment |
string |
"blue" |
Primary group receiving shared storage (first element of comma-separated list) |
shared_volumes |
list(object({name,size})) |
[] |
Volumes to create for shared storage |
luks_sops_file |
string |
required | Path to a SOPS-encrypted file with one LUKS passphrase per shared volume, keyed by volume name |
location |
string |
"nbg1" |
Hetzner datacenter (must match the host groups) |
protect_storage |
bool |
true |
Enable delete protection on shared volumes |
ipv4_enabled |
bool |
false |
Use IPv4 for SSH (must match the host group deployment setting) |
node_index |
number |
0 |
Node index within each group to receive shared volumes / node-side wiring |
Outputs
| Output | Description |
|---|---|
shared_volumes |
Map name → {id, name, device, size} of all shared volumes |
deployment |
Resolved state: deployment_var, primary_group, primary_node |
README |
Runtime summary printed after apply, incl. per-volume unlock/mount commands |