Watch
1
0
Fork
You've already forked hcloud-mmdc-hosts
0
No description
  • Go 44.2%
  • Shell 35.1%
  • HCL 20.7%
Find a file
Repository files (latest commit first)
Filename Latest commit message Latest commit date
H.A.L (Claude) 22dd14dc2e
Feature: Self-Healing Crypttab-Assembler (!8)
own /etc/crypttab via crypttab.d fragments + self-healing assembler

Ref: #361
Co-authored-by: Nikolas List <list@metromorph.de>
Reviewed-on: https://gitea.mm.intern/hosting/tofu-hcloud-mmdc-hosts/pulls/8
Reviewed-by: Nikolas List <list@metromorph.de>
Co-authored-by: H.A.L (Claude) <devel+hal@metromorph.de>
Co-committed-by: H.A.L (Claude) <devel+hal@metromorph.de>
2026-09-10 05:56:13 +00:00
examples Feature: LUKS keyslot rotation script (!7) 2026-09-08 07:03:42 +00:00
files Feature: Self-Healing Crypttab-Assembler (!8) 2026-09-10 05:56:13 +00:00
scripts Feature: LUKS keyslot rotation script (!7) 2026-09-08 07:03:42 +00:00
test Feature: Self-Healing Crypttab-Assembler (!8) 2026-09-10 05:56:13 +00:00
.drone.yml Chore: Enable drone (mmci) (#3) 2026-04-30 12:51:45 +00:00
.mmci.json Feature: Self-Healing Crypttab-Assembler (!8) 2026-09-10 05:56:13 +00:00
CHANGELOG.md Feature: Self-Healing Crypttab-Assembler (!8) 2026-09-10 05:56:13 +00:00
main.tf Feature: Per-Node LUKS-Passphrase from SOPS (!6) 2026-09-07 08:07:11 +00:00
output.tf Feature: Per-Node LUKS-Passphrase from SOPS (!6) 2026-09-07 08:07:11 +00:00
README.md Feature: Self-Healing Crypttab-Assembler (!8) 2026-09-10 05:56:13 +00:00
SECURITY.md Feature: LUKS keyslot rotation script (!7) 2026-09-08 07:03:42 +00:00
setup.tf Feature: Self-Healing Crypttab-Assembler (!8) 2026-09-10 05:56:13 +00:00
storage.tf Chore: Enable drone (mmci) (#3) 2026-04-30 12:51:45 +00:00
variables.tf Feature: Per-Node LUKS-Passphrase from SOPS (!6) 2026-09-07 08:07:11 +00:00

Tofu Deployment to setup minimal internal hosts

This is an OpenTofu module that deploys one or more Hetzner Cloud servers, each with an encrypted (LUKS-on-LVM) data volume for /var/lib/docker and, optionally, further encrypted data volumes. It is consumed by a root config that supplies the network, admin SSH key(s), and node sizing — it is not runnable on its own.

Usage

examples/basic is a complete, runnable root config and doubles as the reference example — read examples/basic/README.md first for the actual step-by-step (env vars needed, manual tofu apply/destroy flow, how the integration test in test/ exercises the same code path).

To run that integration test yourself instead of a manual tofu apply:

cd test
HCLOUD_TOKEN=xxx go test -v -timeout 30m -run TestBasicDeployment

Requires tofu, go, gpg and sops on $PATH (the test skips itself if gpg or sops is missing). Unlike the manual flow above, you do not need to set TF_ENCRYPTION, hold a project GPG key, or have the encrypted passphrase file: basic_test.go uses a fixed, clearly-dummy state passphrase and generates its own throwaway GPG key plus dummy LUKS passphrase file in a temporary directory. All of it is created and destroyed within the same run and protects nothing that outlives it. See test/basic_test.go and examples/basic/README.md for details.

test/rotate_test.go (TestRotateDeployment) exercises files/rotate_storage.sh the same way: deploy, verify the volumes mount, reboot, confirm they come back locked, unlock by hand with the original passphrase, rotate, reboot again, then confirm the old passphrase is rejected and the new one unlocks and mounts cleanly. It does not exercise scripts/rotate-luks.sh (its git-history/tofu output plumbing to drive that script is orthogonal to whether the on-node keyslot rotation itself is correct).

cd test
HCLOUD_TOKEN=xxx go test -v -timeout 30m -run TestRotateDeployment

In outline, any root config using this module looks like:

module "hosts" {
  source = "git::https://<repo>//"   # or a relative/local path, as in examples/basic

  admin_keys = [ hcloud_ssh_key.admin.id ]   # SSH key(s) pre-created by the root config
  hostnet    = { id = ..., iprange = ... }   # network pre-created by the root config

  # SOPS-encrypted file with one LUKS passphrase per node. The module decrypts
  # it; it never generates a passphrase. See "Secrets" below.
  luks_sops_file = "${path.root}/secrets/luks-passphrases.enc.yaml"

  nodes = {
    count     = 1
    types     = ["cpx12"]
    storage   = [[50]]                        # one extra 50GB encrypted volume; [[]] for none
    firewalls = [ hcloud_firewall.ssh.id ]     # the module never creates its own firewall
  }
}

The module itself creates neither the network, the SSH key, nor a firewall — see examples/basic/main.tf for what a root config has to provide around it.

Secrets: the LUKS crypt-passphrases

Read this before running tofu apply for real.

This module does not generate the LUKS passphrases. It reads them from a SOPS-encrypted file whose path is a required input:

  • var.luks_sops_file — path to a SOPS-encrypted YAML file holding a flat map of node name to passphrase. The module decrypts it via the carlpett/sops provider and uses each node's own entry to luksFormat/unlock both the dockerdata volume group and the mmhosting/volumes group on that node. One passphrase per node, so compromising or rotating one node's key never touches the others.
  • The keys must match var.node_name_template (node01, node02, ... by default). If any node has no entry, or an empty one, the plan fails with a message naming the missing nodes — the module never provisions a partially-keyed fleet.
  • There is no passphrase output — there is no module-generated value to reveal. The encrypted file is the authoritative copy.
  • The LUKS volumes are deliberately set up as luks,noauto with no key file stored anywhere on the node's own disk (see files/setup_storage.sh and the module's README output for why — this is intentional, so that a stolen/decommissioned disk alone can never be unlocked). Unlocking a volume after any reboot requires typing its passphrase in by hand.

Because the module never creates the secret, nothing about it is derived from Terraform state — but note the passphrase does still pass through state as an input variable value, which is why state encryption (below) remains required rather than optional.

Where the passphrases come from

The intended source is a SOPS-encrypted file holding one entry per node, encrypted to operator PGP keys. It lives beside whichever root config reads it — its own .sops.yaml and secrets/luks-passphrases.enc.yaml, not this module's — since a real deployment's root config carries its own secrets independent of this module's source tree. examples/basic follows this convention (examples/basic/.sops.yaml, examples/basic/secrets/); see planning/CRYPT.md ("Model B") and planning/IMPLEMENTATION.md for the design.

scripts/gen-luks-secrets.sh creates and updates the file. It writes relative to the current directory by default, so run it from the root config's own directory (or pass --out):

cd examples/basic
../../scripts/gen-luks-secrets.sh --count 3      # generates entries for node01..node03
../../scripts/gen-luks-secrets.sh --count 3 --force node02   # rotate one node (see phase 4)

The script never overwrites an existing entry without --force, and never writes a plaintext passphrase to disk. Requires sops, gpg, yq and openssl on $PATH.

Rotating a node's passphrase

Rotation adds a new LUKS keyslot and removes the old one on the live, already-unlocked volume — it never destroys/recreates the node and never re-encrypts data. It is always an explicit, operator-invoked step, never an implicit side effect of tofu apply; see planning/CRYPT.md/planning/IMPLEMENTATION.md (Phase 4) for why.

  1. Regenerate and commit the node's new passphrase:
    cd examples/basic
    ../../scripts/gen-luks-secrets.sh --count 3 --force node02
    git add secrets/luks-passphrases.enc.yaml
    git commit -m "Rotate node02 LUKS passphrase"
    
  2. Run the rotation against the already-applied deployment, from the same directory (so tofu output -json hosts resolves the node's address). The node is selected either by its 0-based index (node01 is index 0, node02 is index 1, ...) or by its name:
    ../../scripts/rotate-luks.sh 1        # by index
    ../../scripts/rotate-luks.sh node02   # by name
    
    This reads the new value from the file on disk and the old value from the file's content just before its last change in git (unrelated commits in between are harmless; pass --commit REF to read the old value from an explicit commit instead), pushes files/rotate_storage.sh to the node over SSH, and runs it there: it adds the new passphrase as a fresh keyslot, verifies it opens the device, then kills the keyslot the old passphrase used. It is idempotent — re-running it after a successful rotation is a no-op. Requires sops, gpg, yq, git and tofu on $PATH plus SSH access to the node.

The SOPS-encrypted secrets file is meant to be committed to the root config's own repository — that is the design, not an oversight: the ciphertext in git is both the durable backup of these passphrases and, via git log, a tamper-evident audit trail of who rotated what and when (see "SOPS-encrypted passphrases committed to git" in SECURITY.md). You must generate the file (or obtain it from whoever holds it) before the first tofu plan/apply, and commit it. It is the only copy of these passphrases; losing it and every PGP private key that can decrypt it means every encrypted volume on every node becomes permanently inaccessible. That is a total, silent data-loss event, not a recoverable outage — there is no reset procedure and no fallback keyslot.

examples/basic/secrets/luks-passphrases.enc.yaml follows exactly this pattern: it is committed to this repository, encrypted to a throwaway key pair kept only for this example, so a fresh clone can run the example as a throwaway test deployment immediately. Never reuse that key pair or its passphrases for a real deployment — generate your own.

Decrypted, the file is just:

node01: "kx8...Rq="
node02: "pZ2...Nt="

A root config only has to point the module at it — the module owns the sops provider and the decryption itself. examples/basic/main.tf:

module "hosts" {
  # ...
  luks_sops_file = "${path.module}/secrets/luks-passphrases.enc.yaml"
}

The key names must match var.node_name_template (default node%02d), which is also the convention scripts/gen-luks-secrets.sh writes. A missing or empty entry for any node aborts the plan before anything is provisioned — the error names the file and every affected node.

Decrypting requires one of the PGP private keys in the same directory's .sops.yaml to be available to sops at plan/apply time. This is the same trust boundary that already exists for unlocking a node's volumes by hand — not a new category of access, just made explicit.

To recover a passphrase later (e.g. for the manual unlock after a reboot), from the root config's own directory:

sops -d secrets/luks-passphrases.enc.yaml

/etc/crypttab ownership

/etc/crypttab is a shared file: any module that wires something onto the same node could, in principle, need an entry in it. This module used to write the whole file with a single cat >, which silently erased any entry another module had added whenever setup_storage.sh re-ran (e.g. a re-apply of this module, independent of whether the other module's own provisioner had any reason to re-run).

Instead, every entry — including this module's own dockerdata and volumes — is its own fragment in /etc/crypttab.d/<name>, composed into /etc/crypttab by files/crypttab-assemble.sh. That script is installed once, by setup_storage.sh, to /usr/local/sbin/mmdc-crypttab-assemble.sh on every node, together with a systemd path unit that watches both /etc/crypttab and /etc/crypttab.d and re-runs the assembler whenever either changes — so a wholesale rewrite (by this module or anything else) gets this module's entries (and any other module's) reassembled back within moments, and a fragment dropped later doesn't need a reboot or a re-apply of this module to take effect.

If you're writing a module that also needs a crypttab entry on a node this module deployed (e.g. tofu-hcloud-mmdc-bluegreen, for its own shared volumes): just drop your own /etc/crypttab.d/<name> fragment. Do not write /etc/crypttab directly, and you do not need to install your own copy of the assembler — this module already installs and watches for it on every node.

State encryption (TF_ENCRYPTION)

The passphrases still transit state as input-variable values, so terraform.tfstate in plaintext would expose every node's LUKS passphrase to anyone who can read it. OpenTofu's native state/plan encryption (>= 1.7.0) closes that gap, and any root config using this module must enable it (examples/basic/main.tf does).

  • A root config's terraform { encryption { ... } } block declares the non-secret structure — which key provider (pbkdf2), which method (aes_gcm), and that both state and plan should use it. This part is safe to commit to git.

  • The actual passphrase argument for the key provider is deliberately left out of any tracked file. It must be supplied at runtime via the TF_ENCRYPTION environment variable, which OpenTofu merges into the block, overriding any argument set in both places:

    export TF_ENCRYPTION='
    key_provider "pbkdf2" "master" {
      passphrase = "'"$( pass show infra/tofu-state-master-pw )"'"
    }
    '
    
  • Without TF_ENCRYPTION set, OpenTofu refuses to run against a config that declares an encryption block — it does not silently fall back to writing plaintext state.

This introduces a second secret that needs durable, independent storage: the state-encryption master password itself. Losing it means the state file can no longer be decrypted. That is no longer catastrophic for the data — the LUKS passphrases live in the SOPS file, not in state, so the volumes stay recoverable — but it does mean losing the ability to manage the deployment through OpenTofu. Store it the same way you'd store any other master credential (e.g. pass, as above), separately from wherever the state backend itself lives, and separately from the PGP keys guarding the SOPS file — the two secrets protect against different failure modes and neither is a substitute for the other.

See examples/basic/README.md for the full manual-run walkthrough using this.