- Go 44.2%
- Shell 35.1%
- HCL 20.7%
| Filename | Latest commit message | Latest commit date |
|---|---|---|
own /etc/crypttab via crypttab.d fragments + self-healing assembler Ref: #361 Co-authored-by: Nikolas List <list@metromorph.de> Reviewed-on: https://gitea.mm.intern/hosting/tofu-hcloud-mmdc-hosts/pulls/8 Reviewed-by: Nikolas List <list@metromorph.de> Co-authored-by: H.A.L (Claude) <devel+hal@metromorph.de> Co-committed-by: H.A.L (Claude) <devel+hal@metromorph.de> |
||
| examples | ||
| files | ||
| scripts | ||
| test | ||
| .drone.yml | ||
| .mmci.json | ||
| CHANGELOG.md | ||
| main.tf | ||
| output.tf | ||
| README.md | ||
| SECURITY.md | ||
| setup.tf | ||
| storage.tf | ||
| variables.tf | ||
Tofu Deployment to setup minimal internal hosts
This is an OpenTofu module that deploys one or more Hetzner Cloud servers, each with an
encrypted (LUKS-on-LVM) data volume for /var/lib/docker and, optionally, further encrypted
data volumes. It is consumed by a root config that supplies the network, admin SSH key(s), and
node sizing — it is not runnable on its own.
Usage
examples/basic is a complete, runnable root config and doubles as the reference example —
read examples/basic/README.md first for the actual step-by-step (env vars needed, manual
tofu apply/destroy flow, how the integration test in test/ exercises the same code path).
To run that integration test yourself instead of a manual tofu apply:
cd test
HCLOUD_TOKEN=xxx go test -v -timeout 30m -run TestBasicDeployment
Requires tofu, go, gpg and sops on $PATH (the test skips itself if gpg or sops is
missing). Unlike the manual flow above, you do not need to set TF_ENCRYPTION, hold a
project GPG key, or have the encrypted passphrase file: basic_test.go uses a fixed,
clearly-dummy state passphrase and generates its own throwaway GPG key plus dummy LUKS
passphrase file in a temporary directory. All of it is created and destroyed within the same
run and protects nothing that outlives it. See test/basic_test.go and
examples/basic/README.md for details.
test/rotate_test.go (TestRotateDeployment) exercises files/rotate_storage.sh the same
way: deploy, verify the volumes mount, reboot, confirm they come back locked, unlock by hand
with the original passphrase, rotate, reboot again, then confirm the old passphrase is rejected
and the new one unlocks and mounts cleanly. It does not exercise scripts/rotate-luks.sh (its
git-history/tofu output plumbing to drive that script is orthogonal to whether the on-node
keyslot rotation itself is correct).
cd test
HCLOUD_TOKEN=xxx go test -v -timeout 30m -run TestRotateDeployment
In outline, any root config using this module looks like:
module "hosts" {
source = "git::https://<repo>//" # or a relative/local path, as in examples/basic
admin_keys = [ hcloud_ssh_key.admin.id ] # SSH key(s) pre-created by the root config
hostnet = { id = ..., iprange = ... } # network pre-created by the root config
# SOPS-encrypted file with one LUKS passphrase per node. The module decrypts
# it; it never generates a passphrase. See "Secrets" below.
luks_sops_file = "${path.root}/secrets/luks-passphrases.enc.yaml"
nodes = {
count = 1
types = ["cpx12"]
storage = [[50]] # one extra 50GB encrypted volume; [[]] for none
firewalls = [ hcloud_firewall.ssh.id ] # the module never creates its own firewall
}
}
The module itself creates neither the network, the SSH key, nor a firewall — see
examples/basic/main.tf for what a root config has to provide around it.
Secrets: the LUKS crypt-passphrases
Read this before running tofu apply for real.
This module does not generate the LUKS passphrases. It reads them from a SOPS-encrypted file whose path is a required input:
var.luks_sops_file— path to a SOPS-encrypted YAML file holding a flat map of node name to passphrase. The module decrypts it via thecarlpett/sopsprovider and uses each node's own entry toluksFormat/unlock both thedockerdatavolume group and themmhosting/volumesgroup on that node. One passphrase per node, so compromising or rotating one node's key never touches the others.- The keys must match
var.node_name_template(node01,node02, ... by default). If any node has no entry, or an empty one, the plan fails with a message naming the missing nodes — the module never provisions a partially-keyed fleet. - There is no
passphraseoutput — there is no module-generated value to reveal. The encrypted file is the authoritative copy. - The LUKS volumes are deliberately set up as
luks,noautowith no key file stored anywhere on the node's own disk (seefiles/setup_storage.shand the module'sREADMEoutput for why — this is intentional, so that a stolen/decommissioned disk alone can never be unlocked). Unlocking a volume after any reboot requires typing its passphrase in by hand.
Because the module never creates the secret, nothing about it is derived from Terraform state — but note the passphrase does still pass through state as an input variable value, which is why state encryption (below) remains required rather than optional.
Where the passphrases come from
The intended source is a SOPS-encrypted file holding one entry per node, encrypted to
operator PGP keys. It lives beside whichever root config reads it — its own .sops.yaml and
secrets/luks-passphrases.enc.yaml, not this module's — since a real deployment's root config
carries its own secrets independent of this module's source tree. examples/basic follows this
convention (examples/basic/.sops.yaml, examples/basic/secrets/); see planning/CRYPT.md
("Model B") and planning/IMPLEMENTATION.md for the design.
scripts/gen-luks-secrets.sh creates and updates the file. It writes relative to the current
directory by default, so run it from the root config's own directory (or pass --out):
cd examples/basic
../../scripts/gen-luks-secrets.sh --count 3 # generates entries for node01..node03
../../scripts/gen-luks-secrets.sh --count 3 --force node02 # rotate one node (see phase 4)
The script never overwrites an existing entry without --force, and never writes a plaintext
passphrase to disk. Requires sops, gpg, yq and openssl on $PATH.
Rotating a node's passphrase
Rotation adds a new LUKS keyslot and removes the old one on the live, already-unlocked
volume — it never destroys/recreates the node and never re-encrypts data. It is always an
explicit, operator-invoked step, never an implicit side effect of tofu apply; see
planning/CRYPT.md/planning/IMPLEMENTATION.md (Phase 4) for why.
- Regenerate and commit the node's new passphrase:
cd examples/basic ../../scripts/gen-luks-secrets.sh --count 3 --force node02 git add secrets/luks-passphrases.enc.yaml git commit -m "Rotate node02 LUKS passphrase" - Run the rotation against the already-applied deployment, from the same directory (so
tofu output -json hostsresolves the node's address). The node is selected either by its 0-based index (node01is index0,node02is index1, ...) or by its name:
This reads the new value from the file on disk and the old value from the file's content just before its last change in git (unrelated commits in between are harmless; pass../../scripts/rotate-luks.sh 1 # by index ../../scripts/rotate-luks.sh node02 # by name--commit REFto read the old value from an explicit commit instead), pushesfiles/rotate_storage.shto the node over SSH, and runs it there: it adds the new passphrase as a fresh keyslot, verifies it opens the device, then kills the keyslot the old passphrase used. It is idempotent — re-running it after a successful rotation is a no-op. Requiressops,gpg,yq,gitandtofuon$PATHplus SSH access to the node.
The SOPS-encrypted secrets file is meant to be committed to the root config's own repository — that is the design, not an oversight: the ciphertext in git is both the durable backup of these passphrases and, via
git log, a tamper-evident audit trail of who rotated what and when (see "SOPS-encrypted passphrases committed to git" inSECURITY.md). You must generate the file (or obtain it from whoever holds it) before the firsttofu plan/apply, and commit it. It is the only copy of these passphrases; losing it and every PGP private key that can decrypt it means every encrypted volume on every node becomes permanently inaccessible. That is a total, silent data-loss event, not a recoverable outage — there is no reset procedure and no fallback keyslot.
examples/basic/secrets/luks-passphrases.enc.yamlfollows exactly this pattern: it is committed to this repository, encrypted to a throwaway key pair kept only for this example, so a fresh clone can run the example as a throwaway test deployment immediately. Never reuse that key pair or its passphrases for a real deployment — generate your own.
Decrypted, the file is just:
node01: "kx8...Rq="
node02: "pZ2...Nt="
A root config only has to point the module at it — the module owns the sops provider and the
decryption itself. examples/basic/main.tf:
module "hosts" {
# ...
luks_sops_file = "${path.module}/secrets/luks-passphrases.enc.yaml"
}
The key names must match var.node_name_template (default node%02d), which is also the
convention scripts/gen-luks-secrets.sh writes. A missing or empty entry for any node aborts
the plan before anything is provisioned — the error names the file and every affected node.
Decrypting requires one of the PGP private keys in the same directory's .sops.yaml to be
available to sops at plan/apply time. This is the same trust boundary that already exists for
unlocking a node's volumes by hand — not a new category of access, just made explicit.
To recover a passphrase later (e.g. for the manual unlock after a reboot), from the root config's own directory:
sops -d secrets/luks-passphrases.enc.yaml
/etc/crypttab ownership
/etc/crypttab is a shared file: any module that wires something onto the same node
could, in principle, need an entry in it. This module used to write the whole file with
a single cat >, which silently erased any entry another module had added whenever
setup_storage.sh re-ran (e.g. a re-apply of this module, independent of whether the
other module's own provisioner had any reason to re-run).
Instead, every entry — including this module's own dockerdata and volumes — is its
own fragment in /etc/crypttab.d/<name>, composed into /etc/crypttab by
files/crypttab-assemble.sh. That script is installed once, by setup_storage.sh, to
/usr/local/sbin/mmdc-crypttab-assemble.sh on every node, together with a systemd path
unit that watches both /etc/crypttab and /etc/crypttab.d and re-runs the assembler
whenever either changes — so a wholesale rewrite (by this module or anything else) gets
this module's entries (and any other module's) reassembled back within moments, and a
fragment dropped later doesn't need a reboot or a re-apply of this module to take
effect.
If you're writing a module that also needs a crypttab entry on a node this module
deployed (e.g. tofu-hcloud-mmdc-bluegreen, for its own shared volumes): just drop
your own /etc/crypttab.d/<name> fragment. Do not write /etc/crypttab directly, and
you do not need to install your own copy of the assembler — this module already
installs and watches for it on every node.
State encryption (TF_ENCRYPTION)
The passphrases still transit state as input-variable values, so terraform.tfstate in
plaintext would expose every node's LUKS passphrase to anyone who can read it. OpenTofu's
native state/plan encryption (>= 1.7.0) closes that gap, and any root config using this
module must enable it (examples/basic/main.tf does).
-
A root config's
terraform { encryption { ... } }block declares the non-secret structure — which key provider (pbkdf2), which method (aes_gcm), and that bothstateandplanshould use it. This part is safe to commit to git. -
The actual
passphraseargument for the key provider is deliberately left out of any tracked file. It must be supplied at runtime via theTF_ENCRYPTIONenvironment variable, which OpenTofu merges into the block, overriding any argument set in both places:export TF_ENCRYPTION=' key_provider "pbkdf2" "master" { passphrase = "'"$( pass show infra/tofu-state-master-pw )"'" } ' -
Without
TF_ENCRYPTIONset, OpenTofu refuses to run against a config that declares anencryptionblock — it does not silently fall back to writing plaintext state.
This introduces a second secret that needs durable, independent storage: the state-encryption
master password itself. Losing it means the state file can no longer be decrypted. That is no
longer catastrophic for the data — the LUKS passphrases live in the SOPS file, not in state,
so the volumes stay recoverable — but it does mean losing the ability to manage the deployment
through OpenTofu. Store it the same way you'd store any other master credential (e.g. pass, as
above), separately from wherever the state backend itself lives, and separately from the PGP
keys guarding the SOPS file — the two secrets protect against different failure modes and
neither is a substitute for the other.
See examples/basic/README.md for the full manual-run walkthrough using this.