Spinbox on AWS

A Spinbox installation on AWS, as one module:

module "spin" {
  source       = "github.com/spin-stack/spin-terraform-aws?ref=<release>"
  spin_version = "v20261019.02"
  domain       = "example.com"
}

<release> is one of this module's releases, v<YYYYMMDD>.<NN> tags: the newest is the first on its releases (gh release list -R spin-stack/spin-terraform-aws). spin_version is a spin release.

Every machine boots Spin OS of that release, one image per role - the control plane's, the runners' and the proxy's, each laid with that role's part of the release and signed by that role's Secure Boot key: the images spin-stack/ami builds and publishes. Where each is, role by role and region by region, is images.json - { "<release>": { "<role>": { "<region>": "<ami-id>" } } } - which the ami repository's publish updates here by a pull request. A module ref is then a set of releases with their images: a release with none in the installation's region is refused at plan, naming the regions it has one in, and image_ids names images of your own build instead, one per role.

The root composes two modules, and each can be used on its own (github.com/spin-stack/spin-terraform-aws//modules/controlplane?ref=<release>) where the root does not expose what an installation needs to decide:

• modules/controlplane - the VPC, the buckets, the database's archive, the control plane's machine, the proxy's on its own, the installation's CA and secrets, and the document each machine starts on.
• modules/runners - an autoscaling group of runners that join by themselves, spot by default, that the control plane sizes: none until a workspace waits for a host, and none again once nothing has run for runner_idle_minutes (at once in the quiet_hours). It takes the control plane module's outputs whole, its name among them.

A variable of the root left unset is the module's own default: each defaults to null, which the modules read as their default, so a default is said once. examples/complete is an installation, examples/minimal the least one is (a release and a domain), and each module's inputs and outputs are in its own README. task lint validates both examples, lints with TFLint, scans with Trivy - each finding this installation decides against is ignored at its resource, with the reason - checks those READMEs against the code (task docs writes them), and runs each module's tests against its plan.

Before you start

• OpenTofu 1.11 or newer, the AWS CLI signed in to the account (aws sts get-caller-identity answers), and the Session Manager plugin for a shell on the control plane's or the proxy's machine. A runner has none: its serial console, its boot log and its metrics say what it does.

• A domain, or a subdomain of one, that is the installation's alone: its zone is made here, in Route 53 ($0.50 a month), and whoever registered it points its nameservers at the name_servers output.

• Room for runners. Each is a spot machine of 32 to 48 vCPUs with a local NVMe, of any type and size runner_instance_types matches (C8i, M8i and R8i by default: one generation, and their d variants while the data is on the instance store), in any zone of the region, and a new account's spot vCPU quota may be lower than one of them (Service Quotas → EC2 → "All Standard Spot Instance Requests"). The quota is the account's: a CI fleet in the same account and region draws on it too. How likely spot is to give and keep one in the region you chose, per zone, from 1 to 10 - below 7, widen runner_vcpus, give the runners an EBS data volume (runner_data_volume_gb, which lets the types without a local NVMe in too) or use on-demand (runner_spot = false):

  aws ec2 get-spot-placement-scores --target-capacity 1 --region-names <region> \
    --single-availability-zone --instance-requirements-with-metadata \
    '{"ArchitectureTypes":["x86_64"],"InstanceRequirements":{"VCpuCount":{"Min":32,"Max":48},"MemoryMiB":{"Min":65536},"LocalStorage":"required","AllowedInstanceTypes":["c8id.*","m8id.*","r8id.*"]}}'
  

Installing

Once per account and region, a bucket and a KMS key for OpenTofu's state (the state holds the installation's CA key, so it is encrypted and never a file on your machine):

cd bootstrap
tofu init && tofu apply

Then the installation. examples/complete is one; copy it anywhere, or use it as it is:

cd ../examples/complete
cp terraform.tfvars.example terraform.tfvars    # and fill it in
$(tofu -chdir=../../bootstrap output -raw init)
tofu apply
tofu output -raw next_steps

next_steps says the rest, in order: the DNS records, how to tell the installation is up (about ten minutes the first time) and where to look if it is not, and the first sign-in - admin@<domain> (admin_email) with a one-time password this apply wrote to SSM.

Updating

Move the module's ref to a release of spin-terraform-aws - or a commit - whose images.json has the new spin release in the installation's region, change spin_version in terraform.tfvars to it, then:

tofu plan     # the image, the launch templates and the documents change, and nothing else
tofu apply

spin-terraform-aws has releases of its own, v<YYYYMMDD>.<NN> tags a person cuts from main (gh workflow run release.yml --ref main, which names the next one today): an installation that pins one (?ref=v<YYYYMMDD>.<NN>) gets an update when somebody decides this module - its shapes and the images its images.json has - is ready for it, not when main moves. The newest is the first on its releases. main is what spin-stack's own installation, spinbox.dev, runs first.

For the installations spin-stack runs, that edit is a pull request nobody writes, and every step of it is a workflow run, with its log: spin-stack/shipset, which coordinates them, dispatches spin-stack/ami's release workflow for each spin release, which builds and publishes its image and opens the pull request that adds it to spin-terraform-aws's images.json; shipset merges that once its checks pass, and dispatches spin-terraform-aws's .github/workflows/propose.yml for each deployment its shipset.yaml names - on main for one that follows main, on the newest release's tag for one that follows the releases - which opens, or rewrites, the pull request there that moves both. A person merges that, and applies.

The control plane and the proxy are each replaced beside themselves. The old proxy serves until the new one does. The control plane's database has one writer, so the old control plane serves until the new one asks for it, then archives what it wrote and stops: the installation has no control plane until the new one has restored the database and started - seconds, which spin waits out for a read and a workspace does not notice. How it went is tofu output update_status: InProgress, Successful, or RollbackSuccessful with the reason. A new control plane that failed before it asked for the database leaves the old one serving; one that failed after hands the name back, and the old machine's watch starts its control plane within a minute, which restores the database from the archive first - or takes the name back after fifteen minutes when the new machine died without handing it. tofu output boot_log - one command per role, since a role's machines are one stream prefix - says why the new one did not come up. Until the launch hook gives up on a new machine that never comes up, the refresh holds the group and an apply that changes it is refused; tofu output update_abort, by group, is the command that rolls it back now. A change to anything else a machine starts on - installation_config, the autoscaling settings, the control plane's or the proxy's instance type - rolls out the same way. A change to a machine's document alone reaches it when it next starts; to start one now:

aws autoscaling start-instance-refresh --auto-scaling-group-name <group>

A runner cannot take a release in place either - its root is read-only and verified - and how it gets one is runner_rollout. idle, the default, moves no workspace for it: the control plane empties the group once nothing runs, starts or is still being saved to the bucket, and the next host it starts boots the new release; until then a host keeps the old one, which its control plane serves for 90 days (spin's compat.Window). rolling is for a fleet that never empties: the hosts are replaced at once, one at a time with the new one first, each old one leaving by its drain - its workspaces suspended to the bucket and resumed on another host.

A release older than the one that last started the database refuses to start on it: going back past a change to the schema is a restore of the database from its archive to a point before it, not an update.

Operating it

What drain, activate and revoke do to a runner, and how to undo each, is spin's docs/runners.md. What is AWS's is replacing a runner's machine - after a revoke, or when one is wedged: terminate its instance in the group without lowering the group's size, and the group starts another, which joins by its role as a new host.

aws autoscaling describe-auto-scaling-instances \
  --query 'AutoScalingInstances[?AutoScalingGroupName==`<name>-runners`].[InstanceId,LifecycleState,HealthStatus]' --output table
aws autoscaling terminate-instance-in-auto-scaling-group --instance-id <i-...> --no-should-decrement-desired-capacity

Drain it first if it still runs workspaces you want kept. With nothing waiting for a host the control plane may have sized the group to zero already, and then there is nothing to replace: the next workspace that waits starts one.

Several installations in one account

An account holds as many installations as it has names and regions for: two of different names in one region, and two of one name in two regions. name is what tells them apart within a region - every regional name is under it: the buckets (<name>-volumes-<account>-<region>), the groups, the security groups, and the parameters and log groups under /spin/<name>/. IAM is the account's and not a region's, so what it names carries the region too (<name>-<region>-controlplane), and so does the audience a runner signs its join for.

One name in one region is one installation. The name is claimed before anything is made under it (/spin/<name>/claim): a second installation of that name in that region stops at the claim, with nothing of it made. A name is 2 to 26 characters, which keeps the longest bucket name within S3's 63.

What the account has once, the installations share: the Spin OS image of a release (read, never changed, by each), and the EC2 vCPU quota of a region, which each installation's request_quota checks against its own runners alone - where several share a region, leave it to one of them.

Removing an installation

An installation keeps its data past a plain tofu destroy: the volumes, the database's archive and the logs are in buckets a destroy does not empty, and every bucket refuses object writes and deletes from outside the VPC - the root account's included, so they cannot be emptied from a laptop either. Removing one is said first, applied, and only then destroyed:

1. Set decommission = true on the module and apply. Nothing but the buckets changes: their VPC-only policies go, and each is marked to be emptied by a destroy - every version, the GOVERNANCE retention bypassed.

   tofu apply   # with decommission = true in module "spin"
   

2. Lift every legal hold. spin holds what each image is rebuilt from, and the provider's destroy (hashicorp/aws 6.66) lifts a hold and then deletes that version without the bypass, which the retention refuses: the destroy stops on the volumes' bucket.

   for b in $(tofu show -json | jq -r '.. | objects | select(.mode? == "managed" and .type? == "aws_s3_bucket" and .values.object_lock_enabled? == true) | .values.bucket'); do
     aws s3api list-object-versions --bucket $b --output json | jq -r '(.Versions // [])[] | [.Key, .VersionId] | @tsv' |
       while IFS=$'\t' read -r key id; do aws s3api put-object-legal-hold --bucket $b --key "$key" --version-id "$id" --legal-hold Status=OFF; done
   done
   

3. The KMS keys - the encryption key's, the volumes' and the identity's - and the CA refuse to be destroyed, because an apply that replaced one would leave an installation nothing can open. Note the keys, take them out of the state, destroy, and schedule each key's deletion (the destroy takes their aliases):

   keys=$(tofu show -json | jq -r '.. | objects | select(.mode? == "managed" and .type? == "aws_kms_key") | .values.key_id')
   tofu state rm $(tofu state list | grep -E '\.aws_kms_key\.|\.tls_(private_key|self_signed_cert)\.ca$')
   tofu destroy
   for id in $keys; do aws kms schedule-key-deletion --key-id $id --pending-window-in-days 7; done
   

The apply has to come first: a destroy empties a bucket as its state says, so decommission set only on the destroy changes nothing - it stops with BucketNotEmpty. A destroy that already stopped takes the same apply, limited to the buckets and policies the state still has: a bucket named that the destroy already removed would be made again. Its plan is those buckets' force_destroy turned on and their policies removed, and nothing else; then step 2 again:

tofu apply $(tofu state list | grep -E '\.aws_s3_bucket(_policy)?\.[a-z]+(\[0\])?$' | sed 's/^/-target=/')
# step 2's loop
tofu destroy

What is left are the keys, scheduled for deletion in seven days. The state bucket and its key (bootstrap/) are every installation's in the account and region, and are removed apart, after the last of them. .github/workflows/fresh-install.yml in spin-terraform-aws runs all of it every week - bootstrap, an installation, its first sign-in and this removal - in an account of its own, and says what that account needs.

What it decides, and why

• A machine is its release's image, and is told its role and where its document is. Every machine boots Spin OS of spin_version: a read-only root the kernel checks every block of, into which spin laid what its release.yml signed, checked against the signature when the image was built. A machine fetches nothing. Its user data is two systemd credentials - its role and the S3 object its document is - which systemd-imds imports and nothing runs. Everything else - where the database's archive is, the name it takes, the collector - is in that document, and every step is spin-boot's, in Go with tests (spin's internal/installation/boot); a document naming another release than the image's is refused. Nothing is a container: each role is a binary under systemd, as its own user.
• This apply writes every document and parameter, and no machine writes one. The documents, the installation's configuration, the CA, the encryption key and the first administrator's password are all this module's; the permissions boundary denies every role of the installation ssm:PutParameter and all but a read of the documents' bucket. A machine that was taken cannot leave a value for the next one to start on. The documents are objects, not parameters: the control plane's carries what each image's release signed of it, past the 8 KB a parameter holds.
• The encryption key and the first password never reach the state. Both are ephemeral values written through write-only attributes, once (value_wo_version). The key is, with the database, the installation: a new one is a database nothing can open, so nothing here makes one again. The CA is in the state on purpose - its certificate is in the proxy's and the runners' documents - so encrypt the state (OpenTofu's encryption block; a KMS key is the simplest).
• Runners join by who they are. A runner signs a GetCallerIdentity with its instance role, bound to this installation and to its host id, and the control plane asks STS whose signature it is (spin's internal/domain/hostjoin). The installation's configuration names the runners' role and the policy its machines start under (runner_policy: a spot reclaim announced by AWS and 100 seconds to empty itself). There is no token to mint, rotate or publish, and the runners' role reads its own document and nothing else.
• A runner runs its image's release, and nothing else: its machine fetches nothing and hands over to no other spin-boot. Whether the control plane serves a runner of that release is the control plane's to say when the runner registers (spin's internal/controlplane/compat: a host no older than the compatibility window, and none newer than itself), so while an update replaces the control plane - or after one is abandoned - the old one may go on serving the runners it can. A runner of another release is a runner of another image.

Hosts outside the group are still added the ordinary way - a one-time token from Admin → Hosts and spin-install runner - and are policed by the same control plane.