Somewhere real

Part 4 of five in StellarView Alpha, a series about delivering a solved application as an operated platform.

docker compose up works on a laptop. A real account is the step off it.

Nobody handed it a template

Phase five was told to deliver infrastructure as code, a cloud deployment, and a rollback that had actually been run. That is not a summary. It is the phase’s title, decomposed out of the requirements document and sized at twelve work units before a line existed:

It wrote the Terraform itself: thirty two files, including network.tf, database.tf, loadbalancer.tf, secrets.tf, observability.tf, registry.tf, dns.tf, plus backend-staging.hcl and backend-prod.hcl and an environments/ directory.

Two hundred and seventeen lines for compute alone.

The trap, and the machine’s own warning about it

Sixty five resources stand in us-west-2 right now. The certificate is real and practicaicloud.tech resolves. The application returns 503.

We did not have to go looking for that. The platform had already filed it against itself. Top of its own Control Room, marked P1:

Here is why, from infra/variables.tf, written by the build:

variable "container_image_tag" {
  description = "Tag of the aqueduct image (in the ECR repo created by registry.tf) to deploy. Defaults to \"latest\" for convenience, but real deploys must override this with an immutable commit-SHA tag (the ECR repo has image_tag_mutability = \"IMMUTABLE\" — see registry.tf), since that immutable tag-per-commit is the whole basis for rollback-by-revision."
  type        = string
  default     = "latest"
}

Read the description, then read the default.

It states the constraint precisely. The ECR repository is set to IMMUTABLE, deliberately, because an immutable tag per commit is what makes rollback-by-revision trustworthy. Then it defaults to latest, which an immutable repository can never resolve, and the service fails to pull:

CannotPullContainerError: ...aqueduct-staging-app:latest: not found

The decision that makes rollback trustworthy is the same decision that guarantees the default cannot work. It wrote both halves and did not notice they were in tension. Neither did the ten phase gates, and neither did I.

The fix is the one variable that comment names:

terraform apply -var image_tag=643462b

The block that protects a rollback from Terraform

This one is the opposite of a mistake, and it is the reason I trust the rest of the file. From infra/compute.tf:

# from immediately trying to revert task_definition back to whatever
# var.container_image_tag currently resolves to in this root's own state.
lifecycle {
  ignore_changes = [task_definition]
}

I called that a bug twice before I read the comment above it. It is there so that a rollback performed outside Terraform is not undone by the next terraform apply. The infrastructure is declarative; the deployed revision is operational; and the file knows the difference.

The rollback, honestly

The rollback was demonstrated against local containers by image tag:

=== deploy 528ca49 — the current revision ===
  healthy   POST /api/users -> HTTP 201
=== roll back to 8c06137 — the revision before commission ===
  healthy   POST /api/users -> HTTP 400
=== roll forward again ===
  healthy   POST /api/users -> HTTP 201

Eight seconds. A rollback proven by the defect coming back, which is the only version of the demonstration worth watching. Every other rollback demo shows a green checkmark, which proves that something completed, not that anything changed.

It is not a production incident and we are not going to call it one.

The pipeline that blocks a merge

Ten jobs: lint, unit tests, a dependency audit, a secret scan, tenant isolation, ticket isolation, retrieval isolation, observability, wiki drift, and conformance. A merge waits for them.

The conformance job carries a confession in a comment, which we left in:

# THE STEP THAT EARNS ITS NAME.
#
# This step used to be called "Run RealWorld conformance suite" and ran
# scripts/conformance.mjs — 24 checks we wrote against the endpoints we
# built. A green tick claiming a third party's name for our own assertions.
# Nobody reading the workflow would catch it without going and fetching the
# actual suite.

It now clones the real suite inside CI and runs it on every pull request. A stranger’s tests are the merge gate.

What is true, and what is not

Sixty five resources stand in a real Amazon account. The application is not serving. It is one command away and it costs about seventy dollars a month sitting there.

Five obstacles were hit getting there, including a VPC limit at five of five and an earlier teardown that reported success while resources stayed live.

What is proved is the boundary, and that is in the database rather than in the code. Same connection, five tenant contexts:

-- SELECT count(*) FROM articles, on one connection, five contexts
no tenant set          0 rows
as tenant-a            2 rows
as tenant-b            1 row
as conduit            95 rows
empty string           0 rows
SET ROLE aqueduct      ERROR: permission denied to set role "aqueduct"

The last line is the point. The application cannot opt out of its own isolation.

---

Conduit is the reference application of RealWorld, created in 2016 by Eric Simons and Albert Pai and funded early by Thinkster. It was revived in 2021 by Gérôme Grignon and Manuel Vila. It is maintained today by c4ffein and Manuel Vila. MIT licensed, © Thinkster and © c4ffein.

We are not affiliated with the project. Passing its suite shows conformance to its specification and nothing more.

Next, Operated: somebody else’s test suite, and it failed by one request.