An AWS Deployment Setup I Actually Like

This article will catalog a proper CI/CD setup for docker builds and deployment with AWS CDK. This is my special blend developed over many fortnights and many failures. I use this setup to run my current projects, and I hope to share the system I’ve worked hard on to save other people the headache.

Let’s define two terms upfront. The term environment is used to distinguish between local development happening on your laptop, called the development environment, and any deployment in AWS, called the production environment. Tier distinguishes between the client-serving deployment (prod) and ephemeral PR deployments created by pull requests for testing. There is no staging.

Trunk Based Development

It starts with Trunk Based Development, meaning there is one long lived branch, master. Every unit of work branches off master and then merges back into it. Master is prod and a merge deploys to the prod tier through CDK.

Monorepo With CDK

All code is held in a monorepo. Application code and CI live together and ship together. The repo contains two CDK stacks:

• The app stack, which both tiers deploy from the same code
• The base platform stack which holds VPCs, RDS, ECR registries, certificates, and other shared resources.

Platform Stack

These shared resources live in their own stack because most of what they do is not wanted in each app’s CDK. Opening a PR deploys the app stack into the PR tier, so new CDK that reaches prod on merge has already run.

The platform stack is where the tiers differ, deliberately.

VPCs

Prod and PR tiers get separate VPCs with identical subnets for two reasons:

1. It keeps the tiers as close as possible.
2. Connecting to both VPCs at once is impossible, maintaining separation between tiers.

RDS

The platform stack also owns the RDS instances, one for prod and one for PR. Each PR deployment gets its own database. This is because:

1. RDS deployments are slow, and one shared instance speeds up deploys.
2. A database template seeds real data into each new database, so migrations get tested against real data and PR deployments come ready to test with data.

Since prod gets its own database as well, none of this touches it.

Certificates

Certificate generation takes the longest by far. Sharing one certificate across PR deployments cuts more time off a deploy than anything else. We mint two certs:

• example.com for prod
• *.dev.example.com for PR deployments, each deployment takes a pr-X.dev.example.com

ECR

Both tiers share ECR registries, meaning:

1. Container builds all push to one place, simplifying CI.
2. The app stack always pulls from the same registry.

Implementation

Talking about all this high-level can be hard to understand, so let’s get technical. This is not a complete implementation; it only covers the relevant parts to the setup I described above.

CDK

Both CDK stacks live in infra/.

infra
├── bin
│   └── app.ts
├── cdk.json
├── lib
│   ├── constants.ts
│   ├── constructs
│   │   ├── db-tasks.ts
│   │   ├── web-service.ts
│   │   └── monitoring.ts
│   ├── context.ts
│   └── app-stack.ts
├── package.json
├── package-lock.json
├── platform
│   ├── platform.ts
│   ├── cdk.json
│   ├── platform-stack.ts
│   └── README.md
├── README.md
└── tsconfig.json

Platform Stack

This exists as a directory inside infra/ and shares dependencies with the app stack. The platform stack does not have automatic CI; it is run manually. It contains everything a fresh AWS account needs, OIDC for CI, ECR registries, and all items mentioned above. As this stack prepares the account for automatic CI, it itself cannot be automatic. Further, changes to this stack already require heavy manual intervention, and manually deploying makes that process easier by giving the developer direct control. Any changes or additions to the AWS account, such as IAM roles, should be within this stack since it keeps the repo portable between accounts.

App Stack

This is the application’s stack, made of parts like a load balanced web service, CloudWatch monitoring, and importantly the database migration component. It receives a very important input: TIER. TIER (prod or pr-X) is what tells the stack what to set its stack name to and how to choose the VPC, RDS instance, and certificate. It’s also used in the naming of the DNS records:

• prod gets example.com
• pr-42 gets pr-42.dev.example.com

This TIER is the mechanism by which the separation and differences between prod and PR are carried out.

Further, an IMAGE_TAG variable picks which image to pull from ECR.

Database Migrations

My database migration tool is built as a Docker image, sent to ECR, and deployed with the stack. It bundles the migrations generated by our ORM and a few other bits to get the database setup, such as creating access roles and pg-boss queues.

During a deployment, it’s triggered by a lambda function before the rest of the stack deploys, and gets the database ready and migrated for the containers.

During an update to an existing stack, it’ll ensure migrations are run before new containers are spun up, and the stack will revert if the migrations fail.

Workflow Setup

I see many articles which try to separate CI from CD, but to me they are the same.

.github/workflows
├── cicd.yml
├── pr-cleanup.yml
├── reusable-cd.yml
└── reusable-ci.yml

cicd.yml

cicd.yml is run on push to master and pull_request to master. Four jobs:

1. First, tests runs which performs a typecheck and checks for missing database migrations. This gates deployments on tests to ensure things will work before we waste time trying to build and deploy.

2. A generate_docker_tag job doing exactly what the name implies. This Docker tag is in two forms: push-[run #]-[short-sha] for direct push to master and merges, and pull-[run #]-[short-sha] for pull requests into master. This differentiates images for the prod and PR deployments.

   1 2 3 4

   short_sha="$( echo ${{ github.sha }} | cut -c1-7 )" event="$( echo ${{ github.event_name }} | sed -E 's/_.+$//' )" docker_tag="$event-${{ github.run_number }}-$short_sha" echo "docker_tag=$docker_tag" >> $GITHUB_OUTPUT

3. build runs for each container build needed. These builds use the reusable-ci.yml workflow.

4. A deploy job to deploy the app stack using the reusable-cd.yml workflow.

reusable-ci.yml

This workflow builds a container, with several inputs:

• DOCKER_TARGET to differentiate what image to build as my repo has one Dockerfile.
• ECR_REPOSITORY to know where to push to.
• DOCKER_TAG (sourced from the previous step) to tag the image appropriately.

OIDC creds are used to authenticate to AWS, and the setup for that is in the platform stack above.

reusable-cd.yml

This workflow deploys the app stack to AWS. For PR deployments, prior to deployment it runs a step to create the database from the template. (I host my GitHub runners in the PR tier VPC, meaning they have network access to the database)

1
2
3
4
5
6
7

EXISTS=$(psql postgres -tAc "SELECT 1 FROM pg_database WHERE datname='${DATABASE}'")
if [ "$EXISTS" != "1" ]; then
    createdb -T app_template "${DATABASE}"
    echo "Created database ${DATABASE}."
else
    echo "Database ${DATABASE} already exists, skipping."
fi

The database name is determined from the PR number, i.e. pr_42_app. The app_template is created manually, and nothing actively accesses it so it stays available to be cloned.

When the deploy step runs, it passes the TIER and IMAGE_TAG variables to the app stack.

pr-cleanup.yml

This step is responsible for clean up after a PR has been closed. It runs on pull_request of type closed on master. The single job runs cdk destroy --force to remove the PR’s stack, referenced by its name passed as TIER, and drops the PR’s database.

Closing Notes

I hope I’ve effectively laid out a realistic and digestible CI/CD setup which earns it’s complexity. Write to me with questions; I love questions.

This setup pairs very well with an accompanying TypeScript monorepo, which I may write about later.