← all projects

Case study · EC7204 Cloud Computing · architecture review and rebuild

Cloud Auction Platform

An online auction platform built as cloud-native microservices: NestJS services behind one gateway, RabbitMQ events between them, live bidding over WebSockets, and auctions that start, close and pick a winner on their own. It began as a review of the first version, which found 26 issues, among them a race that could accept two conflicting bids. The rebuild addresses every one, and the whole system is designed to run from one command, on a laptop or the AWS free tier, at no cost.

  • NestJS
  • React
  • TypeScript
  • PostgreSQL
  • RabbitMQ
  • Redis
  • BullMQ
  • Socket.IO
  • Prisma
  • Docker Compose
  • Kubernetes (k3d)
  • Terraform
  • AWS
  • GitHub Actions
  • OpenTelemetry

Overview

Version 1 of the platform was a Spring Boot microservice system built for the EC7204 cloud computing module: five Spring Boot applications, RabbitMQ, Redis, PostgreSQL, Prometheus and Grafana, deployed to Azure Container Apps. Its patterns were sound. It had a gateway with circuit breakers, a database per service and a transactional outbox. The business logic and security, though, had gaps that a live demo would expose quickly.

Version 2 keeps that shape and rebuilds the substance in TypeScript. It has an API gateway, user, auction, bid and notification services, with a wallet service as stretch scope. A React single-page app receives live updates, and the deployment costs nothing to run. This case study covers the review, the architecture, the three hardest problems (concurrent bids, auctions driven by time, and reliable events), and how the system is shipped.

  • 26issues found reviewing v1, 6 of them high severity
  • 1command from a fresh clone to a seeded, working demo
  • ≈1.1 GBmemory budget for the core stack
  • US$0to run locally or on the AWS free tier

Reviewing v1

Before designing anything new, I reviewed every branch of v1 and recorded what was actually built, what worked and what was broken, with a file and line reference for each finding. The diagram below shows the v1 containers with each finding placed on the part of the system it affects.

The v1 system as built: a static UI served by a Spring Cloud gateway, four Spring Boot services each with its own database, RabbitMQ and Redis, with red finding markers F1 to F26 on the parts they affect.
v1 as built. Red markers are the findings; the dashed red boxes (a payment service, Zipkin, MongoDB) were configured or designed but never actually ran.

The most serious findings were:

  • Authorisation holes. The gateway protected /api/auctions/create/**, but the real endpoint was POST /api/auctions, so any logged-in user could create auctions. Any user could also start or close any auction, and the seller id was read from the request body.
  • No auction lifecycle. Nothing started or ended auctions at their scheduled times, and no winner was ever chosen. A "Start" button in the UI did the job by hand.
  • A race condition. The highest-bid check and the insert were not serialised, so two concurrent bids could both be accepted, including equal or lower amounts.
  • Hardcoded admin credentials were committed to the source code.

Below those sat a long tail: tokens in OAuth redirect URLs, stored XSS through innerHTML, users silently logged out after 15 minutes because the UI never refreshed its session, polling everywhere, and a schema managed by ddl-auto: update with no migrations. The v1 design diagrams also described lots, payments and settlement that were never built. v2 treats those diagrams as requirements.

Constraints for the rebuild

  • Stack: NestJS, React, pnpm and Docker; Kubernetes only where it adds value; AWS for the cloud.
  • No recurring cost: no EKS, load balancer, NAT gateway, managed broker or managed cache, because all of them are billed by the hour.
  • Small footprint: the whole stack must run on an 8 GB laptop and on a 2 GiB AWS t4g.small.
  • Portable: Windows (WSL2), macOS on Apple Silicon and Intel, and Linux.

Architecture

Bidders, sellers and administrators all use one web app. Google sign-in, e-mail delivery and a payment provider are optional integrations that switch off when they are not configured, so a laptop demo needs no external accounts.

C4 system context diagram: bidders, sellers and administrators use the auction platform, which optionally connects to Google Identity, an e-mail service and a payment provider, with GitHub hosting code, CI/CD and images.
System context (C4 level 1). Every external system except GitHub is optional.

The design follows a short list of principles:

  • Right-sized microservices split along the v1 bounded contexts, with one database and one database role per service.
  • Synchronous calls only where an answer is needed now (gateway proxying, bid validation, wallet capture). Everything else goes through events.
  • At-least-once and idempotent everywhere: an outbox on the producer, an inbox on the consumer, idempotency keys on commands.
  • One write path per aggregate. Every change to an auction goes through one service and its state machine, whether it comes from HTTP, a timer or an event.
  • Secure by default: a deny-by-default route table, identity only from verified tokens, and no secrets in images.
  • The same artefacts everywhere: the same images and configuration model in Compose, Kubernetes and AWS.
C4 container diagram: a React SPA calls an API gateway, which routes to user, auction, bid, wallet and notification services. Each service owns a PostgreSQL database and exchanges events through RabbitMQ; Redis holds ephemeral state; Prometheus, Grafana and Jaeger form an optional observability stack.
Containers (C4 level 2). The gateway is the only public entry point. Solid lines are synchronous calls, dashed purple lines are AMQP events and WebSocket pushes, and the wallet service is stretch scope.

Why TypeScript

Moving from Java to TypeScript was partly about using one language end to end, but mostly about memory. A Node service uses roughly 80–150 MB, against 300–500 MB for each Spring Boot service, and that difference decides whether six services plus infrastructure fit on a 2 GiB instance. A pnpm and Turborepo monorepo holds every app, and a shared contracts package of zod schemas validates API input, types the SPA and defines every event payload from one source.

Inside the services

Each component maps directly to a NestJS module, so the component diagrams double as the module plan. Inside a service, dependencies point one way: controllers and consumers call application services, which call a pure domain layer that imports nothing from Nest or Prisma. Lint rules enforce those boundaries.

Placing a bid

Bidding is the hot path, and it is the part v1 got most wrong. In v2 one use case handles every bid. It gathers the facts, takes a lock, runs a pure policy, and commits the bid and its event together.

  1. Gatewayrate limit, verify the JWT, check the role
  2. Idempotency + rulesreplay a known key; read the rules through a 5 s cache
  3. Lock + policyone lock per auction; rules checked against database time
  4. Commitbid, outbox row and idempotency record in one transaction
  5. Fan-outoutbid notice, price projection, live push
The bidder gets an answer in one round trip. Everything after the commit happens asynchronously.
Sequence diagram for placing a bid: the SPA posts to the gateway, the bid service checks the idempotency key, fetches bidding rules from the auction service, takes a PostgreSQL advisory lock, evaluates the policy, then commits or rolls back; afterwards the outbox relay publishes bid.placed to RabbitMQ for the notification service, the auction service and the gateway's WebSocket push.
The full bid sequence. Steps 9–12 are the fix for the v1 race; steps 16–21 are the asynchronous fan-out.

Serialising competing bids

v1 read the current top bid and then inserted the new one, so two requests could read the same top bid and both succeed. v2 opens a transaction and takes a PostgreSQL advisory lock keyed on the auction. Competing bids on the same auction wait for microseconds, and bids on other auctions are not affected. The time rule uses the database's now() inside the lock, so the server clock is the only clock that counts.

bid-service · PlaceBidUseCase transactionsql · design sketch
BEGIN;
-- one lock per auction, released automatically at COMMIT or ROLLBACK
SELECT pg_advisory_xact_lock(hashtext(:auction_id));

-- read the current top bid while holding the lock
SELECT amount FROM bids
 WHERE auction_id = :auction_id
 ORDER BY amount DESC LIMIT 1;

-- BidPolicy (pure TypeScript): status = IN_PROG, now() < end_at,
-- amount >= max(starting_price, top + min_increment), bidder <> seller

INSERT INTO bids             (...) VALUES (...);
INSERT INTO outbox_events    (...) VALUES ('bid.placed', ...);
INSERT INTO idempotency_keys (...) VALUES (...);
COMMIT;

Each click in the SPA sends a fresh Idempotency-Key, so a retried request returns the original result instead of creating a second bid. When a rule fails, the transaction rolls back and the API returns a 409 with a stable error code and the minimum next bid, which lets the UI offer a one-click "bid minimum". If the auction service is down and no cached rules are available, the bid is refused with a 503 within a second. Bidding fails closed rather than guessing.

The acceptance test for all this is blunt. k6 fires 200 concurrent bids at one auction, and a SQL assertion checks that no accepted bid breaks the increment rule and that the final price equals the highest accepted bid.

Auctions that run themselves

In v1, nothing moved an auction at its start or end time, and no winner was ever chosen. v2 makes time a server concern. The auction's lifecycle is an explicit state machine, implemented as a pure function from (state, command) to (newState, effects), with a unit test for every transition. An invalid transition returns a 409.

Auction state machine with states OPENED, IN_PROG, HALTED, POSTED, ENDED and CLOSED, and numbered transitions 1 to 11 between them.
The auction lifecycle. Transitions 1–9 keep the numbers from the v1 design; 10 and 11 (green) are new in v2. Bids are accepted only in IN_PROG and only before endAt.

When a seller creates or edits an auction, the service schedules two BullMQ delayed jobs, auction:{id}:start and auction:{id}:end. Their ids are deterministic, so an edit replaces them safely. At endAt, the auction service closes bidding and publishes an event. The bid service ranks the distinct bidders (ties go to the earliest bid) and publishes the ranking, and the auction service decides the outcome:

  • no bids, or the reserve price not met → CLOSED, and the lot is revoked;
  • otherwise → ENDED, the top-ranked bidder wins and the lot is sold;
  • with the wallet feature on → the settlement saga described below.

Timers alone are not enough. If Redis is flushed or a worker crashes, a job can be lost, so a reconciler runs every 60 seconds. It finds auctions that should have started or closed and replays the transition through the same code path, which makes the lifecycle self-healing.

Sequence diagram of the auction lifecycle: BullMQ jobs trigger start and end transitions, the bid service publishes a ranking, the auction service decides ENDED or CLOSED, bids are settled, notifications go out, and a reconciler repairs missed transitions.
Lifecycle and winner determination, including the reconciler that catches anything the jobs miss.

Lots, bids and accounts

The other three state machines also come from the v1 design, which v1 never enforced. A bid's "leading" or "outbid" status is derived from the ranking, so outbidding someone never updates existing rows. When an account leaves ACTIVE, every session it holds is revoked.

Settlement (stretch scope)

With the wallet feature on, the outcome becomes a saga taken straight from v1's "winning an auction" flow chart. The service tries to capture the winning amount from the winner's wallet. If the balance is short, the auction moves to POSTED with a top-up deadline and a payment reminder. If the deadline passes, the defaulter is suspended and the next-ranked bidder becomes the pending winner. The saga's state lives on the auction row, so it survives restarts, and the whole feature sits behind a flag so it cannot put the core demo at risk.

Sequence diagram of the settlement saga: capture from the wallet, on failure move to POSTED with a deadline, retry on top-up, and on deadline suspend the defaulter and try the next candidate.
The settlement saga: compensating steps (suspension, next candidate) instead of a distributed transaction.

Events and data

Services talk through RabbitMQ: one topic exchange per producing service, one durable queue per consumer, and about 20 event types. A failed message goes to a retry queue with a 5-second delay, up to five attempts, and then to a dead-letter queue, so a poison message never blocks the others.

Outbox and inbox

A service never publishes directly from a request. The use case writes its business rows and an outbox_events row in the same transaction. A relay polls every 500 ms with FOR UPDATE SKIP LOCKED, publishes with publisher confirms and marks the row as sent, which is safe with several replicas. On the consuming side, the handler first inserts the event id into an inbox_events table. If that insert is a duplicate, the message is acknowledged and skipped. Together these give at-least-once delivery with effectively-once handling. v1 already had simpler versions of both patterns. v2 hardens them and shares one tested library across every service.

packages/contracts · event envelopejson
{
  "eventId": "0192c7f0-6a1e-7c3e-9a51-2f0d4e1b8a77",
  "type": "bid.placed",
  "version": 1,
  "occurredAt": "2026-10-21T09:12:31.204Z",
  "producer": "bid-service",
  "correlationId": "req_7f3c…",
  "traceparent": "00-4bf92f…-00f067aa0ba902b7-01",
  "data": { /* type-specific payload, validated with zod */ }
}

Ordering across messages is not guaranteed, so handlers tolerate reordering. The price projection only ever moves upward, and a state event is applied only if its transition is valid from the current state. The traceparent in the envelope carries the trace across the broker, so a single bid appears in Jaeger as one trace from the gateway through the bid service, RabbitMQ and the notification service and back to the browser push.

When something breaks

failurewhat users seerecovery
Notification service downBidding, browsing and the lifecycle carry on; notifications are delayedQueue holds messages until it restarts
RabbitMQ downBids are still accepted; live updates pause with a bannerThe outbox relays the backlog when the broker returns
Redis downRate limiting fails open (logged); scheduled jobs pauseThe reconciler catches overdue transitions
Auction service downAuction pages and bids return 503 within a second; other pages workCircuit breaker half-opens after 15–30 s
User service downNew logins fail; existing sessions work for up to 15 minutesCached signing keys keep tokens verifiable

Data model

One PostgreSQL server hosts five databases, and each service connects with a role that can reach only its own. There are no cross-database foreign keys: services reference each other by id and keep their copies in sync through events. Money is stored as numeric(12,2) and sent as a string ("125.00"), times are UTC timestamptz, and ids are time-ordered UUID v7. Schemas change through reviewed Prisma migrations instead of v1's ddl-auto: update.

Entity-relationship diagram grouped by database: user_db, auction_db, bid_db, notification_db and wallet_db, linked across databases only by logical ids.
The data model, one database per service. The auction service keeps a denormalised price projection for fast list pages; the bid service stays the source of truth for bids.

Security

Most of v1's high-severity findings were security problems, and most of them came down to trusting the wrong thing: the UI, the request body, or a secret that every service shared. v2 takes identity only from verified tokens and denies anything that is not explicitly allowed.

v1 findingv2 fix
Any user could create auctions: the route rule did not match the endpointOne declarative route table generates the proxy, the role checks and the rate limits, so they cannot drift apart. Services check roles too.
Seller id taken from the request bodyIdentity comes only from the token's sub; request schemas reject unknown fields
One HS512 secret copied to all five servicesRS256: only the user service holds the private key, and the others verify through a JWKS endpoint
Access and refresh tokens in the OAuth redirect URLThe callback sets an HttpOnly refresh cookie only; no token ever appears in a URL
Stored XSS through innerHTMLReact escaping, dangerouslySetInnerHTML banned by lint, and a strict CSP
Login revealed whether an e-mail was registered; suspended users could still log inOne generic error, a dummy hash for unknown e-mails, and account status enforced on login, refresh and bidding
Users logged out silently after 15 minutesAccess token in memory, rotating refresh cookie, and a single-flight silent refresh
Hardcoded admin credentials in the sourceAdmin seeded from environment variables; gitleaks in CI

Sessions

Passwords are hashed with argon2id. Access tokens last 15 minutes and are kept in memory only. The refresh token is a random 256-bit value stored as a SHA-256 hash, sent in an HttpOnly, SameSite=Strict cookie scoped to the auth routes, and rotated on every use. If an already-rotated token is ever presented again, the whole session family is revoked, which is the standard defence against a stolen refresh token.

Authorisation in four layers

  1. Gateway: the route table declares authentication and roles for every path. Unknown paths and anything under /internal return 404.
  2. Service: every controller verifies the JWT and its roles itself, so services never rely on gateway headers alone.
  3. Domain: ownership checks ("is this your auction?") live inside the application services.
  4. Tests: a generated test calls every endpoint as anonymous, USER, TRUSTED_USER, ADMIN and owner, and asserts the expected status.

Role and status changes take effect within seconds rather than at the next login. When the user service publishes a role change, the gateway marks that user's older tokens as stale, and the SPA silently fetches a new one. When a user is suspended, the gateway blocks them and closes their open WebSockets.

Running it for free

Docker Compose is the primary runtime, and the same images run in Compose, Kubernetes and AWS. A fresh clone starts with one command and no configuration. A one-shot container generates the JWT signing keys, others run migrations and seed demo users and auctions, and healthchecks order the start-up.

deploy/scripts/demo.shbash
./deploy/scripts/demo.sh up          # app :8080 · mail :8025 · RabbitMQ :15672
./deploy/scripts/demo.sh up --obs    # + Grafana, Prometheus, Jaeger
./deploy/scripts/demo.sh up --wallet # + wallet service and settlement (stretch)
./deploy/scripts/demo.sh reset       # wipe volumes and reseed
./deploy/scripts/demo.sh down

Every container has a memory limit, and every Node service caps its heap so V8 stays inside that limit.

containerlimittypical
gateway192 MB110 MB
user / auction / bid / notification (each)160 MB85–120 MB
PostgreSQL192 MB90 MB
RabbitMQ256 MB130 MB
Redis, Nginx, Mailpit128 MB33 MB
core total≈ 1.5 GB≈ 1.0–1.1 GB

AWS on the free tier

The public demo is a CloudFront distribution in front of a private S3 bucket for the SPA and a single t4g.small EC2 instance running the same Compose stack. The instance accepts traffic only from CloudFront's origin-facing IP ranges, has no SSH port (operators use SSM Session Manager), and reads its secrets from SSM Parameter Store at deploy time. Terraform creates everything with pnpm aws:up and removes it with pnpm aws:down, and a US$1 budget alarm watches the bill.

AWS deployment: visitors reach a CloudFront distribution, which serves the SPA from a private S3 bucket and forwards API and WebSocket traffic to a single EC2 t4g.small running Docker Compose in the default VPC; IAM, SSM Parameter Store, SSM Run Command, Budgets and CloudWatch sit alongside, with optional RDS and SES.
The AWS deployment. No NAT gateway, no load balancer and no hourly-billed managed services.

CI/CD

Pull requests run lint, type checks, unit and integration tests, contract checks and security scans, building only the packages a change affects. A merge to main builds multi-architecture images (amd64 and arm64 for Graviton), scans them with Trivy, attaches an SBOM, pushes them to GHCR tagged by git SHA, and runs the full demo storyline in Playwright against the new images. Deploying to AWS is a manual workflow that authenticates through GitHub OIDC, so no cloud keys are stored anywhere, and it rolls back automatically if the smoke test fails. A nightly job adds load tests and chaos scripts.

CI/CD pipeline: pull-request checks, a release workflow that builds, scans and pushes multi-arch images then runs end-to-end tests, a manually approved AWS deploy with OIDC and rollback, a teardown workflow, and nightly load and chaos runs.
The delivery pipeline, from pull request to a CloudFront URL and back down again.

Results

The result is a complete design that the build can follow without re-deciding the architecture. It consists of a 26-section architecture document, 20 diagrams drawn in plain HTML and CSS (rendered to the PNGs on this page), 16 architecture decision records, and a backlog that breaks the work into tasks with acceptance criteria, priorities and dependencies.

  • 59functional requirements, each traced to a service and an epic
  • 16architecture decision records
  • 167backlog tasks across 17 epics and 6 milestones

Every requirement is either carried over from v1, taken from a v1 design that was never built, or new, and every requirement is prioritised. The 110 "Must" tasks (about 131 person-days) make up a demo-ready milestone. Stretch work such as the wallet and Kubernetes only starts once that milestone is reached.

Targets the build is held to

qualitytargetverified by
DeployabilityFresh clone to working demo with one command in ≤ 10 min, no manual configTimed E2E run in the release workflow
Correctness under concurrency200 concurrent bids on one auction: 0 accepted bids breaking the increment rulek6 plus a SQL assertion, nightly
PerformancePlacing a bid p95 ≤ 300 ms at 50 concurrent usersk6 bid-storm scenario
Real-time latencyOther viewers see a new price within 1 s (p95)Playwright timing and a push-lag histogram
ResilienceBroker outage loses no events; notification outage leaves bidding unaffectedChaos scripts, nightly
Security0 secrets in the repo, 0 critical image vulnerabilities, every endpoint covered by an authorisation testgitleaks, Trivy and the matrix test in CI
CostUS$0 locally; AWS inside free-tier allowances with a US$1 alarmAWS Budgets and the teardown workflow

The acceptance test is a ten-step demo storyline that doubles as the Playwright end-to-end suite. Two bidders outbid each other with prices updating live in both browsers. A seller's auction starts by itself. An admin halts and resumes it, then ends it early, and the winner, losers and seller each receive the right notification. The notification service is then stopped mid-demo, bidding carries on, and the missed notifications arrive when it comes back. The last step shows that bid's trace in Jaeger, crossing every service involved.

What I learned

  • Authorisation in the UI is not authorisation. Hiding v1's "create" button did nothing, because the gateway rule and the real endpoint had drifted apart. Generating every rule from one table, and testing every endpoint against every role, removes that drift instead of hoping to catch it.
  • A check followed by a write is a race unless something serialises it. The fix did not need new infrastructure. One advisory lock in the right transaction was enough at this scale, and it was the smallest change that was provably correct.
  • Time is a feature. Anything that must happen "at the end time" needs a scheduler, a reconciler for when the scheduler misses, and a single authoritative clock.
  • Cost constraints shape an architecture as much as patterns do. The move to Node was decided as much by memory per service as by language, and it is what makes the free-tier deployment possible at all.
  • Design diagrams that never get built are debt. v1's state diagrams were a good specification that nobody implemented. Treating them as requirements gave v2 most of its domain logic.
  • Next time, I would build the walking skeleton before the features. Distributed bugs are hard to find without tracing, so the v2 plan puts tracing, CI and one event flowing end to end into the very first milestone.