Two application pools helped a timed-exam platform grow. The harder work was proving readiness, draining old nodes and keeping one version of the service.
Autor
Anichur Rahaman
An hour after the frontend DNS change, the old server still received about 36% of recorded traffic. The team had lowered TTL to 300 seconds and checked the new route through a hosts-file override. The migration nevertheless had two live entrances. Keeping the old entrance working mattered more than declaring the DNS change finished.
NovaCommerce is an anonymised course-and-exam platform. This second part preserves September–October 2026 case observations; they are not StoreConsole benchmarks or independently re-run tests. Part 1 identified rate-limit identity, Valkey connection pressure and then backend CPU as different constraints. This part follows the application tier.
The implemented design used one load balancer and autoscale pool for the frontend, another pair for the backend. Separate DNS names already existed. Independent health checks, node sizes and replacement policies made this a reasonable choice for this deployment; two balancers are not a universal requirement for every application.
Tag-based backend discovery replaced a fixed list of Droplet IDs. New tagged nodes could enter the intended pool and firewall rules. A tag is configuration, not proof of application readiness or authorisation. The historical combined load-balancer budget was $48/month, not a current quote.
Both pools were configured for 2–10 nodes. Frontend nodes used nginx and two Next.js containers on 2 vCPU/4 GB; backend nodes used nginx/PHP-FPM on 4 vCPU/8 GB. These were measured deployment choices. A single Node.js event loop does not prove every other CPU is idle: worker-pool and native work also matter.
The historical frontend run recorded about 356 requests/s, p95 0.05 s, zero reported errors and 55–60% CPU on two nodes. That supported a cheaper shared-CPU trial; it does not establish all routes, noisy-neighbour behaviour or write-path capacity. The old $28/$56 monthly node figures are historical planning inputs.
| Historical configuration | Frontend | Backend |
|---|---|---|
| Pool range | 2–10 nodes | 2–10 nodes |
| Per node | 2 vCPU / 4 GB; 2 Next.js containers | 4 vCPU / 8 GB; PHP-FPM |
| CPU target | 70% | 70%, later 55% |
| Health boundary | Next.js response checked | nginx-only; app readiness separate |
| Other state | Public cache needs coordination | One fixed worker outside pools |
The 5 October 21:17–21:33 k6 run used a GET-heavy replay: 300–450 requests/s, then roughly 750 for six minutes. Frontend and backend each grew from two to three nodes. Reported server errors/timeouts were zero, but about 4% of responses were HTTP 429 from a shared test IP. A rejected student request is still unsuccessful; this was not a clean business-flow pass and did not test durable answer writes.
Overall p95 was 132–153 ms, while API p95 was 320–705 ms. The case recorded 5–8 minutes of metric lag and about seven minutes from 99% backend CPU to the first extra node in another run. Those are observations of this configuration, not a provider-wide timing guarantee. The team pre-scaled roughly 45 minutes before exams and checked healthy capacity. Kubernetes was considered, not implemented; it would not guarantee instantly ready application capacity either.
The first template rollout caused a short 503 burst. A later procedure waited for replacements, admitted them to the balancer, then drained old nodes before deletion. The recorded 30–40-second waits belong to those health-check settings; a safe gate checks actual admission and active work, not just a timer. Long requests, WebSockets and provider-initiated scale-in need separate handling.
The backend /lb-health endpoint was nginx-only. That is useful for liveness and a drain switch, but cannot prove PHP or a database-dependent route works. Pair it with bounded application readiness and real-path probes without making every health check an expensive query. The later rollout recorded 219 probes, every two seconds, with zero errors; this sampling does not prove zero interruption for every user.
Keep immutable, tested images and compatible database/queue changes. For multiple Next.js instances, coordinate the build, applicable action keys, cache invalidation and version skew. Cache only eligible public responses; never share a student session or private page. Restrict trusted proxy headers to controlled ingress. A forwarded IP is not a student identity or permission.
Sessions/cache/queues were externalised to Valkey, uploads to Spaces and backend logs shipped off-node. A 24-hour observation of no backend file writes supported the design; it did not prove every future route is stateless or make abrupt deletion safe. Queues and scheduled work remained on one fixed worker, a known failure point; centralising them does not itself give exactly-once effects.
The old frontend had to remain compatible while old connections and cached addresses persisted. A 300-second TTL does not close established connections, and the 36% observation alone cannot identify every cause. DNS reversal is not instantaneous rollback and must not create two database writers. After new writes, recovery must preserve those writes.
The useful result was a repeatable readiness-and-drain procedure, not the promise that nobody will ever notice a replacement. Next, Part 3 follows the database cutover and the legacy client the team missed.
Reviewed 6 November 2026. Anichur Rahaman, StoreConsole founder. Historical case observations are retained from existing project records; general guidance is supported by the primary references, not a new live test.
Über den Autor
Anichur Rahaman
Weiterlesen
