Situation

The monthly maintenance job started at midnight and ran for more than six hours.

Parents are awake and messaging by four. By seven they are working out who collects the child from school that day.

Every one of those conversations happened while the platform was still rewriting their accounts and adjusting their available credits.

My client’s mediation platform rolls over credits and usage data for every account at the start of each month, and Knack’s task system needed all six of those hours to work through the records.

So parents were reading credit balances and usage figures that were halfway through an update, in the app a court requires them to use.

Complication

Six hours was not a slow query that I could tune. It was what Knack’s task system could do with that volume of records, and no amount of rewriting the job inside Knack was going to move it.

The same ceiling showed up somewhere else entirely.

The platform runs a custom JavaScript and CSS layer on top of Knack, and Knack cached those assets hard enough that a deploy reached some users immediately, others eventually, and some not at all. There is no cache to clear on the server in that arrangement. The stale copy sits in each user’s browser.

Bugs I had already fixed kept coming back as reports from people still running cached code that contained them.

The obvious fix was to try bypassing cache when the app updates but this traded one complaint with another. Bypassing the cache on every page load did bring everyone up to date on the current code, but it cost one to two seconds on every page view. Users noticed immediately.

The remaining option was asking every user to clear their browser cache after every deploy. That is not a fix. It fails the first time one person ignores it, and it puts a technical chore in front of a parent who is only there because a court told them to be.

Two unrelated symptoms but the same root cause. The platform had outgrown the infrastructure that came with the tool it was built on.

Investigation

The asset problem stopped looking like a caching question once I framed it properly. Correctness wanted no cache. Speed wanted cache. Both are satisfiable at the same time if the URL changes whenever the code changes, because then a browser can hold a file forever and a new release is simply never the same file. Nothing to invalidate, nothing re-downloaded that has not actually changed.

That turned it into a build and release problem.

The heavier decision was where any of this should run. GitHub Actions alone was the free option and I looked at it seriously, but the work involves enough API call processing that runner minutes were a real risk of exceeding quota.

A VPS at five dollars a month removed that ceiling. It also brought the things Knack could not offer at any price: scheduled jobs outside Knack’s task system, webhook endpoints, and somewhere to keep API secrets that is not the client-side bundle.

Five dollars against a job that was corrupting the customer experience one morning a month.

Resolution

The monthly rollover moved onto the server, running against the API on its own schedule rather than inside Knack’s task system.

The deployment pipeline runs in two chained stages.

A push to main, beta, or alpha triggers minification of all JavaScript and CSS, committed back automatically. That run completing triggers the deploy, which calls an authenticated endpoint on the server rather than handing CI any shell access. Branch determines environment, so production, beta, and alpha each deploy from their own branch.

Assets are served from their own subdomain, cached with expires 1y and Cache-Control: public, immutable. A release is a new file, so a browser never has to ask whether its copy is stale.

The rest is deliberately ordinary. systemd for process and cron management, nginx in front, Let’s Encrypt for TLS. Deploys shut down gracefully, waiting up to five minutes for in-flight webhooks and background jobs to finish before the process exits, so a deploy never kills work midway. Logging runs through Pino, so a production problem is traceable rather than guessed at.

The monolithic front end that all of this deploys was broken into modules separately, which is its own story.

Results

• The monthly rollover went from over six hours to fifteen minutes. It finishes long before anyone is awake to be affected by it.
• Page loads dropped from one to two seconds down to roughly 50ms, measured across pages. Around a 95% reduction, with no cache-bypass workaround left in place.
• Deploys reach everyone. A release is a distinct file at a distinct URL, so there is no version of the app where some users are months behind.
• API secrets moved off the client and onto the server, along with webhook handling and scheduled jobs Knack’s task system could not run.
• Three environments, production, beta, and alpha, each deploying from its own branch.
• All of it runs on a five dollar a month VPS.

Back to Case Studies