deploying on a legacy financial system without breaking anything

the assignment

Add seven features to a digital wallet system in operation: profiles with differentiated permissions, an admin dashboard, search across the account registry, audited editing of account holders, expanded reports, receipts with holder details, and full refund of funds. The system was in production, moved real money, and was subject to audit by a public agency.

what made it hard

The system came from another vendor.

• There was no test environment: what everyone called "staging" was production.
• The code arrived with no history: seven components delivered as folders of files.
• The web application was built from a repository the client had no access to.
• The documentation did not match reality: the manual declared one framework version and production ran another, two major versions newer, because the dependencies were not pinned.
• The container platform is discontinued: a deleted service cannot be recreated.

Zero margin for error.

how i solved it

I rebuilt the entire system on my machine at real volume: the exact production images against a database with 1,500 accounts, the same total balance and 131,000 transactions. Not a mockup: a replica.

I rehearsed the full deployment four times, verifying three things in every rehearsal: that the previous version behaved exactly as it did in production, that the database updates applied without moving money, and that the previous version kept working against the already-updated database. That third point is what makes the whole operation safe: if anything fails, you roll back in seconds without touching the database.

The rehearsal caught a real problem before production: the previous version would not have been able to create users against the already-migrated database. I wrote a specific migration and rehearsed again until it was closed.

I defined a truth metric before starting: the total money across accounts, the number of accounts, the held balance, the transaction count, and the timestamp of the last one. Five measures, taken minutes before and repeated at the end.

I left three verified rollback paths before touching anything: copies of the databases, the previous versions of every service tagged server-side, and the web artifact downloaded. None of them depended on my computer staying on.

the result

I also verified that what was published was exactly what was tested: the files the site serves have the same cryptographic fingerprint as the ones I generated locally. Not an equivalent build, the same artifact.

what the client kept

• Their code repositories, with history and automated scanning for leaked credentials, and a clear path to bring them under an account of their own.
• A report of 31 findings ordered by severity, each with its impact and a recommendation.
• A documented, tested deployment procedure, so that next time does not depend on me.
• Six security fixes for issues that were live in production, among them a page with no permission check and an export that allowed downloading the personal data of the entire registry in a single operation.

how i work

I rehearse before touching production: I reproduce the system with data at real volume and run the full procedure, rollback included, until it comes out the same several times in a row.

I verify the artifact, not the process: a command exiting cleanly does not prove the result is correct; I inspect what actually ended up inside the image, the database and the server.

I define the truth metric before starting: in a system that holds money, that metric is the money. If I cannot prove it did not move, I cannot say the deployment went well.