The Problem
I rolled out of bed on a brisk summer morning in Phoenix (95F) to walk my dogs before the heat would force us into our daily hibernation routine (seek shade; stay inside; dream of a home in Greenland).
Checking my texts as my eyes adjust to the aggregiously flamboyant sun peaking over the McDowell Mountains, I see a frantic, somewhat hyberbolic SMS from a director saying:
Whelp, I've poured myself a glass of Belvenie on ice and I'm tendering my resignation.
We made breaking changes to the org. reports. It broke for Client X because we didn't give them enough notice aannd their entire operation will be halted at 9AM EST (east coast based client) due to inability to process the new reports, costing them a huge amount of wasted time and money.
They emailed our CEO and I [the director] was cc'd. Nice knowing you, Jason. It's been a good run.
That's not how we wanted to start the day. The rest of the timeline:
- 4:15AM - Give doggos special-contingency frozen marrow bones. Sorry, doggos, no walk today. (They were pleased with their consolation prize)
- 4:20AM - Instruct director not to jump off any nearby tall hills
- 4:30AM to 5:00AM - Upload previous version of org. reports to every client's SFTP servers before 8AM EST
- 5:00AM - 5:30AM - Share post-mortem with team and implement new SDLC process specific to Change Management around the org-reports system
- 5:45AM - Drive to said director's home and have them pour me a glass of Balvenie such that we might lament together (and so they don't feel tempted to do anything irrational around surrounding, aforementioned tall hills)
- 6:00AM - Chin up and continue iterating on SDLC whilst building epic products
The Solution
- Now, organization reports have a 90-day communication period with clients for breaking changes
- Any new fields added to these reports are optional for pre-existing clients who are subscribed, but can be enabled at their discretion. For new clients, all fields are enabled by default
Lessons Learned
- Change Management processes around Production systems can be a critical step as an organization grows
- Ensure you have someone who can supervise and ensure those Change Management processes are being followed
- Heavily focus on documentation and training so that Product and Engineering teams are familiar with Change Management for these systems
- Use a
CODEOWNERSto ensure the right supervisors are notified anytime these systems change and ensure there are clear policies around merging code in these systems.