So your team has finished wrapping up the sprint and everything’s ready for the big day: Release Day. Hopefully, this final step will be as peaceful as a walk in the park, but maybe you’re on the completely opposite side of the spectrum. Maybe for your team, Deployment Day is not a joyous day, but one of stress, agony, and suffering. In this article, we’ll try to give you our (opinionated) vision of release management best practices and how the entire process should flow in order to ease Deployment Day as much as possible.
Pre Release: Be the Ant, Not the Grasshopper
The title already says the most important concept we want you to remember from this section: be the ant, invest time in actions that will ease the process before the actual release, and make it as fail-proof as possible. The pre-release part is therefore the most important step and you should start investing in it even before any word of the first release is even uttered.
We don’t know your specific build system, nor the processes you follow, but we’ll try to give you generic actions, which will surely improve your release and help you adopt some release management best practices (unless you’re already using them).
There are two main actions that will ensure, for the most part, the success of the process: automation and the Release Script.
Automate everything
Let’s start with the first one, automation. Any manual task is prone to error, and as our friend Murphy says, anything that might happen, will happen, so any manual task will fail. We’ll either completely forget to do it, or we’ll simply do it wrong, but it will fail. Hence the need for automation.
Regarding automation, if you’re not already doing Continuous Integration, you should consider it. Building and testing your product should always be as automatic of a process as possible. There are a lot of CI tools out there and nowadays the most used cloud repositories include pipelines. Investing the time and effort in creating a good CI infrastructure will save time and mistakes during the release process. Below, you can see a list of the most used CI/CD tools in the market compared:
We also advise using different environments, at least the main 3: development, test/staging, and production. The steps for deploying your product to any of these environments should also be automated, and the step for promoting from development to test/staging should be the same as for promoting from test/staging to production. This will help you catch problems sooner.
The Release Script
The release script is simply a list of steps to follow before, during, and after the release. It sounds simple, but it will be your most precious artifact to ensure the success of your release.
It should contain every step, no matter how insignificant, that will be required during the release. The principle that applies here is the same as with the automation one: anything not written down is prone to be forgotten in the heat of the moment, so it’s always best to have it written down.
The release script must:
- Be accessible to every member of the team
- Be maintained and updated as necessary (it must not be set in stone)
- Include a rollback plan (disaster recovery is VERY important on a production release)
Although the release script can be generic to all releases, it should include any specific step, necessary for every specific deployment (config, data, new servers, third party systems, networking changes, etc).
Helpful Actions
In this section, we’ll include some of the release management best practices we follow to ease the release processes.
Probably the most beneficial one is to implement a blue-green deployment system. It will hugely ease the deployment process by allowing you to deploy the new version (and testing it in production), while maintaining the previous version to run for your clients. Any problem that might arise will not be detected nor suffered by your clients.
We’ve talked about blue-green deployment, but there are lots of other release methods that can help you reduce the risk when deploying: canary deployment, feature toggles, and any other method that could arise.
Rotate the deployer role. There shouldn’t be a deployer in the team, every member of the team should know how to deploy (the Release Script helps a lot in this). This will improve the process as the more people involved, the more ideas and error detection you’ll get. It will also protect you from not being able to deploy when the “deployer” gets sick (because they will get sick).
Share your release plan and schedule to anyone whom your deployment might affect (stakeholders, other teams, QA consultants, …) way before the release date. This will help you find out any side effects you can suffer from or create, that you may not even be aware of (maybe there’s a huge infrastructure change already scheduled, and it could be better to postpone your deployment). It’s really useful to have a channel dedicated to releases in your messaging app of choice for these kinds of things.
Prepare the smoke and acceptance tests with the QA team. They should coordinate with POs and stakeholders to prepare the best tests so that to assure that after the deployment is done, everything is doing what it’s supposed to.
Avoid Fridays and weekends. Remember our friend Murphy. You don’t want your release failing miserably on a Friday and you simply don’t want to deploy on weekends. You’ll have a reduced team and any problem that might arise will be harder to handle (maybe the only one that knows how to solve it is climbing a mountain).
Directly implement a Continuous Delivery system. Just as Continuous Integration will reduce risk by continually building, integrating, and testing your software throughout the whole development process, continuous delivery will also reduce your release process to a minimum. Surely the effort of having to create the process is there, but the benefit is to nearly reduce the release risk to zero. You can see more here.
Use semantic versioning. It’s a really simple thing, but it helps a lot to maintain a good history of what you’re actually deploying. You can check it in detail here, but let’s summarize it: anything you deploy must have an associated version with the format MAJOR.MINOR.PATCH that will increment with the different deployments. And when we say for anything you deploy, we mean for ANYTHING. Even if the deployment includes only a line of code. If you deployed version 1.0.0 and suddenly you have to change a line and deploy again, then you should deploy version 1.0.1. That way you’ll have the historic information of what you deployed.
Finally, The Release Day
So you have followed all the advice and finally we get to the release day. If you’ve been a good ant, and you’ve followed the release management best practices for the most part, there should be really little to be done today. Some advice:
- Be verbose about it. Notify it again on the release channel. Things could have changed from yesterday and suddenly your deployment should be postponed for who knows what reason.
- Follow the Release Script. You’ve invested time compiling it, now don’t forget to follow it or you would have simply wasted your time. Share it on the release channel and update it as you get to complete the steps.
- Be alert for errors and always be ready to rollback.
Post Release: We’re Not Finished Yet
Well, it seems everything went OK and the new version is already deployed to production. The first thing to do should be for the QA team to start the smoke and acceptance testing. If you’re using blue-green deployment, this could be done in the new environment before switching (which will save a lot of problems in case of errors).
Only after your QA team gives you the OK, should you accept that the deployment is actually a success and proceed to notify the stakeholders and celebrate.
The worst has passed, so take some time and maintain the release script. Did something unexpected occur during the deployment? Write it down. Did someone have a good idea during release? Write it down.
Also, don’t forget to do the post-deployment tasks:
- Merge the master branch downstream or the expected step at this point for your repository workflow.
- Close the version on your tasks tracking software.
- Whatever your team needs to do after releases (maybe going for beers?)
To Hotfix or Not to Hotfix
There is one final point we want to touch on. So, let’s say you’ve deployed your new version, it’s already being used by users and suddenly someone finds a really small error. It’s really easy to fix, just a couple of lines of code. Should you proceed and fix it and deploy it in the heat of the moment?
The short answer is no. You might think that it’s small and easily fixed, but let’s be honest, you really don’t know. Our advice would be to either analyze whether the error can stay in production while your team fixes it (in this case you don’t need to rollback) or if necessary rollback directly and start again.
As always the long answer is maybe. In the end, you know your product and your team better, so maybe in your case, you can do the hotfix as hot as you want.
Summary of Release Management Best Practices
To summarize the release management best practices mentioned above:
- Be an ant, be prepared, invest beforehand, it will bring ROI
- Automate, use any kind of technology/workflow that might help you, it will bring ROI
- Create, maintain and follow the deployment script, it’s your best weapon
- Notify, inform, keep people in the loop, it will keep you away from surprises
- QA after you’ve deployed, the deployment is not really finished until they say so
- Be careful with quick hotfixes
Now enjoy, you’ve successfully deployed your next version. See you on the next deployment.
—
This article is written by Alvaro Fernández González, Senior Software Developer at Zartis.
In case you have any questions or request further information on release management best practices, feel free to reach out to us at any time!
