Using Canary Releases and Early Life Support to improve Production Releases

We've all had that moment where a friend comments (or complains) on social media about the new layout of a particular site. But you still see it the way it has been for months, and you have no idea what they are talking about. This is because the company that runs the site is likely using a Canary Release. This type of release is named after the early 20th century practice where canaries were used as sentinels in coal mines to detect carbon monoxide. According to Wikipedia : "Animal sentinels, or sentinel species, are animals used to detect risks to humans by providing advance warning of a danger". Thus the canary would get sick from the carbon monoxide before it could seriously affect the men, giving them an early warning sign to be able to evacuate the mine before they, too, became sick. The Canary Release allows the product release to be deployed to an early warning system, where it can be tested before releasing it across all of the production environments. ITIL refers to these types of releases as Phased Releases. Canary Releases can be an important tool for a company when undertaking CD or even Continuous Deployment . By rolling out new features quickly, and testing them with live users, feedback can be implemented quickly and efficiently. Testing with actual users in a production environment also allows the company to gain insight into actual performance numbers, rather than just in a simulated testing environment. As Jez Humble states , one of the principles of low-risk releases is to have incremental releases. "With this pattern, rather than upgrading a whole cluster to the latest version all at once, you do it incrementally." Canary Releases allow you to incrementally install and test new releases , without risking your entire production environment.

To implement Canary Releases, it is imperative that the deployments be a) automated , and b) the configurations in production be set up in such a way that releasing to a limited number of production nodes is possible. Your deployment automation should be configurable in such a way that the application can be deployed to a single set of servers. Using a deployment automation solution, like CloudBees Flow Deploy , will allow you to select the environments on which to deploy your software release, and then scale from there, if everything looks good. A Canary Release will set up a small number of environments to test in production, before rolling out the release to all of the production nodes. In this way, the sentinel release environment(s) serves to allow the team to watch over the release until the system can be brought up to a capacity that will match production. The environments used to test the production release can be taken out of the production server rotation (remove them from the load balancer), have the new version installed, then traffic can be routed to them in a controlled manner, allowing the testing to be managed, and brought up to production activity levels slowly. If the tests are successful, other environments can be removed from the load balancer and have the new version installed until all systems are made consistent with the latest release. If any problems are found during the Canary Release, since it is a limited number of environments which have received the new release, it will be easier to revert those environments to the previous release than it would have been if all of the production environments were involved. To revert, these Canary environments can be taken out of the server rotation again, and then returned once they are reverted back to the last working version of the product. This type of release ensures that not all customers are affected if there were problems with the new released version.

Another factor to improve the Release Process is to use an Early Life Support (ELS) process . In this instance, a Deployment Team is organized, with well defined and documented criteria on what constitutes an acceptable release. The team monitors the release for a specified amount of time, determined by the company, to ensure that it meets the exit criteria for the release. The ITIL Service Transition standards have a good description of the ELS Deployment Team. "The deployment team’s aim is to stabilize the service for the target deployment group or environment as quickly and effectively as possible." With the ELS Deployment Team in place, the company can closely monitor the release, until the exit criteria are met and the release is considered stable and ready to run without monitoring. The Deployment Team is another early-warning device that allows for a quick reaction team to respond to any issues found in production. While this may not be as effective in an Agile environment, when multiple releases can occur in a single day, it can be very effective in larger companies, where resources are more available and larger releases are done less frequently.

Using a combination of Canary Releases and Early Life Support, companies can work to find the right balance to ensure that their Product Releases are stable and ready for production, improving the overall customer experience for their users. So now you can tell your friends that this is how they might still be seeing the old Facebook interface, for example, while others may be commenting about their experiences on the new interface - since their instance is served from the Canary server.

Canary Releases are just one way to test your deployments and ensure a successful release into Production. Other patterns are Blue/Green deployments, Feature-flag, Dark Launches, Environment Promotion, and more. To learn about other common patterns to deploying mission-critical apps, the differences between them, and what does each mean in terms of your architecture, code base or Operations- check out the video replay of one of our recent #c9d9 episodes below:

Continuous Discussions (#c9d9): Episode 25 Strategies for Deploying Mission Critical Apps at Scale