In this webinar, Software Consultant & Architect and Pete Hodgson explains the pros and cons of feature branching vs feature flags an explains when each of these software development techniques should be used.
Here is the full text of the webinar: Feature Branching vs Feature Flags Webinar Thank you, welcome everybody, so in this webinar we’re going to be talking about different ways that development teams can manage multiple independent streams of work, part of what we’re going to talk about is there’s this kind of inherent tension that we’re going to discover between managing merge conflicts and dealing with conflicts between those streams of work and being able to release those streams of work independently, and we’ll look at a technique called feature flags as one potential way, or one potential tool to help resolve that tension, and then we’ll wrap up by talking about how we can be successful with this technique or tool of feature flagging. So let’s get started with a hypothetical scenario of a new team, and this is a team of one, we have a mobile developer called Alice who’s writing a mobile app, and when Alice is working on her own she’s happy committing to controller, so this is kind of a visualisation of Alice committing to the main line, to trunk, to controller in good terms, and all is well with the world, very very simple, Alice just writes code, checks it into her repository and she’s probably releasing to the app store directly from this controller branch. So things go along like that for a while, over time Alice’s app is successful and her teams grows to multiple depths, and this is a fairly common picture once you get multiple depths in the picture. We have Alice is working on her workstation and Bob is working on his workstation, they need to collaborate because they’re working on the same code base, and so on a regular basis both of those developers will pull code from the controller, a shared controller branch, do some work locally and then once their feature is ready for release, maybe they’ll push their changes back into that shared controller branch. So that works fine, except sometimes when we try and push code into this shared controller branch there’s a merge conflict, so perhaps Bob has made some changes while he’s been working on his feature that conflict with some of the changes that Alice has been making while she’s been working on her feature and so there’s a merge conflict that we need to resolve, and this is a big pain point in software development and I’ll talk a little bit later on about why it’s perhaps more serious than we think it is. One of the things that I hear a lot, recently, is kind of like, oh well, now that we’ve got modern distribution version control systems like Git’s that kind of made branching and merging really easy, are merge conflicts really still an issue, I don’t think we really get merge conflicts anymore. I don’t think that is true, I think merge conflicts really are still an issue for almost every development team. Part of the reason for that is that Git’s merging algorithm and the merging algorithms that are used by other version control systems are kind of line oriented, so they’re great at merging changes to different lines of the same file, where they struggle is merging changes that have involved reorganising where things are in a file, they struggle with the merge algorithms, even these sophisticated merge algorithms in Git will really struggle with more structured files, so anyone who’s had to merge a big JSON file or a big merge multiple, you know, concurrent changes to a big JSON file or a big Excel file will have felt the pain that comes with that. And then the last issue that Git does not help with at all, is semantic merge conflicts, so I want to talk briefly about what a semantic merge conflict is. So imagine we have, in our code base, a function called log error, and when it was first created someone did a typo and no-one noticed and now in many places across our code base we have this log error function that’s spelt incorrectly. Alice is working on a feature, and while she’s working on a feature she finally decides, you know what, it’s so stupid that this thing is spelt wrong, I’m just going to fix it, so she renames the function, and of course she also renames all of the places where that log error function is called because she’s changed the name and so she has to update all of those call sites to use the right name. So that’s fine, and she checks that into her feature branch maybe, meanwhile on a separate branch Bob is creating a brand-new feature, totally unrelated to anything that Alice is working on, but in Bob’s new file that he’s creating he calls that log error message, that log error function, and in Bob’s world that log error function is still misspelt because Alice’s changes haven’t been integrated with Bob’s changed. So at this point both Alice’s reality and Bob’s reality are valid, when they go to merge these two changes together nothing is going to happen to tell Alice or Bob that they have this semantic conflict between their codes, so what’s going to happen is the merged version of these two changes is going to have a renamed function declaration, that Bob’s changes are still going to be calling the old function name, so Bob’s still going to be calling log error with those three Rs in the middle and that function no longer exists. Now if you’re working in a statically typed language, you’re in a happy place and at least like your code will now not compile and you’ll notice that there’s an issue, but if you’re perhaps working in a dynamic language you could never realise that you’ve actually got a serious bug in your program until this unlikely scenario that Bob wants to log happens. So this is a really quite scary concept, right, that we get this semantic merge conflict that not only is not sold by Git, but there’ll be absolutely no warning in some cases, you’ve just got to really hope that you’re really really good at your, like writing those automated tests to protect yourself from this scenario. So this is an example of a semantic conflict and it’s one of the reasons why working without integrating your code on a regular basis is a very treacherous undertaking. And this I think is the main reason, I think this highlights what, like the biggest danger with merge conflicts, which is developers are actually less likely to do these kind of small clean up tasks in their code, after someone has gone through a couple of these really painful mergers, they are less likely to want to do one of these broadly sweeping changes on a branch because they know that one of these cross-cutting changes is likely to result in a merge conflict that’s going to be painful to resolve. There’s this thing called the Boy Scout Rule which says try and leave this world a little better than you found it. And a software development kind of guru called Bob Martin, Uncle Bob, has a kind of, a saying, or says that we should be taking the same attitude towards our code bases, that if every developer left the code base a little cleaner and a little better factored than when they found it, then our code bases would be trending in a positive direction. And I think this is very true, I think this is a really important aspect of sustainable long-lived code bases, code bases that actually get better over time rather than slowly atrophying over time. Is teams following this principle of trying to clean up little issues when they find them. And merge conflicts, and infrequent integration act against that principle and it tends to add just enough friction that a developer might not fix the little problem they see, and over time the camp ground deteriorates until we’re all very sad about working in that code base and we decide we need to a bit re-write, so that’s why I think merge conflicts are such a bad thing. So let’s talk about what teams can do, what are some ideas that teams could use to avoid these scary merge conflicts? Well one thing that we could do is just do more frequent integration, so if Alice had made that wide sweeping change and immediately integrated to a shared controller branch and if Bob was also integrating on a frequent basis to this shared controller branch then there’s less likelihood of these big scary merge conflicts. So this is a totally legitimate approach to avoiding merge conflicts is frequent integration, so this is the fundamental principle of continuous integration which is perhaps slightly poorly named, but integrating your code to a shared branch once a day is the kind of working definition that most people have of continuous integration and it really does help you avoid these merge conflicts. But there’s a problem here, if Alice’s changes and Bob’s changes are frequently being integrated together, these changes are now coupled together, so Alice is working on feature A and Bob is working on feature B, let’s say perhaps on a Wednesday Bob is finished with feature B and he would like to release a version of the application with his new feature. Unfortunately Alice is still only halfway through implementing feature A and so there’s some half-finished implementation that is now on that controller branch because they’ve been continuously integrating into this shared branch. So what’s happened here fundamentally is Alice’s feature, Alice’s stream of work has become coupled with Bob’s stream of work and that means we now can’t release Bob’s feature independently of Alice’s feature and we have to wait for Alice to get done before Bob’s feature can go live. So this is a second big problem for teams that once we release on a regular basis, is the ability to release features independently. So how do teams solve this problem of wanting to release independently? Well a very common pattern is a release branch or a feature branch, so in this pattern when Alice is starting to work on her feature she creates a branch, when Bob’s starting to work on his feature he creates a branch, when Bob is ready to release he merges it into controller and at this point here where Bob has merged but Alice has not, we could release a new version of the app without Alice’s changes because they’re in a separate branch which hasn’t been integrated into controller. That’s great in terms of independent releases, but oh no, we’re back to the same problem we were just talking about, we now have these big potential merge conflicts because we are no longer practising continuous integration, if Bob’s feature takes perhaps four days to build and Alice’s feature is perhaps taking four days to build, then that’s multiple days’ worth of work which are going on on separate branches without any integration and so we’re back to the risk of big merge conflicts and a slowly degrading code base. And if you think about it, this shouldn’t be a surprise, because this pattern here of feature branches with Alice working on one branch and Bob working on another branch is actually exactly the same pattern as we saw earlier where Alice was working on her machine, Bob was working on his machine and they were integrating together, if you squint and look at these two diagrams you’ll see this is actually exactly the same sequence. And if you're using a distribution version control system like Git, these two ways of working will literally create the exact same commit graph, these are exactly equivalent. And so it should be no surprise that by using this like feature branches we’re back to the same challenges that we have when developers are working on their own workstations, and not integrating together and not pushing code into a shared branch on a very regular basis. So one pattern that teams sometimes use to try and resolve this issue of not integrating frequently is to do something called promiscuous merging, and promiscuous merging is the idea of everyone pulling from controller on a regular basis, so that even if you're on your own branch you’re still pulling in changes that are coming from other developers as they work on their features, and the idea here is to try and kind of accelerate, or to pull forward that integration work, so that you’re integrating lots of small changes rather than one big change once you’re done with your work. This can help a little bit with merge conflicts, but it is not going to solve them and let’s go talk through an example of why that’s the case. So here’s a team that’s practising feature branches, Sam has created a feature branch ‘cause he’s working on his feature, Jesse has created a feature branch ‘cause she’s working on her feature, and development goes on a pace. And Sam and Jesse are both practising promiscuous merging and so what they’re doing is as they’re working on their features, they are pulling from controller into their branch to try and kind of pull any changes that are arriving from other developers. And Jesse here has completed her work and she’s merged her changes back into controller, and everything’s fine, no conflict, so that’s a happy place to be. Unfortunately, as soon as Sam now does his next pull from controller he’s going to get this merge bomb that Jesse created, so you can see here we’re still stuck with the same problem, that but even though both developers are pulling from controller, because their changes aren’t being integrated into a shared controller branch on a regular basis, we’re still at risk of these large merge conflicts, or large merges kind of showing up out of the blue and surprising people with merge conflicts. And I talk about this because this is a solution that people often propose when they say, when they talk about long-lived feature branches, oh it’s fine, we’ll just pull on a regular basis from controller, that only works if you don’t have multiple parallel feature branches going on and that most teams that I’ve worked with they are practising some version, if they’re practising feature branches there’s usually parallel feature branches going on and so this promiscuous merging pattern is not a silver bullet, it might help in some cases, but there’s definitely still a risk of this kind of scenario. So one of the patterns that teams will sometimes use to try and avoid this issue is they’ll say, okay, well if we want to keep in sync with each other’s changes, and we don’t want these merge bumps to show up, maybe Alice should just be pulling directly from Bob’s branch as Bob is developing his feature. In this way Alice is seeing, is continuously integrating against Bob’s changes, even though Bob’s not pushing to controller. So this is a kind of, sometimes called cross-merging, and the idea here is to try and avoid those merge bombs by pulling features across branches, so that we’re doing something closer to continuous integration. And this will work in terms of avoiding a big merge bomb coming from Bob and landing on Alice’s branch when she tries to merge. But we’re now back to the same issue we had earlier where our releases are coupled, if Alice wants to release her changes, she cannot do so if Bob has half-finished changes, what you can see here is Bob’s feature branch has kind of become coupled into Alice’s feature branch and if Bob’s in these blue commits here includes some half finished work, Alice now has no way of releasing her feature until Bob’s feature is complete because she’s been pulling in Bob’s half finished work as she goes along to try and avoid those merge conflicts. So we’re back to our other problem of not being able to release our streams of work independently. So, what’s going on here, is this a kind of a fundamental conflict, it kind of feels like it is, like we’ve tried a bunch of different ways to resolve this and we keep on landing on either one side or the other, on the one hand the longer we go without integrating our changes, the more likely we are to encounter these costly, expensive merge conflicts. On the other hand when we do integrate our changes together we’re coupling the deployment of those changes together, so it seems like we can’t have our cake and eat it too, we can either have independent streams of work, but at the risk of merge conflicts, or we can avoid the merge conflicts by continuously integrating our streams of work, but now they’re no longer independent and we can’t release them independently. But what if there is some way that we could integrate these code changes without coupling together the feature release, and that’s what we’re going to talk about in a moment when we talk about feature flags, and the fundamental thing here is separating the concept of merging code with the concept of coupling releases together. So what is a feature flag? So a feature flag fundamentally is the ability to choose a code path at run time based on some configuration. So here’s a very very simplistic implementation of a feature flag, and it’s really really a very simple idea at its core, if a configuration says to do one thing, then send the code down one code path, if the feature flag says to do the other thing then go down another code path, very very simple idea but very very broadly applicable, and it's something that can really help us with this kind of tension that we’ve been talking about, using a technique called release flags, or release toggles. So here’s an example of using a release toggle, so let’s say we have a new feature, and but it’s not complete yet, we can configure this feature flag to say my new feature is disabled, and in that scenario, even though our feature is in the code base, and the implementation is kind of live in the code base, it’s living latent in the code base, that feature’s code is not ever being exercised. And this allows us to ship unfinished work as latent code without exposing it to our end users, and this is the fundamental idea of a release toggle is the ability to ship half finished work, without exposing it to our users. And this resolves our conflict, this is a really big benefit in that conflict that we talked about between merging our changes, and keeping our changes independently releasable, so let’s kind of talk a little bit about why. So the model that we looked at earlier on was this idea of frequent integration into a shared controller branch, so this is the picture of continuous integration essentially. And when we were talking about it earlier, we discussed how there’s a fundamental challenge here where because Alice’s changes and Bob’s changes are landing on the same branch we can’t deploy just Bob’s work or just Alice’s work, and that’s still true, we can’t just deploy Alice’s code or Bob’s code, but what we can do with something like a release toggle is we can deploy both Alice’s unfinished work and Bob’s completed work, but only expose Bob’s completed work to our end users. And this is how we resolve our conflict, by giving ourselves tools that allow us to merge changes including half-finished changes into a common repository and therefore practice continuous integration, but not expose those changes, not release those changes to end users, we’re able to resolve our conflict, and this is what in continuous delivery terms is referred to as decoupling, deployment from release, so we’re able to deploy the code behind a feature, even if that code is half finished, without releasing that feature to our end users by using a feature flag. And this idea of using feature flags in this way is not new, some of the pioneers of this technique were companies like Flickr and Etsy in the early 2000s, and they were talking about it for a long time and it’s become a very broadly accepted technique that’s used by a lot of kind of high performing technology organisations, so Github and Facebook are two examples of organisations that have really embraced not just the idea of feature flag for release, but feature flags in general as a way of managing how their code base, how the features in their code base are exposed to their end users. And as I kind of hinted at there, feature flags is actually a much broader technique, or is a technique that has a much more broadly applicable use cases than just managing this release issue we were talking about. So those organisations like Github and Facebook and Flickr and Etsy use feature flags for a lot of different things, and a lot of high performing technology organisations do this. So we can use feature flags for canary releasing, where we decide to only expose a feature to maybe 1% of our users and watch their behaviour and validate that the feature is not causing any issues and then release it to the general population. We can use feature flags as operational toggles, or kill switches, so we can protect a feature, maybe a feature that’s potentially costly and put it behind a feature flag and then in the case that our servers are under high load, or some back-end system is having a bad day, we can turn off the feature that uses that back-end system. Very common use of feature flags is AB testing, so again the fundamental idea here is the ability to send someone down one code path or other code path and that’s conceptually exactly what an AB test is, is showing one treatment to one cohort of users and another treatment to another cohort. And if you kind of think more broadly about this capability of showing some things to some users and other things to other users, you can start using feature flags for things like admin only features, or premium features that you only expose to your paying customers. So what I see a lot with teams when I’m working with them is they often start using these feature flags for something like managing release, the kind of the continuous delivery idea of decoupling release, or deployment from feature release, but once that capability is in place you can really broaden the uses and start using it for all of these different capabilities. And my advice in general with getting started with this stuff is to start small, so if feature flagging is something that is new to you as an organisation and you want to try it out, it’s really really easy to try this out, this is how I normally… a variant of this is this example, is how I normally introduce feature flagging to a new team or a new code base that hasn’t worked with it before, literally a hard coded Boolean flag that protects a feature from release, and we set that flag to false, we’re able to merge this new feature into our controller branch and release it to production, confident that this feature will not be turned on and then once this feature is ready to go live we make like a one line change to flip that Boolean to true, and merge that one line change into controller, and then the next time we release, that feature is now live in production. This is a really simple technique to get started with, I don’t recommend keeping this simplistic technique as you start to scale out your usage of feature flags, it’s important to use a framework and I’ve seen a lot of teams try and roll their own and I don’t recommend doing that, but I do recommend starting in a very lightweight way. So feature flags are great, but they’re definitely not a silver bullet, there are some drawbacks to using this technique as with all things in software engineering. So feature flags lend, tend to lead to messier code, a flagged system is on average a little bit messier, or at least the implementation of the flags adds some messiness to the code base because you have to kind of switch between these two ideas, rather than using branches to manage that you now need to manage that in code, so there is going to be a little bit of overhead in terms of the code base. There’s some testing challenges here, if something can just be turned off or on at runtime you’re testers need to be aware of that and need to be testing both scenarios, but this testing challenge is not as big as professional testers sometimes suspect it will be when they first get started. Not all changes lend themselves to feature flagging so broad refactoring work is something that can be quite tricky to pull of using a feature flag, and my recommendation to a lot of teams, particularly if they’re new to feature flags is if you’ve got some broad cross, lightweight but broad change, like that renaming of a function that we talked about earlier on, just do that on controller, just do it as a small change and check it in directly to controller, don’t try and use feature flagging to protect yourself because you’ll spend more time setting up the feature flags than you will doing the clean-up. The biggest advice I have to teams is make a plan to clean up those feature flags, a lot of teams when they get started with this technique will get pretty excited at how powerful it is and start putting feature flags all over the place, and after a few months there’s suddenly 20 or 30 tests, we’re not even quite sure which flags are still in use anymore, so you do need to be kind of explicit and intentional in making a plan to clean up those flags once something is actually released into production, for a release flag you should have a plan to retire both the flag and the logic that’s protecting that feature partly because it helps you keep your code base cleaner. And I’ve already touched on this a little bit, but kind of the idea of using feature flags, or feature branches is in my opinion a false dichotomy, both have their place in a software development life cycle, so I believe that even teams that are using feature flags are still going to use branches for some things and conceptually everyone is always using a branch in that whenever you’ve got a change that’s on your local machine that hasn’t been pushed to a shared controller, that’s conceptually a branch. So every team is using branches, unless you’re doing all of your coding by directly editing files in the Github UI which I suspect you’re not, you’re using some kind of concept of branch. So there isn’t this kind of false dichotomy between branches and flags, you can use both and be successful with both. So in summary, I think what I’ve been trying to describe today is this tension between avoiding merge conflicts, by practising continuous integration, versus keeping your features independently releasable and I’d argue that this is a bigger issue than some teams realise. Merge conflicts are kind of, I described them as kind of like an iceberg, the real impact of managing merge conflicts is really a lot bigger than a lot of teams realise, but is non-obvious and the big impact that I’m talking about is the impact upon the overall quality of your code base because of that, that Boy Scout Rule, and not being able to follow that Boy Scout Rule because the cost of merge conflicts is just too intimidating. Feature flags are one rule to help resolve this tension, between continuous integration and decoupled release, and get us past the idea that we can’t do both at the same time. And of course like all things in software development, feature flags are not a silver bullet, but they are easy to try, and so as I showed a couple of slides ago, it’s very very lightweight, you can use a very very lightweight approach to get started with this technique, try it out with your team if you’re not already, and take it from there and if you do start using it more then get serious about it, but to get started it’s pretty straightforward. And that concludes my presentation, I’d love to take questions from participants if they have any? Thanks so much Pete, so if anybody has any questions now you can go ahead and do them in the Q&A section, we still have some time for Q&A, I do actually have one question myself Pete, I’d love if you could answer, you had mentioned that you don’t recommend for people to roll their own solution to deal with feature flags, and I’d love if you could provide some insight into why you don’t think it’s a good idea to create such a solution yourself, in-house, I mean it seems kind of simple, it’s just a simple flag on off, what are some of the things I guess down the line that kind of got you as an unexpected issues, if somebody were to try to go down and create their own solution? Yeah, part of the reason why I say it is because I hear a lot from teams, it seems kind of simple, exactly that sentence that you had, and I think it’s a legitimate thing to be kind of thinking when you get started with this, is like, oh it’s really straightforward, you’re just choosing between two options, and maybe some configuration there. I think that the thing that people don’t realise when they get started with this technique is that it becomes so broadly applicable, and as it becomes more broadly applicable, as you move from just using it for managing release, for using it for maybe AB testing or for canary releasing, that the use cases change and so the capabilities of whatever framework or tool that you’re building or using, need to get a lot broader. And what starts off as feeling like a pretty simple system ends up needing a lot more capabilities, and in my experience teams or organisations that are using a hand-rolled solution, that hand-rolled solution is not really fit for the broader use cases, it is often created by a developer team who is focused on just release toggles, release flags, and starts to get used in a much broader way, but there isn’t really a product manager or someone who’s like thinking about this tool and thinking about the use cases, it’s generally like devs who’s built it for their uses and it’s not really fit for the broader purpose where the people using this tool might be product managers, or operations folks, and in those scenarios they want to be integrating with analytics tools, they want to be getting feedback, like good logging and metrics as to where these feature flags are on and when they’re off, like there’s a lot of hidden requirements in a feature flagging system that people don’t realise when they first get started with them. Excellent point. So also I see another question here, regarding, okay, so the question is, if you can address regarding the mobile space versus the non-mobile space, I know you didn’t really get into it during the presentation, but it’s an excellent question, I do think there’s some specific points maybe to give there, just call out, or just kind of wondering your views on how is this applicable to a web based, where making changes are kind of instant, versus a mobile where you have that extra layer of, well this is now versions which are live and not live, etc, etc. Yeah, so I think interesting, I think that feature flagging as a technique is probably more applicable in the mobile space, so there’s a tension here, so I think feature flagging as a technique a lot more applicable in mobile because you have less control over your release, so if you’ve got a feature that you want to release to your users, and you’re just managing that by releasing it through the app store or the play store or whatever, it’s actually pretty hard to manage that release without going through the app store for everything. If you’re not using something like feature flags, every time you want to turn a feature on or maybe even roll back a feature that’s gone sideways, you’re kind of, you’re beholden to the ability to move a release through that app store process, and if you think about that compared to the web world where, you know, a high performing dev opsy kind of team, can push a change out to production and then pull that change back in the, you know, the space of a few minutes, that’s a very different world than the mobile world. So I think actually feature flagging is probably more important in the world of mobile where you’re not able to just push changes and essentially push out a change to 100% of our population within a few minutes, and then, you know, worst case scenario, roll back that change in the course of a few minutes, with mobile you’ve got a lot less control and feature flags help bring back some of that control I would say. Great. Okay, so I’m not seeing any additional questions, thanks again for all the participants and thanks so much to Pete for really providing with this excellent webinar, I have to say myself, I learned a lot myself. As I said beforehand, this is being recorded and we will be sending out the recording to other participants, if you have any questions feel free also reach out directly to CloudBees Feature Management at support.rollout.io, or go to rollout.io if you want to also check out our product which happens to be feature flagging, so again thanks everyone and have a great day.