This podcast features people doing interesting work in the world of Cloud. What is the state of the technical world? Let’s first focus on the up or down, on or off function of feature flags.
Today, we’re talking to Heidi Waterhouse, a technical writer turned Developer Advocate at LaunchDarkly, which is a feature flag service - a way to wrap a snippet of code around your feature and make it into an instrument to turn on or off. It lets you turn things on and off in your codebase quickly without having to do several commits. LaunchDarkly provides a way to manage your features at scale with a usable interface and API.
Some of the highlights of the show include:
A feature flag allows you to hide items before you want them to go live on your Website. You hide it behind a feature flag, doing all the work ahead of time.
You can test at scale to gain authentic data. However, no matter how good your integration tests are, there’s always wobbles to watch for in the system.
With implementation, there are a few paths that can work, such as the massive reorganization path.
LaunchDarkly thinks in the Cloud as the surface because it mostly works with people who are doing Web-based delivery of features.
Major companies, like Google and Facebook, offer services similar to feature flags for their own development.
Companies use feature flags on the front-end and other purposes.
Do not focus on documentation. Create a customized experience.
Feature flags effectively manage and minimize risk.
LaunchDarkly holds monthly meet-ups called, Test and Production. People share their use case regarding continuous integration, continuous deployment, DevOps, etc.
Resources Mentioned in this Episode:
Corey: Hello and welcome to Screaming in the Cloud. I’m Corey Quinn. Today, I’m joined by Heidi Waterhouse of LaunchDarkly where she’s currently a Developer Advocate. Welcome to the show, Heidi.
Heidi: Thanks. I’m glad to be here.
Corey: Your back story is fascinating to me. You were a technical writer for a couple of decades. At one point, I think you mentioned to me that you used to write Patch Tuesday Release Notes, which I think is the definition of a thankless job to some extent. In your most recent role, you’ve pivoted into developer advocacy. How did you land there?
Heidi: It turns out that if you take to the world giving talks and do it 28 years, people begin to think that you might know what you’re doing and invite you to come give talks. Instead of being my marketing side hustle as a technical writer, writing blog posts and giving technical talks is my full time job, which is like a dream come true.
Corey: Was that a role that you found? Or did you have to have it built when you started talking to them?
Heidi: I applied to be a technical writer and they asked me to come in and interviewed to be the developer advocate. I’d never done that role of really. I knew a bunch of dev advocates and dev rel people, but I had never thought of myself that way. But LaunchDarkly asked me to come in and give them a 15 minute presentation on feature flags to see what I could do with that. I ended up giving them a 20 minute presentation on how you could use feature flags to do documentation better.
Corey: That sounds like a fun story, but let’s rewind a little bit first. When I first met you, I knew you as the person who was redoing a bunch of live tweeting at a conference entirely from an iPad which I found fascinating. I’ve tried to live tweet since then. Usually I just inflict a bunch of nonsense and what I consider to be witty on my audience. But I also took away the idea of doing an entire conference trick presentations and all from an iPad. That is something that turned into a surprisingly interesting area.
Heidi: Isn’t that great? I find it so fascinating that the iPad really is a powerful enough computer that that’s all you need to travel with. I’m also so glad I have that method because it turns out that the USB-C connection on the new MacBooks is not 100% reliable for HDMI and can fail not just frustratingly, but like catastrophically.
This happened to me, I’d just gotten a brand new laptop for this new job, I’m doing my first conference as a Dev Advocate. I plugged in my laptop and it got this weird jaggy sideways traffic and failed to project. I’m sitting there thinking, “Good, good. This is good, this is a brand new talk at a conference where I don’t know anyone for my brand new job. I have just bricked my computer.”
Corey: I feel like the only appropriate response there is, “There’s the metaphor.” Just wait for the applause.
Heidi: Exactly. I actually ended up giving a 35 minute talk purely from memory and adrenaline, and fear sweat. But after that I was like, “MacBook, you are fired for old time. I’m going back to my iPad.”
Corey: Back to topic a little bit, what is a LaunchDarkly?
Heidi: LaunchDarkly is feature flag as a service. It turns out that a lot of people want to be able to turn things on and off in their codebase really quickly without having to do a lot of commits but they have a lot of trouble tracking it when you get over, let’s say, a dozen flags. What LaunchDarkly is providing is a way to manage your features at scale with a usable interface and also an API.
Corey: For those who don’t have a background in software development, namely me, what is a feature flag?
Heidi: Imagine if you are creating a piece of music and you know those big sample boards that the DJs use, and they use in theaters? What we wanna do is instrument every piece of code, every sort of individual sound of code and give it a knob that you can turn up or down, or on or off to be able to blend it with others.
Feature flags are a way to wrap a little snippet of code around your feature and make that into an instrument to turn it on and off.
Corey: Wonderful. The advantage of doing this as opposed to a fresh codedeploy would be…
Heidi: Speed and risk. I am old enough that I remember when code and products came on CDs, actually floppy discs, but we won’t talk about that, Lotus 123. But, it used to be a big deal to push out a deployment and then that was all you’ve got. What feature flags do is make it possible for you to push out a deployment with things hidden, we call it launching darkly.
Heidi: Ahh… See? Then, when you’re ready, you can turn it on. Imagine you wanna do a big website refresh in September. You want it to have all the things that you’re going to need for Black Friday on it. You don’t wanna show the Black Friday stuff yet so you hide it behind the feature flag, doing all the work ahead of time. Then, on Thursday at midnight, you can go ahead and turn it all on instantly without the risk of pushing untested code into your production.
Corey: That makes a stunning amount of sense. Wonderful.
Heidi: Yes. We’re all about avoiding risk, I think this is our motto this year, eliminate risk. [inaudible 00:15:50] because you can’t eliminate risk, but you can make it much less risky.
Corey: As far as doing this in production and minimizing risk, pushing it further down the deployment chain, how does this start to impact larger scale environments?
Heidi: I think one of the exciting things about cloud and scale is that you’re doing things across servers and time zones and areas of control. You don’t know exactly what’s going to happen in production.
There’s no way to test a massive distributed system except in production. But if you’re doing that, you would like not to be showing everybody your testing. Imagine you have a massive enterprise grade system, you want to know if this new feature, let’s say a toolbar, is gonna work right. The first thing you do is you deploy it with nobody able to see it. Then, you turn it on just you and your team can see it.
Then you turn it on so that only people in your company can see it based on IP. Then you turn it on for 10% of your customers, then you scale up the percentage of customers who can see it. This whole time you’re doing sampling in metrics and analysis to make sure that it’s not causing edge case problems or somehow causing your system to fail, conflate, or fall over. Testing isn’t binary, it’s a degree.
Corey: How does this apply to other methods of testing large scale distributed software in production or otherwise?
Heidi: One of the ways that we’ve tested large scale software before is develop bunch of big data through. The problem with big data is that it’s big and frequently sanitary. It’s sort of like trying to test whether an antiseptic works but only in a sterile environment. You’re just not gonna find out because you’re feeding it data that isn’t contaminated and grody. Being able to test production means that you’re going to get authentic data.
Another thing that’s important to remember when you’re testing at scale is that no matter how good your integration tests are, there’s always going to be some wobble in the system. I think about, say, Autodesk. I was working for a company doing cloud integrations stuff a couple employers ago.
Autodesk would spin up thousands, thousands of servers, and then spin them down again really rapidly because they were using them for scaling user 3D printing stuff. If you couldn’t handle the fact that you’re spinning out 2,000 servers all at once, it was a real problem. But it’s hard to get the testing capacity to do that.
Corey: There are some technologies that you tend to see that make an awful lot of sense for certain use cases, generally at software companies startups based in San Francisco. If you try and take that model to something like I don’t know, we control all of the ATMs in North America. A lot of the paradigms that work when your Twitter for pets start to fall down at bank of the world. For example, when a dog can’t tweet for two minutes, that tends to be a different failure domain than the ATM is now spinning out wrong balance information to a subset of users.
Heidi: [inaudible 00:19:55].
Jason: Oh, yeah, exactly. That depends entirely on what level of happy or sad you want your users to be. But the question I’m getting at here is are feature flags something that maps reasonably to most workloads? Or is this something that is better suited for stuff that errors won’t really leave nasty marks and bruises?
Heidi: We certainly think that it is enterprise grade. I cannot talk about a lot of our customers. I have to go through and see how we can talk about. But I will say that [Lassie and Algeria 00:20:38] are using us, which I think is a pretty significant use case. I bet a lot of your listeners have a confluent somewhere.
We think that it is exactly because you can do feature flagging that it makes it safer to do with enterprise grade. Because if you would have a button, a knob that can turn on a feature, you also have a knob that can turn it off instantly. We think it’s about 200 milliseconds from the time you hit what we call the Kill Switch to the time servers stop delivering that broken feature. Imagine the power to say, “Wow. We’re spinning out 20s from our ATM. Let’s roll that back right now.”
Jason: That’s compelling, which I guess leads into the next big question here. How do you get there from where many shops are today? It’s easy to implement something like this relatively, easy to implement this in something completely greenfield where you’re not necessarily going to be having to retro finite the things. But in practice, we rarely get to work with environments like that. Here’s a 20 year old PHP app, time to go ahead and re-architect it to take advantage of something like feature flags. What does that journey look like?
Heidi: People can take a couple of different paths. There is a massive reorganization path, which is not ideal. Nobody enjoys it. It burns a ton of developer time and your value ad is very small right away. But if you’re doing something like using feature flags to do price tiering where you’re showing people the same page but with different features depending on whether they’re paying you or not, it’s what you have to do.
Most of the time, what we recommend is that people start using it for new features. As your new coding practice, as your best practice, whenever you create a feature, instead of necessarily making it a branch, you just wrap it in the feature flag and add some if this is on and this is off defaults. Then, go ahead and write your feature. You know that it’s hidden behind the magical feature flying curtain until you’re ready to turn it on.
We say like this incremental just new features or just new code bases, it’s still gonna help you a ton. You’re still gonna see a lot of benefit from it without disrupting and randomizing a working cold fusion environment.
Jason: Which makes sense. The next logical evolution of that question, given that this is Screaming in the Cloud, how does rolling something like feature flags out change when you’re doing it in a cloud based context, instead of the traditional on-premise style deployment, or does it?
Heidi: Interestingly, when I say we’re feature flagged as a service, we are like cloud first organization. We are tightly linked with the CDN, we’re all about the distributed network. It’s not like we’re having your request come all the way back to our servers to be evaluated or your servers to be evaluated. We’re evaluating on the edge which gives us a lot of power, but it also means that it was a little hard for us to move into on-prem we do have some on-prem installations, but it’s less powerful because we don’t have that edge service. The edge of the cloud is giant, and your on-prem. It’s sharp.
The on-prem servers are small and localized. We can do it, but it’s sort of not how we think. We’re very much thinking in the cloud as the surface, because mostly we’re working with people who are doing web based delivery of features. We’re working with companies that are giant web pages, or retailers, or people who really need to have always on control of their features.
Jason: Which definitely does tend to make sense as you look at the current [graph 00:25:05] of companies and the historical migration pattern that we’re seeing as companies move out of on-prem and into cloud.
A question I have for you though is whenever I hear someone say something as a service, in this case, a feature flags as a service, my immediate instinctive major reaction is oh, okay. How long is it until AWS comes out with the confusingly named offering around this that tries to eat your lunch more or less, but somehow manages to have 15 different billing dimensions so no one knows what it’s going to cost by the end of the day.
How aligned is what you’re doing with how these large cloud providers start to see either an offering that they can build out for their customers or how tightly integrated to a particular vendor’s offering it becomes?
Heidi: Interestingly, there is a level of customer or a level of organization that’s not interested in buying us because they’re doing it themselves. I think it’s possible AWS is not offering this to customers. But I do think they’re using something very much like feature flags for their own internal development. I know Google is, I know Facebook is. They’re operating at such a giant scale, they have entire teams that are already doing this so that they can serve you the 15 degrees of confusing, badly worded pricing. Because they’re serving you your 15 confusing things, but they’re serving someone else 15 other different confusing things. The only way that could be happening is if they’re doing feature flags.
Jason: Gotcha. This is A/B testing taken to extreme level.
Heidi: Yes. This is A/B testing but I like to call it on beyond A/B testing, on beyond Z, because A/B testing is just one of the ways that you could be manipulating what people are experiencing.
Jason: Wonderful. Breaking the level of alphabets to name that across that any dimensions we’ve run a reasonable risk of inadvertently summoning a demon. As a general rule, a feature flag is considered to be a frontend technology, or is this something that starts to work its way throughout the rest of the stack?
Heidi: It works through the whole stack. Our customers are using it for frontend page delivery, but also pricing tiers, and also just say for deployments. If you were doing a significant backend revision like I was reading about how Slack upgraded their database backend by basically putting a new database on top of their old database, then switched over slowly almost like a blue green, but they weren’t identical. That was done using feature flags so that they could slowly shift traffic from one to the other without having anything irrevocable.
Jason: Fascinating. I guess this ties into my next question rather neatly which is you mentioned at the beginning that you’ve done some documentation work, the feature flags, you’ve mentioned that in your interview. What sort of wacky things can you do with feature flags and aren’t continuous integration or delivery based?
Heidi: You can use it to do white labeling. Imagine if instead of having 15 different custom websites that are slightly different when you have to maintain, you have one website and you’re just calling the customized look and feel, the CSS, out of using a feature flag. I think you could use it for some really interesting localization and market segmentation.
If you wanted to target all the people in Germany who have previously expressed an interest in Hamburg United, you will be able to say deliver that to them. I’m working on a blog post right now, my CTO is a little dubious about this idea, but I think you could use feature flags to do some really interesting localization stuff to pull out different files and do your localizing on the fly using flags instead of having to do browser based dependencies.
Jason: Fascinating. Getting back to what you said originally about using feature flags for documentation, how does that work?
Heidi: I don’t think you should have to read documentation for anything you don’t own. Every feature should have its documentation tied to it committed as code. Then, if you don’t have the extreme module, you’ll never see the documentation for the extreme module. That just won’t appear because we’ll turn that flag off. Being able to synchronize exactly the code that you’re using, that exactly the documentation you get would really cut down on the mount of documentation people want to read, because if 20 years of technical writing taught me anything, it’s that nobody wants to be reading documentation.
Jason: It seems sometimes that no one wants to read it all to the point where RTFM has become almost the trope in our space. “Oh, how do I just read the manual?” “Yes. I tried. It’s an encyclopedia. Could you be slightly more specific, please?”
Heidi: I used to call IBM to get the helpdesk to give me the page that I needed to be reading. Your indexing is really bad, people. I think it helps if we remember that everyone who’s reading documentation is already a little bit angry because they couldn’t figure it out. All documentation is essentially a failure of user interface at some level.
Jason: I like the idea that everyone who’s reading documentation is already slightly upset. They’ve had a negative experience. Other than removing parts of the documentation that don’t apply to them, with your background on technical writing, how whilst can you see reasonable ways to address that? I know that I spend more time swearing into various cloud provider documentation bundles than I’d care to admit publicly.
Heidi: I think that you should be able to have a customized experience. That would mean that not only it is tough that you’re not using hidden from you, but also, you would get a synopsis of things that you already know.
For instance, if you would log in as Corey Quinn, expert AWS person, nobody’s going to explain to you how the certificates work because you’ve done that enough times. You know that because they like a consolidated summary. Then we’ll go into how certificates don’t work in these particular circumstances. That part would be expanded. I think you’d be able to do level setting to answer these three questions then will give you a more accurate representation of what you’re actually trying to get an answer for.
Jason: You flatter me. It tells me my marketing has worked. I have no idea how most of the certificates stuff works. I smile, shrug, hand wave over it, and hope no one process me too closely on it.
Heidi: My website was down for 10 days because I can’t figure out how to renew certificates.
Jason: Welcome to the eternal joy of anything involving infrastructure.
Jason: Something you mentioned earlier was using feature flags to effectively manage and minimize risk. How does that wind up progressing as companies start to embrace the idea of feature flags?
Heidi: The thing that we’re trying to do is accept that there is always risk in the world. What causes disaster is not one failure, it is a multiplication of failures. This goes wrong and this goes wrong. It’s not just that the o-ring got too cold, it’s the PowerPoint made it difficult for people to explain to their bosses that the o-ring was too cold than the space shuttle might blow up.
All of the failure analysis you’ve ever read involves a lot of different factors. What we’re trying to do with feature flagging and continuous integration, and continuous deployment is break these monolithic releases up into tiny bite sized chunks that we can make go forward or go back. If you think about it, it’s less like putting all of your money on one color of the roulette table and more like putting it all over the roulette table. Your odds of something catastrophic happening are much lower.
Jason: Gotcha. Effectively, what you’re doing is reducing failure domain by having fewer deltas at any given time?
Heidi: Exactly. If we say I’m only changing this one particular thing, then you can track what’s happening with that feature. If something goes wrong you have a way to back it out or I like to call it bullet forward. If you have it deployed, it has 20 features at it. That’s a really huge deploy in the CICD world but has 20 features in it.
One of them goes wrong, the old style was to panic and push out the old version, effectively rolling back all 20 features. Feature flag style is you go, “Oh, feature X is not working. We’re gonna turn that off. All the rest of the features are fine. They’re gonna roll forward, we’re gonna go find out what happened with feature X.”
Jason: Got you. It seems almost counterintuitive in some ways where you have to deploy, things are broken. It’s a terrifying moment. The instinct is since that hurts so badly, companies genuinely wanna do fewer releases as opposed to more releases that have smaller change sets.
Heidi: Right. I think this is why people have trouble with weightlifting. It turns out very few of us can actually lift 300 pounds, but a lot of us could lift 30 pounds 10 times or 3 pounds 100 times. I want more companies to be lifting 3 pounds 100 times when they release instead of trying to lift this one massive 300 pounds.
Jason: I like the metaphor quite a bit.
Heidi: You’re gonna hurt yourself if you try and lift that.
Jason: Yes. I don’t even try to lift the 30 pounds 10 times. Good Lord. Yeah. That’s not really my skill set these days.
Heidi: Yeah. How old is your baby?
Jason: Eight months at this point.
Heidi: Not quite to 30 pounds yet.
Jason: Not quite. Hopefully we’ll get there for a while but we’ll see. Is there anything else that you’d like to talk about or mention that you’d like people to take a look at, participate in, throw fire and brimstone upon, etc.?
Heidi: I’ve got a couple things. If you’re in the Bay area, my company does a monthly meetup called Testing Production. We talk more about these things and we have people come in and talk to us about how they’re testing in production and what their use case is for continuous integration, continuous deployment, DevOps, that sort of stuff. It’s super fun.
I would like people, if they have time to write me in the stories about Trunk-Based Development versus Branch-Based Development because I think it’s a philosophical split that we haven’t explored a lot in the DevOps industry yet.
Jason: The way that I’ve always found that worked very well to get people’s feedback is to stick out a strong opinion on one side of an issue or the other, then just wait. You never need to give them addresses. They will come back and give it to you themselves without prompting.
Heidi: It’s true. Let’s just say Trunk-Based Development is probably a better way to go for your enterprise organization than Branch-Based Development.
Jason: Thank you very much for joining me today, Heidi. I’m going to disagree with you vehemently as soon as we stop recording. My name is Corey Quinn. This has been Screaming in the Cloud. Thank you for joining me.
Heidi: Thank you. Have fun.