Have you ever had high expectations about a new software product? Did you think it was going to be spectacular? Instead, did it become less about solving a problem for you and more about reaching a bunch of billable consultants? The dynamics of open source communities and the Cloud platform can make or break software products.
Today, we’re talking to Andrew Clay Shafer, who was a notable voice during the days of OpenStack. He had high hopes for OpenStack, which was an effort to bring a democratized solution of Cloud computing to anyone’s data center. He describes the importance of understanding the challenges associated with open source projects in order for them to be successful.
Some of the highlights of the show include:
Open source is not a business model; capture value for customers, or they’ll go with a different solution
Openness/Closure: Every open source project has its own community dynamics
Losing sight of level of expertise for profitability and easy path to useage
Whether to become a product or service company - difficult to be both effectively or go from being one to the other; build partner relationship, focus, and say “no”
Lack of awareness about AWS Outposts admitting public Cloud is no longer a viable business model
Amazon relentlessly focuses on what its customers want and tries to keep promises about what it can and can’t do
Cloud Native: Not where you run, but how you run; confining variables
Self-fulfilling prophecy to under deliver when you make the bad decision to under source IT across the board
Cloud Native, DevOps, SRE: Buzzwords that equal one thing and work together
Dilemma of not building everything and buying some things, but you can’t buy everything; humans like to shop and go with the easiest option
Full Episode Transcript:
Corey: This week’s episode is sponsored by Datadog. Datadog is a monitoring and analytics platform that integrates with more than 250 different technologies, including AWS, Kubernetes, Lambda, and Slack. They do it all: visualisations, APM, and distributed tracing. Datadog unites metrics, traces, and logs all into one platform so that you and your team can get full visibility into your infrastructure and applications. With a rich dashboard, algorithmic alerts, and collaboration tools Datadog can help your team to learn to troubleshoot and optimize modern applications. If you give it a try, they’ll send you a free t-shirt. I’ve got to say I love mine. It’s comfortable and my toddler points at it and yells, “Dog!” every time that I wear it. It’s endearing when she does it and I’ve been told I need a leave their booth in re:Invent when I do it. To get yours, go to screaminginthecloud.com/datadog. Thanks to Datadog for their support of this podcast.
Welcome to Screaming In The Cloud, I’m Corey Quinn. I’m joined today by Andrew Clay Shafer, who is himself famous for screaming at clouds. Having him on the show was really just a matter of time. Andrew, welcome to the show.
Andrew: Do we scream now?
Corey: We absolutely can. It’s fun. There are a couple of companies I wound up encountering at re:Invent, who wound up casting their company names entirely in capital letters, so at some point I feel like there’s going to be a gag or every time I mention their company name I actually scream, and then I realize exactly how unpleasant that’s going to be for someone listening to this on their commute. If you wind up just screaming mid-sentence out of the blue and someone rams a bridge abutment, you kind of feel bad for that.
Andrew: You can also level the volumes there.
Corey: That sounds like something smart people might do.
Andrew: Post production.
Corey: I’m told that’s a thing. You have an interesting history. Historically, you were a notable voice in the days of OpenStack. Is that something we can talk a little bit about or is that still too painful of a memory?
Andrew: I had such hope for OpenStack at one point. What do you want to talk about? I think OpenStack was an effort to do something interesting, to try to bring this democratized solution of cloud computing to anyone’s datacenter. Unfortunately, (1) that’s a very, very hard problem, and (2) the dynamics of the project just kind of got turned over to essentially marketing way too early in its evolution. The other byproduct of that was it sucked the oxygen out of a number of other projects that probably deserved a little more oxygen than they ended up getting. I don’t know. What specific thing to talk about OpenStack?
Corey: I loved what it was. I thought there was something spectacular going on with it. I started playing with Eucalyptus, which is a java-based competitor that was nowhere near as open, and that worked a bit better. The problem is, every time I try to get into it, it felt like there were 800 ways to do it and everything was built around a construct that seemed to be less about solving a problem the customer had, and more about building work for an army of 500 billable consultants for the next 18 months. That meant that when I was trying to stand things up in my test lab, it didn’t take long for me to go from, “This looks interesting,” to, “Nope, nope, nope,” and that was the end of it.
Andrew: There’s a lot to unpack. One is there’s these dynamics of open source communities, which might also be interesting to talk about, but you mentioned Eucalyptus. Eucalyptus essentially had the mantle and the right to be this open source cloud platform that everyone sort of made OpenStack or centered on OpenStack.
But they didn’t understand a few things. One is the real distributed systems problems you need to solve to do this the right way. Eucalyptus was, from the most part, a java monolith and that was not the problem. I mean, it was a problem as you start to scale and try to build these things but when they were in the early phase and interacting with some of these organizations that were interested in having that BS solution that were running up to the scalability that that solution as a java monolith could handle, then they did a very poor job or they didn’t understand the dynamics of open source, such that they essentially created OpenStack by not fostering that community.
I know a number of cases and we won’t name names but there were people that were trying to contribute to Eucalyptus that were prevented from doing so because the Eucalyptus mindset was, “Oh, those are features that will be in our commercial product,” and that ended up essentially creating the gap that OpenStack stepped into.
Corey: That seems very similar in almost a retelling of what we’re starting to see now in a different form, in that, you’re seeing a number of open source projects that are having a bit of an existential challenge as far as they build something, they make it available for everyone, and their business model is to either sell an enterprise version or support around the thing that they released. And then large cloud providers like Amazon come in and build an entire package offering around that and wind up effectively selling the thing that a bunch of people have contributed to, and making money out of it, and not giving anything back. That’s completed permitted by the license.
We’re starting to see things like relicensing and the existential question of how am I going to make money off of this when large vendors will take what I built, repackage it as a service, charge money for it, and never contribute a single thing back, be it code, be it money, be it support, any of it. Where do you land on that?
Andrew: Well, for the most part, I’ve made my living for the last decade off of open source in some form. I’ve been on both sides of this and I think you just have to understand that open source is not a business model and that it doesn’t matter what the quality of the open source is or what you’re creating, if you don’t solve a way to capture value in a way that it’s conducive to your customers getting it, then they’re going to get a solution from someone else.
There’s so many nuance things that are happening and people, especially once they’ve gone and taken a lot of investment with the expectation of an outcome for their investors, then you end up really, really struggling to find that balance. I think it’s actually worth backing up. When someone says ‘open source,’ that’s not a thing. That’s not a single thing that you can define. Every open source project has its own community dynamics, has its own relationship with those licenses. Yeah, they might have the same text but their openness in regard to what the code is only one part of the story, like what’s their openness with regard to their community? What’s their openness with regard to the way that they accept contributions or that they help those users?
In the last six months, we’ve seen a number of these things relicensed like you said. We’ve also seen an interesting dynamic in the Closure community. I don’t know if you’re following that but there is a post from the main author of Closure about how essentially it’s not his responsibility to care about the community. At least that’s my short version of interpreting what he said. I don’t know. There’s so many threads we could pull apart here. Maybe it’s worth refocusing on one of them.
Corey: I think you’re right. It’s also one of those areas that I am acutely aware of, that if I put a foot wrong, I will wind up with 8000 people in my inbox and four at my house. I’m not in this to slaughter anyone’s sacred animal, regardless of what that tends to be. I guess my concern is it’s challenging.
We see this with Docker, for example, where they released an amazing, transformative technology. They were never quite clear on what their story was going to be around monetization, and it turns out VCs generally want to see a profit story come out of this. I never saw a container story that involved ‘Step 7: Give money to Docker,’ in any time I wound up building things around that particular technology.
When you take a look at things that have relicensed, for example, aspects of ReTiS have done that recently. I believe MongoDB did, but don’t quote me on that one. I haven’t been following this with wrapped attention probably deserves and I understand where they’re coming from.
Take the ReTiS story. They watched their service from ReTiS Labs, the product from ReTiS Labs be turned into an Elastic app service. Click button receive a relative baseline ReTiS environment that just works. Suddenly, that need for that level of expertise in many shops is either not there as strongly or is perceived not to be there as strongly.
That’s a scary thing but I think you nailed it when you said open source isn’t a business model. It’s a way in many ways of achieving certain outcomes in getting other people who aren’t just you involved in something you have going on. It’s one of those things you lose sight of over time.
Andrew: More threads you talk about. You can’t talk about OpenStack and some of these things are helping now without also talking about foundations. I’m not sure that’s worth totally dissecting right now but the foundations end up in this weird standoffish relationship where some of them are kind of paid to play and they try to have this blanket umbrella, neutral Switzerland approach of things but that’s not always entirely true.
On this notion of a business model, open source that comes out of a company like ReTiS, like Confluent, like Docker, we have a single company that’s kind of trying to make a go at profitability and it even has taken investment. It’s fundamentally different dynamic than what you see coming out of Google, or coming out of Netflix. These are the things where they’re getting leverage from open source as an effort, as an investment, that is different than the profit. They’re not trying to generate a profit directly out of TensorFlow, out of Kubernetes. It’s essentially, in some ways, a marketing exercise as much as a business.
To go back specifically to Docker, I think Docker is interesting. There’s a couple of good lessons in Docker. In some sense, we use the word ‘technology,’ and some people will take issue of what I’m about to say, but I don’t feel Docker was really technology. I feel Docker was unveiling or revealing fundamental features that had been in the Linux kernel for a while and as a consequence of that, what they basically did was to fix the usability bug with the way that cgroups in namespaces were managed, and gave you sort of this same defaults, easy path to usage that the average developer could finally use containers.
Before that, if you want to use LXE, even OpenVZ, or some of these other container technologies, you kind of had to be a system-level person and you kind of need to make sense of these really long convoluted command-line arguments to set up the root file system and all the flags or whatever to get the right behavior. Docker kinda wrapped that up in a usability layer that gave you access to it. It turned the world onto what these possibilities were but didn’t actually create a new technology and they also have almost no moat to protect their business.
Corey: Right and past a certain point, when you fix the usability problem, it’s one of those challenges where it’s very hard to turn resurfaced awareness of this thing and gave slightly better tooling and visibility into what’s going on, then turn that into something that’s reliable, sustainable, and as you say, has a moat. It is a defensible business strategy.
Andrew: It kind of goes back to the comment that you made about the consultants for OpenStack or the cloud stuff because what happens in these dynamics, and I experienced this with Puppet to some degree as well, where you create something that is intrinsically valuable and easy to use. You almost get less leverage-to-capture value. You have to be very, very careful if you’re trying to build these businesses on open source. Assuming you’re not printing money because you’re Google, to do something where you have something that is going to enable those transactions that you ultimately hope happen.
Corey: Yeah and that’s the sort of thing where you also wind up with companies who are trying to navigate the question of do they become a product company or do they become a service company? It’s very hard to be both effectively and it’s even more challenging to go from being one to being the other in any sort of scale.
Andrew: I think that there’s a little bit of a false dichotomy in that. This is a quick aside but if you look at the way that that’s been projected and you also look at the reality of enterprise IT, there is no successful enterprise product company that doesn’t also so a massive amount of services. The question is and really the bright line in my mind between a consulting company that is only a consulting company and a product company, especially in enterprise IT, is being able to focus and say, “No.”
In the mercenary-for-hire, anything-for-anyone IT business, then you’re chasing the shiny coin and that’s ultimately what prevents you from capturing value from a product but there is no product company, there’s definitely consulting companies that don’t have products, but there is no enterprise IT that doesn’t have services.
Corey: Where would you put enterprise IT companies that handle the service aspect exclusively through partners or doubt those exist?
Andrew: I think that those definitely exist or the proponents of their work will be through partners, assess and lead the way that the empire was built and Microsoft to some degree, Oracle probably as well. I think that the case there is you’re basically choosing to do things through the partners. My point is not necessarily that you can’t be more pure as a product company but you have to enable that partner thing. You couldn’t do that until you get to a mature market.
What I see time and time again in some of my smart friends who made these mistakes before, is you want to be a product company or you made these promises to your investors to be a product company. You’re trying to be a product company which means you don’t want to do services or that actually means is you don’t have relationship with the customer. You don’t have good feedback loops into your products.
I think at the point where you have mature products that are at product market fit, then you can leverage these partners to grow that into the new world domination strategy. But if you don’t actually understand that life cycle of how the partners are going to execute on this thing that’s mature, you’re still on this nascent phase trying to build it, and you don’t own a partner relationship, you’ll end up in a really bad place and your product development suffers as a consequence.
Corey: I think that’s probably one of the more astute assessments of this that I’ve heard, which sort of makes me wonder why so many companies seem to stumble in this area.
Andrew: People are terrible at product management.
Corey: Ugh. If you want, I’ll have something engraved in a tombstone early. I think that might be a great epitaph.
Andrew: I’m hoping for better, but I’ll put it on the list.
Corey: Absolutely. Taking a bit of a different tack for a second here. AWS Outposts has caused a bunch of kerfuffle. For those who are not familiar, AWS Outposts are effectively sealed racks that AWS will put in your data center if you pay them enough money and run AWS hardware inside of them, and AWS services float on top of them, and now you have the AWS APIs on front.
A take that I saw around this was, “Aha, they’re admitting that only public cloud is no longer a viable business model,” and I expect that on Twitter but I didn’t expect it in headlines of Business Insider, for example. I’m a little disappointed at the lack of awareness a lot of journalists are bringing to this. What’s your take?
Andrew: I’m just going to laugh. Journalists are going to be journalists. They don’t necessarily know what they’re talking about. The reality is that cloud to me, cloud-native, or whatever you want to say this thing is, is not where you run but how you run and you’re literally getting the APIs in your data center. It’s essentially what you’ve wanted from OpenStack in the first place.
I think from the other side of that equation, if you look at what Amazon is really trying to do, and something I’ve heard you say and other people are fond of saying it, you can meet people where they are. You’re giving them the same workflow and the same APIs in this data center that they manage, and you know at some point, that they’re going to put that stuff in your cloud, in your public cloud because you’re going to manage it better than they can. Guaranteed.
Andrew: They’ve already got some investment and practice with some of the other things they did earlier, Snowball or whatever, where they have started putting hardware in places and this is just expanding. This is in some ways dovetails back to the comment I just made about cloud management. You can fault Amazon about a few things but you cannot fault them for their relentless focus on the customers. That customer focus is really what drives all these things, so if you have a bunch of customers, they’re saying, “Give us this thing in your data center,” then yeah, it might take a while—years—but here we are and they delivered it. They focus on what their customers want but they also try to keep promises about what they can and can’t do at the point where they could keep those promises, they ship those racks.
Corey: Absolutely. I think there’s a story here about how they also that same week launched Ground Station, which provides radio telemetry reception for satellites in orbit. If the total addressable market of people with satellites in orbit is a large enough market segment for them to focus on, I’m going to go on a limb and hazard that the people who are not comfortable with public cloud for all of their workloads, is a larger total addressable market. Meeting people where they are in solving customer needs is absolutely what AWS has historically been about.
Andrew: Not to me under too much but the track of the historical evolution of Amazon, building a business, selling books on 1% margin gives them advantage to explore all these things. If you have a margin and Amazon thinks they can get something similar to market for half of that, they’ll do it. It’s not just about the cloud. I get $10 a day from Amazon Prime, some days thanks to my wife, but the stuff they’re doing like Amazon Basics is basically an Amazon-engineered brand where like, “Hey, we can make these things. We can do these things. We have whatever understanding of that market. We have whatever understanding of the manufacturing. We can basically put it in a spreadsheet, expecting to sell this much, and it’s a done deal.”
Corey: It also, and I thought it was very astutely delivered from marketing perspective, where they didn’t have to say the reason on-prem environments tend to suck is because people cheap out on the hardware and then they mismanage the crap out of it. By shipping a sealed rack to customers, they’re effectively reducing the customer responsibility to more or less power and cooling and that’s not something Amazon has an offering around this year. But they’re really moving things up the stack even in environments that they can’t control. They’re still trying to eliminate as many variables as they can. They’re not offering their control plane on your own custom crappy hardware, for example.
Andrew: I think that’s the key to understand here and this to me defines cloud-native. I don’t confine cloud-native to just Amazon. It’s really about confining those variables. If you looked at the way Google’s built data centers for the last 15 years, you’re talking about football field-sized data centers with racks and rows of identical gear. By collapsing those variables at the bottom of the stack, you can invest more of your complexity on the higher order features and applications.
Corey: Oh yeah. There’s a story here about the economies of scale. I used to rent from an individual landlord. When the refrigerator broke, it would be a week until a repair person came out. I moved into a corporate building that had hundreds of units. They’re all identical and I reported a broken refrigerator and they were there with a brand new one within an hour. They have a store room downstairs with five new refrigerators, another 10 of them in spare parts.
At some point, when everything winds up looking identical, it really starts to be a, “Oh, you just need a number seven. Got it,” as opposed to everything being bespoke and difficult because it’s the first time or the only one of anything you’re running. It’s the hardware equivalent of pets versus cattle.
Andrew: Another great point to tease out and this is probably the take-home that drives a bunch of the bad decisions in a lot of these places, you said that they didn’t have the right hardware, they underinvested in their hardware. I think that the reality is for enterprise IT in most places, especially where they told themselves they weren’t a tech company, that they’ve under resourced IT across the board. In the worst case, there are even managing IT, CTO reporting to the CFO. When you treat these things as a call center and your only lever is cost management, then it’s a bit of a self-fulfilling prophecy and you're always going to under underdeliver.
It’s a little bit of a misnomer or I think it’s a fallacy when people think that these IT departments which are under resourced and calcified for decades, are going to become this digital transformation center overnight because they decided that they wanted to become more savvy, they want to compete, whatever. Until you re-frame that mindset at the highest level, and that has to do with comp, it has to do with this investment in the hardware, everything to do with how that money gets spent, then you’re going to have a bad time and especially if you’re competing with the Amazons, the Googles, the rest of the world that has sort of made that leap and views IT and technology as a weapon.
Corey: I think you’re right. I think that it also gets a bit to do with Simon Wardley been talking about was value change. Where the differentiation that makes your company succeed is probably not going to be in stuff that’s already heavily commoditized, which is why it frustrates me to see companies that sells stocks, more or less, focusing all their energy on building their own customized container orchestration system. It tends not to be the path to victory in anything of the extreme long-term which is not the planning horizon most companies are operating in.
Andrew: This is hard to navigate. I’m a huge fan of Simon Wardley and if any listener is not familiar with Wardley maps and Simon’s work, then I encourage you to go waste a day familiarizing yourself. The reality is that you have not just okay to decide what’s commoditized, but also what will be you thing. I do think that there’s a need to focus on the high order of functionality and certainly Amazon, the other cloud providers, and some of the vendors are trying to help people do that, but the reality is not everything is off-the-shelf right now. What is a commodity today will be a commodity tomorrow but what is not a commodity today will probably be a commodity tomorrow. It’s like figuring out when you want to make that jump, to make those investments or not make those investments at, that’s a hard choice. It’s not easy is what I want to say.
Corey: It really is. Out of curiosity, where do you see cloud-native fitting into the landscape along with things like DevOps, SRE, and other buzzwords that wind up causing people to get all hot and bothered?
Andrew: I’ll give you the short version. Well, there’s a much longer version, this will still probably seem a little long but the cloud-native as a moniker was something that I specifically used as part of Pivotal Messaging for a lot time. Before there was a cloud-native computing foundation, Pivotal was using cloud-native as a term.
To me, it’s really subsuming all the buzzwords. I think that when you look at DevOps, I think of it as a much bigger kind of systems thinking thing but in terms of how you can explain to someone and not end up sounding like a new age guru, it’s really just this interface between developer and operations and trying to smooth the deployment and operation of software.
When you get to cloud-native, I think you open up into a much bigger story around architecture, some of these other patterns, and then I also think that there’s a huge cultural element. One of those things I just mentioned around the way framing of IT ends up being costly versus competitive manage, so to me the cloud-natives, they definitely use DevOps as a way to do things, whether they want to call that or not.
Certainly, you have SRE as a development. Some people have a fun time making these link-baitey arguments about SRE versus DevOps but in reality, it’s really one thing to me. I would also note that these are not things that people set out to do. People didn’t set out to do DevOps. People didn’t set out to do SRE. It was a function of the Darwinian pressure to deliver these things at scale with reliability, with speed reliability, scale, whatever, all the -ilities and -enities that you get when you’re a Google or an Amazon.
That Darwinian pressure had a solution and that is very, very common. You can find common patterns, not identical patterns but certainly common patterns across all of these companies. Google, Amazon, certainly when you get into the Twitters, the Facebooks, some of these others, they manifest the same patterns.
To me, that evolutionary cohesion that gives you the ability to deliver software, reliably at scale, rapidly iterating to serve the customer, that’s what cloud-native means to me. That’s tech and that’s process and that’s people, and that’s everything.
Corey: I think that there is a serious misunderstanding in the larger industry, sector, world, whatever you want to call it around the idea that the solution to your problem is something you can buy, where you just cut a check to someone and magically the problem goes away. Cultural change never works that way, not that it crosses improvements. It’s something you have to internalize as an organization. Believe me, I wish it were otherwise. I would love to sell people the magic solution in a box, answer to all their problems, but I just never found it,
Andrew: This is reinforcing the point I was making a minute ago when you brought up Wardley maps because on one hand, you wanted people to realize that you don’t want to build everything and that there’s some things you should buy. On the other hand, you can’t buy everything. People like buying things because it’s more concrete than this abstract notion of culture.
We start to talk about true digital transformation, especially an organization that’s done something for decades, then the idea that all of a sudden, you can buy this tool or this certification or implement this thing and you’re going to be done, it’s absurd but you see that happen literally every generation. It literally reminds me of the ‘lose weight with one weird trick’ click ads that you get on whatever website. People are always somewhat susceptible to the easy answer and I think that’s a dynamic that’s developed unfortunately.
Corey: It’s human nature. I’m not sure there’s any patch for that, that people will find even remotely acceptable. Last question for you before we wind up calling it an episode. You’ve given a number of talk ideas. Some years, you’ve given a number of talks, others you’ve gone suspiciously quiet. I guess I’m wondering, where do your ideas come from? What gets you excited or fired up about, “I need to go on stage and ‘give a talk’ about this,” which, similar to me, often translates into, “I going to rant into a microphone for 45 minutes,” which frankly I appreciate.
Andrew: Sometimes, I get focused on tech and those are easy talks, actually, because you want to just teach people and share how this works or I gave some talks about cgroups and namespaces, whatever. The ideas that I get most excited about, I often get them connecting things that are maybe from outside and then I just let them marinate and I try to do whatever kind of real world experiments I can in my organizations or the people I’m working with at the time.
But the talks I’m most proud, however, they came from these organizational learning researchers, reading some paper, some psychology paper, the Maslow’s hierarchy or Maslow’s classification around burnout. All these stuff kind of gets me excited reading the psychological or the management theory or whatever, and then trying to do those experiments in Vivo and see what comes out of it.
Corey: That tends to be a much healthier perspective than where my talks come from, but usually starts as a random story I’m telling someone and they’re laughing and they say, “That’s great,” but you will never turn that into a conference talk. I take that as a personal challenge.
Andrew: Challenge accepted.
Corey: Exactly. Did you build that entire conference talk around that awful pun at the end? Maybe?
Andrew: If someone walked out to me and said, “I bet you couldn’t turn that into a conference talk,” I’m pretty sure it would be a conference talk. We’ll just throw that out there.
Corey: Exactly. If people are excited by the things you’ve been saying and want to hear more, where is the best place to find you?
Andrew: It’s really easy to find me on Twitter, a little idea on Twitter, and then andrewclayshafer, LinkedIn, whatever. Don’t you say you want to connect if you send me a note, you want to introduce an interesting topic, something you want to throw back and forth, I love to express ideas, especially little ones.
Corey: I’m right there with you. If someone wants to connect with me on LinkedIn, I should at least have had a long enough conversation to remember your name. If you saw one of my talks on video perhaps, and then send me a LinkedIn request, I’m thrilled to connect with folks but also give me some context. What do you want to talk about?
Not to be unkind but convince me that you’re not a recruiter who’s just really trying to scrape my network because somehow, if you’re connected to me, you’re somehow perceived as being more trustworthy. No one should ever trust someone who claims to know me. Just the opposite, in fact.
Andrew: Honestly, you never have to talk to me but just give me some context. Show me that you’re interesting or interested in something.
Corey: Exactly and you can even just wind up on the grid with people you should connect with, clicking until the API limits stop you.
Andrew: They’ll never stop us.
Corey: Exactly. Thank you so much for taking the time to speak with me today.
Andrew: It’s a pleasure. Hopefully we’ll do it again sometime.
Corey: Absolutely. This is Andrew Clay Shafer, famous for screaming at clouds. I’m Corey Quinn, who’s screaming at the cloud as we speak. This is Screaming In The Cloud. Stick around.