Open source activism tends to focus on running on hardware you can trust and avoiding Cloud computing. The problem with some Cloud providers has to do with a conflict of interest between serving customers and how they generate revenue. It’s important for the customer to have control of their computer and their data in the Cloud. But what about their security and privacy?
Today, we’re talking to Kyle Rankin, chief security officer at Purism and writer for Linux Journal. He is a Linux expert who decided to work at Purism because of the company’s belief in free software and the Linux community.
Some of the highlights of the show include:
Cloud providers have faced challenges when it comes to data privacy and who owns what.
The word “Cloud” is overloaded, and it is unclear who is in control.
Cloud providers can sabotage efforts to make programs work together.
Cloud providers may not troll through data and exploit it. Yet, they develop tools for customers to be able to do that.
Even though Linux Journal stopped being printed and went digital, and was going under, it’s now back and taking a new approach!
What matters to new readers and Linux users is now different than what was important to original readers.
The more time you can spend to understand what’s happening behind the scenes will make you much more marketable and adaptable.
Kyle explains whether Amazon Linux is becoming a viable concern and if distribution matters anymore. Now, it’s about running an application, not thinking about what it’s running on.
Are there gangs of Cloud users? Do people look down on Azure users? The target is always moving and changing.
Check out Kyle’s book, Linux Hardening in Hostile Networks: Server Security from TLS to Tor.
Kyle Rankin’s book - Linux Hardening in Hostile Networks: Server Security from TLS to Tor
GorillaStack (use “screaming” for discount)
Full Episode Transcript
Corey: Thank you for joining me on Screaming In The Cloud. My name is Corey Quinn. Today, I’m speaking with Kyle Rankin who many of you may know from his long-standing writing and work on Linux Journal. He’s currently the Chief Security Officer at a company called Purism and has been one of the most prolific and long-running contributors to the open-source space in the form of speaking, writing, and otherwise, hurling opinions that you’re likely to find. Welcome to the show, Kyle.
Kyle: Oh, well thanks for having me.
Corey: What are you up to these days? You joined Purism relatively recently. I don’t know much about them yet. What are you folks up to?
Kyle: Yeah. It was interesting. Purism and my work at Linux Journal sort of dovetailed in two ways. I first became aware of Purism a couple of years ago when I did a review with their very first laptop for Linux Journal. I met with the CEO and discussed the products and the vision. It was a crowdfunding campaign for this laptop. The idea was to have this laptop–like some other laptops that come pre-installed with Linux–have carefully selected hardware that works well with Linux. But in addition to that, has hardware that can run with completely free-software drivers which isn’t always the case with other laptops you can get. That was the mission with that. In addition to that, to focus more on privacy and security to have these kill switches that allow you to turn-on and off your Wifi, radio, and then also turn-off your webcam and microphone. All of that I thought was really interesting.
I did the review but the laptop was a giant sport utility laptop, a little bit too big for me and so later on they came out with a 13 inch laptop, reviewed that, I was like, “This is really cool.” I ended up buying one and using it. Now fast forward to this past year–in the meantime, I had been advising them every now and then because I really believed in what they were trying to do. Then Linux Journal, analysis that they’re going under. That hit me pretty hard, given I had been writing for them for a decade and so I started writing this farewell letter to Linux Journal that turned into this weird manifesto by the end.
I started talking about we really have to get back to the Linux community. I need to do something myself to support this community more. Around that time, I was talking to Purism about maybe taking on a more full-time role there and it seems like this is the perfect time for me to put my money where my mouth is kind of and work for a place that really is–everybody there truly believes in free software, believes in the Linux community, and believes in all of the things that are on the front web page. I was like, “This is what I need to do.” They, right now, have the laptop, two laptops, 13 inch and a 15 inch. We just did a crowdfunding campaign this past summer for a phone. It’s the first phone that run not Android but Linux for a long time.
Corey: That’s fascinating in that you generally tend to see a lot of the free and open-source software activism, generally tends to go hand-in-hand with, and that’s why you should run on a hardware you can trust which invariably is a decade or two old. To completely avoid any form of cloud-computing, don’t use any third-party providers, run your own gear in data centers. It doesn’t sound like that’s the direction that you’ve been going in lately.
Kyle: Yeah, that’s accurate. I think that the problem with some cloud providers has more to do with where the conflict of interest between serving the customers and how they generally generate revenue. If you have a company that’s mostly making money off of customer data, they have this sort of perverse incentive to collect more and more of that–that obviously can be an issue. But that’s not the only way to make money on the internet and that’s not the only way to make money with cloud services either.
To me, where you withdraw the distinction and say that these aren’t completely [...] concepts is there’s a notion with free software and there’s a notion just in journal with the work I’m doing at Purism now that it’s important for the customer to have control of their computer, or in the case of cloud it would be important for the customer to have control of their data.
If you talk to most people, they don’t like the idea that most cases of all their data being out there, they may not care because they haven’t been hurt by it yet. But if you talk to them a little bit about some of the implications, they usually don’t like the idea. But then there’s also this resignation of, “Well, there’s nothing I can do about it now. I don’t really have any control over that.” To me that’s pretty problematic if people are going around and saying data’s the new oil, then I guess that means that we’re providing everyone with this new source of energy or this new source of revenue, and it’s weird that we’re doing that for free. I suppose we’re getting services out of it in some case but we’re providing all of this and we don’t really feel like we have control over it.
There are some cloud services out there that are taking a different approach to the customer data and focusing on privacy. There’s others that just don’t have one way or the other to do with customer data, like for instance with S3, you can completely encrypt your data and upload it and they’re not mining that data for anything.
Corey: What’s fascinating to me is if you take a look historically at the behaviors of Google, Amazon, Microsoft, Oracle, effectively all of the cloud players more or less, they have had challenges with respect to data privacy, with who owns what. But when it comes to providing a platform for companies to use in the world of cloud-computing, there’s never been any doubt or equivocation that a company’s data remains their data. In fact, I can think of no better way for these companies to dynamite their business than by exposing all of their customer data or any of their customer data to third parties, that would be doing massive cloud exodus.
For better or worse, I’m not seeing that between these different companies, as we start to horse race them against one another, that there’s a meaningful difference in any of the tier-1 providers to with respect to their respect for data privacy of the customers. Would you agree with that?
Kyle: Yeah. Again, because they’re making money off of CPU hours or the amount of data you’re storing, they’re not making money directly off of farming customer data, they have a different incentive. I think that’s a model that you could see in other platforms.
The other problem is that the word cloud is so overloaded. It applies to 50 different things. When people talk about the horrors of cloud services and the cloud is someone else’s computer, a lot of times there’s this notion of storing your data somewhere else and that person’s making money off of that data.
For some reason there’s all these discussion about the fact that Netflix and Amazon are competitors in the video space, but Netflix uses Amazon. But naturally, Amazon would never shoot themselves in the foot by doing anything to prohibit Netflix from using their platform. Because for the most part, the interest of the customer is in their mind in the sense of protecting their data or at least not harvesting their data, I think it’s just a different conversation.
There are other concerns outside of data sometimes with these cloud platforms just because there’s a notion of control. Whether or not you are in control of your data or your services–one thing that’s a little concerning for me personally has more to do with risks of interlocking than it does, specifically, storing your personal data because while all these platforms nowadays typically run Linux, and underneath the hood you can see this gooey sensor of Linux code and free-software, at top of all it all, they’re made to inter-operate well with each other on a particular platform but not really made inter-operate well with other cloud platforms, then get something like that, you need some extraction later on top of it. It reminds me of a lot the corporate IT space in the 90s and the odds where you had Windows servers that were dominating at that time in corporate offices–IT, and you had these services that weren’t designed to be inter-operable. They worked easily and great as long as you stayed within–in that case–the Microsoft ecosystem.
But if you wanted to set up your Linux server, you had to strain and scramble to try to get Samba working at first until things are really smooth and then there’d be other sabotaging with proprietary exchange, APIs and things. To me, we have the same thing in the cloud but it’s not really discussed very much because your average person doesn’t really worry too much about mobility between cloud providers or if they do, they use an obstruction later on top of it.
Corey: Right. To some extent, I think you’re also seeing these providers kicking the can to one more level in that Amazon is certainly not going to troll through all of the S3 data that it stores then exploit that itself. But they’re building higher level platform tools for their customers to do exactly that. But it stays within their customer’s approach. The client who is using AWS can do terrible things from a privacy perspective to their own customers. But Amazon is providing the tools to build a data lake. It’s not a data public pool.
Kyle: Right, yeah, yeah. From a data security perspective, there’s at least the tools–it’s like within this sort of tool, you’re given the ability to do good or bad things with it. These cloud providers directly, from a data protection standpoint, it’s less concerning than how difficult it is to move between providers potentially or there’s less of a resistance that interlock in, I’ve noticed among administrators these days back maybe a decade ago.
Corey: I think you’re very astute as far as what you’re seeing in the space. I will say it’s pleasant to have conversations with aficionados of free and open-source software who can–how do I put this without sounding incredibly offensive–who can understand the differing pressures and different directions the industries are going and not taking a hard line perspective around what should or should not be done for everyone. Thank you for your reasonableness.
Kyle: Happy to be reasonable. The thing is I’ve built quite a few types of server infrastructures for different employers on mostly the US. There are ways to do it where it’s so easy to get drawn and then locked into a provider, they make it so easy. If the easiest path is to use an Amazon service that’s already there for you to do everything, if you go that route, again, you have a very easy way to do it, it’s typically very scalable and you can back off and mostly focus on writing glue code for the APIs. That’s fine as long as you don’t need to transfer to another provider. That’s when, there’s all of the built-in assumptions that you make may come back to bite you.
But you know a lot of the platforms I’ve created, you’d probably call kind of a hybrid because they use more of a traditional approach to building an infrastructure with modern DevOps tools, but on the cloud without using cloud specific services for a lot of things.
Corey: Fair enough. I think that transitions us to the second topic I wanna cover with you. Before I wound up diving into cloud economics, I spent entirely too much of my life as a Linux systems administrator. For most of that time period, I was a subscriber to Linux Journal. I would get an issue every month, it would show up in my mailbox, and I read it on planes and what not, I’d live through it back in the day when you couldn’t use electronics until you were at altitude.
One day, I wound up getting a terrifying email from Linux Journal that they were no longer doing physical printing, it was going to be digital only. Now, I don’t know who’ve ever tried to discipline a Chihuahua by smacking her across the snout with a rolled-up iPad but it really doesn’t go very well. There was a loss of a discipline dog tool, there was a loss of having a physical magazine I could take with me. I mostly came to terms with that, societies evolve past dead trees to some extent.
Late last year, I received another email from Linux Journal saying that after however many years of being in print, it was going under, so long, thanks for all the fish. That brings us to today. I’ve heard interesting rumblings that it’s not dead after all, what’s the story?
Kyle: Yeah. I, like everybody else, assumed and thought that it was dead and broke my lament about that. But shortly after that was published, Linux Journal was contacted by PIA–a private internet access which is this UK firm of VPNs and things along those lines. They apparently are a major sponsor of free note and supporters of free software.
They reached out and said, “Hey, we want to fund this. We want this to continue, we don’t want Linux Journal to die.” Negotiations went around the things and yes, so now, it’s alive. That surprises everybody else, I was spending most of December just trying to figure out what I was going to do next with my writing. But yeah, they’re back, now it’s truly interesting because ever since the print publication went away, which by the way, every when within Linux Journal 2, all of us really missed the print publication too, it’s just at the time with the way the publishing was going on new stance, there was no way to make financial sense.
Corey: I think the only person that rejoiced when the print version went away was my dog.
Kyle: Yeah. As a result, everyone was making deals with the digital magazine. But because of that, now we started this decline where some people were bothered by that. Now it’s almost like we have this opportunity to do it over again and do it. There’s been this new sort of shot on the arm, this new energy in the magazine and the ability that taken to approach.
I wrote an article recently for Linux Journal talking about it as kind of a refactor where you get to look at everything, keep the things at work, throw away the things that don’t work, and respond a lot to customer feedback. That’s a lot of what we’ve been doing over the last couple of months.
I took a more active role recently to join the editorial team because I saw this as a really good chance to write a Linux Journal Magazine that’s both relevant for all the past readers who want to know a lot of traditional system and tips or want to know about Linux Journal programming or things like that and deep dive into those subjects. But also more articles that impact newer Linux users who maybe have only been using Linux because of the cloud or through the cloud and maybe aren’t even really interacting with Linux directly.
It provides this really exciting way for us to reach out to new users, find ways to be more relevant and write new articles. Because there’s a lot of how to’s out there on the internet right now about using Linux, setting up the thing quickly, but there’s not nearly enough deep dives and expert opinion, I tried this over the weekend, I drew on my notes but there’s not as much expert, I’ve been doing this for a long time and here’s how you do it correctly out there.
Corey: Right. That’s really why I wanted to have you on the show is to talk about how does a magazine that historically has been hot aimed at hands on hardware Linux systems administrators build and maintain relevance in 2018 where most people that I work with in a professional context in many cases have never touched a server in a data setter, they’ve never known the hell that is 20 million fans all spinning at high volume. It’s about, in many respects, now I’m dealing with people who don’t even care what the underlying operating system is let alone the disk row where it doesn’t really matter to them, they just care about a container based workload or they wrote some code in Python that as long as Lambda or where cloud functions or whatever it is that you are using to fire it off server can understand it that it doesn’t matter to them. This is no longer something that has maintained relevance.
It’s similar in some respects to what it takes to install a web server. When I first got started with this, it required three days to compile Apache and an in-depth knowledge of GCC flags. Then it became RPM-I to install it then Yum install, then ensure installed with something like Puppet. Then Docker run Apache and now it’s effectively S3 does this out of the box by checking up tick mark or adding a line to a Yamo file, things simplify overtime. Being able to configure Apache and get that up and running is no longer a skillset that’s going to carry through and make someone’s career. There is no job for that person in most jobs. How do you still remain “Linux Journal” but also address the emerging needs of today’s audiences?
Kyle: Right. No, that’s a great question. When I read Linux Journal myself and wrote for it–I both read and wrote for it from the perspective of a systems administrator but that really was never the full magazine, there was always at least half of it that was aimed more directly at the developer. Now, it was the developer who valued free software or open-source software, but it was still aimed at a developer. I think a lot of that will still remain similar because where that code is going, whether it’s a desktop or a cloud service, from a developer’s standpoint, there are implementation details but other than knowing it might be going on the Linux server in the cloud, they’re not as concerned with that.
For their perspective, it would be similar except that instead of talking about creating a plugin for some open-source web software, we’re talking about, again, deploying to S3 or using AWS services or some other cloud services as a developer.
Now as a sysadmin, it is very different. I think there’s still a place for a lot of it although it’s interesting. To me, there’s a traditional sysadmin that will still have a job at either super large companies that haven’t migrated to the cloud or the cloud providers themselves who somebody has to reconstruct servers. But in between, I think there’s the shift toward, we’re calling it DevOps, but it’s really in the ends, what I see in 5 or 10 years, is the majority of people really need to learn and be strong in one or two programming languages that aren’t batched. Because the DevOps engineer, five years from now–if not today–in some areas, is mostly riding API glue code which is a very similar job to what a lot of software engineers end up doing, gluing up together other people’s libraries.
I think to remain relevant for those people, I think there’s a couple of things. One, helping to bridge the transition for the traditional sysadmin and this sort of what the new reality of what it’s like to deploy a server. I think there’s been a lot of guys that bring someone from zero up to deploying on the cloud without talking about all the things that came before, but I think there’s a huge value in understanding all of the architectures that came before, the way that servers used to be deployed before, and so I think a magazine that can write articles that bridge that gap would be really valuable. Because if without learning all of those past mistakes, you’re bound to make them again.
Corey: The challenge too is if you take a look at how people who are senior engineers, or architects today got there, the path that most of us walked is no longer there. There are very few jobs today that involve someone being just the systems administrator. First, there’s a requirement that someone know how to code in most jobs. There’s also no requirement anymore for a lot of specialties that we all had to at least had some passing awareness around. There were storage administrators back then and that was an entire area of focus, an object stored didn’t exist, it was, let’s figure out how to manage a send.
Network engineering, the way I came up, is now effectively an API call away and is usually entirely removed from the day-to-day operation of an environment. Now, those skillsets are rusty but they come thundering back when there’s an outage that requires that level of expertise.
The challenge I’m running into as I think about what the future starts to look like, where do you see the future DevOps engineers–if you’ll forgive the term–coming from when they don’t have roles that expose them to the baseline fundamentals that we can pull out from the attic and throw out to solve esoteric problems?
Kyle: I think it will require a special kind of engineer who has this curiosity that drives at least some people into the industry where you’re not satisfied with sort of clicking next through life or at least through your work where you wanna understand how does that work. There’s some people who are fine which is letting something work and other people, it kills them to not understand what’s happening underneath the hood.
Nowadays, we have many levels of hoods, we have these Russian nesting dolls of obstruction layers. I think for someone who wants to be eminently marketable in the future, the more that you can spend time–unfortunately, because like you said, a lot of this doesn’t require at work time because everything’s so obstructed away–but if you can spend time understanding what’s actually happening behind the scene, you’ll be much more adaptable to wherever the direction the future goes. You’ll be way more marketable and then when something breaks, you may actually have an idea about how to fix it instead of just calling support.
This isn’t completely different, even when I was just doing straight of IT, you had the IT people who were past data and how all of this stuff worked and linked more on the system inside. Then you had people that lean more invoked with network engineers too. You have some that lean more on helpdesk type site where they were more focused on support contracts, getting really good support, because when it broke, they will just call support and be on the phone for half the time, they weren’t interested in figuring it out.
To me, one of the beauties of a lot of open-source software at least is that you can set up most of the stuff at home, now I’m not exactly [...] AWS necessarily, but there’s at least some aspect where you can, for free, explore a lot of these avenues, you can understand how networking works at home and you can understand a lot of these concepts without having to have on the job training. I think a lot engineers are going to be at a loss because they never had to learn those stuff.
Corey: It’s going to be a fascinating number of years as we start to see these industries evolve. Before this, I was saying the same thing about the loss of help desks. When you start seeing massive outsourcing to third parties rather than having an internal company helpdesk for basic IT, you sort of lose the breathing grounds where a lot of these future senior engineers start to distinguish themselves and are placed into environments where they can articulate the Why questions in a way that can lead to answers.
I think the future’s going to be fascinating in the space. I have you here–even if you don’t call yourself one, the rest of the world does–you are considered an expert in Linux. Five years ago, Amazon Linux as a distribution was something that I viewed as a punch line. Here in 2018, I’m not laughing anymore, it’s become something that for my use cases has been extremely reliable, something that is relatively easy to work with. Do you see first that Amazon Linux is becoming a viable concern and something to look at not only from a competitor perspective but from a what can the rest of the world learn from this? Two, does distribution matter anymore?
Kyle: First, I would say, what Amazon Linux demonstrate is the power of defaults. The fact that that’s a default image that a lot of people, when they’re using cloud services, they may not be exposed to Linux before this, like we talked about already, for many people, the distribution doesn’t necessarily matter. For some of us who use Linux outside of API calls, it still can.
To me, the new distributions are the cloud services themselves. Let’s say, we compare AWS and Google Cloud for instance, there are similar services that do similar things, they’re called different names in some cases, they mostly operate in similar ways but then depending on which one you choose, you have these extra features that one or the other doesn’t have. To me that feels a whole lot like using Red Hat versus Deviant where one of them gets some new feature and of course the files are always in a slightly different place and the services have a slightly different name, and it’s a little annoying. But if you used one, at some point, you can switch back and forth between a Red Hat and Deviant base system and figure it out and it’s not this huge problem.
You may have a preference but you can work through the differences that’s like getting into a different car where most of the controls are in a similar place. I think distributions may matter in other areas, but I think in the cloud at least everyone’s trying really hard to make them not matter. There’s a lot of effort being put into you’re just running an application and you’re not really thinking about what it’s running on anymore.
The notion of Linux has dominated the cloud is true in one way but in its other weird way, it’s been so obstructed away and pushed down that you don’t really have to interact with it at all to get your work done, and for some people that’s fantastic, they didn’t want to anyway. If you’re someone who finds that interesting, then you sort of lost something along the way, I guess.
Corey: In a world where distributions no longer matter, what can we look down our noses at condescendingly at other people who use something different and feel better about ourselves over?
Kyle: Sure, yeah. It’s always important because especially at conferences, you have to call out some sort of approach in a talk and shame everyone who doesn’t agree. We still will always be able to shame people about language choices. That’s always a win but one thing I’m concerned about is you can no longer have really strong holy words about conflict management, I mean, yeah it still exists, but it seems like in a lot of cloud services, that’s not necessarily the primary thing that people are focused on.
This is a real problem actually, we need to work on the next generation of things to become descending about maybe cloud provider choices. Do people generally look down the nose at Azure users versus others, are there gangs of cloud users that go around with their letterman jackets and rumble? I’m not sure.
Corey: There would have to be. But I specifically live in places where those folks don’t go.
As a general rule, I think that there’s always going to be something that we can be snooty about, but it’s always a moving target. When you build workloads that are completely cloud agnostic, and use Kubernetes or similar to schedule them into various places, that’s great. But at that point, we’re gonna argue about what exactly you’re using to schedule or how you’re going about doing it. It’s turtles all the way down, condescending, obnoxious turtles.
Kyle: I think the key is as long as there are software trends or application trends, there will always be condescension. It’s part and parcel with an approach being considered the right way to do it, whatever that is, and as long as it have enough hype about it that everyone feels bad for not choosing the approach, and that lasts for a while, and then someone says no, this is the new approach that you must do instead, and then everyone who’s still using the old approach that was trendy last year is looked down upon and people kick sand in their face.
Corey: Thank you very much for your time today, Kyle. Before we go, is there anything that you’ve been working on lately that you’d like to talk about?
Kyle: Sure, yeah. We’re talking a lot about the cloud. This summer, I did work on my first book that was 100% about security. I’ve been doing a transition over the years more into security role than like an infrastructure architect type role. I ended up doing a brain-dump on all the things that I had learned over the years of hardening services, in particular for the cloud, because there’s a lot of safety assumptions that maybe they’re not good assumptions but they’re assumption’s all the same when people have servers in their own little private data center, there’s this notion or parameter that everyone else think doesn’t exist. But in the cloud, you’re faced with it, it’s driving your face that there’s not as much of a sense of parameter.
I ended up writing this book called Linux Hardening and Houseful Networks and its whole premise is that there isn’t really a parameter, have you hardened Linux in a houseful network with this one, all the networks are essentially a houseful network and you have to start with that assumption and harden things with that assumption.
Corey: Very reasonable. Especially with the shadow of specter and meltdown looking at us recently with the spade of S3 bucket negligence of words of companies leaving confidential data in publicly exposed buckets. You can’t really punch on security. I think it’s important to remember that you can outsource work but not responsibility in a lot of these cases. Being able to dive into that in a more technical level sounds fantastical. I’ll link to that book on the show notes.
Kyle: Alright, great thanks.
Corey: Perfect, thank you for spending the time to speak with me today. Join us next time on Screaming In The Cloud. I’m Corey Quinn.