Do you have some spare time? Can you figure out an easier way to do something? Then, why not build some software?!
Today, we’re talking to Ian Mckay of Kablamo, an Amazon Web Services (AWS) consultancy. He is the author of Console Recorder, which is a browser extension that records your actions in the Management Console to convert them into SDK code and infrastructure as code templates.
Some of the highlights of the show include:
Timeline to build Console Recorder
Infrastructure as Code: How to code repeatedly without starting over and take ownership of what you built by hand
AWS vs. Individual Achievements: People asked AWS for years to create something to record console click-throughs that Ian did in his spare time
Console Recorder support for any browser that exports Web extensions
Sharp edges of what’s expected of Console Recorder to speed up development
Management Console’s unreadable responses require reverse engineering
Console Recorder: Recommended use cases and areas
How to alleviate security concerns with Console Recorder
Changes to Management Console that may break things
Ian’s past, present, and future projects and products
Words of Wisdom: If you don’t like something, just fix it yourself
Full Episode Transcript:
Hello and welcome to Screaming In The Cloud with your host, cloud economist Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming In The Cloud.
Corey: This week's episode of Screaming In The Cloud is generously sponsored by Digital Ocean. I would argue that every cloud platform out there biases for different things. Some bias for having every feature you could possibly want offered as a managed service at varied degrees of maturity, others bias for, "Hey, we heard there some money to be made in the cloud space. Can you give us some of it?"
Digital Ocean biases for neither. To me, they optimize for simplicity. I called some friends of mine who are avid Digital Ocean supporters about why they're using it for various things. They all said more or less the same thing. Other offerings have a bunch of shenanigans, root access, and IP addresses. Digital Ocean makes it all simple. In 60 seconds, you have root access to a Linux box with an IP, that's a direct quote albeit with profanity about other providers taken out.
Digital Ocean also offers fixed price offerings, you always know what you're going to wind up paying this month, so you don't wind up having a minor heart issue when the bill comes in.
Their services are also understandable without spending three months going to cloud school. You don't have to worry about going very deep to understand what you're doing it's click button or make an API call and you receive the cloud resource. They also include very understandable monitoring and alerting. Lastly, they're not exactly what I would call small time over 150,000 businesses are using them today, go ahead and give them a try or visit do.co/screaming and they'll give you a free $100 credit to try that, that's do.co/screaming. Thanks again to Digital Ocean for their support of Screaming In The Cloud.
Hello and welcome to Screaming In The Cloud. I'm Corey Quinn. I'm joined this week by Ian Mckay, who's currently working at an AWS consultancy called Kablamo. The reason I invited him here is a piece of software that he wrote and I've tripped over, over the past week on Twitter, which is the source of all things that we find in the internet, called Console Recorder. Let's start at the beginning. First, welcome to the show, Ian.
Ian: Thanks, Corey, happy to be here.
Corey: Always great to talk to new and interesting people doing exciting things. In your own words, what is Console Recorder?
Ian: The Console Recorder is a browser extension that lets you record your actions in the management console and convert that into SDK code and infrastructure as code templates such as CloudFormation and Terraform.
Corey: It winds up going through effectively what people click through while they're building out their infrastructure and converting that into CloudFormation.
Corey: I have to start by asking, was this something you did yourself or are there other people involved in building this?
Ian: No, it's just something I did in my spare time. It started off with me using the media live other libraries and they create channel function for that, it's just tremendously long. It's like six or seven pages just for the request. I spent two hours just trying to develop this and spec out all the properties that are needed. I thought there's got to be an easy way for this. I did some searching around and couldn't find something. I just decided—hell, why not just do one myself?
Corey: When you say you did this in your spare time, how much time are we talking about?
Ian: I probably started this in around September and it's taken about this long just to reach about 100% coverage with the CloudFormation templates. I'm sitting around 40% coverage with Terraform and about 20% with the APIs.
Corey: Does this only work on specific limited services at this point or is it more or less everything in the console?
Ian: Yeah, my target is to support everything in the console and support everything for a CloudFormation template at this time, which is about 330 resources. Terraform, I'm trying to spec out now, it should be done in about a month out – estimate, that's got about 450 resources and supporting about 180 there. As far as the API's go, there's about 5300 different calls in the AWS landscape and I'm supporting about 17% of that at the moment. I started off by focusing on the confirmation side of things and then I'm working on the other side stuff.
Corey: I think one of the reasons that this has resonated as it echoes around the internet. At the time of this recording, I think Jeff Barr referenced it somewhat recently, as well. It's taken on a bit of a viral aspect to it. I think a lot of that seems come from the fact that many of us have been asking for years for something like this from AWS.
For those who aren't familiar, what generally tends to happen when you build something in the console as you click around, you experiment until you get something that works the way you expect it to, cool.
Here in 2018 or over the past decade, there's this concept of infrastructure as code. How do you wind up doing this repeatedly without having someone else clicking in the console in the same place as you did and the official answer from AWS has largely been – okay, take everything you built in the console, throw it away and start over and do it in CloudFormation and other things as well do support this as you've referenced Boto and Terraform, and the new CDK. But there's never been a direct take ownership of the thing I just built by hand. That's been a sore spot for an awful lot of people for a very long time.
Ian: Yeah, I agree. A lot of feedback I've got just recently when this was starting to blow up a bit on Twitter is that, constantly, I'm hearing – why wasn't this a […] fade shout? Why wasn't this just here to begin with. I spoke to one of the solution architects when I was in reinvent recently, and they said – we looked into this two years ago, and they were just deemed to be too hard to implement, just because there's so many different variations in the way that the console works, because these are in siloed teams.
Corey: I hear what they're saying and I believe them that they're being sincere when they say that, but as you said, you did this in your spare time, the last couple of months and you've now gotten into a workable state as a single person. I get that AWS is built around the concept of small teams. But it seems to me that it wasn't as much heavy lifting, for lack of a better term that has to be insurmountable for a number of years. What did you see that I guess they didn't or was this just a matter of you deciding that you'd dive in and see what it took and it turns out surprise, you're incredibly productive when you start working on these things. And by spare time, you mean the 16-hours a day that you're not at work?
Ian: Yeah, that's right. There's a couple of different ways you could take this in that you could record the actions that the console is doing from a click point of view. But the way I implemented it is to record the network responses that come in and out of the console and then use that to translate into what the SDKs and the templates and those sort of things expect.
Corey: You've also built this into an extension for both Chrome and Firefox?
Ian: That's correct. It should support any browser that exports web extensions, technically it would still work in Opera, Brave, or any of those browsers as well.
Corey: Yeah, realistically, once you get out of the major browsers that you tend to see in almost every environment, then it starts to be a little bit of a point of diminishing returns. I'm astounded at works on something other than just one. The fact that you've supported multiple browsers and multiple output languages is kind of astonishing.
Ian: Yeah, it's interesting. I started off with just the Python and the CloudFormation support but as I saw different tooling that is in the community, I've decided to expand it out a bit. The most recent one being the CDK, which is not even in production release yet, but is being actively developed. I'm working closely with that team to make sure that that is up to date.
Corey: One thing that I always praised GCP or Google Cloud for has been that, when you build something in their console, they spit out what amounts to a, "Congratulations, good for you," here's how to do what you just did programmatically. As you go ahead and click through the console, which I admit is my preferred way of experimenting with them trying new services and things I haven't really wrap my head around yet, and turning that into something that you can use programmatically. It just seems like you've effectively duplicated that particular that trigger feature, which has been how hand people's wish list for a long time.
Forgive me, if I if I'm coming at this from a place of it seems too good to be true, what are the sharp edges on this? Where is it not going to necessarily do what someone might expect to do?
Ian: It's important to remember that this is doing as best as it can to record the actions from the console, which may not necessarily map to an SDK call or an API call.
It can be particularly verbose, especially if you want the very Bare Bones Configuration in your templates or your calls. If it says, I'm going to be explicit with this property that nobody really uses, it will map that property over. Sometimes it can be just a bit too verbose, but it's meant to be a good baseline for people to speed up their development. It's not just knocking out of the box tool, like most people should have a bit of tweaking to do with it after it's been generated.
Corey: Yeah, I guess that's my next question. Given that I've been too busy marveling at the fact that this exists, credit where due, I've been a little slammed the last couple of days and haven't had time to play with it myself directly. How usable is what it spits out as far as being able to take that drop it into an existing environment and iterate forward that? Or, does it require a fair bit of manual massaging to turn it into something that is repeatable and usable?\
Ian: I think out of the box, it's very well supported. Most calls and templates should be able to be upserted and tried out of the box assuming those resources that you just created in the console don't exist anymore, there will be some slight deviations and there's obviously going to be some bugs because every single qualities manually mapped. If anyone sees any bugs, then happy to reach out and prove that and fix that in the code.
Corey: As you went through this and iterated forward on it. I presume that you're experimenting on this yourself as well as this. Did you encounter any bugs in the console itself and how did that unfold?
Ian: Probably, perhaps not some bugs, but the way the console works and the way the network calls is that it's so diverse for each different service down to the point where some network responses from services are borderline unreadable. CloudWatch, VPC, Data Pipeline, I'm looking at you guys like the console responses are insane, quite a significant amount of reverse engineering in some cases, just to figure out what's going on there. There's definitely some cleanup that we can see in the console responses. So you see, uppercase and lowercase deviations a lot with – there's that classic problem with ARNs and AMIs and the community where it deviates between the different CloudFormation Boto, and then this is just one layer on top of that.
Corey: Thank you, incidentally, for pronouncing AMI correctly as listeners to the show, that's something of a sticking point for me. I think you're right, I think that if we take a look culturally, at what drives Amazon, and how they build things internally, they're famous for their small teams working on individual services.
That works really well at getting a particular service from conception at 1.0 that to that the world can use in a relatively short period of time. But whenever you wind up seeing the shared services that everyone has to use, it tends to wind up in a bit of a mess. The example that I live with, in my consulting, is always the bill where all the different services interact with different billing dimensions and it turns into a bit of a mess.
The console is another good example. People who know what a frontend is supposed to do, which I am not one of them, have been complaining about the user experience in the Amazon console for a long time. Some of us who aren't that deep into that space, just tend to accept that it's going to take a little bit of work and be difficult to massage into place the first few times you do things. We view it as an incentive to start doing things programmatically with CloudFormation or one of its similar services.
I guess, what I'm wondering is – are there certain use cases that you see where this would be ideal for? And/or are there certain areas where you would absolutely recommend no one ever try this and expect it to work?
Ian: It's good for most use cases. I don't see any particular services where this doesn't work out very well. There's definitely not 100% support in the CloudFormation console for all services. You won't get any recognition calls or any of that sort of thing, you still need to use that. But I haven't seen any services where it's definitely just – don't use this.
Corey: Perfect. One question that I would be remiss if I didn't ask, are there security concerns that people should bear in mind when installing this extension? Because they read about it on Twitter or heard about it on this podcast or saw Jeff Barr talking about it. I guess, how should they reassure themselves that this is something that is not going to destroy their company out from under them?
Ian: Yeah, that's a very important question. I think, the first thing to know is that this is in a record-only aspect, with the exception that you can potentially block you to book calls as an option, which is not by default, but it's a record-only extension. Look, there's always going to be this concern where potentially the extension can be taken over by a third party. For those people, it's always available to look at code and it's always available to download directly from the GitHub Repo, analyze the code, and make sure it's not sending data externally or that sort of thing, and you can install that directly locally.
If you do have those concerns, you can basically vendor the extension as it is now and download it and use that. Many can see exactly what's going on.
Corey: A pattern that I've used myself and would recommend if people have a sufficient level of security concern is, I disable console access for myself in my production account for the things I use, the power of this podcast, my newsletter, my consultancy, etcetera. But I have a developer account where I can click around and do all kinds of interesting things, but contain no actual data of my clients, or people who subscribe to the newsletter, etcetera. The rule is, something has to be built programmatically to get into the production account. There is no, I guess, production secret in the developer account. To that end, if someone winds up putting secret credentials or whatnot, into a into the console, does that wind up getting captured by Console Recorder?
Ian: Yes, it will. That's part of the post that we put on security—you should always check your templates for any credentials, or any secrets that might not be necessarily good to keep in code, or committed, and make sure that they're extracted out at that time, it's a very brute force tool, it's not going to look into any specific code, it's going to say—this is what you put into the console, this is what you're going to put into your code.
Corey: If you go ahead and do a recording of Console Recorder yourself and then hand the output to me without any tweaking or any editing, ignoring the credential piece for a second. Is that something I can take and run and have success with almost immediately? Or do I have to go through and factor out things such as hard coded account numbers, specific availability zones? For example, how I guess portable is it between accounts?
Ian: Yeah, it tries to do the best job it can in recording resources that connect to each other. For example, if you were to enable the response in a sector and then create a security group and an instance at the same time, it will link them in the confirmation template, you don't have a hard coded SG reference or an I-instance identifier.
Corey: This is a question that isn't necessarily going to help with anything, but if you would go back in time, and we're starting this project again today, what would you do differently, if anything?
Ian: When I started this, there was a lot of change in the way that console responses came out. I made the conscious decision that any recordings that come in would be purely mat as code. Basically, a bunch of statements, if the rejects matches this, then this is the expected output. I'd like to do this a lot more programmatically in the future but I haven't determined a way yet that I can do that – that cross references it in a nice way, because there's always a bunch of data meddling that I have to do sometimes to map it to the correct response. When the console gets back in ARN and the CloudFormation wants a name, then I have to do that conversion.
Corey: One thing that wasn't particularly surprising to me, though I suppose, it maybe to folks who are not as, I guess, overexposed to Amazonian culture. When I saw this and talked about it with a couple of my friends who work at AWS, you might expect their response to be angry or bitter start pointing out problems. But the response across the board that I heard was – Isn't it amazing.
People are thrilled to see something like this come out and I absolutely agree that – I count myself in that list, as far as people who are pleased to see it. I just keep going back to the fact that there's nothing even remotely close to this that's being offered natively. It stings a little bit, just from a usability perspective, it seems, and this is unfair to the excellent work that you've done. But it seems like it's not that hard to have developed over the past, what is it now 12, 13, 14 years that AWS has been a going concern.
Ian: Yeah, and the advocacy from anywhere has been incredible. There's nothing but support from the PMs and the solution architects that reached out and said—this is amazing.
Corey: It really is an amazing accomplishment. I don't want to even slightly come across this downplaying what you've built. This is one of those tools that I will be shouting from the rooftops anytime it's even slightly relevant, which is a disturbing amount of the time.
Ian: Yeah, and we are seeing a bit of a consolidation in the console. Just recently, a lot of the console experience is being a lot more shared, a lot more common UIs. I think RDS and Lambda were one of the first to implement this new UI response in any new services, get this new UI look, I guess you could call it and it does actually map to the network calls. Network calls are a lot easier, they usually are just chasing formattable. Any new services, they're pretty easy to map out of the box. I can usually get things like Transit Gateway, Day One support, pretty easy to put in. But it's lot more of these older services that are in this legacy mode, where it's a lot more chaotic.
Corey: To that end, do you foresee a future in which there are changes to the console that wind up breaking things, for example, they release a new service, or they re-factor the UI for an existing service and that winds up causing problems for the extension?
Ian: Yeah, I've seen it already. Recently, the ACR console, got a new overhaul and what came with that was a new network mapping throughout the whole experience, I had to redo all that mapping by itself. Things will break in this when AWS iterates in their changes. But the idea is that I'll keep up with them.
Corey: Are your plans just to keep this as an open-source project that you work on as a hobby? Are you looking at potentially turning this into a business to some extent? Are you using this as a portfolio project? If you don't mind the question, what are your plans for this other than look at this neat thing I built?
Ian: Yeah, it's mostly just a portfolio project. I plan to maintain it as much as I can. I've got a lot of other development tools, which I'm trying to give just as much love to. I don't plan to commercialize it. I just want it to be a good tool for the community.
Corey: It certainly is at that. For people who are listening and having the same reaction that I am right now. Oh, my word. What else have you built? Is there anything relevant to AWS or cloud that you've built in the context of a developer tool?
Ian: Sure, before I started this project, I worked on a Visual Studio Code extension, which synchronizes with Cloud9 environment, live synchronization by files, but also current positions, you can do a Google Docs experience from both the Cloud9 experience and your Visual Studio Code ID. That was a fun project and we got a lot of feedback from there as well.
Corey: I must have missed that one entirely but that sounds almost as, I guess, Earth shaking is this one. I am wondering if AWS is going to acquire you personally, at this point, where it's just—yeah, we've been decided to acquire, and your responses along the lines of—I'm a person, you don't get to acquire people like that, and their response is—watch us.
This is one of those things where it seems like a single person has been going after some of the more annoying aspects of working with a cloud provider and just picking them off one by one.
Ian: Yeah, I'm always of the opinion that if you don't like it just fix it yourself, whatever that might cost.
Corey: I guess words of wisdom, for lack of a better term, would you have for someone who sees an annoying problem like this, looks at it and says—Oh, well, if AWS has done it, or someone else hasn't done, it must be too hard, and not worth tackling.
Ian: Oh, just for sure, make a POC, do a proof of concept, figure out whether it's viable. There's a lot of times where it just won't be and then you can move on. But if it's a viable concept, you can open source things, you can commercialize it, just do it. It doesn't have to be, this huge styled, clean, tidy project that you do, it can just be this little demo. A lot of the times the community will tell you whether it's a good idea or a better idea.
Corey: Did you communicate with folks in the community as you were building up this out conceptually by hand. I'm looking at the history of this on GitHub, it very much looks like it was developed completely in the open, but it doesn't appear that there have been other people submitting code to it.
Ian: Yeah, I haven't got any other contributors to this yet, there's been a couple of issues raised, but no other people have really contributed to it and core requests are welcome. But I haven't really exposed this. I've just been working towards my coverage numbers, actually, to make sure that it's a viable enough project before I really tell people about it. But yeah, a couple of the engineers in AWS seemed to have picked it up and I know it's making the rounds at the moment. Yeah, I guess it's out in the open now.
Corey: It absolutely is something of beauty to look at, it's neat. If for no other reason, if you never touch the AWS console, or even AWS, I still think there's a value to this in that it shows that a single developer has an idea can release something that one of the largest companies in the world has not been able to get out the door. There's a lesson somewhere in there, I'm not quite sure what it is yet. But it's absolutely inspirational from where I sit.
Ian: Yeah, it's great. It's one of those things where everyone's just like—why isn't this embedded, this is great, and it's just really good to get that sort of community feedback.
Corey: Well, it's something that I think has been a very painful spot for far too long. If people are interested in other things you have going on or the words you have to say, where can they find you on the internet?
Ian: Yeah, I'm primarily on Twitter and GitHub.
Corey: Your usernames are?
Ian: You bet, I'd post a link.
Corey: Fair enough, it will be in the show notes. Thank you so much for taking the time to speak with me today, I appreciate it.
Ian: Cool, thanks for having me on.
Corey: Ian Mckay, currently the sole developer behind Console Recorder. I'm Corey Quinn, and this is Screaming In The Cloud.
This has been this week's episode of Screaming In The Cloud. You can also find more of Corey at screaminginthecloud.com or wherever [...] is sold.