If you’ve been doing DevOps for the past 10-20 years, things have really changed in the industry. There’s no longer large pools of help desk support. People aren’t climbing around the data center and learning how to punch down cables and rack servers to gradually work their way up. Now, entry level DevOps jobs require about five years of experience. So, that’s where internships play a major role. But how can an internship program be set up for success? Where is the next generation of SREs or DevOps professionals coming from? Where do we find them?
Today, we’re talking to Fatema Boxwala, who has been an intern at Rackspace, Yelp, and Facebook. She’s a computer science student at the University of Waterloo in Canada, where she’s involved with the Women in Computer Science Committee and Computer Science Club. Occasionally, she teaches people about Python, Git, and systems administration.
Some of the highlights of the show include:
Mentors made Fatema’s intern experience positive for her; made site reliability and operations something she wanted to do
Academic paths don’t tend to focus on such fields as SRE, and interns tend to come exclusively from specific schools
Fatema’s school requires five internships to graduate and receive a degree; upper-year students are already very qualified professional software engineers
Companies don’t have time to train and want to find someone with an exact skill set; instead of hiring someone, they spend months with an unfilled position
Continuity Problem: You can’t train someone to be a systems administrator, if you aren’t willing to give them certain privileges due to inexperience
Use a low-stakes environment to train, where mistakes can be made; most systems aren’t on a critical path - don’t keep people away from contributing
If you have never broke production, that means either you’re lying or you’ve been in an environment that didn’t trust you to touch things that mattered
Internship should mimic the kind of work that everyone else is doing; give them responsibilities where their work has an impact
Bad mentors lead to bad internships; person in charge of your success doesn’t have the necessary skills; needs to be a good communicator, set expectations
As the intern, ask about possible outcomes of internship early on; mentors should be clear about expectations, feedback, and offers
Full Episode Transcript:
Corey: This week’s episode of Screaming In The Cloud is generously sponsored by DigitalOcean. I’m going to argue that every cloud platform out there biases for different things. Some bias for having every feature you could possibly want offered as an added service at varying degrees of maturity. Others bias for, “Hey, we heard there’s some money to be made in the cloud space. Can you give us some of it?”
DigitalOcean biases for neither. To me, they optimize for simplicity. I polled some friends of mine who are avid DigitalOcean supporters about why they’re using it for various things, and they all said more or less the same thing. Other offerings have a bunch of shenanigans, root access, and IP addresses. DigitalOcean makes it all simple, “In 60 seconds, you have root access to a Linux box with an IP,” that’s a direct quote albeit with profanity about other providers taken out.
DigitalOcean also offers fixed-price offerings. You always know what you’re going to wind up paying this month, so you don’t wind up having a minor heart issue when the bill comes in. Their services are also understandable, without spending three months going to cloud school. You don’t have to worry about going very deep to understand what you’re doing. Its click a button or making API call, and you receive a cloud resource. They also include very understandable monitoring and alerting.
Lastly, they’re not exactly what I would call small-time. Over 150,000 businesses are using them today. Go ahead and give them a try. Visit do.co/screaming and they’ll give you a free $100 credit to try that. That’s do.co/screaming. Thanks again to DigitalOcean for their support to Screaming In The Cloud.
Corey: Hello and welcome to Screaming In The Cloud. I’m Corey Quinn. I’m joined this week by Fatema Boxwala, who is currently a Computer Science student at the University of Waterloo in the faraway land of Canada. Welcome to the show.
Fatema: Hi, Corey. Thanks for having me.
Corey: I met Fatema a few weeks back when we were both at SREcon MEA in Germany. You gave an interesting talk about your experience as an intern and how to structure internships in ways that made sense, both the companies as well as for the interns themselves. You’ve so far been an intern at RackSpace, Yelp, and Facebook, correct?
Corey: Awesome. One of the recurring themes that we’ve discussed in previous episodes of the show has been around the idea of where the next generation of SREs or DevOps professionals or people who want to violently fight about either one of those terms because nomenclature debates are what this entire field is about. What is that next generation come from? Where do we find those people?
For those of us who’ve been doing this for 10-20 years, the roads that we walked down are largely closed. You don’t have large pools of help desk support the way that you used to. You don’t wind up with jobs of people climbing around at the data center and learning how to punch down cables and rack servers and gradually work your way through from that perspective. Now, it’s almost one of those entry-level DevOps jobs require about five years of experience in some respect and trying to get around how we wind up building that has been a recurring theme here. What I really liked about your talk is how you approached it from a perspective of setting an internship program up for success.
Fatema: My talk was about my experiences being an intern in the site reliability and operations space, the things that I took away from them, and the ways that mentors really made the experience positive for me and made it something that I actually wanted to do when I graduated, like a field that I’d want to enter straight out of college, which is clearly not something that’s very common. Most people end up doing regular software development or application development or something like that when they graduate and it’s not real.
Corey: Right. They wind up doing operations due to a confluence of poor life choices.
Fatema: Yeah. It’s not really something that we get introduced to. The internships that I did were the only way that I knew it was something that existed. The only reason that I got my first internship at RackSpace that was more operational was because I had a friend who had graduated. She was the systems administrator for one of our clubs on school and she set up this really, really great system for us. Then from we graduated and left, basically no one in-charge those servers, so I kind of had that responsibility thrust upon me. That’s where I learned basically my low-level systems knowledge, just like resetting people’s passwords and making sure that everyone had user accounts and stuff.
She’s the one who introduced me to RackSpace because she’s working there full-time. She thought that I would be a really good fit for the internship. That’s how I got introduced to this space, mostly by accident, I just fell into it. There wasn’t really an academic path to get me there.
Corey: I have to ask. When I talk to interns, who are at various client sites of mine both in San Francisco and elsewhere, there’s usually three or four schools that they come from almost exclusively. You have Waterloo, of course, and you also see RIT. Is this something that is done systematically at the university or they just doing terrific marketing of their student’s career first?
Fatema: Well, for Waterloo, in particular, the reason is that we have a Co-op Program. We actually have to complete five internships to graduate. Most students end up doing six. If we don’t get those internships, we don’t get the degree. The pressure is much higher for us to get jobs but it also means that by fourth year or third year, most students are extremely qualified professional software engineers because they’ve already been doing it for a while. It makes especially upper year students really, really attractive to these high-level companies for internships especially.
But I also think it’s just confirmation bias. They only hire interns from these five schools and they’re like, “Wow, these interns are really great,” and we only ever hire them. They don’t really see that there are other schools that have really high-quality students. There’s not that many companies that will go out of their way to recruit from different schools. If you’re never going to interview them, you’re never going to hire them and see how good they are. But because they’ve been doing it from these five schools for so long, it’s really hard for them to change.
It’s a really big problem, actually, because obviously there’s lots of extremely intelligent students at schools that aren’t these five schools. A pipeline starts somewhere and that somewhere is probably high school. If you’re not going to a very good high school then you’re probably not going to get into one of the top tech colleges. If you don’t get to one of the top tech colleges, you probably won’t get an internship at a top tech company, et cetera, et cetera, et cetera. Obviously, good high schools are dependent on the income level of neighborhoods so it’s a pretty big problem, overall.
Corey: There’s a definite systemic problem in that company say, “Oh, we don’t have time to train anyone. We want to find someone with the exact mix of skills that we need them to have.” So instead of hiring someone and getting up to speed in three to four months, they spend nine months with the job rack open because they can’t find the exact thing that they’re looking for and it’s thinking through exactly what’s that going to look like in terms of return on investment, time to market, and being able to get someone up to speed and productive.
It’s a growing problem and I’m not entirely sure that the traditional approaches are going to continue to work. This is compounded more so in traditional development by the fact that when you’re dealing with operations in any meaningful capacity, very often what you want to hear from your ops team when there’s a problem is, “Oh, the last time I saw this, we solved it by,” and then you get to fill in the blank there. It’s not something that necessarily follows an established curriculum in the same sense, in my experience. I’m very curious to know how you wound up approaching this from an academic perspective.
Fatema: I guess that’s true. It really is something that you build up from experience. One of the problems we had in the systems committee that I mentioned for our school server, the women in Computer Science committee owns four or five servers, basically, and we just used them for administrative stuff and we offer a little bit of computing resources to women in Computer Science. We would not want to give first-years systems committee privileges. We wouldn’t want them to have sudo because they hadn’t ever worked in a Linux environment before and we didn’t think we could trust them with the power or whatever.
But it made a really big continuity problem because there was no way to train someone how to be a systems administrator if you weren’t willing to give them the privileges of the systems administrator. I had to deal with being the only person, after my friend graduated, who had any idea of what to do with those servers. I would try to train people but asking them to learn how to do systems administration without giving them any tasks that they could actually do was extremely difficult. I had a lot of trouble getting anyone to sign on board with the committee or stick around for training or anything like that.
Eventually, I just gave someone privileges and I was like, “Don’t do anything bad but also if you do, it’s not the end of the world. I think that approach worked for her and she learned a lot but it’s definitely not the kind of thing I would imagine being able to teach someone in class. It really only worked because it was an extra-curricular and it was student-led. Mistakes weren’t going to ruin anyone’s lives or anything or take down an application in prod. It would just mess up some student’s projects may be for a few minutes. I think having that kind of low-stakes environment for someone to train in is really, really helpful and I think it could be recreated in a classroom, maybe. It’s not an impossible task. I just don’t think that anyone has on about doing it or putting on a training program, at least in my school, to get it done.
Corey: Yeah and this is somewhat heresy in the field of operations to some extent but if you take a look at what most companies do—there are exceptions to this of course—for most shops, a service going down for a few minutes is not the end of the world. It cost some money, it leads to some unhappy customers, but as a general rule, nobody dies because the Netflix movie won’t start, or the ad won’t display on a page, or you can’t get an update from what your friend is up to this weekend.
Now, there are exceptions to that. It’s, “Yeah, we go down, suddenly planes fall out of the sky, traffic lights stop working, and suddenly, the power plant is no longer regulated.” That’s a whole separate pile of nonsense you generally want to be careful around. But even on those environments, there aren’t too many systems that are on the critical path. This idea we come up with, that things must stay up at all times, regardless, in some ways isn’t helpful when it winds up keeping people away from being able to contribute or participate in coming up to speed on those things.
One of my favorite interview questions, when I was running ops teams, was always, “Tell me about a time you broke production,” and if people didn’t have a story like that, it generally meant one of a few things. Either they were lying about never having broken anything—let’s be honest because all of us have broken something—or they were in an environment that never trusted them to wind up touching the things that mattered. That wasn’t necessarily a deal breaker but it didn’t mean they didn’t have necessarily, I guess the scar tissue that would make them successful in a role where they were about to have that.
It’s one of those interesting areas where, as an intern, it sometimes strikes me as a bit of a weird model where when I dealt with interns in the past, some companies do a terrible job at it. They give them trivial tasks that don’t really matter the “busy work” and if they do fantastically well, or horrifyingly poorly, it doesn’t move the needle on anything. I have to ask based upon your talk, what makes for a good internship? If you’re considering building an internship program, how do you set that up for success?
Fatema: I think there’s a lot of different things that are factored to making an internship good but in terms of the work that you’re having your intern do, I think especially for ops teams you really want it to mimic what you would have like a junior engineer or full-time engineer be doing on your team. I think the problem with that is that interns are only there for four months so they don’t have the time to ramp up completely as a junior engineer might, but they still have enough time to do something significant. You could think of a project that they could work on that would in the last two months or three months after they finish ramping up, but you want to make sure that they are doing the kind of work that everyone else is doing.
For example, Yelp, there was an on-point rotation. Whoever was on on-point for that day, their name was in the topic of the IRC channel for ops, and then any developer that had a question for ops that day, the on-point person was the person that had to handle that question or figure out where to send them or take that triage or ticket queue or whenever. It was difficult, a hard job to do and it definitely wasn’t the most fun. But I thought that being part of the on-point rotation as an intern really help me understand lots of the different parts of the code base, lots of different responsibilities that the ops usually handle, and it was something that I think a lot of companies or a lot of teams could shy away from putting an intern on. It’s not something that has a lot of deliverables. It’s not presentable at the end of the day as I finished this chunk of code. It also is stressful and a little bit annoying to have to do that for another team but doing it is really, really valuable and it’s also something that would have to do when you are actually an ops engineer.
Make sure that your internship incorporates all that kinds of stuff. If you have an on-call rotation that your junior engineers would normally be a shadow on, then letting your intern shadow your on-call is also a good idea. If everyone has to triage the ticket queue every now and then, asking your intern to take on some of that work when they have loads in their productivity or whatever is also a really good idea. Generally not shying away from putting your intern on the kinds of things that you think a senior or junior engineer would be working on is good. Also, making sure that if they do have a project-project, something that they’re responsible for those four months, that is not something that’s going to get thrown away immediately after they leave.
Just because you think that it might be hard or really difficult or not doable in four months, any project can be scoped down. There’s no point of coming up with a project that is small enough for them to get done but is basically useless. When you could put them on a larger project, maybe that everyone else is also going to work on when they leave, and that way they learn a lot about doing really good hand-off, working within your team, and working with other people. Just make sure that their work is actually going to have an impact once they’re gone.
Corey: To be very clear, these are all paid internships, correct?
Fatema: Yes. I do not advise trying to hire unpaid internships in tech. That’s definitely not the trend that we’re moving towards at all. I think internships are just getting more and more highly paid, so it you’re offering an internship that doesn’t pay well, I don’t think any CS student is going to take it at this point.
Corey: And at least in the United States, in most states, the Board of Labor is going to want to have a conversation with you about your hiring practices. It seems to be an interesting problem to solve for in that when you’re building out an internship program, you’re trying to make yourselves competitive and get people to say yes to an internship offer, with no real expectation that after three months or so that they’re going to necessarily stick around.
Of course, the idea is to where the company to extend an offer after the fact and to get a full-time employee, that often is one of those, “Sure, in 18 months when I finish school, I’d love to come talk to you.” Companies generally don’t tend to think that far ahead. Do you find that that tends to wind up being a bit of a sticking point for companies as they set these programs up?
Fatema: I think it can be a sticking point for smaller companies like startups and medium-sized companies that haven’t actually started building an internship program yet. But I think that what you have to think about when you’re thinking about building an intern project is not to have tunnel vision to what will be good for your company or startup or whatever. Your company is one thing but the ops community and the site reliability community at large is another.
We really have to focus on getting young people into site reliability engineering especially, and also probably operations in general because there’s just not that many. Actually, don’t train the next generation of engineers, in general for any company, that you’re not going to have a pool of talent to hire from.
Thinking about it more as an investment in the future of the community unless there’s an investment in the future of just your company would probably help a lot with that. But I think though, the Facebooks and the Googles, they already have these internship programs and they already have literal billions of dollars. So it’s not much of a problem convincing them to run these internship programs but I think it definitely is harder to convince a smaller company to take an intern in operations.
Corey: Here’s a question I am hoping it doesn’t throw anyone under the bus. What makes for a terrible internship?
Fatema: The most common thing that I see in terrible internships or terrible intern experiences is bad mentorship. Usually, the way that these internship programs are set up is, you have your manager like your team manager, and then you probably also have someone in the role of your intern mentor or your intern buddy or whatever, and that person is supposed to be in charge of your success. They’re supposed to have come up with a project. They’re supposed to be talking to you throughout your internship, helping your build that project, checking in with you regularly, et cetera, et cetera, et cetera.
People sign up to do this and don’t have the requisite skills to be that person. Being a mentor is not just the technical work of teaching someone your code base. It’s a lot of emotional labor and it’s a lot of the skills that you would need to be an actual manager. A lot of mentors will sign up for this responsibility and not just be able to handle it. That leads to really bad experiences for interns.
I’ve known people crying at the end of an internship because they’re like, “I don’t know what my mentor wants. The expectations have never been clear. They’ve never given me feedback that I’ve been able to understand. Now this internship is ending and I don’t feel I’ve made a successful project and I don’t think I’m going to get this full-time offer,” etc, etc, etc. It’s usually just because the intern manager was not a good communicator, didn’t set their expectations correctly, and wasn’t able to pivot with them, their situation changed in the internship. I think the number one thing that will make for a bad intern experience will be a bad mentor and that’s just because people aren’t trained to be good mentors for interns.
Corey: At what point in an internship is it appropriate to wind up asking about the possibility of conversion or getting an offer or what the end of this is going to look like? It seems like a weird thing to ask before you find out where the bathroom is but I’m also a big believer in approaching things at the beginning with an understanding of what a successful outcome is going to be and magically pretending that, “Oh, yeah. This is just about having a good summer. It’s not about getting a job offer, my career,” or anything like that. It seems disingenuous. But you also don’t necessarily want to be off-putting. When’s the right time to have that conversation?
Fatema: I think that’s funny because I generally feel I’m very aggressive about this personally. When I was working in my first Facebook internship, after the second one-on-one that I had with my mentors—this is maybe two weeks in or three weeks in—every one-on-one after that I would wait until he was done talking or whatever and I’m going to be like, “So of this internship ended today, would I get a return offer, yes or no?” Obviously, for a long time, he didn’t really have an answer for that. He’s like, “Well, you haven’t done anything. You started working here last week,” but I just felt it was something that I had to ask regularly because that’s what I cared about. I was there to build my career and I wanted to be judged my development on the metric on what they were judging it on, which was am I going to get a return offer. I wanted to be synced up about that but talking to other interns has made it pretty clear that’s not something that everyone can do and is really not something everyone is comfortable with asking.
I think something that as companies and mentors should really, really be clear about right from the get-go is what this internship is about. You’re right. It is disingenuous for an intern to go in and be like, “Oh, I’m just here to have fun. I don’t care what happens.” That might be true but it’s probably not. it’s extra, extra disingenuous for a company to be like, “We’re just here to teach you operations and have a great time,” but no. You’re paying me, I’m an employee, I’m here to provide labor to you and you’re going to pay me for it and then you’re going to make a decision about my performance.
Corey: Yeah. Treating it otherwise is toying with people. It’s similar to the performance review thing where people care initially about a few things. One, am I getting fired? Two, am I getting a raise, and if so, how much? And putting those parts of the conversations at the end, after a 30-minute spiel of what you’re performance has been and individual examples, no one is listening or caring until they get to those particular points.
Fatema: Yeah, that’s true. I think as a company or a mentor or your managing an intern program, you have to be super clear with your intern about what’s going on. Is this the kind of internship where a full-time offer is on the table at the end? Is this the kind of internship where you get a return internship offer? Is your company actually having a higher increase so none of those things are available for your intern? You have to be really clear about that with them right when they get there and then whether or not you’re going to continually be talking to them in the frame of are you going to get hired or not? That’s a conversation you have to have with your intern. If that’s the kind of feedback that they’re interested in, then that’s what you should be providing.
Corey: It really seems to me that it comes down to a question of setting expectations up front.
Fatema: Yeah. I think that was the pivotal point of the talk that I gave was that you really have to be clear about the expectations and feedback all the time. I think anyone that’s worked in a company and had a manager will recognize that that’s true. There’s just a lot of bad culture around that, I guess. I don’t know how it is in other industries but in tech, it just seems really weird. We’re all supposed to buy into this idea that we’re just here to make the world a better place or write an app that helps rich people find the good sandwich. We don’t care about our performance or feedback or anything. That’s not true. It’s not true when you’re an intern. It’s not true when you’re a full-time engineer either.
When your intern is only there for four months and the entire time they’re super confused about what your expectations are and you’re not giving them any feedback, that makes way worse experience then. Maybe if they were a full-time and they have another four months after that to really buckle down and be like, “What’s going on,” it would have as much of an impact but when you’re in intern, it’s just really scary, especially because you’re probably young and you’ve probably not had that many jobs before which makes it extra intimidating.
Corey: And standing up for yourself in those scenarios is challenging for people even who aren’t in that precarious of a position. I mean, you see a scenario where terrible management patterns get amplified in internship scenarios. I will not name the person but the worst manager I ever had spoke only in metaphor. That works for some people but it doesn’t work for me. I think that as someone who’s running an internship program or managing people, you have the responsibility to deliver feedback on the same wavelength as the same person listening, and everyone wants to hear feedback differently. That’s something that often gets lost and I think winds up making some internship programs relatively inhumane.
Fatema: Yeah, that’s definitely true and it’s hard to get to know someone when they’re only going to be there for a short amount of time in your company, but it’s not hard to be direct and ask the questions that need to be asked. Just asking your intern, “Do you want feedback in a written form or do you want to have a one-on-one every week?” People never ask that and some people do prefer to get that kind of feedback in an email. But on the flip side, some people really do not want that. I personally hate getting any feedback because you don’t see the person’s face, which is really scared to me because anything can be taken in any tone of voice when it’s written down and I always assume that the tone of voice is negative. But for other people, it’s really what they prefer and that’s just one example of something you that you would want to talk about in one of those first meetings with your intern or any employee.
Julia Evans, she makes zines online and her Twitter handle is @b0rk. Anyway, she made a zine in really recently about talking to your manager about feedback. It’s really, really good and I think everyone should read it. It has a lot of really good tips about the fact that you’re a person and your manager is a person too. Sometimes you just have to work through these things together. There are ways for you to make your manager’s job easier and there’s obviously lots of ways for your manager to make your job easier. I really like the way she phrased it in that zine.
Corey: I’ll put a link to that in the show notes right there with you on that. I have to ask, did the internship programs in your case worked, by which I mean, are you considering taking a job with one of the companies in your internship or would you consider working with them or instead is it one of those stories of, “Well, after three months of learning how the sausage gets made, I hate all of those places and I’m going to go raise goats instead”?
Fatema: I had one internship that I don’t even mention in my bio and stuff. It was just my first internship and it was really, really bad. Basically, every internship I’ve had since that one has been pretty positive for me personally. I’ve gotten a lot out of them and I’ve really enjoyed the teams that I’m working on. Actually, I am going to be returning to one of the places that I interned at and the reason that I’m returning to that place, in particular, is because I had a really great experience with my mentor, I had a really great experience with all the other people that I’ve met at that company, and they’re going to pay me the amount of money that I want to be paid, which matters to me a lot, obviously, and isn’t talked about that much.
Corey: Is it considered taboo to talk about wanting to work for money? Yeah, except that that applies to approximately everyone. I hate this side of this model where talking about money is somehow considered uncouth. It doesn’t benefit employees at all. The companies know what everyone makes, internally and externally, because people tell them. The only people who aren’t aware are the people getting paid.
Fatema: Yeah. I try to make a point of sharing what I signed for at any particular internship or job or whatever offers I have pretty widely within my circle, at least, so that when other people are up for the same jobs or are negotiating similar positions, they can actually point to numbers of someone else’s gotten an offer or whatever. There’s been Twitter hashtags about it and (Jackie Wu) does a really good job of getting people to share anonymously what they’re making at different companies. I think that a culture shift is happening really slowly, which is good because I really think that it’s really important.
Corey: I would agree wholeheartedly. Where can people find more about you, more about your thoughts, anything interesting you have coming up in the near future that you want to talk about at this point?
Fatema: People can follow me on Twitter. My handle is @fatty_box. It’s mostly shit posts. It’s occasionally funny shit posts. You can also go to my website if you want to see any of the talks that I’ve given. If they’ve been uploaded, which a lot of the times those videos don’t come up very quickly, but fbox.ca is where I post those links once they’re up. I think my internship talk should be up soon. That one will be there eventually. The slides are all there, though.
Corey: Perfect. Thank you so much for taking the time to speaking to me today. Sounds like internships have really come a long way from the crappy version that I used to deal with back when I was coming up. It does sound like a very viable path forward if your company can sustain something like that.
Fatema: Yes, definitely.
Corey: Thank you once again. This has been Fatema Boxwala and I’m Corey Quinn. This is Screaming In The Cloud.