In this Q&A episode, guest co-host Amando Abreu and I answer listener questions about automatic rollback, how to prepare for an interview, and the meaning of life.
Solving big problems with small teams
Speaker 1: Ladies and gentlemen, the Tiny DevOps Guy.
[music]
Jonathan Hall: Hello, and welcome to another episode of Tiny DevOps. The show where we believe you do not need 1000 engineers to do world-class DevOps. Today's episode is a little different than what I've done before. We're doing a question and answer with questions that came from you, the audience. I have a co-host with me today, Amando, who will hopefully provide some wisdom and guidance in answering some of these questions. Amando and I have been friends for several years. We met several years ago when we were both living in Lisbon, Portugal.
Since then, we both traveled around the world quite a bit. Now we're in very different parts of the world, but he joins me today. He is the CTO currently of digital Expo, but Amando, why don't you tell us a little bit more about yourself?
Amando: Yes. Basically, right now I'm CTO of Digital Expo. Previously, I was just contracting. I've contracted for- as a developer, as DevOps, and several things. Before that, my longest employment was Trivago in Germany.
Jonathan: How long have you been involved in development? Has it been your entire career, I suppose?
Amando: I taught myself how to code when I was a kid, and I didn't actually realize I could turn it into a career. It's all I ever did, except for last summer, because I didn't have a job. Because I didn't want a job and I was just playing around in summer just in a camper van, but I ended up breaking my collarbone. I ended up having to get a job with a friend of a friend's girlfriend's uncle that owned a restaurant. I started off just as a kitchen assistant, but I'm not the type that can just do that. I ended up replacing the chef with a broken collarbone, by the way. I can cook all sorts of super traditional Norwegian dishes from the mountains, which people don't [unintelligible 00:01:50] Portuguese guy to be able to do. Here we are.
Jonathan: When's the movie coming out? That sounds interesting. [laughs] You're living in Norway right now. You're originally from Portugal, where of course, you and I met several years ago. Now you're a CTO. That's interesting. That's good. Today, we're going to go through some questions and answers or we're going to go through questions and hopefully provide some answers. I don't like to be the only person giving answers. I hope that we can make this interesting. Our first question today comes from Dave who asks, "Why is it so hard to persuade people not to put passwords tokens or other secrets in a Git Repository?"
Amando: It's a tough one. It's a mistake I've made. If people hadn't told me like, "Hey, be careful," because it's something that you don't really think about. You assume, because it requires quite some technical knowledge to actually do anything with it, but if it's in the public repository, yes, you're asking for trouble. Yes, otherwise passwords, app secrets, API secrets, et cetera. I'm not certain. It's an interesting one. People just overlook it completely, because it feels like information that no one can do anything with when it's in fact, clearly the opposite.
Jonathan: Yes, I would imagine that it-- First, many developers, I think probably every developer at some point in their career doesn't even realize it's an issue, right? As you mentioned. Once you do realize it's an issue, what's the alternative? That's not always obvious. You need to keep that stuff somewhere, probably. You don't want to keep it on a post it note on your desk. Where do you put those secrets? Of course, there are solutions that provide solutions now these days, but yes, they're not obvious, aren't they? If you want to use HasiCorp Vault or you want to use 1Password or LastPass or one- any of those sorts of things. I think my answer to the question is- why is it so hard to persuade people is first you have to educate them, and that's complicated.
Amando: Yes, it is. You said that there are tools to do it, and people put it in Git repositories. If you actually search for in URL, trello.com, and password-username, you find public Trello boards with passwords and usernames for Facebooks, Shopifys, eBays, et cetera, because sometimes people just don't know what they're doing. Development is complicated, so you end up spending more time in the actual development, and then you just overlook this, because you can't necessarily justify it to a project manager like, "Yes, it's going to take us a week to figure out where to store passwords. That's not something that you can easily sell, I suppose."
Jonathan: Yes, that's true. Actually, this is something I'm dealing with on a project I'm helping with right now. We have credentials stored in Git and we know we need to remove them. We haven't gotten public yet. There's no public users using the service yet, but we're trying to move that way in the next couple of weeks and we need to- one of the backlog items is for each service, remove the credentials from Git, recycle the credentials, so the ones that aren't in Git are no longer useful. Honestly, I don't know where I'm going to be storing all these passwords yet.
Some of them I'll probably put in-- GitHub is like- environment variables in GitHub. That's one easy solution. It's not very scalable, but it gets you a little bit closer to security than in the Git Repository. Because then it's only on Git it's not on everybody's laptop, right?
Amando: Exactly.
Jonathan: Yes, it's a really big hairy problem. I'm sorry, Dave. We can't answer this really clearly for you. We did talk about this, by the way, in Episode Two of tiny DevOps. My guest talked about the time that they accidentally committed some credentials to the repository and they made the decision-- It was a private repository, but it was still-- I think it was GitHub, but it was private. They still made the decision to scrub the repository, recycle those credentials at great cost. Just because they wanted to be careful enough not to expose any problems, if that would ever become public knowledge.
This one comes from Jerry, who submitted the question by LinkedIn. He says, "One thing I have discussions with people about all the time, is if developers become better DevOps engineers than those who are with an infrastructure background."
Amando: Depends how opsy the DevOps role is. I've been called DevOps for roles that were front end, and I've been called DevOps for roles that were just operations. In the just operations roles, I was a little bit out of my depth, because I'm not the best operations person, but the way I see it, DevOps is good enough to deploy, but not good enough to secure everything properly. Someone with a proper background in infrastructure is probably going to make a very ops-heavy DevOps role better than someone with a dev background.
Jonathan: Yes, I can see that. I've often been of the opinion that development is more important for DevOps. Before I make that claim, let me clarify what I mean by DevOps, because you started on that topic. When I think of a DevOps engineer, I'm really thinking of an operations engineer or a site reliability engineer or something like that. I know not everybody means that, but it is very important to be clear about what we mean when we use a vague word like DevOps. What I'm talking about is somebody-- In fact, I'll just say it like this.
I'm thinking of a site reliability engineer, which isn't always clear in everybody's mind either, but I think it's a little more clear than DevOps. I based this on the Google SRE book. That's what I have in mind. Not that that's a strict definition, but something in that vicinity of responsibilities. With that in mind, I tend to be of the opinion that a development background is more important than an infrastructure background. However, I wonder if I'm mistaken about this. Because there's a lot of infrastructure out there that developers don't necessarily know about.
I've spoken to some people recently who challenged me or challenged this concept. Developers don't tend to think about things like upgrading firmware on routers, and what version of the disk driver are you using, and does that SCSI interface with- those sorts of things the infrastructure people probably do think about more.
Amando: That's why I said depends on how opsy the DevOps role is. Yes, I've been out of my depth in some DevOps roles, and it required quite a lot of on the side learning and studying, but someone that would already have that background, then they wouldn't have to, but let's face it, 80% of the work was just development, not the ops 20% of the work was the ops, but it was quite deep ops. Yes, it's an interesting point, but you're still saying that development is more important. Then you're countering your own points or--
Jonathan: Yes, that's been my view or that's been my view or my approach up until recently when I've been challenged on this, and now I'm having to reconsider whether I was wrong all these years, so to speak.
Amando: Yes. It depends on each-- Each company has a different idea of what DevOps actually is. You saw the LinkedIn post I posted with the project manager poking the DevOps guy like, "Do DevOps." That was going to be a part of an article where I just asked people what they understand by each word, but it got too complicated to find an interface that people would just do it seamlessly and easily. I just ended up not doing it, at least not yet. If you ask what is DevOps to 10 people, you're going to get 10 different answers. It's funny that we talk about this now, because even at work, we're all annoyed that everyone has a different idea of what each word actually means. We thought about making a book that everybody could agree on, "This means that. We respect the book ISO-36164 for development terms. If you don't like it, go away."
Jonathan: [laughs] I like that idea. That would be very helpful with a lot of terms, not just DevOps. Product manager, product owner, agile.
Amando: Exactly.
Jonathan: Jerry, I'm sorry. Once again, we don't have a definitive answer, but we're being great consultants here and telling you the answer depends. [laughs]
Amando: It clearly [unintelligible 00:10:50] consultant backgrounds.
Jonathan: Another question from Dave, "Do you have an automatic process for rolling back failed deploys?"
Amando: No. It's the one thing that like-- It's important, but not urgent. It tends to just get-- It tends to get kicked down the stairs or what's the expression again? Yes, "Down the road." That's the one thing that-- every time something happens and we have to roll back, we think, "Okay, next time I have to write this process,", but then you never end up doing it, because other things have priority.
Jonathan: How do you handle rollbacks manually then? How are you doing that now?
Amando: Depends on which service. Each- some slightly different, but it's way too manual. I'm embarrassed to share.
Jonathan: That's why they hired you to fix it. What would the ideal look like for you? If you had the time to invest in this now, what would you go towards?
Amando: I would just do blue-green with either the blue or the green. I don't know which one comes first. Only deploy to a small amount of people. Then if that fails, you simply at the load balancer level, just stop it and send 100% of the traffic to the previous one, but this is not something for shipping every day kind of thing. It could be, but blue-greens are expensive. I'm sure there is a happy medium somewhere. Of course, the project manager in me just says, "Just don't deploy anything that needs to be rolled back." That's it.
Jonathan: Problem solved. We can close early today. [laughs]
Amando: [unintelligible 00:12:31] we did it, boys.
Jonathan: Of course, blue-green, won't work for every project. It depends on, you probably need at least two instances of your project for that to be applicable. If you have something which can't be run as multiple instances, that's a problem. My approach-- Of course, right now I'm not working on any projects that have automated rollbacks. The main project I'm working on is very early stage, as I described earlier. We don't have automatic rollbacks, but I suppose, the closest we have is automatic roll forward. In the sense that if you revert the code in Git, it will be deployed automatically. We're doing continuous delivery or continuous deployment, if that makes sense. If--
Amando: You could just open up your terminal, revert to a certain commit push, and it just reverts on the server.
Jonathan: Yes. That would, of course, revert it in the repository, which in turn would kick off the entire release process again and you'd have a new version on the server. I actually like that. I prefer that approach to actual rollbacks when it's feasible. It isn't always feasible, but I think it can be feasible in many cases. I like that better, because you don't have two processes to work on whether they're either manual or automated. It's just a simpler thing. You just have one process, every piece of software change goes through the same process. If you need to revert this, do it at the Git level rather than worrying about your Docker images or whatever it is rolling back. What do you think?
Amando: How would an automated process actually be? Is it something that- I don't know- measures the amount of 500s and if there a spike after a release, it does all this automatically? What you described is automatic already, but is it something that the texts that it needs to be rolled back or just push of a button and it rolls back? You essentially just described that by reverting to a previous commit.
Jonathan: In this case we don't have anything automatically doing a rollback. We need that dictionary again to define automatic. What are we automating? Are we automating the detection of failure or just the resolution once it's detected? What I just described does not detect all my activity 500s or, "The service is down, so roll back." I don't know if Dave was asking about that level of automation. One other question from Dave, "How do you find meaning and satisfaction in the indifferent existential vacuum of modern life?
Amando: That changed the tone. It's just a simulation. Nothing really matters. It's a tough one to answer, but just do what makes you happy? Don't do drugs. Stay in school.
Jonathan: I try to avoid the indifferent existential vacuum of modern life by focusing on my family and friends and aspects of life that are not indifferent. To each his own. I'm sorry Dave, if that's your outlook.
Amando: This is essentially why I want to have kids.
Jonathan: So somebody will like you? [laughs]
Amando: Instead of finding friends, I just make them now.
Speaker 3: What do you want to be when you grow up?
Speaker 4: I want to be married and have a 100 kids, so I can have a 100 friends and no one can say no to being my friend.
Amando: Actually, I do have a good answer. I think creating is the answer for everything, because you're made to create. Except on the seventh day, we need to rest, but you're made to create. Software developers make things, but it's very abstract. You can't hold it, you can't grab it. When I converted the van into a camper, it was the first time I did things with my hands for a long time. To actually make a camper van, it's not easy. Then the first trips I went on, I'd just press a button and my heater goes on in the camper that I built myself. That's pretty cool. I think creating is the trick. If you're feeling this existential vacuum, maybe you're going through a phase of just consuming too much and you just need to close the books, close YouTube, close Netflix, and just go do something. Actually create something, make something. Could be, kids could be a house, could be a camper van, could be a drawing, painting, photography, whatever, but you need to create something. Something that you can grab is better than something you can't.
Jonathan: That's an amazing answer. I'm going to have to change this podcast from Tiny DevOps to Tiny Philosophy DevOps or PhilDevOps or something. [laughs] What do you do for interview preparation? You've recently landed a new job. Maybe this is fresh on your mind.
Amando: I don't. I actually got this job-- It's an interesting one, because I just read- I came across one of your newsletters, which was pushing back for the code test. That was-- I just do that naturally, because I'm lazy, but for certain roles, that's actually a good thing. This happened to be one of them, so yes. Basically, they were like, "Yes, we're interested. We want you to go for the next phase. Please do this case study. Please create this presentation, and then present it to these people," and so and so. I was working full-time at the time and I didn't need another job. I was like, "Actually, I'm not going to do that. It's too much work. Why don't you just put me in front of the team and everyone can ask whatever they want. If they like me, then good. If not, no."
Then they said, "Yes, but all the other people that applied have already done this process with the case study and the presentation. We think it would be unfair for them." I said, "I also think it would be unfair for them. Let me know if you need an architect, sometime down the line." Then yes, I thought that was it. Then a week later they were like, "Okay. We'll sit you in front of the team and they can just ask you questions."
Jonathan: That's a nice story.
Amando: Then to prepare for that, there was some new concepts to me. There were event-sourced, and CQRS, and all these shiny things. I had to prepare for that a little bit, but I'm honest. I said, "Okay, this is the first time I heard the terms. This is what I read just now." Then explain it to a golden retriever. If you can explain it to a golden retriever, you can go deeper as you need to. I haven't had to go deeper yet, because my position is super high level. I know enough about it to explain it to a golden retriever, which sometimes there are a lot of golden retrievers on the board.
Jonathan: Let's hope none of them are listening to this episode, right? [laughs]
Amando: They know I say this, but it's okay.
Jonathan: What kind of research did you do on the company before you accepted the role?
Amando: Yes, of course. I read the website, and of course my first interview is prep for the next one. You read the website that says one thing, and usually it's a way fur-- Especially in start-ups, what they say on the website is way further ahead than what they actually have. A good question is what you actually have, where do you want to go? Size of industry, stuff like that. A good one is doing some background on the people that are going to interview you. You can check Facebooks, Instagrams, LinkedIn, et cetera, find stuff to build rapport. Yes. If you connect with the people, usually that's what it takes to land the job. Not in a psycho way, by the way. Just like an actual genuine interest, friendly way.
Jonathan: I think you and I probably think fairly similarly. When I'm preparing for an interview, I basically don't. Part of that is my own philosophy. I think there's two aspects to it. One is I feel like at this stage of my career I don't need to study for an interview. Maybe when I was fresh out of school, I would have felt I needed to. A lot of people do, but at this stage I've been programming for 15 years. If I can't do an interview now, I'm never going to do an interview properly, so I don't feel the need. Second, I strongly feel that the purpose of an interview is for me to get to another company and the company to get to know me. If I'm putting on a facade, that's not going to happen.
Amando: Sorry to interrupt you here, but I worked with an American guy in Dusseldorf called George. He ran a comedy club, an Expat comedy club. Hilarious guy. Then he wants to go into project management, so he took the PMP course, et cetera. The interview, he went wearing a suit and he was very serious, and that's not who he is. That wasn't his authentic self. Then they turned him down, because they said they thought he was too serious, but he runs a comedy club. [laughs] He's not as too serious guy. He just acted serious to try and get a serious role, but he was turned down for being too serious.
Jonathan: I wore a suit to one job interview early in my career, and I don't know why they turned me down. It may have been for the same reason, but I know I was so nervous at that interview, and I've never worn a suit before or since. I've never felt as nervous before or since.
Amando: I like wearing suits. I like to make the effort, but if you're clearly overdressed you can always just take your blazer off and fold your sleeves back and there, you're casual-ish.
Jonathan: Back to what I was saying, I don't prepare in the sense of cramping or studying ahead or perfecting my smile or practicing a speech. I don't do any of that stuff, because I don't think it's my genuine self. Generally do at least a minimal amount of research, maybe not for the first interview, because the first interview is often just a screening and they don't know anything about me. It's okay if I don't know about them, but by the second time I should at least know what products are selling and who the CEO is. Some general things like that. I think the most important thing I do is I have a mental checklist of what's important for me and what are my deal breakers. I'm really bad at this sometimes, especially the what's important for me thing.
I downplay a lot of those sometimes. I'm talking to somebody like, "What salary do you require?" For example, I'm like, "I would like to maybe-ish have this salary." It's important especially after you've had a job or two, and you're not just looking for the first job that will come along. I think it's important to know what salary you expect, to know if you're willing to commute and how far. To know whatever is important to you, to know those things and be firm. Polite, but firm about those in the interview. Don't be wishy-washy and pretend that, "Yes, that's 5,000 less than I want, but it's probably okay as long as we can negotiate on-- Is there a ping pong table," or something like that.
Amando: Yes. I don't think so much about what I want, because what I want that I know, so it's okay, but it's usually like you mentioned like a pricing models. Did you mention pricing models? Maybe you did.
Jonathan: No.
Amando: A pricing models strategy, goals, they actually have. Their current MRR is X. What would they like to go to? This also shows a certain business mindedness, which for a role like mine it makes perfect sense, but for a specifically developer job it might not, but it's also good to be budget conscious. It's all about money in the end, yes. To avoid the topic doesn't make much sense.
Jonathan: One last question, is it's related to the job search. It's from a different person. This one is from Joshua, and he says he's in school. I'm just reading through it here. "If you don't mind me asking what is the best company to work for as a beginner? I'm trying to look to get in a couple of company types." He's considering a consultancy, where he can work with B2B clients. He's also looking as a direct hire as a DevOps engineer for more "Real world experience," he says. First, he says that learning is more important than the salary, because he believes if he has enough skills, he can learn more in the future. I would agree with that. Then he says he's concerned about switching jobs in the future. If he works as a consultant, will it be harder to transition to an engineering role in the future? There's a lot there. The general question is as a beginner out of school, what kind of jobs should you look for?
Amando: It's a good one, because consultancy is attractive because of the money. Usually you stay at a company for a few years before getting into consultancy, because in consultancy you make choices, but you don't see the consequences of those choices, because it's usually shorter lived employments, like six months a year, et cetera. Whereas if you're in the company for two, three, four years, you get to see the consequences down the line of the choices you made. That makes you better all around, and then just making bad choices everywhere and never seen the consequences of it.
Jonathan: That's really good insight.
Amando: Yes. Of course, salary for consulting is good, but you can always go to a company, be an employee for 4 or 5 years or not even so much. Two or three is probably good enough to get a good idea of it, and go into consultancy after that.
Jonathan: That's really good advice. That's an angle I had not thought of.
Amando: Really?
Jonathan: Yes. I've never worked in a consultancy per se. I don't know if you have, but I do consulting now, but after having had 15 years experience.
Amando: It also depends on what you mean by consulting.
Jonathan: That's true. [laughs]
Amando: Do we need that dictionary again?
Jonathan: We do.
Amando: That's the the third time.
Jonathan: That's it. [laughs] All right. I think we've made it through our list of questions for the day. Thanks for joining me today. It's been great. Remember, if you have any questions that you'd like me, Amando or some other future guest host to answer, please send them to jonathan@jhall.io. I'm going to say that again, because of my tongue got messed up. If you have questions that you'd like me to address on this show with possibly Amando or some future host just send them to jonathan@jhall.io. Once again, if you record a video or an audio of yourself asking the question, it'll go to the top of the list, but a text format is perfectly fine too. Until next time.
[music]
Did you know you can watch this episode and other Tine DevOps content? Search YouTube for Tiny DevOps to see all of my guests' beautiful faces. My thanks to Riley Dave for the Tiny DevOps theme music.
[00:27:06] [END OF AUDIO]