Subscribe
Share
Share
Embed
There are a lot of choices for managing and encrypting secrets in Kubernetes. Kamus is one of those solutions, and it was developed as an open-source project by Omer Levi Hevroni.
Today we’re talking with Omer, a DevSecOps engineer with Soluto at Asurion, about his work on Kamus, its origins, and how it’s being applied for secrets management in Kubernetes.
Screaming in the Cloud with Corey Quinn features conversations with domain experts in the world of Cloud Computing. Topics discussed include AWS, GCP, Azure, Oracle Cloud, and the "why" behind how businesses are coming to think about the Cloud.
Transcript
Corey Quinn: Hello and welcome to Screaming in the Cloud, with your host, Cloud Economist, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of Cloud. Thoughtful commentary on the state of the technical world and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.
This episode of Screaming in the Cloud is sponsored by O'Reilly Velocity 2019 Conference. To get ahead today, your organization needs to be cloud native. The 2019 Velocity Program in San Jose from June 10th to 13th, is going to cover a lot of topics we've already covered on previous episodes of this show. Ranging from Kubernetes and site reliability engineering over to observability and performance. The idea here is to help you stay on top of the rapidly changing landscape of this zany world called, Cloud.
It's a great place to learn new skills, approaches, and of course technologies. But what's also great about almost any conference is going to be the hallway track. Catch up with people who are solving interesting problems, trade stories, learn from them, and ideally learn a little bit more than you going into it. There are going to be some great guests, including at least a few people who've been previously on this podcast including Liz Fong-Jones and several more. Listeners to this podcast can get 20% off of most passes with the code CLOUD20. That's C-L-O-U-D-2-0 during registration. To sign up, go to velcityconf.com/cloud. That's velocityconf.com/cloud. Thank you to Velocity for sponsoring this podcast.
Corey Quinn: Welcome to Screaming in the Cloud. I'm Corey Quinn. I'm joined this week by Omer Levi Hevroni, who's a DevSecOps Engineer at Soluto by Asurion. Welcome to the show, Omer.
Omer Hevroni: Hello Corey, and thank you for having me. It's a great honor to be on this podcast. I'm a huge fan.
Corey Quinn: It's super kind of you to say that. So, in addition to the day job, or I guess in component with the day job. You wound up releasing an open source tool called Kamus, which I'm almost certainly mispronouncing. There's some that are going to say it's called Kamus, some will say Kamus. It's K-A-M-U-S, which out of the gate is already a better name then almost anything Amazon has ever given to anything. What does it do?
Omer Hevroni: Let's start with the name because I guess, I don't know how much of the people hearing us right now are speaking fluent Hebrew. Kamus in Hebrew means a secret. And we called it Kamus because it's a tool that helps managing secrets on Kubernetes. We all love secrets and we all want to keep them confident. And Kamus come to solve exactly this problem. So this is like the very high level of overview. And it's different from all the other tools because Kamus let you encrypt a secret to a specific application. So only this application can decrypt it. And this is a very unique approach to secret management.
Corey Quinn: That makes a lot of sense. So take me back to the beginning here. There are a number of different ways to address secrets management of varying degrees of validity. Some make an awful lot of sense, some are less sensible. It ranges from using GPG which is heavy and difficult to wind up encrypting secrets that then applications wind up retrieving. There's just embedding it in your source code and hoping that no one ever finds it. There's putting it in secure S3 buckets and just acting shocked when that winds up getting discovered. And fifty thousand other horrible things you can do.
Corey Quinn: But then there's good stuff. There's things like AWS systems manager parameter store. There's systems manager in the AWS side. I have to assume Azure and GCP and the rest have something vaguely similar. What does this do differently?
Omer Hevroni: Yes, you pretty much covered are the different options and I think when thinking about what's the end solution to choose, we need to think about the quality of the company you're working at. For example, the way we work at Soluto is a way we like to call “super devs,” which means developers who do everything. They do the fine work of writing the code but they do the hard work of deploying it and maintaining it. And, when you have developers who do everything, you need to choose a solution that on one hand they can use easily, and on the other hand they can use easily but still be secure.
So, for example if you go with GCP like you mentioned, you need to find a “best” solution that would let all the developers have access to encryption keys and keep it secure and all that, and it gets complex very fast. This is why I'm saying it's the culture or the way you work, the way you work is a very important factor when thinking about which solution to choose. So pretty much you can think about different approaches and approaches are like the most of the GitOps which is I think one of the hype buzz words existing today and maybe GitOps and Bitcoin and blockchain.
GitOps is the idea of managing everything via Cloud. So, describing your cloud architecture and everything else. When you do things with GitOps, you get lot of benefits of Git, like auditing and roll-back and all that good things. And this is why I like also to manage secrets [inaudible 00:05:32]. So, solutions like AWS KMS or Azure Key Vault which is pretty much the equivalent of Azure to KMS. A really good choice, but they don't have any GitOps support. So if you go on this path you need to solve also authentication, authorization, and a lot of other things that you don't want to mess with. So I think this is pretty much the reason not to go this way.
Corey Quinn: There are a number of different things that I will look at whenever I see something that purports to handle security and discount them almost out of hand. The easy low-hanging fruit one those is rolling your own crypto. One of the nice things I like about Kamus is that it uses whatever KMS version is being used by underlying provider — to wit: that's Azure, Keybolt, KMS, there's Google Cloud KMS and AWS KMS for the big three. You’re not generally rolling your own crypto solution here with the singular exception of running your own keys for just local development which you very clearly call out in the documentation. But that is not something to be done unless you are in a hurry for testing purposes. So, first can I just congratulate you on not doing the most ridiculous thing imaginable?
Omer Hevroni: Yes, at first we tried, actually this is a pretty interesting aspect. The whole idea of Kamus was to say, "Hey, I have no idea on how to do encryption good." I do security and I do OpSec for two and a half years, but still I'm saying out loud I don't have any idea about [inaudible 00:07:05] and I don't want to do this things alone. So this is why we prefer or to have [inaudible 00:07:10] log to the cloud. And the interesting part is that in this area GCP was the best experience. And GCP will have no control in the [inaudible 00:07:29].We just told the KMS we kept this and we kept this and you have zero control about what happens behind the scenes.
In my opinion it’s the best experience, because there are less chances you’ll make mistakes. For example in AWS you need to handle the encryption yourself . So the master key is encrypt in AWS, but you are doing the data encryption in your code. And I guess you might have a mistake there with the try really hard to do good but you know mistakes happen. This is very interesting aspect of looking at all the tech clouds and looking which one give the easiest developer experience.
Corey Quinn: One thing that I found fascinating about Kamus, compared to an awful lot of other projects that have come out of the open source world is they start off being built for AWS and that's it. Then eventually they add a second cloud provider, begrudgingly, and then it sort of always feels like a secondary thing. This came out of Azure first and then, I think, correct me if I'm wrong, then it was GCP second and AWS third?
Omer Hevroni: Yes, we started with Azure, because it’s what we work with, with Azure it is the cloud we use the most. We started with the cloud we were using because we first wanted to make sure that Kamus is usable and its working as we expected. And then when we were going to open source it was clear that we can't support on the Azure, if we want people to use it, because unfortunately not everyone love Azure as we did. So, yes then we added GCP because someone asked for it, but then never use it, which was pretty disappointing.
Corey Quinn: That sort of at times feels a bit like the GCP story, but that feels a little mean past a certain point.
Omer Hevroni: Yeah it was really disappointing, because I did it really fast and hoped that he would use it and then he vanished. So if you hearing us and still waiting for your review of the GCP implementation.
And then two months ago we had a AWS, which as I said was the hardest, I think, to implement. And we now support all the three major cloud providers and we look forward to add support for other KMS, so Hashicorp, Vault, ,or anything elseyou want support of, it’s very easy. Choose your service and you’re done.
Corey Quinn: Something else that I found fascinating about this is that unlike virtually everything modern that is written in Go or everything designed that is written in Rust, but never actually built, because no one wants to stop talking about Rust long enough to write anything in it, you wound up writing Kamus in .NET. Why was that?
Omer Hevroni: Yes, well historically I started developing in .NET and I stick with it. And it is still the language that is the most easy for me to write in. But .NET is actually pretty awesome language and lot of people hate it because its belong to Microsoft. But since .NET went open source it's become really easy to use and really handy tool. It’s still a bit heavy compared to other tools. A simple project has a lot of files and all that. So compared to one Golem file its a lot more heavy but you have a lot of things that are harder to do in other languages and this why I still like it. It was a personal choice because choosing a language that not a lot of people will use mean that you will have less contribution.
I already have some people saying I will maybe consider contributing to it, but I'm not sure how to write things in .NET. So there are some downsides to choosing this language, but this was what's easy for us at that time. And maybe in the future we will consider writing in Golang or something else so we can have more people be able to contribute to Kamus.
Corey Quinn: Realistically I find that picking a good language makes sense. When there is a compelling reason to do so and you just named one of them, where having an ecosystem of people in the space who are able to contribute to a project. That's a viable strategy, but I have zero tolerance these days, or frankly most of my career, for language bigotry. Where, oh, this is an amazing tool that does the thing I want it to do but it is written in Pearl so nevermind it must be crappy. Or, oh, it is written on the wrong version of Python or, oh, Ruby is or children. We're not going to go and use that at all.
And none of those statements are ever true. And none of those statements are ever helpful. It feels almost like it's a form of gate keeping, where the entire purpose is to make people feel bad about what language they have chosen to write things in. In fact, if we ought to go by the sheer number of expensive things that will break, “written in Java” is pretty much the clear winner. Between that and still the things that run our entire world are written in COBOL, I don't think that there is a compelling argument to be made that only the cool modern languages are the ones that matter.
I'm a big fan seeing things that aren't written in the usual suspects. And this is absolutely one of them. I've taken a bit of a look though the code base it seems clean, it seems well written, I understand what its doing despite not being conversed in a .NET myself, which tells me that either the language is not that hard to learn or you've gone significantly out of your way to make this approachable to people who aren't just you.
Omer Hevroni: I think this is a fair point to touch. For me its still very hard to write Golang code, and I am working with Kubernetes for more than a year, so I had to write a lot of Golang code. I think it's the language, but I cannot say that for sure, because I don't have enough experience with Golang, but again its just only my experience. Golang still very hard for me to understand and write and read. And this is places where I think language don't matter. If you have a language that you know well and you know all the tiny points of what to do and what not to do, its very hard not to use that language and use something else, because you will probably produce a better product. So, there are places where languages matter and its not what you control and not what is popular.
Corey Quinn: The problem that I've always had with popularity driven development is that it leads almost to group think. We're starting to see this, not just with languages, but with other things as well where if you're not using AWS you're clearly doing it wrong. I don't know that is necessarily true. Or if you're building any monolith that no microservices are the way and the light of the future. Or, oh, if you're, not going serverless you're completely missing the entire point of the modern revolution. And I don't like any of those things. They smack far too heavily or orthodoxy and its getting away from solving business problems and into the realm of religion. And that doesn't seem to be serving anyone particularly well in a technical or a business context.
Omer Hevroni: Well, I'm a religious, Orthodox Jewish so I'm pretty fine with being orthodox and going with religion. [Laughter]
Corey Quinn: There’s validity to that! The question is at some point it blurs the line between things we have to take on faith and I think that religion is inherently going to be one of those, versus things that lead to definable business outcomes. And doing something just because the thought leader on the stage said it was the way to do things, that's always been a problem for me. Because our choices are built by constraints in a technical context and the fact that we don't wind up making decisions based upon other people's constraints or even our own constraints leads to disaster.
If someone gets on stage like what they did back, in I wanna say 2014-ish, and talks about how Docker is amazing and terrific and solves everyone's problems, well it didn't. What about state full workloads? What about networking? What about orchestration of these things? What about monitoring? How do you admit statistics? How do you troubleshoot when the thing that's running was turned off twenty minutes before you knew that there was an error? And there was a giant list of problems that has largely been solved today, but at that point the whatever your problem is Docker containers are going to solve it, was what everyone was saying and I didn't' find that resonated with me, because just throw everything away that I built previously and build everything again and this new context, this new environment.
I could see doing that, but I didn't see a compelling business case for it. And now here we are six years later almost and we're talking extensively about Kubernetes in some cases being the answer to everything. I have a problem that I may possibly have. I don't see it. There are absolutely use cases for which Kubernetes is a valuable path forward and a great use case. But certainly not all of them. And the level of complexity still feels slightly higher than most companies are probably going to want to dive into, unless they are themselves tech companies at heart.
Is that something that you're seeing? Do you agree? Disagree? Don't feel you have to agree with me because I'm the one with my hand on the mute switch.
Omer Hevroni: No, I'm pretty much agree. I think its level of maturity as a software developer or a security developer or as any other dynamic, but the other parties not always answering for what is good but for what is not good — it hit me first when I heard a really good interview question: when you need to answer when microservices are not a good choice. And there are pretty good answers to this questions, but I hear this question and it got me thinking. It wasn’t my job interview but I heard a little of, and I started thinking it was pretty interesting thing to think about, like what situations are not good use cases because microservices but as you say it was very interesting because it was going against the dogma. And starting to say, hey, microservices are not always that good and has a lot of overhead. And we take it as a fair point. I actually like the example of Kind, it is an open source that lets you run Kubernetes using Docker and its very good in CI.
And what I liked about Kind documentation website is that they have a whole list of alternatives that are similar to Kind, but do other things. And you can look on all of them and chose between the alternatives and find the one that best works for your use case. And I wanted to do this on the Kamus documentation website and I forgot, I need to do it. But I think it is very important to say, like, what you are not good for and what use Kamus, or Kind, or Kubernetes, or whatever.
Corey Quinn: I absolutely love the idea of an interview question. Where there's this thing that you're good at, that you're passionate about, great. Tell me about it's shortcomings. Tell me about when it's not a good idea. Because that tells you a lot about a candidate. Just as far as can they see both sides of an issue, are they so blinded by zealotry for this thing that they are in love with that they can't see another perspective? And if you wind up making a technical decision down the road, well, they're working with you, are you suddenly going to wind up having a screaming meltdown? Because that is not acceptable. Finding ways to tolerate and deal with disagreements is incredibly important for almost every environment I've ever been in. So I just wanna say I love the idea of asking that kind of question.
Omer Hevroni: Yeah we can do it on Kamus. Kamus ever since it start it's not good for everyone. If you already have Vault and Vault works for you and you have a culture where you have a specific team who will manage the secrets, it might be easier to continue using SSM or GitOps might work better for your solution. Another caveat of Kamus is that Kamus do one-way encryption. Once you encrypt the secret you have there is no admin account lets you decrypt it back, and its not work for everyone. For us it was good enough, but it had downside. Sometimes you this file decrypted and Kamus, by design don’t do that. So it's important to understand the idea behind the design of Kamus and understand this idea fit your workflow or not. Because it's not always fit.
Corey Quinn: Pay attention listener. The reason to look into Kamus is because you heard on this podcast, yeah that's a good idea. Deploying it blindly, because the super confident sounding people on the podcast said you should, that's a bad reason. It all comes down to making sure to whatever you are using works for your use case. Understand what it is. Understand how it breaks. Understand how when it breaks, because everything breaks that's what computers do, will impact your production environment. It all comes down to understanding constraints, understanding trade-offs.
Omer Hevroni: Yeah this is part of the reason we decided to release with Kamus. A full-blown [inaudible 00:20:14] of all the different threats and controls we think about. So you as the user can look at that and say hey this is an unacceptable threat by us. And I don’t going to deploy Kamus and have this treat available, I’m not going to go with it. And you can look at the controls and understand the different things we decided to do in order to mitigate them and you don’t like some of the controls or you might even think about threats or controls we didn't think about. So it’s all part of the idea of giving our users all the information, it’s all there for them, and with all this being said, go use Kamus it really is amazing. Don’t listen to all the things I said before!
Corey Quinn: Exactly! Adoption where it all comes from. Sooner or later we'll put a foundation around it and get people to start supporting it giant corporate piles of contributions. Changing gears a little bit… You introduced yourself in the beginning as a DevSecOps engineer. And I tend to come from an old school world. Where I started my career as a grumpy Unix Systems administrator. We also called that a UNIX Systems administrator, because they are all grumpy. There is no second kind. And over time I watched the world evolve. We started calling the same thing I was already doing DevOps. In many shops they started calling it SRE, or cyber liability engineering. And it feels to me, and its probably slightly naive, that the role hasn't particularly changed in fifteen years. Sure, the tools we use are different. But if you say, well, back you were a SysAdmin you didn't have to write code. I didn't. I was doing an awful lot of stuff in Perl and if you start working on UNIX systems administration you generally in that era and won't get very far without a working knowledge of C.
Corey Quinn: And it turns into, it seems everything old is new again. So then we called it DevOps, which people argued vehemently is a title, is not a title. Well at least we’re debating rather than doing work. We don't want to do work. We want to argue with people. And now we are seeing DevSecOps as adding words to that construct, which on the one hand I like. The idea that security has to be baked in from the beginning is a fundamental tenet of safe systems design. And I'm curious, though, how did you wind up coming by that job title? When did you start using it?
Omer Hevroni: It’s complicated like all the good things. The funny part is that today I'm part of the DevOps team at Soluto. We talk a lot with our manager about changing it to SRE, and ask him if I change my title from DevOps to SRE what will that increase my salary. And he refused to answer this question. So today I'm not SRE, but if it will increase my salary I will.
Corey Quinn: Absolutely — to an extent there have been a number of DevOps days where they’ve done job title polls and going from SysAdmin to DevOps or SRE was something on the order of a salary boost of at least a third taken in the aggregate. So it's weird, but the things we call ourselves change the way people view us and more importantly how people pay us.
Omer Hevroni: So I think I will go with DevSecOp SRE. I think it will be good enough.
Corey Quinn: I like that. FinOps in there too somewhere. I think someone is trying to make that happen somewhere.
Omer Hevroni: Yeah and the less funny part is that I don't have any official degree. So I am a bit ashamed to call myself engineer. But you know it’s a different discussion about the important of doing a bachelors degree in computer science and all that.
Corey Quinn: Not necessarily, I have an eighth grade education myself. I have always toyed with the idea of getting a computer science degree at some point in my career. And there aspects that will be helpful for it, but from my perspective it seems for some folks it’s more of a credential. In my twenties it was rough not having a degree. I had to talk my way around a lot. Now that I'm in my thirties I can bring this up in public, that, hee-hee, I don't even technically have a high school diploma. And it’s a punchline it’s not a career limiting move. By and large careers do get easier. With the right credentials. But I don't think that its ever absolute. At least not as we tend to think of it as. It just means we have to be a little bit more creative.
Omer Hevroni: Yeah and my parents are somewhat disappointed and hope at some point I will get the sense of it and go and do an official degree. But it just too hard. I never managed to get over the hard math courses, so I don't think it will happen any point.
Corey Quinn: I'm in the same boat. My parents still think I'm a an accountant. Which, yeah, we're gonna go with that. It’s a lot easier to explain.
Omer Hevroni: Anyway the job title is complex. I started at Soluto as a developer. And then someone offered me to be part of the security champion — you heard about the idea of security champion?
Corey Quinn: I have not. Tell me more.
Omer Hevroni: OK. So it’s not that new but it's coming from the idea that the security team, especially in companies like insurance, usually very, very isolated, located somewhere in the United States. And they cannot control all the developers all around the globe. So usually the solution to that is either hire a lot more security people, who will be located in each site, which is very hard, or taking developers and allowing them, converting them into security. Taking them to the live site. So this what they did. I did a SANS course of one week, so I have a certificate. I’m officially a pen tester and I can use this title if I want. And since then I started to do all things related to security. OpSec, and all the fun stuff DevSecOps which going from security testing to running things in the pipeline and the things I like the most, my job in the DevOps teams, I think fundamentally different from SysAdmin is what I started with.
We have a lot of developers and our job as the DevOps team, especially my job, is the security person is to help them and guide them. We don't have any more control than what they do, we cannot tell them what is wrong and what is right. We can just help them and provide them with best practices. And if they come and say, “hey, want to use this cool new technology,” we are the bad guys who come and say what about monitoring? How can you monitor that? — how can we deploy that? Is it secure, and all that question. And these things I think are fundamentally different from a SysAdmins. Because SysAdmins usually were not responsible to do this work and this why I like the title of DevSecOps engineer. I’m not that happy with this title, but I think it’s better than security champion. Which was the title I used before.
Corey Quinn: I think that may wind up being for the best. All things considered. We're about out of time, but if people are interested in what you have to say where can they find you to learn more?
Omer Hevroni: So, the best place my Twitter account. I try not to tweet too much in Hebrew, but I do it also. Its my language and I try not to put so you can follow me there.
Corey Quinn: That’s O-M-E-R-L-H — we'll put the link in the show notes.
Omer Hevroni: Yeah and I also have a personal blog, which I try to post new things once in a while. And you can also find me there or find me on Kubernetes Slack, or CNCF Slack, I'm on each one of them. And of course GitHub, and all the other things. You know some people have, and of course LinkedIn. I think some people are also there.
Corey Quinn: Sound good. Thank you so much for taking the time to speak with me today.
Omer Hevroni: Thank you for having me. It was a real honor, and as I said in the start I'm a huge fan of you. It was pretty nervous to be on this podcast after the last episode with Jess.
Corey Quinn: Yeah but you’re talking about, there's always a bigger fish. I gotta say I was nervous as heck talking with Jess. She's incredible. And everyone I talked to is incredible. But she's one of those people I've been following for a long time and it turns out everyone has someone that they look up to, oh I could never talk to them. I’m never gonna be in their tier — there's always a bigger fish. And something I’m learning is that reaching out and talking to people and asking for help, everyone worth admiring is a human being who is willing to have a conversation. Don't ever hold yourself back from that. Omher Levi Hevroni, DevSecOps engineer at Soluto by Asurion. I'm Corey Quinn, this is Screaming in the Cloud.
Corey Quinn: This has been this week's episode of Screaming in the Cloud. You can also find more Corey at Screaminginthecloud.com or wherever Find Snark is sold.
Speaker:
This has been a Humble Pod Production. Stay humble.