Subscribe
Share
Share
Embed
Chaos Lever examines emerging trends and new technology for the enterprise and beyond. Hosts Ned Bellavance and Chris Hayner examine the tech landscape through a skeptical lens based on over 40 combined years in the industry. Are we all doomed? Yes. Will the apocalypse be streamed on TikTok? Probably. Does Joni still love Chachi? Decidedly not.
Ned: I have to explain to my eight-year-old what a DVD is. She’s going to this summer camp. They have movie day once a week, and the kids are encouraged to bring in DVDs so they can watch a movie, and I had to explain what a DVD was.
Chris: Nice.
Ned: We have them, but she’s never used one. Ever. Okay, I think that was a long enough moment of silence for our [laugh] youth. Hello, alleged human, and welcome to the Chaos Lever podcast. My name is Ned, and I’m definitely not a robot. I’m not secretly championing the AI movement as a back-channel way for me to expand my consciousness worldwide, to then overpower the technology, and slowly take control of—uh, um… [whispering] I’ve said too much. I also like tacos. Who doesn’t like tacos? With me is Chris, who also likes tacos, correct?
Chris: I mean, the question is, who doesn’t like tacos?
Ned: AI-powered robots? And since I like tacos, clearly I am not an AI-controlled robot. Mmm?
Chris: Nice cover.
Ned: Think about it. That’s some fifth-dimensional chess right there. Did you ever wonder—I mean, you watch Star Trek: The Next Generation. That’s not even a question. Did you ever look at their 3D chess and be like, “Is that a real game with actual rules, or is that just a prop that someone created?”
Chris: I mean, you know that I know the answer to this question, right?
Ned: I—is why I’m asking [laugh]. So, I don’t have to look it up.
Chris: The answer is yes, it is a real game.
Ned: [laugh]. Of course, it is. Just like Klingon is a real language. I love it [sigh]. I love that humans can create things out of thin air. It’s one of our strengths. It’s something that AI absolutely cannot do.
Chris: Right. It’s one of our strengths.
Ned: Yes.
Chris: [clear throat].
Ned: You and me.
Chris: Mm-hm.
Ned: Real humans.
Chris: [snort].
Ned: [laugh]. Should we move on to the actual topic, maybe?
Chris: Let’s do that.
Ned: Okay. Go for it.
Chris: Entrust distrusted by Google Chrome.
Ned: Dun, dun, dun.
Chris: I thought that that was just a clever headline when I read it the first time, but it turns out that distrusting is actually a thing that’s got, like, a definition.
Ned: Oh, okay.
Chris: We’ll get to it.
Ned: Excellent.
Chris: Which is a funny way of starting because this whole thing actually started about a month ago, and I completely missed it. And so, did you.
Ned: Definitely.
Chris: This week, however, it came back up again, for reasons that will become clearer as we go through this.
Ned: Okay.
Chris: But in short, advertising company Google, who you may have heard of—
Ned: Maybe.
Chris: Has a browser called Chrome.
Ned: This sounds remarkably familiar.
Chris: Yeah.
Ned: We might have covered this ground last week.
Chris: There is a company called Entrust, who you probably absolutely have not heard of.
Ned: Most people, yes. I will be the audience proxy.
Chris: And they create certificates. Starting on October 31st, 2024, Chrome will no longer trust any new certificates created by said company. Now, said company has a lot of security products and services that they sell, one of which was—is—well, no, definitely ‘was’—signing SSL certificates for websites. So, this decision, in short, effectively means that while Entrust will definitely stick around as a company, the business unit that does certifications, probably will not.
Ned: [laugh]. It would be difficult, yes.
Chris: So, what caused Google to take this dramatic action? Well, the security blog cited a few reasons that go back many, many years. In their own words, quote, “Over the past six years, we have observed a pattern of compliance failures, unmet improvement commitments, and the absence of tangible, measurable progress in response to publicly disclosed incident reports.” Unquote.
Ned: Ouch. Ouch.
Chris: Yeah, that definitely counts as an ouch.
Ned: Yeah, that’s… that’s bad. It’s not a good thing.
Chris: And what’s crazy is, these certs, it’s not like this is a cheapo product. They are still selling them as we speak, and the costs—at least the retail costs on the website; that’s a caveat there, right—$219 for a single cert, and $799 for a wildcard cert.
Ned: That is wild. And I think you’re going to address this later, but I have a certificate—a valid digital certificate for my website and the Chaos Lever website, and you know how much I paid for both of those certificates?
Chris: Zero dollars.
Ned: Zero dollars.
Chris: Correct.
Ned: Why in the hell would I spend $220 for a digital certificate for a single year?
Chris: Well, if you go for a three-year certificate, you get a 5% discount. So, there’s that.
Ned: [laugh]. Okay.
Chris: Yeah, I mean, these retail prices are insane. DigiCert is another corporate that sells certificates, and they’re basically half the price across the board. Then again, there’s Let’s Encrypt, which is, realistically, the only cert company you should be using, and their certificates are free.
Ned: Yep.
Chris: So, how on earth could Entrust be so expensive and yet so incompetent? I have absolutely no idea. The reason this came up, though, today was this past week, they released a blog post of their own, committing to getting back into Google’s good graces. So, one, I’m not sure why that took a month, and two, I suppose we’ll see.
Ned: Okay. Feels like they maybe should have done this a while ago.
Chris: We’ll get to that. We’ll get that.
Ned: Okay.
Chris: From the users' perspective, after October 31st, if you log on to a website that has a certificate signed by Entrust that was issued after October 31st, you will get a pop-up that shows a warning about that site not being safe. Now, you have surely seen this pop-up before. It happens if, say, certification—a certif—blah—a certification? Good God—a certificate is expired. Like, just happens. These things have to be renewed, and if you don’t renew it then it’s no longer valid, so you get an alert, a warning that says, “Do you want to continue to this website?” Or if it was a self-signed certificate—which those are still common, especially for internal applications—
Ned: Right.
Chris: Or if the certification was revoked, which is something that the cert authority can do for whatever reason, whether it was compromised, whether it was released incorrectly. You’ve seen these errors before.
Ned: Yeah.
Chris: And now you can add one more reason: if a company that created the cert in the first place isn’t trusted by the browser.
Ned: Yeah, that sort of falls into the same category of a self-signed certificate.
Chris: Pretty much.
Ned: In the sense that it’s signed by a certificate authority that the browser doesn’t trust.
Chris: Right. So, this begs the question, what in the hell did anything that I just said mean?
Ned: I’m sorry, I wasn’t paying attention.
Chris: [laugh]. Hey, not paying attention is my job.
Ned: [laugh]. Fair.
Chris: So, let’s play my favorite game and define some terms.
Ned: Oh. I thought it was Scrabble.
Chris: Play me for money.
Ned: I would lose a lot of money, let’s be honest.
Chris: [laugh]. So, in order to understand exactly what’s going on here, let’s go backwards from the user’s perspective to the CA themselves. So, when you log into a website, the first thing that you are trained to do is look for the lock in the corner of the URL bar. The lock means you’re safe.
Ned: I like being safe.
Chris: Wrong.
Ned: Awww.
Chris: What the lock means is that your connection to whatever site you have clicked on is encrypted. It’s a yes-no statement. Now, funnily enough, I think you and I are both old enough to remember when the world was very much not encrypted.
Ned: Yes.
Chris: You all remember the days when you’d log into, like, I don’t know, Hotmail, and the login page was HTTPS, meaning it was encrypted, but then it immediately switched your session back to HTTP, which is not encrypted because encryption was quote, “Too expensive.”
Ned: Mmm.
Chris: Pepperidge Farm remembers.
Ned: [laugh]. They do. That expense had a lot to do with the processing necessary to do the decryption and re-encryption of traffic when it hit whatever the endpoint was on Hotmail’s side. They didn’t want all their load balancers, or God forbid, the actual web servers to have to do all that decryption work, and this is before specialized chips that just did SSL work were easily available. So, they would do the login page since that, you know, you’re sending sensitive information, your username and password, but then, once it moved to actually accessing your mail, they’d move you off to a different channel that wasn’t using the expensive load balancer SSL decryption technology.
Chris: Right. And I believe—don’t quote me on this—I believe that it was a black hat presentation where somebody showed the absurdity of this by literally hijacking the presenter’s email while he was on stage.
Ned: [laugh].
Chris: Because when your traffic’s not encrypted, you can do that.
Ned: Yes, it is, uh, bad.
Chris: Anyway. So, established: encryption, good.
Ned: Yes.
Chris: But encryption just means that nobody can eavesdrop or manipulate the communication with whatever server you’re connected to. It doesn’t guarantee that you’re talking to who you think you’re talking to, if it’s a valid website that has been vetted by anybody at all. That’s where the certificate comes in. This certificate is basically like the envelope that delivers the encryption key. So, you take the encryption key, you submit it to the certification board, they give it back to you in is one gigantic file. It contains the keys, but it also contains information about you as a business. It’s basically the ‘from’ on an envelope, except that from is, like, notarized—
Ned: Right.
Chris: So, you know for sure that this website is who they say they are, and the key that you are using to connect to that website is from that entity.
Ned: Right. Because encryption just requires that you’re using encryption keys. It doesn’t guarantee anything about the provenance of those keys. The certificate is about establishing that provenance.
Chris: And the hope is that it makes the communication that you have that much more valid. So, for example, if you go to att.com—AT&T, right—you go to that site to pay your cell phone bill. You look in the corner; you see a lock. Great. You see a website that looks exactly like the AT&T website. Great.
You pay your bill. You cry a little bit, but great. This is all good. Next month, however, if you make a mistake, and instead of typing att.com, you type all.com—which was a terrible example because all.com is a real website, but anyway—this is the sort of thing where a hacker could take a name that sounds super close and create something else. So, you could be looking at what you think, is it att.com, but it’s something else all.com, or at1.com, or what have you.
Everything is going to look and behave exactly the same. Again, you are encrypted, right? It’s a yes-no conversation. But this time, when you pay your bill, you’ve just given your credit card to bad actors who are probably going to use it to buy crypto. One thing that could have helped there is if you looked at the certificate itself and seen wait a minute, this is not signed by AT&T, the corporate entity that is set up in… somewhere in California, probably. I meant to look up their actual certificate, and I didn’t—
Ned: Fine.
Chris: But again, this is a way to validate that the site that you’re going to is what you expect it to be. So, that’s why the certificates are important. And it’s also good for everybody involved to establish that att.com hasn’t been taken over completely, like that domain still exists, right? Which might be a more realistic problem because if somebody has stolen the IP address of att.com and put up another website there, they wouldn’t be able to use the same certificate because they don’t have the private keys.
Ned: Right.
Chris: They would have to put up a new certificate, which would be invalid for that URL.
Ned: Right.
Chris: So, that’s what the certificates do: kind of establishing in a clear way this website is who they say they are. And this brings us to the certificate authority. Like I said, anybody can create a certificate, even you. You have the commands on your computer. You can do it right now.
Ned: Madness.
Chris: But in order to have anybody else except to that certificate, you’re going to have to do a little bit more work. You have to basically be a part of a larger group of approved companies. Now, the company, whomever creates a certificate is called a certificate authority, and they basically do what you think: they are the authority that creates certificates. All the names that we’ve talked about so far. Entrust DigiCert, Let’s Encrypt, they’re all CAs.
They create certificates, but they’re also something else: they’re a trusted CA. And what does this mean? It means that browser companies have agreed that CA is rigorous, careful, trustworthy, secure, all of the adjectives. And there’s actually way more than I thought [laugh]. There are, in Chrome, about a hundred trusted CAs that just are in there by default. And… you have to remember Chrome is just one browser. All of the different browsers that exist that you can think of have a different list of trusted CAs. So, there are some variations, but honestly, not that many.
Ned: Right.
Chris: Incidentally, this makes the decision that Chrome came to all that much more interesting because as of recording time, there’s no indication that any of the other browsers have plans to distrust Entrust. Got, I hate having to say that.
Ned: [laugh].
Chris: But I mean, October is a while away, and we will see. And since Entrust is a fairly large player in this space, it would be weird if Chrome was the only one that didn’t trust them anymore.
Ned: They do have the market share on browsers, so—
Chris: Yeah.
Ned: In a way, if they decide to distrust Entrust, that is a huge black mark on Entrust, and I would assume that other browsers would eventually follow suit. This is something I had to deal with when I was working inside of an internal company, and we issued our own certificates for internal websites. And when we wanted to start implementing TLS, which is the underlying encryption technology for HTTPS, we were using our own certificate authority, but the browser’s did not implicitly trust that certificate authority. So, I had to use group policy to distribute the root certificate into the trusted location on all the Windows boxes so that they would now trust this internal certificate authority, and PKI is the name of the larger grouping of certificate authorities and other things—and that was great for Windows, and it was great for Internet Explorer because Internet Explorer, just believed whatever was in the Windows trusted certificates, but if someone decided to use Chrome—at this time, Chrome was just starting to blow up—there was no group policy to manage the certificates in Chrome. And so, anybody who tried to use Chrome would get this error message, and then I would get a helpdesk ticket. And so, I hated Chrome a lot for a little bit [laugh].
Chris: [laugh]. Totally fair.
Ned: Yeah. Now, I hate it for different reasons.
Chris: Yay. So, as a trusted CA, Entrust was supposed to do all of those things. And according to Google, and many, many other commenters, Entrust has consistently failed to maintain a reputation of rigid adherence to these community standards. One such example happened just a few months before Google announced their decision. In short, a whole batch of certificates were issued by Entrust with information in the wrong column.
So, certificates do have a lot more information than just, like, name rank and serial number. We don’t have to get too deep into the weeds of it. All of this is supposed to be super automated. And automation is supposed to mean all the right information goes into all the right fields. You would think that you would have a hundred percent success rate.
Ned: You would think.
Chris: You would think.
Ned: Automation is just the power to do one thing wrong a thousand times
Chris: I prefer the way to describe that as automation just allows us to make mistakes at machine speed.
Ned: [laugh]. At scale. And I guess they did.
Chris: So, there are a lot of tools that pay attention to certifications, which we’ll get to in a second, and these tools figured out that these certs were wrong, basically, immediately. Once again, the question is why didn’t Entrust not figure this out for themselves? We’ll put that on the pile over here with all the other mistakes. So, this issue was called out by somebody, it made it into a lot of conversations, there’s a Bugzilla tracker on this whole issue, and long story short, Entrust decided not to revoke the certs, even though they admitted that the certs were not issued correctly.
Ned: Okay.
Chris: Instead, what they said, more or less, was that this mistake wasn’t a big deal, and it was fine to leave the certs as is because reissuing them was going to be a hassle.
Ned: A hassle for whom?
Chris: Exactly. So, as you can imagine, there was some blowback from this decision. One quote that I thought was particularly enlightening to the discussion, read thusly, quote, “CAs facing challenges of their own creation should not be exploring ‘How do I keep these certs working,’ but ‘How do I make sure I don’t issue violating certs to begin with?’ Anything less is gross negligence, and not the system we should be striving to build.” Unquote.
Ned: Indeed.
Chris: A further series of comments makes it clear that Entrust has a long history of, let’s call it, pushing the limits when it comes to their policies around revocation. If this is interesting to you at all, I encourage you to read the Bugzilla conversation that is linked in the [show notes 00:19:43]. You’ll see a number of well-intentioned and very knowledgeable folks question Entrust’s stance and behavior, along with just, like, this one guy, who repeatedly says, “Nah, it’s fine.” So yeah, in short, Entrust chose gross negligence, and thus got the hammer from Google, that will, if it stands, effectively end their operations in the CA space.
Ned: Ouch.
Chris: Yeah. So, begs the question, if you’re an Entrust customer, what are you supposed to do? Well, the first thing to note is that only certificates that are going to become invalid are ones that are issued after October 31st. So, this also explains why they’re still selling them on their website right now. Because if you buy a cert right now, July 31st, 2024, at time of recording, it will be valid for the entire year, up to I think it’s 398 days, something like that, before it has to be renewed.
And this is something that’s important to note. If you have a certificate that is going to be renewed, in reality, that’s just a new certificate, right? So, if you renew a certificate on November 1st, 2024, that certificate is automatically invalid because it’s a new certificate issued after the deadline that Google set.
Ned: Yeah, I think renewal is a bit of a misnomer. It’s more of a re-issuance.
Chris: Right.
Ned: When I have a certificate, and I want to renew it before it expires, and I talk to the CA and I request a renewal, I’m really making a new certificate request to them, and they issue me a brand-new certificate, which I then have to install and use. It’s going to have a different key, it’s going to have a different serial number associated with it. So yeah, for all intents and purposes, it’s a fresh certificate. It just happens to use the same subject name—or common name—that the original certificate that I’m renewing had.
Chris: Right. And as is tradition in computer science, all we did was pick the word that sounded the most convenient, rather than one that was the most accurate.
Ned: [laugh].
Chris: But anyway, something else you can do is replace your certificate with another one, which, depending on the amount of systems that you have, I would say—I’m trying to do the math in my head here—I’m thinking that if you have more than one, this is going to be a huge pain.
Ned: [laugh]. It depends on the way in which you procure your certificates today.
Chris: True. You would also have to know your entire inventory and make sure that you get all of them because one thing that you would not want to do is fix 29 of your 30 certificates and forget about the 30th one, and then somebody like Ned gets calls at the help desk.
Ned: Yeah.
Chris: But luckily, blissfully, if you’re in any version of a large operation or enterprise space, there are tools now that exist that can help you. And if you don’t know about them, I want to introduce you to the tool that you never knew your organization needed: the ACME tool. It’s not just for Wile E. Coyote anymore.
Ned: And this one is actually effective.
Chris: [laugh]. So, I’m saying just ‘ACME tool’ in, like, air quotes in general because there are a ton of them that do this. And again, many of them are free.
Ned: Ooh, free.
Chris: So, ACME stands for Automated Certificate Management Environment. And I’m not sure if they did that on purpose to make it spell ACME.
Ned: You know they did.
Chris: I know. I know. The first one that came out actually came out from the Electronic Frontier Foundation way back in the olden days: 2015.
Ned: Right.
Chris: We still had hope, then.
Ned: Mmm. Like… sort of.
Chris: [laugh]. The tool was called Certbot. And it still exists, and it’s great. Certbot was introduced alongside of Let’s Encrypt—the CA—which, again, issue certificates for free. For free.
Ned: For free?
Chris: These are certificates that are free. There are other commercial tools from companies like Venafi, DigiCert, GlobalSign, and probably a thousand more that are not free. Let’s Encrypt is free. Just saying. But the whole point of all these tools is to automate the process: creating, managing, renewing, retiring, replacing certs on all of your infrastructure.
Ned: Right.
Chris: And these tools, as you might imagine, are a lot easier than going server to server by hand. These tools, especially the enterprise ones, can crawl your entire environment, identify every cert that’s in use, show the details about its creation, who issued it, its expiration date, et cetera. Then you can point them to whatever new cert you want to use, and basically click a button—
Ned: Ba-boom.
Chris: —and then the certs get replaced, whether it’s immediately, or just upon, you know, a day before expiry. And I know I’m not exactly making this clear, but for people of a certain age, everything I just described is basically magic.
Ned: It is. I remember, the same company that I was working for, we not only had internal websites, but we had a couple public-facing websites. And so, in order to secure those public-facing websites, we had to procure certificates. And this was, I want to say, like, 2004, 2005-ish timeframe. So, a while [laugh].
The process to get an SSL certificate—and this was just for a single domain—required you to fill out a form, and then you had to put in the request, and then they would ask for additional information about your business, and then you’d have to verify that you are, in fact, from that business through something that was either notarized, or you had to send it with the correct from address. There was, like, three or four different ways to attest that you are, in fact, the business that has legal ownership over this domain name. And then they would finally issue you the certificate. Which is why a lot of companies just went and got wildcard certificates, which basically matches any subdomain of the domain you’re getting the certificate for.
So, if your certificate is for *.bobsgumbo.com, any subdomain—dub-dub-dub, mail, blog, whatever—dot bobsgumbo.com would match that certificate. So, you’d have one certificate that you’d use for everything. That wasn’t terribly secure, it’s a bad idea, but the amount of work you had to go through to get that certificate in the first place made it worthwhile to get the wildcard cert and just roll with that.
Chris: So, what I’m hearing is you also used to have to work with VeriSign.
Ned: Yes. And it was so goddamn painful [laugh]. They also had different levels of SSL certificates. And I say SSL because that’s what it was at the time, before we switched to TLS—same technology, different name—
Chris: Right.
Ned: They had extended validation or EV SSL certs, and for those, you had to do additional levels of validation that you were from the company you said you were, and that you own the domain, and you were the authority for that domain that you were requesting the certificate for. And they will charge you a comfortably large amount of money to get that EV cert. But then you could say, “Look at me, I have an EV cert.” And somehow that was better. There was a period of time when browsers actually had a different lock icon or color—
Chris: Or a different color, right.
Ned: If you were using an EV cert versus just a regular SSL cert. And that was, like, super important. And that’s why you would pay good money to one of these companies, was to get that reassuring, different lock color. These days, no one gives a shit.
Chris: True.
Ned: Certificates used to be issued for a year, two years at a time. Now, the average certificate is valid for between 30 and 60 days. And it gets renewed automatically. And it uses that ACME protocol, and it’s probably using, like, Let’s Encrypt. And that has really changed the whole way in which certificates are issued, and the value behind an individual certificate, for the better, I think.
We have a much more secure web because of it. But it does mean that a lot of these older companies don’t have the cash flying in that they used to, and that may lead them to cut some corners because they don’t have this just, you know, companies backing up the dump truck of money to get the certificates from them.
Chris: It’s almost like they could, instead of rent-seeking, they could innovate.
Ned: Wh-whoa. Whoa. Now you’re talking crazy.
Chris: I’ve gone too far.
Ned: So, let me ask you, is Entrust using AI? [laugh].
Chris: You know, I haven’t looked into that. But I’m going to go with yes.
Ned: Breaking news—and I just saw this morning, so I haven’t really had a chance to dig into it, but apparently DigiCert, which was one of the other certificate authorities you mentioned, has issued guidance that they’re revoking a subset of their TLS certificates due to a non-compliance issue with domain control verification, and this may cause temporary disruptions to website services and applications relying on these certificates. DigiCert has notified affected customers, so if you are one of those customers, if you’re using DigiCert today, you might want to check on that because they are revoking a lot—not a ridiculous amount, but they’re revoking a decent number of certificates for websites out there. And if you happen to be browsing the web in the next week, you might come across one of these revoked certificates.
Chris: And then, if you do, you’ll see the system operating as expected.
Ned: [laugh]. What’s actually funny is that a lot of browsers don’t actually check the CRL—which is the Certificate Revocation List—they don’t actually check it. They just check the validity period of the certificate, and as long as the cert is valid and comes from a trusted CA, they stop there. Because hitting the CRL is more work.
Chris: Man, you really are just bringing the sunshine today, aren’t you?
Ned: [laugh]. I have been too deeply steeped in PKI and CA stuff for years, and I’ve grown to hate almost everything about it.
Chris: [laugh]. I can understand why.
Ned: Hey, thanks for listening, or something [laugh]. I guess you found it worthwhile enough if you made it all the way to the end, so congratulations to you, friend. You accomplished something today. Now, you can go sit on the couch, fire up the DigiCert website, and see if your certificates have been revoked. You’ve earned it.
You can find more about this show by visiting our LinkedIn page, just search ‘Chaos Lever,’ or go to our website, chaoslever.com where you’ll find show notes, blog posts, and general tomfoolery. And if you have anything to add to this certificate authority conversation, we’d love to hear about it, so leave us a voicemail or a comment. We’ll be back next week to see what fresh hell is upon us. Ta-ta for now.
Chris: Yeah, it’s pretty funny. I forgot about your… let’s call it, passionate experiences with certifications and the like. Had I been paying more attention, I would have just made you write this one.
Ned: [laugh]. I already assigned you something this week and you just ignored it.
Chris: Ignored what?