It's Time for the VPN Industry to Innovate (Obscura Interview)

You don't have to trust Obscura—you just have to trust that not both Obscura and Mullvad are compromised. Henry sat down with Carl, former Bitcoin Core developer and founder of Obscura VPN, to discuss how it's the first VPN that mathematically can't log your activity, what makes it censorship-resistant against networks like the Great Firewall, and what it really means to build privacy that's more than "a pinky promise."

🔗 SOURCES & LINKS
• Obscura: https://obscura.net
• GitHub: https://github.com/Sovereign-Engineering/obscuravpn-client
• Trust, 2-Party Relays, and QUIC: https://obscura.net/blog/bootstrapping-trust/
• Mullvad: https://mullvad.net

⏱️ TIMESTAMPS
00:00:00 INTRO
00:01:50 CARL'S BACKGROUND
00:10:47 APPLE'S PRIVATE RELAY
00:17:11 TRUSTLESS MODELS
00:19:05 MULTI-PARTY RELAY PARTNERSHIP
00:21:39 TRADITIONAL MULTI-HOP
00:23:39 CENSORSHIP RESISTANCE
00:27:33 MULLVAD VS. OBSCURA TRAFFIC
00:29:35 EXIT IP
00:30:18 BLOCKS, CAPTCHA, & SPLIT TUNNELING
00:31:16 CUSTOM DNS
00:32:41 PARTNERING WITH MULLVAD
00:36:05 OBSCURA VS. PRIVATE RELAY
00:37:10 OBSCURA VS. MULLVAD
00:38:33 OBSCURA VS. DECENTRALIZED VPNS
00:40:57 OBSCURA VS. TOR
00:41:58 REPRODUCIBLE BUILDS
00:50:23 CLIENTS & DEVELOPMENT TIMELINE
00:55:24 SPEED
00:59:06 DEFAULTS VS. CUSTOMIZATION
01:00:48 PRICING
01:02:43 OPEN SOURCE
01:03:34 THE OBSCURA TEAM
01:05:39 THOUGHTS ON THE VPN INDUSTRY
01:07:49 OUTRO

🎥 VIDEO
Watch on YouTube

🧡 SUPPORT TECHLORE
Keep Techlore Talks independent & growing: ★ Support this podcast ★

Creators and Guests

Host

Henry Fisher

Runner, artist, musician and digital rights activist. Owner of Techlore

Guest

Carl Dong

Obscura VPN

Editor

Tori

Techlore

What is Techlore Talks?

Techlore Talks brings you in-depth conversations with the experts at the forefront of privacy, security, and digital rights. Hosted by Henry Fisher, founder of Techlore and long-time digital rights educator, each episode features meaningful discussions with the people building, researching, and advocating for digital freedom.

From cybersecurity researchers and privacy tool developers to open-source advocates and digital rights activists—if they're shaping how we protect ourselves online, they're on this show.

Topics include: privacy tools and technologies, cybersecurity threats and defenses, open-source software, surveillance and digital rights, encryption, tech policy, and digital sovereignty.

New episodes released regularly. Subscribe and join the community at techlore.tech.

You don't have to trust Obscura for your privacy. You just have to trust that not both Obscura and

Mullvad is compromised. Hello, everybody, and welcome to Techlore Talks. Today, I have the pleasure

of inviting Carl from Obscura, the VPN provider, to talk about what Obscura is, what they're trying

to accomplish, how they are actually somewhat like Mullvad, but a bit different. In fact,

they use a lot of Mullvad's infrastructure for what they do, the importance of censorship resistance,

and what they're really bringing to the table,

which is something genuinely unique in the VPN space I haven't heard before.

So get ready for a really insightful discussion about VPNs,

but not just VPNs, the underlying technology that makes them happen

and what they really mean for this world where we're seeing age verification pop up,

as well as tools that a lot of people are relying on

to gain better privacy in light of those things.

So let's dive into it.

Welcome on the podcast, Carl.

It's really great to have you.

and you're now the second VPN person who's ever been on this podcast.

So do you want to introduce yourself and who you're from?

Yeah, I'm Carl. I work on Obscura VPN.

We're trying to be the first VPN that can't log your activity

instead of just promising with a pinky promise.

I used to work on Bitcoin Core.

I was a Bitcoin Core developer for three and a half years.

And yeah, I'm all about championing privacy tech, and that's why I'm here.

Yeah, so do you mind, you know, we need to kind of establish context and maybe the behind-the-scenes tech to kind of break down what you mean by, you know, we know that there's a lot of trust you hand over in your VPN, but what exactly does that trust mean?

Because you say they're pinky promising, but what exactly does a typical VPN provider actually see, and what are you trusting them on?

Yeah, maybe we can just start with sort of more on my background and how I came to this, right?

Like it's not every day that somebody wakes up and is like, I'm going to make a new VPN today and everything else.

What is it?

So I grew up for a part of my childhood in China and in London, actually.

And so I was in London for a few years for primary school.

And, you know, we did book reports, you know, I sort of paraphrased Wikipedia as everybody

did for a time.

And, you know, I had my friends on Facebook and everything else.

And I went back to Shanghai for middle school.

And, you know, all of that was taken away from me, basically.

The Great Firewall had come up.

I wasn't able to access YouTube or Facebook or any of these niceties.

And so sort of by necessity, I had to try out all the commercial options back then.

And these were the old protocols, PPTP, IKEv2, you know, the really old ones that you can still use on iOS nowadays.

And, of course, you know, experimenting with building my own for my family, for my friends and things like that.

Right.

I came here to the States in high school.

And actually, it was also networking problem because it was a boarding school and you weren't

allowed to be on the internet after 10 p.m.

And they did Mac blocking.

So it was blocking based on Mac addresses, which me and this other Japanese kid figured

out that if you get the Mac addresses of the library computers and spoof your Mac address,

on your laptop. Anyway, it's all fun stories like that. And of course, after college, I spent a few

years on Bitcoin Core. I worked on mostly user trust. That's sort of the theme of what I like to

work on, right? The question is sort of, even if a piece of software is open source, right?

How can you trust that what you download corresponds to the source code that you're seeing?

Because they could just have good source code, and then before they compile it into a binary,

they put a little backdoor in there and publish a binary that has a backdoor.

And so when I was working on BiggerCore, I was working mostly on this reproducible process.

We even made it bootstrappable in the end, so working on user trust then.

After I took a break from Bitcoin Core in 2022, I was looking around and I was actually one of my good friends, Matt, who was telling me about Apple iCloud Relay.

And, you know, through all these years, like every time I go back, visit relatives or whatever, I like test out the great firewall.

I'm like, hey, what new tricks have you got up at Sleaze?

At a certain point, it started.

So like OpenVPN was good for a bit and everybody, oh, OpenVPN, pretty good.

Well, I had the TCP over TCP problem, but whatever.

OpenVPN was good for a bit.

And then it started being able to fingerprint OpenVPN, right?

And then start blocking that.

So you want to move to something else.

WireGuard came out and was like a huge, you know, that was monumental, I want to say, what Jason Donnell did.

And then because it was UDP and there's better performance, yada, yada, yada, emulates IP stack.

And then the Great Firewall started being able to fingerprint that because that was kind of easy to fingerprint, I will say, the WireGuard handshake.

And so throughout all of these learnings, I think what we found and now there are better protocols with obfuscation and things like that.

But the main learning basically from all these years of fighting against the great firewall is that people who restrict networks still want to let some good traffic, what they consider good traffic, out.

And that is what you need to exploit.

It's not like we're going to have magically the best cryptography in the world and it's going to look indistinguishable from a pseudo random stream and there are no headers, nothing to fingerprint.

To someone with a restrictive network, they could just be like, this looks too random.

I'm going to just drop this.

I'm just going to drop this connection.

What you really want is you want some obfuscation so you look like normal traffic.

You look like regular HTTP traffic.

And this was what's so brilliant,

but I don't think they ever stated it in the Apple iCloud relay paper,

is that because it's a scheme that's based on the Qlik protocol,

it looks like normal HTTP traffic.

So sort of on the obfuscation side,

that was an innovation that they had that they never told anybody,

I don't think because, you know, you don't like to say these things, but, you know, anybody who understands networking is like, oh, this is great.

Right. And then the the big, I think, improvement that is sort of at the core of obscure as well is this concept of a multi-party relay or two party relay, let's say.

So normally, when you use a VPN to access the website, you have you, you connect your computer, your laptop connects to the VPN, right?

And then the VPN connects to the VPN server, and the VPN server connects to Google.com or something like that, right?

And so that VPN server being the only man in the middle, they can see who you are, and they know exactly where you're going, right?

They could correlate this information, build sort of information on this.

And a lot of times, a lot of these VPN companies, they collect like your credit card information,

your email, like they know where you live.

They know everything about you, right?

And they're this wholly unaccountable, privately owned company.

And I don't think that I'm not saying that companies do this.

I'm saying that for even the trustworthy companies, right?

All it takes is a breach.

All it takes is a breach that they don't notice for this to be leaked, even if they are trying

their best to protect their information.

I think that's a very important point.

The innovation then of a multi-party relay is that the first hop, in our scheme, the first

hop is Obscura.

Obscura sees who you are, right?

We know that you've paid, you know, if you pay via credit card, then we could technically,

you know, see your address and things like that, right?

Because you put it in.

But we can see who you are, but we can never see where you're going.

Right. We have no idea where you're going.

And by the way, with Obscura, you can pay via Lightning and soon Monero.

So we don't even know anything about you at all.

Right. But the basis of it is that we can see who you are, but never where you're going.

And the second hop, which is Mullvad, our partner, doesn't know who you are.

It just sees it's, you know, one of a thousand users coming from Obscura, let's say.

And they're the only ones knowing that you, like, one of a thousand obscure users is going to, you know, wick your feet or something like that, right?

And so that multi-party relay separation really separates who you are from what you do, meaning that no single party can piece together that information.

Even if, like, we were hacked, let's say, the hackers won't know where you're going, right?

And that was like core innovation that I think made using a VPN and having privacy when using a VPN require a much lower level of trust in the provider.

And I was like, this is the basis for the next generation of VPN providers.

And I'm a VPN nerd, so I got to do it.

Yeah, thank you for that.

And so I think the first question that people will ask, and my mind goes to as well when you break this down, is this sounds a lot like a traditional...

Actually, before we get into multi-hop, because that's where I was going to go with VPNs and how Mullvad, I think, has their multi-hop proton, etc.

So I'd like to touch on the difference there.

Can we touch on private relay first?

Because I think it's one of the most misunderstood technologies, especially in our community.

I think a lot of people wrote off Private Relay because it's from Apple, which is very fair, very justifiable.

I get why our community doesn't love Apple.

They do a lot of crappy things.

And Private Relay in itself, I think, does have a lot of crappy things because it's not system-wide.

It's only in the browser.

It only works in Safari.

You have to be in the Apple ecosystem to use it.

So I get those criticisms.

But if you actually read the white paper and all the people who know what they're talking about, like you alluded to,

absolutely think that private relay is like this genius new thing that no one's ever really done before.

So do you mind speaking a little bit more to private relay's tech and why it's cool?

Oh, 100%. I mean, this is, and I will say I had the same reaction.

Because like, I remember, everyone I know in your position had had like a, whoa, this is kind of crazy new tech.

I'm saying like my initial reaction, let's say, because I didn't even read into it because I was like, oh, Apple's doing something, you know, probably not that interesting, you know, I'll table it.

And then, you know, Matt's a good friend of mine.

You know, he's a longtime Bitcoin core contributor.

And like we nerd out about networking things all the time.

And like he was telling me, hey, read this.

I was like, OK, I better take this seriously.

And so I actually read into what it was and I was like, oh, there's no way that this.

So I think for people who are sort of feel a little bit iffy about the Apple side of things, read up on mask.

Mask is sort of the generalized, M-A-S-Q-U-E.

Mask is sort of the generalized protocol that Apple iCloud Relay is operating over.

And Mask is sort of, there's an IETF working group now working on Mask.

And, you know, we take a lot of inspiration from Mask.

We've made a little bit of, a little more optimizations on Mask, you know.

But, you know, mask is generally the scheme.

And mask is exactly what I describe of, you know, these multi-party relays, right?

Being able to set up through open protocols, multi-party relays that are privacy preserving for your IP address.

And mask works over quick.

And the quick part, I was talking before a little bit about how quick looks like, well, it's what HTTP, what's underlying HTTP3, right?

So quick allows your connection to look like regular internet traffic, right?

If you go on Google.com right now, you're actually using quick to connect to Google.com because most of these big sites have already enabled HTTP3, which runs over quick.

And so to a network operator, right, your connection over Qwik to Obscura or over iCloud

Relay looks very similar or is much harder to distinguish from a Qwik connection to Google

to, you know, regular sites and things like that.

So that's one big win for using Qwik as the tunneling protocol, tunneling transport protocol.

The other one is, yeah, the other one is in the old days, before Qwik, we basically just had TCP, right?

HTTP 1 and HTTP 2 was over the TCP transport protocol.

And so if you want it to look like HTTP, you need to basically tunnel traffic through TCP.

But tunneling traffic through TCP is one of the worst decisions that you can make.

because it leads to, how can I explain this?

It leads to a conflict in between the TCP connection

that's carrying your packets from your computer

to the VPN server and the TCP connection

that's between you and the end site.

Because you're basically tunneling one TCP connection

inside another and TCP is a protocol

where it tries to do retransmissions and tries to adjust its own windows.

And it really is meant to operate over something like IP, something that is unreliable, as we say in the networking world.

And basically, it all just leads to a lot of stuttering, a lot of like, you know, when you see in the video, it just goes, it just sort of fast forwards a little bit.

And so TCP was terrible. And that was the other innovation is in Qwik, you can do unreliable datagrams, as I was saying before.

So you avoid completely this problem of things not being smooth, of videos scrolling forward, the jitteriness of the network tunneling, and you preserve the fact that it looks like regular internet traffic, which was so genius.

So a cool backstory on this.

I remember I was in love.

I was in love with mask.

And I remember I was looking, I looked at the IETF RFCs, right?

The internet drafts and at the RFCs that I was like, okay, I got to implement this.

So I implanted the mask UDP extension in both Rust and Go before I was like, okay, this is actually like something that works.

And then I went to the IETF conference and talked with them about it.

And to confirm, let's say that I'm, you know, knocking on the right door.

I don't know.

I'm going down the right rabbit hole, let's say.

And yeah, totally.

It's a great innovation.

Yeah.

So one thing also about private relay that I've seen and maybe how does it compare maybe to your model where it is a trustless model for Apple?

Like you don't have to trust Apple in private relay because they include a partner.

So is that similar to what you guys are doing?

Yes, exactly.

So in private relay, what happens is the two parties are Apple and Cloudflare.

Oh, well, there's Cloudflare, Akamai, and Fastly, I think are the three.

They might have added more recently, right?

So basically, you don't have to trust Apple.

The two parties are such.

If both of them were compromised, then that's a different story, let's say.

But you don't have to trust Apple itself.

It's multi-party.

the one you have to you have to make basically make sure that not both of them are compromised

that's how it works so in obscura it is also the case and that you don't have to trust obscura

for your privacy you just have to trust that not both obscura and all that is compromised and

sharing data which you know it wouldn't make any sense for Mullvad to like risk their well-built

reputation for like little me let's say yeah yeah and i guess um for people who are

because you know one of the cool things about being in the position i'm in is over time getting

to know people like you behind the projects a bit better and seeing that 90 percent of people who run

these projects are like wonderful 90 plus percent you know that's not like nine out of ten it's like

a majority of people are like extremely awesome people who are doing some really good work. They

really care about this. And they would never jeopardize their users in a way that was intentional.

No, 100%. But to someone who is more critical, they want like absolute peace of mind, how do they

know? Like, you know, when you say partnership, it sounds like that also could equally mean, well,

we also collaborate on user data. So what does like the partnership look like?

Yeah, interesting. So I'll make two points on this, right? One is what we have with multi-party

relay is strictly better than a single-party relay, right? There is no case in which we are worse in

terms of privacy and being able to access your data than a single-party relay. And when I mean

single-party relay, I mean every single other VPN out there, right? There is no case. So it's a

strict improvement, right? If you want to sort of look more closely on sort of how can you trust that,

you can go to the Obscura VPN app. I mean, you can look at your app on your computer and you can go to

the locations page and you can look at literally the public key, the WireGuard public key of the

Molvat exit that you're connected to and what you're encrypting your data to.

So unless they have, and by the way, you can, there's a link there.

You can click and it links to on Molvat's own server website where they publish the key for

that exit server.

So unless they have, I mean, I think anybody, unless they have given us the WireGuard private

key of their servers where all of their other customers are also on, we cannot see your data.

Like that is basically how it is, right? And of course, I think there's always things that we can

do to improve it. And that's what we want to do, right? We want to push forward this industry. For

example, we want to do reproducible builds. Now we've been very busy getting all the other features

out there like iOS, Monero and Android and everything else.

But we want to do reproducible builds because at the end of the day, you want to look at

our source code.

You want to make sure that that is the app that we are releasing.

Right.

But I think our partnership with all that, I think, is very simple.

We pay them money for, you know, to be able to like send traffic to their servers and that's

it.

You know, it's very simple.

Yeah.

And then do you mind speaking a little bit to the multi-hop now of a traditional VPN provider?

So I use some of these features myself.

So I'm not here to just say, don't use these.

And I'm not just trying to make them look bad.

But I think it's important to talk about what the key differences are.

So like Proton VPNs, multi-hop, Mullvads, multi-hop.

I think even Nord and the more popular VPNs provide similar functionality as well.

So how is you connecting to two different servers from the same company different from what you're doing with Mullvad?

Yeah, I think, you know, when we were just talking about the security, it's all about security model, right?

When we were just talking about the security model, we were like, oh, in two-party relay, as long as both parties aren't collaborating, you have better privacy, right, than a single-party relay.

I mean, in a multi-hop, it's by definition true that the two parties are collaborating

because it's the same entity, right?

It's the same company.

In certain scenarios, it might offer better privacy, but I think it's just a little bit

-- it's qualitatively different.

What are those?

What are the scenarios where it would give you better privacy in like a traditional multi-hop

setup?

let's say you have the two hops in jurisdictions that are not friendly with each other.

So if there's enough people doing multi-hop between these two locations, you may have a big enough

anonymity set versus a global passive adversary. That's sort of the steel man case for multi-hop.

But I think in most, I don't want to be crass, but I think in most cases, it's just you're just adding a little bit of latency for it.

I don't know why.

I think for, you know, I have to believe in my own product.

I think, you know, two-party relay is really the future here because you want separate entities.

Okay.

So, you know, as we've already been talking, I've been laying out some extra questions here.

So, you know, we started this with talking about censorship resistance and kind of your original motivations of trying to bypass the great firewall and then, you know, spoofing Mac addresses and all this fun stuff.

So does Obscura add any layer of censorship resistance that just regular Mullvad would provide you? Is there any difference here? Do you add more or less or is it equivalent?

No, I think we definitely add more in that, you know, the censorship resistant part comes

from our obfuscation, right?

Comes from the fact that to a network operator, we look like HTTP3 traffic, which is much harder

for them to differentiate.

There was actually a new paper that came out about the difficulties of quick censorship and

everything else like that.

They were measuring across different nation states that tried to censor QUIC and see sort

of ways to get around that.

And it sort of paints a picture that people are not doing a great job, let's say, at censoring

QUIC because QUIC is pretty hard.

It's QUIC.

It slips by you, I guess.

And so when you connect using WireGuard, let's say, which is what every other sane VPN company would do, when you connect using WireGuard, it's quite easy to fingerprint in that it's UDP traffic.

It has, you know, like the handshake.

I mean, we are UDP traffic too, but it's UDP traffic with a very fingerprintable handshake pattern.

And the packets are very well distinguished.

And there's basically only one reason why you would be using Wirecard in that fingerprintable of a way, right?

Versus for Quick, you could be using Obscure, you could be going on Google.com, you could be doing anything, right?

And that's where the censorship, this is a concept called collateral freedom.

I encourage every listener to look up, there's a Wikipedia page on it.

I think collateral freedom is very important in building censorship resistant tools in people

participating in a way such that we sort of raise everybody up, such that our normal usage

makes everybody more censorship resistant. And I think that's sort of one of the genius strokes,

let's say, of Mask and Quick. Got it. And so for a listener as well as myself, because

my brain, networking and my brain just definitely don't click as well as it might for you.

Okay, sorry. So for the hop, so the censorship that would happen

would be between you and Obscura slash the VPN. So if I was traditionally connecting to Mullvad

and I was in a place that didn't want me to connect to a VPN, it would say, hey, we recognize

this is VPN traffic, so they're going to block it, and then I can't connect to Mullvad. With you guys,

you guys have a different protocol so that when I'm connecting to Obscura instead of Mullvad VPN

directly, I'm getting far better ability to blend in with other traffic, so it's harder to block that.

But then once it hits you, you can freely transfer that data to Mullvad without any censorship concerns.

That's exactly it.

Got it.

Okay.

So it's that first hop that you're getting the main benefit.

It's that first hop.

Sorry, I shouldn't explain that.

It's that first hop.

That's the most important hop, right?

Got it.

Because you're probably, you know, connecting through, I don't know, your company Wi-Fi.

You know, it could be anything, right?

A nation state Wi-Fi, you know, whatever, right?

And yes, we blend in better.

We look like normal traffic.

And so it's less likely to be blocked at the first hops.

So what would, if I was a network administrator and I worked for my local company and I was just, you know, snooping on my employees' web traffic, what would Molvad VPN look like on the network versus Obscura?

Like, what would a network administrator see?

So I actually have this on my, I run Ubiquity at home, which is a bit nice.

And you do get a little dashboard.

And so before I built all of this, it was like, I don't know, 90% literally identified WireGuard traffic.

And then now it's like 90% quick traffic with sort of no identification as to what it is.

There is technically a way to identify what host name you're connecting to via this thing called TLS-SNI.

I won't go into the technical details.

It basically just tells you like you're going to google.com or going to youtube.com or whatever,

right?

But there's one guy that I talked to at Google that made a very cool scheme.

So in Google Chrome, this part of the quick packet gets chopped up and then shuffled around.

And so most routers just give up.

They just don't try to even read that part.

And for us, we'll probably implement that at some point.

But also for us, we sort of, because we control the networking stack and we can just set this

to whatever, we sometimes set it to example.com or like just an empty string or something like

that.

And so nobody sees it.

So is that customizable for an Obscura customer or is that just done by default?

That's done by default for them.

We try to make the defaults sane.

And, yeah, and, you know, it's only because we're like a bunch of networking nerds and protocol nerds that we're able to sort of get all this together.

So I'm very happy with it.

Got it.

And then kind of a final question here.

So in this hop, you know, trying to visualize it.

So, you know, someone gets the censorship resistance and they connect through you, which adds a party to remove trust even from Mullvad.

And then you forward traffic to Mullvad.

So then is your exit IP when you're accessing all the websites online going to be Mullvad where

if I access Mullvad's website, it says you're using Mullvad even though I'm using Obscura?

Yeah. Okay.

Yeah. Many people have found this out in that they go to the Mullvad.net/check and it's like

read checkmark and I'm like, yeah, that's how you know that we're working. We're not just like,

oh we're using mulvac you know it works yeah and um i guess one thing obscura probably doesn't

deal with then based on the way that works is you're still going to get blocked on websites that

block mulvac traffic you're probably still going to get captured online so obscura doesn't help that

specific no we don't we don't help with that specific problem right now i think you know we

We're doing research into the best ways to do split tunneling.

Because I think we've had now a couple of VPNs give their try into split tunneling.

You know, some of them are per app.

Some of them try to do per URL.

But that's very, you can leak a lot if you're doing per URL.

And so we're trying to make sure that we do it in the right way.

And I think there will be big news when we do announce what we have.

So stay tuned.

A bit of a niche question, more for myself.

I think for people who are really into networking, they're like, this is a horrible idea.

Never do this.

But I do enjoy using a custom DNS with my VPN connection.

I like having my own filtering.

I don't mind if sites can see like, oh, this one user is using a different DNS from all the other people connected to the same IP address.

I'm fine with that if I need better fingerprint resistance

I'm using Tor. So for me the VPN isn't as hardcore of a tool for that specific use case and I

absolutely love Next DNS and AdGuard and all these tools. Can you still do that with you guys?

So we've I've I did I'm personally not like a custom DNS kind of user so I was kind of

overwhelmed with emails from people. I use the next DNS profile, but it sometimes works with

the material and sometimes it doesn't. This is our next big project after, well, not big project,

medium-sized project, let's say, after we are done with our post-IOS cleanup. We're going to do

custom DNS. We're going to do on-demand connections on iOS. This is definitely, that's top of mind.

And yeah, I think, you know, people should be able to customize their system.

I think, you know, advanced users should be able to do whatever they want because we love

networking nerds, you know.

Cool.

And then, you know, I guess I should have asked this earlier.

Why did you guys pick Mullvad?

I know there's a few other trusted VPN providers in the space.

I'd normally say like the most trusted overall seem to be iVPN, Mullvad, and Proton.

Yep.

So I don't know if Mullvad was just a better fit for you guys.

Were they more open to a partnership?

I'd love to hear a little bit more about why you picked Mullvad for that second hop.

It wasn't much of a pick for me.

So I think to me, when I was still in college and using all these different commercial options

and trying them out and everything else like that, right?

When Molvad was like, hey, we don't need your email.

We just need a random number.

That really showed to me like where they're coming from.

Right.

And it's the same stance that we try to take, which is like, what is the least amount of

information we can have about you and still be able to provide you the service?

Right.

And that like, just generate a random number, you know, like why do we need an email?

And it made me, you know, it carved out a special place in my heart for them, let's say.

And I think in 2020, I think a COVID weekend project was, you know, they were accepting Bitcoin already.

And, you know, I was like, hey, you guys got to accept Lightning or something.

And so I just emailed them and I was like, hey, can I just resell Molvad for Lightning on my site that I put up in a weekend?

And they replied with a PGP encrypted email being like, yeah, sure, let's do this.

And I started doing this with Molvad and every few years I would top up that little plain HTML website and it worked and people wanted it.

And so when it came time to look at what we should do for Obscura, I was like, you know, there's no, they're my first choice, let's say.

They're my first choice.

And I think they keep doing things that I think make them almost unbeatable in terms of trustworthiness.

Although I will say Proton and IPPN do a lot of things as well.

But they keep doing things like, let's just deprecate OpenVPN.

And to someone who like, think about these things a lot, right?

OpenVPN is a gaping hole in security of any VPN company.

OpenVPN is so complicated of a protocol and of an implementation with so many extensions and all those things that I don't think anybody can audit it.

WireGuard, you can audit, let's say a sufficiently good auditor can audit, let's say in a few days.

OpenVPN would take months. And so it's very different. And that's why we picked them. Yeah.

Great. And then a few more questions that I have. I want to start by just kind of doing a quick

comparison. So maybe just like 30 seconds of just explaining the different users that these tools

are geared towards. So first, if somebody's all in in the Apple ecosystem, why would they pick

Obscura over just using private relay? Right. I think two main things, right? One thing is that

Apple iCloud Relay is only for Safari and only for, you know, your Apple devices. You might have

other devices. Not only is it that it only covers the traffic that gets sent out by Safari and all

those things. So it's not a whole system. It's not a whole IP stack relay. It's only for Safari,

whereas we are a full-on VPN. Every IP packet that gets sent out goes through our tunnel,

and so it'll cover all of your apps. I think a second thing is that it depends on what your trust

model is, right? The two hops in Apple iCloud Relay is Apple and, you know, those Akamai,

Fastly, and Cloudflare, right? Whereas for us, the two hops are Obscura and Mullvad. So,

depends on what your trust model is there, too. And then how would you compare maybe

just regular Mullvad to Obscura? Yeah, regular Mullvad to Obscura, I will say, you know, if you use

Molvat, keep using Molvat. They're a great partner and we love them. I would say the regular Molvat

versus obscure, I think the main thing is the two-party relay, right? You no longer have to worry

about, okay, what if one of Molvat's servers get hacked, let's say, right? If a Molvat server,

if you're just connecting to Molvat, if a Molvat server gets, because I feel like they're the most

trustworthy ones. If the molvat server gets hacked, then in a way that molvat doesn't notice,

I'm sure if they notice, they wouldn't fix it, right? If they get hacked in a way that they don't

notice, then that attacker could be logging everything that's coming in, logging everything

that's coming out, correlating them, building sort of a data set from that, right? Whereas with a

two-party relay, if we get hacked, they still get nothing. If molvat gets hacked, they still get

nothing. It really has to be a double hacking, let's say, and a correlation of things across

different entities for them to get anything. So it's a much, it's a strictly better security

model. And I think a much better security model. Got it. And then two more quick comparisons here.

So one, how about something like a decentralized VPN? So I know IVPN acquired Safeng, who has the

SPN, how would you compare yourself to something like that?

I don't know the specifics of SPN and how they work, but I will say traditionally, and I've

sort of tested some of these, I just haven't tested SPN.

So traditionally, decentralized VPNs have the problem of almost the same as Tor of the exit

nodes are run by volunteers.

Less so on Tor's side, but decentralized VPNs, let's say, they're run by the nature of being

decentralized, let's say.

The exit nodes are run by random people, and all it takes is for a cat to trip over a power

cable and for there to not be a UPS or something like that in the house for your connection to

be interrupted, right?

They might also have limited bandwidth up and down.

It's just not the most reliable thing.

However, I think they offer sometimes better, you know, more hops and things like that for

absolute anonymity and things like that.

But I think no VPN is going to be perfect for every use case.

The use case that we want to target is someone who is the use case where the person can have

the VPN on 99% of the time, where it's fit for daily use.

I don't think something like Tor or something like a decentralized VPN is fit for everyday

use because of reliability, because of performance, because of everything else.

And I think Obscura is because I'm talking to you through it right now.

You know, we made the optimization.

We made sure that we have bare metal servers, all optimized Rust software.

We did the work to make sure that is good for daily use.

And the point I also want to make is I think we want in the privacy space to make more tools that are suitable for daily use and suitable for normies.

I don't like that. You know, it's a derogatory term, but, you know, suitable for normies because that's that's the way that we move our cause forward.

Right. That's the way that we expanded and and sort of expanded as a fundamental human right is to make it available for everybody.

Got it. And then is the fourth one was going to be Tor? Is there any specific you want to add to Tor

or is it kind of the same answer as decentralized VPN? Yeah, I mean, I basically covered it all. I

mean, I just wouldn't use Tor to, you know, watch a 4K HDR video, but I do it on Obscure every day,

right? I wouldn't use Tor to do a video call. Although, like, amazing work that the Tor Foundation

of like donate, please.

Like it's amazing work, just different use cases.

What's very interesting is in the Tor config,

there is like a configuration line

that's like just do two hops or something like that.

And if you set it to true, it's basically obscure,

which is really funny.

And it's meant for Tor but better performance,

which is like, okay, that's what we are

with professionally run hops, let's say. So it's funny. Let's pivot over to the clients a little

bit more. Reproducible builds. This is something that I like to approach carefully because there's

a lot of very intense people in the community who want best practice everywhere. And they see any

software that's not reproducible and they say, well, I just can't trust it. And then you look at

the actual amount of privacy and security software that we all know to be trusted that is not

reproducible. And it's almost everything. There's very little reproducible software. So can you speak

to the challenges of being reproducible, why you're not currently reproducible, and how you intend to

be reproducible? Okay, so I can go on for days about this. This was my main work on Bitcoin Core.

I think our work in Bitcoin Core basically pioneered bootstrappable reproducible builds.

And so I can go on for days about this.

But what I will say, the challenge...

Do you also mind explaining what reproducible builds are for people who are new to this?

Oh, yes, yes, of course.

Sorry, I should have also asked that as well.

Yes, yes, yes.

So reproducible builds is, I've given the story before of you have a piece of software, let's

Let's just say Bitcoin, right?

Let's just say the Bitcoin Core software and it's all open source and you can see the source

code.

You can look at it.

You can audit it.

You can see that there's no malicious backdoor that's going to steal your money or whatever.

But when you run that software, you download, I don't know, a disk image, a zip file, a whatever,

from Bitcoin Core.org or GitHub or whatever.

How do you know that what you're downloading is what the source code is showing?

that way you're downloading is a binary, is an executable, let's say. They could have just before

they produced this binary, put a little back door in there, you know, compiled their little thing,

and then just distribute it to you, right? And for, let's say, you know, trillion dollar market cap,

digital cash, let's say, I think that's pretty important to make sure that nobody can do that.

And so reproducible builds is one where you have a set process such that given a code repository on GitHub,

anybody on Earth can run a command that would build that repository into the final disk image.

And the final disk image would be bit for bit reproducible.

So every single one and zero would be exactly the same.

And so you can have, you know, dozens of people all over the world run the same build and sign their own build and then compare and see that everybody got the same result and nobody was putting a backdoor in it.

Right. This is the beauty of reproducible builds.

However, I will say it is one of the, I get a little joy out of doing it just because like, I'm one of two dozen people in the world who like, you know, enjoy doing it and have contributed to the reproducible builds like website and everything else like that.

But it gets like, for example, in C, you can put in underscore underscore file underscore underscore, which just embeds the current path of the current file inside your program.

Guess what?

Everybody checks out their Bitcoin core directory to a different folder.

Some people put it in documents.

Some people put it in their own folder, whatever.

That makes it unreproducible, right?

Sometimes some programs are like, yeah, I'll just put a timestamp in there.

What's the harm?

I'll just put it.

I produced this thing at this time.

I'll just put it in there so somebody can see it, right?

Guess what?

That makes it unreproducible because one guy is compiling on this day.

The other guy is compiling on this day.

There's a whole slew of things.

And you basically need to look through every build tool and fix every little thing before

it gets to be reproducible and everybody can trust it, right?

So that is why it's such a challenge.

And when tools are not immediately reproducible, especially with like small teams and whatever,

I'm like, I get it.

This is like two people's full-time jobs just to get it to happen within like a year and

a half.

You know, I totally get it.

The reason why we're not reproducible is because this is a harder challenge even than Bitcoin

core because we are working we're building mac os and ios apps and not only are we building mac os

and ios apps we are using their system extension and network extension system which expects files

to be laid out in a very particular way and to be signed in a very particular way

And that verification differs from one operating system to the next and can break unexpectedly on offline systems.

So if you don't do it with Xcode, which is the worst piece of software ever written, if you don't do it with Xcode, you are bound to get it wrong, basically.

And so we have to do it with Xcode.

And Xcode is not something where I'm like, oh, there's this piece of reproducibility problem.

Let me upstream my change.

Let me change Xcode and upstream.

No, I have to work around it somehow.

I have an experimental branch and it works okay, but it is a lot of pain and I need to

rebase over our new changes and everything else.

It's quite a lot to set up.

So I don't blame people.

I think Molvat has now taken the lead in some reproducibility.

They've done it for Android, I think.

Yeah, I think they have it for Android now, which is, you know, props to them.

We should all be aiming for that.

And hopefully Obscura gets enough new users that we can get more people on the team and get more reproducible builds going.

Yeah, it's a tough problem.

I mean, literally some of the most trusted projects in this space that people love and adore still aren't fully reproducible.

So it's something I also want to see more of.

But right now, I think it's a very vocal minority, but there's these people that are like, well, if it's not reproducible, then I can't trust it.

And it's like, well, what's your stack then?

You're not going to have a very thorough stack because there's like three pieces of software I can think of off the top of my head that are properly reproducible.

Geeks is the only properly reproducible.

So there's actually, for those freaks out there who are listening, who have that mindset,

you know, like the iceberg, the reproducibility iceberg, an iceberg level lower is bootstrapable

builds.

You should Google what bootstrapable builds are.

Bootstrapable builds is, and this is what I did for Bitcoin Core, by the way, bootstrapable

builds is protecting against an attack called the trusting trust attack that Ken Thompson,

you know, of Unix fame proposed, whereby if you poison one generation of a compiler,

you can have that poison propagate through future generations of compilers, even if the code for each

compiler is good and doesn't have any bugs.

That is sort of like an undetectable virus that can go through.

And bootstrapable builds counters that to a certain degree.

But anyway, that's my little thing for the freaks.

Great.

Now everyone listening is going to go, "I'm not going to trust anything anymore."

No, no, no.

You can at least trust Bitcoin Core, let's say, because we do have bootstrapable builds.

There's actually a ticket open on the tour project that's like, look at what Carl did

for bootstrappable builds for Bitcoin Core and try to copy it.

And I was like, happy to answer questions.

That's cool.

Yeah, very cool.

Good timing, actually.

The next thing on my list here is clients.

So you brought up macOS and iOS.

A lot of questions might come from that.

A, oh, why did this come up 54 minutes into the interview?

I should have probably asked it earlier.

Why did you start on iOS and macOS?

Is your plan to extend to all major operating systems like Linux, Android, etc.?

Yeah, just speaking on that, I guess.

No, Henry, we're just going to forget about the rest of them.

And, you know, we don't want to have a sustainable business.

No, of course, we're going to get everybody.

And this is sort of our main sprint right now.

It's always hard, I think, running one of these software businesses,

like, you know, balancing everything, right?

Like, we want to do Android now,

but we also want to do custom DNS,

but we also want to do on-demand.

We want to do everything

and across all of the platforms.

I will say the reason why we started

with macOS and iOS

is because there's something called the,

I was just mentioning this,

the network extension framework.

And the network extension framework,

I'm, you know, I have love,

love, hate relationship with it for sure.

But the network extension framework allows us to run a VPN in a special sandbox mode

in which we don't have access to sort of the rest of your files and everything else.

And that's very special to macOS.

And macOS also has some integrity things built in, in terms of, which also causes problems, let's say.

What I mean by is program integrity things, right?

So somebody can't just publish an obscure VPN, you know, before we have reproducible builds, can't just publish an obscure VPN on SourceForge.com, on SourceForge and say this is obscure VPN, but it also actually does something bad, let's say.

We know that it comes from us because we go through notarization and things like that.

This actually happened in 2015 to Xcode itself.

The vulnerability was called Xcode Ghost.

So somebody through SEO got the top link for Xcode download and put a malware in there.

And everybody was downloading it.

And this was found in like, and the virus was found in some major apps like WeChat, like the biggest chat platform in China.

Like that was, that had that virus in it, right?

We just don't want any of that.

And we looked into the Linux options.

I am a Linux nerd.

There was not that many good sort of program integrity options.

And of course, the network extension being a little bit sandbox is somewhat unique as well.

And so we thought to release on macOS and iOS first.

But also one of the biggest things is that unfortunately, I use macOS.

and as a macOS, I do not feel comfortable shipping anything

that I haven't used and don't use 24-7, basically.

That's sort of my bar for things.

And I think that's the only way to make sure that we serve our users well.

And so if I'm going to be on macOS, that's what I want to be on.

But I will say to the listeners, we are working as fast as we can

to get it on all other platforms.

And do not worry, all of our engineers are Linux and Android freaks

with one person who uses Windows quite a lot.

So there will be a mutiny before we abandon any of the other platforms.

And, of course, we do want to do them.

Yeah, and not to add pressure,

but do you have a very loose generic timeline

on when to expect kind of those other options?

Yeah, I'm happy to say loosely, let's say.

Android is the next one.

Android is our next big thing.

Windows, I'll be honest.

Windows, we will definitely do it.

It's just way more difficult.

And so we expect it to take much longer to do.

Windows is just not POSIX, right?

It's not Unix.

It's not anything.

It's a custom network stack.

So we'd really...

And hey, if somebody knows the VPN subsystem of Windows very well,

please reach out to us. We would love to learn from you. But Windows is going to be pretty tough.

It's going to take a while longer. So next up after some cleanups for iOS is going to be Android.

Great. Okay, I have just a few questions left. Kind of the next section here is more about like

the day-to-day usage. And then the last thing is maybe a little bit more about you guys as a business

and who your team is. So to start speed. My first thought, one thing I do know about networking is

there's a lot of physics involved and that physics shows up for me as someone who doesn't understand

it as the more hops equals the slower it's going to be. So what's the obscure speed drop off,

if any, when you're doing this extra hop to Moldad servers? Yeah, I think we did do a...

Oh, I don't know to mold that servers, but I think last time I measured versus sort of line speed as in like no VPN was something like 15%.

That was sort of the drop off.

And I'll say that for us, we have made all the precautions to make sure that this is not noticeable at all.

What we've done is we've picked locations and peering locations for our servers that are good for Malvan and for the rest of the internet.

We've made sure that we write everything in Rust.

We try to make sure that we're not incurring any latency where we don't need to be and things like that.

And so what I will say is that for we have gotten, I don't like to toot my own horn, we have gotten emails.

I actually tweeted about this.

There was one guy who emailed me who was like, you know why I don't trust your VPN?

Because it's too fast.

Because I feel like I'm not even protected at all.

And I'm like, can I please use this as marketing?

Like, this is the best testimonial.

He like wrote a whole rant about it.

I was like, thank you, I guess.

That was, yeah.

And, you know, this is my experience, right?

Like, you know, because I use it every day,

this sort of goes back to the OS choice, right?

Because I use it every day.

If anything slows down, I am on my team's case.

Maybe they don't like it, but I'm on my team's case

because I want to make sure that as smooth as possible.

I will say one thing that might be interesting to more technical folks,

which is there is this concept of a performance enhancing proxy.

This is a Wikipedia article.

You can look it up.

It's actually not the case that more hops,

not always the case that more hops equals worse throughput.

The throughput is very important here.

Latency, of course.

Throughput is very important here because in a performance enhancing proxy, basically, if you have stream-like protocols like TCP, what it likes is either have, God, I think it's, what is it called?

the latency bandwidth multiple, something like that.

So basically, if you're on cellular, right,

if you're on cellular and you're connecting directly to google.com,

you have a link that has, sorry, packet loss.

Yeah, you have a link that's high packet loss and high latency, let's say.

However, if you have a hop in the middle, let's say,

a data center that's close to the ISP that the phone's connecting to, you can have one link that

is high loss and low latency. Yeah, exactly. Basically, you trade off one for the other on

one link and you trade off the other part for the other and you get overall better throughput. I'm

sorry, I'm explaining this terribly, but please look up performance enhancing proxy. It's an amazing

concept for the nerds out there. Very cool. And then defaults versus customization. You mentioned

earlier you want the defaults to be beginner friendly. You want people to be able to just use

it. But you also said you want to offer things like custom DNS tools. So currently, how do you

balance this? Are there major features that you allow users to customize at the moment? Yeah, yeah,

100%. So to me, it's like a UX thing, right? This is sort of just the user experience thing. We have

same defaults. And then you can sort of just have a little, you know, like YouTube has stats for

nerds, let's say, right? You know, like just a little, just a little accordion that drops down.

That's like, Hey, here are like the advanced options. Like if you don't know where, and,

you know, for the advanced options, we usually, there's actually one in, in our app right now,

I can look it up. Yeah, exactly. So under network, we have this strict leak protection thing that's

as opposed to protect against tunnel vision, right?

Right now, the UI isn't great.

We have like unstable and then like a whole paragraph explaining like

why you probably don't want this for normal users

because of macOS and iOS bugs always.

And so I think we should always give users the options

to do whatever they want,

but present them in a way such that they can make their own choices.

Now, have I gotten support emails about strict leak protection from people who don't fully

understand what it's doing and have not read that paragraph that we read for them?

Yes, but you know, such is my life and I'm okay with it.

And I'm happy that they're at least using our service.

Yeah.

And then I saw on your website that it's $8 a month.

So it's a little bit more than Mullvad, I assume because you're adding, you know, you still have

to pay Mullvad.

So there's a little extra there.

Do you have annual plans?

Is eight going to be kind of the flat rate?

How many devices does that include?

What's kind of the pricing structure there?

Yeah, so $8 is basically flat.

What we do if you do a year is just 12 times eight.

We don't like to sort of mess with these things until we're further down the line and sort of

understand our users some more.

I also don't want users to be in doubt that we can support ourselves and that we can maintain

this service going forward.

It does cost money to run these servers.

What that gets you is what we say is three active connections.

So if you're connecting through the apps, you can register, you can have 500 Macs and 500

iOS devices, let's say, signed in, but only three can be online as in connected at the

same time.

I think that's a pretty reasonable trade-off there.

We also generate WireGuard configurations.

So for anybody who's not on macOS and iOS, we have WireGuard configurations.

This is a compatibility mode, right?

We were just talking about how the first hop we use Quick.

This is a compatibility mode where we use WireGuard for the first hop.

Obviously, you don't get any of the obfuscation, you know, magic that comes with using our app.

With the WireGuard configuration that you download, that takes up a slot permanently, as in it doesn't matter if you're connected or not, because, you know, WireGuard sort of loosely has, but doesn't really have a concept of connection, let's say.

Got it. Now, just kind of, you know, zooming out and learning more about you guys, the company, etc.

and kind of finishing this out here.

Are you guys open source?

I meant to ask this earlier.

- Oh yeah.

I mean, we have our source code on GitHub.

We need to link it from more pages.

I was just telling--

- I'm on the homepage and I didn't see it there

which is why I wanted to ask.

- That's right, that's right.

It's github.com/sovereign-engineering/obscurevpn-client.

Anyway, we'll put it in the show notes.

I think we have a sync from our internal code base, like every hour or so, or maybe every

push I've forgotten.

So it's all there.

Even in the footer, like in the resources or...

Yeah, I was just talking to our team about it yesterday.

I was like, we don't have any place that links to this and this is completely public.

And yeah.

Cool.

No, that's exciting.

I'm glad it's open source.

And I also wanted to ask about you guys.

So I see some photos as well here and it looks like there's you and three others.

But like, what's your team look like?

When did you guys formally start and kind of become the formal team you are today?

So I'd love to learn a little bit more about you guys.

Yeah, yeah, 100%.

So it's me and four other engineers.

It's so fun.

I went on my whole vision quest after exploring Apple iCloud Relay and Mask and everything else

where I was implementing everything, making sure that it was at all possible.

And then after that, I have a very particular way of finding people and hiring

in that I go to GitHub global code search and just search up people who are using the APIs

and the libraries that we're using and look them up and look up like, you know,

have they opened issues?

Like, you know, are their concerns legitimate?

Like how smart they are?

And very slowly, I, you know, I email people and they come back.

And so I built a team around that.

And they are all, you know, they're all really great engineers.

And that sort of shows, right, in the quality of our software, the quality of our network connections and everything else.

It's funny, somebody made an observation when we were meeting together as the whole team.

Because we're a fully distributed team all across the world.

So we come together sort of twice a year that sort of every walk or every talk is like a tech talk.

You know, because somebody will just start talking about some random obscure corner of the Linux, you know, user space.

And, you know, everybody will listen and learn.

Very cool.

Well, is there anything you feel like that we missed that you'd like to inform our audience about or just any general takes on VPNs?

Or do you think we touched on those things?

I think we touched on most things.

I think that, to my eye, it is time for the VPN industry to level up.

I think it is time to level up to a higher level of trust.

I think the next generation of VPN companies should really have, should at the very least have their source code on GitHub.

I think that's like a piece of software that's processing.

It's rare.

Yeah.

I saw this week that Nord didn't have their app on.

It blew my mind.

I was like, what are you talking, like a piece of software that's processing all of your packets should be at least open source.

Hopefully sandboxed well and a stretch goal would be reproducible.

I feel like that's not too crazy of a claim to make.

I'm sure you saw the viral tweet with Nord.

Someone was like, why is the Nord Mac OS app 430 megabytes?

Right. That was, that was, that's, we are, I think, about 40. And most of that is like two video

tutorials we made for people so that they can navigate the permission prompts in macOS,

which are a little horrendous. But anyway, yeah, I think we should ask more of our VPN providers.

I think they really need to have a good trust story.

But also, I think you VPN providers should have a good story for the trust in their logging

policy, right?

Can we establish something where we don't need to trust that your pinky promise that you don't

log is the truth, right?

where we can have sort of cryptographic, mathematical, or engineering proof that you don't or can't

log.

I think I look forward to seeing everybody's answers to that.

Great.

Well, thank you so much.

And where can people find you?

Go to obscure.net.

Everything is there.

Follow us on Twitter, Mastodon, BlueSky, and Nostr for people who are interested in.

And that and yeah, we look forward to seeing you.

Awesome.

Well, thanks for the great interview.

I learned a lot and hopefully others did too.

Leave any questions down below or go ahead and reach out to Obscura.

I don't want to flag your support team.

So if you are.

The support team is me in my pajamas trying not to fall asleep while answering emails.

But that's fine.

I love talking to people.

Yeah.

Well, thank you, Carl.

and hope to have you back sometime in the future, of course.

And that, my friends, was the end of the interview.

Thank you all for tuning in.

I want to thank Carl for taking the time to be here

and explaining his service to all of you.

And I want to thank you all for learning a little bit more

about how the technology works behind the scenes

so that you can be better educated and make better moves

to better empower yourself and the people around you.

If you enjoy these podcasts, you can support TechLaur down below

by becoming a TechLorean, which gives you really cool,

exclusive access to things in our community and helps keep all these podcasts free to the public.

And of course, we have free ways to support us as well. I'll see you next time on Techlore Talks.

Thank you all again for listening.