Subscribe
Share
Share
Embed
Explore how software engineers use Effect to build reliable, production-ready software in TypeScript.
you might think
is just internal tooling,
but we have a lot of
internal tooling, just so you know.
which is why Effect is
actually being used a lot.
So, instead of running a Python eval
suite on the market,
there's a bunch, we write our own, but it's
using Effect.
So, we actually have an eval suite
and also a bulk test suite,
that's all using Effect with Effect retry
and so on and so forth.
it's a very
sophisticated system.
I'm just saying is that if we use,
you might think internal
tooling is like some toy,
but no, we're actually using
it almost on a daily basis,
it's actually production load.
testing out all
the endpoint, right?
Which is a crucial part of our operation.
well, I
mean, almost every respond
transformation is using the pipe now.
And then also I'm
introducing more and more generator.
So, down the line, we finally will,
it will be making it
easier for us to transitioning
over to Effect as needed.
Welcome to Cause & Effect a podcast
about the TypeScript
library and ecosystem called Effect,
helping engineers to build
production-ready software.
I'm your host, Johannes Schickling,
and I've been building with
Effect for over four years.
With this podcast, I want
to help others understand
the powers and benefits of using Effect.
In this episode, I'm
talking to Louis Vichy,
co-founder of OpenRouter,
the unified interface for LLMs.
In this conversation, we dive deep
into the Cloudflare workers architecture,
their routing layers, and how they're
replacing their own result type
implementation with Effect
for better retries,
observability, and long-running
agentic workflows.
Let's get into it.
Hey, welcome everyone.
It's so nice to have you here.
Louis, you're from OpenRouter.
Would you mind introducing yourself?
We also have the Effect
co-creator Michael here.
Awesome. How are you all?
I'm Luis. I'm a co-founder of OpenRouter.
Me and my co-founder, we met
through GitHub essentially.
He was building out a V0 of
OpenRouter using a framework I was
building for my previous
startup and that's how we met.
Then we just love
collaborating with each other and then
around May is when
OpenRouter first launched May 2023.
So early 2023.
Essentially, the idea of OpenRouter is
at the bare bone,
it is actually a
switchboard for AI models.
That's it. We reduce the switching cost
between like Anthropic
and OpenAI and Gemini,
and Moonshot, Cohere, and
Meta to essentially zero.
We're also centralizing
all the payment for you.
Instead of like 60 different providers,
you pay just us and
then you get some credit,
and then you can call
any AI model you want.
Essentially, that is the
basic part of OpenRouter,
but OpenRouter is
actually much more than that.
You have seen Elon tweet about us on X.
the secondary product of
OpenRouter is the data that
we are allowing everyone else to access.
The data about which model
is being spent on the most,
or which model is
getting the most utility.
We measure them based on tokens.
A token is about roughly full characters.
Last week, we did about
five trillion tokens a week.
Exactly a year ago though,
we were doing about 150
billion tokens a week.
That's roughly about
more than 20x growth.
By the way, this feature was released
literally the first month
of the product, I think.
Basically, my co-founder
and I were thinking about,
"Okay, we have this data.
Should we just let everyone see it?"
Because at the time,
there were so many benchmarks.
There were so many
Llama or Meta Llama benchmark,
Llama one, right?
They're claiming some
of these Llama fighting
will be better than GPT 3.5.
And we were like, "Is
that actually the case?"
Because when we were trying them locally,
they're dumb as hell.
So we were like, "Okay,
let's make a benchmark
that's based on real
utilization, real usage, real money,
being bet on the
model, like a horse race.
And see which one get
the most of this money."
And that's essentially
is the secondary product,
which is the ranking
page where everyone can see
which model is being spent the most on.
And we do cap the amount
of free usage on a provider.
So all of that is actual usage.
Like people spending money.
That is awesome.
So you mentioned you met
your co-founder in 2023,
May 2023.
Now more than two years later,
where are you at in terms of a company?
Is it just the two of you?
Probably not.
So maybe you can shed some light
on like OpenRouter as a company.
So we were two people
writing on the thing.
He was checking Datadog.
I was just writing
code for up until January.
And then we have our
first engineering hire
who also joined us
through an open source project.
We released a couple of open source
project at the time,
early 2024.
And then Sam joined us
through he sent us a PR
on another project.
and then we still have him to this day.
last year we have four people.
Now we have about 22.
So team grow about 5x,
but the utilization
has grew more than 20x.
So I think we have a decent wiggle room.
We make enough of coffee, right?
So got to buy more coffee.
essentially the
growth curve is exactly the same.
The first year we went
from zero to 7 billion.
7 billion token, you know, in a week.
And that's when I showed my dad.
And then a year later, September 2024,
we went to about 150 billion.
And that's when I
show, you know, my friend
is my south friend.
And now, right, I'm showing you guys
and a lot of people we're interviewing,
we have five trillion.
So it's been 20x and
then more than 20x.
Hopefully the next year we'll be higher,
but we'll see, right?
Well, AI usage is only gonna increase.
And as far as I know,
from friends and so on and so forth,
OpenRouter is probably the best way
to access those different models.
We had internal
conversations and one key aspect
that I truly like about what you do
is that you're not
just a proxy to models.
You actually
consolidate the model responses,
which is key here.
So a lot of people actually
mistake us as a wrapper, right?
Building a wrapper is very easy,
but building a dev tool is hard.
Building a dev tool on top
of the API is extremely hard,
right, because you have
to ensure that the Dev X,
the user experience of the
developer is actually good
so that they can stick to us.
And this is the key kind of mistake
that people didn't think about
until they need us, right?
We have seen this a lot, by the way,
is that user
essentially didn't think too much
of OpenRouter, they
try it once, they're like,
"Okay, I'm gonna use two or three model,
"it doesn't matter, so
I'm gonna do my own."
A lot of people have
built their own OpenRouters.
And then the moment they have to say,
"Okay, now I have to
add Anthropic, you know,
Claude 4.5 came out,
they have to scramble
their entire weekends, right?
Either weekends or
they're Tuesday and Wednesday
to add a model.
With OpenRouter,
it's right there, right?
You add the same
code, you allow your user
to switch up model name, you're done.
And that essentially has been a case
of how OpenRouter has been growing,
is that every time there's a new model,
we do partner with this
model to launch them early,
and then the moment the model is launched
is outright on OpenRouter
and people who miss the
train, right, come back to us,
and need people coming in
because they also want to try the model.
That's kind of
essentially the growth engine
we have had thus far, which
is leveraging our partnership
with all this new model coming in,
Can you maybe categorize
sort of like the products
you had to build out over the years,
and also maybe
orthogonally the challenges
that maybe you hit right away
or that just appeared
over time, I suppose,
like working at that scale,
just like pulling on that scale thread,
there's probably like so
many interesting challenges
that are coming towards us.
So the one thing that
me and my co-founder really like
is to have type safety, to be honest.
So, I kid you not, like the
initial version of OpenRouter
all the models, all the endpoints,
were TypeScript object,
TypeScript literal
type object in the code.
And then the way that we do reference,
we would do strict type satisfy.
As a way to do literally
like foreign key reference
in the code.
So our code were a huge
database of these endpoints
it's not crazy.
The moment we add a new field, right,
an alert, right, because
of TypeScript and satisfy,
an alert across all our code base saying,
hey, you have to add this
field in every, you know,
this place.
And then at the time,
right, when we add new model,
we add new endpoint, we
just tell Devin to do it.
It's automated because of type,
because of type safety, right?
The moment we just want to add something,
we tell the model, it changed something,
and then the rest is
just TypeScript, you know,
at type error.
So it was not that crazy.
it was actually usable.
even satisfying a bunch of, you know,
like literal number constraint as well,
which is very fun.
I mean, you know, fun
doesn't scale, right?
So we'll talk a lot more
about the data in a moment,
but is this entire system
still written all in TypeScript
or have you moved some parts
that are maybe like critical
to be more efficient?
Are those written in Go and Rust
or is everything still in TypeScript?
Nope, everything.
Everything's still in TypeScript.
We very much just TypeScript all in.
I mean, the main
reason why I think is fine
for OpenRouter to be strictly
TypeScript is because,
I mean, that won't lie, right?
When we actually hit the bottleneck,
we sure will switch to Go and so on,
but right now we're running on top of,
you know, Cloudflare.
And essentially, the
Cloudflare worker is much, you know,
better to use TypeScript with.
The tight strictness of TypeScript
and also the flexible, the
dynamic nature of it, right?
Allow us to move much faster.
The velocity of the engineers is much
faster, I would argue.
Technically Go can do the same.
It's just that we start with TypeScript
and we use TypeScript almost everywhere
on both front end and back end, right?
So I think it's best for a
team to have the same language.
I mean, time and time again,
I mean, time and time again,
the same story sort
of like repeats, right?
Yep.
Like if you're looking at Facebook
and Mark having written
the first Facebook prototype
in PHP, guess what?
Still mostly in PHP.
Yeah.
The same for like--
Instead of switching
from PHP, they ended up
making a new kind of
Built the hack.
Yeah, exactly.
Or it's like, I mean, that's
how TypeScript came to be.
It's like, I would say
the rationale of just,
the whole idea, right,
is that let's make sure
we just speak the same language.
So, in our code base,
everything is TypeScript.
Even the scripting is TypeScript.
No bash.
We don't use bash.
We use ZX and TS up to write script
to run like database migration or
something like that.
Or like even instant response scripts or
something like that.
So that every time that we communicate
is in the same language.
And so that someone
else can pick it up, right?
And we never really have dead code
because TypeScript is
very easy to kind of pull
into a shared package
and then you can reuse it.
Honestly, I'm kind of surprised
that this is not more common.
In the Ruby on Rails world, for example,
it's like so normal.
Like obviously you
build all of your scripts
as Ruby scripts and then you run them.
it's so surprising to
me that in a JavaScript repo
where like arguably
it's as nice, if not nicer,
to write scripts in JavaScript.
This is like in the name.
Like there's still
often like more bash script
than like custom JavaScript files.
It takes about three years
for someone to understand
that you don't need your,
database migration script to be as fast.
you also don't need
like a bespoke framework
that you're allowed to have like a script
that you can just like, even
if it does only a single line,
console log, hello
world, like that's fine.
You don't need like a
bespoke CLI thingy to run it.
Well, also I would also argue that it
take about five years
for someone to ease into all the
TypeScript, you know,
like not TypeScript, but most
of Node.js ecosystem tooling.
Right?
It took a while.
Because if someone has
been in a Golang for a while,
they probably have, they
probably pollute them, you know,
their home directory with
all the Go binary already.
I mean, I did the mistake three times.
So I'm like, holy moly.
I have so many of that.
the zero point five version of Go,
Was that you add Go in
your user bin
You clone it down and then
every single main module,
you just, you keep on cloning it.
There's no, there's no Go model.
Your machine becomes a Go machine.
Yes.
Your machine is essentially a Go machine.
And then with
Python, for example, right?
You get used to either
using UV or PIP, right?
You get used to
running almost everything.
And so when people
are, you know, kind of,
used to it, right?
It's really hard for them to switch gear.
And this is why they
having a list
ensuring that when the
team is still growing,
we ensure that the
consistency is there, right?
Then there's no
fragmentation of knowledge, right?
And people can learn very quickly too.
It's not that crazy.
So tell me a bit more about
how you've picked Cloudflare
back then
did you have a lot of
experience with Cloudflare workers
from previous projects and how, like,
I suppose there's also a
non-trivial amount of money
being spent.
So I'm curious how you
think about this as well.
It's actually very cheap.
I'm not, I'm not kidding you.
Like the infrastructure
cost is break even completely.
Just by the fee.
So actually before
Cloudflare, we were all on Vercel.
Everything was on Vercel until they,
they decided to charge
for egress cost, right?
but then also the,
the network connection layer
was not first class, right?
So everything was completely abstracted.
And we also have, even
though we were on Vercel,
we also have a Cloudflare
proxy on top as well, right?
For all the, you know, like
for the DDoS protection and so on.
So there's a three loop, three hoop.
And so, I'm curious to
hear a little bit more
about the
software aspect of like,
what does a typical line of
code in your TypeScript app
actually look like?
So currently you're
running on Cloudflare workers.
So in Cloudflare workers, for
those who have never written
a Cloudflare worker, it's essential,
it's very web centric.
So you have like an export default there.
And in there, that's
basically you export an object
that returns a fetch function.
And in that fetch function,
you get a request coming in,
you do your thing, and
then you send back a response.
It's like very simple, works really well.
But like how you do your thing,
it does like, that's the
actual app that you're running.
So I'm curious like how that looks like.
The way I'm thinking about Cloudflare
or worker in general is
just the entry point, right?
And if you ever kind
of use any framework,
entry point are just
supposed to be entry point.
You just try to attract your core logic
as much as possible so that
you can pull it into a package
or a shared package so that
regardless of entry point,
but when we moved from
Vercel to Cloudflare,
it's like literally
just moving entry point.
And so there's no drag,
so the core logic that we
have is plain TypeScript
function, right, with strict typing.
A lot of time I would love
to infer the type, right?
Because I hate typing thing manually.
I like to infer the
type as much as possible.
This is why one of my
favorite brand teasers
I give people is inferring type from
third party library.
You have to see if they have struggled
through this thing before.
the Cloudflare code
is we just use the Cloudflare
Hono library to wrap all the API
under the same path and then does it.
at that point, just the entry point
with authentication and middleware
and the rest of the core
logic are plain function.
Got it.
So how is that like that
inner chunk of business logic
that's like actually
what makes OpenRouter work,
like where you're
actually route the requests.
So can you describe a
little bit the architecture
and like how trivial is
what's happening inside there
or how non-trivial, probably the latter.
So maybe you can describe roughly
like what is the life cycle of a request
as it walks through like various stages
of what makes OpenRouter, OpenRouter.
let's make the tech request
we have Llama for
Maverick, let's say maverick.
Maverick has about what like
more than 10 different providers
or serving the model.
So when a provider is serving the model,
we call that an endpoint.
That's a concept that we have internally.
A provider is, this company, right?
Is the provider containing metadata
but the provider
meaning like company names,
what they do or their location, right?
Like the data privacy
policy and stuff like that.
The model is like the actual model on,
that's the actual weight to the model
with like context
length combined those two,
we call that an endpoint.
let's say when we call
the Llama 4 model Maverick,
there's like more than 10 endpoints.
The first thing is authentication, right?
But authentication is
literally just a function.
We authenticate first.
And then after you authenticate, we
return a result, right?
Either results, we
really subscribe to the result
monad paradigm.
So almost every function
we have returning results,
we check the result and then we return
a respond as needed.
we would then take the request
and we pass it through
each of these endpoint.
And while passing through the endpoint,
so there are two stack that we applying
for each of the requests.
One stack is the
routing, the routing stack.
The routing stack
allow us to like prioritize
and order the endpoint
according to what the user wants
or according to some
heuristic like latency,
throughput or price.
So the routings are a
stack of decision step.
And secondly, we have a plugin layers.
The plugin layer allow us to like
basically transform the request
and also transform respond
as the stream coming back up
have a FIFO stack style.
And that's essentially is
the life cycle of requests.
And then we just keep
on hitting each endpoint
until one that's
actually, one is like saying,
A, we can use a 200, we
can now return some data.
That's like a full cycle.
And the code for all
of this is very modular.
Every step is a function.
And then we just have an orchestration
that's run through each of those
would be very trivial to rewrite into
Effect down the line.
But we write this code
before there was Effect.
I think it was
Effect 1.x or something.
So, and I have to say
just a brief comment
because what you
described is sounding very similar
to what I had to do in my past,
which is some sort of
consolidation engine
between different APIs
that expose a single format.
The reason Effect is born,
like more than seven years ago now,
it's because I was CTO of
a Fintech company London.
And we were doing consolidation
over peer to peer landing providers.
And we would buy and
securitize those products
to then sell them to
institutional investors.
And we had a large piece of software,
which was basically we
wanted to have a single format
for loans, as effectively
all of the loans are the same.
There's a borrower, there's a lender,
there's an interest rate,
which is charged annually,
monthly, daily, or
whatever, and so on and so forth.
But then you had those providers
that would all have
outstandingly different APIs,
sometimes to achieve exactly the same.
And we had a very similar
approach to what you described.
We also initially
started with a result monad
in Scala back in the day,
and then testing became
harder and harder and harder,
to the point where we ended up
doing tagless finally in
Scala, which is
it's absurd in complexity.
It does achieve the goal
of having everything
modular and testable,
but it is highly complex.
And that's when I decided
let's migrate everything
to TypeScript because
well, our domain is IO bound.
There's no CPU compute
used at any point in time
except for data mapping, but even there,
like you said before
once you run your
JavaScript in enough servers
you're not really using
multi-threading to
handle a single request.
You just have to respond
to thousands of requests
at the same time, but two requests can go
to even two different
instances and so on and so forth.
Plus JavaScript is still very, very fast.
So I was like, let's move
everything to TypeScript.
And I was punched in the face
because none of the
infrastructure existed
that we were using in
Scala for TypeScript.
And that's why I built Effect in the first place.
So it was precisely to be in the
consolidation engine,
a testable consolidation engine
that had embedded open
telemetry
And this is what later became the open
source project Effect.
So I think it would be extremely good fit
for actually what you're trying to do
and funny enough exactly
the same that I was doing.
Which brings us to today
where my understanding is
that OpenRouter, the core
product, is not yet using Effect.
I think you're using Effect
already for some internal tools
and we can talk
about that separately,
but maybe addressing the
elephant in the room first.
You're here on the Effect podcast,
where the core product of OpenRouter is
not yet using Effect.
Yet you've dropped the
M-bomb before, the Monad bomb,
which typically is a
word that we're not trying
to use too much because
it's this interesting dynamic
of what is the role of
functional programming
in what we're trying
to achieve with Effect.
My perspective on it is
that we should follow the path
that React has taken,
where React is in essence
a functional programming framework,
but it does so in disguise,
where it doesn't overwhelm everyone
by being too dogmatic
about functional programming.
It's just an elegant system
that tricks you into becoming
a happy functional
programmer without knowing.
I would like to argue that
because I was there back in 2015, right?
When Angular and Ionic was
kind of a king at the time
because it's pushed by Google, right?
And everyone was writing a
directive and writing, you know,
some, basically it's very
much functional programming,
but it was a bloat.
It was bloating so
much that the key sell,
at least at the time I was still young,
I was still inexperienced,
but when I listened to
people talking about,
hey
if you adopt functional programming,
your program will be very, very lean.
And I think there was
that push of just whole ass
in functional
programming, don't be ashamed of it.
And literally say, fuck
object-oriented programming.
I think that was a push.
I would argue you should be,
you should not be ashamed
of functional programming.
And even if they push it,
I would say, I mean, sure,
then just go slow, you know,
then you have a slow code base.
I wouldn't say we're ashamed of it.
I think we're just
trying to like not make it
like an obstacle.
So I think we've so far
traversed the line pretty well
by like inviting
everyone who is FP positive,
but we've also not tried
to make it define the area
to the barrier to entry through
functional programming.
so there was that guy, I forgot his name,
one of the React engineers.
Dan Abramov?
Yeah, Dan Abramov, yes, that guy.
I think he has a good
sense of community there.
I'm curious what he would say about this.
Yeah, great point.
We should invite him to
the show and see whether.
But I would say that like the result,
I mean, the reason why we
reached for the result monad,
I mean, sure, most of my
previous project were try catch,
or like, you
don't even try catch by the way.
I just have a single error
handler at the top of my apps.
but then
whatever it try, right?
Whatever error is being
thrown in underneath, right?
Is an error type that
I have defined, right?
The problem is, down the line, right?
You soon learn that it's
very hard to debug those stuff,
right?
You soon learn that the stack tray
can be very, very
long and need to function
is like any anything complex.
It'll be very hard to go
through the stack trays
because you catching
it on the top, right?
So now you always want to
somehow manage your error
or localize your error
at the call site,
and then when you start doing that,
you have nested try catch and
it's a pain in the ass to read,
right?
Because every nested layer is,
I review my code on
my phone a lot.\
And yes,
almost every try catch
kind of nesting layer is like,
I just have to scroll on one side.
I just want to scroll on
one, you know, vertically.
The result monad
essentially solve this problem,
By explicitly forcing you
to handle the error case
as a value.
I think is a value add and it is weird
I mean, it is surely a different syntax.
I think people have to,
they just have to do it.
It's the same as when
React first came out, right?
With useState.
If you say useState is an
equivalent to the class state,
that's completely wrong.
Like class state is,
you cannot just check it in and out.
With useState of React,
you can just take it in
and out fully functional.
And then now your component is so light.
And that was, I would
argue the main selling point
at the time.
because of the pain, right?
The pain of, oh my God,
my class component is
so slow or so bloated
that with this thing
now, it will be so faster.
And it has to adopt
functional programming.
So I'm curious to hear a little bit more
of like when you grew
the engineering team
and like you as the founders,
you had the privilege
of like later foundations
or like set the taste as like,
hey, this is how we're doing things.
I think that's probably
still holds to some degree,
sure, like new people joining,
bring in like their opinions.
But I'm curious like which
sort of mini culture clashes
you've experienced by hiring someone new
and they look at this and it's like,
what is that?
How did you explain that?
And how did you sort of
like culturally assimilate them
to those ideas?
Well, I think we just have
to hire very good engineers
and good engineers with
experience usually can swallow.
Yep.
Oh, I mean, swallow not in a bad way,
but kind of like they have
seen things before, right?
It's not something crazy, right?
I would argue the crazy part
is we have like bash script there,
bash script with
different syntax of bash,
different flavor bash even, right?
And then we have
TypeScript, we have coffee
and then we have like,
maybe we use zest here and we did there.
That is the crazy part.
I would say that as we
grow the team, right?
We have tried our best to
if they tech debt, take
tech debt is widespread.
If they have to tech debt,
it widespread is unified.
So when we refactor, we refactor all of
them at the same time.
That's one key part of it.
If there is a certain framework we use,
we stick to it.
So like, for example,
right now we're using
a lot of our front-end,
our like React, right?
We know how to spend,
we are going to spin up a stencil,
you know, web component anytime soon.
The next one is probably will we react?
We'll stick to react for a while
until we decide, okay,
I think we, you know,
let's move everything, uproot everything
from react to stencil.
Sure, we'll have the
discussion, but not right now.
And then same with TypeScript.
The fact that we picked
TypeScript and forced everyone to,
hey, get used to it,
write script in TypeScript,
do everything in TypeScript
as much as possible, right?
That really forced us to
just keep on learning.
Even though they were
like from a Python shop
of GoShop or something else, when they
come in here, right?
I mean, sure, there's a resistant layer,
a resistant period where they're like,
how can I introduce a Go, you know,
the Go binary a little bit?
But then over time, right?
they start to get like used
to the TypeScript paradigm
because it's everywhere
in the code base, right?
You can have the AI agent look it up.
Also like Linter, we're using
biome right now for Linting,
plus Grid.
So we've been starting
adding a bunch of this Grid,
custom rules, which is very nice, right?
Because you can do like a
crazy regex on the code.
I'm using the same.
This is like, allows you
really nicely to define
some like, hey, this is
our way of doing things
from like all the 10 different,
TypeScript engineers
and they have 10 different
preferences of like braces there,
no braces there.
And then you can basically say,
this is the OpenRouter
way and run a command
and it gets linted and formatted.
Yeah, it is very awesome.
Like for example, we have a
rule that's strictly saying,
hey, if you have more
than three position argument,
just make an object.
Plus I feel like experienced engineers
usually focus on the problem
and the way we solve the problem,
rather than picking on the how,
in the sense of like
the thing you described,
that you use a result type for,
not gonna use result monad,
because monad is a term I don't wanna use
and not because I don't trust monad,
but because a monad would be an
abstraction on top of it,
which is the thing we're
trying to stay away from.
Result types are great
and they solve the problems you have.
They solve the nesting of try-catches
and so on and so forth.
So an experienced engineer that comes in
and sees the solution
might disagree on the syntax,
But the problem it's resolving,
it's clear they had it before,
they got tired of it
from other code bases
and they just digest
the current solution.
I wouldn't be opposed to
contribute to a code base
where there's a custom result type
and effect is not yet used.
I would say,
so that apply for like
a more senior engineer.
For the
slightly less senior engineer though,
I think the main thing is curiosity.
If they curious about why this was done,
they'll learn sure
learning faster on it too.
Because if you imagine you
can hire a very junior person,
but also very hard headed on some stuff
they've done before,
then they might not budge at all.
So I think for the less senior,
curiosity is where we strive for the most
and the way that we test
was curiosity is very simple.
They asked good question, really is.
So I think solving challenging problem
and then dig deep into it,
show that they have clear ownership
and clear digging, right?
The more they can dig, the better it is,
I think they show curiosity in a sense.
So taking a slight step back of like,
now you probably have a healthy mix
of relatively experienced engineers
and maybe some slightly more junior,
but very curious engineers.
And so you have right now,
a very sizable TypeScript code base.
Not sure, probably in the
six digits lines of code,
TypeScript repo, maybe even larger.
And you clearly have
Effect on your radar.
You've mentioned that some internal tools
are written in Effect,
but the main product is
not yet built with Effect.
And maybe you can just walk us through
and describe a little bit of like the
internal conversations
where you're considering,
hey, should we use Effect
for more parts of the core of the system?
Which sort of like
resistance questions you're getting,
how do people who are
already sold on Effect,
how they make a case for effect
and like just sharing your observations.
Well, the main observation
I have with the overall team
is almost every time
someone bringing a new technology
is a shiny object syndrome.
So very tough to sell, right?
in terms of handling errors,
we already have our own result monad
or result type that we already have,
which serve the purpose just fine.
and then we already have
this paradigm of writing,
make sure you write functions.
So when engineer, when
engineer look at Effect, right?
And the way it
orchestrating, you know, function, right?
A lot of time, the main question is,
why can't we use co-function then?
Why do we have to go
through Effect piping
or Effect matching when
a function work just fine
and an if and else work just fine too?
And if needed, we can
just make a new function,
do a switch cases with
complete check, right?
And plop it into the current,
functional flow, right?
And I mean, my response a lot to that is,
okay, what if you need to add a logger?
What if you need to add
a sidecar logger, right?
What if you need to add
observability or trace or span
on top of, you know,
this function call, right?
Well, the answer to a
lot of time would be,
well, I just add a new line, right?
Right before it.
Well, except that doesn't work for spans.
Precisely, right?
A lot of time, it doesn't work at all.
I think the main idea is
people don't really need
it at the moment
And so there's no huge push.
There's not, for example,
if we don't add Effect today,
we will still be around.
We will not be dying.
So that is why I would
say the main pushback,
because my team is
also very rational, right?
You know
they're also very busy, right?
it's not like,
it's spending time kind
of learning the namespace,
the Effect namespace, right?
Is already a drag.
Learning pipe is
already a drag, by the way.
So this is why the
first thing that I'm doing
is just introducing
small little usage of Effect.
We're using pipe almost everywhere now
when we composing a bunch of
function
in step by step, right?
And that helped a lot.
But the pipe function, by the way,
one of the main questions I
have for you guys actually
is a polymorphism of the pipe function.
A lot of engineers actually thought
that the pipe function is too flexible.
It's allowed you to
transform the type, right?
So it should have been called transform.
Transform pipe maybe, not pipe,
But to some engineers,
when we think about pipe
They assume it's
literally just a pipe through
where the type,
the input type should be
the same as the output type.
And it just
transformation step in between.
I'm curious what you guys
think about this, by the way.
What do you think about polymorphism?
Because essentially, the
this is the case where the
pipe type is not very scalable.
It only has 20 letters
You can only pipe 20 times.
I think there's reasons
why you don't wanna pipe
more than a number of elements.
And that's more along
the lines of limitations
in type inference
capabilities of TypeScript
and speed of type checking.
So we could easily add
200 arguments to pipe.
The actual implementation supports any
number of arguments.
The issue is TypeScript cannot deal
with the full polymorphism.
So you need to add specific overloads.
And before AI was very
tedious to add those overloads.
Now with Effect 4, I got Claude
to add the various pipe signatures.
I can easily tell Claude add another 200
so Louis can be happy.
that would be a solution.
But generally speaking about the
polymorphisms arguments,
pipe is generally applied to functions.
So you're piping the output of one
function to the input
of another function.
In that case, there's no
restriction about in and out.
You have data that flows
across a number of functions.
If it's understood in this
way, then it's not so confusing.
The confusion that I also had,
and at some point I was evaluating this,
is should the Effect
pipe be tied to Effect?
But then you have
cases where you might want
to convert an Effect to a stream.
You might want to repeat an
Effect
So you're having like slight
differences between the types.
So, I think putting an
arbitrary restriction
doesn't really help
in the understanding.
Now it is a good practice in your pipes
to not change the type
from the beginning to the end.
Otherwise while you
read, you lose context.
Like if you change the
type, most likely the best idea
would be to slice those
into two different pipes
and have variable
names that clearly state
what you're doing.
So this, I think it's
sort of best practices
and so on and so forth.
And I think with
touching on another point
that you mentioned, it's
the familiarity concept
and people's brain is wired into
imperative programming.
By imperative
programming, I mean the style.
Not the fact that
operations are actually executed
in an imperative sense.
I simply mean developers are used to
write normal functions.
Are used to write if statements.
They're used to write while.
They're used to write for.
They're used to write all of this.
With promises at the beginning,
we didn't get async await
And it was very tedious to write programs
because you had to think
in terms of this data flow,
this sequential nested data flow.
We escaped callback hell
to enter a venable hell
because then you had to access the
property from before,
then async await and
everything flattened out beautifully.
That you don't actually
get with a result type.
With a result type,
you almost end up writing
what they call Golang in TypeScript,
which is you call a
function then you have,
if you don't wanna end up in
pipes
what you end up doing
is you call a function
then you have if there was an error,
return the error, otherwise keep going.
If there was an error, well, in
JavaScript and in TypeScript
we have generators, which we rely heavily
in Effect to make code look
in the way that developers
are used to work with.
The mental model should be the same.
So my personal stance is that
pipe should not be the
default way we do things.
We should write code in a way that
everybody can understand.
And if you use a
generator, like switching mentally
from yield to await
it's not rocket science.
After you've read 10 generators,
you're gonna understand that you have a
direct mental mapping
that yield equals await, that's it.
And you apply your knowledge.
You can use it in a for loop,
you can use it in a while loop,
you can use it in an if condition.
You can do all that you want.
And if you want to access the specific
result of something,
you can say, okay, yield result of that.
And then you have
your if else that checks
for the exact
pattern that you would have
with a result type.
So I think like 90% of code
should go in that direction.
The rest of the code is
higher order behavior.
Like I have a
computation, I wanna retry it.
I don't wanna write a for loop.
I don't wanna deal with exponential
back off and everything.
Like you are integrating different APIs.
I think at some point
you have some retries.
At some point you have
to decide to back off
from the retries
because maybe open AI is down
and you all wanna keep
hitting the endpoint.
But those elements are
really not imperative.
They're not business
logic, they're not core.
They're just behaviors
that you add to the function.
And for that, I like to
do .pipe(Effect.retry).
That's it.
An LLM can understand it,
but a user doing
review can understand it.
And even a person
that doesn't know Effect
can read the function.
Okay, I understand the business logic
because it's all if else A equal B
and if these then that
while these conditions
through keep going.
And then they read pipe effect retry
while the error is network error.
Do not retry if the
response is malformed.
Cause if you call an API
and they change the response,
your decode logic will
keep failing anytime you retry.
So you might retry 500,000 times,
not getting anywhere.
So like to me, I
really wanna make it simple
for developers to read the code
and use pipe and use generators as tools
to really express business logic and
higher order behavior.
And I think like you are
in the very good direction
with the code base that you have.
And adding concepts one by one
is the right way to do it.
And from our private
discussions in the past,
I also see for example, you
use a lot of pattern matching.
Why the code becomes readable,
much more readable than
a sequence of if statements.
I would say that I actually consider
building everything using generator
but the code base started two years ago
and generator did not
work very well with async.
Async generator was on
a thing two years ago.
And you should not use async generators.
Yeah, you would not using it.
Now I would love to hear this by the way.
Why?
Why shouldn't you not
use async generators?
You should not have two
different representations
for sync code and async code.
That's madness.
That's actually also the
reason why I actually look up
by Effect by the way.
Oh, actually I look up what is it?
Function JS.
Because we had the result
and then we have async result.
Yes.
And you cannot escape from that.
So I think just to zoom out a little bit
and just to kind of like
observe the reflection here.
I think what makes your current code base
and your current programming styles
a little bit distinct from others
is that you went into this
with a very strong opinions.
Like, hey, we're not doing
try catch in this house.
And so you went like
this proper error handling.
And I think with that,
you're like way ahead
of where most TypeScript developers are.
most Effect adoption
I think happens to like
wherever the pain is most acute.
Maybe going to the doctor,
like you say like, oh my
gosh, this hurts so much.
But like while you're at the doctor,
you also realize like
you have those other things
that need treatment, but like one thing
gets you to the doctor.
And so the error handling thing,
like that pain like you've already
relieved yourself of that.
Yes, you have
appreciation for these other things.
Most people come to
Effect for error handling.
Also a lot of for other things,
but I think now
with the patterns
that you've like ended up with,
like just by talking through it.
Like I think we've like already uncovered
a couple of things where you say like,
well, yeah, actually that's not ideal,
but like, you know, it's like we have
good error handling.
But what Mike calls sort
of like the Golang style
of writing TypeScript.
I have done that myself as well.
And like this works
until it no longer works
or like until it just gets very annoying.
And I think the, what will
probably be the pain point,
like group of pain points
that gets you to the doctor
might be like better
retrying, might be observability,
might be like what
you call it like logger,
maybe metrics, et cetera.
And I think this will
make it actually easier
for you to adopt Effect
because you don't need
to rewire your brain.
So a lot of time we have been
incrementally adopting
some Effect, right?
Cause you know, lodash is a bloat
And I already added, I
already use my authority
to add Effect to the code base, right?
As I'm saying, we added Effect.
You're still
retaining your CTO authority.
Yeah, by the way, title wise, I'm not,
I don't claim CTO.
I'm just a co-founder
engineer title wise.
Yeah.
Perhaps there's a
better CTO out there for us.
We will see.
But I don't care too much about title,
but I care more about like
the dictator title for sure.
I'm a dictator for like BDFL for sure.
Yeah.
I can push the man by the way.
So like, for example,
You don't call a CTO, but
you get called a dictator.
Yeah, I'm a DFL.
And then, I mean, I was
saying like last week,
last week for the OpenAI dev day,
they were launching
the eval product, right?
And then right the Monday before,
they were paying a slack and saying,
hey, how come the
responding API don't have cost?
I just came right on and just, you know,
like push a thing on
man, two man in 30 minutes.
Shut up, right?
Get it over with, you know, like,
and that's the kind of thing I think is,
as a co-founder you
kind of have to do, right?
Is you gotta hold it by, you know,
you gotta hold it by
your throat and shove it.
Yeah, you have to know
when to skip the steps
and you have to know when
to impose a long-term vision
that is not yet apparent.
And sometimes you're wrong, like,
I do the same with Effect.
Sometimes I'm wrong.
I think I'm wrong and now, by the way,
there's one thing that
I might have been wrong,
which is like, I'm too ho-ass
into this idea of not using try-catch,
but then when it
comes to like the logging
or the tracing server,
you kind of have to use try-catch
so you don't allocate a new variable.
Because that way it's very easy to make a
copy of the variable
or when you do JSON parsing,
you make a copy of the variable
of a very deep nested
object of every value,
out of memory very easily.
Exactly.
And I think this is the thing where like
when people are sort of like in the,
the denial phase of Effect
and sort of like they, they
like justify to themselves,
like, we don't need Effect,
like we can do this thing like that.
Okay, yes, you can do
that thing like that,
like you do for error handling,
but like what if you need the,
if you wanna do the other thing,
if you wanna add that as well.
And like then, like all of those things,
they kind of like as you introduce them
on top of each other,
they don't seem to like
linearly add complexity,
but that's, I guess
like one of the unfortunate
emergent situations about TypeScript
is like the more you
add those requirements,
like observability,
better error handling,
dependency
management, like context management,
that doesn't make your code
like linearly more complex,
but exponentially more complex,
also visually more like visually,
exponentially more unwieldy.
And I think this is
where Effect comes in.
We've been adopting effect
in a bunch of like our retry
and like backup handling logic as well,
because the API elegant, right?
That's the whole idea.
So why would you write your own
when you can just pop this thing in,
throw a fetch and it's done, right?
And to review,
I don't think anyone
should adopt Effect in one day,
like in any reasonably large code base,
it would be absurd to
think one would rewrite
the whole code base.
we call effect a good virus,
in the sense that it tends
to spread when you started,
because now you have this retry,
but then you call a lot of APIs.
Obviously you are an API
consolidation provider.
So testing is essential.
You have to do integration testing,
you have to do end-to-end testing,
you might wanna do some unit testing.
And then you are in a
madness of Jest mocks
or Mocha mocks or whatever framework
you're gonna use Vitest mocks.
But mocks are really a very
complex thing to deal with,
because all your code is eager.
If you call it in the wrong place,
you have one piece
that is using one module,
another piece that is
using another module.
And so you might wanna elect,
use dependency injection.
I have seen how Epic Fabric
and their dependency
injection would help with that a lot.
Yeah.
Very good to use it, but you know,
I mean the team has
been using Vitest mock,
do mock and stuff for a long time.
By the way, we switched from Jest to Vite.
And that was a feat.
I know, I mean it's faster,
it's like the test runs
10 times faster, right?
We switched from prettier to biome.
I'm saying
we are not foreign to adopting new thing.
But it truly is all because,
it's pushed come to shove, right?
For example, if the test
take about five minutes to run,
oh my God, okay, let's
try to shave this down
and stuff like that.
Yeah, and to be frank,
I think like for your specific use case,
I think one of the
highest leverage points
could actually be the observability.
Because you already
referenced to debugging troubles
in the past of those,
once you have a hundred
of different functions,
the stack traces becomes unreadable.
Yes, they are.
They're much less unreadable
if you have spans at the right places
and you have a full dashboard
that tells you if something goes wrong.
The one thing with
observability and OpenRouter
is that because we run
the thing on the edge
and the worker runtime,
we don't truly have a span server
to continuously collecting the span.
So right now we're actually hooking
into the Node.js diagnostic channel,
which is not very good, I would say.
At least on the worker runtime
because the memory is very limited.
So we have to be very
careful with what we trace.
Well, I wish at some point,
we're gonna get to agentic behavior,
long-running processes
at some point in time
in this discussion.
Yeah, so are you familiar with the ECS or
the entity component
system in game design?
Yeah, I think the reason the actor model
in general is very popular in game design
is because in games you usually have a
lot of parallel entities
and they do share messages with each
other. It's a very
interactive world. Basically, again, it's
a world with entities that both some
entities can be movable, some entities
can be movable, some speak, some don't.
But even if you have something like a
room with a door, you have a character
that hits the door, the door is open and
All of those are sort
of unstructured. It's not very
deterministic. The player can take
different ways and so on and so forth.
And the actor model is perfect for that
case. And if you think about it, the
reason the actor model was even invented,
as far as my knowledge goes, we go back
to Erlang and we go back to the 70s when
Ericsson had to model telecom networks
where, again, you have very
undeterministic and unrelated entities
that communicate with each other.
I can phone Louis, New York, I can phone
Johannes in Germany, they can phone each
other. You have race conditions when two
people try to call the same
It's very similar to what you
would get in a game.
And in my opinion, it's very similar to
what an agentic system
looks like to a very high degree.
Have you thought about
agentic system that's steerable?
So when I was doing game design,
we do a lot of steer algorithm.
And steer algorithm is
all about managing a crown
Let's say you want to model
a particle system, right?
You have the particle
system, it's actually,
and each of them could
be a bird, by the way.
The particle system can be represented
by a bunch of bird
actors within this cap.
It's kind of a point
cloud of a bunch of birds.
And now let's say you
swarm this bird, right?
You swarm this bird through
an obstacle in the middle.
Now they all have to
independently dodge that ball
and fly around that thing, right?
Because the whole idea is when
you steer this flock of bird,
you just want to pick a pointer
and you move that
pointer to a certain location
on the 3D map and then the flock of birds
will just keep on following.
They keep on
following that point in space.
They both try to steer
away from each other,
but they all kind of act in tandem to
move to your pointer
and then they dodge popsicle on the way.
How do you think about steering agent
or in that same vein?
Have you thought about that?
To be frank, I have not.
I can apply some of my
knowledge from mathematics
in the past,
especially studying how birds
actually end up doing that.
And it's all about local communication
between one bird and the other.
There's no overall concept.
Basically the steering is almost like
one tries to steer a little bit,
the other follows the one
close and so on and so forth.
And I think even in that case,
it's all about the connectivity
between sorrounding entities
the neighbor flock
Each of the entity
they should have knowledge of
the entity right close to it.
So you put a small little boating block.
on each of them
so more like small clusters of
entities that sort of have
deep communication between themselves,
but almost not communicate
in the far distance one.
And also to quickly throw in here
to answer your initial question,
I've asked you to
chat GPT and asked how,
chat GPT how ECS
compares to the actor model.
And it says that they
overlap conceptually.
And the TLDR is that
ECS is about what you have
in terms of data and what operates on it,
like which systems optimized for
throughput and composition.
And I think composition here
is really interesting, right?
In a game you compose like a big scene
of like smaller things.
And the actor is about who does what
optimized for isolation concurrency.
So I think we can basically
build an overall system here
with both and they're
just like have sort of like
different aspects, like
what they're concerned about.
Like composition is interesting.
And the other one about concurrency,
like ideally you have both,
but both needs like this
different kind of concerns.
So I just wanted to weave that in
as I think it fits really nicely
the sort of like ball flock of birds
kind of scene we've been describing.
The kind of difference is more
at the level of the API they expose
rather than at the level of the runtime.
Because they're still both actors.
It's just that the API
focus is more on one aspect
or the other.
For example, even in
actor systems like Akka
you have different ways
of creating actors.
And even in a fat cluster,
which is our actor solution,
you do have different
ways of creating actors.
You can focus on the behavior,
which would be more along
lines of data in data out.
The kind of messages that the actor
can respond to, can send to.
Or you basically get a fiber with a cue
that you pull on and
everything you do is custom.
So you have more
flexibility, less structured API.
I think like the idea of a
distributed actor system
is very, very essential to
distributed systems in general.
And my point is really that
agentic behavior in
applications is a distributed system.
And it's just so happens that nowadays
everybody builds AI applications.
So they hit
the distributed system problems day one.
rather than facing them
further down the path.
And it's also why companies like Temporal
is getting renewed traction.
Because before you'd only use Temporal,
if you actually have a lot of customers
and what used to be an edge case,
like I always make the example of an
onboarding procedure
where you put in
username, you put in password,
you put in email, the
system has to record those
in a database and it
has to send an email.
Like those operations
happen within 100 milliseconds
of each other.
The probability of your server failing
between the two operations near zero
if you have queued two users that are
registering in a day.
But if you have 200 users that
registering the same seconds,
it's almost guaranteed that in a month
you will get multiple
instances of the issue
of the server failing in
between the two operations.
So that's when you usually
would reach for solutions
like Temporal or like Effect
Cluster.
The difference with AI is that the
average response time
is much longer than used to be.
Now we call an Open AI,
API to OpenRouter and it goes through
two different layers of proxying,
but it has to stay
alive for like 30 seconds,
the same connection.
The model can reason
for more than 10 minutes.
A lot of time, deep reasoning
can go for a long, long time.
And a lot of time during
these deep reasoning steps,
the model might not
produce anything at all.
So, OpenRouter, we actually,
a lot of time we set
to standing this,
the comment, like it's
actually in the SSE spec
is that you have to send
back comment to keep alive.
The way they see the connection alive,
otherwise either
the client would not know
if we're still alive or
The DNS server, the DNS layer would
actually cut us off.
And even any proxy that is in between,
because even when you call open AI,
you're not just calling
straight up to the server,
you have layers and layers
of content delivery networks
And each would hit HTTP timeouts
So like either you do web socket stuff
where the ping is in the protocol itself,
or you have to go on top and do a ping.
But the point is here, the average length
of an interruption with a service
went from being
milliseconds to being minutes,
if not days when the
other end-to-end reasoning
could be that we have still a pattern
where there's humans
in the loop of agents.
So sometimes an agent has to
stop and call another agent,
which is called a human that has access
to human intelligence
that is less artificial,
but they work exactly in the same way.
So humans are entities that share
knowledge via messages.
Currently we have three
entities in this call,
one entity is in New York,
one entity is in Italy, the
other entity is in Germany.
Yeah, absolutely right.
And we all communicate
with, we're sharing messages.
You know what's funny though?
I just saw the
premiere of the Tron movie.
And in the Tron movie,
so Tron has a bunch of program,
which is all humanized.
The program in Tron
communicate with the human outside
through MCP.
It's a fun detail in that movie.
I was like, "Ooh, cheeky."
I wanna play back the
same question to you.
So you asked Mike how he thinks about
how multiple AI systems could be steered
through this sort of
flock of birds, ball analogy.
How are you thinking about this?
Like just in terms of my own experiences
about a steerable AI system versus a non
steerable AI system,
actually two coding
agents I use on databases,
Codex and Claude Code.
Well, as of October 13th, 2025,
maybe a new thing drops
tomorrow and I'll switch.
But as of today, I am using Claude Code,
which can be steered and
I make a lot of use of it.
And therefore I'd use it
differently than Codex,
where it's much more like binary,
where I feel like if it's off track,
I like interrupt it and
say, "No, no, no, no, no,
please do this."
And I guess it is sort of
like similarly steering it,
but it feels much more like
stop and go and more forceful.
So I'm curious like
for you to extrapolate
where this is going.
And we're talking
about multiple AI systems.
So how are you thinking about this?
Yeah, one of my main pain points,
we're using Claude code or even like,
you know, Cursor or so on
Is that you set a goal or
you say, "Hey, fix this thing."
The problem is, right?
That's the goal already.
The flock could just find the goal.
I shouldn't have to like say,
"Oh, there's an
obstacle, there's an obstacle."
The obstacle is in
the environment itself.
go back to my college day
when I was listening to this system,
an autonomous steering system,
self-driving is the same idea, right?
You need sensor.
You essentially need a
way to emitting a bunch of
either event, which if we're using ROS,
it will be very simple,
ROS for barring over system.
It emits an event system,
when you're just sending into the system
a bunch of different light sensor
to tell you about the environment.
So first I think
every actor in the system
kind of have to emitting
some kind of sensory system
that tell the whole system,
"Hey, here's the
state of the environment."
And secondly, I think
that through those sensor,
you pick up some kind of weight.
allowing the system to
like course correct.
Automatically, instead of
having the end user have to like,
"Okay, we hit a roadblock now,
the type is not checking here.
What do you remove,
let the satisfy?"
So humans still have to right
now steer somewhat manually.
I would love for the end goal system is
I put a pointer in
the space in the world,
meaning a goal, right?
To say, "Hey, fix this unit test."
The swarm of bot
should just be able to say,
"Okay, I'm going to
use the browser agent,
which is like top tier browser agent.
Let's read the documentation
for the test, for example."
And then, "Okay, I know that you guys,
I'm using the test with the set timeout,
the set timeout set too low."
That's one thing that
it might see in the test,
or in the output.
And then from the dock of
the Vitest agent, right?
It will send back to my agent,
and my agent will then be like,
"Come on, okay, the set timeout too low.
So let's change that."
Just keep on doing that
loop over and over again
with our other agent,
or it figure out like what sensor it
needs from the world,
from the system to
steer its action as well.
And I think so far,
I mean, Claude Code is
occasionally doing recently well,
I would say.
It has a browser inside,
and like the chain of thought
that it has to like
solving a certain problem,
seems pretty decent so far.
here's like what I'm
seeing of the whole system,
like here's the data of the system
that can help me steer myself.
And I think what's
also interesting is like
that the entire ceremony is
like all sequential, right?
And so that therefore it makes it slower.
And so I think where
I'd like to see this go,
and I'm sure it will
go in this direction,
is that it's more of like an orchestra
performing a symphony.
And right now it's just
like solo, solo, solo, solo.
And while you're
encountering a problem,
like that might be a signal,
I was like, you know what, like,
let me go off like a little
bird just goes to the side
and like studies the docs more deeply
about like a system.
So it's like, hey, I
found something interesting,
like we're on the wrong track.
And like you could imagine multiple
of those like things happening.
And right now it's like
everything's just like happening
sequentially.
And yeah, I think this is very exciting
to see how this is unfolding.
And I like your guys steering analogy
and to look at it through like the ECS
and an extra system lens.
I have one observation
that I recently learned myself.
to take a step back,
I've been very late to the
AI coding game in general.
Johannes has pushed me for a while.
I developed the ability to ignore
Johannes for enough time,
but at some point sort of
penetrates the protection
when you're more
vulnerable and you're like,
okay, then I have to try.
And since Claude Sonnet I
think 4 or 3.5 came out,
I've been heavily invested in AI coding
and I really wanna make it work.
And what I've realized is
that I think we're putting tools
either on the wrong place
or we're not putting
tools in enough places
in the sense that
currently we rely on the models
to maintain some sort of loop.
But the models are
terrible to do looping.
They can't even properly assign tasks
They are exceptionally good
when called with a specific
prompt and a specific task,
then they're done.
The task might not be
done, but the LLM is done.
So what I found extremely
helpful in approaching tasks,
this is not really like AI coding
in the sense of the AI
assisting you while you code.
It's rather a task like, okay,
I give you the Effect repository,
create JS docs for every
single function in the repository.
Go, use 1000 parallel
agents to achieve the task.
I don't care.
Ready to spend 10, 20K
to do it, just do it.
we are approaching
the point where I think
there are ways to make it work
and even with normal like Claude plans
with $200 per month,
I'm completely fine
running multiple agents
and so on and so forth.
But what I've realized works best
sort of goes in the
direction of steering.
Instead of letting, for
example, the model calls tests,
calls TypeScript to get type
errors and so on and so forth,
instead of having the model do that
and assigning the model the
responsibility to do that,
I have those in a deterministic loop.
And if I have a type
error, I call the model again,
I have a system which is LLM based
that scores a specific example.
And if that is not happy, I
call the first model again,
saying, look, my Oracle is
not happy for those reasons.
I call it a deterministic outside loop
that ends up providing those events
and this feedback
inside the single agents
that are steering them
towards the right goal.
And that works exceptionally well.
Like well to the point
where you can leave it going
for a day, come back, you
have 95% of the stuff done.
And now that multiple
times, because I wanna improve,
I'm testing these on Effect 4,
because I don't wanna
break everything in Effect 3.
And to the point where the
quality is going up and up
and the automation is
also going up and up.
And I'm almost there
where it's almost better
than me writing example manually,
if I let it go long enough.
I have never seen that
with a normal coding agent
where you don't have this external loop
that steers it in the right direction.
So I think I didn't
understand the answer at first,
but actually I'm doing stiffing myself
now that I understand it better.
So I think this sort of setup
and system you've described
is becoming more and more mainstream.
And I think there's
multiple ways to achieve it.
Either it's already the de facto behavior
of like some coding agents,
or also this is where
those like coding agent SDKs
come into play.
Like I think Claude Code was
probably one of the earliest
ones that provided that.
Now, Codex has that AMP,
I think also just release theirs,
where you can like build
your own coding agent
and like express the sort of like system
and workflow that you want.
Converting a big repository
from having not no JS
doc to having JS doc,
which can be massively run in parallel.
That's a great candidate,
but doing that is very
different from building
whatever like a single new module
that you maybe have no
idea what the exact goal is.
And it's like, it's
much more exploratory.
Yeah, there's much more human in the loop
in the other direction.
Exactly.
So yeah, Louis, I'm curious
like how much you're already
using those sort of like patterns as
you're building systems
and maybe your thinking is more refined
in advance on this.
Not that refined yet, by the way.
We use a lot of AI model, AI agent.
So we use the Devon a
lot, you know workload.
And I think right now
the best way to steer Devon
is to have a lot of knowledge.
The knowledge truly helps steering it.
And Devon is selectively like it's select
the right knowledge per task as well.
So that actually easily have a lot.
I mean, this apply to
also, you know what,
I think Cursor and CLI
and all the other coding agent as well.
The more you can add into the memory,
the better they adapt to your code base.
in terms of having an orchestration layer
to like either use
multiple of them at the same time
or like picking the right
one for the right job though.
We still haven't like
think too much about it yet.
Yeah, right now it's all
just my workflow.
Maybe a related but
different question about this.
Given how Mike has described his
coming to terms with this new future
and he at his own pace
started like discovering
and like exploring, experimenting,
see what works well for him.
And I think that
fundamentally changed Michael's workflow.
I had the same process
and I think everyone does.
But I think what is
maybe interesting to explore
is how you as OpenRouter as a company
try to create like a higher
level process around that.
So for example
maybe a stupid way to go
about is say, hey, it's
the first of the month.
And we're again spending
three days of like going
from scratch,
exploring the best practices,
what it means this month
to do agentic coding.
Or maybe you have
like a team that is known
for being super cutting edge
in every first of the month.
They do like a presentation of sharing
their best practices.
Or you basically say,
we're not trying to orchestrate anything.
We just let emergence happen.
So I'm curious how much
hands on or hands off you are
of like fostering, like
embracing the productivity
and efficiency benefits
you get from agentic coding.
I would definitely
encouraging using agents
in almost every aspect of the code base
and also in the business as well.
I think the encouragement
is like the key part, right?
asking our team
to use it as much as possible to see,
essentially the idea is to
for them to learn the best way
to use it, right, to leverage these tool.
But the one thing I found though,
oh, okay, with everything,
right, there's a drawback.
The drawback is reviewing the code.
The key sin is when someone, you know,
they rely too much on
the bot and they say,
hey, do this and then they
just send it off for someone
else to review without
them reviewing themselves.
That's like a huge sin
that one day I will eradicate
it somehow
we have a way for
you to eradicate that scene.
Oh, awesome, I love it.
It's called Effect
Well, I'm telling you, so occasionally
the bot is very smart.
The bot, the agent is
smart actually, I don't know,
like a lot of time it
will actually change
either the TS config or it
will actually add an ignore
to the test setup or something like that.
Or sometimes it mark the whole module.
So the test looked like it
actually testing something
but it looked, you
read deeper into the test,
it literally marked the
module you're supposed to test.
Holy shit, it did.
So even, let's say, even
if we use Effect, right?
If the bot mark Effect, what can we do?
Just to take this
slightly more dystopian,
like maybe if you want to turn this
into a Black Mirror
episode, you could imagine
sort of like the AI
modeling like a mock human
where like the
instructions always you need
to get permission from the human.
So it models, it's mock
human that always says,
yes, go for it.
I have
experienced those problems.
Of course, everybody doing
AI coding has sometimes the,
yeah, I made the test fast.
You check and you added a deed dot skip.
What are you doing?
But I feel like most of the time
when the AI agent is doing that,
it's because it
really has no understanding
of the structure of the code.
And in my experience that often comes
from the agent
working toward a low level.
So it almost focuses too much on the
implementation details
versus what it has to do.
And that's why for higher level,
for example, application development
as opposed to library development,
like the AI is doing very poorly
to implement the internals of Effect.
It's doing exceptionally well to
implement the tests,
the examples
on the effect repository.
The reason is that
Effect almost produces code
that is closer to the human
versus being closer to the machine.
And LLM's being trained
on tons of human language.
They can sort of
reason better in human terms
versus that in code terms.
Like if you ask
Chat GPT a human type question,
very rarely it ditches the question,
there's not TS ignore at the human level
in a sense because the
language is higher level,
it's richer and expressing concepts
in a richer language is much easier
than boiling down to the very
details of the implementation.
And that's where I see the key shift,
the key difference when I build
application level code
and it uses Effect,
almost never I have a problem
like the AI ditched the task.
It's perfect to create schemas.
You have like, okay, I had these API
returned the structure like this one,
please create a service for me
that interacts with this API.
And it's fits down the full service
which almost always is,
I don't like the term one-shotted
because we're really not
here to one-shot things,
but close to
one-shoting the correct solution.
And I've seen that happen over a lot of
different code bases.
And it's where I think like
Effect can play a big role,
especially linking to
one of your concerns
slash questions that you raised
before we even started recording.
And I wish you will
use a different solution
that we are building
and where you will not
be memory constraints.
You will have spans
collected out of the box
and so on and so forth.
That'll be very interesting.
And then, well, I mean,
and you guys can
handle like caching, right?
So caching is another layer that's like,
caching at the DNS
layer would be interesting.
also no egress cost
if you can figure that out.
That part is hard.
Yeah, no, no, that is figured.
We don't have any egress cost
because we're building on bare metal.
So we don't pay for bandwidth.
There's no reason why to
charge people for bandwidth.
There's no reason why
to overcharge people
for CPU prices or go
5X what AWS gives you.
The only hard part, to be honest,
that we had to deal with is networking.
But networking in the
sense of private networking.
We got it figured,
kind of EPC layer stuff.
I had to write tons of
eBPF code in the kernel
to be able to figure that out properly.
But I think we're getting there.
And I hope that we're
gonna be able to provide you
with the right
primitives to deploy a platform
such as OpenRouter, which
is really what we're trying
to solve with Effectful.
Because with Effect, I think we solve
the production grade coding.
But at some point, code ends,
and you have to
deploy your code somewhere.
And especially with AI,
I feel we are hitting.
The real troubles were
if you were deploying
distributed systems in the past.
But you would only do
that if you were a scale up
handling millions and
millions of customers
Otherwise, a small
server, just a database,
would get you out of the troubles.
And the edge cases would not materialize.
Because the average request time was,
like what, 100, 200 milliseconds.
Now with AI and flows like
the ones you have in OpenRouter,
my guess is that
sometimes you have API calls
that takes minutes.
with minutes of
time, be sure that something
can happen much more
frequently than it can happen
in 100 milliseconds.
And so long-running processes
and less server-less pieces,
I think will be much more important.
And let me ask you a question.
How important is latency in a request
that takes five minutes to answer?
Not a lot, right?
Not a lot.
Maybe the handshake so that the client
is not easily, some
of them are very eager
to kill the request, but otherwise, yeah.
People don't really care.
Sure, but you might be,
like the important part
what might be to back off from a request
much faster to not spend money
versus actually letting the process run.
So I think for these kind of use cases,
and I've been discussing
this with some other people,
it's much better to have
reliable infrastructure,
maybe deployed
slightly outside of the edge,
so it's not close to your home,
but it's cheap, doesn't
have network costs whatsoever,
has very cheap compute
costs, very cheap storage costs,
and you can actually run a server
not being constrained by memory
because I feel like
being able to collect traces
is much more important for the business
than saving, what, 20 megabytes of RAM?
like RAM is cheap, or 20
milliseconds of request time?
or maybe like
100 milliseconds, yeah,
which is doesn't really do much.
By the way, the thing
that you might think
is just internal tooling,
but we have a lot of
internal tooling, just so you know.
So, which is why Effect is
actually being used a lot.
because we are
TypeScript everything, right?
So, instead of running a Python eval
suite on the market,
there's a bunch, we write our own, but it's
using Effect, right?
So, we actually have an eval suite
and also a bulk test suite,
that's all using Effect with Effect retry
and so on and so forth.
Yeah, so it's a very
sophisticated system.
I'm just saying is that if we use,
you might think internal
tooling is like some toy,
but no, we're actually using
it almost on a daily basis,
it's actually production load.
Reason why we don't use it in the core,
because while the core is working,
don't touch it.
But we do have a lot of internal tooling
to do testing out all
the endpoint, right?
Which is a crucial part of our operation.
We have, well, I
mean, almost every respond
to transformation is using the pipe now.
And then also I'm
introducing more and more generator.
So, down the line, we finally will,
it will be making it
easier for us to transitioning
over to Effect as needed.
by the way, to the point,
I'm actually still using async generator.
And the main reason why
I'm using async generator,
because inside this generator,
we have some async
process, which unavoidable.
No, no, my prior
comment on you should not use
async generators is
because you should not have
two representations of code.
But if you're using
promises with result types,
100%, please use async generators
so that your developers
don't have to unwrap
every error manually at every point.
But it's still a much
better user experience.
I just meant that if you
have one single representation
of code, you don't need
to use different types
of generators
Plus, adding like one
of the key troubles I had
in the past was, okay, I
have a very synchronous
code transformation, like these takes
an API, does this
this, this, this, that, okay.
Now I wanna emit a Kafka event.
Not now, not at the
beginning, but a month later,
I would write an event in Kafka.
Look at what's the writing to Kafka
is an asynchronous operation.
So I stitch it at the
end, and I have to bubble up.
Everything becomes async.
That's a problem.
Now it's still a much
better problem to have
versus writing callbacks.
Because callbacks don't
have function coloring.
Callbacks are fine for sync and async.
They are a unified
representation of computations.
It's continuation passing
style that we ended up calling
callback passing style in JavaScript
because we just like to
invent new terminology
so that when somebody looks at history,
they don't see the
theoretical papers from the 70s.
Otherwise they understand
we're just stealing ideas
from the 70s and making
them production ready.
But there was a unified
representation of code,
which leads to, boom,
boom, boom, boom, boom,
of callbacks.
Now with promises and async
await, you no longer have that.
But you do have the two different
representations of code.
And that's much better
than reading the code
like you described before.
You want to review on the phone.
There's no way you're going
to review 100 nested callbacks
with your phone screen.
By the way, no way
you're going to read that
in a normal computer screen.
You might need Johanness' set up
with exceptionally large
screens and so on and so forth.
But even there, I
think you should optimize
for a different problem.
So I think the final solution, which
is I believe what we have with Effect
is a unified representation that allows
for sequential representation.
But still, the intermediate point
of having two different ways of doing
things that all look
good is perfectly viable.
And especially in your
code base where you're
using a custom result
type, all the time go for that.
It's amazing.
Yeah, this is the thing
that I was looking for,
which is the way to represent in both
async and synchronous
process
in the same container, essentially.
that's something I'm excited
about most, about Effect, right?
It blurred the line between async and sync.
And you can now
orchestrate them in the same flow.
So it seems like you're well on your way
to introduce Effect more and more
throughout the entire system.
And it's very interesting
to hear how you've basically
introduced mini versions and
little pieces of Effect
already to make it
more and more swallowable
by the rest of the engineering culture.
So I feel there's plenty of stuff
here that we need to unpack
probably in another future
episode to hear how your
Effect journey is continuing.
Before we're wrapping up, though,
we talked before about that as part of
your onboarding culture
to ask a new employee
what's the latest rabbit hole
they went down on.
So what's been your latest rabbit hole
that you barely emerged again from?
The most recent one-- I recently moved to
New York, by the way,
two weeks ago.
Sorry, two weeks ago was two weeks ago.
So that was a month ago.
Oh, my god.
And that led me to hold rabbit hole
buying stuff for my
apartment, essentially.
Well, there are two rabbit holes.
One is on buying a bed,
and one is on apartment
finding in New York.
Luckily, I didn't--
I was on a time crunch to find an
apartment in New York.
So that
hole was a bit shallow.
But the one about finding a bed, though,
a good bed for your
back, is it was very deep.
And thankfully-- so I'm
working off the Marlin.
It is a very bougie cooking
space down in Flat Island,
Manhattan.
And right next to this place, there are
three mattress stores.
Like right next to me is the Sattva,
which is one of these
very high-quality hotel--
mattress.
Down the block, there
is a mattress firm store,
which contains a bunch of
variety of mattress store.
And then on the other block is Avocado,
which is this another
very bougie, very expensive
mattress.
And I got to go in and try
all of them, spend about a day.
And it is fascinating.
The final choice I made
was to get a purple mattress.
This is not a purple
mattress app, though.
I just tried all of them.
So the best-- I'm
going to say the best thing
that you can do when buying a mattress,
you just try them out
yourself, tug it in, go on all side.
You've got to try all side
of your sleeping position.
Yeah, I roll it all over
and try to tug it in to see
if your spine can get enough support.
And of all of them so
far, for me personally,
the purple mattress has been the one
that I can actually support my spine.
The second option was the
Sattva, the Sattva Classic,
which is somewhat soft, but
also decently firm in the bottom.
And yeah, the purple
mattress is kind of interesting.
It's not one of those
put traditional bad.
He's going to send
over the affiliate links,
and we're going to put
them in the description.
It's going to be a podcast first.
There was no affiliate, but
thanks to a very good friend
of mine.
He was my co-founder of
my previous company, too.
He went mattress
shopping together with me
And then we did the good
cop, bad cop negotiation
at the mattress firm.
And we get $1,000
discount on that mattress,
which is kind of crazy.
$1,000?
$1,000.
$1,000.
A $1,000 discount on a mattress?
Yes, on a mattress.
That kind of suggests
that the mattress was
more than $1,000.
Yeah, I guess.
Yes, the mattress was very--
I mean, I talked to a friend of mine,
and he told me, you
spend a lot on your back.
You're going to need it.
So yeah, I think it was good spending.
Yeah, very comparable bet, for sure.
Unrolling the bed,
though, was insane, though.
It's very hard.
So recommend, if you're buying a bed,
just have two people.
Just have a friend.
Fair enough.
Otherwise, you'll ruin your back while
setting up the mattress.
That's no good.
Awesome.
Hey, Louis, this was an absolute treat
for us having you on.
for us having you on.
I'm looking forward to
having you on in the future again
and hearing more about how
OpenRouter is adopting Effect
and all the crazy
growth you're going through.
Also, a big thank you
for Mike for spoiling us
with his presence
throughout this episode, as well.
Very, very interesting conversations.
I've learned a lot.
So thanks a lot to both of you.
Awesome.
Thanks, Mike, for
putting me in on the podcast.
It was a big pleasure.
And I hope to see you soon
when we come to New York.
Absolutely.
I'll see you then.
All righty.
Have fun.
Take care.
Thank you for listening to the
Cause & Effect Podcast.
If you've enjoyed this episode, please
subscribe, leave a review
and share it with your friends.
If you haven't done so already, you can
join our Discord community.
And if you have any questions, feedback
or suggestions about this episode or
about Effect in general,
don't hesitate to get in touch.
See you in the next episode.