Subscribe
Share
Share
Embed
In this episode, Attila Večerek, Tech Lead & Staff Engineer, joins our host Johannes Schickling to discuss how Zendesk incrementally adopted Effect in a polyglot environment with a large codebase.
Effect is an ecosystem of tools to build production-grade software in TypeScript.
Effect is an ecosystem of tools to build production-grade software in TypeScript.
#Effect #TypeScript #Zendesk #softwareDevelopment
Song: Dosi & Aisake - Cruising [NCS Release]
Music provided by NoCopyrightSounds
Free Download/Stream: http://ncs.io/Cruising
Watch: http://ncs.lnk.to/CruisingAT/youtube
Song: Dosi & Aisake - Cruising [NCS Release]
Music provided by NoCopyrightSounds
Free Download/Stream: http://ncs.io/Cruising
Watch: http://ncs.lnk.to/CruisingAT/youtube
- (00:00) - Intro
- (03:13) - Being an engineer at Zendesk
- (06:05) - Challenging the status quo
- (13:10) - Introducing TypeScript at Zendesk
- (20:22) - Adopting fp-ts
- (25:19) - The transition from fp-ts to Effect
- (31:00) - DX adopting Effect
- (37:15) - Implementing a Kafka consumer with Effect
- (42:18) - Dependency injection
- (48:33) - The power of TypeScript & Effect
- (53:03) - Onboarding developers to Effect at Zendesk
- (01:15:37) - Excitement for Effect Cluster
- (01:19:30) - Outro
Creators and Guests
What is Cause & Effect?
Explore how software engineers use Effect to build reliable, production-ready software in TypeScript.
like the dependency injection,
like dependency management as well.
It's just so great with Effect.
I feel like I'm much more
incentivized to write unit
tests and I don't know how other people
think about engineers at companies like
Zendesk and like Facebook and Google, like big companies
like how they
deal with these things
on a day-to-day basis.
In the end, like it doesn't matter
how skillful or experienced you are.
It comes down to incentives all the time.
Like you do things that you
are more incentivized to do.
So, If a language or a framework or a library
makes something really easy to do,
you will do it regardless, whether it's
the right thing to do
or the wrong thing to do.
Welcome to "Cause & Effect," a podcast
about Effect, the TypeScript library,
and ecosystem helping engineers build
production-ready software.
I'm your host, Johannes Schickling, and I've
been building with
Effect for over four years.
With this podcast, I want to help others
understand the benefits
and powers of using Effect.
In this episode, I'm talking to Attila
Večerek, a tech lead at Zendesk
and long-term user of Effect.
In this conversation, we talk about the
challenges of building production-grade
software in TypeScript
and how Zendesk ended up adopting Effect
in one of their main products,
serving millions of users every day.
Let's get into it.
Welcome Attila to the very first episode
of the Cause & Effect Podcast.
How are you doing?
Thank you.
Thank you, Johannes.
I'm so happy to be here.
Uh, I'm doing pretty well.
Thank you.
How about you?
I'm doing fantastic.
Really looking forward to doing
this podcast and to
do it together with you
since I think we've been in
touch now for quite a while.
We met like half a year ago, also
for the first time in person at the
first Effect Conference in Vienna.
And yeah, really excited to do this
together and share more
your story of like
discovering Effect and using Effect.
So you're using Effect at Zendesk, but
would you mind giving a quick
introduction of who you are?
What is your role at Zendesk?
What have you done before?
And then we can dive deeper.
Sure thing.
Hey everyone.
My name is Attila, Attila Večerek and
I've been at Zendesk
for almost seven years.
It's going to be seven years
exactly this year in October.
Currently I'm a tech lead at a
team in the Guide organization, which
is responsible for the
help center part of Zendesk.
So, if you see any help center
articles or community
and stuff like that,
that's, that's what I do.
And most recently I also got the
responsibility of re-vamping
our customer satisfaction feature.
which is one of our most used
features besides like the core product,
which is ticketing itself.
And yeah, I'm very excited about that.
And that's actually the feature
which is built using Effect.
That sounds awesome.
Yeah.
I'm excited to hear more about that and
go in depth, but would you
mind describing a bit more of, given
that, there are, I think,
several thousands of engineers at Zendesk
and not all code there is written
with Effect, that might change at
some point, but would you mind giving
a quick overview of, what it means to
be an engineer at Zendesk?
So Zendesk is really large, we
have many sub orgs and, or
departments and each of them, even at the
team level, like there are huge
differences, we're truly a polyglot
company when it comes to languages
and different kinds of tech stacks.
I still think most of our code is
still written in Ruby.
The company started with
Ruby and Ruby on Rails specifically.
We have two large Ruby monoliths.
both of them I have to
interact with from time to time.
That's, that's always
an interesting
experience, especially
when you've been working with Effect
for a while and then you have to go back
to Ruby and like, Oh, okay.
Things are a little bit different here.
But besides Ruby, there's also like a
ton of Java, some Scala, some
Golang, yeah, and obviously
TypeScript mostly for
front-end, but, we
have also some companies that we acquired
who brought in a lot
of backend TypeScript.
And yeah, we have some, some of the
TypeScript on the
backend in Guide as well.
So it sounds like Ruby has always
been the majority of code and still is.
Is there any technology, any
language that you feel like is really,
on its way in and might replace Ruby
as the primary
language maybe at some point?
That's really hard to tell.
I don't think Ruby will go away ever.
These huge monoliths, they're here to
stay with us for
until Zendesk will exist.
I'm pretty sure of that.
We had some initial thoughts of breaking
the monoliths down, but it's really
hard to justify these efforts, when
it comes to product and, and like the
business needs of the company.
So for all the startups out there,
like if you start with a technology
and you become like large company and
really largely successful, all the
tech that, that you accumulate for all
those years, that's going to probably
stay there for, for a while.
So yeah, maybe, maybe think about twice
about what you start with.
Not all this tiny decisions, but like the
major ones, it's, it's good
to put some thought into it.
Oh yeah, for sure.
But I mean, Zendesk is a very,
profitable and very great company.
So I suppose it wasn't
the worst stack decision.
If it led Zendesk to this point
today and given that you, are also
now leaning more into TypeScript, I think
there there's, you can probably use the
best tools for the appropriate jobs.
But speaking of TypeScript, I think this
is one example of like a theme
that I'm noticing, which is that you seem
to challenge the status quo a bit.
when it comes to engineering
within Zendesk where you see, okay,
this is how things are done.
And then, you seem to, send some
ways to do things possibly better.
And this is, I think we're using some
colleagues, if I understood
correctly have, at some point looked
at TypeScript and said, actually,
this is a bet worth making.
Can you share that anecdote?
Yeah, sure thing.
So maybe just to understand for maybe
people who are not familiar
with Ruby on Rails, Ruby is a
dynamic programming language.
It's great for really one
engineer or tiny teams that know the code
base by heart, it makes
them really efficient.
So productivity wise,
it's, it's terrific.
So starting with Ruby for Zendesk was
definitely a good bet.
It allowed them to have like a
velocity in the beginning that was
absolutely crucial to become successful.
But you know, working with a dynamic
type language, especially like
with a heavy OOP focus where you have
lots of like design patterns.
I'm not sure if you have worked with
teams where people would argue
like, Oh, you know what, for this
feature, we should
use this design pattern.
And then the other side would be arguing,
no, no, no, uh, look at,
switch your mentality.
Like, you have to look at the problem
this way and then they
bring up adapters and whatnot.
And DDD and all these things.
At a certain scale, it becomes fighting
an uphill battle all the time because you
have to be fully aligned with all
the people who work on the code base.
You have to agree on these things.
Like what are the design
patterns that we are introducing?
Okay.
Are we shifting to a
different design pattern?
Are we re-architecting
redesigning our modules?
Are we taking out and
extracting reusable modules?
And how do we do that?
Are we doing some, you know, some
module boundaries within rails and
use rails engines or some other ways of
enforcing these module boundaries?
So all of these decisions, like you
really have to pull the same rope in the
same direction and then it's good.
But at some point, now let's say you have
this huge monolith, which we have,
and you have 50 people or even hundreds
and thousands of
people contributing to it.
Having this alignment across the
entire organization is really hard.
So what you end up having is a mixed bag
of things and, you know, parts of
the repo, use one design better.
Another part uses another one.
And then there's this new design better
that you're pushing for some
modularization and
then that's a migration.
And then there's tons of migrations going
on at all time, like five, 10, 15
different types of
migrations in a code base.
And then let's say you're a contributor
who just wants to implement like a tiny
bit of a feature, which is necessary in
that repo so that the other service can
communicate with it.
I don't know.
You, let's say you want to emit some
events from
the main monolith and then over
Kafka or over whatever message bus so
that you can consume
those events in your microservice.
Well, suddenly you'll see five or six
different examples of how to emit
domain events, or Kafka.
So what do you do then?
How do you find the people, that know
what's the current status quo, or
will you just blindly pick the one that
is the most prominent?
I mean, that's, there's no problem going
with that route either, because if it's
the most prominent thing and let's say
it's something that
we're moving away from.
You can just commit to that and it will
be moved away from with the rest.
But then, you know, if you pick something
in between, then that also brings the
risks that the people who are doing that
migration will just
have bigger and bigger
headaches because now they need to
migrate different kinds of ways of doing
things into this one,
one way of doing thing.
so these are the biggest problems
that I see with Ruby, alignment,
constantly fighting an uphill battle.
You introduce a new feature, but at the
same time you introduce a bug because
you miss something and there's constant
action at the distance.
You change one part of the application,
which has some
unforeseen or unpredictable
effects on another part of the
application because
everything is reusing everything.
That makes a lot of sense.
So it rather sounds like you're,
you're running into some like social
alignment issues where you just have so
many engineers, like an all probably very
brilliant and with like the best
intentions trying to design it with their
understanding of the world.
But if you have thousands of engineers,
this can lead to some, some very wild
different directions and to enforce that
alignment, I think if
you're, if you don't
have a types type system at your
disposal to help with
that, to be even aware of
like, Hey, here's like some divergence.
We're unopigniated maybe, or we
tolerate both, but even to have some
visibility into this and
help drive the alignment.
I think this is where a
type system can help a lot.
And I suppose that is what's what has
made you so interested in TypeScript.
Yes, absolutely.
So that, that was one of the biggest
driving forces behind the
adoption of some language
that is statically typed.
it has tons of
benefits, not just like this alignment.
Very recently, I just actually had to
go into one of these large Ruby
monoliths and there's a feature flag
that I introduced and some different
behavior for when the feature flag
is enabled and now this feature flag has
been rolled out fully
for, for some months now.
So I was looking into removing the,
if.else branch for the feature flag
to keep only the logic that's basically
used now that it's fully rolled
out and by removing that one line of
code, I broke like
hundreds of unit tests.
Because the unit tests had no idea that
there was like a dependency on some HTTP
call somewhere down the line.
So now I have to find like a place where
I can maybe permanently mock this call to
a specific endpoint for all the
tests forever, because I
cannot pinpoint like the
hundreds of different places
where this is exactly called.
So I have to mock it
like on a general level.
So, so these are also kinds of issues
that you bump into
and using languages and
frameworks like this which is going
to be another segway into Effect, I guess,
later on with the
dependency injection.
Yeah, most certainly.
So it sounds like you're building
production grade software at scale at
Zendesk in many different flavors,
many different technologies, but at some
point you've introduced TypeScript and
that's the foundation for the services
and products you're building.
What was that that journey like?
So you now have a type system to help
with many things, but I suppose it still
has not given you the silver bullet that
makes it super easy to build production
grade software.
So what are some of the challenges you
were still facing and possibly still
face today and were
you reaching for Effect?
So if I have to reflect back on the
time when we were introducing TypeScript
at guide for the first time, I think
obviously what we wanted to avoid where
all of these problems that come with, uh,
shared monoliths. We just wanted some
some service, which is nicely isolated
and we have our own
deployment schedule, which
is basically on demand.
We don't need to deploy it on a
weekly or biweekly cycles.
And then of course the maintenance,
improvements,
because now you have a type
system, so if you make a change in one
place, the compiler
will aid you in updating
all the other call sites.
So that, that was
obviously one thing.
But we also wanted to make sure that
we don't get into these like reliability
issues that we were having with the box.
So like type safety was
really important for us.
Then we're looking into how to, because
there are many different ways how you can
write TypeScript, right?
Like you can be really
lenient and allow any.
Right.
You can typecast here and there.
The language is quite flexible.
You can use as much type safety as
you want, or you can
go towards the other end
of the spectrum and use like the most
brutal compiler
options that are out there
and make it super, super type safe.
And then you even start thinking about,
okay, how do we do
end to end type safety?
Right.
Because there are certain things that
come out of your system.
Like on these boundaries of your system,
you have a HTTP request coming in.
Maybe HTTP request going out.
Kafka message coming in.
Like you need to parse these and at
the inbound of your system, you have to
be able to work with
it with a static type.
Right.
So, so how do you do all of these things?
Rolling it manually is really tedious
with all the type checks at runtime.
Especially if you have like really
complex objects it
is really hard to do that.
So obviously you can compose like start
at the leaves and then
build up all of these,
but that's a lot of work.
So we started looking into other,
like maybe even, even other programming
paradigms like with, with Ruby, like of
course there's like a
ton of OOP stuff, but
then with TypeScript, we were realizing,
oh, okay, maybe we can actually leverage
like, some functional programming
concepts to build
more composable software.
Because one TypeScript
service was not our goal.
We were already thinking in multiple
TypeScript services and in the end it did
become reality and then we wanted to
share code in between them.
And then how do you do that?
Well, we came up with a mono repo of
different libraries that we can use that
implement like Zendesk concerns, things
like sharded database
clients that know how to
connect to our sharded database and yeah,
many, many other
libraries like this, these
internal ones, these
productivity booster ones.
So we started implementing those and then
we're like, okay, but
they are like different
libraries like suddenly they have this
library has this API.
This library has another one.
I want to use them in combination.
And then you think about how to
hype all the APIs
in the way that they
nicely flow.
So that's when one of my colleagues found
fp-ts and that sounded really great.
Obviously it had a huge
learning curve for us.
None of us were really much into
functional programming.
I only remembered some stuff from
university, but definitely
have it, haven't had like
hands-on experience and then learning fp-ts
like people who
learned it, they probably
know how hard it is because you don't
have like real documentation.
You have like API
interface documentation.
What gave you the confidence to take the
leap of faith to go
to reach for fp-ts and
make that the
foundation for your libraries?
Um, desperation.
I don't know.
I was just really, I think we all just
wanted to avoid another monolith.
And obviously we, we cared a lot about
these tiny libraries
that were meant to be
shared across the different services and
different services need to do different
things, right?
Like they're in a different domain. So it's really hard to
predict what API is the right one.
So as long as we can stick to something
and make it
composable, that should work for
most of the cases.
I mean, that was at least our idea.
The execution was rough.
We were iterating
on this group of
libraries and we followed,
versioning a strategy that would version each library.
At the same time.
So even if we make a change in one
library, we would bump
the version of all the
libraries just to make it easier for the
consumers to understand what version of
one library works with another.
So we opted for this and then the
iteration was so fast on these APIs.
We just kept in introducing breaking
changes all the time because we
discovered new and new ways
of using these libraries.
And so within the span of three years, I
think we went from
version zero to version
27. Our consumers were
not happy about this.
Let's just say this. Right.
So I suppose fp-ts at that point was
already quite the pill to swallow for
you all where you convinced yourself to
do it, but then to have other API
consumers that did not intentionally say
like, yes, we want to bring in fp-ts
into our life, but that's just the way
how you offered it to them.
That was probably even, even trickier.
So, but that's what you started out with.
And I suppose you, you chose that for
the, for very
understandable reasons, such
as the dream of composability and code
reuse and making all your systems more
resilient, but it came at a price of
like a rather esoteric flavor,
particularly for the ones who were not
getting up in the morning saying, yes,
I want to do functional programming, but
you just want to do proper programming,
uh, maybe in TypeScript.
So, and if I understand correctly, this
is where you have been kind of like
running into growing pains of fp-ts and
building larger systems out of this.
And at some point you found Effect.
So tell me about that.
Oh yeah.
So that, yes.
So basically with this fp-ts we
were trying to build
something similar to Effect.
We wanted to have a system which allowed
our consumers to
do proper dependency
injection with all the config that we
provided so that when they start their
program, like if the config is wrong, it
will just not deploy.
We want it to eliminate as
many foot guns as possible.
Obviously resource management was
another big thing that
we were looking into.
Like how do you properly handle like all
the interrupt signals coming from
Kubernetes when it's trying to kill
or restart your container.
These things.
They require a lot of effort and they're
not easy things to do, especially if you
want highly composable solution.
So I think one of my colleagues,
was my partner in crime when it came
to introducing fp-ts to the
organization at
large, he found Effect.
I don't know how he found it, but he
found it and it was maybe in 2021.
I don't remember.
I'm so bad with years, especially because
it was like the pandemic there.
And suddenly like two
years felt like one or half.
But around the time, it was weird.
Time flow flew, flew differently.
But he did found it and it wasn't
like we were jumping on it immediately
because at the time there wasn't really
such a huge buzz around it.
Like it is nowadays.
Also
didn't have a stable API.
So we were for about at least a year, we
were mostly observing Effect from the
sidelines, we didn't talk to also no
documentation at that point yet.
Yeah, exactly.
It was just another fp-ts for us
from the sidelines looking at it, but
it looked way better than fp-ts it
looked like, okay, this is not just
abstractions, this is
also some implementation.
Finally, something that
makes our lives easier.
It's like a general programming
framework, how to build like resilient
services and all the dependency injection
resource management,
configuration management,
building the layers of dependencies.
That's all taken care of for you.
So, so that looked really promising.
And as I mentioned, like for a year, we
were just looking and watching,
Effect and its development.
And then later when it felt like, okay,
now, now it seems like it's, it's
like stabilizing
around some sort of an API.
We still don't really agree with some
namings, of some concepts, which
then also changed name and everything.
And now it's like much, much better.
But we were, that that's, that's when
we started to seriously
consider adopting
it on some, some project.
We didn't really know
which project should it be.
We didn't really want to go right into
the migration of an existing fp-ts
project, because we
didn't know how to do it.
It was a little bit weird because
from fp-ts we had all these reader
task, eithers, and we, we could
intuitively see how it would map to an
Effect, but we didn't quite understand
how, how it would work with our flavor
of fp-ts if you know what I mean?
Like we built those abstractions for
managing dependencies and
resources and whatnot.
Like how would that, how would we make
the step from that to Effect?
So what we ended up doing in the end, my
team got this opportunity to rebuild
the existing customer satisfaction
feature and we are making
it like more flexible, like it's almost
on our end in the backend,
we treat these customers satisfaction
surveys as, as generic surveys,
because like, what is the customer
satisfaction survey?
It's like you ask a bunch of questions
and one question has to be how satisfied
you are about whatever you're measuring
against in our case, like satisfaction
about how your ticket was
handled by the support agent.
But in the end, it's just a survey.
It's a bunch of questions and different
types of questions and whatnot.
So it looked like a perfect opportunity
for us to try Effect.
Especially because there were, there
was already like Tim Smart's
SQL package, SQL effect, which is like
the predecessor of effect SQL.
So that, that was really good because our
service needs to,
interact with the database for sure.
So, so that, that gave me
a lot of confidence there.
obviously, and now
we're using Effect SQL.
I'm trying to like being lockstep
with all the development
with Effect itself, the core
library, and then also like all the
satellites that the level
one and layer one abstractions.
So it sounded like this was like the
initial prototype where you gave Effect
or like the initial trial project where
you went from observing the Effect
development to actually getting your
hands dirty and seeing, okay, is this
fulfilling all of our dreams, how
our world already got better through
fp-ts but we've introduced a couple of
like new problems and maybe some of our
teammates or some of the other teams, uh,
have a hard time with it.
And that could be addressed by Effect.
And so now you've adopted it.
How did that go?
How quickly were you up and running?
How quickly were you productive?
And, uh, also very
importantly, what did other teams think?
So before I actually get into that, let
me just tell you that for us getting rid
of fp-ts or moving towards something else
is, was existential.
Like there was no universe where we would
be able to push for fp-ts, at
least in our usage of fp-ts in a way we
did at a larger scale.
Like if we wanted this TypeScript service
framework to fly and be useful for not
just our two teams that used it, but also
outside of our org for all the TypeScript
native teams, who we acquired
through the business deals and stuff.
We really had to get rid of it.
We had to find like an alternative and
unfortunately there was like no real
alternative, beside Effect.
But I'll get into why this is
actually a pretty good alternative also
for teams that only
use native node JS TypeScript.
But yeah, so, after I
tried Effect for the first time, so I
think it was last year in June that I
made the first Effect commit ever at
Zendesk, getting up and running was
fairly easy because I could get a
skeleton of an application using our fp-ts
based service framework.
And then, yeah, I think one important
thing that I need to mention is that,
we're mostly building these services
on graph QL and then we have graphQL
Federation, so yeah, like lots of
microservices, each owning their own
domain and we call them sub graphs and
then they compose into a super graph through
the Apollo
Federation solution.
So this survey service
was just another sub graph.
So once I got to the point where I could
start implementing the resolvers for the
different graphQL objects and
mutations and queries and whatnot, um,
I was already, at that point I could
finally use Effect.
So I was using Effect in the beginning
only at the resolver level of my
application.
So, so that went pretty well.
Startup was very easy.
And then obviously like I created like an
application skeleton that I'm familiar
with, and then the resolvers were Effect.
And that nicely adapted into the
promise based resolver because with
an effect, you can run it as a promise.
So that was perfect.
And then after that, it was just
like, okay, let's, let's start with this.
Let's start with a simple Effect.
Let's start with a dummy resolver.
And then next step was
let's add some validation.
So we brought in Effect Schema and yeah,
now that I mentioned Effect Schema,
I think that's probably the
best gateway drug into Effect.
Like everybody needs input validation.
Everybody that it's almost impossible to
imagine a piece of software, like a,
you know, an HTTP service that doesn't
need input validation unimaginable.
Yeah, I definitely agree.
I mean, this is where, why Zod as a
standalone library is also getting so
popular, but I think people
will see that their
requirements will grow beyond just input
validation where you maybe need to run some
effectful
code, like some, some
code that can't go wrong,
et cetera, as part of this validation,
but then also critically, not just
parsing and validating the code, but at
some point you might also want to
re serialize it as you, you don't just
receive data, but you want to ship it
over the wire to another
service or to your front end.
And then you're scratching your head
because, okay, you have now this beautiful
sod system that can validate and parse
data, but now you want to go the other
way around and that is impossible.
And if you use a whatever, like you've
created a class, a user class that you
parse things into, and now you have your
data expressed through like those
classes or more complex data structures.
And you want to call
JSON.stringify on it.
And suddenly stuff blows up.
And this is where Effect Schema comes
in and I completely agree.
This is going to be my primary bet for
what will be the, the main gateway
drug that brings people to Effect.
And it's has gotten so nice.
I think this is where in the past where
you've been using fp-ts
you've probably used io-ts
Effect Schema is written by the same
person, but I feel like
It's a new incarnation, of the
same person who just made everything
so much nicer to use so
much easier to understand.
So yeah, huge fan of that part.
Absolutely.
Same.
I'm a huge fan of Giulio who does
all of this, this work.
It's just amazing.
-big shoutout to Giulio. -yeah, absolutely.
fp-ts and io-ts helped us so much in the
beginning and now seeing him also do all
of the work, for Schema at Effect and
doing a lot of Effect documentation
work as well, it's just, uh, yeah, it's,
it's amazing to see that.
So sounds like last year, last summer
you've written your first Effect code at
Zendesk, and I think you've been
by the end of that year, you've been
getting to a point where it could be
rolled out and we've been just chatting
before where you mentioned that this year
it's getting released in GA.
So, I suppose across like all
production traffic, et cetera.
So how's that gone?
And how did your team like it?
How, how was your experience?
Yes.
So my personal experience was, was great.
Let's start with me and
then we can go to the team.
And then we can go
like outside of the team.
Yeah, I was taking
really baby steps with effect.
So after input parsing and
validation the next big step was to
interact with the database, obviously
relying just on SQL effects was not
enough because we have
a sharded database.
We have like this big shared sharded
database instances that every
application interacts with.
So we had to write like a wrapper,
for SQL effects that would read
the configuration, create one SQL
pool per shard node.
So basically one for a writer, one for a
reader, and then we would have like this
big collection of writers and readers
group by shards, and then every time a
request comes in, we would have to figure
out what is the shard ID reaching to the
collection handle like short shard not
found errors and stuff like this.
Finally getting the shard, getting the
pool for the writer or the reader
that you currently need at the
point where you need to
like call the database.
And then that's when we get the SQL
effects, like the SQL client from the
library at the time, like the pool is
basically provided by that.
And then that was great.
once we got there, because then we
could persist events.
So one thing that I want to mention is
like, we use event sourcing where exactly
this two way bi-directional schemas,
work really well, because on
one hand, like when the request comes in,
we do the input parsing and then that
input represents almost the exact
payload that we then write to the
database which represents our event.
So once we did the parsing, obviously we
need to serialize it into, into a
string that we can then push into a,
into a text column in SQL.
So yeah, that was a big thing.
And then obviously the next step
after the database client, there was
like the HTTP layer where we might
need to call like different HTTP
services, especially around the feature
flags and stuff like that.
So yeah, really baby steps.
And then eventually we added some
level of observability,
logging, tracing metrics.
In the beginning.
It wasn't really anything sophisticated.
We were relying mostly on the data dog,
tracing library defaults, which
provided some level of detail.
It didn't work very well.
We didn't get very detailed traces,
but it was good enough to see some
traffic and what's going on at the, maybe
let's say the GraphQL level.
So everything that is an Effect,
obviously we couldn't get any visibility
into that, but then we switched to
Otel eventually, and then things
improved from there and then as the more
I learned about Effect, the closer
I was getting to turning the application
into an end to end Effect.
And today it's like fully Effect.
So it's effect from top to bottom.
Right.
And I think this is a pattern that I've
seen a lot both in the apps that I've
been building and where I take, I've
taken my first steps with Effect, but
also in what I see how other people are
approaching effect is like how
nicely it can be adopted incrementally.
Where if you think about your program as
a tree, you can very easily start
at like any layer of the tree, really
like at any position, you can start
out with the leaves and
write those as effects.
And then at some point, just run them as
promises and assuming your, your other
code is already like promises or
async await based, you can just call your
new effect code as good old promises.
Or you can do it on the other way around.
And at the root of your program, you
can rewrite some
parts that are maybe more
for governing and overseeing the
execution of your program.
You can rewrite that with Effect, for
example, request handlers, but then,
still call out to promises and everything
else of your program is still promises.
Or you can do it somewhere in the middle
and say, okay, at some point we're like
in async await, we're going to rewrite
some part in with effects, but then the
stuff way further down,
we'll call again as promises.
And so you can mix and
match as much as you want.
But what I've typically seen is that
people get very
productive very quickly
with like some incremental adoption.
And then they see
like, Oh, wait a second.
That part is still not as nice.
And that is some of the
leftover promise code.
And then in no time
that's rewritten to effect as well.
I think what is typically
the case is the more
you rewrite to effect
that kind of collapses the amount
of code typically like by half and
shines a very bright light on some of
the holes that you haven't yet
covered such as error handling very
often, like Effect kind of shows
you a mirror and say like asks you like,
Hey, here stuff could go wrong.
Right.
What have you done so far about it?
Not really much.
And so it forces you to do the right thing
That sounds very familiar.
What you've described.
Absolutely.
Writing Effect has
a lot of knock on Effect.
Pun not intended. On how you write
maintainable code on a day to day basis.
Things like obviously the error
handling it's amazing.
Very recently I implemented a Kafka
consumer with
effect, um, using Kafka JS.
So I just wrapped Kafka JS in an Effect,
API and then had like an
abstraction for a consumer and then
obviously the consumer can consume
things, um, either message
by message or in batches.
So I just needed a message by message,
type of consumption.
So that's what my abstraction does.
It creates a consumer and you pass in
like the effect that hand a function
that returns an effect that handles like
the incoming message, the payload.
It was just really beautiful to see
how the different error cases
all bubble up to the very root of
your message handler.
And then in Kafka, like when you consume
the messages, it's super important to
make a decision at that point.
Like, do you drop the message
or do you retry reprocess it?
Right.
You need to be able to make this decision
for every single error case.
And you know, you just have effect dot
catch tags and then you have like a big
dictionary, like a big, big object
containing the tag of
the error on the left side.
And the error handling basically die
or, or succeed with some error
logging or whatever.
So if you die, Kafka
will reprocess the message.
If you, if you succeed and you log it,
well, you can either put it into
like a dead letter queue or something.
If you want to keep
those messages around.
But if that's not necessarily
important because
it's like time sensitive
and if you don't process it now, then it
doesn't make sense to process it like
five days later, then you
just discard the message.
And before I would normally implement
Kafka consumers like in Ruby or maybe in
fp-ts, it was also, it was better, but I
don't think I had it to this level
of granularity because what we ended up
doing in fp-ts was taking like the
shortcut route where we would just return
a generic error interface.
Just to have like, yeah, you
know, it can error, but we didn't know
what kind of error later on as I got
better with fp-ts, then obviously you
would have different types of errors,
which would also compose.
but I haven't written a Kafka
consumer with that,
approach in fp-ts,
but with Effect, it was just like so easy
and it's so useful because you don't
think about all these different kinds of,
ways your program can fail.
And my consumer is very simple.
It receives a message.
It parses it.
writes an event and
then calls another service.
So basically three things, but there's
like 15 or 20 different ways it can fail.
And it's just like, wow, now I know
everything every single way, how it can
fail
so what took me the most
time was thinking about whether to
discard or reprocess the message in the
whole implementation
proportionally the most time,
but that makes a lot of sense.
And I think that makes a difference
between resilient and
non-resilient software.
I think a lot of TypeScript programmers
are blissfully ignorant of the
unhappy path of a program, but this is
what ultimately makes your users very
unhappy if stuff goes wrong.
and a lot of times that is completely
unrelated to whether
the user has done something wrong as
whether your servers have a bad day or
whether you've migrated some infra and
like some things is like hitting capacity
or you're just hitting a blip somewhere
and things go wrong and a user gets like
an endless spinner or gets like undefined
is not a function in their face.
This is where like some backend
engineers haven't done their homework.
And I get it.
It's really tricky.
If you, if you're just like dealing
with like catch error and then
you have an any or unknown
thing and what do you do with it?
Yeah, maybe you bubble it around, maybe
you log it, but to work with it as
nicely as like structured type data that
you return from a function, that's what
you get with Effect and
you can handle it so nicely.
That was actually fun to
deal with the non happy path.
Yes.
And I think you
touched upon it very well.
It's fun.
You don't feel like it's something that
you have to do, but the language or the
library like doesn't really provide a
good way for you to manage.
And then it's like just a headache.
It's really fun.
To handle these cases in Effect.
And it just has such a great effect on,
on your reliability, on the
reliability of your software.
The other thing that I noticed was also
in terms of
maintainability and testability,
like the dependency injection,
like dependency management as well.
It's just so great with effect.
I don't have to worry
about mocking things extensively.
I just build a fake dependency and
provide that instead of, instead
of the real one, the live one.
And I feel like I'm much more
incentivized to write unit
tests and I don't know how other people
think about engineers at companies like
Zendesk and like Facebook and Google,
like big, big companies, like how they
deal with these things
deal with these things
on a day-to-day basis.
In the end, like it doesn't matter
how skillful or experienced you are.
It comes down to incentives all the time.
Like you do things that you
are more incentivized to do.
So if a language or a framework or a
library makes
something really easy to do,
you will do it regardless, whether it's
the right thing to do
or the wrong thing to do.
It's just like this.
I don't know if there's like a law or
this, but it's
definitely a psychological effect.
This, this is one of my favorites
paradigms or
guidelines in programming
or in life, in principle,
which is like make the right thing easy.
And that is, I think Effect makes
some really hard things
that are the right thing
as easy as they can be.
And so easy that it's fun doing things
like error handling or structuring
the hierarchy of your program in a
nice way, in the as nice way as you could
possibly do it in TypeScript, that's a
huge lift and that's
what Effect enables you.
And I think sounds
like dependency injection.
That's probably if you're in your Effect
adoption journey, that typically happens
sort of like maybe in the second week
when you're using Effect
after you've like rewritten
a whole bunch of like
promise code, et cetera.
Maybe you've now like finally cleaned up
some of your tech depth
around error handling.
And then you realize, okay, there's still
like some
global state we mutate
to kind of pass things around, or we just
have like this big blob of
like properties we're
like kind of pulling
through like all of
our function vacations.
Or maybe we have this like monster
monstrous context object,
which maybe has a whole bunch
of like, either it's untyped or it has a
whole bunch of like
nullable properties and you
kinda pray that it's there.
That's like all like CME principles.
I'd argue this, the most principled
approach about that is
like having like a bag of
properties that you just like lift
through your function
calls, but Effect gives you
the best of both worlds.
It gives you a very principled approach
and a very convenient approach.
And I think dependency
injection has a kind of a bad rap.
I've used it in
various programming language.
I've used it like a lot
of in PHP in the past.
I've used it like in GoLang and
other programming languages.
It never, it felt like, okay, this is how
things should kind
of, there is a solution
somewhere there, but all the solutions
I've used so far where
it kind of like had such
big foot guns that at some point I said
like, okay, no, I've hurt myself too much
with that.
I'll do it manually.
And I think Effect finally gives you the
cake and lets you eat it.
And I think that's
really hard to explain.
And I think you have to build a little
thing, refactor it and
like, then throw in a
little bit of like type save context,
which is all there is to it.
Really.
It's like react context,
but type save and much nicer.
So that's, I think something you have to
try for yourself to
see that how nice it is.
But I agree.
This is one of the best things about it.
Yeah, absolutely.
And why does it have a bad rep
if I think
about it, I think it's
again, comes down to incentives.
If the language makes it hard to do
dependency injection,
because it's never just a
singular case, like, Oh, I'm going to do
dependency injection on this function.
And this one function will be, you know,
they're written the right way.
Well, you know, your program usually has
a certain depth, like there's like a
function and then that function calls out
that of other functions.
Those functions call out other functions.
And then the dependency needs to travel
all the way down because it's
called somewhere at the leaf.
For example, a database call is made
somewhere at the leaf of our program and
wiring through all these layers and then
adding the dependency in the list of
arguments all the time.
Well, I'm not surprised
that people get tired of it.
And I started thinking about it a few
weeks ago, you know, like, what are these
approaches?
And I started calling them just
internally for myself,
my own, for my own sake.
Like there's this explicit dependency
injection where you like to do all the
wiring and like yourself.
Then there's some languages like Scala,
which somehow give you like some, some
language features, which allow you to
implicitly wire through it.
You need to have a big brain for that.
And it feels a little bit magical, right?
This implicit dependency injection.
I don't know if it's
through traits or something.
I'm not a big Scala user,
but I did see some, some of it.
And then you have Effect, which is like,
it's somewhere in the middle.
Like it's, it's kind of implicit, but
it's also very explicit in a sense.
Like you, you do declare, you see where
you inject the dependency by providing
the implementation.
And then you also see the place where
you're calling the functions and the
stuff that's on the dependency because
you have to yield it or yield star.
So it's, it's kind of implicit because
you don't have to wire it manually.
You just use it at the, at the, at the
site where you need it.
I think it's the best of both worlds in a
very similar way.
How I think TypeScript is the best of
both worlds where it very elegantly
does type inference in
most places where you can.
a lot of static languages,
like ask you to write type annotations
everywhere, and that I think also causes
like some fatigue when
you use a typed language.
And I think TypeScript makes it so nice
that you can get away with
like just type annotations in
the minimum amount of places.
Sometimes even for argument types, if you
have a default value, for example.
So most things where possible can be
inferred and that's totally fine.
And so think about the context, the type
dependencies of an Effect of a
function, think about it the same way like
if it's
used, it can be inferred.
If you return something from a function
that looks like an object with a property
user, then the type can be inferred.
That's because you return it.
And what's so cool about Effect is
like, if you use a thing in a function
and using, like you said, like if you
yield something, so the equivalent
of like an await, then Effect and like a,
on the type level wires things up
nicely with TypeScript that in the type
signature, you say like, aha, here
we need the database client.
And also during runtime makes sure,
okay, there's behind the
scenes, the context objects where we have
the database client.
So it picks it up and not just even that,
but also when you finally get to run your
program, it makes sure that at some point
you supply your database client.
And I think that is so elegant when you,
when you use it, but it's, it's hard to
to kind of grasp it if you, if you
haven't take a look at that with
code and like try to refactor a little
something, but I agree.
It's one of the most
elegant things about Effect.
Absolutely.
People just have to
get their hands dirty.
There's no other way of learning and
understanding Effect.
Like obviously you could read the
documentation all day long, but then
you get fatigued because there's just so
much that Effect provides.
I often see people being very confused
about what is Effect?
Like, "I don't understand it.
it seems to do
everything", because it's such a big
departure from the tiny libraries with
very well-defined responsibility
in the JavaScript ecosystem.
And then you can like pick and choose and
you can build your own tech stack,
upon certain libraries, and then that's
your definition of production
grade software, but then
you have Effect, which seems
to be like the glue code.
It's a really a generic
programming framework
Right.
And I suppose in a parallel universe
Effect would have been a different
programming language, but I think
now we sort of have the best of both
worlds in that regard as well, because
TypeScript is darn good.
Like, and so many people
already love TypeScript.
I love it.
It has so much structure and has so much
amazing tooling around it.
VS code just works super well with it.
You have like LSPs that
work in other places.
So at this point, you need to have a very
good reason to create a new
program language, and I think good
reasons could be superior runtime
performance, what like Rust is giving you
or what other program languages give you.
But if you can't provide those unfair
advantages, then I think you
gotta stick with TypeScript for now.
And TypeScript is so elegant and
so flexible that you can bring all of
those semantics that you would get from
something like reason or a re-script.
But you can bring it
directly into TypeScript.
This is where Effect has
struck this really nice balance.
But I agree you need to rewire
your brain a little bit.
And people maybe
don't immediately get it.
And I've seen an interesting correlation
that people have the easiest time
getting what Effect is about if they've
experienced the problems that Effect
solves before and they
have sort of like a lot of scar tissue
from trying to solve those problems
themselves, like trying to do proper
error handling, trying to do
observability, trying to do interruption.
What you've mentioned
before with Kubernetes.
So the more problems an engineer has
experienced in the past, particularly
TypeScript engineer, I feel like
for them
Effect clicks
most quickly, but yeah, I'm curious, what
was the experience talking to other
engineers at Zendesk so far?
What have they been confused about?
What has clicked for them?
So far I mostly talked through
my own experience.
Then I had my immediate team members and,
with them, obviously it's a journey
because, they have to learn it.
It's also different from fp-ts.
Also fp-ts, they didn't
really bother learning that much.
Like as long as they could kind of
understand in terms of the code
review, what's going on, that was
already a good enough level for them,
To be productive and
help me with the reviews.
If I write some code, also my team in the
past one, two years, like we've had this
unfortunate situation where
we had some churn in the team.
So often I was like the only backend
engineer on the team while being
the tech lead as well.
So I really needed like my front end
engineers to be able to review my
code, and Effect is just doing
really well in this regard as well.
Because once you have the generator
syntax, where you have the gen and
yeld star, which you can easily map
in your mind to async and
await, you can build up this
adapter layer, in this mental
model for you once that clicks,
it's very easy for them to review code.
I'm not talking about stuff like,
you know, database queries and, you
know, how to set up proper indices for,
for your table and like these backend
concerns, purely backend concerns, but
like all the business logic that you
write on the backend and there's a ton of
it, that's not an issue in terms of
review.
So that's sort of like the 10 second
onboarding like, Hey, this stuff going
to look a little bit weird.
Just everywhere you see yield, think
that's await everywhere.
You see that gen thing, I think that's
async and you should be able to like,
just read that code as like your
traditional async, await code go.
I think that is sort of like the hail
Mary 10 second onboarding
where someone can get really far.
Yeah, exactly.
And that's like totally
Pareto, like, this 20% of effort
gives you 80% of the results.
Like after that, obviously they're going
to have questions
like, what is this layer?
What is this runtime?
Uh, what do you do when you catch tags?
What are tags?
Like there will be questions like this,
but they're, and yeah, they require maybe
more nuanced explanations, not just
like a one-to-one mapping from a
new concept to a well-known
well-established other concept.
But, but it's that
20% of the, of, of the productivity
that you're achieving
with the 80% of the effort.
So already with the 10 second onboarding,
you're so far ahead that the
reviews just work already.
And then I like this idea of like,
exposing someone
to Effect first through
like reading
code and doing code review.
Since this is where someone through
the context that they are already
familiar with, maybe through a refactor,
maybe through a new feature, they
have all of the context that they need to
understand what the problem is about.
And now they can focus
on the implementation.
And I think what's also so nice is
depending on where someone
reviews the code, possibly ideally in
their IDE, this is where you can also
use all of like the type inference
benefits to help you understand
what's going on, if you hover over an
effect and see like, Oh, this is where
we can have an error that is about maybe
the user wasn't found or maybe
another service is done.
This can add so much to the picture to
understand what's going on.
Where before everything was just like,
an implicit sort of wake
thought, and I feel like this is where
someone just by also being exposed
can pick up so much.
And then you have seen at the end of the
day, a lot of code users are
very similar to each other.
And this is where in someone get now
takes that step to writing their own
Effect code, they probably have already
seen two or three places that are very
similar, so you can go copy some of that
code, go over there, adjust it, and
bring the usual programming muscle.
And it's works going to work just as well
and probably even better
since you have improved type safety.
Yeah, absolutely.
Also, I really love the way you can work
with generators because anything
that's within the function body of a
generator, it's basically your happy
path because all the error cases just
short circuit the happy path.
And then you just do a quick pipe after
the Effect where you
handle all the
possible failure cases.
And I don't know why, but I just love
this style of writing programs.
Here's my happy path.
Everybody can understand what's going on.
And then now in this pipe, I'm going to
handle all the errors.
Right.
This way can like, sprinkle a little bit
of like extra sauce on top of it, where
you can, I often do
also like timeouts there.
I add a little bit of like Otel
instrumentation around that, or maybe do
like a retry for an error
but yeah, as you say,
like in the generator,
this, and I think this is so beautiful
about it is like, you can nicely separate
sort of like signal from the other stuff
and say like, okay, here's my business
logic and here's like,
here are those other concerns.
I think like in the future, if we have
like a next
generation of IDEs, et cetera,
and maybe like even more AI assisted,
maybe that can help you and say like,
Hey, hide everything that is not about
the business logic or
hide everything that,
or like highlight everything that is
about concurrency or highlight everything
that is about error handling with Effect.
You already put in sort of like the
structural effort and I think we're going
to see some, some big rewards even beyond
what we have right now.
That's very interesting.
I never thought about this, but, uh, it
makes it enough sense.
Yeah.
The, the tooling that you can build on,
on, on top of these like static
descriptions of a
program is just like limitless.
Yeah.
Interesting.
Yeah.
This is something I'm
very, very excited about.
And we've, we talked briefly before about
the launch of the Effect Playground.
I think it's super nice to have like an
environment where it can just play
around a little bit, get
familiar with something.
I use it on a daily basis to maybe
understand an API surface a bit better
and just play around with it, have fun.
And we also threw in support for some of
the effect dev tools in there,
notably also the trace viewer.
And this is where you can get real time
feedback for what does it mean for
my program to run this is where it may be
a certain thing took like a second
and then should just time out, et
cetera, like visually
see what's going on.
There's so many tooling possibilities
that are coming.
And that's gonna just kind of like the
gift that keeps on giving as like
you adopt Effect, and there's like so
many benefits that just fall out of that.
I think we're still at the beginning and
it's already very rewarding for at
least in my experience
and what I've seen so far.
So you shared your
experience using an adopting Effect
and also how you help your own team adopt
Effect and be productive with it
through code reviews and helping them to
refactor code and build new Effect
programs.
But given that you built this service
framework that is used, I think all
across Zendesk when it comes to
TypeScript code, there's now more and more
people that are exposed to effect.
So how was their experience?
Maybe you got a little
bit of like questions.
What is that thing?
Uh, maybe some similar concerns that
people asked about fp-ts
So which sort of questions did you hear.
Yes.
Uh, well, that's a great question.
So let me start with another
team, uh, not my team.
That's the closest to our team.
And they have some services that are
fully written in fp-ts
and using our service
framework, so they're looking and
watching us from the
sidelines, writing Effect
code line, because we're enjoying this
opportunity of building a completely
new service from scratch.
And, uh, they're, they weren't so lucky.
So they're still stuck
with their fp-ts project.
And they're just looking at us, uh,
and are maybe a little bit jealous,
of us that we're writing effect
already because they've been looking
forward to writing effect as well.
But, but I'm helping them try to
figure out how to migrate, fp-ts to
effect also incrementally
it's, it's a bit tough.
Especially if you have your own
abstractions and own ways of doing things
with fp-ts, so it's really slow.
And also it's really hard to justify, to
spend the time to fully migrate a
rather large project in one go.
So it really has to be incremental.
So that's, that's a
positive feedback from them.
But then we also have teams that are
outside of our immediate organization
and they are, let's say more TypeScript
native teams and they have completely
different requirements from, from my team
and from the other Guide team.
Because Effect was not their choice.
It was our choice for, for the
TypeScript service framework.
Right.
And the service framework does
provide a lot of value, but without
knowing Effect necessarily or fp-ts
even it's really hard to tap into
that value and, and use it immediately in
your project, which knows nothing
about fp-ts or Effect and the engineers
know nothing about fp-ts and Effect.
So here Effect actually brings some
really good tools, that can
help bridge between the two
requirements and
that's the adapter layers.
So basically when you have an Effect, you
can run it as a promise or you have
a promise and then you can wrap it into
an effect using different APIs.
So in our service framework, this is
something that we're going to be leaning
on more and more because we want to
provide the benefit to all the users
regardless whether
they choose effect or not.
So for every effect API, we can have a
rule that we will also be able to a
promise based API, which is fully built
on top of the effect, because we're just
going to satisfy all the dependencies at
the time and, and run it as a promise.
And then they can always look up what
kind of failure
modes there can be because they can just
follow by convention, or by
inspecting the implementation.
They can see which effect APIs,
rather which service framework effect
based service framework API is wrapped
and then discover the
type signature there.
So that's, that's one way how they can
reap the benefit of knowing
what kind of errors there are.
They don't have to
inspect all the depths.
I don't know how even people were doing
it with like regular
type script libraries.
You know, how do you discover what
kind of errors you may encounter?
Like, I think typically you don't and you
discovered during runtime and logging.
Yeah, exactly.
It will be so nice to know like, Oh,
here's the documentation page.
It lists all the 150 ways of my program
failing or my library failing.
But this, this doesn't exist.
I at least I have not seen a library
documenting their
implementation or
their API's to this level.
It would also be really terrible to
maintain the documentation for this.
I mean, we have the perfect primitive for
that, which are types and I
guess more modern programming languages,
such as Rust, et cetera, they have
figured this out and they return results
in case something can go wrong.
And I mean, Effect is about the same idea
that you don't just return
just the success value, but also you're
returning the, the errors just through
the return channel as, as other things as
well, but coming back to
the point you were just making, I liked
that approach that basically for
the folks in your organization who are
already excited and interested about
Effect, they can already start consuming
the effect API's for the ones who
are still on the fence or are not quite
ready to make the jump yet.
They can still stay in the Promise land
and need to deal with,
errors, et cetera, the, the good, bad
way, um, the old bad way.
Which sort of questions do you typically
hear when someone is as
confronted with Effect, I suppose
there's a full spectrum of people.
You immediately get it and are excited to
dig in to people who are maybe curious,
but don't quite get it.
And then maybe people who are much
more skeptical and maybe this
reminds them of like some other bad time
they had in the past and why they have
reasons in mind, why
they don't want to adopt it.
So tell me more about the different kinds
of reactions that you've seen.
Yes.
So I've done a few
Effect related presentations
at Zendesk already.
I presented at our annual tech
conference in May.
So I had the opportunity to actually
get some of those questions and lots
of people are actually skeptical.
Maybe due to their own
experience with something
similar that they tried or
just, it just looks too functional for
them and they're more familiar in the
OP and dynamically typed languages.
They don't necessarily understand it.
Like at Zendesk, we have lots of
engineers who have experienced all
the issues related to scale
maintainability,
testability, reliability,
all these things, but still this alone is
not necessarily
not a huge selling point for them necessarily
because they already
have their ways around it.
Like they have years of experience doing
Ruby years of experience doing whatever.
And they, they know their way around it.
Maybe they don't even care necessarily
about super reliability because
there's like feature flex.
So you can basically not break
everybody at the same time, but you just
break a tiny bit of, uh, of the
customers, which is understandable.
If you don't have any other option
because you're limiting the blast radius.
But it's also not something I'm
really a big fan of.
Like I really want to catch the errors
and possible failure cases even
before I commit my changes.
Like that's the ideal thing.
I don't even want to push it somewhere to
a CI and then waste CPU cycles,
of a CI just to make it fail.
Um, and then repeat rinse and repeat many
times possibly, because also sometimes
it's really hard to run all the things
that the CI runs locally due to different
limitations, but yeah, so we have these
people who know their way around.
So for them, maybe a bigger concern is
usually, okay, but
listen, we have so many
different technologies at Zendesk, we have
to consolidate like, why should this be
the thing that we consolidate on?
How will you align with
all these hundreds of engineers
on the single one technology?
Obviously we have some processes like
ADRs and whatnot, but if it comes
to a big change like this, obviously
there's going to be also resistance
because people just are accustomed to
to the status quo.
And they're their way of operating and
they don't necessarily want to switch.
Which is totally reasonable.
And I don't want to change anybody's
mind or I don't want to force
anybody to now, you know, forget about
promise land and
start incorporating
Effect into your code
base starting tomorrow.
I truly believe it should be like a
choice that everybody
can make for themselves.
But then you have, you know, the company
incentives to try and consolidate to,
to not go into too many directions
because if you want to be productive at,
at the top level, you know, at an
organizational level, like the more
people pull in the same direction, the
better you are and the more productive
you are.
So these are also a little bit
political, uh, you know, influence
and political, it's a
question of politics as well.
Like how can you influence
without being in a big leadership
position and stuff like that.
Have you found some arguments like from
the many things that Effects can offer
that has resonated still with the people
who are more in the skeptical spectrum?
Yes.
Um, so sometimes I do, because let's say
you're a company and then now you
had this financial crisis where the
interest rates went up and now, uh, you
figure out that money doesn't grow on the
trees and you have layoffs and whatnot.
And suddenly you stop backfilling for
positions, you know, which, which
came up because of churn
and you're not used to pairing.
So you have hiring
freeze or whatever suddenly.
And, but the expectations of productivity
are staying the same.
So basically you have
more workload per person.
That's, that's the final result.
The company still expects you to deliver,
but now you are fewer people to do so.
It's, it's a hypothetical one.
Right.
So what do you do?
Well, I think TypeScript is positioned
really well because
you can have TypeScript
both on the front end, which in many
cases you do, and then
you can have it on the
back end as well, which
is, isn't a terrible option.
Especially like, you know, you have NodeJS
or Dino or whatever, nodeJS with
its event loop runtime, perfectly suited
for i/o heavy operations.
And as far as I'm aware, like 90% of what
Zendesk does is i/o heavy.
We take a message, no matter where it
comes from, we do some processing and
we send a message somewhere else.
Like maybe it's like you, you, you start
a record in the database or you emit a
thing in Kafka, maybe you have like a
MySQL connector with Kafka, so you can
do these things
in a single transaction.
So basically you manage your distributed
transactions this way.
So you do a ton of like taking things
from here, pushing there.
A lot of waiting time, a
lot of i/o, what do you do?
Like with Ruby, obviously.
And this is another thing
that I often mention is cost.
Like if you have nodeJS handling
traffic, like large concurrent
traffic, heavy traffic, you can save a
lot of cost because with Ruby.
What's your option there?
Well, yes, you can use threads, but then
your memory consumption goes up because
threats are not for free.
Or you can scale horizontally.
So when you put the horizontal pod
autoscaler max it out at 24 replicas
or whatever you figure out the rules
around how to increase the
replica count by how much, when, what is
the signal that you're looking at?
You know, you can figure out all of these
things, or you can
just have a few replicas,
maybe one per availability zone, 2
per availability zone for extra
redundancy
of a node, uh, process.
And then suddenly you
have a throughput of what?
Tens of thousands of requests per second.
So, so it's also money, you know?
So when you, when you talk to high, uh,
to the leadership, you have to
convince them with some
hard hitting facts.
And it's not just, obviously you can say,
ah, in theory, this works.
No, you have to sit down, do the
analysis, maybe set up
some project which can
demonstrate how much more cost
efficient it is compared to other similar
workloads, put it into
money, uh, values, right?
Uh, convert it into dollars or whatever,
and then show the difference.
And then once you do this, you know, you,
you, you won the golden ticket or
something because it comes down to
money in the end always.
Yeah, totally.
And I agree with that approach that
you can basically like, let the
actions, actions speak louder than words.
And you're doing
the right work already.
You're are shipping things in production.
You're appreciating and leveraging all
the benefits that
Effect provides to you.
And I think the, your team and some other
peer teams have a great
experience already with Effect.
And I think those will show the results
and that might make a case for itself
and prove out that it's not just words,
but it's actually an improved
reality that makes teams more effective,
more efficient, happier, and
possibly also then saves resources and
money when it comes to running those
services,
saves down times, et cetera.
So I'm sure the, the more time continues,
the more all of those arguments
going to resolve
themselves in your favor.
And I applaud you for being
on the early adopter train.
Thank you.
I, I do hope that it
plays out really well.
I'll do my part for sure.
Perfect.
So maybe that leads me to the last
question that I'd love to hear your
thoughts on, which is what gets you most
excited about the future with Effect.
Yes.
Ah, that's a good question.
I haven't put a lot of research into
Effect Cluster, but it's definitely
something I'm observing again from the
sidelines and look forward to using
in the future, maybe for some
use cases like backfills, let's say.
I have my event sourced service and now I
evolve my, uh, event schema from version
one to version two,
maybe two version three.
And now I feel like, okay, my switch
statements where we, where I, where I
switch between the versions of the schema
and then, uh, the way I'm reducing the
events into a single aggregate, it's
getting a bit cumbersome.
So let's just migrate some of those old
schema versions to the latest one.
So having like millions, maybe billions
of records, it could take quite
some time to do this sequentially.
So having like a solution where I can
have set up some workers, which can agree
on, you know, the scheduling and how
they're going to partition the database
table among each other, uh, and do it in
parallel, that, that would be just the.
You know, perfect dream come true.
I don't want to start the backfill in
every cluster one after another, or even
in parallel, and then like having to like
watch tens or dozens of, you
know, monitors to see the progress of
each individual
backfill on every Kubernetes
cluster and then managing that for hours.
You know, if that could be like maybe a
10 or 20 minute thing, that would be
just the perfect dream, right?
So I'm looking forward to Cluster.
Yeah, me too.
This is, uh, one of the, as I said, like,
uh, be the, the gift that keeps on giving
and we're going to have like many layers
built on top of the foundations that
we already benefit from and the Effect
Cluster and Effect workflow primitives
that are in the work and the
systems that are in the work.
Uh, I think that's gonna, yeah, that
that's going to be literally next level.
This is going to unlock some benefits
that you see from systems like Temporal
Temporal IO, not the new time standard,
but temporal IO, which is about durable
workflows and, and workflow scheduling
and running long lived things.
You can already do that in Effect
combined with temporal, but Effect
is the perfect foundation to do that, uh,
natively with the effect primitives.
When you think about rerunning something,
if something has failed scheduling, some
work, um, processing work across multiple
workers and massively
parallelized systems.
This is where we have like amazing
foundations for that.
And that's being systematized with the
effect cluster and effect workflows
project, which is now
in development for, I think also like in
research for a few years now.
And I think it's alpha grade right now.
I think some people are
already starting to use it.
I'm actually also planning to give it a
shot soon for the music app that I'm
building, and I think it will take a
little bit of more time to be fully
production ready, just because it's also
a very ambitious project, but it's very
principled and I'm very
excited about the potential for it.
And I think we're going to hear a lot
more about that in the months and years
to come and possibly the next year's
Effect Conference already.
So yeah, super excited that
you're excited about that.
Because I think you have some really
interesting use cases for that.
So, Attila, thank you so much for
taking the time today to doing the
initial episode of the Cause & Effect
Podcast with me and taking the time.
So that's much appreciated.
And thank you so much.
Thank you.
It's been a great honor to be the first
guest of this amazing podcast.
Perfect.
Thank you.
Take care.
Take care.
Thank you for listening to
the "Cause & Effect" podcast.
If you've enjoyed this episode, please
subscribe, leave a review,
and share it with your friends.
If you haven't done so already, you can
join our Discord community.
And if you have any questions, feedback,
or suggestions about this episode or
about Effect in general,
don't hesitate to get in touch.
See you in the next episode.