Phil: Hello everyone, and welcome to another episode of APIs.
You won't hate the podcast, it's the audio version Instead of blog
posts, we're here talking about APIs.
And today I've got Joel Claremont with me, who is well known around
the Laravel PHP community but might be new to some of you folks.
So Joel, Hello.
what's going on?
Joel Clermont: Not too much.
Yeah, I've listened to some of your episodes here and I'm
like, I am definitely not in the API dev tooling space.
That is not my specialty.
, I'm coming at this more as the, the, in the trenches practitioner
building, shipping, supporting APIs and not, not like the higher
level tooling person that's
building things for other people to
Phil: Well that's, that's fine.
So we're not, we, we have, because I have been working in the kind of open API
dev tooling space, a lot of the content ends up being quite specific to that.
And that's not really what this
podcast is meant to be like.
If you cast your
mind back to way back when of the origins of this, of APOs, you
weren't hate as a an anything before it was a community in a podcast.
I was writing that book.
Build APIs you won't hate, which was pretty much how to make rest Dish APIs
in Laravel and all of it was Laravel.
So
it's lovely to be chatting to Laravel people and I still build APIs in
Laravel and I think it's important that we don't, like, I don't want to
kind of separate out that there's the, the API community and then like all of
the, the programmers who are actually building APIs in various frameworks.
They're all, it's all overlap overlapping
circles.
And, whenever I'm talking to people
about like.
One of the main things in, in API derail, I always try and explain to folks is that
like if you go to API conferences and you only ever talk about APIs and you
sell your products to other people who care about APIs, it's a very small world.
But if you go to Laravel, PHB meetups and and things like that, then there's actual
people who might end up using your stuff who have never heard about any of the
wonderful, lofty things you've been talking about and really should have.
So that's kind of why I'm talking to you today of, yeah, you're
posting about like how, how you like to make API specs with Laravel
or how you don't, so you wrote an article called I have It right here.
Why I Don't Like Generating API Specs, which is written nail on the head.
do you wanna tell me a little bit more about this?
And you said there's a series of blog posts and like what led to this?
Tell me more.
Joel Clermont: Yeah, it, it actually started like, I think the, so, so, first
of all, for a little context, I, I have a site called mastering laravel.io and
I publish, for some reason, I committed to a daily newsletter Monday through
Friday, and I, I cross post to the site.
But so any, any random thought that pops into my head, I'm like, oh,
that could make an email, right?
So I'll, I'll, I'll write it down.
And I was working on a project recently building an API for a Laravel company.
And we were writing the spec and, and the client, like they, they're
an existing product and they're like, we've never written a spec.
Like we only have an internal team.
I.
And so that kind of started the discussion, which is
like, I think it's useful.
In fact, I know it's useful to have a spec even if you don't have a giant
team, even if you don't have like a separate frontend and backend team.
I think like even if it's a full stack small team or a single dev,
it, it is nice to have a spec.
It, it sort of keeps you honest.
And so that, that was the, the start of the whole discussion in, in the
newsletter, but then it ultimately led to, okay, hopefully I've convinced you.
Now how do I get started?
And where most developers like to go is, well, let's use code to do it right?
Like, that's what we do.
And I personally have never had success with that approach outside
of like demos and a proof of concept or maybe like breaking
ground on something to get started.
But the code first approach.
Has been problematic for me.
And so that, that was, that was where my experience was coming from and I
was trying to share that, like, it's not that complicated to like just write
the file yourself and, and I think there's actually some benefits to it.
Phil: I mean, so many developers are running about, like, everyone has
to learn Kubernetes and everyone has to learn these 25 different things.
Comically.
Most of those do end up being slightly overloaded.
yaml DSLs.
But the idea of learning this one other yaml DSL is completely
unacceptable to anyone and we'll never do it.
And, That's pretty funny.
So just in case, there are people who haven't heard me
banging on about open API in the past.
Joel says spec, he means API specification, which is one other
term for like API description or API design document.
There's a million different names of this stuff, but yeah, basically it's
like.
I'm gonna, I'm gonna describe the endpoints and properties and parameters
and request bodies and all the bits of the, the interface of the API with
some sort of machine language, which.
tools can use generally for docs.
Pretty handy for mocks and loads of other stuff.
And I've built a spectral API scanner.
So literally you put your open API in and it goes, oh, that's
gonna get you hacked buddy.
Watch out for that.
There's a million things you can do with, with an API spec when you've got it.
But yeah, when, when left are the developers.
I'm picking up on what you said a minute ago.
Like the, if you leave it to the developers they're just gonna
write a whole bunch of code.
And if you, if they ask you, if you ask them for documentation,
they'll go and write down like 10 lines somewhere and it'll be bad.
So at some point over the last I.
I don't know, eight five.
Over the last five years specifically, I feel like open API has picked
up in popularity massively.
And that's partly 'cause people like me bang on about it all the time.
But it's bloody useful for everyone to know what's going on.
And you've got that machine learnable format, so you, and, and you've got this,
this spec where you can generate some docs and you can use it for contract
testing in a million things to, and that helps keep those docs up to date.
But some folks don't like
to write.
That YAML out by hand.
'cause that is pretty boring.
There's a bunch of other ways of doing it.
There was gooeys and things, but people don't like to write that by hand and
instead they try a bunch of creative ways to avoid having to write that.
You were talking about in
Laravel, there's a few kind of Laravel specific tools.
How, how do, how do people do it in Laravel?
How do they get, how do they get their code spitting out open API
so they don't have to write it.
Joel Clermont: Yeah, it's what you might expect if, if, even if you're not in
Laravel, you know, where, where there's some, some element of static analysis
and then that can be enhanced with some sort of annotations or like inline
documentation to, to enhance it further or to specify like examples of things.
Phil: Mm.
Joel Clermont: and I probably should say like I've.
At least not super recently.
I've never actually tried to use one of those packages in earnest.
Like, I will try, somebody will recommend one, and I'll be
like, oh, I'll give it a shot.
And I usually get a couple days into it.
I'm like no, this is, it is just, it ends up, you're just conforming to a
different layer of abstraction in the sa in the same sort of specification.
And so, it's, it's really a personal thing.
Like I'm sure there are teams out there that do this and it works great and
I'm sure there are use cases where it, it's beneficial, but I'm generally not
working on massive, massive projects.
You know, like, like maybe there's 50 to a hundred API endpoints and a lot of
times there might not even be a public consumer.
Is it
okay.
I mean, it's not stripe.
Phil: right now, but yeah.
Fair enough.
Joel Clermont: Okay, well I'm counting like get versus post too,
right?
So, you know, in terms of
resources it's probably a lower number, but, but yeah, it's, it's
not, I don't work on teams of like 50 developers and you know, that
like, so there's probably some efficiencies that become more.
Important and beneficial where the trade offs might skew the other
direction toward code first tooling.
IJI just am not there and
I here, maybe this is a question for you, Phil.
Are there any tools that you've seen that do spec generation,
that actually leverage the spec?
Like things like refs and, you know, things like that Or is does
it end up spitting out some really flat, verbose document that no
human would ever write in that way?
Phil: Yeah, that's the problem.
Most of those tools will just kind of just vomit something upwards, which is not
particularly.
Helpful.
I mean, it generally, it, that sort of stuff doesn't really
matter if you are just putting it straight into a documentation tool.
'cause the documentation tools will flatten it.
so yeah, the, the way, the way that it seems to me is.
Somebody has said, Hey, we've, we've started using open API to
have one kind of documentation hub, a developer portal, whatever.
We've got this, we've got this thing, and we need all the teams to make open
API and we don't care how you do it.
I've been that guy.
I've, I've done that.
And I'm like, you know, some of them are like, oh, we've got it all in Postman,
so we'll convert it into open api.
I, I'm like, sure, whatever.
And some of them are like, oh, we've got open API two or Raml or these
different old kind of defunct formats.
And I'm like, whatever, I can
convert that for you and, and or, you know, and then eventually you end up
with open API and everyone's like, great.
Some of it's weird and some of it's ugly and some of it's not entirely right.
But that's a step one is we now have one documentation hub for the entire thing.
Great.
But then over
time it's a case of like, well, how do we get this to be more useful?
And.
And how do we use this for more verticals as well?
Like if you are just shoving it into docs, then it kind of makes
that nice stripe, like three coal API stuff that everyone wants.
But
if you try and use it for, you know, code generation and a bunch of other
use cases, it, it kind of falls apart and you can't it gets a bit hard
to like add stuff to it as well.
So.
the one thing I've seen, which helps, if, so, if you've got a API spec that's being
kind of just a magic appeared out of your API one thing, that can be pretty handy.
Have you heard of open API Overlays?
Joel Clermont: Mm.
No.
Phil: It's this like extension thing.
So open APIs started doing like working groups or special interest groups.
And these special interest groups create different things.
So they're doing like workflows so you can document entire kind of workflows
of, of API calls and things like that.
And one of them is overlays where you can basically.
It's kind of like a spectral rule set.
If you're familiar, you kind of have like a J path and you're like,
go to this bit of the document and then like add this or remove this.
And so you can theoretically have some horrendously vomited out, open API and
then be like, and I'm gonna add some descriptions in here, which are like
10 paragraphs long and have a bunch of markdown so you can have your technical
writers kind of wedging that stuff in.
And so.
My, my initial like, no one should ever do code first ever, because then you have
this useless thing is, is somewhat kind of abated a bit where it's like, okay, you,
you've, you've given us this useless thing and now some technical writers can jump
through some quite horrendous hoops to make it a bit less useless, but that still
isn't like the most shining endorsement for it.
Joel Clermont: you know, like you mentioned, if if, if the only people
reading the spec are doing so via like a generated HTML portal, then
it, it doesn't probably matter internally if it's using like refs for
shared schemas and things like that.
But
I, you know, just like I'm thinking and recommending you write it by hand at
least once, at least know how to do it.
I also.
I think like, I like to look at the spec by hand too.
Like
the, the tool, the tooling is great to auto generate fancy, nice branded
documentation, but I, I do find some value looking at the spec directly and
then especially seeing, oh, this, you know, this specific type of request
or this, you know, form payload is
used more than one place.
Like that tells me something about the overall API design that you don't
see if it's just all flattened out.
Phil: Mm.
Yeah, I mean, that's a useful thing.
So kind of having intent.
Written somewhere else outside of the code, not only helps you spot
obvious mistakes from a wider pool of people that don't necessarily
know how Laravel works but
it, it just becomes a, it becomes a lot more useful.
Like you can have a bunch of people doing design reviews that
literally don't know Laravel and don't know its random conventions.
And, and it also means that if
you accidentally have a slip of the keyboard and, and change some code.
You didn't necessarily change all of the annotations that came with it then.
I, the graphic I used on a blog post talking about this was like a, a
drinks cabinet with a labeled that as milk and it's clearly orange juice.
Like people kind of confuse the like proximity with
accuracy if like these annotations are somewhat near the code I'm describing.
So I'm sure someone will definitely remember to update them both.
And so.
Kind of having it in that second place means you can then kind of you,
you have those extra checks where you, you use that open API as an
assertion in your contract test that says, does my code match what I think
it does and what the design review team think it does, and what everyone
involved so far has thought it would.
And so that, that for me is kind of why I'm a big fan of the approach.
But it's, it's just really hard to get a PA developers to do another thing.
They're like, I'm a coder.
I write
code.
I would like to write some code for
this.
And yeah, it's, it's a weird one.
Joel Clermont: One of the benefits I think there is in writing the
spec separate from the code is like.
Especially if you start with the spec is it forces you to think
through the requirements in a level of detail that you don't often do.
You know, if you're getting a requirement from some stakeholder
and they're like, Hey, I want an endpoint to do this, this, and that.
Okay, well, like who can do that?
You know?
How long can that field be?
Like there's all these constraints that.
Somebody has to make a decision on, and like that spec becomes the
forcing function to decide like, well, what is the maximum length?
What, what is the format on this date time?
Or, you know,
just things like that that otherwise you might get to later
in the middle of implementation.
And a dev just like, well, I guess we'll just do it this way.
And it wasn't actually discussed or decided on.
Phil: That's a really good point.
That's I think where I was heading was that like the API spec being thought of as
a documentation mechanism is kind of the problem here because it's it's basically.
It's not always a problem.
There are, there are use cases where you have an API and you
need to like play catch up, right?
There's the there's, there's API design first.
There's API code first, and the, and I've done polls where it's
like, are using API code first.
API design first.
Switching from one to the other or some awkward combination.
And every time I do this, every couple of years, the answer's
always like 25% of each.
So people, there are some teams who are doing code
first, some are doing design first, and, and it's just all over the place.
But yes, like those, those tools that can help you kind of grab a whole
bunch of metadata out of your code through static analysis and maybe
you add a few other things in there.
Can really help you get a head start.
And then you have the open
API that you can go back and put assertions into your code base
to make sure that it's correct.
And then you can expand that.
And if you want to add a new endpoint, then you can design first
on that, on that open API file.
You can add a new endpoint that's completely imaginary, start working on it.
Have everyone agree whether that's a good idea or not.
Whether it solves, solves the needs or not.
Run it through a mock generator, a mock server so that people can
interact with it before it even exists.
And then later on.
Code it and make sure that it matches the contract test for that.
But do you think, do you think that's how everyone should work?
Joel Clermont: I like your use case because I, I do join a lot
of, I'll call 'em legacy projects where you don't have documentation.
In fact,
I hate to say this, sometimes you don't have tests, right?
Like,
and so that I could see as a useful time to generate something just,
just to get a starting point.
And then edit it from there, and then like, you know, write contract tests
around the little bits as you go.
And, and so I, I gotta remember that because a lot of times when
I join a project, like decisions have already been made and now
you're just trying to catch up.
And I, I think code generation could be an excellent use case for that.
Just to, to get you to where you wanna go faster.
And to not have to write all of that by hand and, and reverse engineer it.
You know, like try to
figure out, well what are all the overlapping validation rules
or schema constraints in the database layer or things like that.
A static analysis can get you 80% there.
Phil: for sure.
I think.
An interesting third way I've come across, and we've written about
it a bit on the blog, is kind of using HTP traffic inspection, right?
So I've, I've definitely inherited APIs where no one
knows how the heck this works.
You've gotta like decompile the iPhone app
and there's some third party client that you can't even look at the
code of and, and like no one can get into the compiled Java app on
some bloody API that people haven't got the password for anymore.
It sounds silly, but I've been in these silly situations that help out a lot of
struggling
Joel Clermont: boy.
Phil: And everyone's like, right, we've gotta rebuild this API.
And you're like, what API if I can't find any of the endpoints, like,
everyone, check your browser history.
What are we doing here?
And so being able to
kind of wrap it in a, in a traffic proxy that essentially
says like all the requests.
Going in, or you can literally, you can wedge them into the browser.
And, and then kind of run around clicking things.
There's, there's a bunch of these different tools for sniffing actual
traffic going through the system.
And then
you can kind of take all of that and output an open API
file and say, there you go.
And I feel like if we're, if we're talking kind of the catching up
with design first, then I'm almost, I'm mu I'm, I generally feel more
interested in going with the traffic sniffing approach because it's real.
I.
Like you can still look at some, look at some stuff and, and get it wrong.
Like if, if when you've got these complicated dynamic payloads where
there's this or this or this, that could come through and you're like,
it's definitely only this 'cause you've only seen it come, you know, you haven't
noticed any null values in there.
So you said it, it is a required property.
Mistakes like that can be made depending on how much
the sample size goes through.
But I, I feel like the promise of having these kind of framework
integrated static analysis.
Tools is potentially misleading unless they can hook into enough
where you literally don't need to add other stuff yourself.
'cause as soon as you ask the humans to get involved, that's when they're
just introducing human error.
Have you come across anything like that?
Joel Clermont: Yeah.
Um, Not directly.
'cause again, I've not worked on a project
recently that has done the automated tooling, but, but I've,
I've experienced drift for sure.
And, and honestly that's one of my fears with Code Generation, I, you alluded
to it earlier, is like if you make an unintentional change, well, and it
just regenerates the spec to match it.
Now your tests are still passing, but I.
Unless a human catches that in code review or design review of some sort,
like you could unintentionally change your API even in a breaking way,
which to me is like the worst possible outcome of of, of, of an API change.
So I don't know if there's solutions to that, but yeah, the, either
the intentional or unintentional drift is just, just an ongoing
problem I have experienced,
With that traffic inspection like that's super useful for like sort of
those black box scenarios you were
describing.
Like, I'm gonna have nightmares about that.
But I, I'm almost wondering too if like that in conjunction, like when
you're, when you're, when I'm joining a project, like I'm coming into some
project, if I, if I just use static analysis, that doesn't tell me.
How much dead code is in there,
right?
And like now, maybe I'm wasting a week of work trying to document and test something
and then I, ship it to the client.
They're like, oh, we haven't used that in five years.
Like that.
Well, that was just a wasted week of effort.
So I could see pairing the static analysis with the traffic analysis
just to like, even to prioritize.
These are the paths that get the most traffic.
Like
let's focus on those.
These endpoints, they never got traffic.
Oh, I'll ask the client even before I start working on it.
Oh yeah.
We can just delete that.
Right?
Like, so I could see a hybrid approach there being useful too.
Phil: Yeah.
Interesting.
The I've seen a few frameworks that are.
me love the idea of code first a little bit more.
I wrote
an article recently and I can't remember the names of any of these things.
I think it's like Hu Hugo Oh, Hummer in, in Node and Fast.
API I.
I'll get some problem links in the show notes when I remember.
But there, there are, there's this kind of new idea of like
open API Aware frameworks, right?
Like built.
Into the core of the framework is in the past, in the past you've got
like swagger, PHP, which I hated.
Like I, I, when I, when I first got involved with swagger, when I first
tried to start trying to figure out how to document an API, I looked
at swagger, PHP, and went, ah, for all the reasons we've discussed is
literally you put a bunch of doc block
annotations in your code and hope that it's correct.
And, and I ran away and then I ended up doing other things.
But that's still like the main premise.
But with modern tools, like like Scribe is a really cool tool and like props
to the people that have made this.
But they, I.
They've essentially got rid of quite a few of the annotations because the framework
is kind of saying that this is a poster or, or a get, and like they've, they know
what the URL is from looking at the code.
They're doing a lot of that static, static analysis,
but it's still kind of hacked into the framework a bit.
And there's still a certain amount of duplication.
And so what I like about the frameworks that are.
Coming at this from like the framework itself will tell you what
your open API is, because just by using the framework, we are building
up all of the bits that you need.
That seems really cool.
Like when, when you write the validation rules for your
models, they all line up with an appropriate validation in open API.
So when you're doing that, it sounds that that, that to me seems like it
would be pretty much just fine because.
The, the main, the main thing of like a PA design first is it's more about planning.
You should like plan this thing and spend a lot more time in the planning
stage past the whiteboard, but before you start writing loads of code.
But if the framework can
make itself simple enough that it, the framework is, is.
Just like a config DSL where you are, instead of writing open API in yaml,
you're writing open API in this kind of a bit of PHB code and it's easy enough to
read and it makes sense, and then you just shove a bit of business logic in there
later that, that to me would be fine.
That kind of absolves the differences, removes the differences between open API.
Well between design first and code first.
It kind of makes them become the same thing.
You just kind of like add your business logic some other time.
But none of the tools in Laravel that I've seen are that, and it's
quite rare to actually find them.
And at this point I worry that those projects are kind of just like hobby
projects and not a whole thing.
So
there's a bit of hope in there, but I don't think anyone's taken the
idea too seriously at this point.
Joel Clermont: Yeah.
The, the reason this, this came on my radar recently is there is a
newer package, like it's not even at 1.0 in Laravel called Scramble.
I
don't, I don't know if you've seen this one.
It will look at, like in, in LAR you have a form request with validation rules.
It will statically analyze those rules and.
Set like max links or field types or things like that,
purely based on that with like zero annotation, which is, I,
I do like the promise of it.
I really do.
Like I, I get the appeal.
Um, one, one other thing too I was thinking about is I.
You know, when you mention whiteboards and upfront design, like I cringed a
little bit because like I, I've come over the years to recognize like you
never can plan super far in advance.
And, But I, you know, so one thing I wanted to clarify is like, even
with the design first approach, I generally do it like literally
end point by end point, right?
So I, I design it, I write the spec, we agree on it, I implement it.
Then we repeat for the next one, not like, well, let's design the
entire API and, and see what happens,
because that, that just never materializes correctly.
Phil: man.
Yeah, it's, it's something that I really, I need to do a whole episode on this,
on like, how do you actually plan an API, because the stuff I said in my book
was, was pretty immature at the time.
I think I was like, just think of all the endpoints you need and
then like post and get next to 'em or whatever is, is a bit dumb.
But, but like the phases, the phases, of of planning and that's like
literally stakeholder interviews, which is like a, a, a concept from
user research and, and, and ux.
and then kind of taking everyone's needs and requirements, and not
necessarily too literally, but distilling down the essence of what.
They want and what they need and, and how multiple different stakeholders
can ask for different things.
But finding the middle ground that will be just fine for everyone, like
doing all this really complicated work.
And then another group of people or, or you, with a different hat on, can
then start to say, well then these are the end points that I need.
But even then, like, is the, is jumping straight into
endpoint endpoints premature.
Like, do, do you need to think a little bit more about, about kind of the
workflows and the, and the type of API that could be used and like, should
this even be a, a endpoint based API or should, should it be event driven?
There's, there's so many stages to the planning and it's hard to figure out.
Where you should start and open API and design the spec first.
Stuff like it doesn't help you with any of any of that.
Like you, literally, your planning starts with Slack calls and whiteboards
and like Miro boards and sequence
sequence flow diagrams and like chatting to a load of people and
having a spreadsheet of feedback and all this crazy stuff that no one,
no one that I know is really working on, apart from the tools that I've
mentioned, but they're like generic.
And then once someone has says.
Someone has said, you definitely need to make a HT PAPI and it
should probably do these things.
That's when it's time to get the open API out.
'cause you're literally turning the words they've just said right there of like,
make an API that does this, this and this.
Then you kind of are going, well okay, we'll have that end point, that end
point and that then end point and.
Where design first gets really handy there is you can say, okay, I'm gonna turn
those words into, like, I'm gonna reshape them into a little bit of yaml and I'm
gonna like, the A PO will look like this.
Then it's much easier to change that little bit of YAML than it is to go and
change a whole pile of Laravel code.
You're like, oh God, now I've got rename all of my controllers and that means
I've gotta rename all my models and all the relationships are gonna be broken.
And I, you know,
that's, that's really daft.
And so
pretending that.
Code first is quicker 'cause you just add some annotations to some code that you
wrote is predicated on the fact that you wrote exactly the right code first time.
And when does that happen?
Joel Clermont: It doesn't.
Right, exactly.
At least not for anything.
Non-trivial.
Phil: Yeah, well, alright.
Pretty decent agreement on that.
Screw.
Screw that.
Joel Clermont: Awesome.
Phil: That was
Joel Clermont: See, I I we're far too reasonable, Phil, in our, in
our older age and maturity we're, we're mellowing and we're, we're just
seeing things much more reasonably.
Phil: Absolutely.
Absolutely.
I haven't had a fight about framework interoperability in a long time.
Well, fantastic.
Let's, let's call that where can people find more of you?
You are, you're all over the internet doing all sorts of things.
What, what what are you up to?
Joel Clermont: Yeah, my, my main hub of activity now is mastering laravel.io.
You know, on there we have a podcast for Laravel and PP
devs, the, the mailing list,
some books and courses if you wanna, like, keep me fed, like,
you know, all, all that good stuff.
But that's, that's really the central point for all of it.
Phil: Nice.
I will have to I need to get on that newsletter because I am using
Laravel more and more and they've gone, they've gone on like a rampage
of major versions and I'm like, I, I remember thinking four and five are
quite nice and now version eleven's out.
I'm like, I don't know what's happening.
Brilliant.
Joel Clermont: Yeah, it does tend to move quickly.
Phil: Cool.
Alright, well thank you very much and yeah, pleasure having you on the show.
Joel Clermont: Yep.
Talk to you later.