Leo Dion (host): Hello and welcome
to another episode at EmpowerApps.
I'm your host, Leo Dion.
Today I am joined by Sven Schmidt and
Dave Verwer of Swift Package Index.
Thank you guys for coming on.
You've both been on separately.
This is the first time
I've had you on together.
is awesome.
Thank you for doing this.
Dave Verwer (guest):
Thanks for having us, Leo.
Hey, Leo.
Yeah, thanks for having us.
Leo Dion (host): Before we begin,
I'll let you introduce yourself for
those who don't know what the Swift
Package Index is and what you do.
Dave Verwer (guest): Sure.
So my name's Dave Verwer with Sven
I run the Swift Package Index.
also run a newsletter called iOS Dev
Weekly various other sites have directory.
I always have jobs, basically
everything starting with
iOSdev something to do with me,
.
Sven A. Schmidt (guest): Yeah.
Hey, I'm, I'm Sven, I'm an
indie developer podcaster.
Even most recently we've started out
together, Dave and I I've been working on
the Swift Package Index this past almost
three years now, together with Dave.
Mm-hmm.
. Awesome.
two other apps that I have, but the
Swift Package Index is, is really
the main thing that I'm doing.
Leo Dion (host): So you,
do, you do a podcast?
Is this this related to the Twitter
spaces or is this something different?
.
Dave Verwer (guest): That's
certainly how it started.
Yeah.
So we.
decided few months ago now to, to do
little Twitter spaces, to to basically
just chat through a little bit of
what's been going on with the package
index and then also to recommend
some packages that we've seen fast
on the, on the feeds basically.
So whenever packages get updated,
We can subscribe to an rss.
In fact, anyone can subscribe
to an RSS feed of those updates.
We check over those updates every
week and every couple of weeks.
We did a Twitter space to talk about
those packages and also talk about what
was happening with the package index.
went down reasonably well, and we decided
to kind of upgrade that a little bit
into an actual podcast because we did
get some requests for people to Yeah.
Who wanted to listen to
it in a podcast player.
So yeah, the last three I think,
or three or so episodes have
been proper podcast episodes.
Okay.
we, we experimented with live streaming
to YouTube and then we experimented
with stopping live streaming to YouTube.
now it's just a regular.
Podcast.
Yeah, I'm not, we still do have
a YouTube version, but, but
the podcast is the main way.
Leo Dion (host): Yeah.
Yeah.
I'm not, I'm not Yeah.
Risky enough to, to willing to do live
streams as well with the, with this stuff.
So yeah.
I commend you for doing that experiment.
I gonna say?
Yeah.
Well, I'm glad you didn't
go the other route and go to
Clubhouse, I guess, so if that's
Dave Verwer (guest): still a thing, right?
Yes.
Is that still a thing even now
Leo Dion (host): so Dave, you
spoke recently at the wonderful
Server-Side Swift conference that
Tim Condon had put that seemed like
that conference was a huge success.
Apple revealing that they're open
sourcing or making available the
foundation in the library, I believe.
But I want to kind of just do a little
brief redux on your talk that you did
on how you accidentally ended up running
the largest open source Vapor site.
I love that title.
And
Dave Verwer (guest): so, I mean, one
of the jokes that I made in the talk
was if you put enough conditions on
anything, you can win any competition.
That's right.
That's right.
. Yeah.
. Open source Vapor . So, yes, it
was really, it was really little
bit of history about how the
Swift Baggage Index came to be.
it was previous to being
a Swift package index.
It was actually a little rails
application that was a closed
source rails application initially.
then through a conversation
that, and I had over email.
that the Swift Package Index.
And, and it was an open source project
from day one because we knew that if, if
it stood any chance of becoming really
significantly becoming the place to find
Swift packages, it had to be source.
And if it was gonna be open
source for the Swift community,
it had to be written in Swift.
So was, it was very clear immediately
that we, that when we were talking
about in fact, , we were talking
about redoing it in Swift because of
various other conversations that had
gone But it was very clear that it,
that it, it should be a Vapor project.
Leo Dion (host): Yeah.
Yeah.
And of the things that I think you
mentioned was the fact that like, Vapor
was, was, what was it at the time, really?
not sure where Kitura had stood, but
Perfect, at least had been discontinued.
So it just made an obvious choice.
Dave Verwer (guest): I think as well
Spen had some experience with Vapor
already, which was huge bonus because
I had actually , I had actually
started, I'd made a new Vapor project,
but it was also in that period
between Vapor three and Vapor Four.
I'd made a Vapor three project
because there was better
examples and documentation.
But then when Sven kind of came on board
with some Vapor experience, everything,
everything changed from that point.
So maybe, maybe Sven Can can talk
about it from his perspective.
Sven A. Schmidt (guest): Yeah.
I had been running a small.
Vapor 3 project before in, in production.
Like it was doing a real thing
and I had good experience with
But also there was, it was good
documentation, which I had worked with.
And the Discord community, the
Vapor Discord community was there.
I knew about it.
I, I was in it.
I There'd be enough people to
answer questions and, and I
made heavy use of that early
on, and that was really helpful.
at the time there weren't
really that many other options.
You mentioned couture perfect.
Which was sort of, you know, if not
already discontinued on the way out.
Smoke I don't think existed yet.
Amazon's smoke and, and Hummingbird by
Adam Fowler didn't exist yet either.
So Vapor.
Pretty obviously obvious thing
to pick if it's swift spec'd.
I had previously run two Python services
in Flask and also another one in Django.
So I had some experience
running that kind of service.
I think that was also similar, but I have
to say in hindsight, I would definitely
pick Vapor and Swift over these, over
these backend frameworks because.
of issues that I faced with the Vapor,
with the Python based services we
don't actually have in production.
Like I remember fighting with Python
and the service just dying, running
out of memory, that sort of stuff.
We don't see any of that
with with the Vapor Service.
It's rock solid in production.
runtime issues just don't exist because
they are compiled home issues in Swift.
works just really, really well.
I think some smaller.
issues early on we had mostly around
foundation not being exactly a
hundred percent the same between
macOS, where we develop and
Linux where we, where we deploy.
Yeah.
But once you sort of know what,
these are mostly around formats
and stuff, you can work around it.
Plus UCI catches that.
I mean, you're never going
to accidentally ship that.
Right, right.
In Python, that might be a problem.
You know, you, you don't hit
that code path, you ship it and
then in production you hit it.
But the compiler makes sure you can't
actually ship that sort of stuff in Swift.
So I, I think it, it works really well.
The, the, the other downside perhaps
is that the package ecosystem doesn't
have the same depth as it does on
other platforms that are older.
You know, like Ruby, Ruby
on Rails, Python again.
guess also go.
, I don't think breadth is
necessarily what's lacking in Swift.
I, you might argue that there aren't
many variants of different services,
you know, to, to choose from.
But of all the stuff that we've been
doing, there's always been a package,
and that package has, has pretty
much always been really high quality.
You didn't have like five to choose
from, you know, three of which
might be questionable and, and not
maintain, but you had often had that
one and, and that was good enough.
mean, our service is probably
not the most edge, you know, the,
you know, on the bleeding edge.
We're not, we're not doing
authentication at the moment beyond
some, some basic basic types.
We're not doing image video
uploading, transcoding, that sort of
stuff, but I know others are, and,
and it, it works perfectly fine.
Yeah.
Talk to the
Leo Dion (host): folks at Amazon.
.
Dave Verwer (guest): Exactly.
Yeah.
I think there's, and Tim,
yeah, I think there's two
sides to this question as well.
first is certainly Vapor and running a
web application on Swift is a very stable
environment the, to, to in production.
And, and we've seen that over, over
years now of, of running the site.
But I think the other
side of it is we also.
Build a lot of stuff into
making sure that's the case.
So extensive test suite monitoring,
good logging and a lot of those
tools really help that reliability.
It wouldn't be unreliable without
those, but it's more than just
picking the right framework.
It's, it's how you approach
development of your entire service.
I think that that makes a stable service.
Leo Dion (host): Yeah.
And one of the things I remember spend
when I had you on was talking about
some of those libraries that you use,
like prometheus and snapshot testing
make sure that the site is running and
doing, you know, is staying stable.
Yeah.
.
Sven A. Schmidt (guest): Yeah.
And, and one of the reasons why I'm so
confident that Swift is better, you know,
from my experience with, as opposed to
the Python frameworks, is because we
did exactly the same things in Python.
I mean, I'm, I'm a test
driven nurse, okay?
I, I do that in all projects because
otherwise I can't, I feel I can't
deploy unless I've run a test suite.
So I have these in place and, and yet
still I see a lot of issues not appearing
in Swift, that that did appear in Python.
. So yeah, that's, that's my 2
Leo Dion (host): cents on that.
Yeah.
And I think this is the gear.
If you're, I, this is probably
the second year I've said this,
but like, yeah, I mean, there's no
reason not to go with a server side
backend and swift at this point.
So
Sven A. Schmidt (guest): is it
the year of Swift on the server?
Well, that
Leo Dion (host): was, that was the episode
I did last year and I can't, I can't
do that again, unfortunately with Tim.
So,
Dave Verwer (guest): This last year has
been all about documentation and not
the documentation of the Swift package
index, but so a little while ago I
can't remember exactly when it was,
but it was probably almost a year ago.
We shipped the first version of.
DocC documentation generation
support on the package index.
So just a quick recap of what that is.
If you have a Swift package that is
already in the package index and if your
Swift files have documentation comments
that DocC would understand and would
produce documentation from, and of course
you can test that locally with Xcode.
Swift Package manager.
Locally you can indicate to the Swift
package index that we should build
and host your package documentation.
and we do a couple of things, which
think in my, my opinion, takes
us above and beyond, beyond even
what, what people would potentially
set up with their systems.
So the first thing that we do is all
you really need to tell us is what
target or targets that you would like
documentation to be generated for.
. And then as we do our compatibility
builds because behind the package
index is a full, almost a kind of
full CI system for every package
where we verify compatibility with
platforms swift versions by building
each package hundreds of times.
the end of that process, we tacked on the
documentation generation step, so, Package
authors have let us know that there are
three documented targets in a package.
As we build that package, we'll
also generate documentation, upload
it, and host it automatically.
Like literally, you don't need
to do anything apart from,
from it's awesome targets.
Yeah.
And then secondly we keep every version.
Documentation.
So as your package is moved forward
through releases and especially major
releases we, we update the documentation
every time you release a package.
And then also if you upgrade a major
version, version two to version
three, for example, we'll keep
the old major version, the latest
old major versions documentation
around and put it in a little
dropdown on your documentation site.
So when you make breaking API
changes, Generally get indicated
by a major version change.
We'll keep that old
version around for you.
And again, all of that just
happens completely automatically.
You just release your
versions and that's it.
It's done.
So that's a, a very quick
recap of what the feature does.
of course, To provide that level
of simplicity for the end user
and also for the package author.
There's an enormous amount of work
maybe can talk a little bit about
of the stuff that we've, we've
worked on with that this year.
Right.
Yeah.
Sven A. Schmidt (guest): So the,
the DocC hosting, as Dave mentioned
is, is a tack onto the build system.
And, and the tricky bit there is, you
know, to pick which targets actually
the docs are built for the uploading
to S3 for that, and especially
the proxying around because we,
we are adding a, a custom header.
The, the thing that Dave just described,
the bar on the, on the top of it,
where you have a chooser of which
target you're viewing docs for, and.
Which version you're looking at.
normal DocC site doesn't have any of
that, so we have to fiddle a bit with the
structure of the webpage to inject that.
just to mer make everything work
with us hosting that site you know,
and, and doing the redirecting and,
and proxying of all those URLs.
Well, I was just gonna
Leo Dion (host): say the DocC building
is fantastic on Swift Package Index.
Set up stuff with Netlify
on hosting my documentation.
But like the fact that you do
automatically, and like Dave said,
has all these quirks with like major
version, I didn't even think of that.
Like yeah, if you update to a new major
version, you have breaking changes.
Like people might wanna
pull up the old docs.
Just yesterday, speaking of like put.
I updated an old repo that I had and
renamed it, and like I was able to
get this DocC documentation working
in minutes, and it's fantastic.
That's great to have.
Yeah.
I want to jump a little bit
and talk about you add a
package, two Swift package index.
It's simple.
You know, Dave, you're a big
fan of doing pull requests.
that's, that's it.
That's all you have to do.
You put it in a pull request,
it's still almost working.
It's awesome.
And like you have to do.
But then there's some custom quirks that.
You know, swift package Index, like you
said, DocC You know, we've talked about,
I think watch kit testing, which used to
be a thing, but now it's native anyways.
But there's a few things that you
can do in swift Package Index to
customize using this like YAML file
you've set up, you want to explain
what that, what purpose that serves.
Sure.
And how it works with
Swift Package Index.
Dave Verwer (guest): So, from the very
beginning we had w w we knew that at
some point we would need ways for package
authors to tell us bits of metadata
that we might need to know that were
was outside of the Package.swift file..
Because obviously we get a lot
of, we had as much as we can from.
the package manifest, and also services
like GitHub, where packages are hosted.
Mm-hmm.
. we knew proposing changes to the
package manifest is a long and
slow process as it should be.
You know, we shouldn't be, we shouldn't
be changing that file every, every , but
that we'd need a way for package authors
to be able to tell us various bits of
metadata that was not covered by that.
Originally I think the idea for
the Swift YAML file came from us
wanting to add author support,
which we recently just added.
So I'm actually just gonna
go off a slight tangent.
Yeah, go for it.
Talk about this.
We Swift package Index participated in the
Swift Mentorship program again this year.
So we had two mentees come on
board and implement features.
We was mentoring one, and I
mentored the other, and my mentee
basically he implemented author
support for the swift package index.
So passing out git history, finding
the most people with the most commits
deciding who was like a significant
author based on an algorithm and.
putting that information
live on, on the site.
But then of course with anything
automated, it's gonna get it wrong
sometimes or get history as maybe
not as accurate as, as people
or it do maybe doesn't represent
who actually wrote package.
And so we wanted a way for people
to be able to override what
the automated system had done.
And again, we used the Swift package index
YAML file to allow package authors to say,
actually your author information that
you've kind of derived isn't quite right.
Here's the string you should put to
say who wrote this so even though that
feature has only just recently been
implemented we've had the idea for that
feature a very long time ago, and so
we knew that we'd need this package
Swift package index YAML file to do that.
But then the first time it actually got
used was when the compatibility build.
being developed.
So we I think it was the very first
thing we needed to do was custom targets.
Is that
Sven A. Schmidt (guest): Schemes.
Schemes, schemes.
Yeah.
Yeah.
It was schemes.
I, I remember, well, we were
still, we had an open discussion.
We had an issue in our GitHub
talking about what the format
should be of that file.
And you can imagine what
that discussion was like.
, you know, oh, don't use YAML, use JSON.
Well, JSON is terrible.
You can't have comments use
TOML and you know, , while the
discussion was still brewing.
We really just needed some way of,
of authors to tell us scheme to use
because you mentioned it earlier.
There was one issue that you couldn't
build on watch os if there was a,
I think if there was a test, right?
Because of, yeah,
Leo Dion (host): you had needed, you
had the example of the point free I
think it was composable architecture.
Sven A. Schmidt (guest): Yeah, exactly.
Right.
And the way we, in the build system
we had, we sort of had to, we, what
we're doing is we're guessing which
scheme is the right one to build.
There are often many,
and we just pick one.
Certain heuristics, they sometimes
fail and it's less and less common,
but especially early on, people
needed a way to tell us just use
this scheme for this platform.
So effectively we say, all right, YAML,
it is . Everyone's gonna be unhappy,
but this one is, this is what we're
going to choose just to get this fixed.
And, and that's how this file was
born and has lived ever since.
Pretty much.
Leo Dion (host): So
Dave Verwer (guest): yeah, go ahead Dave.
And it's done us really well.
terms We've expanded the amount
of metadata that package authors
can to that file over time.
And that's also where package
authors us which targets our
documents and that kind of thing.
There's two main advantages of doing this
file in this way is we could have made
a system where package authors needed to
authenticate on packaging next.com, log
in, and maybe they get a web view where
they can say, oh, it's these targets,
and it's, you know, this is the correct
author information and that kind of thing.
But because everything comes
from a Git repository, that's the
source of truth for everything
to, to have some of the metadata.
Somewhere else, maybe behind a specific
site, never really felt right to me.
And to have that as a YAML
file in the package repository
feels like the right solution.
It also means that any other tools
can also look at that file because
it's just a file in a git repository.
think as far as I'm aware, we
are the only, we are the app
that does look at that file.
certainly it felt right
to have that metadata.
In the package repository.
course there are downsides to it
because it's much more difficult
to tell people that they should go
looking at the package documentation.
And actually, just to go off
on another tangent, go for it.
The other mentorship project Sven
with his was to document that file.
So maybe you can talk about
a bit about that Sven..
Sven A. Schmidt (guest): Yeah.
I mean, this has been, this, this
popped up quite a few times that.
You know, knew there was this file,
but they didn't know what the format
was, what the, what the actual
things are that you can put in it.
And we had a blog post that introduced
it and a couple bits and bobs here
and there, but no central place
to actually host more document.
I mean, the obvious central place
is obviously the package index, but
we never actually put up a a page on
it that explained that file better.
And then we kind of thought, well, , we
had just built a doc system and we
had pushed all of the, the specs
that, like the SPI YAML file is
actually Codable struct that is
behind it is in a separate package
that we have in the, in our GitHub.
So it's Swift package index
slash SPI manifest, and the
package is actually in the index.
And then the obvious thing was to just add
documentation, oxy documentation, have it
generated through the index, and thenpoint
towards that documentation in the index.
And that's what Azza worked towards
actually
Sven A. Schmidt (guest): adding docs
to that package such that we can host
it and then direct people towards that
file and have Nice, it has examples now.
We chose, you know,
what things you can do.
Yeah.
How you add to, I actually,
I mean, that's the thing.
I had snippets lying around to
tell people what to do, and now
finally I can , I can look it up
Leo Dion (host): myself.
Did you.
Did you really deep dive into
DocC and like do tutorials and
articles and things like that?
Or did you just stick stay with like,
document, like just swift comments?
Sven A. Schmidt (guest): Yeah,
it's, it's, this is just a, it has
a intro just marked down pages.
Yeah.
Okay.
I mean, it's obviously the, the structure
is documented to a certain degree, but
the main documentation for this kind
of thing has to be a markdown file
describing or what the purpose is and,
and you know, I think the, the most
interesting page there is common use
cases where we describe, you know, what
do I do if I want to override the scheme?
Okay, what do I do if I
want to override the target?
How to add documentation?
That sort of thing.
Yeah, yeah, yeah.
Because those are the
main questions people ask.
Leo Dion (host): One of the big
improvements it sounds like you wanna do
this year is to searching on the site.
do you wanna explain how it works now
and what you're hoping to do in 2023?
Dave Verwer (guest):
Well, I think actually.
, I think what you're potentially
referring to is a change that we've
already made , and again, this
was a this was a contribution the
open source project from Joe Heck.
So Joe noticed some problems when
he was searching for think he,
was he searching for Ping or ftp?
One of the two.
Yeah, both of them.
Yeah.
Those were those.
. Those are the two.
Yeah, those are the two problems.
Both of those have very ambiguous not
ambiguous, that's the wrong word, but
both of those phrases occur in lots of
other words in ways that you wouldn't Yes.
And it was putting potentially terrible
search results to the top of the list.
so he originally reported it.
as an issue.
And we talked about it a little bit,
but he was also interested in in,
doing some work to, to solve it.
And he did some amazing work with looking
at some of the Postgres search commands
I forget the name of technology,
maybe span remembers what other
things that they've broken up into.
Stemming the
Leo Dion (host): heuristics.
Okay.
Dave Verwer (guest): Yeah.
Stemming and stemming.
There was something else as well.
I can't remember what it was.
Yeah.
And, so that landed few months ago And
what it does in, in real terms is make.
Vastly improved search results where
if you type ftp it's more, much
more likely now to show you things
that are about FTP instead of that
being part of some of the word.
Okay.
Sven A. Schmidt (guest):
That's really cool.
Yeah.
Just maybe to explain by, by
example what the problem was.
So imagine you search for, for
Ping, you get all sorts of hits
from mapping, you know, ing being
ping, being a pretty common ending.
Ftp, likewise Swift package.
ftp, swift FTP package.
Yeah.
Right.
You know, when that's spelled
together, you get hits like that.
They have nothing to do with what
you were actually looking for.
So what you can do is have run an
index does word stemming so it, it
breaks up words and, and discards
endings that aren't actually, you
know, part parts of the world really.
Right.
that then gives you good search results
if you, if you ranked by those, it was a
bit more complicated than that because we
had to weave it into our existing search
and our, you know, we want to preserve
a lot of aspects of our current search.
Joe did a lot of work.
I mean, Joe actually wrote a SwiftUI
app that took searches that he'd.
, which he had captured.
So he captured the ranking, had a
Swift UI app to actually look at
those results and to compare them.
He went all in on that.
That's awesome because it was amazing.
Leo Dion (host): Is this, and this all
works within Vapor and Fluent or like,
was there some customization in there?
Sven A. Schmidt (guest): So
what saves us here is that
you can drop down to pure sql.
Okay.
I mean, I, I'm sure you.
potentially write this in, not in
fluent, but in, there's, there are
other layers in Vapor below flu.
Right, right, SQLKit, for instance,
where you can, you know, write your
SQL functions and then assemble them.
But in this case, in search, we are
really assemble what we're using.
We are using SQLKit, but a lot of SQL
RAW as well that we then assemble.
that's how we build up
the, the search string.
And it's effectively going through
a materialized view to keep it fast.
Yeah, that makes total sense.
But this sort of stuff you
can't, you can't do with
Leo Dion (host): fluent.
Yeah.
Yeah.
You can get into the raw sequel,
which sounds like what you guys
did and it makes sense with search.
ahead Dave.
Dave Verwer (guest): And the nice thing
about being able to talk about this on, on
a podcast is that that kind of work that
Joe did on that is really thankless, like.
He did an enormous amount of work to
get that pull request in and, and, and,
and to improve the search on the site.
And to anybody using the site, it still
just produces search results, right?
You only that, that's the kind of
change where only really notice when
it goes wrong and it, the fact that
nobody noticed is, is a great testament
to how good that changed from Joe was.
Yeah.
That's a, that's
Leo Dion (host): awesome.
. So one of the other things you've
talked about quite a bit is the
whole we, we just got this whole
infrastructure, I believe it was June
of this year with Swift package plugins.
What, what are you doing with Swift
package plugins and Swift Package
Index and how, how, how do those two,
what's their relationship, I guess?
Sven A. Schmidt (guest): we've passed
this package manifests the whole time.
And, and plugins are a new
type of product that appeared
when they, when they came live.
But initially we were blind
to them because they weren't
a product that we knew.
So we had to had to change that to
actually ingest But I think that
happened pretty early on because if
I recall correctly, not passing them
led to, you know, the packages with
plugins actually failing to index.
So that was an obvious thing to do.
But of the things that, that was
also a contributing change . and
was support for plugin search.
So you may not know we have
certain extensions on search.
You can obviously search for terms, but
you can also apply little qualifiers.
So for instance, platform colon
iOS, which, you know, lists packages
that are compatible with iOS.
But we also have a.
, product colon and then
executable product colon library.
And then product colon Mine, plugin.
Added a support for product colon plugin.
And then you can search for, for
plugins, you know, you can just search
for product colon plugin that lists
you all the packages with plugin.
The package index knows about,
and you can obviously add other
search terms in addition to that,
to to filter that down further.
that was really cool because that
allowed us early on to, to see, you
know, to expose what plugins they
actually are and see them, you know,
the lists grow over the months as
they, as they started appearing.
Leo Dion (host): What are some quirks
that you found with plugins that
you didn't have to run into besides
just the new product type I guess
?
Sven A. Schmidt (guest): I don't
think there were any problems really.
Well actually, let me qualify that
one issue that is, is there, now that
I think of it that is there, and I
don't think there's a, an obvious
fix is compatibility of plugins.
Okay.
Because what do you actually
We currently build a package
to determine compatibility.
I'm not actually sure what happens
if you run Swift Build on a.
Plugin package.
Okay.
Will it just, will it
actually build anything?
Would you actually need to use it
to, to determine compatibility?
I'm, I'm a bit hazy what would actually
happen, but I do recall that we had
packages being flagged and people said,
well, is this should be compatible,
but sort of isn't doing the thing.
But I think they had an example in, in
their repository that we then built.
But the plugin was
actually attached to that.
So I'm not sure what the state there is.
I don't think it's a currently
not a common enough problem
to be, to be an issue.
because plugins are sort of a
dependency when you use them.
Mm-hmm.
, right?
But then you're gonna also not because
they, they do something in your
project, but they're mostly not part
of your built product in the end.
Right?
There is no code that lands in your,
in your package from the plugin
unless the plugin transforms code
and that that's then gets added.
So I think they're a bit weird
in terms of are they, are they
dependencies in that, in that sense?
And how do you test for
compatibility in that sense, I guess?
Yeah,
Dave Verwer (guest): and I think the other
thing with compatibility around plugins is
it's never going to be easy or potentially
never even gonna be possible to represent
the full spectrum of compatibility,
of every part of every package.
Is this Pluggin compatible
with this platform?
Like even just representing that
data would be extremely challenging.
And then for people to read that
data and get what they need from, it
also needs to be relatively simple.
Right.
And I, I would, ima I think in, in my
head at least what we have right now
with that grid is at the very top end.
of how complex we want
compatibility to look.
I would not want it to be any
more complex than it currently is.
In fact, I would like to simplify
that a little bit if we can over time.
Right, right.
And that makes sense.
And so, yes, we may not capture some
of the intricacies of, well, this
package is compatible with iOS, but this
plug-in needs macOS, or you know what?
Whatever there are, there are
always gonna be things there that
we might not be able to To people
who are looking for a package.
so there's a limit to how far
we can really go down that path,
or at least how far we can go
without making things to companies,
Leo Dion (host): right?
Yeah, exactly.
So I wanna talk a little bit about
the future of the Swift Package Index.
In our last episode one of the
things we were talking about
is registry is becoming more
and more wide known and used.
And it sounds like you're doing something
with that for Swift package Index in 2023,
Dave Verwer (guest): Yeah.
So think 2023 will be the year
that people start to hear a little
bit more about package registries.
registries have been, or the idea
for a package registry in Swift has
been around for a few years now,
and in fact, in 5.7 the client,
the Swift Package manager client.
Gained full support for package
registries, but it's probably worth
talking a little bit about what
the difference is between a package
registry and a package index.
Before we go too further, but too much
further, because that's certainly going
to be something that people may not
have been aware of yet because package
registries have been, you would've
heard of them if you were like super
in the weeds on package manager.
but most people are just
using packages, right?
Yeah.
And they just put a git URL in and
off it goes, and everything works.
So a package registry and there
can be multiple package registries.
The Swift package manager supports
multiple package registries.
So GitHub could run a package registry.
Apple could run a package registry.
Amazon could run a package registry.
BrightDigit could run their
own internal package registry.
You're in my mind.
Yeah.
I was just thinking about better.
I was like, yeah, I should try that out.
. The, the, the, the, the concept of a
registry is, is there's no centralized
place by definition for a registry.
And the registry takes care of
all the hard stuff, basically.
So it takes.
Delivering package data it takes
care verifying that the contents of
package data is, is signed and secure
and, and verifiably not tampered with
since the package author of certified
it, which is a problem that is
currently the current implementation
package manager has that problem
where basically if you tag a version
That's the version that the package
manager takes as a specific version.
But with git tags, they're mutable, right?
So you could move a tag around and,
and potentially one person who's
using your package at this version has
different code to another person that's
Leo Dion (host): using the packages.
As somebody who's developed Swift
packages, I've run, I've done, I've tagged
something and then been like, oh crap.
Like forgot to do this and this, and then
I have to go in and I have to delete my.
Package cash, which is horrible.
If you've ever done that,
don't ever do that unless you
absolutely need to on your machine.
yeah, I've totally, I, I totally know
where you're coming from on that.
Dave Verwer (guest): So,
package registries are gonna
solve a lot of these problems.
But they, it, it'll mean that you're
no longer putting git your URLs
into your package manifest because
instead of getting it from a Git
repository, you'll be getting the
package from a package registry.
so there's some changes, you know,
I mean, these are these changes even
though they're supported by the current
version of package manager, they're
probably a little way down the road
because what Apple announced recently
was that they are going to build
a implementation of the package, a
package registry server that will be
compatible with the package manager.
So apple are open source
developing this package.
The only
Leo Dion (host): catch is they get 15%.
You know, I don't know if you,
you wanna use that, registry?
Sorry, I had to make that joke.
15%.
I had to make that
Dave Verwer (guest): joke.
15% of all the, of all the revenue goes
to open sources and actually got brought
Leo Dion (host): somebody, one
of the, one of the owners is just
like, oh yeah, that's a good idea.
Without any knowledge of, you
know, that's open sources.
Dave Verwer (guest): It's a little bit
different to the app store where where it
is a, a, a platform based on transactions.
It open source software is, is a
little different that .So what, how,
where does that leave the package?
. good news is we are going to fully
support package registries as soon as they
become a thing that's out in the wild.
So we will.
Obviously you have to, and the
Swift package manager will also
continue to support, get your URLs.
I I can imagine it's going to support
get your URLs probably forever.
Yeah, hopefully.
Leo Dion (host): I, just sort of segue
question, but like, one of the things I've
found with a lot of other registries out
there, well not registries, but other.
Languages, especially npm and know
Cocoapods had run into this too, is like,
anytime you have one central registry,
there's always like room for trouble.
And what I've liked about Swift is that
it relies, and you probably like this
too, but like it relies on get repo URLs,
whereas's so it sounds to me like what
what you're saying is like a registry.
You can add that as a dependency
with like, A qualifier saying that
it's this specific package in this
registry as opposed to the github url.
Is that correct?
Okay.
Yes.
Okay.
Dave Verwer (guest): And, then you,
it'll pull in so the, the, they'll
still, we will, we will still use.
Get repositories to store our source
code, but they'll, there will now
be this kind of intermediate step.
And how that all falls out is probably,
we're probably a little bit premature
talking about that now because we, we, we
haven't seen what they're gonna develop.
the package index.
Is going to be a centralized place
because at some point if you are
gonna deal with discovery and search
for multiple package registries, you
need to have one place where all of
those registries can be searched.
Otherwise, you'd need to go.
If you were looking for a package,
you'd need to go to each registry.
Not that searches even
a part of a registry.
It's not part of the, the registry spec.
It search is search is not there.
, we'll still need something
to gather metadata from.
Get your URLs and package registries,
bring it all together and make it
searchable and allow packages to be
discovered matter where they're hosted.
Awesome.
And that's what we'll
Leo Dion (host): take down.
That's awesome.
Yeah.
haven't dived into, but now
I think I might dive into
registries on how they work.
like that I think will increase
obviously your audience.
I assume your audience continues to grow.
You can see that, right?
people are using swift
packages more and more.
definitely.
Dave Verwer (guest): Yeah, we,
we've seen consistent and, and
steady ever since we launched.
Really.
And in fact, I was just looking
at something earlier today, and
our traffic has increased by
over 50% since August last year.
So we are growing, we're, we're, we are
growing really nicely, which is great.
What, and that, I mean, that
I think proves two things.
It proves that what we're
producing is useful, but it also.
I think mirrors the adoption
of sif Package Manager.
I think as we go through you CocoaPods
was, and still is totally dominant
in terms of and macOS projects.
It was just, that was the way that
you imported Carthridge was there as
well, but I, Carthage never really
got the adoption that Cocoa Pods did.
And, I think we can see clearly.
Now that package manager is, is
gaining huge amounts of adoption.
CocoaPods is gonna be around for
a very, very long time because
there are lots of projects that
will not upgrade until there's a
reason that they have to upgrade.
Leo Dion (host): Right, right.
You still have objective C stuff
out that will be different there.
If you have anything that's
subjective based course, like
you have to be on Cocoa Pods.
That's it.
There's nothing out there.
,
Dave Verwer (guest): but I think
we're certainly seeing package
manager take more share of that market
Leo Dion (host): Anything else you want
to talk about that's coming out this year?
Dave Verwer (guest): think we could talk
a little bit about maybe what yeah, yeah.
Let's talk a little bit about what
we're planning to do over this.
Yeah.
,
Leo Dion (host): go ahead.
Yeah,
,
. Dave Verwer (guest): think one
thing that, that, oh, sorry.
Go on.
Sorry.
Sven A. Schmidt (guest): No, I was
just maybe picking up on the DocC
stuff that we talked about earlier
because seen a lot of adoption.
just checked earlier, we've
got 270 packages now that are
hosting documentation with
us, which is, that's great.
That's more than We have, however, seen
some issues with some of those packages
because they are, they generate very large
dock setss and that is something that
is, that is a , a feature of, of DocC.
It produces a lot of small
files and they, they add up.
one package in particular has, is, is so
large and has so many files that it, it.
in our current setup, impossible to
manage, to upload it within our timeouts
that we currently, can you tell me what it
is In our bill system ? is swift syntax.
It's Apple Swift Syntax package.
Okay.
Okay.
And it's the nature of the package.
I mean, there's, it's, it's not,
it's not the authors or the packages
fold, it's just the nature Yeah.
If a package,
Leo Dion (host): I've
looked at that before.
Yeah, I can, I can see
where that's coming from.
.
Sven A. Schmidt (guest): Yeah.
Every symbol that's so committed
generates a adjacent file.
Mm-hmm.
with data in it.
It's like, like a couple
hundred kilobytes.
It's not, but because every function
has that, every method, every attribute,
it's, it has 117,000 files that doc
set and is, is, is a gigabyte large,
which, isn't crazy, but because
we need to build the package, build
the documentation, and upload it
all within a maximum of 15 minutes.
that that is just not enough So we
have to work around that sort of stuff.
And we're also hoping that so we are part
of the documentation working group we have
flagged this, and they are, people are
aware that it's, they are some problems
with the, with the size, the sheer
size of Doxies doc sets tremendously.
If, if, you know, that would
be one way to, to, you know,
bring the size down a bit.
But there are also things that
we can do to offload that whole
process and, you know, not try and
sync 117,000 files in that job.
Yeah.
But rather zip it up, upload one
file elsewhere, and then have
a, have a job with more time
budget, deal with the upload.
And that's what we're currently working.
, but these are the sort of things that pop
up when you've shipped the feature and
then it gains adoption and you, you see
all sorts of, you know, Edge cases pop
up, and that's, that's just the way it is.
So that's something we are,
we are dealing with right now.
another very interesting thing that
came out of our participation in a
documentation working group, and that's
preview of upcoming DocC features.
So DocC shipped in its first version
That's two WWDC.
Yeah, of course.
Because in June, June last year,
we shipped the, the DocC hosting.
So it was a year old But
it's, it's still pretty young.
So there has have been lots of
things that have been added since.
most recently there was a, a feature
by Sophia Rodriguez which is a, a
search popup, which is really nice.
So if you are on a DocC Works set,
you can I think it's just forward
slash you know, Command popups you get
when you have Launch bar or Raycast.
So you have a dedicated search
that you can bring up and then you
just start, you know, type search.
As you type, you get results.
You know, that's awesome.
Which is really great.
Yeah.
And we've, so we've set up
a preview system of that
feature with a preview of DocC.
This is like a nightly five eight
chain where we've, where we're, we are
previewing that and we hope, we hope to
get this rolled out potentially before
five, eight chips in, I guess the spring.
Right.
that's the sort of stuff we're
working towards and hoping.
Do you able to pull forward so
people can, can have this feature
you on the Swift package index as as.
Leo Dion (host): Are you doing anything
with like quality or maintenance scores?
Because that's a big thing.
I use Shields io and I like to post
those badges in my repos showing how
good, good of a developer what is it,
a well-behaved developer I am when
it comes to maintenance and quality
Dave Verwer (guest): scores.
Well, I can vouch for how, for
how Well that, yeah, yeah, yeah.
Thank you.
Leo Dion (host): you wanna do something
like that with Swift package Index?
Dave Verwer (guest): It certainly.
So I, I think one thing that is on
my mind at the moment is that we
built the Swift package Index where
it shows metadata and we built that.
, you know, a couple of years
ago now, and then we decided to
tackle compatibility systems.
So we ended up going deep into this,
this huge build system that we have
hanging off the back of the packaging.
Then index and it, it feeds
compatibility results back.
And then we added to that
with documentation generation.
, we've not, I mean, there's been
various improvements of course,
but I, I, I'm conscious that we
haven't done much on the actual core
package page for a little while now.
for example, we did some
initial work on dependencies.
And we are pulling in resolved
dependencies for every package and
we have that stored in the database.
we're not really doing
a lot with that data.
we also, that data kind of
comes with some tricky things.
Like for example, if.
test target declares a dependency,
and we are maybe showing how
many dependencies a package has.
Do people want to know
about test dependencies?
I think most people are primarily
concerned with what dependencies a package
has that might ev eventually end up
Leo Dion (host): in their application
products, executables, et cetera.
Yeah.
Dave Verwer (guest): And so there's
a lot work to be done there on.
A, an, an intelligent display of how
those dependencies actually affect
what you bring in with a package.
Okay?
We, we certainly shouldn't
just ignore test dependencies.
That's not the way to do it.
But to, to, to represent that data in
a readable and a, a, form that elects
people, make intelligent decisions.
what packages to use.
So that's something that, that
think we're, we're looking to
get back to in the next year.
as you say, you, you know, there's
a lot of these metrics that we are
already pulling into the package
page could be determined to give some
kind quick glanceable rating of how
well maintained or how, you know,
what, what's the, you know, how.
It's, it's, I don't really wanna say
how good is this package, but because
it's not about this, but, but certainly
some kind of glanceable metadata that
says, yeah, this is something that
I wanna look a little deeper into.
it, that can come from all sorts of stuff.
Some of that will come from dependencies.
Some of it will come from.
The, the metadata we have about how
well maintained packages in terms of our
issues being closed, our pull requests
being merged, that kind of thing.
Yeah.
Does it have a read me file?
Does it have documentation?
Does it have tests?
There's a lot of things that we
could expose in, first of all, on
the package page, and then maybe
infer from some kind of quality
score or something like that from it.
It was something that Cocoapods did.
They don't do it anymore.
But they Cocoapods had a quality index.
And you, you could go to a, a
little webpage for every mm-hmm.
that had like five or six
different metrics of like, this is
something that it is doing right.
This is something that's that, that
it maybe could improve And it was
things like how complex is the readme?
There was a package that, there was
tool they used Try and get a handle
on how good a readme There's lots
of possibilities for that kind of
be To be used by the package index.
So some something I'm interested
Leo Dion (host): in working on.
Yeah, I mean, one thing I always
look at, and I've done this actually
more with npm, is just like, has
this package actually been maintained
in the last, like two years?
Cuz that's usually a decent
indication that it's either a
dead product or, or still active.
But it sounds like, yeah,
you want to add some of those
metrics, which, which is awesome.
I'm looking forward to that.
.
Dave Verwer (guest): And the other thing
that we won't be able to is a lot of
these package NPM and and others because
they are both the registry and the index
in most cases, like mpm for example,
is, is, is vans, the, the, the package
data or the, the dependency data and it
is the search engine, so it can use how
many people are downloading or using.
Right?
Right.
Dependency as some kind of indicator.
Now we absolutely don't have that
because we have no, we have no way
to to who's using each package.
We don't even know really, you know,
all we can see is, is on how many
visits each package page has, basically.
Right, right.
So we have none of that information.
Maybe that's something that registry
eventually, I don't think it
supports it in its current form,
but maybe it could potentially
start gathering some of that data.
But again, there's good
arguments for it, not I agree.
Yeah.
Leo Dion (host): I'm not sure that
that is useful necessarily but it
sounds like you have already a lot
going on with Swift Package Index.
It sounds like you have
a lot coming this Yeah.
people.
Have I been to Swift Package Index
and you have a sponsorship as well?
Definitely take some time and some support
financially and maybe put in a pull
request if there's something you can fix.
.
Dave Verwer (guest): Well, there
are many ways to support it.
Yeah.
And, and, and before I go into
this, I think like to say, or we
would like to say thank you so
much for the little advertisement
you've been, been running for us.
That's, that's very much appreciated.
And we, yes, we do have a GitHub.
Sponsors account, which you can find
on any of the GitHub repositories.
Or on the package index page itself,
there's a little, a little pink heart
in the top menu bar that you can
click to go to our GitHub sponsors.
also there's many ways to help.
You could also like many people
have done this here as you could
involved in the development, we
have a discord server where we meet
with all of the contributors of.
so far and other people who are
just generally interested in what
we're doing on the package index.
So the link to that discord is on our.
Main repositories.
We, and
Leo Dion (host): we'll put
that as well in the show notes.
Of course notes.
Thank you so much Dave.
It's fun for coming on the show.
I really appreciate it.
It's glad to have you both on.
Thank you so much for your
work on Swift package decks.
dunno what else I have to say,
but yeah, it's been wonderful
Dave Verwer (guest): chatting.
Thanks so much for having us.
I'm, yeah, thanks having us on show again.
Yeah.
Leo Dion (host): As a double actor, where
could people find both of you online?
Dave Verwer (guest): Best place
to find me is daveverwer.com.
ev Links to everything from there
Sven A. Schmidt (guest): I
am on Mastodon these days.
finestructure@mastodon, oh God.
Master on social.
Yes.
. . Don't even know the domain.
It's the, the main side, but fine
structure is, is the username
fine as in, you know, fine and
structure if I any structure.
Leo Dion (host): Well, thank you so much.
People can find me on
Twitter at Leo g Dion.
Maam, I'm Leo g Dion, c dot i
m if you are listening to this
podcast, please take some time.
To post a review and also subscribe to
the Swift Package Index Podcast as well.
And if you're watching this on
YouTube, subscribe and like, thank
you so much for joining me and I
look forward to talking to you again.
Bye everyone.
