Leo Dion (host): Hello and welcome to another episode at EmpowerApps.
I'm your host, Leo Dion.
Today I am joined by Sven Schmidt and Dave Verwer of Swift Package Index.
Thank you guys for coming on.
You've both been on separately.
This is the first time I've had you on together.
is awesome.
Thank you for doing this.
Dave Verwer (guest): Thanks for having us, Leo.
Hey, Leo.
Yeah, thanks for having us.
Leo Dion (host): Before we begin, I'll let you introduce yourself for
those who don't know what the Swift Package Index is and what you do.
Dave Verwer (guest): Sure.
So my name's Dave Verwer with Sven I run the Swift Package Index.
also run a newsletter called iOS Dev Weekly various other sites have directory.
I always have jobs, basically everything starting with
iOSdev something to do with me,
. Sven A. Schmidt (guest): Yeah.
Hey, I'm, I'm Sven, I'm an indie developer podcaster.
Even most recently we've started out together, Dave and I I've been working on
the Swift Package Index this past almost three years now, together with Dave.
Mm-hmm.
. Awesome.
two other apps that I have, but the Swift Package Index is, is really
the main thing that I'm doing.
Leo Dion (host): So you, do, you do a podcast?
Is this this related to the Twitter spaces or is this something different?
. Dave Verwer (guest): That's certainly how it started.
Yeah.
So we.
decided few months ago now to, to do little Twitter spaces, to to basically
just chat through a little bit of what's been going on with the package
index and then also to recommend some packages that we've seen fast
on the, on the feeds basically.
So whenever packages get updated, We can subscribe to an rss.
In fact, anyone can subscribe to an RSS feed of those updates.
We check over those updates every week and every couple of weeks.
We did a Twitter space to talk about those packages and also talk about what
was happening with the package index.
went down reasonably well, and we decided to kind of upgrade that a little bit
into an actual podcast because we did get some requests for people to Yeah.
Who wanted to listen to it in a podcast player.
So yeah, the last three I think, or three or so episodes have
been proper podcast episodes.
Okay.
we, we experimented with live streaming to YouTube and then we experimented
with stopping live streaming to YouTube.
now it's just a regular.
Podcast.
Yeah, I'm not, we still do have a YouTube version, but, but
the podcast is the main way.
Leo Dion (host): Yeah.
Yeah.
I'm not, I'm not Yeah.
Risky enough to, to willing to do live streams as well with the, with this stuff.
So yeah.
I commend you for doing that experiment.
I gonna say?
Yeah.
Well, I'm glad you didn't go the other route and go to
Clubhouse, I guess, so if that's
Dave Verwer (guest): still a thing, right?
Yes.
Is that still a thing even now
Leo Dion (host): so Dave, you spoke recently at the wonderful
Server-Side Swift conference that Tim Condon had put that seemed like
that conference was a huge success.
Apple revealing that they're open sourcing or making available the
foundation in the library, I believe.
But I want to kind of just do a little brief redux on your talk that you did
on how you accidentally ended up running the largest open source Vapor site.
I love that title.
And
Dave Verwer (guest): so, I mean, one of the jokes that I made in the talk
was if you put enough conditions on anything, you can win any competition.
That's right.
. Yeah.
. Open source Vapor . So, yes, it was really, it was really little
bit of history about how the Swift Baggage Index came to be.
it was previous to being a Swift package index.
It was actually a little rails application that was a closed
source rails application initially.
then through a conversation that, and I had over email.
that the Swift Package Index.
And, and it was an open source project from day one because we knew that if, if
it stood any chance of becoming really significantly becoming the place to find
Swift packages, it had to be source.
And if it was gonna be open source for the Swift community,
it had to be written in Swift.
So was, it was very clear immediately that we, that when we were talking
about in fact, , we were talking about redoing it in Swift because of
various other conversations that had gone But it was very clear that it,
that it, it should be a Vapor project.
Leo Dion (host): Yeah.
Yeah.
And of the things that I think you mentioned was the fact that like, Vapor
was, was, what was it at the time, really?
not sure where Kitura had stood, but Perfect, at least had been discontinued.
So it just made an obvious choice.
Dave Verwer (guest): I think as well Spen had some experience with Vapor
already, which was huge bonus because I had actually , I had actually
started, I'd made a new Vapor project, but it was also in that period
between Vapor three and Vapor Four.
I'd made a Vapor three project because there was better
examples and documentation.
But then when Sven kind of came on board with some Vapor experience, everything,
everything changed from that point.
So maybe, maybe Sven Can can talk about it from his perspective.
Sven A. Schmidt (guest): Yeah.
I had been running a small.
Vapor 3 project before in, in production.
Like it was doing a real thing and I had good experience with
But also there was, it was good documentation, which I had worked with.
And the Discord community, the Vapor Discord community was there.
I knew about it.
I, I was in it.
I There'd be enough people to answer questions and, and I
made heavy use of that early on, and that was really helpful.
at the time there weren't really that many other options.
You mentioned couture perfect.
Which was sort of, you know, if not already discontinued on the way out.
Smoke I don't think existed yet.
Amazon's smoke and, and Hummingbird by Adam Fowler didn't exist yet either.
So Vapor.
Pretty obviously obvious thing to pick if it's swift spec'd.
I had previously run two Python services in Flask and also another one in Django.
So I had some experience running that kind of service.
I think that was also similar, but I have to say in hindsight, I would definitely
pick Vapor and Swift over these, over these backend frameworks because.
of issues that I faced with the Vapor, with the Python based services we
don't actually have in production.
Like I remember fighting with Python and the service just dying, running
out of memory, that sort of stuff.
We don't see any of that with with the Vapor Service.
It's rock solid in production.
runtime issues just don't exist because they are compiled home issues in Swift.
works just really, really well.
I think some smaller.
issues early on we had mostly around foundation not being exactly a
hundred percent the same between macOS, where we develop and
Linux where we, where we deploy.
Yeah.
But once you sort of know what, these are mostly around formats
and stuff, you can work around it.
Plus UCI catches that.
I mean, you're never going to accidentally ship that.
Right, right.
In Python, that might be a problem.
You know, you, you don't hit that code path, you ship it and
then in production you hit it.
But the compiler makes sure you can't actually ship that sort of stuff in Swift.
So I, I think it, it works really well.
The, the, the other downside perhaps is that the package ecosystem doesn't
have the same depth as it does on other platforms that are older.
You know, like Ruby, Ruby on Rails, Python again.
guess also go.
, I don't think breadth is necessarily what's lacking in Swift.
I, you might argue that there aren't many variants of different services,
you know, to, to choose from.
But of all the stuff that we've been doing, there's always been a package,
and that package has, has pretty much always been really high quality.
You didn't have like five to choose from, you know, three of which
might be questionable and, and not maintain, but you had often had that
one and, and that was good enough.
mean, our service is probably not the most edge, you know, the,
you know, on the bleeding edge.
We're not, we're not doing authentication at the moment beyond
some, some basic basic types.
We're not doing image video uploading, transcoding, that sort of
stuff, but I know others are, and, and it, it works perfectly fine.
Yeah.
Talk to the
Leo Dion (host): folks at Amazon.
. Dave Verwer (guest): Exactly.
Yeah.
I think there's, and Tim, yeah, I think there's two
sides to this question as well.
first is certainly Vapor and running a web application on Swift is a very stable
environment the, to, to in production.
And, and we've seen that over, over years now of, of running the site.
But I think the other side of it is we also.
Build a lot of stuff into making sure that's the case.
So extensive test suite monitoring, good logging and a lot of those
tools really help that reliability.
It wouldn't be unreliable without those, but it's more than just
picking the right framework.
It's, it's how you approach development of your entire service.
I think that that makes a stable service.
Leo Dion (host): Yeah.
And one of the things I remember spend when I had you on was talking about
some of those libraries that you use, like prometheus and snapshot testing
make sure that the site is running and doing, you know, is staying stable.
Yeah.
. Sven A. Schmidt (guest): Yeah.
And, and one of the reasons why I'm so confident that Swift is better, you know,
from my experience with, as opposed to the Python frameworks, is because we
did exactly the same things in Python.
I mean, I'm, I'm a test driven nurse, okay?
I, I do that in all projects because otherwise I can't, I feel I can't
deploy unless I've run a test suite.
So I have these in place and, and yet still I see a lot of issues not appearing
in Swift, that that did appear in Python.
. So yeah, that's, that's my 2
Leo Dion (host): cents on that.
Yeah.
And I think this is the gear.
If you're, I, this is probably the second year I've said this,
but like, yeah, I mean, there's no reason not to go with a server side
backend and swift at this point.
So
Sven A. Schmidt (guest): is it the year of Swift on the server?
Well, that
Leo Dion (host): was, that was the episode I did last year and I can't, I can't
do that again, unfortunately with Tim.
So,
Dave Verwer (guest): This last year has been all about documentation and not
the documentation of the Swift package index, but so a little while ago I
can't remember exactly when it was, but it was probably almost a year ago.
We shipped the first version of.
DocC documentation generation support on the package index.
So just a quick recap of what that is.
If you have a Swift package that is already in the package index and if your
Swift files have documentation comments that DocC would understand and would
produce documentation from, and of course you can test that locally with Xcode.
Swift Package manager.
Locally you can indicate to the Swift package index that we should build
and host your package documentation.
and we do a couple of things, which think in my, my opinion, takes
us above and beyond, beyond even what, what people would potentially
set up with their systems.
So the first thing that we do is all you really need to tell us is what
target or targets that you would like documentation to be generated for.
. And then as we do our compatibility builds because behind the package
index is a full, almost a kind of full CI system for every package
where we verify compatibility with platforms swift versions by building
each package hundreds of times.
the end of that process, we tacked on the documentation generation step, so, Package
authors have let us know that there are three documented targets in a package.
As we build that package, we'll also generate documentation, upload
it, and host it automatically.
Like literally, you don't need to do anything apart from,
from it's awesome targets.
Yeah.
And then secondly we keep every version.
Documentation.
So as your package is moved forward through releases and especially major
releases we, we update the documentation every time you release a package.
And then also if you upgrade a major version, version two to version
three, for example, we'll keep the old major version, the latest
old major versions documentation around and put it in a little
dropdown on your documentation site.
So when you make breaking API changes, Generally get indicated
by a major version change.
We'll keep that old version around for you.
And again, all of that just happens completely automatically.
You just release your versions and that's it.
It's done.
So that's a, a very quick recap of what the feature does.
of course, To provide that level of simplicity for the end user
and also for the package author.
There's an enormous amount of work maybe can talk a little bit about
of the stuff that we've, we've worked on with that this year.
Right.
Yeah.
Sven A. Schmidt (guest): So the, the DocC hosting, as Dave mentioned
is, is a tack onto the build system.
And, and the tricky bit there is, you know, to pick which targets actually
the docs are built for the uploading to S3 for that, and especially
the proxying around because we, we are adding a, a custom header.
The, the thing that Dave just described, the bar on the, on the top of it,
where you have a chooser of which target you're viewing docs for, and.
Which version you're looking at.
normal DocC site doesn't have any of that, so we have to fiddle a bit with the
structure of the webpage to inject that.
just to mer make everything work with us hosting that site you know,
and, and doing the redirecting and, and proxying of all those URLs.
Well, I was just gonna
Leo Dion (host): say the DocC building is fantastic on Swift Package Index.
Set up stuff with Netlify on hosting my documentation.
But like the fact that you do automatically, and like Dave said,
has all these quirks with like major version, I didn't even think of that.
Like yeah, if you update to a new major version, you have breaking changes.
Like people might wanna pull up the old docs.
Just yesterday, speaking of like put.
I updated an old repo that I had and renamed it, and like I was able to
get this DocC documentation working in minutes, and it's fantastic.
That's great to have.
Yeah.
I want to jump a little bit and talk about you add a
package, two Swift package index.
It's simple.
You know, Dave, you're a big fan of doing pull requests.
that's, that's it.
That's all you have to do.
You put it in a pull request, it's still almost working.
It's awesome.
And like you have to do.
But then there's some custom quirks that.
You know, swift package Index, like you said, DocC You know, we've talked about,
I think watch kit testing, which used to be a thing, but now it's native anyways.
But there's a few things that you can do in swift Package Index to
customize using this like YAML file you've set up, you want to explain
what that, what purpose that serves.
Sure.
And how it works with Swift Package Index.
Dave Verwer (guest): So, from the very beginning we had w w we knew that at
some point we would need ways for package authors to tell us bits of metadata
that we might need to know that were was outside of the Package.swift file..
Because obviously we get a lot of, we had as much as we can from.
the package manifest, and also services like GitHub, where packages are hosted.
Mm-hmm.
. we knew proposing changes to the package manifest is a long and
slow process as it should be.
You know, we shouldn't be, we shouldn't be changing that file every, every , but
that we'd need a way for package authors to be able to tell us various bits of
metadata that was not covered by that.
Originally I think the idea for the Swift YAML file came from us
wanting to add author support, which we recently just added.
So I'm actually just gonna go off a slight tangent.
Yeah, go for it.
Talk about this.
We Swift package Index participated in the Swift Mentorship program again this year.
So we had two mentees come on board and implement features.
We was mentoring one, and I mentored the other, and my mentee
basically he implemented author support for the swift package index.
So passing out git history, finding the most people with the most commits
deciding who was like a significant author based on an algorithm and.
putting that information live on, on the site.
But then of course with anything automated, it's gonna get it wrong
sometimes or get history as maybe not as accurate as, as people
or it do maybe doesn't represent who actually wrote package.
And so we wanted a way for people to be able to override what
the automated system had done.
And again, we used the Swift package index YAML file to allow package authors to say,
actually your author information that you've kind of derived isn't quite right.
Here's the string you should put to say who wrote this so even though that
feature has only just recently been implemented we've had the idea for that
feature a very long time ago, and so we knew that we'd need this package
Swift package index YAML file to do that.
But then the first time it actually got used was when the compatibility build.
being developed.
So we I think it was the very first thing we needed to do was custom targets.
Is that
Sven A. Schmidt (guest): Schemes.
Schemes, schemes.
Yeah.
It was schemes.
I, I remember, well, we were still, we had an open discussion.
We had an issue in our GitHub talking about what the format
should be of that file.
And you can imagine what that discussion was like.
, you know, oh, don't use YAML, use JSON.
Well, JSON is terrible.
You can't have comments use TOML and you know, , while the
discussion was still brewing.
We really just needed some way of, of authors to tell us scheme to use
because you mentioned it earlier.
There was one issue that you couldn't build on watch os if there was a,
I think if there was a test, right?
Because of, yeah,
Leo Dion (host): you had needed, you had the example of the point free I
think it was composable architecture.
Sven A. Schmidt (guest): Yeah, exactly.
Right.
And the way we, in the build system we had, we sort of had to, we, what
we're doing is we're guessing which scheme is the right one to build.
There are often many, and we just pick one.
Certain heuristics, they sometimes fail and it's less and less common,
but especially early on, people needed a way to tell us just use
this scheme for this platform.
So effectively we say, all right, YAML, it is . Everyone's gonna be unhappy,
but this one is, this is what we're going to choose just to get this fixed.
And, and that's how this file was born and has lived ever since.
Pretty much.
Leo Dion (host): So
Dave Verwer (guest): yeah, go ahead Dave.
And it's done us really well.
terms We've expanded the amount of metadata that package authors
can to that file over time.
And that's also where package authors us which targets our
documents and that kind of thing.
There's two main advantages of doing this file in this way is we could have made
a system where package authors needed to authenticate on packaging next.com, log
in, and maybe they get a web view where they can say, oh, it's these targets,
and it's, you know, this is the correct author information and that kind of thing.
But because everything comes from a Git repository, that's the
source of truth for everything to, to have some of the metadata.
Somewhere else, maybe behind a specific site, never really felt right to me.
And to have that as a YAML file in the package repository
feels like the right solution.
It also means that any other tools can also look at that file because
it's just a file in a git repository.
think as far as I'm aware, we are the only, we are the app
that does look at that file.
certainly it felt right to have that metadata.
In the package repository.
course there are downsides to it because it's much more difficult
to tell people that they should go looking at the package documentation.
And actually, just to go off on another tangent, go for it.
The other mentorship project Sven with his was to document that file.
So maybe you can talk about a bit about that Sven..
Sven A. Schmidt (guest): Yeah.
I mean, this has been, this, this popped up quite a few times that.
You know, knew there was this file, but they didn't know what the format
was, what the, what the actual things are that you can put in it.
And we had a blog post that introduced it and a couple bits and bobs here
and there, but no central place to actually host more document.
I mean, the obvious central place is obviously the package index, but
we never actually put up a a page on it that explained that file better.
And then we kind of thought, well, , we had just built a doc system and we
had pushed all of the, the specs that, like the SPI YAML file is
actually Codable struct that is behind it is in a separate package
that we have in the, in our GitHub.
So it's Swift package index slash SPI manifest, and the
package is actually in the index.
And then the obvious thing was to just add documentation, oxy documentation, have it
generated through the index, and thenpoint towards that documentation in the index.
And that's what Azza worked towards
actually
Sven A. Schmidt (guest): adding docs to that package such that we can host
it and then direct people towards that file and have Nice, it has examples now.
We chose, you know, what things you can do.
Yeah.
How you add to, I actually, I mean, that's the thing.
I had snippets lying around to tell people what to do, and now
finally I can , I can look it up
Leo Dion (host): myself.
Did you.
Did you really deep dive into DocC and like do tutorials and
articles and things like that?
Or did you just stick stay with like, document, like just swift comments?
Sven A. Schmidt (guest): Yeah, it's, it's, this is just a, it has
a intro just marked down pages.
Yeah.
Okay.
I mean, it's obviously the, the structure is documented to a certain degree, but
the main documentation for this kind of thing has to be a markdown file
describing or what the purpose is and, and you know, I think the, the most
interesting page there is common use cases where we describe, you know, what
do I do if I want to override the scheme?
Okay, what do I do if I want to override the target?
How to add documentation?
That sort of thing.
Yeah, yeah, yeah.
Because those are the main questions people ask.
Leo Dion (host): One of the big improvements it sounds like you wanna do
this year is to searching on the site.
do you wanna explain how it works now and what you're hoping to do in 2023?
Dave Verwer (guest): Well, I think actually.
, I think what you're potentially referring to is a change that we've
already made , and again, this was a this was a contribution the
open source project from Joe Heck.
So Joe noticed some problems when he was searching for think he,
was he searching for Ping or ftp?
One of the two.
Yeah, both of them.
Yeah.
Those were those.
. Those are the two.
Yeah, those are the two problems.
Both of those have very ambiguous not ambiguous, that's the wrong word, but
both of those phrases occur in lots of other words in ways that you wouldn't Yes.
And it was putting potentially terrible search results to the top of the list.
so he originally reported it.
as an issue.
And we talked about it a little bit, but he was also interested in in,
doing some work to, to solve it.
And he did some amazing work with looking at some of the Postgres search commands
I forget the name of technology, maybe span remembers what other
things that they've broken up into.
Stemming the
Leo Dion (host): heuristics.
Okay.
Dave Verwer (guest): Yeah.
Stemming and stemming.
There was something else as well.
I can't remember what it was.
Yeah.
And, so that landed few months ago And what it does in, in real terms is make.
Vastly improved search results where if you type ftp it's more, much
more likely now to show you things that are about FTP instead of that
being part of some of the word.
Okay.
Sven A. Schmidt (guest): That's really cool.
Yeah.
Just maybe to explain by, by example what the problem was.
So imagine you search for, for Ping, you get all sorts of hits
from mapping, you know, ing being ping, being a pretty common ending.
Ftp, likewise Swift package.
ftp, swift FTP package.
Yeah.
Right.
You know, when that's spelled together, you get hits like that.
They have nothing to do with what you were actually looking for.
So what you can do is have run an index does word stemming so it, it
breaks up words and, and discards endings that aren't actually, you
know, part parts of the world really.
Right.
that then gives you good search results if you, if you ranked by those, it was a
bit more complicated than that because we had to weave it into our existing search
and our, you know, we want to preserve a lot of aspects of our current search.
Joe did a lot of work.
I mean, Joe actually wrote a SwiftUI app that took searches that he'd.
, which he had captured.
So he captured the ranking, had a Swift UI app to actually look at
those results and to compare them.
He went all in on that.
That's awesome because it was amazing.
Leo Dion (host): Is this, and this all works within Vapor and Fluent or like,
was there some customization in there?
Sven A. Schmidt (guest): So what saves us here is that
you can drop down to pure sql.
Okay.
I mean, I, I'm sure you.
potentially write this in, not in fluent, but in, there's, there are
other layers in Vapor below flu.
Right, right, SQLKit, for instance, where you can, you know, write your
SQL functions and then assemble them.
But in this case, in search, we are really assemble what we're using.
We are using SQLKit, but a lot of SQL RAW as well that we then assemble.
that's how we build up the, the search string.
And it's effectively going through a materialized view to keep it fast.
Yeah, that makes total sense.
But this sort of stuff you can't, you can't do with
Leo Dion (host): fluent.
Yeah.
You can get into the raw sequel, which sounds like what you guys
did and it makes sense with search.
ahead Dave.
Dave Verwer (guest): And the nice thing about being able to talk about this on, on
a podcast is that that kind of work that Joe did on that is really thankless, like.
He did an enormous amount of work to get that pull request in and, and, and,
and to improve the search on the site.
And to anybody using the site, it still just produces search results, right?
You only that, that's the kind of change where only really notice when
it goes wrong and it, the fact that nobody noticed is, is a great testament
to how good that changed from Joe was.
Yeah.
That's a, that's
Leo Dion (host): awesome.
. So one of the other things you've talked about quite a bit is the
whole we, we just got this whole infrastructure, I believe it was June
of this year with Swift package plugins.
What, what are you doing with Swift package plugins and Swift Package
Index and how, how, how do those two, what's their relationship, I guess?
Sven A. Schmidt (guest): we've passed this package manifests the whole time.
And, and plugins are a new type of product that appeared
when they, when they came live.
But initially we were blind to them because they weren't
a product that we knew.
So we had to had to change that to actually ingest But I think that
happened pretty early on because if I recall correctly, not passing them
led to, you know, the packages with plugins actually failing to index.
So that was an obvious thing to do.
But of the things that, that was also a contributing change . and
was support for plugin search.
So you may not know we have certain extensions on search.
You can obviously search for terms, but you can also apply little qualifiers.
So for instance, platform colon iOS, which, you know, lists packages
that are compatible with iOS.
But we also have a.
, product colon and then executable product colon library.
And then product colon Mine, plugin.
Added a support for product colon plugin.
And then you can search for, for plugins, you know, you can just search
for product colon plugin that lists you all the packages with plugin.
The package index knows about, and you can obviously add other
search terms in addition to that, to to filter that down further.
that was really cool because that allowed us early on to, to see, you
know, to expose what plugins they actually are and see them, you know,
the lists grow over the months as they, as they started appearing.
Leo Dion (host): What are some quirks that you found with plugins that
you didn't have to run into besides just the new product type I guess
? Sven A. Schmidt (guest): I don't think there were any problems really.
Well actually, let me qualify that one issue that is, is there, now that
I think of it that is there, and I don't think there's a, an obvious
fix is compatibility of plugins.
Okay.
Because what do you actually We currently build a package
to determine compatibility.
I'm not actually sure what happens if you run Swift Build on a.
Plugin package.
Okay.
Will it just, will it actually build anything?
Would you actually need to use it to, to determine compatibility?
I'm, I'm a bit hazy what would actually happen, but I do recall that we had
packages being flagged and people said, well, is this should be compatible,
but sort of isn't doing the thing.
But I think they had an example in, in their repository that we then built.
But the plugin was actually attached to that.
So I'm not sure what the state there is.
I don't think it's a currently not a common enough problem
to be, to be an issue.
because plugins are sort of a dependency when you use them.
Mm-hmm.
, right?
But then you're gonna also not because they, they do something in your
project, but they're mostly not part of your built product in the end.
Right?
There is no code that lands in your, in your package from the plugin
unless the plugin transforms code and that that's then gets added.
So I think they're a bit weird in terms of are they, are they
dependencies in that, in that sense?
And how do you test for compatibility in that sense, I guess?
Yeah,
Dave Verwer (guest): and I think the other thing with compatibility around plugins is
it's never going to be easy or potentially never even gonna be possible to represent
the full spectrum of compatibility, of every part of every package.
Is this Pluggin compatible with this platform?
Like even just representing that data would be extremely challenging.
And then for people to read that data and get what they need from, it
also needs to be relatively simple.
Right.
And I, I would, ima I think in, in my head at least what we have right now
with that grid is at the very top end.
of how complex we want compatibility to look.
I would not want it to be any more complex than it currently is.
In fact, I would like to simplify that a little bit if we can over time.
Right, right.
And that makes sense.
And so, yes, we may not capture some of the intricacies of, well, this
package is compatible with iOS, but this plug-in needs macOS, or you know what?
Whatever there are, there are always gonna be things there that
we might not be able to To people who are looking for a package.
so there's a limit to how far we can really go down that path,
or at least how far we can go without making things to companies,
Leo Dion (host): right?
Yeah, exactly.
So I wanna talk a little bit about the future of the Swift Package Index.
In our last episode one of the things we were talking about
is registry is becoming more and more wide known and used.
And it sounds like you're doing something with that for Swift package Index in 2023,
Dave Verwer (guest): Yeah.
So think 2023 will be the year that people start to hear a little
bit more about package registries.
registries have been, or the idea for a package registry in Swift has
been around for a few years now, and in fact, in 5.7 the client,
the Swift Package manager client.
Gained full support for package registries, but it's probably worth
talking a little bit about what the difference is between a package
registry and a package index.
Before we go too further, but too much further, because that's certainly going
to be something that people may not have been aware of yet because package
registries have been, you would've heard of them if you were like super
in the weeds on package manager.
but most people are just using packages, right?
Yeah.
And they just put a git URL in and off it goes, and everything works.
So a package registry and there can be multiple package registries.
The Swift package manager supports multiple package registries.
So GitHub could run a package registry.
Apple could run a package registry.
Amazon could run a package registry.
BrightDigit could run their own internal package registry.
You're in my mind.
Yeah.
I was just thinking about better.
I was like, yeah, I should try that out.
. The, the, the, the, the concept of a registry is, is there's no centralized
place by definition for a registry.
And the registry takes care of all the hard stuff, basically.
So it takes.
Delivering package data it takes care verifying that the contents of
package data is, is signed and secure and, and verifiably not tampered with
since the package author of certified it, which is a problem that is
currently the current implementation package manager has that problem
where basically if you tag a version That's the version that the package
manager takes as a specific version.
But with git tags, they're mutable, right?
So you could move a tag around and, and potentially one person who's
using your package at this version has different code to another person that's
Leo Dion (host): using the packages.
As somebody who's developed Swift packages, I've run, I've done, I've tagged
something and then been like, oh crap.
Like forgot to do this and this, and then I have to go in and I have to delete my.
Package cash, which is horrible.
If you've ever done that, don't ever do that unless you
absolutely need to on your machine.
yeah, I've totally, I, I totally know where you're coming from on that.
Dave Verwer (guest): So, package registries are gonna
solve a lot of these problems.
But they, it, it'll mean that you're no longer putting git your URLs
into your package manifest because instead of getting it from a Git
repository, you'll be getting the package from a package registry.
so there's some changes, you know, I mean, these are these changes even
though they're supported by the current version of package manager, they're
probably a little way down the road because what Apple announced recently
was that they are going to build a implementation of the package, a
package registry server that will be compatible with the package manager.
So apple are open source developing this package.
The only
Leo Dion (host): catch is they get 15%.
You know, I don't know if you, you wanna use that, registry?
Sorry, I had to make that joke.
15%.
I had to make that
Dave Verwer (guest): joke.
15% of all the, of all the revenue goes to open sources and actually got brought
Leo Dion (host): somebody, one of the, one of the owners is just
like, oh yeah, that's a good idea.
Without any knowledge of, you know, that's open sources.
Dave Verwer (guest): It's a little bit different to the app store where where it
is a, a, a platform based on transactions.
It open source software is, is a little different that .So what, how,
where does that leave the package?
. good news is we are going to fully support package registries as soon as they
become a thing that's out in the wild.
So we will.
Obviously you have to, and the Swift package manager will also
continue to support, get your URLs.
I I can imagine it's going to support get your URLs probably forever.
Yeah, hopefully.
Leo Dion (host): I, just sort of segue question, but like, one of the things I've
found with a lot of other registries out there, well not registries, but other.
Languages, especially npm and know Cocoapods had run into this too, is like,
anytime you have one central registry, there's always like room for trouble.
And what I've liked about Swift is that it relies, and you probably like this
too, but like it relies on get repo URLs, whereas's so it sounds to me like what
what you're saying is like a registry.
You can add that as a dependency with like, A qualifier saying that
it's this specific package in this registry as opposed to the github url.
Is that correct?
Okay.
Yes.
Okay.
Dave Verwer (guest): And, then you, it'll pull in so the, the, they'll
still, we will, we will still use.
Get repositories to store our source code, but they'll, there will now
be this kind of intermediate step.
And how that all falls out is probably, we're probably a little bit premature
talking about that now because we, we, we haven't seen what they're gonna develop.
the package index.
Is going to be a centralized place because at some point if you are
gonna deal with discovery and search for multiple package registries, you
need to have one place where all of those registries can be searched.
Otherwise, you'd need to go.
If you were looking for a package, you'd need to go to each registry.
Not that searches even a part of a registry.
It's not part of the, the registry spec.
It search is search is not there.
, we'll still need something to gather metadata from.
Get your URLs and package registries, bring it all together and make it
searchable and allow packages to be discovered matter where they're hosted.
Awesome.
And that's what we'll
Leo Dion (host): take down.
That's awesome.
Yeah.
haven't dived into, but now I think I might dive into
registries on how they work.
like that I think will increase obviously your audience.
I assume your audience continues to grow.
You can see that, right?
people are using swift packages more and more.
definitely.
Dave Verwer (guest): Yeah, we, we've seen consistent and, and
steady ever since we launched.
Really.
And in fact, I was just looking at something earlier today, and
our traffic has increased by over 50% since August last year.
So we are growing, we're, we're, we are growing really nicely, which is great.
What, and that, I mean, that I think proves two things.
It proves that what we're producing is useful, but it also.
I think mirrors the adoption of sif Package Manager.
I think as we go through you CocoaPods was, and still is totally dominant
in terms of and macOS projects.
It was just, that was the way that you imported Carthridge was there as
well, but I, Carthage never really got the adoption that Cocoa Pods did.
And, I think we can see clearly.
Now that package manager is, is gaining huge amounts of adoption.
CocoaPods is gonna be around for a very, very long time because
there are lots of projects that will not upgrade until there's a
reason that they have to upgrade.
Leo Dion (host): Right, right.
You still have objective C stuff out that will be different there.
If you have anything that's subjective based course, like
you have to be on Cocoa Pods.
That's it.
There's nothing out there.
, Dave Verwer (guest): but I think we're certainly seeing package
manager take more share of that market
Leo Dion (host): Anything else you want to talk about that's coming out this year?
Dave Verwer (guest): think we could talk a little bit about maybe what yeah, yeah.
Let's talk a little bit about what we're planning to do over this.
Yeah.
, Leo Dion (host): go ahead.
Yeah,
, . Dave Verwer (guest): think one thing that, that, oh, sorry.
Go on.
Sorry.
Sven A. Schmidt (guest): No, I was just maybe picking up on the DocC
stuff that we talked about earlier because seen a lot of adoption.
just checked earlier, we've got 270 packages now that are
hosting documentation with us, which is, that's great.
That's more than We have, however, seen some issues with some of those packages
because they are, they generate very large dock setss and that is something that
is, that is a , a feature of, of DocC.
It produces a lot of small files and they, they add up.
one package in particular has, is, is so large and has so many files that it, it.
in our current setup, impossible to manage, to upload it within our timeouts
that we currently, can you tell me what it is In our bill system ? is swift syntax.
It's Apple Swift Syntax package.
Okay.
And it's the nature of the package.
I mean, there's, it's, it's not, it's not the authors or the packages
fold, it's just the nature Yeah.
If a package,
Leo Dion (host): I've looked at that before.
Yeah, I can, I can see where that's coming from.
. Sven A. Schmidt (guest): Yeah.
Every symbol that's so committed generates a adjacent file.
Mm-hmm.
with data in it.
It's like, like a couple hundred kilobytes.
It's not, but because every function has that, every method, every attribute,
it's, it has 117,000 files that doc set and is, is, is a gigabyte large,
which, isn't crazy, but because we need to build the package, build
the documentation, and upload it all within a maximum of 15 minutes.
that that is just not enough So we have to work around that sort of stuff.
And we're also hoping that so we are part of the documentation working group we have
flagged this, and they are, people are aware that it's, they are some problems
with the, with the size, the sheer size of Doxies doc sets tremendously.
If, if, you know, that would be one way to, to, you know,
bring the size down a bit.
But there are also things that we can do to offload that whole
process and, you know, not try and sync 117,000 files in that job.
Yeah.
But rather zip it up, upload one file elsewhere, and then have
a, have a job with more time budget, deal with the upload.
And that's what we're currently working.
, but these are the sort of things that pop up when you've shipped the feature and
then it gains adoption and you, you see all sorts of, you know, Edge cases pop
up, and that's, that's just the way it is.
So that's something we are, we are dealing with right now.
another very interesting thing that came out of our participation in a
documentation working group, and that's preview of upcoming DocC features.
So DocC shipped in its first version
That's two WWDC.
Yeah, of course.
Because in June, June last year, we shipped the, the DocC hosting.
So it was a year old But it's, it's still pretty young.
So there has have been lots of things that have been added since.
most recently there was a, a feature by Sophia Rodriguez which is a, a
search popup, which is really nice.
So if you are on a DocC Works set, you can I think it's just forward
slash you know, Command popups you get when you have Launch bar or Raycast.
So you have a dedicated search that you can bring up and then you
just start, you know, type search.
As you type, you get results.
You know, that's awesome.
Which is really great.
Yeah.
And we've, so we've set up a preview system of that
feature with a preview of DocC.
This is like a nightly five eight chain where we've, where we're, we are
previewing that and we hope, we hope to get this rolled out potentially before
five, eight chips in, I guess the spring.
Right.
that's the sort of stuff we're working towards and hoping.
Do you able to pull forward so people can, can have this feature
you on the Swift package index as as.
Leo Dion (host): Are you doing anything with like quality or maintenance scores?
Because that's a big thing.
I use Shields io and I like to post those badges in my repos showing how
good, good of a developer what is it, a well-behaved developer I am when
it comes to maintenance and quality
Dave Verwer (guest): scores.
Well, I can vouch for how, for how Well that, yeah, yeah, yeah.
Thank you.
Leo Dion (host): you wanna do something like that with Swift package Index?
Dave Verwer (guest): It certainly.
So I, I think one thing that is on my mind at the moment is that we
built the Swift package Index where it shows metadata and we built that.
, you know, a couple of years ago now, and then we decided to
tackle compatibility systems.
So we ended up going deep into this, this huge build system that we have
hanging off the back of the packaging.
Then index and it, it feeds compatibility results back.
And then we added to that with documentation generation.
, we've not, I mean, there's been various improvements of course,
but I, I, I'm conscious that we haven't done much on the actual core
package page for a little while now.
for example, we did some initial work on dependencies.
And we are pulling in resolved dependencies for every package and
we have that stored in the database.
we're not really doing a lot with that data.
we also, that data kind of comes with some tricky things.
Like for example, if.
test target declares a dependency, and we are maybe showing how
many dependencies a package has.
Do people want to know about test dependencies?
I think most people are primarily concerned with what dependencies a package
has that might ev eventually end up
Leo Dion (host): in their application products, executables, et cetera.
Yeah.
Dave Verwer (guest): And so there's a lot work to be done there on.
A, an, an intelligent display of how those dependencies actually affect
what you bring in with a package.
Okay?
We, we certainly shouldn't just ignore test dependencies.
That's not the way to do it.
But to, to, to represent that data in a readable and a, a, form that elects
people, make intelligent decisions.
what packages to use.
So that's something that, that think we're, we're looking to
get back to in the next year.
as you say, you, you know, there's a lot of these metrics that we are
already pulling into the package page could be determined to give some
kind quick glanceable rating of how well maintained or how, you know,
what, what's the, you know, how.
It's, it's, I don't really wanna say how good is this package, but because
it's not about this, but, but certainly some kind of glanceable metadata that
says, yeah, this is something that I wanna look a little deeper into.
it, that can come from all sorts of stuff.
Some of that will come from dependencies.
Some of it will come from.
The, the metadata we have about how well maintained packages in terms of our
issues being closed, our pull requests being merged, that kind of thing.
Yeah.
Does it have a read me file?
Does it have documentation?
Does it have tests?
There's a lot of things that we could expose in, first of all, on
the package page, and then maybe infer from some kind of quality
score or something like that from it.
It was something that Cocoapods did.
They don't do it anymore.
But they Cocoapods had a quality index.
And you, you could go to a, a little webpage for every mm-hmm.
that had like five or six different metrics of like, this is
something that it is doing right.
This is something that's that, that it maybe could improve And it was
things like how complex is the readme?
There was a package that, there was tool they used Try and get a handle
on how good a readme There's lots of possibilities for that kind of
be To be used by the package index.
So some something I'm interested
Leo Dion (host): in working on.
Yeah, I mean, one thing I always look at, and I've done this actually
more with npm, is just like, has this package actually been maintained
in the last, like two years?
Cuz that's usually a decent indication that it's either a
dead product or, or still active.
But it sounds like, yeah, you want to add some of those
metrics, which, which is awesome.
I'm looking forward to that.
. Dave Verwer (guest): And the other thing that we won't be able to is a lot of
these package NPM and and others because they are both the registry and the index
in most cases, like mpm for example, is, is, is vans, the, the, the package
data or the, the dependency data and it is the search engine, so it can use how
many people are downloading or using.
Right?
Right.
Dependency as some kind of indicator.
Now we absolutely don't have that because we have no, we have no way
to to who's using each package.
We don't even know really, you know, all we can see is, is on how many
visits each package page has, basically.
Right, right.
So we have none of that information.
Maybe that's something that registry eventually, I don't think it
supports it in its current form, but maybe it could potentially
start gathering some of that data.
But again, there's good arguments for it, not I agree.
Yeah.
Leo Dion (host): I'm not sure that that is useful necessarily but it
sounds like you have already a lot going on with Swift Package Index.
It sounds like you have a lot coming this Yeah.
people.
Have I been to Swift Package Index and you have a sponsorship as well?
Definitely take some time and some support financially and maybe put in a pull
request if there's something you can fix.
. Dave Verwer (guest): Well, there are many ways to support it.
Yeah.
And, and, and before I go into this, I think like to say, or we
would like to say thank you so much for the little advertisement
you've been, been running for us.
That's, that's very much appreciated.
And we, yes, we do have a GitHub.
Sponsors account, which you can find on any of the GitHub repositories.
Or on the package index page itself, there's a little, a little pink heart
in the top menu bar that you can click to go to our GitHub sponsors.
also there's many ways to help.
You could also like many people have done this here as you could
involved in the development, we have a discord server where we meet
with all of the contributors of.
so far and other people who are just generally interested in what
we're doing on the package index.
So the link to that discord is on our.
Main repositories.
We, and
Leo Dion (host): we'll put that as well in the show notes.
Of course notes.
Thank you so much Dave.
It's fun for coming on the show.
I really appreciate it.
It's glad to have you both on.
Thank you so much for your work on Swift package decks.
dunno what else I have to say, but yeah, it's been wonderful
Dave Verwer (guest): chatting.
Thanks so much for having us.
I'm, yeah, thanks having us on show again.
Yeah.
Leo Dion (host): As a double actor, where could people find both of you online?
Dave Verwer (guest): Best place to find me is daveverwer.com.
ev Links to everything from there
Sven A. Schmidt (guest): I am on Mastodon these days.
finestructure@mastodon, oh God.
Master on social.
Yes.
. . Don't even know the domain.
It's the, the main side, but fine structure is, is the username
fine as in, you know, fine and structure if I any structure.
Leo Dion (host): Well, thank you so much.
People can find me on Twitter at Leo g Dion.
Maam, I'm Leo g Dion, c dot i m if you are listening to this
podcast, please take some time.
To post a review and also subscribe to the Swift Package Index Podcast as well.
And if you're watching this on YouTube, subscribe and like, thank
you so much for joining me and I look forward to talking to you again.
Bye everyone.