Join Dave and Sven, the creators of the Swift Package Index open-source project, as they talk about progress on the project and discuss a new set of community package recommendations every episode.
It's kind of funny, it made researching the packages for this week's episode of the podcast
a little bit more difficult than usual. So what we're talking about is that you may have noticed
if you visited the package index over the last couple of days that readme files are not rendering
on package pages and we had a little bit of technical debt that hit us squarely
in the front of our faces this week. A little bit of background to this is when we implemented the
readme file rendering inside package index, we originally did it with a Markdown processor,
a GitHub-flavored Markdown processor, and so grabbed the raw readme file, rendered it out to
some HTML, and then did some fixes to fix up image paths and things like that.
And then over time there were edge cases and edge cases and edge cases of things
that didn't quite render correctly that we decided we need to have GitHub's
rendering of their readme files really. And so the way that we approached that,
because it was the only way that we thought we could approach it at the time,
was to grab the rendered HTML page that GitHub would serve for the readme, find
the readme file, and then render that through onto our site. And at the point
that we deployed, or just about to deploy that feature out to the site, one of our
contributors, James Sherlock, mentioned that there was an API for this and this
is something that was news to us that we didn't know about. We did know that there
was a readme API, but what we didn't know is that if you specify a
specific accepts header and tell it that you want a rendered version of the
readme information, then it will give you this rendered HTML page in the
correct way, not by scraping, doing it through the API, which is what we should
have been doing. But this feature was implemented. Changing it to the API
version would have been quite a lot of work and it was one of those things that
we put in as an issue. In fact I looked at the issue the other day and it's
from 2021, so a couple of years ago now. And yeah we stuck it in as an
issue and promptly kind of forgot all about it. Right up until GitHub changed
their readme page design, which is always the problem if you scrape anything then
you're vulnerable to if the original page design changes or something like
that then you you have a problem. Now we had some code in there so it fell back
nicely and displayed a link to the readme instead of the actual readme but I
must admit I desperately miss the readme files because it is a real pain to
have to click through every readme file. Yeah there's no there's no worse reminder.
I mean, it was really, it was really harsh.
I mean, I look at package pages often, but, um, I'm not looking for the readmes
every time, but this, you know, this last couple of days, as I, especially as I was
working on it anyway, to, to fix this, every time I looked at a package, I was
like, Oh, God damn it, what should I be doing?
Should I be preparing for the podcast or fixing this?
So it's kind of interesting that, um, when I, when I first noticed this, I noticed
that almost immediately that GitHub must have deployed this change because I was
researching packages for iOS Dev Weekly. It was on Thursday last week that I
first noticed this problem and it was instantly intensely annoying to me that
I couldn't read the reading files on the index and I talked with Sven about it a
little bit and you kind of said, "Ah, it's okay. Well, it's okay for a few days."
And I thought to myself, "This is... I'm not sure it is okay for a few days. It's
Really annoying.
So I'm glad we shared the pain today.
- We certainly did, yeah.
I think part of the difficulty with the readme's
in particular is that it's one of those things
where it's very easy to assume,
well, this is always going to be a readme markdown file,
right, but readme's actually don't have to be.
They can be in a number of different formats.
So it was really a bit tricky to find out where,
This is actually nice that they have a fully rendered HTML version of it that you can get,
because that funnels all the readme formats through a single output format that we can then
tap into. Because I remember, I vaguely recall looking into this at some point and stopping,
because my assumption that it would be a markdown file that we could easily ingest and embed,
because, you know, getting that and adopting it is easy. We're handling markdown already. But then
when I saw, well, actually, you know, it could be different formats. That's when it all stopped and
became more complicated, as it often does.
Yeah. And of course, if we, you know, we still have a plan. It's not a plan that we're working
on actively, but we still have a plan to support other source code providers with the package index.
and certainly GitLab would be a good contender for the second source control provider to support.
But every little reach into the GitHub API like this ties us more closely, so it makes that job
a little harder. So I'm always conscious of that, but this definitely needs fixing.
Yeah, I mean there's to hoping that GitLab, which do have a quite extensive API,
we're using them in our build system so we have some experience with the API.
And my hope is that everything will be available, but you never know, right?
Yes.
There's going to be little bits and pieces that are going to be different or not there.
And if, when the time comes, it'll be interesting how that'll pan out.
Absolutely.
Right, so what other news do we have? I think worth mentioning is that we've crossed 6,000 packages
a couple of weeks ago, I think just after we recorded the last episode.
I actually managed to catch a screenshot of it when it was exactly at 6,000 as well.
We didn't have to edit the package list down to 6,000 to make that happen.
No, it was pure coincidence that I managed to just glance at it because I knew that it was
imminent and so I was checking it every every day or so and I happened to catch
it when it was exactly 6,000 packages. Well there's always our left pad package
to deal with off by one arrows right that we need. Yeah and that's a that's a
huge milestone I mean 6,000 packages it's it's not an enormous number by some
package managers and I'm looking at NPM when I say that but it's certainly not
not an insignificant number and it's something that,
this is 6,000 kind of manually curated packages.
This is 6,000 packages that people have said,
this is something that I would like people to be aware of.
- Artisanal packages, yes, you get them here.
- Artisanal packages.
- In Swift's defense with Codable,
that nerfed probably around another 20,000 packages
that our JSON passes out of existence, so.
(laughing)
- I think there's a few of those still in there, yeah.
- Those 6K are actually worth a lot.
(laughing)
In fact, I did see one, a new JSON package come through this week actually.
There have been a couple, yes, but they do actually earn their existence by adding a
couple of extra bits.
We talked about one, I think last time we recorded or the time before dealing with those
little bits where Codable is falling short a bit for the moment or just in general.
I do have one other little bit of news
that we might want to talk about.
And that was a post on the Community Showcase forum
on the Swift forums, which just actually,
before we talk about the specific post,
I would like to recommend that people check out
that Community Showcase forum.
It doesn't get a lot of new posts,
maybe two or three a week, something like that.
But I get a huge amount of value out of reading
the posts in that forum.
It's a great place to learn about what's going on
in the community.
And I noticed it was actually the week before last,
but there's a post by Neil Jones titled Swift on RISC-V
or RISC-V, which I have been reading the Wikipedia article
and apparently it is pronounced RISC-V.
Now I didn't know a lot about RISC-V
and all I know now is the contents of the Wikipedia article.
So I'm far from an expert on it.
But it's an open standard instruction set,
a reduced instruction set that is being implemented
into hardware and things like that.
It's been around for quite a while.
And Neil is working on a project
to bring RISC-V support to Swift.
And in his own words, the RISC-V ecosystem
growing at a phenomenal rate and 2023 will see the introduction of many new exciting and powerful
devices. So at first I thought, well, is this kind of some embedded, you know, instructions that might
suit an embedded computing or something like that? But it's the whole range of things. It's from
small boards up to really large workstations and things like that. So to continue our talks about
platform support, we may, it's looking like we may need an enormous list of platforms at some point
if this comes to fruition. Oh dear me.
It's just the space on the matrix. It's also, I mean, we just need that stuff.
You know, we need testing for that. It's interesting. I mean, it's great to see that
Swift is branching out like that. Yeah.
What's the vendor? Is that like vendor neutral? Are there specific vendors, multiple vendors that
implement machines?
It's all I know is that it's an open standard and, um, and I believe most of
the work on it is open source as well.
So I, I don't think I didn't see, um, a vendor named when I was reading about it.
Well, I think Apple should just be sending us any, any hardware platform that uses
Swift for compatibility testing.
That should obviously include
He tries to angle for a vision pro headset.
Well, that was my, I wouldn't say no to that, but
I don't believe that runs RISC-V.
It's clearly beneficial.
Well, but it's a platform we support, so...
It is, and I've seen a couple of readmes come through this week
that said added support for VisionOS and things like that, so...
Oh yeah, and we should probably mention, because at the time we recorded last time,
it wasn't live yet, but we have added that and it's completed the processing.
So we're all caught up.
It's all there in the matrix and you can check there if you're not sure about
your VisionOS support of your package.
Yeah.
So, and it obviously, depending on what the package is, a lot of them just
automatically get VisionOS support if they're not doing anything specific to
UIKit or AppKit or something like that.
But, but yeah, you can check your package live on the site as of now.
Exactly.
And the final bit we might want to mention is that the macro search is live now.
And the way that works is you look for a product, product colon macro will list you
all the packages that ship with macros.
Um, and we'd especially like to hear if you find this useful, if you're using this,
we have a bit of, we've had a bit of metrics to see if that's being used because
it is the way we've done it is, isn't quite straightforward.
It's a bit of a performance cost when we compute our materialized view for search.
Um, so if there's not a lot of pickup of that, we might, we might actually take
that out again at some point if we have other things to add, so it's not a, it's
not a zero cost addition, but it is there.
Uh, give it a try.
Product colon macro to see what macro packages there are.
And I think there's a, there's a fair few, I think it was a dozen or so that,
that ship macros.
- I think we've had metrics in there
for whether people are using any kind of filters.
So not obviously just searches,
but filtering searches by platform
or Swift version or whatever.
And I actually don't know how often any of those are used.
My gut feeling would be not very often.
- Not a lot, no.
- Yeah.
- Yeah, that's why I'm curious.
And they aren't used a lot,
and then the product macro search
even, you know, that's even a smaller slice of that already small chunk. And we pay an outsize
price to compute that. I mean, it's not terrible, but it's significant. So that's why I'm a bit
concerned whether that's a long-lived thing. And it's also the way we had to implement that isn't
ideal because we're actually saying macros are products, which they aren't in the strict sense
of the package manifest because they're actually targets and we're sort of faking it a bit.
it would be easy if they were proper products, but that's really an implementation detail.
What's important is to know if people find this useful or if you can think of ways to
expose this feature better. I mean, we do spell out the search extensions below the search box,
so there's a little text explaining what you can do to search in more detail,
but it doesn't seem to be used.
Maybe that's because it just doesn't come up often
or people don't know.
That would be the interesting bit to know.
Whether that's just a case of discovery
or it being really useful.
- I think I'm fairly happy with the way
that we present the discovery of that.
So obviously it's not there on that first homepage,
but as soon as you search,
it's right below the box there and there,
links so that if you even tap on one of those links it pre-fills the search with
a filter so that you can kind of get started with filtering. I think that
search, I think Google has effectively ruined us and what people do for search
is they type in some words and they hit enter. Yeah, yeah that's true. The other
thing is I tend to, especially in places I visit a lot, I tend to turn blind
to extra text on stuff. So I start overlooking small print very easily. Only the first time
on a page and I'm trying to find my way around is when I look at stuff in more detail. And
then once I sort of know where stuff is on a page, I really only zoom into the bits that
I actually need and become blind to all the stuff around. It becomes ornamental to a degree.
It's fascinating how that works.
- Absolutely right.
In fact, I was talking to somebody
about their app the other day
and they'd sent me a test flight link
and I had used the app and pointed out something
that I got stuck with when I was using their app.
And they pointed out to me that it's written right there
on the screen that I got stuck on.
- Oh dear, yeah.
That's the way it goes, isn't it?
- And that's it.
It's just a fact of software is that we,
it's not a software problem, it's a human problem.
We overlook things that are not directly pointed out to us.
But I do have a solution.
I think there is still support in most browsers
for the marquee tag,
so I think we're gonna use the marquee tag.
- The marquee tag?
What is that?
- Oh, the marquee tag was a tag that scrolled text
across the screen. - Oh, right, in a ticker.
- Yeah, that's right. - Right, yes.
- So look for that coming soon.
back to the 90s, excellent.
We might as well add a construction site
little icon to it.
- Exactly, exactly.
Well, it is always under construction, you are correct.
- It kinda is, isn't it?
Right, does that, that wraps up the news section, doesn't it?
- I think it does, yeah.
Should we do some package recommendations?
- Let's do the packages.
What do we have, do you wanna kick us off?
- Sure thing.
So my first package this week is by Sindre Sohus,
and it's a package called doc progress. It's actually been around for five years now, but I
only came across it this week and it's a kind of niche. I think both of my packages this week are
kind of niche, but it's kind of niche in that it's for macOS applications only and it's for
macOS applications that want to show progress of something when the window might not be visible.
So there are many different ways to show progress with this package.
And it rendered takes advantage of the fact that you can render your icon in a
Mac OS app, completely custom code. You know,
you don't have to have an actual icon file at all.
You can just draw into a context and that becomes the icon.
And so what this package does is it has a kind of the first,
the basic one is has a progress bar across the bottom of the icon that fills up
over whatever progress that you're sending it,
or there's one that outlines the icon with a squircle
showing progress in that outline.
The circular ones, there's badges,
there's badges with progress, there's badges with pie charts,
there's lots of different options for showing something.
And it's the kind of thing that is not going to be useful
when someone's actually looking at your application,
but for really long running processes
where people will actually switch away from your app,
this felt like a great way to keep people updated
on whatever it is that your app is doing.
- Nice, yeah, I'm actually staring at one of these right now
because my application folder has a progress bar
underneath it, I guess some app update died.
It's been like this for days now.
- Well, certainly that one might need a reboot.
That's how we fix things, isn't it?
- Yes, exactly, yeah.
But for ones that are not terminally stuck,
this feels like a nice solution.
- Nice, when you said doc progress,
I first thought, oh nice, this is going to tell you
how far along you are with documenting your project.
And then you said five years,
I was like, well, maybe it's something different.
- That would be docc progress, yeah.
- Right, my first package is one
that people may have heard about
because it's a new package by Apple.
It also showed up on the Swift blog and it's called Swift HTTP Types by Apple.
It's a package that collects foundational HTTP types like HTTP request, HTTP response,
and it's for use in client and server packages.
And that's really nice because if you're using like different ways of interacting
with HTTP, you've probably come across quite a number of different implementation of those.
And the goal here is to unify that a bit and use that package as the underlying package
in Swift Neo and Foundation, for example, to make sure you have only one version of
these around and, you know, sort of can use that one implementation and knowing how that
works because they often have like slightly different ways of setting headers, for example,
and stuff like that.
So that's really nice.
And I guess also, you know, it's going to open up to be used in other server-side packages
to adopt this either directly or automatically via Swift Neo.
It's really nice to see more and more of these foundational packages pop up and as individual
packages, you know, not necessarily tied into foundation itself, but they are being there
on their own.
And of course, this is not just interesting
for server-side packages.
Clients also need to deal with this
when they interface with server APIs, for instance.
I just mentioned setting up headers and that sort of stuff.
So it's quite a package with reach, I'd say.
And it's good to see it, good to have it.
- So I'm ashamed to say that I did see this blog post,
but I actually haven't read it yet.
Is this the kind of package
that is going to be purely for use with server-side Swift
and Swift Neo based projects?
Or is it also something that you would use outside of that?
- No, I think you'd use it outside.
I mean, this is going to be the underpinning types
for foundations URL requests.
So when you construct a URL request,
you'll be using eventually, that's not the case right now,
but it will, that's one of the goals is to have that
as the HTTP currency type for URL requests.
So when you construct a URL request,
which you do when you use URL session or allow more fire,
you'll be using this package and it'll be the same type
that you also use on the backend to drive that.
So that's quite nice.
- And so the package itself is just a representation
of those request types.
- Yes, exactly.
Which will then be adopted in those other packages
that I mentioned, yeah.
- That's great.
Well, that's good to see.
And it's good to see that things that,
'cause obviously there's the server side of Swift,
which is primarily open source
or entirely open source maybe.
And then there's the kind of Apple side of Swift,
which is AppKit and UIKit and SwiftUI,
which is primarily closed source.
And of course there are some exceptions
like the new foundation library is completely open source
and that underpins everything in those platforms too.
but it's nice to see more things become open source
that are gonna be used in those platforms.
- Yeah, absolutely, yeah.
- My next package is a package by Darren Ford.
And it's a package that I have confidence
that I will probably never,
no, I'm not gonna say confidence then probably,
I have confidence that I'll never use this package.
But what I really liked about this package
is how I could see the rabbit hole
that Darren went down when implementing this.
(both laughing)
So it's a package called Color Palette Codable,
and it is a codable implementation of color palette files.
The original one is an ASE file,
which is an Adobe Swatch Exchange file,
which is, if you read the read me,
This is actually the file that Darren needed to decode.
So he obviously had a requirement to read one of these files.
And just to be clear what's in these files,
it's a list of colors, names for those colors,
in the, get formed into a palette of colors.
So it was a group of colors.
So he needed the Adobe Swatch Exchange file
and implemented something and thought,
well, maybe I'll make it a package.
And then that, he said that this then extended to ACO,
which is an Adobe Photoshop color swatch file.
And then it expanded to other types.
And there's a smiley face in the read me
because he's implemented support for,
I think 12 or 13 different color palettes implementations,
including PaintShop Pro files,
which was a real blast from the past.
Do you remember PaintShop Pro on Windows?
- Oh, I don't know.
I used Windows a bit, but I switched to Macs in '91,
so I missed out on a lot there.
- So I was Windows up until the early 2000s,
and PaintShop Pro was one of those applications
that was absolutely essential to have on your Windows PC.
Microsoft RIFF, R-I-F-F, Palette Files,
Sketch Palette, which I've never even heard of,
corral draw, every type of color. And I'm confident that even Darren hasn't used in
anger most of these. So I thought as I was reading this read me file, I saw the rabbit
hole you went down Darren and I hope he came out the other side of it.
Nice. It's one of those, if I understand correctly, part of that was to make colors
codable or... Yeah well color palettes codable so it's the combination so as some of these
palettes there's actually a it's a fantastic readme file for what the package does. There's
a table about three quarters of the way down the readme file of all the different files that are
supported. What are they binary? Are they text? Are they JSON text or whatever? Do they support
Do they support named colors?
Do they support named palettes?
Do they group colors?
Do they have color type support?
Do they support color spaces?
So there's obviously a whole load of information
that you can use in these palettes,
but no, it's more than just a single color.
- Right.
The reason I'm asking is because I saw this pop up
in the last week.
I think someone went to the labs at WWDC and asked,
Well, why isn't SwiftUI color codable so they can save color choices to the,
what you call it, the defaults, user defaults.
And the answer is quite interesting because, you know, naively you think,
well, how hard can it be, right?
It's just three numbers, right?
Red, blue, green.
But the answer was, well, actually, you know, it, what actually displays
depends on your display, right?
calibration, that sort of stuff. So it isn't dark and light mode.
So a color red is slightly different depending on whether you have it in dark
and light mode. I believe there's a variant that SwiftUI gives you
depending on context. So a color is context dependent. So when you save
it, how much of that context do you pull in? You know, it's not just that. It's
one of these great examples of a thing that seems to be so simple at first glance, where you'd say,
"Well, why isn't this just codable?" And then, "Oh dear me. Yeah, well."
- If you want to learn more about this, Mark Edwards has a two-part blog post on color spaces,
which is well worth the five to ten minutes it would take to read it.
Nice. We'll add that to the show notes as well. I find stuff like that so interesting.
And I think that's one of those places where sort of experience shows a bit because you become
really a bit more resistant to this. Well, just do this because you've been there a number of times
where you thought, well, of course, this is going to be easy, right? And then like days later,
you think, oh God, what did I get myself into? And I love this sort of stuff where
there's a very, you think, obvious first stab at a thing, and then it opens up this whole world of
complexity that is so neatly hidden. And we sort of get tricked into this a bit as well with our
systems that we like to use, right?
I mean, it's one of these arguments, Linux or Unix versus the Mac, right?
The Mac sort of tends to present itself as simple, but that's, that's mostly
because the complexity is hidden away or choices have been made on your
behalf to deal with the complexity, but it's still there.
Other systems make other choices and, and still have that, expose
that complexity to you.
I mean, it's a trade-off, right?
Because some people prefer to deal
with the complexity themselves
and not be handheld in that way.
And I find that really interesting how, you know,
that's just a fact of life, right?
Complexity is just there.
It's a matter of how you deal with it, really.
- Yeah, yeah.
And color, there's plenty of complexity in color.
- Oh yeah.
So my next package is called Aoxiang
by the user with a handle isaced, I-S-A-C-E-D.
And Aoxiang is a Chinese term,
apparently translates into saw or hover.
This is a small HTTP web server package.
And I think it's interesting, not just in that space
where you probably use something more heavyweight
than this package, but it's interesting as a prototype
or testing tool.
Imagine you're writing an iOS app
And you need to stand up a service that you can connect to.
For instance, if you're testing on device
and you have no other or better means of injecting
JSON's codable structs, for instance, for connecting to.
And you can stand this up as an endpoint you control.
So typically you might use a Python package, for instance,
which has a couple of tools like this
where you can easily do this.
Or I think Node as well,
Node Express is a term that comes to mind.
I don't have a lot of experience with Node.
But this is sort of the Swift equivalent to that.
And that's really nice.
I tried this, you can actually run it in the playground.
So you can bring this up.
It has a nice API, a closure-based syntax.
So you can write little root handlers.
So it's very easy to stick something in.
And then obviously Swift, as an example,
is very easy because you can use the same
codable structs there that you use in your client app.
That makes it then very easy to stand this up
deploy. It has no dependencies as well, that's nice and that makes it easy to use in in like
scripts or a playground. So that's a package called AUGS-YANG by Aisa Khed. And I should also
mention at this point there was another package that does a very similar thing which is called
Swifter by Damian Kołakowski, which is a package that has been around for quite a while,
that's nine years in development, 99 contributors, which is also a package like this that has no
dependencies and works in a playground or in scripts. So for instance, these are packages
you can use with Swift SH, the Swift runner by MaxHole. So two nice packages for standing up a
little web service for testing and prototyping. ALFIE: I think most languages have something like
this. There's a one-liner that I use with the Ruby executable that will, from one line on the command
line, take a started HTTP server and point it at a set of files in a directory. Now this is obviously
more than that because you can actually define not just serve files but define actual code to execute
when these requests come in. And it's also worth noting that it supports async/await as well,
which is great, so you can fire off async commands on your requests as well.
Yeah, exactly. Yeah, be a nice little package. That's great, and I think that
pretty much brings us to the end of another episode. Thank you so much for
listening everybody, and if there is any feedback or any packages that you
would like us to take a look at on the show, please do drop us a line. The
easiest way to get in contact with us is through our Discord server, which if you
go to the readme file of the main Swift package index server repository, you'll
find a link to the Discord there. So we'd love to have you take part in that. And
we will be back in a couple of weeks with another episode.
Yep, see you in two weeks. Thank you. Bye bye.
All right. Bye bye.