038: Finally, found a good use case for Go's plugin

I've restarted active development on my open source Go backend server API StaticBackend. For a long time I wanted to make its CLI size smaller, and I decided to use Go's plugin package to extract a functionality that used a dependency that was accounting for more than 50% of its 170 MB. Go plugin were the solution I decided to use for this and I explain the problem and the solution in this episode.

Links:

• StaticBackend on GitHub
• go-size-analyzer

As always it's appreciated if you can talk about the pod and share. You may also purchase my course(s) if you want to contribute with money, there's a 50% off coupon with those links: Build SaaS apps in Go and Build a Google Analytics in Go.

As always 

Creators & Guests

Host

Dominic St-Pierre

Go system builder - entrepreneur. I've been writing software systems since 2001. I love SaaS, building since 2008.

What is go podcast()?

15 minutes news, tips, and tricks on the Go programming language.

Dominic: 00:01

Hey there. I'm Dominic Senthil and you're listening to Go Fast. Today I'm flying solo and I talk about the usage of plugins in static back end. Hello, Gophers. It will be another solo episode this week.

Dominic: 00:23

I wanted to talk a little bit about something I worked, recently in static back end. So first of all, I restarted active development on on static back end. So if you don't really know static back end is a project an open source project I started in early 2020 or, you know, late 2019, basically. And at first I thought it was going to be a commercial product and, changed my mind at some point because I was not getting traction at all in that in that way. So I open sourced it and still wanted to see if, I could, I could do some manage hosting for the service and whatnot.

Dominic: 01:10

But even that, did not really worked. So you know, it led me to to start looking for looking for founding and whatnot. Because you know, in this space there there is there is you know, well founded, competitors. It's it's pretty hard to compete sometimes when a competitor got, you know, you know, 130,000,000, in founding and whatnot. So and by the way, I'm talking about Supabase.

Dominic: 01:40

I I I had a couple of, of talk with, with the the founder, actually, and, on email and whatnot. Yeah. He he knew about, about static back end. He and he even told told me that the how I implemented the security is is, you know, really it's it's better than than what they have. So those are his words, not mine.

Dominic: 02:06

But, yeah. After failing at also trying to find some founding, you know, I I was I was still reaching, the version up and still maintaining the project. There there has been some really nice contributions from, you know, from, the community as well. But, you know, at some point, it was, it was it was getting very difficult with all those failures in mind and whatnot. And and even the, you know, even the usage is itself.

Dominic: 02:38

So even even though there has been some some small contributions here and there and and, you know, the occasional people talking to me and whatnot. I mean, I I still don't really know exactly if, you know, if anyone were using it. I was using it. But, but yeah. I mean I mean after 1 year of inactivity per se, I started to miss, you know, and and really, really miss working on this project.

Dominic: 03:09

So I really I really like it and I decided to restart, active development. And this time, you know, there's no commercial, there's no funding, there's there's no any path. So it's it's a 100% free and open source project. So I'm I'm going to even change the website all, you know, and it will it will just be the documentation. So you, you know, it will go to static backend.com and boom, it did, you know, there there's there's no there's no mention of any anything that could let people think that there is, there is a commercial, you know, path or or avenue behind this.

Dominic: 03:51

It's it's it's directly the documentation that will load. It's not live yet, I'm working on it. I'm I'm working on rewriting all the documentation, but long story short, one aspect that I always wanted to improve with static back end. So basically, what it is what it is exactly, it it's a back end API. So imagine that you want to build a a new web application for example and you don't really know if your idea is going to work or anything like that.

Dominic: 04:29

So, you don't really want to put too many time trying to architect the best web application, you just want something out there. So this, this project is helping with that because there's a lot of building blocks that can be used right away. So it's it's a back end API written in Go, of course, that offers things like authentication, authorization, you know, if a full CRUD and queries and full text search cap capabilities out of the box. There is file upload, you know, file storage. There's, there's a way to to do event driven.

Dominic: 05:19

So you are building a monolith in a sense. So if you think of a way to build a monolith, but without having to write one line of backend API, This this is it. And I know. I mean, I I'm also, I'm a, you know, a back in API first person. You know, I this this is where I like to be.

Dominic: 05:42

But sometimes, it's also true that in my life I want to put ideas out there. I don't know if people are going to like them or not. And when I need to do that, I don't want to handle, you know, all the middlewares myself. I don't want to rewrite the code that I've been rewriting for the last 20 years and, you know, of my careers. So if I can skip writing the authentication for this specific idea that I just want to know if there's going to to have some traction after, I don't know, 1 year 1 year, 2 years, then static back end is is really where, you know, where where it can be extremely helpful.

Dominic: 06:36

I built it for myself. So this is how I used it. So I I launched basically 2 or maybe 3 SaaS in the last 4 years with with this. So again, I mean in terms of of local development, you basically download a CLI which have a fully fully functional back end server without you having to, you know, to bring up some some Postgres or any any kind of database or Redis and things like that. It's all it's all in memory when you're developing.

Dominic: 07:16

So you just you just download the CLI, you run back end server and and boom. You have your development server with everything that the production server is also offering. So all the real time capabilities, all the full text search. So I mean it's it's it's interesting. It's different.

Dominic: 07:36

It's it's a little bit similar to what you know things like Django and Rails could be on on on those on those languages, but for Go, but but but not really. And and it's also it's also very different Because it lets you it lets you know that your back end is in Go, but you don't really need to to write any any kind of code. And, and if you want, you you can still you can still have another small, web server on the side with with Go and use the the the Go client library as well. So this is this is totally possible and I I've done that as well. So the idea of static back end is that you can you can have it manage multiple of your ideas.

Dominic: 08:28

So, all the the the common building blocks that it that it's handling, all the component that it's handling and you can host multiple web, you know, web application from a single instance if you will. So it's it's a multi tenant application so all of your application, you know, all of your web application will be a tenant in there. You know, they will have their their set of account and users and whatnot. So you may you may build a a Go servers that is a campaign, you know, in addition to, to what what static back end is offering. So one aspect that I wanted to, to improve for a long time is the size of this binary.

Dominic: 09:15

It it was 170 megabytes. Which is you know, it it was a go binary. So it was pretty huge. And there was you know, there is multiple reason for that. And, this is, you know, this is a topic of of the of of the episode of today because how do you how do you trim fat if you will on on the Go binary?

Dominic: 09:39

I mean we it's not like we have a lot of option here. Obviously it's certainly not your code. So static back end is about 18,000 line of code. So it's certainly not that that is taking the the most, you know, disk space on this on this binary, of course. So it has to be your dependencies, but which one?

Dominic: 10:03

So I stumble, on the tool called Go Size Analyzer, which was which was interesting. I mean, it kind of tells you the percentage of space that a dependency is taking and I I, you know, I add my I was pretty sure I was knowing what what it, you know, what it was. But I wanted to be sure and and this tool is is great because now you you see all all of the the dependencies and turns out, you know, just just a side note, the a AWS, SDK seems to be crazy big. I mean, how how big how big, this thing needs to be? So now, when I just ran the the tool the GoSize analyzer I saw I saw that the now that I've removed my my biggest dependencies from there, the next one in line is the, IWS SDK.

Dominic: 11:05

So, I I'm just using the, the storage part. So, I mean, yeah. I will I will need to, I will need to remove that. This is my next target, but I'm diverging a little bit. So returning to this, this CLI.

Dominic: 11:20

So the the CLI, so the local development server is available via NPM as well as Go install of course. It's a it's a it's a it's a normal Go binary in fact. So, there there it's a cobra, CLI application. But one of the feature that static back end was implementing was a way for someone that is building a web web application to say, you know what? I will need to have some HTML turn into PDF at some point.

Dominic: 12:00

So that was one of the feature that I'm I'm kind of usually need needing in an application and and nobody was doing that. So I said, you know, this this is this is still a common ish use case if you think about that. But, the problem is I'm using Chrome. Dp for that. So if you are not familiar with Chrome DP, Chrome DP is a package, a good package that allows you to, you know to automate the Chromium Browser.

Dominic: 12:33

So you basically are able to load you know, to load the page and and do do lots of things, you know, you can you can scrape a little bit of data here and there. You can you can have some CSS selectors and and, you know, all sort of selectors and try to try to inspect the the DOM elements and what not. In my case, you know, I'm basically just loading some page and rendering that to PDF. So this is something that you can do. It's all offer via the Chromium.

Dominic: 13:03

So, you know, since you can do a file print, print to PDF in your browser so you are basically leveraging this functionality in a sense. The problem is the Chrome dp kind of embed its own version of, of Chromium inside or something like that. I mean, and I knew that it was not going to be as popular as, you know, CRUD operation in database. So maybe it was time to think a little bit about how can static back end starts to offer, you know, less common features without bundling them into the binary. So kind of a an opt in scenario here.

Dominic: 13:51

And you know, you're you're probably seeing me coming and and the plugin package of Go was something I wanted to to use for a very very long time. So, I'm all you know, I've always been kind of interested or I I like the fact that an application can load some pieces of functionality at runtime and, you know it's seamlessly executed in the same, you know the same address spacing in the same memory if you will space as the main process. So there there's basically not really a huge penalty for you to doing that. And and the Go, plugin package is something that is offering that for us. I've never really, you know, played around with it because I I to be completely frank, I I was not really having this, this need before.

Dominic: 14:47

But now, you know, as soon as I removed the Chrome DP Dependencies from the CLI. The CLI went from, 170 megabyte to, about 60 megabytes. So it was kind of 110, megabyte just for this. And, and you know, just, just an aside. So then my next target is probably going to be like the AWS, which is taking like 5% of the space.

Dominic: 15:12

So it's not much. It's like, you know, maybe 3 or 4 megabytes, but still, I mean, you know, 3 megabytes for, for a dependencies that you're using just a tiny bit. So that, you know, that that brings me to question now, you know, when we are building, you know, when we are building our application. So I I suppose that the compiler is is not doing any kind of removal of dependent, you know, I don't know, you know, types or or files, maybe entire files and things like that that we are not using. So it's probably it's probably bundling all all all of the, the dependencies as it is.

Dominic: 16:03

So, just seems to be interesting. Could could there be any kind of, something similar to a tree shaking in JavaScript where you are just bringing your your, you know, your required dependencies. I've never really interested myself to be frank to, to the the, you know, the deep down details of the of the Go compiler. But now now that I I've seen that, I'm a little bit interested to see. I mean, wow.

Dominic: 16:32

Could there be something, you know, in the long run? Is it is it something that they are thinking? I I don't I don't know. But, but again, continuing with my story. So once I removed this dependencies, I was like, well, yeah, you know, you know what, you know, 60 megabyte is a little bit more acceptable.

Dominic: 16:51

So it's still very big. So why is that? So I'm, you know, static backend is is is like an event based server monolith that that you can you can use. It it can be deployed on multiple instance. So, that, you know, don't, it's it's something it's something that I wanted from day 1.

Dominic: 17:13

I wanted to let people have one instance and have a great experience. And also, if they need to to scale a little bit. You know, yeah. Let's let's unpack that a little, because we need we need we need a little bit more information here, but if if, you know, if for some reason someone needs to go to 2 or 3 instances, then it's possible to do that. So there is inter inter process communication and it's not using gRPC.

Dominic: 17:47

But, but it's possible it's possible to scale, so that's why I'm I'm I'm thinking of static back end as, yes. It's kind of a prototype, something that helps you build an an MVP or v one, but it also can, reach like 20, 20,000 concurrent user easily. I mean, so so it it will not be your first problem if you have some success for sure. So even though even though it's kind of a low code, you know, I I won't say prototype builder. It's it's really something that can can help, you know, launch a v one very quickly.

Dominic: 18:26

But, it's it can also can also, be there until until your v three for for instance. So, but yeah, I mean this is, this this is that. And and and I I do I do have some other dependencies like I was saying. So there there's, there's, I don't know how to pronounce that. I think it's it should be go Goja probably.

Dominic: 18:51

So, g o g a, which is which is a JavaScript runtime, basically. And I'm having so I I always wanted to to have static back end as a like I like I was saying if there is one instance being used you can you can still publish and and subscribe to events all over the system. So, events that the developer that wants to publish and subscribe to and as well as system event. And what what are system events? Well, they could be, all all the database CRUD.

Dominic: 19:29

Well, not CRUD, but you know, like create, update and delete events. There there's a couple of built in events that that can be triggered. And when they are triggered, they can hide or call a URL. So if you want to have, you know, your own handler for that, then it's possible. You can you can just said, you know what?

Dominic: 19:55

When this happens call call this URL, but you can also run a what I call the server side function. So server side function are just JavaScript very tiny, piece of it's basically one function which is called handle and this thing is receiving the the HTTP response body. There's the HTTP headers, and there's a third parameter which for some reason I'm I'm kind of forgetting at the moment, but, you know, the the point is that you can you can build, you you can build an event system inside this this tiny server without without complication, without anything. So, you just you just create a function and you receive you know you receive the the proper parameter and and from there you you can you can just do anything that you want with the system. You can you can write data, you can read data, you can upload files and and things like that.

Dominic: 20:56

And you are inside a a super simple server side function. And yes, it's sad that it's only in JavaScript. I'm I'm currently inspecting or at least, exploring if it could be possible to to write that into into something else. Go could be extremely nice. I'm, I'm looking at maybe having a a WASM runner at some point.

Dominic: 21:22

So, you know, service side function could be just, build in in in mostly most any any kind of languages as long as they output to WebAssembly and, and the binary could be, could be executed inside the inside the runner. But, yeah, I'm not there yet. So, so this runtime is also taking some kind of place but I, you know, I don't I don't I don't I cannot I cannot do much about that. So this the you know, there there will have there will have dependencies that you can you you have to to to keep there for sure. But yeah, I mean the plugin, the the plugin package from from the go library is very simple, really nice to, to play with.

Dominic: 22:10

It seems to be only working on Linux and Mac though. So that's that might be a huge downside so may Yeah. Maybe in my case as well. I'm not sure if, you know, if if there is some, some people that that is using Windows to develop. Yeah.

Dominic: 22:34

Probably they they won't be able to load this, this plugin. So basically, when you are building a plugin you just you just pass a build mode parameter to the Go build command and you say you know what this is going to be a plugin. So create a a file that can be that can be loaded at runtime in your main program. And this this, this mechanism maybe maybe it's the it's the it's the file that gets built or I I don't know what what it's what exactly is not compatible with Windows, but, but, yeah, this won't work. You know, but anyway for for people in in a, you know, using Windows, you know, you should probably investigate to to run your, your development things in in WSL.

Dominic: 23:25

Anyway, it's very, I I think I think it's better. I I needed to switch, to Windows a couple of years ago due to, I needed to start, using a screen reader, you know, daily and whatnot. And I I was using, at first, I was using Arch Linux for a long time. And at some point, I, after 3 or 4 years, I had some issues and I, I said, you know what? I screw that.

Dominic: 23:52

I I will I will just go with my old friend DBM which is, you know, always has been very stable for me and I ran for for multiple years without any issues. In fact, this this machine is still is still up. So, kudos to to Debian, but I'm I'm still diverging the yeah. The first thing I did when I when I needed to switch to, to Windows was to to make sure, I I was going to to pass the, you know, 80 80 to 95% of my time, on WSL because, yeah, this this is, this is where I am familiar and I like to be. So, yes, the plugin, if you have, if you haven't look at that it's interesting for sure.

Dominic: 24:42

It could be it could be tedious if if you are planning to use that and you you you you you are thinking about doing a you know a lot of how can I say that? You know data passing between your main program and and the plugin. This might not be very comfortable because even even though they are they are 2 separate Go, binaries if you will. So your, you know, your main program is calling, is is directly opening this, this thing. You cannot you cannot easily share types between, between the 2.

Dominic: 25:20

So you can export types from your plugin and functions and variable and things like that. But you you will still need to, to have them define on your main program. So, I, in my case, it's not very a problem because I'm basically just passing, you know, slice of bytes. I found that it's it's it it really worked well for my use case and also was was kind of the simplest thing to do. So, basically, when I'm calling a function.

Dominic: 25:55

So in our case, if you recall the the function in question that I was trying to, to remove from the the main go binary was you know having having a URL converted to a PDF file. So, yeah, I'm using some kind of bite bite slice to to communicate. So back and forth. So basically, I'm passing the the response body directly and I'm casting that or at least the JSON unmarshal that into into something that the plugins knows about. It do it, you know, it does its thing convert to PDF and return the the the PDF bytes if you will to, to the color and and and boom.

Dominic: 26:49

That's it. I mean, it's working. It's nice. And of course, I mean, for people that wants to use this, they they will still have, you know, roughly the same amount of of, you know, of binary size because now you need to, you need to bring this plugin the binary along with with the main program binary. So, if you are going to to deploy a static back end on a server for example and you need this functionality then on your server you will have you know the back end binary as well as the plugins.

Dominic: 27:27

But since it was very easy to do, it opens, you know, it it it it opens a nice at least, possibilities for me from the future to say things like, you know what? This thing is not really common enough that it should be, it should be part of the, of the core of setting backend core, but it can certainly live into its own opt in, extension. If people want it then they just bring the plugin. If they don't want it, then they do, you know, they just don't. So, yes, I think, I think it's nice.

Dominic: 28:08

I was, I was really, really enjoying discovering this package because I wanted to to play around for a long time. I recall that I think I think in C there was something very great about, about plugins, but now it's been it's been so many so many years like 10 or 15 years that I I'm not really recalling exactly, but I I think I you know, I think this is something fairly common, or at least you you will encounter that once or twice in your career. That a plugin system is really is really what you are looking for. So, if you haven't check it out and you you need that then by all means. If you want to, you know, to contribute a little bit in in a go project, I mean, join join me with, with static backend.

Dominic: 29:02

It's, it's very friendly out there. There there's already a couple of peoples, you know, people that that did some, some contribution. I have started to, to recreate some kind of roadmap and whatnot. I mean, it's it's a it's a cool project. I think it's it's still still fairly small ish.

Dominic: 29:25

There's a lot of cleanup to, to be done, but but, you know, there's a decent amount of unit test. The the application works. I I I used it 2 or 3 times in production myself. But is it, you know, is it super friendly and things like that? No.

Dominic: 29:44

Not really, but I'm I'm trying to I'm rewriting the documentation. So I mean, yeah. If you if you are looking for, for some fun a little bit, you know, join, join me. There there's the discussion tab on GitHub. I I'm I'm trying to trying to revive this project.

Dominic: 30:02

I I really, you know, I really missed working on it. It's nice. So on that, take care. Alright. That's it for this week.

Dominic: 30:18

I would really appreciate if you can talk or share about this podcast. It's it's always helpful. Also, another way to support this show is by purchasing my course. There's always a link in the show notes. So on that, see you next week.