Toying with static analysis of HTML templates

After last episode with Templ maintainers I was really pumped to try Templ and see if it would work for me. Without spoiling too much I believe it would have been easier to start from scratch with Templ vs. trying to migrate an existing project.

This led me to try and see if I could add static analysis of my templates in my library tpl. I don't really have a PoC yet, but kind of getting close to it. If everything continue I should be able to capture errors in using of wrong field in template, like typos in field name that are caught at runtime at this moment.

Links: https://github.com/dstpierre/tpl

Also if you want to support this show, this is a 50% discount on my courses: Build SaaS apps in Go and Build a Google Analytics in Go.

Creators & Guests

Host

Dominic St-Pierre

Go system builder - entrepreneur. I've been writing software systems since 2001. I love SaaS, building since 2008.

What is go podcast()?

15 minutes news, tips, and tricks on the Go programming language.

Dominic: 00:01

Hello, gophers. So there will not be any kind of intro for this podcast. So, basically, we are September 30th, today 30, And it's the International Podcast Day apparently. And, my podcast Ooster, which is a transistor.fm is doing some kind of, special day today. So I mean, every every one of their clients that is posting an episode today might, have the chance to win a Shure SB 7, mic, which which I would would love to to get.

Dominic: 00:38

So hopefully, I will be the the lucky one or whatever. But, you know, this is not what we will be talking today. So you might have listened to the last episodes where I received the maintainer and the main, one of the main contributor of the temple tooling library. And, after after this talk, I was extremely pumped and I'm currently building a software as a service called Kiwi Polys. Basically it's, well we we have pivoted the last week basically, but I mean it's it's targeted now at Homeschooler and it's a planner.

Dominic: 01:19

There's there's a huge twist in there where there is a 2 d, game where the kids can, do their activities and whatnot with small serious game. But, suffice to say that the UI that I have at the moment was well, it it's built on the, you know, the the standard HTML templates. I have created my my own small helper library called tpl just to add some kind of structure and some parsing to, to the mix, just to make it a little bit easier to, you know, to to have some partials and reusable, you know, quote unquote component, even though they are not really component. But let's just say, if you have used, Yigoo in the past for static site generation, then it, you know, it's a little bit similar where you can have some some partial here and there in your views or in your pages. So I was extremely interested to see if I could start transitioning to Temple, because I, to be frank, after after the the the confer after the conference after the the podcast episode, I was kind of, you know, very interested to see if well, first of all, if if the tooling would work with my screen reader.

Dominic: 02:51

So that was one aspect that I was not sure about because if you are not aware, yeah, basically basically you're not really writing HTML or at least you're not really in a quote unquote HTML, the language server or at least maybe the I'm using Versus Code. And from what I understood from the interview, it seems like the, you know, Vim user might have a better experience than, Versus Code user because in, in Vim and Neovim, you can have, you can have multiple language server for a specific file type or something like that. And, it's difficult, it's difficult to do or impossible to do with Versus Code. So I I was curious to see, well, first, is this thing is going to to be working with how I'm working with HTML. Because when you're blind and I don't want to repeat myself too much, but basically when when you're blind and you're working with HTML, it's, it's non trivial to move part of your document structure here and there.

Dominic: 04:01

So I'm using the the the, you know, the collapsible feature of Versus Code a lot. Not to hide anything, but just let's say let's say I I have, I don't know, 3, 3 div container horizontally and I want to move the middle one into the left one. So one easy way for me to do that would be to collapse the left one. After that collapse the center one and just you know cut the middle one and paste that just before the left one. So when they are collapsed, first of all, there there's there's audio there's audio sound in in Versus Code that tells me that you are you are in in a in a container that is that is collapsed and it's it's doing the same.

Dominic: 04:50

So I'm I'm I'm not using, collapsible, you know, feature, like like they are supposed to be to be used, I guess. It's, you know, for me it's just to prevent the screen reader from, from having to traverse a lot of development. Because as you might expect, when you have a screen reader everything is read to you, so it's pretty difficult to, to know, especially in an HTML file, well where you are exactly. So that part worked, you know, decently well, I must I must admit. What I mean by decently, well, I I did not add have many problem.

Dominic: 05:35

It seems like the HTML edition or I mean I mean, the way that you write HTML in the temple file seems to be, you know, it seems to be pretty pretty similar to what I am having when editing, you know, quote unquote normal HTML file for example. So so that that, that part works for me. So the the setup aspect was extremely quick. The second the second, you know, piece of concerns that I was having was, you know, is is it is it going to be straightforward to, you know, to have an intermediate, steps in between having my HTML that now needs to, you know, it needs to kind of compile to some Go code and now the language server, or at least yeah. The language server is supposed to give you some kind of richer experience when you are accessing the data that you are receiving.

Dominic: 06:47

So for me, it did not work. So that was the first the, you know, the first thing that I I was I was finding it a little bit, you know, sad, if I can say that for lack of a better word. I I I don't I don't know what to say exactly, but I I I was kind of, a little bit disappointed about that. So to me, maybe it's my, maybe it's it's my slow computer. I do have I do have a very old machine and and I know that screen reader is eating a lot of of resources and whatnot.

Dominic: 07:22

I have like 16 gig of RAM, but, the processor are are not great at all. So so to me, I I was I was, you know, inside a a component in in Temple, and I I was not having any kind of suggestion. Let's say let's say we are receiving a struck, like, I don't know, a user struck or whatever, and you have a a name field and an email field. So I was not I was not having anything there. During, you know, when I was writing my, my markup.

Dominic: 07:59

So that was a downside and I was like well, okay, you know, that's not a huge thing as long as, I I guess, there there will be some some compilation error at some point that that will tell me, you know, you have you have tried to use a field that is not, written correctly or things like that. So I started to move my my my pages and and my not not my component, like like I was saying, in in my in my library TPL, I I do have a underscore partials directory where it's it's just a bunch of small HTML snippet that are just available to all of the the pages. So so let's say let's say we are building a dashboard, so you can have something like, I don't know, a a user a user snippet. Let's say it's called user dot HTML, and this thing will print the avatar of the current user and their names and maybe a drop down menu for, well, maybe not the drop down menu. But let's say let's say we have a lot of places in our application where we will have a a user display like that with their avatar and their name.

Dominic: 09:19

So so basically that that is the kind of component that I was expecting to to to be building. So I started to to move, you know, to move my HTML around and was I was simply doing, you know, mostly, you know, in in HTML templates. I have I have a a define, directive at the top. Let's say, define, quote unquote user. And inside that, I have a small small piece of HTML, that just displayed the avatar and and the name.

Dominic: 09:52

So I just copied the HTML in there and now, was was also creating the pages. And I I was having, let's say let's say the the main the main root, the root component with, with the dashboard page and and a couple of components. And I started to to, to get a lot of problem, from the the temple compiler. So I I I at first, I was like, good. This is, you know, this is interesting.

Dominic: 10:32

This is nice. So maybe there's there's some form of formatting things or maybe there's some just straight all e errors in my HTML. You know, it's possible. So I started to to try and and see, you know, what was those error and that, you know, the the the error line that that was, that was sending me the the compiler was was like a directly slash a, for example, or slash div. The error was extremely difficult to understand, you know, extremely vague, like, non empty node or something like that.

Dominic: 11:12

Not not non empty node. It's it was not, you know, I I I don't recall exactly. It was it was not the, because you you need to terminate your, let's say you have a a b r for example or or an image tag, so you need to you need to terminate them with a slash. But I already already always do that mostly. So I was having a lot of formatting issue I think from what I would say my partial and my pages to to the temple view.

Dominic: 11:44

Maybe it's because and that that would probably sounds extremely to to a lot of you, but I am using tabs in HTML. I'm I'm I'm always using tab as mother as much as I can for indentation. So is it is it that that caused an issue because their, their compiler is expecting, I don't know, spaces and whatnot. I'm also I'm also using Pretyfire, to, you know, automatically format my HTML, you know, pages and and things like that. And it was kind of lost, from from the temple, point of view as well.

Dominic: 12:29

So I mean, I I was yes. So to me, this is this is where this is where things started to to not work for my situation. So that being said, I I I if if I think if I had started with Tempo from the beginning and try to, you know, go with the the compiler as as quickly as possible, fixing issues, as you are typing or as you are creating your new component. I think that would have been easy versus trying to migrate what I have at the moment and just copy pasting, you know, what I would consider valid HTML to, to temple. Because at the end of the day, I was, I was mainly, you know, replacing the range, directive with for the if, you know, with good old, if, statement and, you know, calling the data itself just, just using the, the brace the temple brace, having the field in there.

Dominic: 13:42

So yes. I I I think I think if I had started from the beginning, so my my my be a choice. I I even I I even considered it a couple of months ago. I I think I started this project like 3 or 4 months ago, 3 months. It was, it was on the table.

Dominic: 14:06

But yes, I think, I, I, I have to, I have to say that the the amount of of time that it seems to be taking, for for the compilation or at least to get the proper error, seems to seems to be, to be a little bit too long for me at least. I I can sense it. I I feel that there's there's something there and, and yes. So that that brought me to to the idea of, you know, maybe at the stage where I am with with my library, maybe it might be an an interesting, idea to investigate if I could, include some static analysis in there. And I started to to play with that a little bit.

Dominic: 15:03

But be before, you know, before I go there so I'm not I'm not saying at all that that the temple is is, you know, it's good. It's really, really nice. And to be frank, I would have really enjoy if it if it if it would have worked for me. But again, I mean, it's I'm I'm just one person and it's it's it might be extremely localized to my situation maybe. Yeah.

Dominic: 15:30

I don't know. I don't know exactly, but but, yes, it's, it's interesting. I I have searched a lot and it seems to it seems to, you know, it seems to be a common common problem. So there there's a couple of people, there there's some people that also have this, you know, some issues with. And I it's totally understandable.

Dominic: 15:51

I mean, it's it's kind of kind of huge and crazy, what, what they are trying to do. And, if I were to restart a project today, I'm I might I might be tempted to to try it from scratch. So I would be extremely interested to see if my experience would be different going from, you know, just starting with Temple directly. I kind of imagined that it would, it would be a little bit smoother and probably, probably, you know, easier, compared to what, what I was trying to do, which is like convert converting, converting a lot a lot of things. And especially if if the formatting is is not is not done properly now, the error message is extremely misleading.

Dominic: 16:41

So I mean, it it was, yeah, it was difficult. So so, yes, it it led me to to start thinking about my my small library, which I, you know, I I'm building it for myself, to be frank. I'm not not really expecting that, to, to be extremely popular. That being said, if I am able to complete this static analysis part, might, you know, it might start to, to be a little bit more interesting maybe for for others. So what I'm what I'm thinking at the moment, what I what I do have at the at this time is that since the library is already parsing all of the HTML, you know, pieces that that an application have, so again, if we, if we do a small detour back to to to TPL, it kind of forces you to have a directory structure for your your templates, with kind of layout pages and views.

Dominic: 17:51

So those are the 2 main, building block of of of this. So you you have a couple of layout pages. Let's say let's say we have a a public facing layout pages called layout. HTML, and we have an app. HTML when the users are signed in, for example.

Dominic: 18:10

So from there, you will have some views inside a subdirectory call layout and a and some views inside a subdirectory call app. So I'm I'm already I'm, you know, I'm already parsing everything there. So I was able to, to start playing with the, you know, the AST. So we we do have we do have access to the AST directly when we are parsing templates. So that was extremely interesting to me.

Dominic: 18:42

I I did not knew that, on runtime we we were able to access that. So what I did is that for now I'm kind of extracting all the fields that are used in all the pipelines. So all the directive in in a in a HTML page, and I'm storing that into a file. Yeah. I I know it's it's, you know, bear with me for for one second.

Dominic: 19:09

It's it's just a it's just an idea at the moment. So when are those templates are created? So at the end at the beginning of your when your application is is starting, you are forced to parse your template. So you call a function called tpl. Parse.

Dominic: 19:27

So this is already like that in my library. So at this moment I'm able to capture all the fields that are used in a specific template or a partial page. So if we, if we recall, in TPL you have views, let's say we are in the dashboard. So you, you might have a dashboard dot HTML inside a app directory. And we have our underscore partials directory, which will, you know, will have all the the small pieces that you want to, to share across all your views.

Dominic: 20:05

So my I'm able to, you know, to extract all the fields. So let's say let's say our dashboard is receiving, I don't know, some kind of some kind of structure, whatever call it the dashboard data. This is a struct that have, a list of to dos, whatever. So I'm I'm now able to to capture, you know, all all the usage of that that structure. And it's possible to do that if you compare t p what TPL is doing compared to the HTML package, it's that it's wrapping the data that you want to send to your views.

Dominic: 20:44

It's wrapping that into its own structure. So this structure is already having like the current user, the XSRF token, for example. There is a couple of things that are just reused on on, you know, mostly all all the HTML views that I've I've seen in my career so far. So basically that I wanted something that encapsulated yes again, the the language of the user, the current user, the, there's a data field which the data field is is what you would want to send to your views itself. But there's a couple of other, you know, fields in that in that struct.

Dominic: 21:33

So first of all, the you know, this structure is known, this is this is a defined type, and and it's pretty easy to, to to do some static analysis because now I know that all of the pages and the partials well, not the partials, sorry, but all the all the views at least will receive this, you know, this wrapped, structure. So it's pretty easy to to do some static analysis for for that. Now the the data so there's there's 2 fields, in TPL, on that structure that you can you can send some custom data. Well, first of all well, no. Yeah.

Dominic: 22:16

There's 3. So the current user is is is a is an interface. The data field is an interface, and and there's an extra as well. So I mean, again, this is my library. I mean, yeah.

Dominic: 22:28

This this is what I was knee needing. I find myself that sometimes I want to, to pass something more than just the structure that that the views is expecting. So if we were to build a an e commerce website for example, you might want to pass a product in the product page, but in the extra you might want to pass the store, the current store of of, you know, of this particular particular store where there is the the the main menu, the categories and and a lot of information. So I mean, yeah. So again, returning to this idea that TPL is already parsing all this thing, already knowing all the fields that that is used, so what I needed to, to connect the dot at this point was how can I, you know, how can I parse all those fields or at least or how can I store those fields and now have some kind of process or small a small, you know, small CLI that might that might be watching some files that could that could just run the static analysis, when when, you know, files are changing or at least when you this when when this file that contains all the combination of views and, the fields that are used in that field, or or or, you know, when this file change?

Dominic: 24:03

So again, I use the the go slash parse. So so so Go have, you know, it's it's all built in. There's there's a package there's a package to access some e AST, and kind of browse the the, you know, I I I could I could have some package, sent to, to an input of a CLI and just, you know, just fetch that package for exported fields and whatnot and compare that to the the file that was generated. So I mean let me rephrase that. So when we are parsing the template, when TPL is parsing the template, it saves a file.

Dominic: 24:48

So let's say let's imagine just a kind of a key value file where the key is the name of the template and and the value would be like a a slice of string that just contains all the the data or all the fields that are used in that template. And now the, the CLI is able to to take that with without even touching the, the template itself and without even having to know anything about the template because that part is done. Now it's able to say, okay, you know, we have those templates, those templates are using those fields. Now am I able to inspect the the AST of of the go of the go package, that that is passed to to, to the command line and see if there is something, that is wrong, just a typo there. So that my my goal at this point is just to start detecting.

Dominic: 25:51

You know what? You have used dot emails and the there's no field in in the user struct called emails. So now how would the, you know, the static analyzer knows about the type? Well, at first, I wanted to to use the comment system of the templates. So that that was my idea.

Dominic: 26:17

Well, I wanted to to just add a comment in, you know, on on the top of each views and, maybe use a an old, an old RazorView, syntax where where they they they are using an at symbol and, they have the the word model. I would, you know, I was going to use a couple of different things, but just having some, some comment that explain, you know, these views is going to receive a user, for example. The user being a struct, accessible in in in the package itself. Unfortunately, the AST that is that is received, when you are using the HTML template, parse, the comments are just omitted from the AST. So I I did not add access to that.

Dominic: 27:12

So I said, well, strangely enough and since the this is my package, I could do anything. So what I'm doing at the moment is I'm already, you know, I'm already having a func map and you might you might already know what where I'm going with that. But I'm already having a func map where I have the, you know, all the functions for translation and couple of, internalization functions for dates and currencies and whatnot. So I mean, I said, you know what, let's have a TPL type function there which which just accept a, you know, a type in string, and returns nothing. And this is how I'm now able to specify, you know what?

Dominic: 28:02

This views is receiving user, for example, or this view and and this function is, accept, the the you know, it's a varietic, argument. So it it it accepts the type for the data, the current user and also the extra fields of the wrapped structure. So now the crazy part is that I am now able to ensure that if a field is used in the header of those three things, now I, you know, from an external CLI, I know that, okay, you know, this, the the the the dot data is supposed to be a user. It at this moment, they are calling it dot emails, and that does not exist in the user. So I'm I'm kind of very, very close to be able to, to write that down, from the CLI to just say, you know what?

Dominic: 29:02

You you have an error. You are using an emails. I I unfortunately, I am losing the line number and whatnot, at this time. Again, this this is all just a huge huge quick prototype. But, I don't know.

Dominic: 29:20

It feels, it feels interesting at least. And of course yes, yes. It's it's a little bit strange that it's it's kind of needing an intermediate, you know, files in between the parsing of the templates and and the static analysis. But there's a good reason for that. There's a lot going on when people are parsing their, their templates and one aspect is that they can they can also pass a func map as well to just happen, what they want as functions.

Dominic: 30:00

So I would not have access to that from the static analysis CLI. So that's that's mainly why the parsing is not done there. Because if if the parsing would have been done in the CLI, I would have been able to specify, you know what, you will you will include the comment as well in in your AST, you know, a template, please give me the comment. But yes, at this moment and it's, you know, it's totally invisible. So the way that you are doing that and I'm I'm going you know, again, this is my library, this is this is just me trying to find a good good way for me to build my Sass.

Dominic: 30:44

But but, I mean, if this works, maybe maybe it it will be useful to to others. I don't know. So there's a config that can be passed to the to the TPL, and now there's there's there's 2 there's 2 fields. First of all, there's one called enable static analysis. It's a boolean.

Dominic: 31:02

And the second one is that where you want this file to be to be safe. So I imagine myself doing I already have some kind of, you know, app environment, you know, the stage. So when I'm in development, I will be generating that file, which, by the way, it's it's negligible. It's it's it's I don't even know, but it's it's it's just it's just taking the AST and and just, basically, encoding that and writing that to, to a file. And and the the analysis tool will will be a separate TPL.

Dominic: 31:43

It will it will be called TPL. It will be a CLI, very, very, you know, quick and dirty CLI that just have to, to function well, at least 2 argument or 2 commands, let's say, let's call that a command. So it will there will be the init, which will generate all the the the directory structure for for the library to work correctly. And, there there will be this, this analyzer, command or whatever the name, which will want to accept the file that that was generated when parsing the template and also the root package for where it should start to look at, you know, will try to find the the structure itself. And, and yeah, I mean, this is, this is what I'm working on at the moment.

Dominic: 32:35

I'm having a little bit of fun. It's not like I'm, I'm, you know, I'm not working on that 40 hours a week. Of course, I'm I'm building a Sass. So but I can see myself having a decent because at at the end of the day, to me, the HTML template, what what is what I miss from from that, to be frank myself, only is those small errors that I'm doing sometimes when I'm using a a field. Let's say let's say there there's a I don't know, there's a struct that gets refactored and there are some fields that change.

Dominic: 33:16

You will only see those those errors in, in run time. So now if I have this, static analysis tool working and and it's able to find that in development, I mean, yeah, that that's still that's still pretty pretty decent. I have started a branch, called, well, I guess it's called static analysis. I don't know. On on the TPL repository, if you are interested interested to see what it looks like, if you want to to jump as well, if you want to contribute.

Dominic: 33:49

I mean, I'm I'm def definitely not, not an expert in all of that, that thing, but but seems to be, it seems to be possible to be, to be able to do that. Is it going to to to capture all the issues? Not at all. I mean, I I'm I'm not even sure if if it's if it will work. But, but just by by be able being able to type the you know quote unquote type.

Dominic: 34:18

It's it's not it's not real type, but it's it's just saying that you know what, this interface that these views is receiving, it should be I don't know a to do list or it should be whatever, x y z structure. And, and from there, I mean, it should be, should be possible to do. Again, this is, you know, this this is interesting and I've I've, you know, I've been having some some fun doing that with, some tests and, it's nice sometimes to build something else than, a web server and Go. To be to be completely frank, I find myself being in a in a position where, you know, 80, 85 percent of what I'm building well, maybe 80% is mostly web based, web backend related stuff. You know So yes, it's, it's interesting sometimes to just, not be a web backend developer and and put a put a another hat that just say, you know what, I I will build this tool for me and and see, first of all, see if I can do that, and second of all, maybe there will be some, some interest, at some point.

Dominic: 35:41

I don't know. But again, it was fun. So Go is, extremely nice. I still love it even after, 10 years plus. And, yeah.

Dominic: 35:52

If you want to, to reach out, if you want to come to this podcast as a guest, I mean, I would be extremely happy to to talk to you if you have an open, open source project, if you have something, that you want to talk about. Just, you know, just reach out to me. The best way is probably via, you know, Twitter at the moment. But, but yeah. On that, have a great week.

Dominic: 36:20

Bye.