What does it mean if someone says your app is slow?

The dreaded bug report: this app is slow. Ok, but what is slow? I have so many questions!

In today's episode, we discuss our approach to getting more information out of that initial bug report, and a methodical approach to locate and quantify the slowness.

If you'd like help setting up tools like Xdebug profiling, send us a message.

Creators & Guests

Host

Aaron Saray

Host

Joel Clermont

What is No Compromises?

Two seasoned salty programming veterans talk best practices based on years of working with Laravel SaaS teams.

Welcome to No Compromises. A peek into the mind of two old web devs who have seen some things. This is Joel.

And this is Aaron.
I went to your site the other day, Joel, and it was really slow.

First of all, I don't have a site. We have a site.

No, I went to joelclermont.com. It was just slow, I think you-

Oh, I can't blame you.

It was just so slow.

All right, Aaron, you need to be more specific. What was slow? Maybe it was your internet connection.

Oh, I don't know. It was just slow, it wouldn't let me do anything.

Oh, boy. He's roleplaying the typical user bug report and it's triggering me a little early in the morning here. But I think you're trying to make a larger point.

I'm trying to talk about when someone says something is slow, what does that actually mean? We need to dig in on that a little bit.

Sure.

And as developers, I think it's our job to have that conversation back and forth a little bit with the stakeholders. So that they understand when they say something is slow, you need to define a little bit more. Like, what were you expecting it to be? How fast were you expecting it to be? And what did you think was supposed to happen? And what are you comparing it to that you feel it's slow? All those different things are the first sort of level of stuff that we want to focus on. And then I think then after that you kind of go into more of the technical questions.

Yeah, and I want to throw one more on there. Unless maybe it overlaps with something you said. But is this newly slow or has it just always been slow and you're finally mad enough to say something about it now? Because I think that's useful information. Like, "Oh, it's slow today. Yesterday it was fine." Or, "No, it's been slow for a year," that's where my brain goes. I would like to know that.

"It's been slow for a year, but I just ignored it and didn't tell you until I got really angry."

This happens, this is very common.

No, I think that's important context when we're talking about slow websites to understand how long, what the context is, and all that stuff. And then we want to move over to the actual troubleshooting. I mean, that's probably why most developers are listening. Like, "Okay, we get it. Something is slow," but what are the parts that we need to start checking for that are actually slow now? And what does slow mean? And how do we even understand how we would fix something that's "slow"?

Right. I was just going to say that I know you did this unintentionally, but actually, my site is slow sometimes and I know why I just haven't fixed it yet. But I'm like, "Aaron's going to call me out right now? What's going on here, man?"

It's probably your picture that spins upside down when you hover over it. That's what slows everything down.

Spoiler. It is a CDN thing, but anyways.

I think the first thing to take a look at is when we're ourselves measuring the speed is, are there any sort of metrics that we can look at, first of all, to even understand if we're going to make this better. So if someone says it's slow, that's great, but how do I know I made a difference? And how can I tell them when I've made that difference that it's improved? Because, hey, if you increase the speed by 50%, that's huge. And also going from 30 seconds to 15 seconds for a page load, the customer probably won't notice. They'll notice, but they won't care.

It's a move in the... What?

Having these metrics at least help you have a story to tell as you're troubleshooting and building a solution.

Yeah. And we might be proud of that 50% improvement but it's still bad, right? It's a move in the right direction but you still have some work to go.

Right. The first thing that I would look at for something like that is more of the front-end. Let's look at the inspector in Chrome or DevTools or whatever, and see if there's any delays there. If I'm not familiar with that tool in a network tab I want to do a little research into that, understand if there's some longer-running stuff in here. Because you can usually see if there's maybe a request that's taking a couple seconds. So is it JavaScript that's blocking it? Is it the page just coming from the server? Is it all the different assets? And we know especially, well, HTTP/2 and HTTP/3 technically are supposed to help with some of these issues. But until everyone supports all that, is there too many files coming at one time? Should we combine some of those? There's all these different sort of front-end questions is the first place that I'd go look.

Well, let me see if that's step one, is there a step zero? Which is, you obviously implied in this is you try to reproduce the slowness. If you can't observe it yourself and you're just relying on somebody else's bug report then you probably wouldn't start digging into the internals, right? You would make sure that you can actually see what they're seeing and you're on the same page as them?

Yeah, I think to an extent. But to another extent, you'll never see what the client or the customer is seeing because everyone's computer and internet connection is different and whatnot. Usually when a customer complains that something is slow, I mean that's 50/50, right? It was their internet or it's your site. One of the two.

Right. Well, but even within the bucket of, "It's your site," it could be their particular account on the site or it could be the time of day they were doing it. Slowness is one of those things that can be really finicky to get your hands around.

You're right. Because It could be an account that loads a bunch of customizations for a theme that's slowing it down or something like that. That makes total sense. Yeah, definitely step zero then is make sure you can identify the problem, you can see the problem yourself.

Okay. But then-

Step one, check that front-end and look at the network inspections and see where the timing issues are and all that kind of stuff. And before any backend programmers go, "Ha-ha," and any frontend-ers flip me off, I'm not saying the front-end is always a problem. I'm just saying there's a number of tools that are available on the front-end that can show us if the front-end or the back-end is more likely the culprit.

And it's certainly worth looking at. If you just skip past that and go straight to the back-end you could miss something that's pretty obvious. And I think it's probably even one of the easier spots to just see the graph and look at it and just know, "Oh, is there something here to work on or not?" and then move on to step two.

Well, speaking of that next step, that step two, that back-end. What would you do, Joel, to sort of troubleshoot a slower thing if you have a general idea it's the back-end?

It kind of depends. A lot of the projects we support have some sort of performance monitoring tool in place. Sentry is one that we use. The reason I hedge there is because even if that's in place, the way those tools work, at least at the pricing levels we're comfortable with, is you're generally only sampling requests in terms of what shows up. So depending on what your sample rate is and how frequently this happens, it might not even show up. But best case, if it's visible in a tool like that that is where I would go first because at a high level-

I'm going to be the cranky old man and say, "No."

Oh, come on.

We should have enough tools in our toolbox, in our tools that we're using on our backend, that we don't have to pay and buy someone else to tell us how our server is going. By the way, what would I do if I didn't have a paid solution like Sentry?

Well, I mean, you could just look at PHP-FPM log or your Nginx logs, things like that, just to see if they're reporting the same slowness. And obviously, we're talking mainly about Laravel apps so Laravel has some tools that are available as first-party packages, like Telescope. We typically don't run... In fact, I don't think any of our projects right now have Telescope enabled in production. But you can try to reproduce it locally or you can turn it on selectively in production as well. But that'll give you a wealth of information too and, for the budget-conscious, it doesn't cost anything. So it's just the overhead of enabling it and making do with what you get in there.

Well, I guess I have to step back to step zero here real quick. It'd be nice to be able to duplicate the problem the customer is facing locally as well.

That'd be awesome.

Because if you can do it locally, then everything is great. Like, you can use any tool you want including other back-end tools. One of the favorite ones that I've used in the past is when things go really off the wall, really hard and you really need to figure out what we're looking at, I'll use Xdebug's profiling and tracing sort of functionality.

That's not one I reach for a lot but that is the tool that will give you the most information. You can actually see down to system calls and things like that. Where if there's something really wacky... If it's not just like, "Oh, you're missing a database index." If it's something weird with file system locks or things like that, that's going to get you the answer. But I would much rather run that locally than try to get that going in production.

I like that you mentioned database locks. Because I think once we're on our back-end side, we start to figure out was it front-end or was it the back-end. Maybe it's the back-end, so now is it our code or is it our database? We can go to our database and then there's tools obviously in there to kind of determine if something is slow or not. In MySQL, for example, slow log file and things like that.

Yeah. And the thing there is, ideally you will have those on already. Depending on how you're hosting your database and how it's maintained, it may not be on because logging does incur a little bit of a performance overhead. But, yeah, if you have to turn it on after the fact you've kind of lost out on a lot of data. And I would also say if you had it on before the fact and you were monitoring it, maybe you would've solved the issue before somebody complained about it. That's the ideal scenario. I think out of the box it's one or two seconds, it's a pretty generous length of time for a database query. But you can even ratchet that down and maybe use like, "Hey, if any database query takes more than 500 milliseconds, I want to know about it." You might not act on it depending on what it is but there's some pretty useful tools that, again, you get for free with the tools you're already using.

So I guess the question really at the beginning then was, okay, the customer complained that the site is slow. What does slow mean? Then where does that slowness, if it does exist and we can reproduce it, where does it reside? Is it that front-end? Should we be looking at some of those resources? Caching, all those different things?

Sure.

Is it the back-end programming? Should we look at per-application performance monitoring tools? Should we look at profiling and tracing? Or are we going to go to the database and explain some of our queries? Look for things like that. Depending on where the slowness is, which the customer obviously can't define where it is, you can then start to develop a good process to troubleshoot slowness.

Or, you know how I like to spend money, you could just buy bigger servers. Right, Aaron?

Oh, Joel.
I tell you, man, kids these days they just don't understand.

Oh, I can relate but I want to hear more.

They just don't understand. They don't understand the type of fun, the crazy fun we had growing up. Actually, this probably still happens these days I just don't hang out at kids parks. I was thinking about bumper cars. I don't know if that's still a thing, but what a weird thing. First of all, young kids, you're still developing, great. Get in this car and try to hit people, possibly get whiplash. While you're doing that, look above you and notice the sparks. That's electrified chicken wire, don't touch that. And just all these different things. Like, what was with bumper cars? What was the deal?

Well, you're missing a step too. Before you get on the bumper cars, fill your stomach with all sorts of candy and greasy foods and stuff too. Just absolutely set yourself up for disaster.

Oh. And then bam.

Well, and it's good for your first experience being behind the wheel to be encouraged to hit other people. That's probably sending a good message for later in life too.

Yeah, but they only get some access to electric cars when they're young.

Nice.

I know I kind of glossed over Xdebug but it's a super important tool that not many people know how to use.

If you'd like help getting that set up in your project, head over to masteringlaravel.io and contact us.