Subscribe
Share
Share
Embed
The Question is a collaborative learning podcast about Design Systems. Smart people like you sign up, answer a few niche questions about design systems for each episode, and then we all get together to unpack the data we've gathered. Each week, I'll invite a new co-host to help facilitate the conversation. After the deep dive, the co-host and I record a recap of what we learned. That means, for each episode, you can listen to the recap and the full deep dive!
If you're a design system practitioner, subscribe today (https://bencallahan.com/the-question) to receive an invitation to each episode. This only works if the community joins in!
Stay in learning mode ❤️
Ben Callahan (00:00)
Welcome everybody to the question episode 68 recap. I'm very excited to welcome my friend TJ here to talk a little bit about design systems as AI context. TJ, we just finished the deep dive and we had over 100 people on our zoom call and in our FigJam file jumping around. was wild. Thank you so much for being here,
TJ (00:20)
Yeah, thanks for having me. It ⁓ was a lively conversation.
Ben Callahan (00:24)
Yes, and we're toying with the idea of how to kind of keep it going because I think there's so much we just barely scratched the surface in that hour. So I appreciate you. There's something I've always appreciated about you. You're sort of like I saw it today when we were in the deep dive, TJ, you kind of come with all this knowledge and expertise, but you come with such a
like openness and like you're just a curious guy. can see that. That's really true to kind of what I want this community to be is like embracing that curiosity. So I just want to say thanks for coming with such like that was just like on your sleeves. Like, I mean, I really appreciated that perspective you brought. So thanks. Yeah. Yeah.
TJ (01:05)
Thanks.
Yeah, like the, I think the, I feel like you can attest to this too. It's like, I feel like we all got to the place that we're at through the people who did this before us. think like open source technology has always been something that's been something that we've always adopted. I feel like help build the majority of us who are in these fields today.
And now it's, it's, it's easy to take the work that you do for granted, ⁓ because like you're, when you're in your own little bubble and you're working and you're doing your things and it's like, who's going to find value and then you had this stuff. But then as soon as you start putting this stuff out there and you get validation on that stuff that you're putting out, feels like, all right, now the tables have turned. I am it's, is, it is my obligation to start giving back now. And I think that's part of the reason why I've just kind of doubled down on.
like just exposing all these internal tools that we do and services and, and, hopefully other people can find value out of it. And, you know, maybe you'll keep, keep passing that on.
Ben Callahan (02:10)
Yeah, they definitely are. People were very excited to have you here. so again, thanks for the time. We asked ⁓ three questions. I'll read those in a moment. But I also just wanted to say, as we kind of open up this chat, that there's a lot going on in the space right now. Folks are feeling all the feels around AI. it is ⁓ in some ways, it's a loaded conversation. There's a lot of emotion here. There are people who are literally afraid for their jobs. There are people who are
ethically concerned with energy and all the things that, you know, the loss of intellectual property and all, there's so much wound up in this topic, right? In just those two words, those two letters, AI. And so as we kind of, you and I, TJ, come into this, I just want to say, like, if you're listening, ⁓ you know, we're trying to approach this conversation with a lot of sensitivity to that variety of emotion that folks bring to this topic. ⁓
The reality is there's a lot for us to learn here, a lot for us to unpack. And so we're trying to stay curious. We're trying to just really dig in together. And that's kind of our goal. So thanks for coming with that attitude, TJ. And ⁓ appreciate everybody who's listening in with that as well. So we asked.
TJ (03:24)
Yeah, absolutely.
Ben Callahan (03:27)
We asked three questions. ⁓ And the first one was, how prepared is your design system to act as reliable AI context today? The second was, have you experimented with having AI generate UI based on your design? And the third was, if yes, how do you feel about the output of your AI? And if no, what's keeping you from trying this?
We actually sent this out to, let's see, 1031 design system practitioners. We got 148 responses this week, which is a record number for us. And if you are a data nerd and you want to go look at the data yourself, you can always click this link or find it wherever you're listening to this to go see the raw data yourself. So you can look at all the anonymized answers that folks submitted this week. We did our own little diagnosis and the first question, it's funny because
because
overall it's kind of like folks are sort of in the middle. It's kind of what this says to me in terms of their preparedness. But you said something very early in the deep dive TJ, which was that this question is actually not getting at the reality of their preparedness. It's getting at their perception of their systems preparedness. Can you just unpack that for us?
TJ (04:41)
Yeah, I think perception is the right word. I think in traditional design systems, there's the perception that if everything looks like it's, from a static point of view, everything looks like it's the way it's supposed to look. And even from maybe a sub-level under that, metadata is in its right spot, and there are states and variants and various properties all put together that it is in a state for developer handoff.
And, ⁓ and without the use of AI tools, it may just go completely unnoticed. A developer may inherit it. The developer may be using the dev mode within, within Figma. And then you might find little discrepancies like detached components and some hard coded token values and things like that. And then just be like, whatever about it. And then just go about their day, realign, remap those tokens to make sure that the code part is in a good spot. And then eventually that ends up getting shipped to.
to production. Now, let's time this by 70. So let's think about like, you know, that by a thousand cuts. So like all these little cracks eventually start to show and then this is what is the drift. This is what causes the drift because eventually there's going to be something that surfaces on the development side from the design side that a decision is going to be made by someone who isn't the person who's responsible to make that decision.
And then lo and behold, you have like a different color focus state on a button that's being used throughout the entire platform. Why I say that I think some of these in the bar chart might be slightly skewed or overestimated was because it's a lot of the underlying infrastructure, the unsexy stuff that goes unnoticed that is noticed by AI. So when we say things are AI ready,
Ben Callahan (06:31)
Yeah.
TJ (06:34)
A lot of times when we run these AI diagnostics and you can even do this with like just the standard official figma MCP and just ask it to like give me the component structure of a card component and then it can tell you whether or not it's finding a nested component button or whether that button is a brand new button that has been detached and is a part of that because what would happen.
is if you ask it to start developing it, it's going to end up developing a brand new button instead of importing the button that you already have within that code base. We learned that the hard way. In the beginning of our explorations, this was the stuff that we were scratching our heads going, wait a minute, so is AI not ready for this? Is this the things that maybe people were talking about AI hallucinating and then as we pulled that thread and worked our way backtrack ⁓ to where
Ben Callahan (07:02)
Yeah.
TJ (07:24)
Where is this AI getting this information from? Lo and behold, this is a specific instance where we're working with a banner that was an icon button. And then sure enough, that icon button was detached. And then we had a fresh new icon button in our code base that didn't belong there. So I think these things, a lot of times, without any sort of linting or diagnostics that are happening at the Figma level, then yeah.
your output's only as good as your input. So if it's absorbing crap, then it's going to spit out crap. And then if you flip that on its head and say, if you're absorbing pure context and that context is in place and then we have these tools in place in order for you to run those AI diagnostics before you allow AI to start making that transition into a code base, then you can feel confident.
Ben Callahan (07:53)
Yeah.
TJ (08:12)
about what you've done on the design side. And then you can also allow AI to help support you in those things that it finds to help you, because not only will it recognize how AI ready are you, it can also assist you in making it more AI ready. So anyway, I think a lot of the people who haven't really dabbled too far into the AI part, who feel as though it might be AI ready, might want to do a little bit more of a under the hood, deep checking to see whether or not it's actually there. And some of those threes and fours might end up as ones or twos.
Ben Callahan (08:40)
Probably, yeah, it's so good. So if I was trying to summarize that TJ, I would say the hallucination or the slop that you see is a signal that perhaps there is more clarity or structure needed in the system in order for AI to generate something valuable. Is that an accurate summary? Yeah.
TJ (08:42)
Ugh.
That's a, yeah, you just,
I wish I can talk that succinctly, but that's exactly what it is. Sometimes I'll over explain things too much, but that's exactly it. It's like, the, if the, let's, if we refer to it as context, like that, that's the context, all of those facets that are happening, all of those pieces that are happening within whatever your canonical source of truth is, whether it's like a UX contract or whether it's Figma, whether it's Penpot, wherever you're getting that information from, as long as that
Ben Callahan (09:02)
No, that's great.
TJ (09:29)
context is intact, then the AI should be able to effectively do its job.
Ben Callahan (09:32)
Yeah. Okay. So we didn't talk about this in the deep dive, but you've just got me thinking. most people, most people don't know this, but when I was in college, so I'm about certain 50, so this is a while ago, but when I was in college, I studied computer science and I actually had to choose a specialization and I chose AI. So this is a long time ago. Yeah. Well, yeah, a long time ago, but a lot of the things that I was learning about
TJ (09:49)
No way, you're ahead of the game!
Ben Callahan (09:55)
Years and years ago are things that you see now much much more refined and the scale is unbelievable, of course. ⁓ But what's interesting about this to me is that a lot of the courses I had to take were like physiological psychology. Like we were studying the structure of the brain because the whole goal with the code was to recreate what humans can do. And so the way you set this up, which is like it's possible that
As a human, it's enough for you to just design some things that look right in Figma, pass that to an engineer. They can sort of infer the intent because they're another human and make good choices. and now when we, when we do that same flow, but instead we, we, we, ask AI to do some of the generation. We're seeing errors that come into play, right? Because the clarity of the context is not defined, but humans can somehow work through that.
to a limited extent. So do you think that there's a future where it's not necessary to have all of the infrastructure, all the clarity where the AI can essentially infer the intent? Is that something you see?
TJ (11:09)
⁓ Yes, but I don't know if we want that. like there's a part where like humans need, this is like opinion on AI, maybe not necessarily with the focus on design systems, because I feel this way about much of AI supportive technologies. ⁓ There needs to be human involvement along the way in order to produce original thought.
And as long as AI is doing the job for you, it's not going to produce slop, but it's going to produce things based on patterns that it recognized. then patterns are things that it has learned because they've already been done before. So it just makes more sense to insert a human in the mix, at least to just be aware. if you are, I mean, I understand in the context of design systems or digital teams, humans are
There's resource issues where they may not be available to be able to be a part of every every every step of the way where AI is involved to monitor it. But when there's big decisions that are being made, I would not recommend AI just handing that off or handing that off to AI. it just AI can infer purpose and intent, I'm sure, because it understands the just general knowledge like.
what a certain component usually does based upon best practices or based upon majority usage. But what if you want to do something different? What if there's something unique that you want to do that hasn't been thought of before? feel like even if it's just oversight, a human needs to be there. But more so, let's think about new things. Let's try to figure out a new way to ⁓ build a switch or something different besides tabbed interfaces. I don't know.
Ben Callahan (12:53)
Yeah.
Right.
Yeah, I love that. I think that's a really smart way to think about all of this. we talked a little bit about this, where I think one of the sort of general concepts that we kind of landed on in the deep dive was it's nice, it's better that the results, the output will be better if we have human and AI interacting throughout the process instead of just.
One part is human, one part is AI. And so I think that's sort of ⁓ an echo maybe of what I hear you saying now. And I think that resonates.
TJ (13:31)
Yeah, yeah, definitely. just to add on top of that, the catastrophic things that can happen that lead to the discussions around AI not working or AI being ineffective or AI slop usually happens when the AI is starting to do something that you're not aware that it's doing, and then you're continuously building on top of the thing that it produced that you were unaware of. And then that's when the slop compounds.
Ben Callahan (13:56)
Yeah.
Yes.
TJ (13:58)
That's
when it's like it's the trough of cement or whatever you have a manure and you're like like bucketing more on top of it because you'll end up with this unwieldy product that no one knows about. Like no one, whether you're coming from the design side and you just design this giant dashboard of product inventory things and you're like, I don't know, but it looks cool. I don't know if any of this would be useful, but then the same side on the development, if you just keep like, if like, that's why I like to use the term context engineering because context engineering indicates that the
Ben Callahan (14:02)
you
Yeah.
TJ (14:28)
person who's responsible for wielding that AI has some sort of insight into what the AI is doing. And I don't like to use the code. Although I am a fan of vibe coding because I think it's a great way to express creativity through people who never had the means to be able to do those things. But I think vibe coding, there was one of the commenters who would also agree with me from the session, but I don't know if vibe coding has any place in like a design system. I think it's great for prototyping thing, but
Ben Callahan (14:42)
Yeah.
TJ (14:58)
I think that the practitioners of their craft are the people who should be responsible for what goes out there so that way you don't end up with a compounding slop.
Ben Callahan (15:06)
Yeah.
That's so interesting. And it connects to a topic that we didn't get a chance to talk about in the deep dive, which is the idea of evals in AI. And they're used at many layers in the process, right? Like in the early training days of a model, there's a lot of evals that are run to sort of like help a model know, it choosing the right things, you know, as it's sort of like ⁓ learning essentially is really what it's doing. But also in implementations of AI inside of an organization, right? We can't just like put it into practice and trust that it's going to always
do the right thing. And so we need checks, we need the evals essentially right in place for that. There's a whole market spinning up around this now with like consultants that are helping organizations do that. ⁓ Is this is this a part of what you're doing in your work or? Okay.
TJ (15:51)
Yeah, it
is. And eval is the new buzzword. so eval's purpose or the evaluation of the response, like just to get to the root of what these things are and what they do, is to reduce risk on output. so whether or not it's like you're using, there's multiple ways that you could do this. There's like.
If you're using Claude, there's skills. There's if you're using any like MCP enabled AI client, you can use MCPs for these things. Or if you're using just general traditional old school things, there's like various testing methods that you can leverage for this type of stuff. But this kind of goes without saying for us, like, because it feels like the in the cascade of how to ensure that the output of AI is what the expectation should be. It's that first and foremost is context.
Like the most important thing with AI these days, even outside of design systems, is like knowing how to articulate. Like I think that's probably the most important thing. So understanding ⁓ what it is you're trying to do, what's the most articulate way of saying it, and then feeding the AI what you think that it needs to do in order for it to do its best job. And that could be as simple as like things where ⁓ it could be a concept.
that you like, like just from a personal perspective, what I do is I take morning walks and then I'll put chat GPT on talk mode and then I'll have my headphones in and then I call it weegeeing. I'll just like banter back and forth and then I'll get, this is my scratch pad in the form of AI weeds out the bad ideas, gives me a good set of ⁓ ideas that I can work with. At that point is when I start like the actual ⁓ development side of things or like I'll start incorporating it into a tool set. ⁓
But yeah, think I lost track of the original question that I had a pretty good answer for. Yeah, so and then when we get to the point of having proper context, then everything else with the expectation of the output should be good. But then depending on the complexity and ask of the AI when you're working with it, then
Ben Callahan (17:48)
Yeah, we were talking evals, but yeah.
TJ (18:12)
It's like it's either you can hand it off to evalves to like do all these like sub agent checks with there's a lot of stuff that that we do depending on the types of products that we're working on but then there's also like if we the in the context of design systems we try to approach things like in the very Russian doll method or very atomic like like a brand like a very atomic way so the more we can ⁓
scaffold up the system by starting with, let's say if we're building a brand new design system on the code side of things, starting with things like badges and indicators and avatars, things that don't have a whole lot of composition or a whole lot of metadata involved with them, and then building that up. Because as you're building that up, the AI that's embedded within that code base is gathering context. So the less you need evals, but evals are important, because there is going to be a moment when you get to,
larger architecture, like bigger structures where we'll have, and we have lots of custom agents and this kind of goes into more of an AI ⁓ deeper architecture discussion, but we have a lot of custom agents that we deploy on things that we think are going to be relatively complex bug fixing. It's like, okay, we've ran into a big issue with like web sockets or something. And it's like, okay, that sounds like a little bit of back of the front end stuff, or that's like a little bit of DevOps-y.
Ben Callahan (19:32)
Mm-hmm.
TJ (19:36)
So let's go ahead and deploy XYZ ⁓ subagents to help check the job of the one that's doing the main work in there. And then most of the time I'll do cloud code on verbose mode so I can actually see all the things that's going on there. And I'm one of those people that just monitor it. And then I always ask for a full report when it's done too, because again, I want to know exactly what it's coming through. But yes, evals are very important.
Ben Callahan (19:46)
Yeah.
Yeah.
TJ (20:04)
Have your AI check your AI and then when your AI is checked send it to another AI to recheck your AI Yeah, and then and then and then in the sense of like a pull request yeah, like put some human eyes on it
Ben Callahan (20:10)
Yeah. Yeah. And then have a human take a look, right? Yeah.
Yeah, this is great. ⁓ Okay, so we hit on a lot of interesting topics, but the one that really resonated, I think, with folks was this shift away from sort of like prompting as prime to infrastructure winning over prompting. And so we got into a lot with this topic. ⁓ So many folks jumped in, but like, how would you summarize that conversation in the deep dive?
TJ (20:44)
Yeah, infrastructure wins over. So this is the whole concept of your design system is already an API. So all of the tooling, all of the functionality, it all exists when there, it all exists within that system. but like how fine tuned is your API if your design system is that API. So if we're saying that we want to do things in leverage AI for downstream, so sort of product engineering design to like, like token mapping, whatever it is, then
your infrastructure is your prompt. Because when the AI is doing its thing, when it's running an initialization and it's doing a diagnostics, it's running through, it's absorbing all the information about the component or the Figma project or the variables tables to understand what it is to enact on, to act on whatever the prompt is, having that is 90 % of it.
to me, like, because without that, your prompt is a generic prompt. And then it's going to assume that it's going to figure it out. But then the results of that response is going to be underwhelming. And then what might lead to people blaming the AI, like, ah, the AI is junk. But having that, having a really tight knit, fine tune design system that acts as, as, know, that's your, that's your infrastructure, your infrastructure equals context. And then once it has all that, you can
you can create some pretty generic prompts and it'll make it all the way through. ⁓ What I like to do in a lot of my prompting is ⁓ I'll send it to certain directories just to say the focus is this or if I need output in certain areas, I'll always ask it to put outputs. Then ⁓ going back to eval, there is one.
Ben Callahan (22:18)
Yeah.
TJ (22:37)
product that I really want to, ⁓ that I don't know what I would do without. It's called Serena and it's a Serena MCP. It's also sequential thinking is another one that's amazing too. So it's like before you even get to the point of needing evaluations on anything before you get responses, sequential thinking does an amazing job of doing AI human thinking.
Ben Callahan (23:00)
Mm.
TJ (23:01)
Like
almost going down, if you picture just like a web and it's going through every single possible scenario, or if you're a Marvel fan, if you've seen Dr. Strange going through all the scenarios in the end game, going like, and it was the one, and that was the one. So that's basically what it's doing is it's webbing, extending all its tentacles out to figure out every possible scenario until it arrives at that one best practice way to do it. And so you can do a little pre-eval going through that method.
Ben Callahan (23:10)
Yeah. Yeah.
Yeah.
Yeah,
that's cool. ⁓ There was a lot to unpack. I'd love to hear from you just like sort of your kind of like closing thoughts, you know, like after having asked this question and sort of seeing the responses coming back from folks, like, what do you think folks can go do tomorrow or ⁓ next week that would actually help in their use of AI right now? In the system space in particular.
TJ (23:56)
Outside of just grabbing a tool, I would say stay curious. curiosity is curiosity without feeling as though you're behind. And just from a personal part, ⁓ not from organizational job pressure. I like to dedicate an hour a day just to try out new tools.
Ben Callahan (24:21)
Yeah.
TJ (24:21)
And
I think it's fun. Like I get a dopamine rush out of it. It's like, is this going to be the one? You know, is this the cheat code? But I think staying curious about what's out there and then like, like it's just like, it's just like anything where it's just like diet or exercise. Like if you want to get better at it you want to see results at it you really want to reap the benefits of the, the, the, the, it is, you know, the, the bigger muscles or the, the, the
Ben Callahan (24:25)
Yeah.
TJ (24:49)
smaller way size, then you got to put in the reps. And I think having ⁓ just a sense of curiosity to start with, to be able to be open to testing some of these methods or tools or products or platforms or reading the articles, it could be as simple as subscribing to a newsletter and just being aware is step one. Step two, if you're in this space, if you're in the design systems community, like my, I think the, my
my gateway drug for the most simple setup would be Claw Desktop. Just spin up simple download, or not even Claw Desktop, Claw Chat. Just go to Claw.ai slash chat. then pay the 10 bucks or whichever model you're at if you're a student. Pay the subscription fee so that way you get the feature set. And then install the Figma MCP.
Ben Callahan (25:42)
Yeah. Yeah.
TJ (25:42)
It's already right there for you like you
can just go in and click the button and then all you have to do is authenticate. So once you authenticate and then you don't even have to have a design system of your own to work from, you can just go to the community in figma. Grab any design system like I recommend. I like GitHub primer. I think it's a great one to test on because GitHub primer was AI ready before AI was even a thing when that design system was set up. Cause I've ran that one through a gauntlet when we were doing a lot of our testing, but.
Ben Callahan (25:54)
Yeah. Right.
Yeah.
Nice.
TJ (26:10)
Polaris is a great one. ⁓ There's a couple of shared CN ones out there. But yeah, just pluck one of those and then build something with it. You don't even need a code editor. Like you can just do it in the browser. You need one platform, a browser, and then you need a link to a component that exists in some other publicly accessible design system and then watch it do its thing. And it's magic. It really is. It's one of those things where, I don't know, after you...
Ben Callahan (26:30)
Yeah. Yeah.
TJ (26:39)
You do your first workout that is going to set up yourself for your your long-term lifestyle change health journey You feel really great after that that that exercise is done. It's like well, I did it, you know, no no, I'm amped and I'm ready I'm gonna do it again. It's in a weird way. It's a similar feeling You're like wow, I just built that whole component set in like like legit code I may not completely understand the code I can use this as a learning mechanism, but I just did it and then and then from then on out I don't know like like
Ben Callahan (26:49)
Yeah.
TJ (27:08)
There's lots of materials that you can go. This is a lot of stuff that we cover too in the AI and Design Systems course. But once you've established that relationship and then you get over that barrier to entry, because a lot of people are just uncomfortable with doing that part, I think that's the easiest and most ⁓ simplest way to start to recognize, OK, I see how effective this might be and how I can potentially start thinking about ways to use this within my own workflows.
Ben Callahan (27:37)
Yeah. Okay. One follow up to that then. So let's say somebody does that and they also are on a design system team and they have their own system. What would you say the most impactful change they could make would be to their system to make it more prepared?
TJ (27:53)
Yeah, diagnostics. and you could do the same thing with the, the, I would say if you're entry level and you don't want to overwhelm yourself, start with the official Figma MCP. it's not just made to create developed components. It is made to understand what's under the hood in these components. So you can do the same thing through, through Cloud Chat.
Ben Callahan (28:18)
Hmm.
TJ (28:21)
And you can send it the canonical link for the component set that you want to discuss and then ask it questions. like, how many components are in here? How many components are in this set? Tell me the properties of this component. Tell me the variables. me the tokens. Are there any hard-coded values in here? So you're asking it for it to provide a report on how well-defined this component is. then so that's like, you're not actually editing anything yet, but you're getting a good
Ben Callahan (28:27)
Yeah.
TJ (28:50)
you're getting a good rundown of the state of the state of this component. And then as that product designer, who may not be familiar with the code side of things, they know that what they can do is be responsible for making those edits in that component. So they can shift on over to their Figma profile, their Figma project. They can realign mapping the token values. They can create the additional variants or states that are needed for the developers. ⁓
And then that is one thing that they can take upon themselves that is fully within their responsibility and realm to be able to do to ensure that developers aren't going to come back and ask questions or don't ask questions, which is the bad part. It's like when they just make assumptions and it's like, okay, well, I'm just going to assume this is going to be green when it should be blue.
Ben Callahan (29:29)
Yeah, right.
so helpful, man. ⁓ What a cool conversation. We're scheming already to figure out how to do a part two. And I just want to say thanks again, TJ. It's so awesome to have you here sharing some of your experiences and doing so with such a great attitude. And I appreciate that a lot. It means a lot. Yeah.
TJ (29:48)
Yeah,
thanks. And yeah, I sincerely appreciate you bringing me on. yeah, anytime you want to talk more about this stuff, here.
Ben Callahan (29:58)
We'll do it. everybody for listening. Cheers.
TJ (30:00)
All
right, yeah, thanks everybody.