Managing Airflow at Scale using the Flowrs TUI

Jan opens 12 browser tabs every morning to check overnight pipelines. Log in, check, close. Twelve times. So he built a terminal app instead.

Flowrs is a TUI for Apache Airflow written in Rust. Navigate all your environments from the keyboard, drill into failed tasks, tail live logs, bulk-mark runs -- no browser, no mouse. In this episode Jan demos it live, walks through the architecture of a TUI (event loop, state, render), compares the main frameworks (Ratatui, Bubble Tea, Textual), and gives his honest take on whether agents will eventually replace tools like this.

"Go forth and mulTUIply. Life is too short to click around."

Resources:
- Install Flowrs: brew install flowrs
- GitHub: https://github.com/janbvanbuel/flowrs

---
Click here to watch a video of this episode.
Full playlist: https://www.youtube.com/playlist?list=PLJ_da7qdfL80rA7byzC_CmyrfJWjcCTnb

• (00:00) - Introduction
• (01:17) - What is Flowrs & Airflow?
• (03:27) - Demo: Flowrs in action
• (10:18) - The evolution of CLIs and TUIs
• (13:07) - Why not just use agents?
• (14:25) - TUI frameworks: Bubble Tea, Textual, Ratatui
• (15:21) - What's up with the Rust hype?
• (16:51) - Building the Flowrs UI
• (19:20) - How to install Flowrs yourself
• (22:34) - Takeaways & what's next

---

Data & AI: Technology Explorations is a biweekly show from Dataminded. Each episode a Dataminded engineer demos a tool or technique worth knowing about -- working code, honest takes, no hype.

Music by Aleksandr Karabanov from Pixabay

Creators and Guests

Host

Jonny Daenen

Head of Knowledge at Dataminded

Guest

Jan Vanbuel

Team Lead at Dataminded

What is Technology Explorations in Data & AI?

Deep dives and practical demos on the technologies shaping modern data and AI development. Join the Dataminded team as we explore, unbox, and critically review the latest tools, from building AI agents and RAG systems to optimizing cloud costs and accelerating data pipelines. We cut through the hype to show you what actually works in real data engineering practice, complete with demo code!

You open up your laptop and it's time to check the overnight data pipelines.

You open up a browser, you log in and check the pipelines.

You open up another tab, you log in again, different environment, same thing.

12 environments, dev production, that's 24 environments every single morning.

My colleague Jan got tired of that, so he built a terminal user interface called Flowers
that allows him to manage all of his environments at once, from his keyboard, no mouse, no

browser, no nonsense.

And in this episode, Jan is going to show us how that works and how you can build
something like this yourself in Rust.

I am Jonny, knowledge theory at Dataminded, and welcome to Technology Explorations.

Hi everyone, today we'll have a look at text user interfaces or terminal user interfaces.

For that, I've invited Jan.

Welcome Jan.

Thank you, Jonny.

Could you tell us a bit more about yourself and your role at Dataminded?

I'm working here at Dataminded as a data engineer.

I'm a team lead currently at one of our clients and I'm also heading uh our academy.

Okay Jan, so you have been building a terminal user interface.

Maybe show us what you built

Sure.

I'm in my terminal and in my case the tui is called Flowrs and you can start Flowrs by
just typing Flowrs.

The name Flowrs is a pun on Apache airflow and flow Flowrs and Because it is written in
rust a lot of rust projects.

They have this RS suffix So that's where the name comes from Flowrs

And so for our viewers, Airflow is an orchestrator for data engineers.

Could we maybe have a look at what the UI looks like normally?

Sure,

So this is the Airflow UI.

And in Airflow, you have your DAGs, your directed acyclic graphs, which represent your
workflow, your data pipelines.

And you can inspect them and see how they are running, what they are doing.

So here you have an overview of all of your DAG runs.

Each...

vertical line is a DAG run.

Each square here is a task your pipeline.

You can also visualize it as this DAG, as this directed acyclic graph of all these
operations that will get scheduled by Airflow.

And so Airflow is basically the babysitter of these DAGs, of these pipelines.

And as you can already see, to navigate in this UI,

First of all, there's a lot going on.

There's a lot of things that I can click.

There's a lot of different buttons and tabs.

What you've now seen was one Airflow environment, one Airflow instance, and that's still
manageable if you need to navigate through the DAGs and through all of the pipelines to

see after a night of batch runs, which pipelines have failed, where do you need to take
action.

But if you have 10 or tens of these environments, it quite quickly becomes a hassle to do
this through the UI by just clicking.

And I got sick and tired of doing that, and that's why I built the Tui.

Yeah.

So you hate clicking around and jumping across all these environments because it takes a
lot of time.

And as a data engineer, you manage multiple ones of these.

Like in this case, we see 12 environments, and you have a development production, you need
to check all of them.

So that would require 12 tabs being opened.

And then even in those environments, you manage multiple DAGs, lots of work.

lots of clicking, of exactly, lots of digging deeper to figure out what happened.

And I wanted to make this easier.

Yeah, so you built your own terminal user interface called Flowrs to navigate it.

So could you show us around a bit in the terminal user interface on how you use it?

Yeah, sure.

So let's drill down on an environment.

So when you go into an environment, you have, first of all, the list of DAGs.

So these are your pipelines, your workflows, and then you can navigate through the DAGs
just using your keyboard.

can use VIM key bindings.

So meaning J is going down, K is going up.

You can also jump to the top of the table by using gg, or you can go to the bottom with
capital G.

And you can also filter if you just quickly want to find a DAG that you're interested in
by using the slash command and then you see a filter popping up.

at the bottom and then you can look for your favorite DAG and in my case I want to have a
look at this DAG.

If you then select the DAG, you get to see all of the DAG runs of the DAG.

So a DAG run is an instantiation of a DAG and DAG is this concept of a workflow or
pipeline.

DAG run is a run of this pipeline.

You get some information about this DAG.

When did it run?

How long did it take?

There's

a line gauge to show how long it took in comparison to other DAG runs.

And if you then drill down, you can see all of the individual tasks of your DAG run to see
how long did they take, did they run successfully.

And then even at the run or the task level, you can drill down and see all of the logs of
that specific run.

I also see a follow mode here.

So you would be able to have a trail and follow along when the pod is running in this
case, where the task is running.

Exactly, So it basically continuously polls for new logs and then it will follow along so
you don't have to do anything.

while you did the drill down, I saw these tabs at Every time you went a bit deeper and you
go from the top level where you have all your instances to your DAGs, to the runs, to the

tasks, to the logs.

And how do you effectively do use this in your day to day?

In my case, what I typically do, there's a few workflows, but we typically get alerted of
important pipelines when they fail.

This is happening in our alerting system.

It's pager duty, And then I just look at the name of the DAG in the alert and I navigate
to the right environment and I start searching for that DAG to see uh it failed.

And then I drill down to see what failed, why did it fail?

And as I mentioned before, you get the application logs for a failed task instance.

So I can easily

figure out what went wrong and try to fix it.

Yeah.

your trigger is an error from a pipeline.

You get an alert, there's something wrong.

And this allows you to immediately dive deep into that error without opening a browser,
opening the right environment.

So it saves you a bit of time, I guess.

Yeah, it does.

can also, if I know which DAG it is,

Here there's a bunch of successful tasks, but I know that there's also a couple of failed
ones.

And then I can also filter on the state.

You also see hopefully that there's like this autocomplete.

It will also cycle if I start typing failed, it already knows.

And then I just have an overview of the failed tasks of my...

DAG run in this case and then I can have a look at the logs and see okay here's some a dbt
job is running and one of the tests actually failed

Yeah.

And also these things like this auto-complete, you built that yourself in this UI.

yeah, the autocomplete is basically a state machine.

It generates the possible attributes of a DAG run or of a task that you can filter on.

very nice.

I think this looks quite slick actually, this UI.

Anything else you'd like to show

Actually, yes, on the UI side, Jonny, I recently added something based on a request of a
user.

So we all know, and you also know that any self-respecting developer uses a dark mode
terminal, but there are some people that have maybe some color issues, color blindness,

and they sometimes prefer a light mode terminal.

So when you would open Flowrs in a light mode terminal, looks at your terminal, tries to
figure out are you in a light mode terminal or in a dark mode terminal.

uh

Because terminals, do they pass on the information in the environment?

well, I cannot give you the full details because I used a dependency to do this.

There is a crate called Terminal ColorSaurus and I like the name already, but that does it
for you.

And if you look at the readme of the project, it seems like it's people really knowing
what they're doing and really knowing terminals inside out, looking at specific escape

sequences and C codes.

It just works in my case.

It correctly can figure out that this is a light mode terminal.

So the extra feature is light mode where we used to see dark modes.

You have a light mode as a new feature.

exactly.

And not only light modes, you can also, because of course we have to over engineer
everything, you can also select these Catppucin themes.

like latte, frappe, macchiato.

And they each have a slightly different color scheme.

And you can also enable

uh Flower supports both Airflow V2 and V3, which is something that was quite useful for us
because we have many of these environments and we were migrating these environments from

V2 to V3.

But we still wanted to manage our DAGs in the same way, right?

We still wanted to navigate through our DAGs, find the failed task instances.

And it kind of looks the same for both V2 and V3 Airflow.

So

So you provide more UI stability than Airflow itself at this point.

Yes, but also less features.

In the Airflow UI, apparently now you can play Doom.

That's something you cannot do with Flowrs.

nice not yet at least.

I wanted to show this escape hatch maybe not everything you want is visible

within Flowrs.

So you can press O on any object on the DAG, on the DAG run or on task instance, and it
will open the Airflow UI of that environment.

you can easily, if you need more features or more graphical features, you can easily dive
in.

Nice.

if you press V, it shows you the code of the DAG and you can

one last thing I would like to show, which I think is quite cool, is if you have a bunch
of failed DAGs, for example, and you want to all mark them as successful, you typically

would do this if there's nothing really actionable about your failed DAG.

Maybe your DAG has failed because there was a timeout or something, but you know that it's
fixed now, and so you just want to mark your DAG as successful.

You can press shift V, you go into visual mode, just like in vim, you can navigate and
that basically selects a bunch of these, in this case, DAG runs.

And you can then press the button to mark them as success, failed I want to mark them as
successful.

So in terms of managing such a large workload or multiple environments, this is a lot
faster than going to the UI.

I see tuis popping up everywhere.

Where is this coming from?

that's a good question.

terminal user interfaces were the OG interfaces of our computers, right?

You used to have only a terminal and some text on it.

This is a VT100, one of the first tele typewriters.

That's what you got, black screen and some text.

And that used to be more than enough for a lot of use cases.

You also have an example of MS-DOS.

Also the first games, I really like this kind of trivia, the first games were also just
text-based games And I think as a developer,

Yeah, you kind of like working in a terminal.

You feel efficient, you feel fast.

If you have to navigate through a file system, I think most developers would say they
prefer doing that than clicking around

You have this element of speed, of being efficient.

it's a lot less.

convoluted and a lot less busy than the typical web UI.

Although of course, if people properly design web UIs, they can also be simple and
focused.

And then there's also these retro aesthetics.

It's more of a personal thing, but I like it.

and it's also available on many systems, right?

Like there's always like a bash terminal available that you can use.

Indeed.

Where I also think it shines is in for these kinds of applications where you have a big
surface and like Airflow is a good example.

You have a lot of things you can do with Airflow with the UI.

There's also an Airflow CLI, which is also quite a big CLI.

You can do a lot of things.

What is then the difference between CLIs and TUIs according to you?

When would you opt for each of them?

a very good question.

And also it can be tied to this new wave of agentic AI and agents doing things for us.

A tui is a very visual thing, right?

We humans are very visual creatures, right?

And we easily spot small changes in uh

in a visual thing, reading a bunch of text, which is typically what comes out of a shell
command that takes time for us.

We don't read that fast.

Now agents do read very fast, right?

So you have all of these agent skills and a lot of agent skills wrap around the CLI, they
use CLIs.

And I really think that's a beautiful marriage between the two.

Agents can read super fast, the CLIs just give text back.

But as a human, you are, I think, more efficient by just having something visual.

In the case of Airflow, it's way easier to spot one red dot on your screen to show that
something has failed, and then read the output of one CLI command.

Yeah, so your Flowrs TUI does it mean it's not as suited for Agents to use?

Yeah, maybe it could be used.

I'm not sure if it just will be very efficient.

And so what is then the reason you designed Flowrs?

Because now we have agents, they can talk to CLIs, you could just ask a question.

Do you still have the need for a speed up that you cannot achieve with agents?

Well, first of all, I started developing Flowrs before there were agents and it was born
out of frustration with having to click in the UI and also curiosity.

I like to know how things tick.

Like how does it actually work?

What does it do?

So I started building this before the whole agentic revolution,

agents already replacing Flowrs in my day-to-day workflow?

Not yet, but I also don't really have a good reason or argumentation of why they couldn't.

But what I think you're still faster when doing it on your keyboard because if you know
your Flowrs then you can easily jump into things that you know.

So it's like second nature to you, seems.

You still win of the AI in this case.

Still out competing the bots.

Yes, that's true.

But if cost is not of a concern, because in the end you can indeed tell an agent, hey,
here's the airflow CLI, figure it out.

And it might take half an hour to figure it out and burn a bunch of tokens to do it.

that might be a trade off you're willing to make.

how do you build such a system?

Where do you start?

Yeah, so when I started building Flowrs, I started first looking for frameworks that
people are using to build these tuis.

I quickly found these three.

You have Bubble Tea, Textual and Ratatui and each are written in a different language
which is also quite nice if you're very familiar with a specific language With a colleague

we experimented a bit with Bubble Tea.

It's written in Go.

It's very easy to use.

It's very elegant as well.

You also, by default, get a nice UI if you just take the default templates of Bubble Tea.

But I didn't choose Bubble Tea.

I thought, well, let's do something more challenging.

And I wanted also to improve my Rust knowledge So that's why I went with Ratatui.

Textual, I think it's also quite good.

despite Python having this connotation of not being a very performant language, I think
textual UIs can be also very interactive.

and very performant.

and what is the reason for you that you're so hyped about Rust?

I hear this a lot.

I haven't looked into Rust myself.

It's still somewhere on the backlog for me.

I think one of the main reasons is that it's fun to get hyped about stuff, In the end,
it's just a programming language, they're all Turing complete.

But it's fun to get hyped about things.

And the other reason is that I like the expressiveness of the type system.

I come from a scientific background, physics and mathematics, and I always liked the
rigidity of proofs of logic.

And then with Python, it's kind of the opposite.

You can write a lot of things and nobody will complain.

And at some point they might crash and burn in production and you will look at the
wreckage and then see, ah, yeah, okay, that went wrong.

I need to fix this in this way.

With Rust, you have

kind of the opposite.

You are developing your application and for the first, in my case it was many days, the
damn thing doesn't even compile.

Some people say the compiler shouts at you.

I now look at it as the compiler teaches you why what you wrote doesn't make sense, is not
fault tolerant, has some issues with it.

So I like the expressiveness of the type system and the guarantees that it provides.

Okay, so it's a static compiled language, I assume, and you get a lot more safety.

yeah.

And one of the selling points and the tagline is always blazingly fast.

You could do a benchmark with some other tuis and you would probably find that it's
faster.

All right, so you have these different frameworks, but under the hood, they're all
actually quite similar.

In the end, it's just one big event loop, and it just...

does three different things.

It listens for input events, when you press a key on your keyboard, or you get a response
from an API call.

And then based on the event, typically updates some state.

And then after the state has been updated, there's a render phase and you display them on
the screen.

And for the rendering phase, you often have...

built-in widgets like a table, a bar graph, and that just repeats in an infinite loop.

So most of these frameworks give you either built-in widgets that you can easily configure
yourself or you can create your own widgets and they typically have a few functions that

you need to implement.

So an example of some of these widgets, this is like the demo or the showcase project of
Ratatui.

So you can create these kind of tabs with it.

You can have maps, you can have sparklines, gauges, whatever you want.

This is really good for in the movies to put on screen and show something really cool is happening,
right?

This is the basic gist of it.

You have your application, which is basically a loop

which first draws or renders to the terminal, and then it starts processing events.

See if something has changed.

In this case, the only thing that can happen is you press a key and then it will break.

And then basically your app will shut down.

But what you would do is here, put all of the logic, like if people press the J key or the
down key, that means you

to go to the next item in a table.

em If there's a tick event, like this clock ticking every 200 milliseconds, for every 2000
milliseconds, make an API call to the Airflow REST API to get the latest DAGs.

You update your state with that, and then the next loop starts basically, terminal.draw,
and it will render.

actually my main takeaway message is go forth and mulTUIply.

It's very easy to build your own Tui and to wrap around it.

And how long were you working on this?

that's a good question.

in the beginning, of course, I didn't know the language, so I had to learn Rust.

I made a lot of stupid rookie mistakes

So I think it took me.

on and off about a year to get like a first UI where I could just see the DAGs and the DAG
runs in the task instances.

But then this filtering, for example, with the state machine, this theming stuff that
became a lot easier because I feel comfortable with the language and we have agents to

help us.

Okay, nice.

And so people can actually use this, right?

Can they brew install Flowrs or how does that work?

Actually that's also a nice thing.

there was a contribution from somebody from the community last week and they added it to
the brew core taps.

So now anybody can just install brew install Flowrs.

So when you open Flowrs for the first time, actually don't see that much, right?

Because works with airflow instances and you need to tell Flowrs where to find those
airflow instances.

Airflow is an open source project, which means you can just host it yourself.

can basically deploy it on any cloud environment on AWS, for example, with some EC2
machines.

But you also have a bunch of managed services.

So actually before you would even uh open Flowrs, you would do something

like Flowrs, config enable -m for managed service and then you can type a managed service.

There are a bunch of managed services as I already mentioned.

There's conveyor which is

Dataminded's own managed airflow offering, but you also have MWAA, which is managed
Workflows for Apache Airflow on the AWS cloud.

You have Google Cloud Composer.

You have Astronomer, which we all know now because of the CEO news story.

um managed service.

have and then it generates a config for you.

Exactly, so you enable it.

In this case, it was already enabled and there is a config file If it doesn't exist, it
will get created and it basically contains some configuration that the TUI manages for

you.

And then you can start Flowrs and it will actually.

uh

find all of the Airflow environments.

If you don't want to use a managed service, you just have your own Airflow instance.

You can also just provide a host name and credentials.

If you use basic authentication like a username and a password or OAuth 2 with the JWT
token, those are all also valid options.

Also in this case you have conveyor, but conveyor manages multiple of these environments.

So it also discovers even the environments that you have.

Cause that depends on the managed service, right?

Exactly.

And on MWAA so on Amazon, it also auto-discovers the environments that you have within
your region, On GCP, you can also select a region and a GCP project, and it will also

auto-discover the composer environments that you have available.

oh Sure.

So here's the repo.

Yeah, exactly.

143 stars!

Exactly and of course it's self-starred.

So this already existed since 2023 and I was mainly using it myself, And I was constantly
poking some colleagues like, hey, have you tried Flowrs already?

I know that you have to work with 10 airflow environments.

Maybe you want to try Flowrs.

And then at some point I got an email

Mentioning like, your TUI is the TUI of the week.

And then also the maintainer of the Ratatui project also shared it on his LinkedIn, I
think.

And also some more people found it.

And I recently posted it in the Apache Airflow community Slack.

So there's some community interaction now, which I actually like.

It's also a first for me.

I built some stuff, but usually just for myself.

What did you learn about this whole TUI process

I learned that I really like Rust as a programming language.

I also learned that it's fun to just sometimes take a peek under the cover.

We all use these applications, figuring out how they tick is a fun and rewarding
experience on its own.

I've also learned that agentic AI, especially the last couple of months, can really speed
up the frequency at which you can push out new features.

And I also learned that you sometimes also need to push people, like you have to be the
annoying marketeer in order for people to start using your app.

Even if you think it's amazing.

I mean, I use it on a daily basis.

I like using it.

Just building it and making it public on GitHub.

That doesn't result in people finding it and actually using it.

You have to push it out and go after your audience yourself a bit.

Yeah, okay.

Yeah, I think it's a very nice tool you showed us, So thanks a lot for sharing what TUIs
are, how you build them, and put them in the market.

if you're using Airflow, check out the tool of Jan.

We'll share the link in the description and in the comments.

So Jan, thanks a lot for explaining this.

Thank you everybody for watching.

Check out Jan's tool and we'll see you next time.

Bye bye!

Bye!