Path To Citus Con, for developers who love Postgres

Melanie Plageman, a PostgreSQL hacker working at Microsoft, and Thomas Munro, PostgreSQL developer and committer also as Microsoft talk with co-hosts Claire Giordano and Pino de Candia. They talk through all the different ways they got started as developers. Does making your first patch to Postgres get you hooked for a lifetime? Do you have to be a tinkerer to be a good software engineer? What is the “toothbrush test”—and how do you make your avocation be your vocation? We hear stories about dropping out of school or dropped out of career fields before they found their true passions in development and Postgres. 
 
Some of the links mentioned in the order they were said: 

Creators & Guests

Host
Claire Giordano
Claire Giordano is head of the Postgres & Citus open source community initiatives at Microsoft. Claire has served in leadership roles in engineering, product management, and product marketing at Sun Microsystems, Amazon/A9, and Citus Data. At Sun, Claire managed the engineering team that created Solaris Zones, and led the effort to open source Solaris.
Host
Pino de Candia
Pino de Candia is a software dev manager at Microsoft since 2020 and is currently working on the Citus open source project. Pino previously worked on the managed PostgreSQL database service in Azure Cosmos DB for PostgreSQL, which includes Citus on Azure support for distributed PostgreSQL. Pino has lived in New Orleans since 2017.
Producer
Aaron Wislang
Open Source Engineering + Developer Relations at @Microsoft + @Azure ☁️ | @golang k8s 🐧 🐍 🦀 ☕ 🍷📷 🎹🇨🇦 | 😷 💉++ (inc. bivalent) | @aaronw.dev (on 🟦sky)
Producer
Carol Smith
Senior Program Manager at Microsoft in the Citus Community team. Previously at GitHub and Google. Horseback rider, cook, and armchair movie critic.

What is Path To Citus Con, for developers who love Postgres?

Path To Citus Con is for developers who love Postgres. Guests join Claire Giordano to discuss the human side of open source, databases, PostgreSQL, and the many PG extensions. Produced as a monthly live show on Discord by the Postgres team at Microsoft, subscribe to our calendar to join us live: https://aka.ms/PathToCitusCon-cal.

How I got started at as a developer (& in Postgres)
===

Claire: Welcome to Path To Citus Con, episode four. The topic for today is: how I got started as a developer and in Postgres. I'm Claire Giordano.

Pino: And I'm Pino de Candia.

Claire: And if you're attending live, I just mentioned, you can join the text chat in the #cituscon channel in the pathtocituson-Ep04 thread. We're joined today by Thomas Munro, who is a Postgres Committer, he works on the Postgres team here at Microsoft. He's based in New Zealand. So it's very early in the morning for him right now, and we really appreciate him getting up so early to join us. And incidentally, Thomas gave a talk at Citus Con: An Event for Postgres, which happened back in April, and it was all about parallelism in Postgres.

And that talk is what inspired us to invite Thomas to join us here today.

Thomas: Good morning.

Pino: Good morning, Thomas. And we're also joined by Melanie Plageman, who is a Postgres contributor, also works on the Postgres team here at Microsoft. Melanie recently contributed pg_stat_io to Postgres 16, and she gave a talk about it at Citus Con.

The talk was titled Additional IO Observability in Postgres with pg_stat_io. I recommend everyone check it out. It was a great talk.

Claire: And also, Melanie gave that a similar talk about pg_stat_io last week at PGCon, if I recall correctly.

Melanie: Oh, no. Last week, I, talked about benchmarking IO also, but using visuals to be to understand benchmarks.

Claire: Got it. Okay. So we can check out your Citus Con talk online, and then I presume you're gonna post the slides from, from last week's PGCon talk as well.

Melanie: Yes, I'll post them at some point.

Claire: Awesome. So I think the first question we wanna start with before we talk about how you got started in Postgres is just how you all got started as developers, and then Pino, I'm gonna ask you the same question and you're probably gonna ask me the same question because all of us started or at some point in our careers were developers.

So Melanie, maybe start with you. How did you get started as a developer?

Melanie: Yeah, so when I was in college, I was super interested in computer science and in programming. I took some classes and I struggled a lot with it. I felt like a lot of it didn't make sense. One of the things they have you do at the beginning is make games and everything was in Java and it just felt like there was a lot of magic and I felt like I didn't really understand, you know, how computers actually work.

So I ended up abandoning it and actually there's a funny story. So when I did an independent study my last year of college, which was like, "why can't Melanie understand programming?" with the computer science department. And I read different research about different people's learning styles and how to understand how to code and that kind of thing, and how your learning style influences that.

Which is, funny looking back on it because the professor of that class sometimes, you know, we message each other and now I'm a software engineer so it's kind of funny. And so I did a startup after college and I just felt really strongly that I'd seen technology make all these amazing changes in the world. Like, I did work with different nonprofits and seeing how blogging platforms really empowered people, I think made me think that's what I really wanted to do: be able to make things that were accelerators or that enabled people, and I thought that was super amazing. So in my spare time, I tried to teach myself to code in various different ways. And then, after going back I did IT consulting for a couple years, and they encourage you to kind of go more towards the sales and mark- you know, sort of like selling, consulting work as opposed to coding and developing products. So I saved up money and said, I'm gonna quit my job and I'm just gonna figure out how to become a full-time software engineer. And so I spent at that job, one of the last things I had done was start to use Postgres. And so when I quit my job, basically my boyfriend at the time was like, okay, let's make a syllabus for you to check off all these things and figure out how to become a developer.

And so we started with, I forget the name of the book, but I think it's called like, it's basically starting from the fundamental building blocks of computers. Like, "what is a processor?" And what are the sort of electrical engineering kind of things. And going up from there.

And so I spent a long time trying to learn the basics and then learning about, okay, what are functional programming languages? You know? And so it was a really interesting way to do it. And what I found was that I learned better from other people and from collaboration and so I kind of came to it realizing that I had a very different learning style and that just, like, reading something in a book, I'm definitely not a solo hacker, just reading something and figuring out on my own, even if it just doesn't work that well for me.

So , what I found is that just coding with other people and talking to other people about coding was how I really learned.

Pino: I find that really interesting. So you sort of need to know what's happening in all the layers to feel comfortable with what you're doing.

Melanie: Yeah.

Pino: Do you find that that's still the case as you approach new technologies and how?

You know, I find there's always stuff that you just can't get around to doing. So I'm amazed that you can go that deep, and how's that been more recently?

Melanie: Yeah. So I think that was why, even though initially my interest in programming was kind of more things that were higher level, like applications that were making it possible for people to have their voice be heard in the Arab Spring and things like that.

Even though that was what initially drew me to technology, when I started actually coding, I found that I didn't like application development at all because I wanted to know how things worked. And so that's what kind of drew me more to systems programming and I think that even though what we do is as Postgres developers is pretty detached from, I mean, databases definitely enable people to do things, but it's a few steps away from people doing transformative things with technology.

But I think I kind of came to this point where I realized what do I actually enjoy doing on a day-to-day basis? And I really wanna understand how things work. And that's continued till now. I mean, I definitely spend more time solo hacking than I did initially, but when I was doing a project for storage tuning for Azure when I joined Microsoft, understanding actually what was happening in the Linux kernel and trying to really understand every part of the storage tech completely was important to me before I could say, "oh, okay, I recommend this tuning" or that tuning, or let me just throw things at the wall. I had to go all the way down until I understood it at an, "okay, what is the driver doing?" What is this doing? What is that doing? And like ended up contributing a patch to Linux.

And I feel like that there's two parts of me. And one is like, I wanna help and enable people, but actually, how my brain wants to process things is like, let me completely understand it. Which are kind of not like disjoint things in some ways.

Claire: I'm still wondering about that class you took in college about the way that people learn. Did you fall into a particular category of how you learn? Was there some epiphany for you there that didn't fit? I mean, all these choices...

Melanie: Yeah. I didn't really fit into any of the categories of like, you know, you learn like auditory, you learn from hearing lectures or you learn from writing notes or whatever.

I think like a little bit, or visual learner, I think if anything it was maybe more tactile learning was something I identified with, like actually doing something. But I think what I got out of it was that talking about it and talking about concepts with people was sort of how I learned.

And I think it's kind of like I'm not a tinkerer, which I always felt like, you can't be a good engineer if you're not a tinkerer. That's like what my perception was. But the things that I understand the best are actually, things that I literally did, but especially things that I did and then I talked about with other people and taught other people or discussed with other people.

Pino: I wanna throw in something. This is a little bit off topic and then we'll come right back. But this morning I was listening to Neil deGrasse Tyson talk, the astrophysicist, talk about his podcast StarTalk. And he has a comedian on with him in every episode of a podcast because laughing and joking and being in a good mood helps remember. That's the theory behind that, so I thought that was relevant to this question of styles of learning and how learning is effective.

Claire: Well, that's why I'm here, Pino, right? I'm supposed to make you laugh.

Pino: Fair.

Melanie: I could see that. You kinda remember something because you joked about it or whatever.

Claire: Yeah. Your brain is just more open.

Melanie: Yeah. I found the link to the book that was super influential. It's called From Nand to Tetris. And it was like, you know, building a modern computer from first principles.

And this was what m ade me finally have an aha moment about computers. Awesome.

Pino: I'd given up on understanding the whole stack. Now that you've put that link there, I'm gonna check it out and see.

Melanie: Yeah, it's really cool.

Pino: Fill the gaps. Yep.

Claire: Okay, I wanna pivot to Thomas for a second and start at first principles, how did you get started as a developer?

What led you there and what kept you there?

Thomas: Okay, well, I'm gonna go back a bit further. I'm a bit older and in the 1980s we had 8 bit computers like Commodore 64's and that kind of stuff. And I was lucky, my mom got me a Sinclair ZX81, which was a tiny, tiny computer.

Probably not known all around the world, but it's a small British computer that I think it costs less than 50 bucks or something. I don't know. It was very small. And back in those days, if you wanted to make a computer do something interesting, and you're a kid, right? And you've got a lot of time on your hands and a small number of books.

There's no internet and like, maybe you've got some finite number of games or whatever, and few magazines, and if you wanna try and do something interesting with that, you really have to tinker a lot. And so, I guess I followed the kind of tinkerer path from there. You know. When I was a student, I was studying linguistics.

I thought I wanted to be a linguist at some point. But I did a fair amount of tinkering and programming, and I kind of wanted to use computers in linguistics. But, at some point a job came up where somebody, another student had left town in the middle of a project, and they were desperate for a C programmer to finish this computer program.

And I took that job and I eventually, that was just a chain of events. From there, I eventually actually dropped out of the linguistics course and just went to hack on computers full-time. That's kind of how I got started with computers.

Claire: So you dropped out of linguistics. Does that mean you dropped out of school altogether or just out of that program?

Thomas: I didn't finish my degree. Yeah. Which I kind of regret, but I sometimes think I should just go back and do that, but...

Melanie: It's funny because you love learning and like reading academic papers and that kinda stuff more than anyone else I know.

Thomas: Yeah. Well, you don't have to be in the program to read the papers, right?

Melanie: Yeah.

Thomas: I mean, and especially these days, right? I mean, it's just information is so easy to get. Back then you know, there's a lot more gatekeeping of information and knowledge and ideas I think, than there is now. Or at least it felt that way to me. I don't know.

Claire: Yeah, I don't feel like my children, I obviously took them to the library a lot as they were growing up. That was like one of our weekly activities, but I don't feel like they ever really understand that you used to actually have to go to the library to find some things out. Like there was no internet. There was no way to discover at home. Like there was no computer sitting on your desk necessarily.

Thomas: So, yeah, I remember one particular purchase, I remember getting a reference manual for the Zilog Z80 CPU that was in my computer. And I remember that the book cost more than the computer, and it was really hard to convince my parents to buy that. Yeah, it's just different times.

Melanie: Your parents, did your parents do tech? Did they like...

Thomas: No, not at all. No, nothing. Like, to them, this was all just some weird amusement. Like they didn't really, I don't think back then, at least where I was from, I don't think people really saw computers as a serious thing yet.

It wasn't a thing that you wanted your kids to get into or anything like that. Maybe they were worried you would you play too many games or something, you know? That was about it.

Claire: Well, so you said there's this chain of events, right? That you dropped out of linguistics, dropped out of school and went to hack full-time.

Thomas: Yeah.

Claire: What kinds of things did you go to hack on? What were you doing?

Thomas: Well the first thing, the first job I got when I was still at university, it was some software for a factory. It was factory automation software where there was sensors all over the factory and it was a Unix system.

It was SCO UNIX. And so, I kind of knew a bit of C because at that point I had an Amiga at home and I had one of the C compilers for the Amiga. And I sort of knew the basics of C programming. I certainly wasn't very good at it, but it was enough to land this job, finishing this project because some other person had left and it wasn't super hard.

So, that was a nice way to get started. And so it was cursors software, like end cursors software, trying to do a GUI on a text terminal screen using fairly simple drawing primitives and stuff. And it was a software for controlling machines that process wool.

And I can't even remember the details, but it was writing code in C on a UNIX system, which really opened the world of Unix to me. And I realized that wow, there's this whole culture of things happening there. And in fact, around that time if you wanted to use SCO UNIX back then, that was System V Unix for, for PCs, for 386 computers or whatever.

It was just around the time that Linux was coming out FreeBSD and things like that were around as well. Around that time too, if you were working in C on one of these System V UNIX systems, I think it costs thousands of dollars per seat to use the compiler, for example, the system didn't come with a compiler.

If you wanted the compiler, it was really expensive. It also didn't come with TCP. And if you wanted that, it was really expensive. And around that time, my boss on that project found out about GCC, the GNU compiler. And we gradually began using more and more of those tools, even though we weren't yet using Linux.

And it was a real gateway to a whole universe of Unix stuff. And that the world of open source software, which kind of exploded from there, really.

Claire: But if I were to simplify your experience, you're a self-taught engineer.

Thomas: Yeah, I guess that's true.

Claire: I mean, with the help of books and other people that you might have worked with.

Thomas: And yeah, there's a huge aspect of it is learning from specific people, like along the way, there's certain people that have been kind of technical mentors and kind of shown the way to interesting problems .

Pino: Were there any times that you thought of taking a break or changing fields? Just curious. Changing directions. Not being a developer.

Thomas: No. No, I wouldn't say so. I mean, there's fields and there's fields, right? I mean, you can use computers to solve all kinds of problems and you can work in different spaces. It's nice to change space that you're working in, right?

Like, at the moment, I'm working on Postgres and I have done for quite a while now, and and I don't plan to change that, but you know, in the past I worked in a applying computer technology to financial stuff, for example, and that was an industry that I was in for a decade.

And, you know, you can change pretty completely while staying with computers as a basic skillset. Right.

Pino: And I've met folks that are primarily interested in an area and others that just think of themselves as developers and they expect to change areas, I wouldn't say frequently, but several times, perhaps, over the course of a decade. And they just think of themselves as technologists or developer, give me anything, I'll work on it. And other folks prefer sort of generalists versus specialists?

Thomas: Yeah. I would say one thing that I've noticed over time is that like if a lot of jobs that you'll get working with computers are gonna involve solving some particular business problem, so you'll start fairly high up and you'll have to understand things about that business.

But I've noticed that some people naturally tend to look down the stack, like, look at the tools they're using to solve it from, and become more interested in those tools than the business problem. And sort of burrow further down that stack. Right? And I guess I've always tended to do that, which is how I finished up working on databases and being interested in operating systems, for example, even though perhaps at the time I was supposed to be working on some user facing app, you know?

Claire: Well I think that's how we first bonded, Thomas. You found out that I used to work at Sun in the kernel group and you just wanted to talk about Solaris for a couple of hours.

Thomas: Yeah.

Claire: So, yeah, you definitely focus on the lower parts of the stack.

Thomas: Yeah.

Pino: Claire, is this a good, oh, sorry, go ahead.

Claire: Ask your question. Go for it, Pino.

Pino: I was gonna ask you, is this a good moment to segue to your, your experience? I wanna hear more about your time. Well, how you got started.

Claire: You know, as a dev, my plan was to become a patent attorney and I was gonna go to college and study engineering and be an engineer for a couple of years and then go to law school.

And so that was the grand plan.

Pino: I have to interrupt you and before you move on from that, because how do you start off planning to be a patent attorney? That seems like not a common dream for folks.

Claire: Honestly, I think, people, children are so influenced. I think by, or at least I was, influenced by all the adults around me who would look at me and see that I was good in math and sciences, but also see how I like to debate and discuss and try to, you know, influence people and they're like, "oh, you should be an attorney."

And so, I don't know, I just got it in my head that that's what I was gonna do. Went to college found out that material science kicked my butt and wasn't gonna become an engineer, but along the way took some computer science classes and just absolutely loved it. Like, my first course was in Pascal and, next I took the data structures and algorithms course and then, you know, just kept taking more and more and more and more.

And before I knew it, I had dropped out of engineering and was studying mostly applied mathematics and computer science. So that's what I got my degree in. And then I had it in my head I wanted to live in California for a couple of years. And back in the day, Sun wasn't doing college recruiting, but I knew someone who knew someone who knew someone.

And they landed me an interview at Sun and, you know, moved out to California, worked in developer tools in the beginning before I went to the kernel group. And just loved being an engineer and decided not to move to the East Coast, not to apply to law school and just stayed in tech.

So I, I don't know, it's kind of boring to me because, it's a story that's very familiar to me and I know it well. But mm-hmm. It was a lot of fun. Sun was an exciting place back in those days. And I mean that's even before we called it Solaris, it was SunOS in the beginning.

And I got to work with absolutely brilliant people. I sometimes think of it as Camelot just those days and that team and everybody was very just tightly connected to each other and to our mission. And yeah, I still think of myself as a developer, but I did gravitate later toward becoming an engineering manager and a program manager and to leading teams of people.

And that's still kind of the role that I'm in. So, I still work really closely with engineering teams, but more on figuring out how do we communicate what we're doing or how do we do community-building around this project.

Melanie: I think it's so interesting the idea of what people tell you, adults tell you you're good at or reinforce when you're growing up, and how that affects what you think you wanna be. So I think that was for me, everyone was always like, oh, you're so good at writing and you're so good at creative things and art and, also people in my family were in TV and filmmaking, and so, that was what I thought I was gonna do. And then once you do it and people tell you, "oh, you're good at this," you know, you like keep doing it. And I think it's easy to decide to do the things that you're good at and or that people tell you you're good at. And so I always find interesting when people sort of deviate from the path that they're on, or they decide to pivot and lean into the thing that they enjoy more.

And whether or not it was the thing that people suggested that they do.

Claire: I had this English teacher in high school and he is very memorable. He used to have a bow tie on every single day and as he taught his class, and he used to always say that his wish for us was that our avocation and our vocations would be one and the same.

And he would share how happy he was that his avocation and his vocation were one and the same. He absolutely loved his job. And so whether the things people tell you when you're a kid stick and whether you do them or whether you deviate, I just think it is so cool when people land in a spot.

I call it passing the toothbrush test, right? Where you're brushing your teeth in the morning and you're looking forward to whatever that work thing is that you're gonna do that day. I know that's certainly what I wish for my kids as they grow up.

Pino: Do you think there's something a little bit unusual about our field though, in that if you grew up tinkering or you eventually ran into software development at some point, into coding, that what you got to experience is it matches pretty closely what you do on a day-to-day basis as an engineer. In other fields, it's very hard for folks to imagine what their day-to-day work life will be like.

Say you're a high schooler thinking about a career and it's just hard to imagine what will you be doing on a daily basis. What kind of problems will you be trying to solve? At least that's the feeling I get that there's less of a gap for software.

Melanie: Yeah. I totally feel like the the thing that I tell people is, the actual thing that I do, whatever, however many hours a day, I really enjoy that thing.

And it's the bulk of my time is spent doing coding and trying to think about interesting things and making things as opposed to , I think there are other jobs where part that you're doing, that's the thing, the meat of what you really wanna do, is maybe less of the hours of your day.

Claire: So before we pivot to Postgres, I actually am really curious, Pino, how you got started. I don't actually know this story.

Pino: So let's see. So I didn't really touch a computer except for learning to type, to touch type. And so until I was in college, so freshman year of college my brother, who is two years older, Leo, he said, you should take a coding course.

I think it was C++, the intro to programming. So I start taking that and I see that, so I always liked math as a student problem solving algorithms. And so I find I'm good at it I'm good at the coding part, you know, writing the code, the algorithms. Not good at all at the setup of the IDE, the build path, all of the things that you have to do around the program or to make the program actually compile and run.

But, fortunately in that first course I had, we worked in pairs and Jerry was a classmate who eventually ended up going to medical school. But we worked in a pair and he could do all that stuff that I wasn't interested in doing. And unfortunately I didn't bother to learn that.

And for many years, I always managed to find someone who would do those parts of the work for me. And so, to this day, those parts of the job remain my weakness. Yep. So maybe I'll add that after college. I'll say that. I, sorry.

Just to finish that, I guess I'd say that I probably fell into software because it's what I knew how to do. It was fun. It was fun, it was exciting. And I found that, so '99, I graduated. I went off for a year and was a volunteer math teacher. But when I finished that, it seemed like the natural thing to do.

I knew how to do it. It was fun. There were startups. This is 2000. There was so much excitement in the field. It didn't even occur to me to wonder, to ask about anything else I could do. And I think momentum carried me for a very long time. Although, it resonated with me that you said you gravitated towards engineering management.

And working with people. I had the same experience maybe I went back and forth a few times. Mostly out of a feeling of, I'll say not entirely because I loved it, but more because I felt I had to keep those skills up. What I really loved was working with people on technical problems.

Claire: Very cool. So the, the title of today was not just "how did I get started as a developer," but "how I got started in Postgres." And obviously this podcast is for developers of all types, but specifically, we have had pretty much all of our guests have something to do with the Postgres community. Sometimes with Citus, often Postgres more generally.

So. How did you get started working in Postgres? And this time we'll start with Thomas.

Thomas: Okay. So I had a number of jobs that involve databases and I finished up using probably most of the big name relational databases over the years. But sometime in the late nineties or mid to, yeah, mid to late nineties, I had a kind of a side project with a couple of friends of mine. I hesitate to call it a startup because it wasn't like terribly business-focused. We weren't really that serious, but we set up this website where you could, when you were traveling around the world, make maps and you could show where you'd been.

And this was around the time when digital cameras were starting to come out and we wanted a place to upload photographs. And you could write a journal entry at each point and you could sort of show where you'd been, travel blogging, I don't think the word blogging had been invented yet, but we, it was kind of travel blogging, right.

And it went pretty well. We got up to, I don't know, like 50,000 users or something like that. I can't remember the exact numbers. But in the end that sort of fizzled out with the advent of Facebook and all those kind of things that kind of wiped out all of the smaller blogging-type things that people used to used to exist back then. But we wrote that as a side project and we used MySQL. And there was a particular moment where we got some coverage, our website was mentioned on some US TV program or radio program. I've forgotten what it was.

And we got a lot of hits and our system completely melted down. And we spent a couple of days figuring out like, what do we need to do to fix this? How can we handle more load? And that was kind of the beginning of a study of locking that led me to Postgres. And I was my first attempt to use Postgres for that.

I realized at the time Postgres was slower than MySQL at simple, sequential queries because I'm talking about MySQL in the days of MyISAM. Back in those days, it could run simple queries much faster than Postgres, but it couldn't run them concurrent, with concurrency because it had a much simpler locking model.

And so we switched over to Postgres and we were able to handle a lot more load just because of the way those systems worked. So, now I know that MySQL since then has completely changed the way it does that stuff because it now uses Inno DB and so on. This isn't really the point of my story isn't MySQL versus Postgres, but I'm just explaining.

There was this kind of moment when it solved a very specific problem that I had. And after that, the system ran really well and I started learning more and more about Postgres from there. But it wasn't until much later, it wasn't until Postgres 9.5, so I'm not sure, what year is that?

Well, a decade later anyway, that I first wrote a patch for it, and I was working for a company that had really large data processing systems using Oracle and Db2, and there was this particular queue processing problem that I've talked about before, even at Citus Con, I can talk about queue processing.

Claire: Yeah. Last year, right?

Thomas: Yeah.

Claire: Queues in Postgres.

Thomas: That's right. And so there's a particular moment where there was a discussion about whether we could move this particular system over to Postgres and it was using the skip locked feature of Oracle. At least, that was an area we realized that Postgres wasn't quite as good at.

And, so I stayed up late hacking on my first patch for Postgres to add that feature. And it was just a really rewarding experience. I wrote a patch to add for updates skip locked, which is a relatively simple thing. If you need it, you need it.

It's not rocket science. It's a pretty simple thing that a lot of relational databases have. But when I posted the patch, I found all these people were interested in it, and other people had thought about this problem as well, and then people jumped on it and reviewed it.

And I was like, wow. I'm not the only one that wants this. And wow. Maybe this is actually gonna work. And after a while, a few rounds of review and so forth, Álvaro Herrera, who you had on earlier episode I believe of Path To Citus Con, I committed that patch.

Claire: Yes.

Thomas: And after that I was like, wow, that was just... I wish I could do this kind of stuff all the time, you know?

Claire: Were you, a complete stranger to the Postgres community when you submitted that first patch, had anyone ever heard of Thomas Munro?

Thomas: I had participated in a few discussions here and there, but not in any significant way.

Not really. I mean, I think I'd reported some bugs or I can't remember really. But no, essentially I was not well known. And then there was a particular moment where I actually got talking to someone at FOSDEM who told me that, like it was the first point, the first moment I realized that there were companies that would hire you just to work on upstream Postgres.

And, not improving it as a side product of what they were doing, but actually just literally working on it directly. That was the realization that it was possible to do that. Maybe not full time, but maybe with a part of your time and I managed to get my first job working full-time on Postgres at EnterpriseDB.

And that coincided with with a big move. We moved from London where I'd been for a long time back to New Zealand where I'm from. And I kind of knew I wasn't gonna be able to work in financial technology out here in New Zealand because there just isn't that much of that sort of industry.

And so I was really looking for a change and it was around that time that I heard about the possibility to work for EnterpriseDB and that plan worked and I finished up getting to work on work with people who were working full-time on Postgres, on really interesting problems.

Pino: And Thomas, I guess because of the way the PG community works, being on the other side of the world wasn't an impediment.

Thomas: Not at all. I mean, everyone that I interacted with was living wherever they wanted to live, right? I mean, maybe a lot of people are in the States, for example, but they're not all in San Francisco or something like that. They live, they work from home if they want to, and it could be anywhere, right. So, my time zone's a little weird, but there's plenty of overlap.

Claire: Mm-hmm. Now, in this pivotal conversation that you had at FOSDEM with someone where they planted the seed in your mind that you could work for a company that would pay you to contribute to Postgres open source. FOSDEM is an easy place to get lost at. At least that's my experience. The first time I went there, it was just overwhelming and spread out and distributed.

And were you in the Postgres dev room and is that where you were connecting with Postgres people? Or how, how did that happen?

Thomas: Well, I was lining up to get a beer actually, and there's one of the great features of Belgium, of course, a lot of beer. There were actually there.

Well, I wouldn't say lots of beer, but you know, there's a place where you can queue up and get some Belgium beer and there were actually two different people around that time who I saw and suddenly realized that they were living in this way that was suddenly attractive to me.

One of them was a friend of mine who works on GCC, he went to work for Red Hat and you know, I'm not a compiler guy. I wasn't particularly interested in that specific thing, but he moved around the time that I was moving away from London and he was moving to live in a small English village with a cricket green in the middle and live in a kind of a different way and could work from home and work on compilers, or standards libraries and things like that, it's his thing. And I was like, wow, that's pretty eye-opening.

There's a lot of possibilities out there, right? I mean, if you can contribute to open source software, you can live wherever you want.

Melanie: Did you feel strongly about the open source aspect of it? Because I noticed throughout your story there's a thread of the contrast between how inaccessible, and expensive being part of closed-source versus open source.

Did you want to move to working on open source specifically after leaving the financial industry, or was it more like that was what was possible in New Zealand?

Thomas: Oh, definitely. I mean, it gets you in touch with so many more people and it gets you outside of a small, closed environment, you know I'm not like religious about it or anything, like, I'm not saying proprietary software is bad or anything like that.

It speaks to me much more to be part of an open community where people are collaborating. I just love that.

Claire: How do you feel about it, Melanie? Is it important to you that Postgres is open source?

Melanie: Well, yeah, definitely. To the question of, how getting involved in Postgres, I was not working when I started trying to learn to code and get a job as a developer.

And so I didn't have access to a closed-source code base. So when I started working on Postgres code, I wouldn't have been able to do that if it was closed-source, right? So it was this moment where, I mean, I wasn't working in tech when the open source movement first started happening, and so I just saw the contrast because when I did IT consulting, we mostly implemented, well pretty much entirely implemented, closed-source software at different client sites and you know, of course we couldn't see the code, because we were just professional services basically not working for the company directly.

And then seeing how if I'm not being paid to work on something at all, that I can still see this code and learn from it. And I mean, there's resources online to learn how to code, but, there's a big difference between seeing toy programs or small projects and actually being able to see production code of something that's being used by people over the world.

And, I guess like my initial interest in technology was as an equalizer and as something to empower people who didn't have power, you know? And, while I will not pretend that tech is completely egalitarian and anyone can do it and that kind of thing, I think the principles behind open source really align with some of the things that we've been able to do with technology.

And I think it's so important, and that in some ways I think we need zealots like Richard Stallman, that kind of thing, to really move things forward, to hold the line about open source. I mean, I still use closed-source software, so I'm not religious about it.

I'm definitely like, this is convenient. But I think there's been times throughout my journey working on Postgres where I thought, I don't know if this is gonna work out, I dunno if I can keep doing this. And you know, what would I do instead? And I think it would be hard.

It would be hard for me after having the chance to work at open source and feeling like I'm part of something that has the possibility to include everyone to go and work in closed-source. I think it would be hard for me. And, one thing that I've noticed is, when you talk to people that were part of open source when it started, things were open source for different reasons in some ways than they are now.

So now, companies talk about open sourcing stuff for strategic reasons. Like, there's this really cool open source conference in Raleigh Durham every year called All Things Open. It's like thousands of people go to it and it's different than FOSDEM.

It's on the other end of the spectrum, I mean, I haven't been to FOSDEM, but I get the sense that's kind of the original open source mission around like free open source collaboration, that kind of thing. But now companies open source software strategically for different reasons, right?

It's part of a business plan and it's something that's packaged in a product of like, whatever, capitalism, right? It's a thing companies do. So Facebook and Google and all these places are doing that, and I think it's good. It's making more software open and more people can use it, and you can build a website with completely open source components on a laptop in college or whatever. And I think it's great, but it's also people that are coming up in open source now are seeing a very different core mission or ethos of open source. And I don't know like what the implications of that are, but I think that that's interesting.

Claire: Yeah, there are a lot more reasons to open source software now and many more different motivations. But I think the cool net effect of it is there's a lot more open source software.

Thomas: I mean, certainly Postgres and and I think Ingress, that it's predecessor were open sourced, or given a BSD license for strategic reasons back in the beginning. I think you can say because they were developed at Berkeley University and then also turned into commercial products by their creators, the BSD license was part of that.

And the ability for people to be able to make money from things while also releasing the source code, that's something that happened right at the beginning of Postgres' story, right?

Claire: You know, I realize...

Pino: Would you agree with this statement? Companies decide whether to open source, partly based on talent, but there's always a business justification on the other side of the market, now that there's so much open source, there are more and more developers that want a guarantee that they'll spend at least some of their time or most of their time coding open source projects. Would you agree with that?

Thomas: I think that if you're asking me, I think that's true. Yeah. I hear a lot of people who want some kind of time carved out for contributing to open source. Yeah.

Claire: Okay. So I realize that we actually didn't get Melanie's answer to how she got started in Postgres. We talked about your perspective on open source and the motivations, but was there a moment, is there a story? Were you doing something else altogether different and then pivoted to Postgres?

Melanie: Yeah, so I was doing IT consulting at PWC, at PricewaterhouseCoopers, and there was a project we were doing that was related to data quality and different proprietary software you can use to, essentially what they're doing is writing SQL to query things about the customer's database.

So like, using regular expressions and things like that to check if they have more than one record for the same customer. You signed up with an email address twice and you spelled your name slightly differently or things like that, right. And so we had decided the proprietary tools we were using were difficult because we were trying to collaborate with people from different companies and the team had changed a lot. There's new people joining and other people leaving. And just the licensing and dealing with everyone trying to have access to the proprietary software was slowing us down so much that I was like, let's just, I mean, this is probably against the rules now that I think about it.

I was like, let's just dump this data into a database and then used regular SQL to query it because we're spending most of our time in meetings talking about how no one can access this data in order to do these queries. Right. And so then we ended up writing a bunch of very similar kinds of SQL queries and it felt like we were wasting a lot of time on boilerplate.

And I decided to use Postgres. I think this was 9.6, but I'm not sure. I'd have to look at the exact version. Just because I like Googled around and talked to some friends that were in tech and they're like, "everyone uses Postgres, it's the best!" open source stuff because I wanted something that I didn't have to pay for and just didn't get permission to really do the project this way.

So I was like, okay, what's free? What can I use? And I found that also what I wanted to do was, so, Postgres has a string function called FORMAT and it lets you do string interpolation and so I wanted to do something closer to templating to create SQL queries, but I wasn't building a whole application and I originally didn't want to use like Jinga2 or whatever.

So, I think I ended up doing that eventually. So I wrote a Postgres extension, and at the time I was trying to figure out how do I make FORMAT do the thing I want, or do I need to use another piece of software with it or whatever. And so someone told me that Postgres has this extensibility thing and you can write extensions and they can access internal Postgres code and do cool things with it if you write them in C.

So, I tried to figure out how to do that and so I wrote this extension that basically was a souped up FORMAT function that let you interpolate. You could make named parameters and then you could pass JSONB or hstores to it. And so you could like have the key that was in your hstore or JSONB, and then it would interpolate the value.

And I just thought it was so cool you could just do that. You could just make it and then you could use it. And that was my first time looking at Postgres code and I had this feeling that I hadn't had before where I was like, this just makes sense. And so I leaned into it and then I remember I was at All Things Open and there was a developer advocate for Mozilla there talking about Rust.

And his talk was about "why do systems programming?" And he made the argument that if you feel excited when you understand how something works, like under the hood, kind of like what Thomas was saying, right? Like, you just want to keep traveling further down the stack, and you sort of lose interest in the business problems at some point.

Or just are less engaged, that may be a sign that you want to be a systems engineer. And so I was like, that's the feeling that I'm feeling. I stopped caring about templating those SQL queries and I cared more about how does this work, you know? And I decided at the time, looking back, it was pretty crazy that I thought this was something that was possible, but I was like, "I wanna get paid to contribute to Postgres" and "I'm gonna work at a database company" because I want to do this.

And I didn't study CS, had never had a job as a programmer. And like, all right, I'm not gonna take a job that's not that. So just kind of nuts looking back on it because there aren't that many jobs like that. Right? Just like the number of people that do that is not very many.

Claire: Yeah. So you left Pricewaterhouse and eventually landed at, what was your first database company?

Melanie: Pivotal working on Greenplum. Yeah, I wouldn't be here if it wasn't for that, because like I said, I've learned from collaboration and from working with other people much better than through reading or of taking a class and listening to lectures.

So I saved up enough money to give myself runway for I thought a year, but then living is expensive. And basically said, I'm gonna learn everything I can and then I'm also gonna try to make a portfolio basically. And so, I mean, I was familiar with some database concepts from the job that I'd had in consulting with data quality.

But I did the Postgres extension. I sort of polished it and, made it open source and then did more hacking on Postgres. And then I was looking for jobs and I applied, I think Vertica, I applied to HP, is it? Yeah, I was just looking for databases and I preferred open source, but I was like, okay, I wanna work on a database.

So I applied at a bunch of different database companies. I interviewed at Second Quadrant actually. But Pivotal, it was just, they were so focused on collaboration and pair programming, and that was exactly what I was looking for. And I mean, in a lot of ways they took a big risk on me, right?

Because I didn't have a track record of being a software engineer at other companies or, you know, the bunch of people that worked there had PhDs in query execution and just completely different level of qualifications than I had. And for months of working there people just would pair program and teach me things.

And at that point was like, okay, I'm gonna go back to school and start taking classes on some of the fundamentals. So I did some Stanford classes on databases and then I did some of CMU classes on OS you know, learning about operating systems and I just spent all my free time... okay, so Greenplum is a distributed fork. It's a distributed database that's a fork of Postgres. So you know, what are distributed systems, how do I learn about that? And then all my spare time, I was like, how do I learn more? How does Postgres upstream development process work? Let me look at patches and everything.

Claire: Well plus one to the hiring manager at Greenplum who took that risk on you, obviously it paid off. I always think it takes a special kind of courage for a manager to hire, not because this person has already done the thing that we're hiring them for, but to hire someone who's got the skills, the motivation, the ability, and the willingness to learn and learn and learn.

And I sometimes feel like those types of gambles, like that's where the joy is. And those type of gambles can often pay off really well. But not everybody does that.

Melanie: Yeah.

Claire: I'm glad they did it because now you're here at Microsoft and still working on Postgres and just contributed something pretty, pretty sweet to PG 16.

Melanie: Yeah, definitely grateful.

Claire: So are other Postgres contributor stories the same as, as yours, Thomas and Melanie? Like as is there a certain pattern?

Melanie: At least -

Thomas: I can answer that. Oh, go.

Melanie: No, go for it.

Claire: I kind of want you both to answer, so just take turns.

Thomas: I would say it, it's very non-uniform. People all have very different back stories. That may be true in any group of people in any project. I'm not sure, but certainly with the people that I know in the Postgres community have all different sorts of backgrounds and there's a lot of people, for example, who didn't study computer science or something like that.

If that's what you're getting at. Yeah, Melanie, what's your-

Melanie: I think I often ask people this inside who didn't, you know, study CS undergrad and I would say the majority of people that you talk to that are Postgres contributors did do some sort of studying of IT or computers or that kind of thing.

But certainly not all of them, and I think I would argue there might be a disproportionate number of people that sort of fall into that solo hacker category and didn't finish school or didn't necessarily stick to that one specific path or study that exact thing. And I think Postgres being open source and having a lot of enthusiasts be the people that were working on it in their spare time and that kind of thing, I think is a big part of the background of people.

Claire: We have a set of questions that we wanted to ask you that Pino and I collaborated on beforehand, and neither one of us will own up to writing this question down. So I don't know how it got on our list, but it's: "is being a developer all sunshine and roses?" I'm just gonna ask it anyway.

Thomas: Well, I could say that there's an enormous amount of maintenance work involved in projects like this. Investigating bugs. I mean, the number of users of Postgres is absolutely enormous, right? I mean, we don't even have a way to begin to estimate the number of databases out there, I don't think.

But the number of users of Postgres, but it's gonna be a very large number and that's kind of stressful, right? You know, if bug reports are coming in and you don't yet have any clue what the blast radius is, yeah. You kind of drop everything and focus on that for a long time.

Maybe trying to reproduce something that's timing-based or something like that. And there's a huge amount of work like that can be quite stressful. It's not all developing.

Pino: Could you list some of those Thomas, in retrospect of course it's obvious, but reminders are great about the maintaining the build farms.

Ummm... The commitfest software. Right. Could you give some more examples of stuff that isn't right? Isn't the core software but makes stuff go right? The build, the CI system?

Thomas: Well, there's a whole process for.... I don't know how many people contribute to Postgres exactly. There are different ways to estimate it, but you can say there's a couple of hundred people who are actively working on patches at any given time, I think. And those people are all trying to see certain patches they've worked on get in, and they're trying to get other people in the community to build consensus and agree that a certain feature is ready to go in or whatever.

Or just trying to get people to review things and there's just a huge amount of human interaction there and a huge amount of expectation and disappointment. Sometimes things just don't go in for a whole year or two years or something like that. Or sometimes things are outright rejected, sometimes things just don't work out.

They get committed and then they get reverted and we never get back to them, or you need to go around again and it takes another whole year. And there's a lot of processes and pain and emotions and all those kinds of things involved. I guess maybe like any collaborative enterprise, but it's not just, "wow, we made this amazing new feature" and everyone's happy at the end.

It's an ongoing process and even when you manage to get things in and they work and everyone's happy things, query planners, for example, are incredibly complex things and you can make some people happy and other people sad when you make a change. You know, maybe blow back comes for years telling you that if you look at something like the the git query compilation stuff that happened a few years ago now that's really amazing technology and it makes a whole lot of long running queries run a lot faster, but it also slows down some workloads and you've gotta tune things to turn it off sometimes, because, it's a really complex system.

Melanie: I think one that's interesting. Oh, sorry. Go ahead, Thomas.

Thomas: No, and just to answer Pino's question about the stuff like the build farm, Postgres is uncommonly portable.

I think that's partly to do with its age. It comes from the time of the, you know, Cambrian explosion of Unix operating systems that used to run on all these different systems. And gradually we prune the set of systems we support, but we still support 10 or depending on how you count 10 or 11 operating systems.

And so we have, including Solaris and AIX and various systems that some people care about. And we have this build farm that tests everything on hundreds of computers and runs our test suite and it fails sometimes, for timing reasons on some obscure computer system, and we have to try and figure out why.

You know, is it a problem with that particular machine or is there a bug? Probably the answer is there's a bug somewhere or some race condition somewhere. And so you can just look at the Postgres build farm results and you can find problems to investigate that will take days and days of your life and will turn out to be because of some order of operation things somewhere, or more likely just a timing bug in the test itself or something like that.

And there's there's huge amounts of different types of work involved in keeping this thing running. Melanie, you guys...

Melanie: One of the things yeah, I was just gonna say, the thing that's interesting about working in open source, is that there isn't some specific roadmap and then you work on a feature and we've all agreed it's the thing that we're gonna work on, and when it's done it gets to go in. People pick up projects and try to build consensus around them, and that's true for core Postgres code, and it's also true for tools and frameworks and things that the Postgres developers and other people build to try to make development easier.

There's a thousand little factors that can make it, okay, this thing actually isn't something that we can support and merge. And I think the calculus for whether or not to work on something can be difficult. Figuring out all of the different things that you need to do and whether or not you should do each of those things and whether or not it's actually the right use of time and all of that. It's not something that's spelled out and dictated and then these are the priorities and then we just do them. And everyone in the community is doing, you know, three projects on the side or maintaining a tool or something like that that you didn't know about that's outside of their core work that's contributing to the community, like Thomas developed cf-bot, which is something that pretty much we completely rely on in the commitfest to show if past patches are registered for commitfest are passing. And that's just something that was like, oh, this is a problem and you know, let me just write something to solve it and it wasn't a simple, trivial project and he didn't know if people were going to use it or not. Right? It's interesting to try to find motivation and figure out what is the right way to use your time, because there's a thousand things that you could do. Like what he said, there's a thousand projects on the list of projects that I would like to work on, or tools that I thought of making that would help with Postgres or ideas that I had or wanted to try.

And I think most Postgres developers have that, or most engineers probably have that: there's all these things I could work on. And I think typically if you're in a corporate environment, there's very clear business priorities. If you do these things, they will get merged. And it's different when you work in open source.

Claire: Well, one of the things I've observed is a lot of the developers, whether they're committers, contributors, they also contribute in ways that go beyond code. And if there's college students listening to this or trying to imagine what their future jobs in open source software might be like, people focus on helping with error message translations, right? Across a ton of different languages.

You both were at PGCon last week, right? You both gave talks. So there's the whole notion of giving conference talks to help other people understand either a new feature or your perspective on a problem. The demos that people create and even, there's a whole bunch of Postgres contributors who help to organize conferences to bring people together. Because as you said, Thomas, before we started today's podcast, that it was so awesome to see people in person again last week. So there's a lot of ways that people end up contributing to a community that go beyond code. I mean, another example is Thomas, you reached out to me recently to try to figure out how can we get Windows... I think you wanted Azure VM credits on Windows to help with the QA efforts for the next version of Postgres, right?

Thomas: Mm-hmm.

Claire: So figuring out who the right people are to ask and make something like that happen that ends up being on some developer's plates too.

Thomas: Yeah.

Claire: So I think what that means to me is, and tell me if I'm right or wrong, I could be wrong. That the way you spend your days isn't always the same, the same, the same.

Thomas: Definitely true. I mean, I guess one of the things Melanie was talking about, how if you if you work on a more traditional software project where your goal is to produce, I know this is a over simplification. Nobody's job is like this, but supposing you were to compare with a job where you're gonna produce this one widget and you know, maybe you've got one boss, and you've got one product line, and, you know, maybe there are multiple projects underneath that, but it's still guided by a kind of hierarchical approach.

Maybe no job is really like that, but still, I find working at open source tends to be very different from that. Because you're communicating with a lot of different people about a lot of different things. Everybody wants something slightly different and you've gotta figure out which things to say no to, which can be quite tricky, you know, which things, the only way you can focus on something is to not focus on something else, right? So choosing how to allocate your limited time and resources when you're communicating with so many people about so many things can be a little tricky.

Claire: So, you're saying that your time management skills are important, like improving those, being aware of those and fine tuning them?

Thomas: I guess so. I mean, I wish I could be better at that myself. But trouble is there's so many interesting things as well. You can be genuinely interested in all of these different things and you want to see them advance. So figuring out how to help with individual projects that are happening in the community without getting completely lost in one of them and then not doing the other things.

You know, that's really something that I struggle with personally. I guess I'm still kind of responding to, is it all roses and honey or whatever it was.

Claire: Sunshine and roses.

Melanie: I think there's also stages, at least I've found for myself in working in the community where at the beginning, every project you do, no matter what it is, you're gonna learn something from.

And so it doesn't matter what it is, you don't have to be that strategic in what you work on because like you're just trying to scramble and cover some enough surface area to get a handhold, you know?

Thomas: Mm-hmm.

Melanie: And there's lots of, you know, basically anything you do is a good use of your time almost.

And then you kind of get to the point where if you want to see work go in or you kind of have to think more strategically about how to spend your time. I mean, there's so much of the Postgres code base I don't know, and I still have so much to learn, but I think that transition to when you have to think about which projects to work on and be more strategic, I think is kind of an interesting thing. And that applies to tools and to sort of things outside of core as well, or community engagement. You know, do I wanna give talks this year or do I wanna volunteer in one of the committees for the next developer conference?

Or do I wanna run a workshop for prospective developers? You can't do it all. Do I wanna try to be a commitfest manager, and figuring out how to spend your time, again, without explicit... You know, someone telling you, okay, this thing is worth five points and this thing is worth 10.

Or, this is a business priority and this isn't, I don't know how to navigate that still. Because it's not just what is the highest success rate, or what has the highest likelihood of succeeding and it's not just what am I interested in and what'll make my day most interesting.

It's like some combination of all of that plus what'll help enrich the community the most. And yeah, I don't know that formula, but it's hard.

Pino: So if I were to try to summarize what you both are saying when asked about "is it all sunshine and roses", you're saying there's a lot of different kinds of work, all of it meaningful and how much you like it depends on your tastes.

You both gravitated towards this.... I don't know if it's existential anxiety, but a certain kind of angst around how you choose what to do in the absence of top down hierarchy or direction. Yeah. Is that right? Because I mean, I think that's great. I mean, I walk away saying okay, that's interesting, the psychological thing going on, but otherwise it's mostly sunshine and roses or are there other aspects that are not sunshine and roses beyond that angst?

Claire: Well, roses have thorns and sunshine is sometimes blocked by clouds.

Melanie: I have fun pretty much every day, like even if I'm stuck on a bug or I'm on an email thread that gets annoying on hackers I look forward to working every single day pretty much. Yeah.

Thomas: That's the toothbrush test.

Claire: Yeah. Yeah, that's right. What about you, Thomas? Do you pass the toothbrush test? Oh does your postgres work...

Thomas: Most days. Most days. I do love my job. I think we're very lucky to be able to work on something that we know how to improve and that a lot of people depend on, and that it's satisfying to work on. Yeah, definitely.

Claire: Well, big thank you to both of you for joining us today. This podcast will be made available within the next couple of days as a recording on YouTube, on YouTube podcasts. And we'll post that link on Twitter and Mastodon and LinkedIn and all the places Carol Smith will take care of that. We're also gonna be making this podcast available across all the podcast platforms that people tend to prefer. Everybody has their own favorites. So Carol and Aaron are working on a plan to do that as well. So it'll be available on YouTube this week but other places later. I also wanna throw a shout out to the fact that the next episode of Path To Citus Con Episode 05 is gonna be on Wednesday, July 12th at 10:00 AM PDT.

And I wanna thank Carol Smith and Aaron Wislang for producing this event, because without them we wouldn't all be here today. But Thomas and Melanie, I could talk to you for another hour on this, and we have a whole bunch more questions that Pino and I didn't get a chance to ask. But we'll have to save those for another day.

Thomas: Mm-hmm.

Pino: Are you all free for beers?

Melanie: Virtual?

Thomas: It's a little early, but yeah...

Claire: It's definitely too early for Thomas, but yeah. Melanie, you're east coast, right? So..

Melanie: I'm East Coast. Yeah.

Claire: Almost have a beer. Maybe?

Melanie: 2:00 PM Happy hour.

Claire: All right, Pino, thank you so much for co-hosting with me.

Pino: Thank you. Thanks to our guests.

Claire: It's a wrap. Thanks. You actually, I do have one more thing, which is to thank, for those of you who are listening to this and who loved it, liked it, as a new podcast we appreciate recommendations. So subscribe to the calendar but also recommend it to your friends so other people can discover it.

And yeah, we'll be back here on Discord on Wednesday, July 12th for the next episode.