Transcript
Abi Noda:
Really excited to dive in to this really interesting journey you've had from starting out as a DevOps organization, DevOps team, then shifting into EngProd, then eventually landing with DevEx. So I want to start all the way at the beginning with some background for folks, all the way back to when you were still doing DevOps. So maybe start by telling listeners a little bit about the old world. What was your DevOps team doing?
Ana Petkovska:
So our DevOps team was started around the same time we started to shift our company and product from on-premise to cloud. And in the beginning, the first version... And I wasn't part of this team, I was... Actually, they got three people together that work on different topics. And they put them and formed the DevOps team. They brought a manager. And then they worked still on their own topics, like one was taking care about the tools, one was more about the release process, another one was more about one of the base component, base VMs for the product.
And then we started to hire people, DevOps engineers, to take upon other things like more tools that are used in the CI/CD process. We started to look into Kubernetes, and Kafka, and infrastructure-as-code and access rights management. So we somehow built a lot of topics on top of each other. So the cognitive load of the team really expanded, and at some point we saw that it's time to change, and to have more people, and maybe more teams dedicated on the same number of topics.
Abi Noda:
And take us back. I know you were personally involved with the shift in creation of the EngProd and then DevEx team, but who originally formed, or created, the DevOps team? Was that your CTO at the time or the CEO? Who sponsored that?
Ana Petkovska:
So there was the CTO that hired the engineering manager, and then they formed the initial team. At that part I was part of another feature team that is working just on one of our components. I already had initiated internally DevOps meetups. We had one of the days during our lunch break just to share some things that we were doing like on Jenkins and for the build and release process that could be shared, not just for our component but also for others. And this is actually how I started to be more in communication with them. And later I changed and moved to the DevOps team as well.
Abi Noda:
You told me this funny story earlier about how after you've made the full shift to DevEx, you looked back at some of your earlier slides and realized that you had already been talking about experience all the way at the beginning. Share this story with listeners, if you don't mind.
Ana Petkovska:
Sure. So when I joined Nextlink, I joined in a feature team. And at that time we had many issues with how we released this component. So one of my first tasks was to improve this. So then we sat and draw diagrams on the different jobs and so on. And then a few months later, we wanted to present this to the whole company, because we had impressive improvements in the build time, the release time. But also one of the last slides that I put is that the rest of the teams should also take on such initiatives, because this improves the engineering experience and how fast we can deliver the product to production.
I think from at that time, the experience part came from two areas. One, because our company was talking about employee experience, and I knew that this will resonate with the rest of the engineers in the company. But also because I saw the benefits that after we brought these improvements, my colleagues were much happier and excited to work on the product because the whole process on releasing got simpler than before.
Abi Noda:
That's really interesting.
You already touched on this before, but what were the specific signs you started to see that led you to start thinking, "Hey, maybe we want to shift from DevOps to EngProd"? What were you seeing, hearing, feeling that led you to those thoughts?
Ana Petkovska:
So as I mentioned, we had a lot of topics under our team, and first of all, it was hard to prioritize things as, of course, we had many different priorities from all the different topics. So it was very hard to dedicate our attention, to know what is the most important thing at the moment.
Then we started to have experts in given topics, like we had experts in the team for the release, experts in Kafka, experts in Kubernetes, so not everyone could tackle any of the tickets that we were starting to have. And then also our support board was exploding because we already had a dedicated support board for all the issues that the teams would raise to us, and the number of tickets was rising, and we were feeling like we are also not doing good job in supporting the teams in their daily work.
Abi Noda:
And as you were experiencing these challenges, what is it that led you to EngProd specifically? What's even the difference between DevOps and EngProd, first of all, but where did you even hear about or see EngProd that made you think, "Oh, there's a movement here"?
Ana Petkovska:
I saw it first on Google's website, the engineering productivity team. Then I started to do a bit of research what's going around. I saw a few videos on YouTube from Google and Netflix that really... One of them I really love because it was mostly about how the team is structured, what is the ratio between the engineers you need to have in the engineering productivity group?
So that got my attention and I started to search like, "Okay, who else is doing this and what is the team mission of all these teams, what they're focused on?" So for me it was also the big difference that I saw is that, DevOps is nice in terms of... And I really like the whole concepts and the type of work, the goals of DevOps, but for me, engineering productivity, or developer experience, is putting that into practice with a bit stronger focus on the engineers.
Abi Noda:
For listeners, maybe you could share, what was the specific video from Google, or talk that you're referring to, and what was it about that video that really struck you?
Ana Petkovska:
So it's the Engineering Productivity at Google video that was given by Michael Bachman, who was the Senior Engineering Director at Google at that time. It's an old video, it's from 2018. The things that really struck me is about what I mentioned, how the team was structured that they had many different teams, smaller teams in their engineering productivity organization, and then they gave what are the benefits of engineering productivity and wanted to have everyone… They had a team for different areas and they want to have for each of this teams dedicated DevOps effort to make them a first-class citizen. It's something that we were missing because I was seeing in our organization a lot of effort on one side, but not all the product teams were receiving the same attention from the DevOps team.
Abi Noda:
And you also mentioned, I think in the talk they may mention the ratio of the developer productivity, or EngProd organization, versus the rest of engineering. Do you recall being impressed by how big the investment was at Google?
Ana Petkovska:
I think it was 18:1 engineer. For 18 developers, you would have one person in the engineering productivity group. On our side, it was much smaller at that time. With the amount of work that we had in our team, it was obvious that we have to add more people, and we were already eight engineers. So it really made sense to split the team, have a dedicated focus with good boundaries, and then add more people to have more capacity to work on all these topics.
Abi Noda:
So you were inspired by this Google video. You've mentioned other material from companies like GitLab and Netflix, and so was your overall impression that EngProd was the future?
Ana Petkovska:
To me, it seems like there is a new discipline that is on the rise, and it's a bit different than DevOps because it's like... I saw a lot that they talk about the metrics and that is data-driven. And for me as a past researcher, and I was working in academia before, that was very tempting, because I knew that if you want to prove your work, and the concepts that you're bringing on the table, you need to have data. So it really attracted me with that aspect.
Abi Noda:
So with the help of these videos and the inspiration from other companies, I understand you started to work with your manager on building a case internally to convince management to invest in this.
So, tell us how did you begin that process? You got together with your manager. How did you begin to construct this presentation or argument?
Ana Petkovska:
So for us, even for the management, I think it was obvious that we need to split the teams, but it was not clear what the split should be. Then yes, with my manager, we built a case that this is where we want to go, there are a lot of things happening here. That we also need to separate all the things that our team is doing that are product related, that went on the side, and everything that is non-product related, and it's just serving the engineers that are our main customers.
Then we built a case. We also discussed with the people in our team to get a bit their sense, to present it to them, to understand where they want to go, because the two teams will have to work with different technologies. Then when we got that, we got a proposal for the management to show them, "Okay, look, like, this is what we think we should be doing because this is happening all around in other companies. These are the benefits. Then this is the proposed split of the team and the topics that each of the team will inherit from the original team."
Abi Noda:
I want to ask you go a little bit deeper into this because this is just something I hear all the time from folks who reach out to me and say, "Hey, I really believe there's a big opportunity at our company to improve developer productivity, developer experience, but I'm having a really tough time convincing management that this is something we should invest in."
So, what advice would you give to others? What were some of the specific benefits or arguments that you made? And it sounds like you probably also made the argument that, "Hey, this is what a lot of top companies like Google are also doing." But share with listeners what are the concrete arguments, or bullet points, benefits that you think make a really compelling case for asking for investment?
Ana Petkovska:
So the first one was this about being really focused on the developers or the engineers in the company to improve their productivity and experience so that the teams can reach their full potential. Then with this, we can also speed up the onboarding of new members because the processes will become easier and more natural and everyone can onboard easily and be more effective faster.
It can help in the retention of employees because we also had some employees that left the company. Of course, when they leave, they tell you what things they didn't like, so these are obviously the things that we need to improve. It's, in a way, we were saying like a cost reduction initiative because when you improve the productivity, you do these things faster, so you can also save on maybe hiring because you can be just more efficient with the same amount of engineers, which is I think quite important in this period where we had a lot of hiring freeze and people are a bit more conservative in hiring.
And it's also, for the product, it brings benefits because if we improve how the engineering organization work, then we can speed up the whole feedback cycle for the product, deliver the value to the customers faster. So it has benefits on many fronts.
Abi Noda:
I really liked the points you’ve shared. In particular, I really liked the one around cost reduction through being able to do more work with the same amount of engineers without as much hiring.
Do you recall anything in particular where you got pushed back or challenged by management on your proposal, and what were those things?
Ana Petkovska:
So one of the things that we got challenged was the ratio of the engineers that we need to have in the group because, of course, it’s one thing what Google does with their size. It’s one thing what we can do with our size. Because, for example, just to give some numbers maybe that can be useful. Now we have about forty-four feature teams, about 250 engineers, and now we are 12 engineers in the DevEx group today. We started with a group of six engineers two years ago, and this year we added a second group of another six engineers.
Abi Noda:
That’s really interesting about the ratio. That’s actually something that we’ve been doing research into. So we’re actually about to publish a report. We’ve been asking developer experience, EngProd leaders across the industry, "Hey, what's the ratio of your EngProd organization versus the rest?"
I don't want to give too many spoilers, but the ratio you're describing and the ratio at Google definitely falls within the range of what we're seeing. We're actually seeing an average of around 17, 18%. So a little bit higher than I think what you have today, but certainly interesting to benchmark that and think about how you even determine what the right ratio should be.
Ana Petkovska:
That sounds really exciting. We'll be looking forward for the report because it can really help into future conversations about the size of our team.
Abi Noda:
Absolutely.
I asked you before the show, what's your team mission, and you were able to read off a little bit to me and I really liked it. I'd love to ask you to share that again here with listeners.
What's your team mission? And maybe share a little bit about your experience coming up with that mission. I talked to teams that spend months defining their team charter. So that it's a challenge in of itself.
Ana Petkovska:
So in our mission, when we were forming it, we wanted to take really the two aspects. So one is delivering the business value to the customers, and this is why... Maybe I'll read it and then I'll go into more details. So it's, "Enable delivering business value to the customers fast and reliably while increasing the developer experience."
So we wanted to get the two parts of delivering the product faster to production, but also in the same time that user centricity of the developer experience and developer productivity that we want to improve the experience of our engineers. And then how we are doing that, we have two small additional things after the mission is that we do this through "Providing frameworks in terms of processes, tools, services to abstract the engineers from the complexity of bringing the software to production”, and that “we gather metrics to analyze the current state of performance and to introduce changes that improve the engineering excellence and velocity."
We really wanted to insert a metrics part so that for the team, it's also motivating that, and it should be a goal that for every initiative that we do, we collect metrics to show what we achieved and what we improved.
Abi Noda:
To be able to convey the value of what you're doing and hopefully grow investment in developer productivity over time.
I want to shift now into the topic of DevEx, and we'll talk about all the fact that there's all these buzzwords in a minute. But as I understand it, after your presentation to management, after you got approval to move forward with your EngProd initiative, later you learned about this new concept of DevEx. Share with listeners. How did you come upon this new concept and what were you thinking to yourself as you dove into this topic?
Ana Petkovska:
First of all, the concept about experience itself, it's not new in my company because our product is based around providing a better digital employee experience. Now, the developers are just one type of employee, and of course for different type of employees, you need to have different methods to improve their experience. I think for engineering productivity and developer experience, the two things arise around the same time, and today we can find different organizations calling their teams with different names.
I think this is a bit what makes it difficult to just gather all the information around this area and decide like, "Should I do this or should I do that?" Because there is no one standard way and terminology and definition that you can just google this keyword and find all you need. We have for DevOps, we have for SRE today. For us that we are working on this field, it would be quite good if we decide, or we join the two terms together, and we standardize this new movement so that we can ease the future creation of such teams and using the practices that are common for all of us.
Abi Noda:
You made an interesting comment earlier today when we were chatting about how you started really embracing developer experience as a focus, it not only helped improve the developer experience of the product teams across your company, but also the developer experience of your team.
Share with listeners what you mean by that.
Ana Petkovska:
For our team, it was also because our mission is try to improve the developer experience of the product teams, but then my mission, and our mission as managers, is to ensure that we have good well-established developer experience team that they can take on these initiatives and bring them to production, to our developers.
For me, I think it was like all the actions that we took really reduced the cognitive load of our team. By reducing the support tickets, we got more time to do meaningful and long-standing work that will bring more benefit. Then it improved the flow because now we are also not pinged for so many issues. By building platforms, we also off-loaded a lot of the work that before it was manual for our team, and we enabled the developers’ teams to have more self-service for different services that our team is offering.
So the teams I think are... Now the people that work in the teams are much more satisfied with where they are, and there is a great energy between them.
Abi Noda:
I want to ask you though, we already talked about EngProd versus DevOps, but what is the difference between EngProd and DevEx to you? What is it about DevEx that meant something different to you as you dove into that term?
Ana Petkovska:
Engineering productivity for me is a bit more focused on the tooling, on the functional stuff. But then, once you do all that part, you want to re-implement, let's say part of the CI/CD pipeline, you need to launch this to the teams. And then the DevEx part is about, "Okay, now when I need to launch it. How do I do it in a good way that it'll be easily accepted and adopted by our customers, the developers?"
So I think the whole developer experience concept is bringing this customer centricity to the whole movement. And I think by focusing also on these things, the changes, the technological changes that we are proposing, they're much more easily accepted, adopted, and then we can, of course, see the benefits faster out of all these initiatives.
Abi Noda:
And how did this translate for you? Meaning, as you embrace this new concept of developer experience and its principles, what were the specific changes or shifts that you made or wanted to make on your team to adopt this?
Ana Petkovska:
For us, it was mostly how we interact with the teams and what is the data that we get.
So first of all, for the interactions, for example, we introduced weekly meeting with the developers. It was initially for the first initiative where we wanted to give regular updates, what is the progress, when we'll get some things done, how they can adopt and supporting them through the adoption? And we saw that this is super useful because we bridge this gap between us and the rest of the teams, and we had stronger collaboration and communication with them. So then we just revamped the name. We just renamed the meeting into DevEx Connect that is today, and now we're using to share weekly news with our product teams on the initiatives that we're leading.
And then, regarding the metrics for engineering productivity, for example, we started with the hard metrics, with the hard data that we can get from pipelines. But then we saw that, "Okay, it's not enough. We want to get the sentiment of the engineers, and then we want to listen to them, what are the problems that they're facing and they're struggling to know where we can have biggest impact."
Abi Noda:
And you've shared this great journey of the shift from DevOps, to EngProd, to DevEx. Share with listeners a little bit of a glimpse into what your team is actually focusing on today.
Ana Petkovska:
So last year, the first initiative, big initiative that we did was to have continuous deployment in one of our internal environments because for us still, we have scheduled releases, monthly releases, and then in between we do hot fixes when it's required, but we want to move to the continuous deployment path. And then...
So last year we achieved the first part about having continuous deployment in an internal environment. This year it's more about revamping the complete process, so we can also have continuous deployment to production.
Abi Noda:
One of the things you touched on earlier as part of the shift to DevEx was not just wanting to focus on tools, but also human processes.
Share more about what you mean by this and how that's been actualized for you in terms of your team's focus and work.
Ana Petkovska:
So I think the tooling and the technical details are just half of the work. It depends, again, on the project and the initiative, but the other part is just on the adoption side. In our team, we often try to see, "Okay, can we do the work in a transparent way for the developers so that it doesn't have a big impact and they don't need to take any action on their side?" But often it's not possible. Sometimes they need to change something in their pipelines or to change how they work, to adopt the changes. So we saw that to have a successful adoption in these cases, it's very important, how you follow up on things and what you bring in the table, how you present these initiatives.
So usually we have one-pager where we have all the required documentation, like "Why is it important? What is the deadline to adopt this? What are the other details that they need to know to take on this initiative?" We have either a table to follow up the adoption, or we share with them some Jira templates to create on their side, tickets with sub-tasks to do the whole movement from one version to another, let's say. And then we have workshop session with them to give them more details, to explain them, to demo the things.
And through these weekly meetings, we can also follow up and ask them, "Okay, does someone has any questions that they want to raise to us that they need support with?", so that we're there with them in this process. I mentioned before also that we had, in the beginning in the DevOps group, we had a lot of support tickets from the team, and one of the good things in the transformation is that we really improved this number, we decreased the number of support tickets that we get. But we also noticed that whenever we launch one of these new initiatives, we have small spikes because this is a time when, again, teams needs us to help them to finalize and adopt the stuff that we bring to them.
Abi Noda:
Ana, thanks so much for sharing this really interesting journey from DevOps, EngProd, to DevEx. I think there's really relatable experiences and challenges and solutions you've shared today that I think will be really valuable to listeners. Thanks again for your time today.
Ana Petkovska:
Thanks a lot for the invitation. It was my pleasure to share this with others.
Abi Noda:
Thank you so much for listening to this week's episode. As always, you can find detailed show notes and other content at our website, getdx.com. If you enjoyed this episode, please subscribe to the show on Apple Podcasts, Spotify, or your favorite podcast app.
Please also consider rating our show since this helps more listeners discover our podcast. Thanks again, and I'll see you in the next episode.
