Software Delivery in Small Batches

Adam connects Deming's trips to Japan in 1950 with modern software delivery. This is an essential episode.

Deming on Small Batches:
Down the Deming Rabbit Hole:
★ Support this podcast on Patreon ★

Creators & Guests

Host
Adam Hawkins
Software Delivery Coach

What is Software Delivery in Small Batches?

Adam Hawkins presents the theory and practices behind software delivery excellence. Topics include DevOps, lean, software architecture, continuous delivery, and interviews with industry leaders.

Deming
Hello and welcome to Small Batches. I’m your host Adam Hawkins. In each episode, I share a small batch of software delivery education aiming to help you find flow, feedback, and learning in your daily work. Topics include DevOps, lean, continuous delivery, and conversations with industry leaders. Now, let’s begin today’s episode.
Hook
John Willis tell the story of his friend telling him “John, John....it all goes back to Deming.” I did not know what this story meant when I heard it at first.
It took me years of self-study to understand. Now I agree. It all does go back to Deming.
Trace the lineage of DevOps, to the evolution of lean, and you’ll find a humble statistician from the United States visiting Japan. That man is Dr. W. Edwards Deming.
Content
In 1950, Union of Japanese Scientists and Engineers (JUSE) invited Deming to Japan as part of the country's efforts to rebuild after World War II.
Industrial quality was low and based on inspecting products after the fact. Deming described the situation as “You burn the toast, I’ll scrape the toast”, with no one asking why the toast was burning in the first place.
Deming envisioned a different future. He wanted to build quality into the processes themselves by addressing the problems of delivering quality to the customer. That was necessary but not sufficient. He said that this was a process of ongoing improvement as customer requirements changed and market conditions changed.
Deming demonstrated this could be achieved by applying Statistical Process Control (SPC) to all processes. Start somewhere, and eventually to all processes in an organization.
This was a drastic change from the status quo.
Deming knew that change must come from top management. Meeting and educating engineers was not enough.
One of these trips included a meeting in Hakone outside Tokyo. Attendees include the majority of top executives of the biggest firms in the country. Imagine being able to talk to all the CEOs of the S&P 500 at a single time, now imagine all of them eager to apply the teachings. That’s quite a force.
Deming emphasized that leadership must think of Japan itself as a system.
Every system must have an aim, so leadership should aim at the developing country for the mutual benefit of all involved. This type of thinking later formed one pillar of Deming’s seminal work: The System of Profound Knowledge.
Deming’s slide decks from these early meetings are legendary. You can find copies of them online. Look for the slide that visualizes all of Japan as a complete and integrated system.
Deming would make more visits to Japan over the years. These trips laid the foundation for the QC (Quality Control) movement which grew into Total Quality Control (TQC).
These education trips directly contributed to the so-called “Japanese Miracle” in manufacturing and industry in the following decades.
Deming advocated for continuous improvement using PDCA, or Plan-do-check-act. This approach challenged companies to think differently about the work and how to engage employees in the business. Fundamentally, Deming wanted to unlock the creative problem-solving ability in each person.
JUSE created a prize to promote competition in quality. They called it the “Deming Prize” to honor his efforts. It’s a coveted prize to this day.
These ideas dispersed across the country finding their way into many companies. One of those companies was Toyota.
Toyota was in a difficult position in the 1950s. The company was not the successful Toyota we know today. Manufacturing was challenging given the limited supplies. The economic situation was difficult as well. The company had just closed a difficult new collective bargaining agreement with their labor union. Something had to change if Toyota would survive.
Toyota had to figure out to generate a profit by selling a small number of vehicles in small quantities with limited production capacity. Their solution to this problem changed the world.
Deming's teachings caught the attention of Taiichi Ohno, an engineer at Toyota, who went on to develop just-in-time manufacturing. Just-in-time emphasized reducing waste. Reducing waste ultimately lowered cost. Ohno applied the idea of continuous improvement to keep iterating on his production lines. This eventually led him to the idea of “pull-based” work. He called it Kanban.
Something else was happening at Toyota in the 1950s and 1960s. QC efforts began improving quality and engaging the entire company towards that aim. This evolved into Total Quality Management (TQM).
These efforts culminated in 1965 when Toyota won the Deming Prize for quality. The company had transformed since the 1950s.
Toyota continued to iterate and evolve the combination of just-in-time, kanban, and TQM through the 1970s. This when the legendary Toyota Production System (or TPS) emerged. In part, this positioned Toyota to expand into the US market with their Corolla model.
Then the 1970’s fuel crisis changed the automotive world forever. Americans wanted different cars that only the Japanese manufacturers could deliver. Decades of continuous improvement allowed manufacturer like Toyota and Honda to capture the market. American car companies simply could not keep up. These Japanese companies had developed the high-velocity edge.
The 1980s saw the peak of the “Japanese Miracle”. Japan completely transformed from producing low quality and low cost products to producing high volume, high quality, and low cost products in a matter of decades. Japanese vehicles and electronics took the world by storm and people knew it.
NBC news aired a primetime special report in 1980 titled: “If Japan Can, Why Can’t We”. The report interviewed Deming about his ideas and work in Japan decades earlier. This brought Deming’s teaching to much broader audience in United States.
In 1984, Dr. Eliyahu Goldratt published “The Goal”. Time magazine considers “The Goal” one of the 25 most influential business books of all time.
The Goal demonstrated how applying the theory of constraints and scientific thinking can turn a struggling plant around. It’s easy to imagine the same story playing out inside Ohno’s machine shop.
Goldratt essentially demonstrated part of “lean” production. However, he didn’t have the vocabulary for it because the term “lean” didn’t exist in the zeitgeist until 1990.
In 1990, the book “The Machine that Changed the World” was published. The authors coined the term “lean” to describe what these Japanese companies were doing. The so-called “lean” manufacturing focused on quality improvements based on SPC, reducing waste, developing people, continuous improvement, and engaging the whole organization in these activities.
In the 1990s and 2000s, Agile software development methodologies further built on Deming's principles. Agile emphasized adaptive planning, early delivery, customer focus, and continuous improvement. Again, echos of the principles Deming brought to Japan in 1950.
In 2001, Toyota published the first version of “The Toyota Way” to standardize how the now international company went about their daily work. The Toyota Way was conveyed as house built on two pillars: Respect for people and continuous improvement. Both of these track directly back to Deming. This publication made “lean” more accessible to other industries.
Our industry, software, took notice.
DevOps emerged in then mid 2010s. The DevOps movement brought the principles of Lean manufacturing to software development and operations. DevOps emphasized collaboration, automation, and continuous improvement, all principles that can be traced back to Deming.
Gene Kim, Kevin Behr, and George Spafford published “The Phoenix Project” in 2013. The book was directly inspired by “The Goal”. It followed the same format: a wise sensei uses the socratic method to teach struggling manager how transform a struggling IT department. By doing so, the manager discovers DevOps.
This was a seminal moment in software because it fused software development, wholistic thinking of business outcomes, and lean manufacturing into a coherent whole that anyone in IT could relate to.
The crucial concept behind DevOps was the combination of development and operations. At the time, many companies had siloed these into different departments or teams. That created friction between the two. One hand didn’t know what other was doing. Importantly, opportunities to learn from each other were systematically removed by the organization structure.
DevOps wanted to combine the two, thus the name “DevOps”, for the benefit of the system with ultimate aim of providing a better product or service to the customer. This is system’s thinking. Deming talked about this in Japan back in 1950 when he encouraged everyone to view Japan as a system and consider interconnected parts holistically.
The DevOps approach was codified in 2016 with the publication of “The DevOps Handbook”. The message was simple: optimize for flow of value to the customer, collect feedback to adjust, and continuously experiment at improving those two activities.
Deming’s ideas were now firmly planted in software by way of lean manufacturing.
Deming published his seminal work “The New Economics of Industry, Government, and Education” in 1993. This book is the culmination of decades of work on what he called “The System of Profound Knowledge”.
Deming did not know it at the time, but he was the only one in Japan in 1950 with profound knowledge. Seventy years later there are many more in the world with profound knowledge.
The System of Profound Knowledge has Four Parts. I’ll defer to John Willis order and introduction of each part.
The first is theory of knowledge. It focuses on how do we know what we know. The short answer by facts, data, and statistics acquired through empirical activities.
The second is understanding of variation. Variation is part of life. There are ranges of acceptable outcomes that may be statistically quantified. This forms the basis of Statistical Process Control as a means of process improvement.
The third part is understanding of psychology. There is a human aspect to everything. We must understand how choices impact the individual and their behavior.
The fourth part is appreciation of a system. Every system has an aim. Every component of the system must coordinate to deliver that aim. Components should optimize for the aim of the system, not their individual success. Appreciating the system enables thinking holistically about structures, connections, and the constraints.
The System of Profound Knowledge is a lens for viewing the world.
Deming famously demonstrated this with his “Red Beads” experiment. I’ve previously covered this on the podcast, so you can pull that from the back catalog.
The System of Profound Knowledge offers a transformative individual mindset. These ideas transformed those engineers and leaders in Deming’s early lectures. Look at the result.
Hell, indirectly, I’m doing this podcast because of Deming. I did not start with Deming, but I’ve come full circle back where it all began.
Deming’s ideas are not limited to manufacturing. Lean ideas have gone to impact health care, government services, and education. I think Deming saw his ideas as a means to transform the entire world for the betterment of all people. That’s continuous improvement at a global scale.
I’m going to close out this episode with one of my favorite passages from The New Economics. Here we go.
The first step is transformation of the individual. This transformation is discontinuous. It comes from understanding the system of profound knowledge. The individual, transformed, will perceive new meaning to his life, to events, to numbers, to interactions between people.
Once the individual understands the system of profound knowledge, he will apply its principles in every kind of relationship with other people. He will have a basis for judgment of his own decisions and for transformation of the organizations he belongs to.
To me, this passage demonstrates how encompassing Deming intended his system of profound knowledge to be. Once understood, an individual can continually grow the boundaries of their system to wider horizons, from the individual, organization, state, country, and global human society.
Now, hopefully, you also see, how it all does go back to Deming. His teachings, his emphasis on understanding the underlying causes of problems, and his holistic approach to improvement would become the foundation for developing lean which led to DevOps, and now into the wholistic concept of software delivery.
Outro
Alright, that’s all for this episode. Go to SmallBatches.fm/84 for links to the past episodes I mentioned and other wonderful stuff related to the legendary Dr. Deming.
I have cool things planned for Small Batches in 2023. I’ve shared previews with friends, and their feedback is overwhelmingly positive. So stay tuned.
Anyways, I hope to have you back again for the next episode. Until then, happy shipping!