Book Culture

Thoughts on “Implementing Lean Software Development”

Reading and summarizing books on lean software development, so you dont have to. Part 3 (see Part 1 and Part 2).

“Implementing Lean Software Development” written by Mary and Tom Poppendieck and published 2007 at Addison-Wesley. The Poppendiecks are quite famous in the lean-agile software development community, as they published the constitutive book “Lean Software Development: An Agile Toolkit” in 2003, the first (recognized) book about bringing the lean principles to the software development space. The book reviewed here is a successor book aimed at delivering more practical advice. As in the last parts, my review will not focus on re-iterating lean and agile fundamentals, but rather focus on novelty aspects, ideas, and noteworthy pieces.

In the foreword, Jeff Sutherland (co-founder of the Scrum framework) introduces the Japanese terms of Muri (properly loading a system), Mura (never stressing a person, system or process) and Muda (waste):

Yet many managers want to load developers at 110 percent. They desperately want to create a greater sense of “urgency” so developers will “work harder.” They want to micromanage teams, which stifles
self-organization. These ill-conceived notions often introduce wait time, churn, death marches, burnout, and failed projects.
When I ask technical managers whether they load the CPU on their laptop to 110 percent they laugh and say, “Of course not. My computer would stop running!” Yet by overloading teams, projects are often late, software is brittle and hard to maintain, and things gradually get worse, not better.

page xix

In their historical review the authors bring a very interesting statistics which should resonate with many of my peers:

Both Toyodas had brilliantly perceived that the game to be played was
not economies of scale, but conquering complexity. Economies of scale will reduce costs about 15 percent to 25 percent per unit when volume doubles. But costs go up by 20 percent to 35 percent every time variety doubles. Just-in-Time flow drives out major contributors to the cost of variety. In fact, it is the only industrial model we have that effectively manages complexity.

page 5

As evidence, two papers are given: “Time -The Next Source of Competitive Advantage” by George Stalk and “Lean or Sigma?” by Freddy and Michael Balle. Managers and engineers increasingly become aware about the not-so-visible cost of complexity, typically by experiencing project failure or long-term product degradation.

For the aspect of inventory, the authors provide a quite good methaphor:

Inventory is the water level in a stream, and when the water level is high, a lot of big rocks lurking under the water are hidden. If you lower the water level, the big rocks begin to surface. At that point, you have to clear the rock out of the way, or your boat will crash into them. As the big rocks are removed, you can lower inventory level some more, find more rocks, clear them out of the stream, and keep on going until there are just pebbles left.

page 8

That adoption of lean practices and mindset is not straightforward and many organizations struggle or fail to do so is explained by the authors by pointing at a “cherrypicking” approach. Hence, only some activities of the lean domain are adopted in isolation, like just-in-time or stop-the-line. Instead, they a classic:

The truly lean plant […] transfers the maximum number of tasks and responsibilities to those workers actually adding value to the car on the line, and it has in place a system for detecting defects that quickly
traces every problem, once discovered, to its ultimate source.

Womack, Jones, Roos: The machine that changed the world, page 99

I think this cannot be underestimated. To seldom I have seen organizations and management really focussing on the “value creators” and the impediments those are facing.

In earlier blog posts I already wrote about the differences and similarities in the lean manufacturing and lean development. The Poppendiecks provide a table putting both side-by-side (page 14):

Later, in a footnote, the authors refer to a paper by Kajko-Mattsson et al. on the cost of software maintenance. The paper’s sources vary a lot, however its obvious that considering a typical big software project it becomes clear that this ratio quickly translates to millions of Euro/Dollar.

The published numbers point out that maintenance costs between 40% to 90% […]. There are very few publications reporting on the cost of each individual maintenance category. The reported ones are the
following: (1) corrective maintenance – 16-22% […] (2) perfective maintenance – 55% […], and (3) adaptive maintenance – 25% […].

Kajko-Mattsson et al: Taxonomy of problem management activities, page 1

On the lean principle of waste, the Poppendiecks make a simple but revelating statement:

To eliminate waste, you first have to recognize it. Since waste is anything
that does not add value, the first step to eliminating waste is to develop a keen sense of what value really is. There is no substitute for developing a deep understanding of what customers will actually value once they start using the software. In our industry, value has a habit of changing because, quite often, customers don’t really know what they want. In addition, once they see new software in action, their idea of what they want will invariably shift. Nevertheless, great software development organizations develop a deep sense of customer value and continually delight their customers.

page 23

Too often have I experienced software development projects who dont know what their product and the value they provide actually is. Of course, everyone has a vague feeling about what it could be, but putting it in clear words is seldom attempted and easily ends in conflict (a conflict which can be constructive if facilitated well).

On the second principle “Build Quality In”, there are some interesting distinctions on defects and the relation to “inspection”:

According to Shigeo Shingo, there are two kinds of inspection: inspection after defects occur and inspection to prevent defects.10 If you really want quality, you don’t inspect after the fact, you control conditions so as not to allow defects in the first place. If this is not possible, then you inspect the product after each small step, so that defects are caught immediately after they occur. When a defect is found, you stop-the-line, find its cause, and fix it immediately.
Defect tracking systems are queues of partially done work, queues of rework if you will. Too often we think that just because a defect is in a queue, it’s OK, we won’t lose track of it. But in the lean paradigm, queues are collection points for waste. The goal is to have no defects in the queue, in fact, the ultimate goal is to eliminate the defect tracking queue altogether. If you find this impossible to imagine, consider Nancy Van Schooenderwoert’s experience on a three-year project that developed complex and often-changing embedded software. Over
the three-year period there were a total of 51 defects after unit testing with a maximum of two defects open at once. Who needs a defect tracking system for two defects?

page 27

The authors are citing two papers by Nancy Van Schooenderwoert (“Taming the Embedded Tiger – Agile Test Techniques for Embedded
” and “Embedded Agile Project by the Numbers With Newbies“). This resonates well with me, because accumulating too many defect (tickets) is very expensive waste. Its a kind of inventory with the worst properties. To break out of this is not straightforward, I have attempted and failed multiple times to establish a “zero defect policy” (i.e. as long as there is a defect no further feature development happens). In that context let me at two more quotes from the book:

The job of tests, and the people that develop and runs tests, is to prevent defects, not to find them.

page 28

“Do it right the first time,” has been interpreted to mean that once code is written, it should never have to be changed. This interpretation encourages developers to use some of the worst known practices for the design and development of complex systems. It is a dangerous myth to think that software should not have to be changed once it is written.

page 29

On the fifth principle of “Deliver Fast” a very important statement is made:

Caution: Don’t equate high speed with hacking. They are worlds apart. A fast-moving development team must have excellent reflexes and a disciplined, stop-the-line culture. The reason for this is clear: You can’t sustain high speed unless you build quality in.

page 35

Very often I observe a dire need for speed. Of course everyone wants to be faster in the software industry. Competition doesnt sleep. However, similar to unclear definitions of value and products, I have barely ever seen a clear definition of speed in a software project. Or, probably more correct: there were competing definitions of speed on people’s and especially decision maker’s minds. Its a huge difference to beat your team to “push out features now” and grind to a halt when quality activities are started, or to maintain a sustainable pace:

When you measure cycle time, you should not measure the shortest time through the system. It is a bad idea to measure how good you are at expediting, because in a lean environment, expediting should be neither necessary nor acceptable. The question is not how fast can you deliver, but how fast do you repeatedly and reliably deliver a new capability or respond to a customer request.

page 238

The Poppendiecks are summarizing those effects in two vicious cycles (page 38):

For all the lean principles, the Poppendiecks also discuss myths originating from mis-interpreting the principles or applying them wrongly. One which caught my attention was the myth “Optimize by decomposition”. Its about the proliferation of metrics once an organization starts to apply the benefits of visual management. All of a sudden, there are tens if not hundreds of dashboards, graphs, KPIs, and such flying around. Their recommendation:

When a measurement system has too many measurements the real goal of the effort gets lost among too many surrogates, and there is no guidance for making tradeoffs among them. The solution is to “Measure UP” that is, raise the measurement one level and decrease the number of measurements. Find a higher-level measurement that will drive the right results for the lower level metrics and establish a basis for making trade-offs.

page 40

Speaking about myths, they encourage readers to check which myths apply to their situation – certainly a worthwile exercise also for you 🙂

Early specification reduces waste
The job of testing is to find defects
Predictions create predictability
Planning is commitment
Haste makes waste
There is one best way
Optimize by decomposition

page 42

Coming back to the notion of value, the authors are asking the fundamental question how great products are conceived and developed. They write:

In 1991, Clark and Fujimoto’s book Product Development Performance presented strong evidence that great products are the result of excellent, detailed information flow. The customers’ perception of the product is determined by the quality of the flow of information between the marketplace and the development team. The technical integrity of the product is determined by the quality of the information flow among upstream and downstream technical team members. There are two steps you can take to facilitate this information flow: 1) provide leadership, and 2) empower a complete team.

page 52

The book has an extensive chapter on waste with many insightful aspects. I dont want to repeat all of them, and instead just provide some examples. For example I found this statement on the relationship of automation and waste/complexity very inspiring.

We are not helping our customers if we simply automate a complex or messy process; we would simply be encasing a process filled with waste in a straight jacket of software complexity. Any process that is a candidate for automation should first be clarified and simplified, possibly even removing existing automation. Only then can the process be clearly understood and the leverage points for effective automation identified.

page 72

In my current position, automation is a key activity, and we try to automate everything in an endeavour to increase speed, quality and convenience. The quote points out, that automation can hide or defer complexity. I can confirm this. Even though my team automated the complexity of product variants in the build process, our customers (e.g. manual testers) dont have a chance to test all the build we produce. Hence, even made with best intentions, our automation is overloading the whole.

Another good comparison between traditional manufacturing and software development is the following table, putting the seven waste equivalents side-by-side (page 74):

On architectural foresight, I like the following statement:

Creating an architectural capability to add features later rather than sooner is good. Extracting a reusable services “framework” for the enterprise has often proven to be a good idea. Creating a speculative application framework that can be configured to do just about anything has a track record of failure. Understand the difference.

page 76

While discussing Value Streams, the authors dig into effectiveness and efficiency. They are of the opinion that

chasing the phantom of full utilization creates long queues that take far more effort to maintain than they are worth-and actually decreases effective utilization.

page 88

This opinion is not speculation, they provide a good analogy to road traffic and computer utilization:

High utilization is another thing that makes systems unstable. This is obvious to anyone who has ever been caught in a traffic jam. Once the utilization of the road goes above about 80 percent, the speed of the traffic starts to slow down. Add a few more cars and pretty soon you are moving at a crawl. When operations managers see their servers running at 80 percent capacity at peak times, they know that response time is beginning to suffer, and they quickly get more servers. […]

Most operations managers would get fired for trying to get maximum utilization out of each server, because it’s common knowledge that high utilization slows servers to a crawl. Why is it that when development managers see a report saying that 90 percent of their available hours were used last month, their reaction is, “Oh look! We have time for another project!”

pages 101f

I think in daily work, management typically does not pay enough attention to those basics. It is not that this is not known that too high utilization of resouces is bad, quite the opposite is the case in my experience. However, the root causes and the remedies are often not considered. Instead, there is a sentiment of capitulation: “Yes I know our team is stressed and overloaded, but we have to get faster nevertheless.”

In order to reduce cycle times, the authors refer to queuing theory, which provides several approaches:

Even out the arrival of work

Minimize the number of things in process

Minimize the size of things in process

Establish a regular cadence

Limit work to capacity

Use pull scheduling

page 103

In the chapter “People”, there is a lot of reference to William Edwards Deming, a pioneer of quality management. Its an iron of history, that this American actually was teaching the fundamentals of what leater became Lean in post-war Japan, while he was “discovered” only in the 1980s by the US (industrial) public. Deming formulated a what he called “System of Profound Knowledge”:

  1. Appreciation of a System: A business is a system. Action in one part of the system will have effects in the other parts. We often call these “unintended consequences.” By learning about systems we can better avoid these unintended consequences and optimize the whole system.
  2. Knowledge of Variation: One goal of quality is to reduce variation. Managers who do not understand variation frequently increase variation by their actions. Critical to this is understanding the two types of variation — Common cause which is variation from the system and Special cause which variation from outside the system
  3. Theory of Knowledge: There is no knowledge without theory. Understanding the difference between theory and experience prevents shallow change. Theory requires prediction, not just explanation. While you can never prove that a theory is right, there must exist the possibility of proving it wrong by testing its predictions.
  4. Understanding of Psychology: To understand the interaction between work systems and people, leaders must seek to answer questions such as: How do people learn? How do people relate to change? What motivates people?

When pursuing change and transformation, it is very important to take the staff on board. This is easier said than done, because the employees have a very fine sense. They realize very quickly, if for example a certain change in mindset is requested from them, but not exercised by their supervisors. In engineering projects, the demands and expectations of decision makers are often antagonistic to their communicated strategies and visions. Just consider if in your organization “quality” is an essential part of your long-term goals, and totally overriden by daily task force death marches.

The challenge to achieve quality is handled in another dedicated chapter. The authors point out the importance of “superb, detailed discipline” to achieve high quality. Here come the famous “5 S’s” into play. The book’s authors transfer them also to the software space:

Sort (Seiri): Sort through the stuff on the team workstations and servers, and find the old versions of software and old files and reports that will never be used any more. Back them up if you must, then delete them.

Systematize (Seiton): Desktop layouts and file structures are important. They should be crafted so that things are logically organized and easy to find. Any workspace that is used by more than one person should conform to a common team layout so people can find what they need every place they log in.

Shine (Seiso): Whew, that was a lot of work. Time to throw out the pop cans and coffee cups, clean the fingerprints off the monitor screens, and pick up all that paper. Clean up the whiteboards after taking pictures of the important designs that are sketched there.

Standardize (Seiketsu): Put some automation and standards in place to make sure that every workstation always has the latest version of the tools, backups occur regularly, and miscellaneous junk doesn’t accumulate.

Sustain (Shitsuke): Now you just have to keep up the discipline.

page 191

I really enjoyed reading this book and can absolutely recommend reading it. It contains a lot of gems, and is probably one of those book you want to read every other year again to re-discover aspects and connect them to new experience.

Book Culture

Thoughts on “Lean Software Development in Action”

Reading and summarizing books on lean software development, so you dont have to. Part 2 (see Part 1).

“Lean Software Development in Action” written by Andrea Janes and Giancarlo Succi and published 2014 at Springer. The authors are scientists at the University of Bolzano, and the book clearly has a more scientific approach than the last one.

As the last – and probably every book – on the matter of lean, agile and software engineering this book starts with an introduction on each of those aspects. Again, I will not reiterate on what lean and agile are, but focus on interesting observations and perspectives exposing new angles on “known stuff”.

The first noteworthy piece is about “tame and wicked projects”. This section is referring to work by Rittel and Webber who came up with a discution between tame and wicked problems (see also). Poppendieck and Poppendieck extended this to projects, and in this book they are described and identified by following ten points:

  1. Wicked projects cannot provide a definitive, analytical formulation of the problem they target. Formulating the project and the solution is essentially the same task. Each time you attempt to create a solution, you get a new, hopefully better, understanding of the project.
  2. Wicked projects have no a stopping rule telling when the problem they target has been solved. Since you cannot define the problem, it is almost impossible to tell when it has been resolved. The problem-solving process proceeds iteratively and ends when resources are depleted and/or stakeholders lose interest in a further refinement of the currently proposed solution.
  3. Solutions to problems in wicked projects are not true or false, but good or bad. Since there are no unambiguous criteria for deciding if the project is resolved, getting all stakeholders to agree that a resolution is “good enough” can be a challenge.
  4. There is no immediate or ultimate test of a solution to the targeted problem in a wicked project. Solutions to such projects generate waves of consequences, and it is impossible to know how these waves will eventually play out.
  5. Each solution to the problem targeted by a wicked project has irreversible consequences. Care must be placed in managing assumed solutions. Once the website is published or the new customer service package goes live, you cannot take back what was online or revert to the former customer database.
  6. Wicked projects do not have a well-described, widely accepted set of potential solutions. The various stakeholders may have differing views of what are acceptable solutions. It is a matter of judgment as to when enough potential solutions have emerged and which should be pursued.
  7. Each wicked project is essentially unique. There are no well-defined “classes” of solutions that can be applied to a specific case. It is not easy to find analogous projects, previously solved and well documented, so that their solution could be duplicated.
  8. The problem targeted by a wicked project can be considered a symptom of another problem. A wicked project deals with a set of interlocking issues and constraints that change over time, embedded in a dynamic and evolving context.
  9. The causes of a problem targeted by a wicked project can be explained in several ways. There are several stakeholders who have various and changing ideas about what is the project, its nature, its causes, and the associated solution.
  10. The project must not go wrong. Mistake is not an option here. Despite the inability to express the project solution analytically, it is not allowed to fail the project.
page 9

These are some interesting observations which resonate with my experience. One might say “many of those points do not apply to our project as we have a quite clear understanding of our product delivery (like an ECU for a car, providing a certain set of functions/features for the customers)”. However I think its not that simple. Many projects of non-trivial complexity I have been involved in do not only have the goal of releasing a product to the market, but there are other, interlinked objectives which give the project in total a semi- or non-defined goal. Besides delivering a good, innnovative product these objectives may consist of e.g. financial goals (better return on investment), efficiency gains, usage of new technologies and approaches and in- or outsourcing of activities. While project I know typically start with those defined on a high-level and people are onboarded or recruited referring to those motivating goals, as soon as the project enters the death march the project’s goals are gradually getting more fuzzy, unbalanced and volatile. I don’t know silver bullets to such situations (yet), but the notion of wicked projects resonates with those and other observations. However, isn’t awareness the first step to improvement?

[…] the quality of the final product is seen as a result of the process pro-
ducing them. This assumption creates a high attention for a high-quality production process according to the credo “prevention is better than healing”

page 41

This lean wisdom sounds trivial, however, I have never seen it realized. I have yet to understand why so many managers ignore the efficiency and effectiveness gains by a proper process and instead decide to continously apply brute force which cost them more money, time, energy, motivation and subsequently of course quality. And, of course, by a “proper process” I dont mean an perfect considers-everything-process which is both unreachable and undesirable.

“The role of standardization”, page 43

I like this illustration showing standardization as a wheel chock for the plan-do-study/check-act cycle. Similar to the paragraph above it strikes me how many projects and managers re-invent the wheel (haha) and then start the whole process from the bottom. Of course no one wants and says this, but this is what often happens.

The result is a development approach in which requirements are not “refined down to an implementation,” i.e., taken as the starting point to develop an implementation that represents those requirements, but where the business objectives are mapped to the capabilities of the technical platform to “equally consider and adjust business goals and technical aspects to come to an optimal solution corresponding to the current situation

page 63

This is another insightful depiction. It provided me a new perspective on requirements, as they help to close the gap between business goals and technical capabilities. This can be a good approach helping in situations in which a project is lacking a good “feeling” on the right amount of requirements, between over-specification and under-specification.

page 80

Referring to studies conducted by Herzberg this shows how motivators and hygiene factors influence the motivation of the staff, and how agile methods very well support/complement those.

Later the authors come to write about the “dark side of agile”, on which they also published a paper. As observed by others before, agile statements can be easily twisted and thwarted in good or bad faith to yield an abomination which leads to the opposite and extreme positions of the original intention. Citing Rakitin’s old paper the agile manifesto can be translated as

  • Individuals and interactions over processes and tools: “Talking to people instead of using a process gives us the freedom to do whatever we want.”
  • Working software over comprehensive documentation: “We want to spend all our time coding. Remember, real programmers don’t write documentation.”
  • Customer collaboration over contract negotiation: “Haggling over the details is merely a distraction from the real work of coding. We’ll work out the details once we deliver something.”
  • Responding to change over following a plan: “Following a plan implies we have to think about the problem and how we might actually solve it. Why would we want to do that when we could be coding?”
page 111

The above is one way to twist the agile manifesto in favor of what the authors a few paragraphs later call a “cowboy coder”. This reminds of the “Programming, Motherfucker” webpage (thanks, Kris). While such sentiment exists, I cant often blame the engineers on mocking the agile manifesto and similar approaches that way. Very often, such engineer’s reactions are preceeded by even more unfaithful perversions pushed by all sorts of management. Just to bring one example to the table: Who has not witnessed a manager throwing new (vague) requirements at the development team every other week claiming this is what agile is about and, of course, everyone has to be faster in reacting. Because agile is faster. I could go on.

In chapter 6 the authors start to synthesise and bring lean and software development together. They start with citing the seminal book of Poppendieck and Poppendieck “Implementing Lean Software Development From Concept to Cash”. I couldnt read it yet as it wasnt available in print when I tried to get a hold of it. So they provide 7 principles for lean software development:

  1. Eliminate waste;
  2. Build quality—we used the terms “autonomation” and “standardization”;
  3. Create knowledge;
  4. Defer commitment—we used the term “just-in-time”;
  5. Deliver fast—get frequent feedback from the customer and increase learning through frequent deployments;
  6. Respect people—we used the term “worker involvement”;
  7. Optimize the whole—we used the term “constant improvement.”
page 131

Nowadays, those principles may sound obvious. However, the Poppendieck book was published in 2006 and I think at that time many if not all of those principles where both not clear nor any best practices and tooling was available back then to realize them.

In a break-out box a comparison between lean and agile is given

  • Agile Methods aim to achieve Agility, i.e., the ability to adapt to the needs of the stakeholders.
  • Lean production aims to achieve efficiency, i.e., the ability to produce what the stakeholders need with the least amount of resources possible.
page 144

After this section which gives some more references to earlier work the book enters its less interesting but extensive part. Janes and Succi present three methods which shall support lean software development. They all have their merits, but I have to admit I dont catch fire for any of them.

Let me sketch those methods in short: First they introduce the “Goal Question Metric (plus)”, short GQM+. GQM+ is based on GQM, which is a methodology to derive crisp business goals. While I find some of the leading questions worthwhile, the overall concept strikes me as overly complex and hard to grasp.

After this, the authors present the “Experience Factory”. This is essentially an extension of the classic plan-do-study/check-act cycle with additional steps and a “sub-cycle”. Its a semi-interesting read, but doesnt convince me in its current form.

Finally, the concept of “Non-invasive Measurement” is laid out. The goal of this approach is to collect data without distracting the engineers. While such non-invasiveness is indeed desirable, the proposal seems overly complex to me. I mean, there are so many ways of analyzing process flows, code quality, efficiency, etc. Why do the authors describe a database scheme for a concrete solution.

All-in-all the book “Lean Software Development in Action” didn’t convince me. Its best part is where lean and agile are described, and the book offers a few interesting new perspectives on them to an already somewhat informed reader. Those aspects I have mostly covered above. The part where the authors bring in their ideas for methodologies which augment existing known approaches is rather weak, probably because its about academic ideas with little (not none!) exposure to real project life.

Book Culture

Thoughts on “Lean-Agile Software Development”

Reading and summarizing books on lean software development, so you dont have to. Part 1.

Besides agile philosophy, practices, processes & methods, lean becomes an increasingly recognized topic around software development. At least I can say that about my peer group. After an initial training on the matter in which I learned about the “general lean” practices from the industrial production area, I had a lot of questions and doubts about its applicability in software development. Of course there are obvious connections and transfers which one could try, but I was wondering about existing experience, studies, research and best practices. So I was checking out the available books and found three. Two of them I already read, and today I want to start with the first (which I actually read second). Please note: I will not provide introductions and details on neither Lean nor Agile, as for such there is a myriad of online resources available, and I assume my readers know at least the agile part very well. Also, as usual in my book reviews, I am less focused on how well-written a book is. My focus is on new thoughts, inspiring ideas, surprising perspectives and generally speaking everything which deserves an application in my professional life (and the ones around me).

“Lean-Agile Software Development – Achieveing Enterprise Agility” written by Alan Shalloway, Guy Beaver and James R. Trott was published in 2010, which means that in the fast-paced software industry its already quite old. For this book this was not a downside, as I could compare their takes against current state-of-the-art.

In the foreword Alan Shalloway makes an interesting observation:

Too long, this industry has suffered from a seemingly endless swing of
the pendulum from no process to too much process and then back to no process: from heavyweight methods focused on enterprise control to disciplined teams focused on the project at hand.

page xviii

I can confirm this from various scales. On a grand scale this has been true when you look at the software development history, starting decades ago. Enterprise processes, which followed the wild-west of the early days of computing, were replaced by agile practices. Even more, even within an organization down to project-level and individuals, the continued conflict about the “right amount of process” is probably the biggest philosophical debate around in software development. Shalloway claims that lean principles can “guide us in this” and “provides the way”. Lets see.

On page xxxviii the authors summarize “core beliefs of Lean”, preceeded by core beliefs of Agile and Waterfall. As all of those are not taken from “canonical” sources, let me share the lean ones here, as they are a first cood summary:

Even when applied to software development, Lean is not limited to software development teams alone. On page 7 a tables lines out the contributions from all parties:

This is easier said than done. In many organizations both business and management are focused on pushing and tracking the delivery team, but spend too less time on their contributions. Another thing noteworthy here is the notion of the “delivery team”. This is not a team supporting, testing, integrating and generally taking care of delivery, no this is actually the software development team. Hence, this seems a synonym to more widely used terms like “feature teams”. I like the term delivery team, and could think about combinding both in “feature and delivery team”. Each term focuses on one aspect, the former more about the product, the latter more about the activity. In modern software development, I think its crucial to combine both in one team. Diminishing on of both aspects will inevitably lead to a suboptimal efficiency because essential parts are outsourced to other teams.

Lean principles suggest focusing on shortening time-to-market by removing delays in the development process; using JIT methods to do this is more important than keeping everyone busy

page 8

This is very valuable statement. Too often I see engineers getting dragged into task forces just because in the moment they are not overloaded. As a consequence, this leads to a culture in which everyone wants to be perceived or actually be busy all of the time. Continuous busy-ness is not sustainable and leads to growing organizational and technical debt. The cited statement instead clarifies that a lean and efficient process doesnt correspond to a process in which everyone is busy all of the time. Essentially we are talking about different dimensions.

Eliminating waste is the primary guideline for the Lean practitioner. Waste is code that is more complex than it needs to be. Waste occurs when defects are created. Waste is non-value-added effort required to create a product. Wherever there is waste, the Lean practitioner looks to the system to see how to eliminate it because it is likely that an error will continue to repeat itself, in one form or another, until we fix the system that contributed to

page 10

While reading this, it comes to my mind that everything which is not automated which can be automated is also a waste. Manual execution is inherently more error-prone in any software process.

Developers tend to take one of two approaches when forced to handle some design issue on which they are unclear. One approach is to do the simplest thing possible without doing anything to handle future requirements. The other is to anticipate what may happen and build hooks into the system for those possibilities. Both of these approaches have different challenges. The first results in code that is hard to change. […] The second results in code that is more complex than necessary. […]

An alternative approach to both of these is called “Emergent Design.” Emergent Design in software incorporates three disciplines:

  • Using the thought process of design patterns to create application architectures that are resilient and flexible
  • Limiting the implementation of design patterns to only those features that are current
  • Writing automated acceptance- and unit-tests before writing code, both to improve the thought process and to create a test harness

Using design patterns makes the code easy to change. Limiting writing to what you currently need keeps code less complex. Automated testing both improves the design and makes it safe to change. These features of emergent design, taken together, allow you to defer the commitment of a particular implementation until you understand what you actually need to do.

Page 11f

Conflicts around the aforementioned two approaches are indeed quite common. Both sides typically are able to throw business needs into the ring (pragmatism vs. sustainability). Even more, I often observe conflicted parties which did take the opposite position in the last conflict. Hence, emergent design sounds like a promising middle ground. I already have ideas in which conflicts I may bring it to the table.

Table 1.2 lists a good transfer of the industrial production costs and risks to the software world, something my first training on lean was missing out on:

On assigning persons to multiple projects at the same time, the authors cite an interesting study by Aral, Brynjolfsson and Van Alstyne. This study showed that the overall productivity of on person is reduced by 20% for the second and third parallel project, each. This is huge, also considering that often the better engineers are pulled/pushed into multiple projects/teams to rescue them. As a result, the best engineer’s capacity is reduced and thinned.

In the chapter about “Going beyond Scrum”, there is a good summary of Misunderstandings, Inaccurate Beliefs, and Limitations of Scrum:

Misunderstandings commonly held by new Scrum practitioners

  • There is no planning before starting your first Sprint.
  • There is no documentation in Scrum.
  • There is no architecture in Scrum.

Scrum beliefs we think are incorrect

  • Scrum succeeds largely because the people doing the work define how to do the work.
  • Teams need to be protected from management.
  • The product owner is the “one wring-able neck” for what the product should be.
  • When deciding what to build, start with stories: Release planning is a process of selecting stories to include in your release.
  • Teams should be comprised of generalists.
  • Inspect-and-adapt is sufficient.

Limitations of Scrum that must be transcended

  • Self-organizing teams, alone, will improve their processes beyond the team.
  • Every sprint needs to deliver value to the customer.
  • Never plan beyond the current sprint.
  • You can use Scrum-of-Scrums to coordinate interrelated teams working on different products.
  • You can use Scrum without automated acceptance testing or up-front unit tests
page 84

I will not comment on each point. The first two sections I would confirm entirely. The last section is pointing at some “missing” aspects in Scrum, but I think just because e.g. test-driven development is missing, its not a limitation. Scrum is not claiming to be describe every aspect of software development.

In general: tables. This book really contains some nice side-by-side comparisons in tabular form. Table 5.1 compares “Scrum and Lean Perspectives”:

The book is also strong in naming typical anti-patterns in agile execution, especially when those anti-patterns clash with lean mindset.

Some common anti-patterns for Scrum teams are

  • Stories are not completed in an iteration.
  • Stories are too big.
  • Stories are not really prioritized.
  • Teams work on too many things at once.
  • Acceptance tests are not written before coding starts.
  • Quality Assurance/Testing is far behind the developers.

Here are questions we always try to use.

  • Does the team’s workload exceed its capacity?
  • When was the last time you checked your actual work process against the standard process?
  • When was the last time you changed the standard process?
  • Where are the delays in your process?
  • Is all of that WIP necessary?
  • How are you managing your WIP?
  • Are developers and testers in sync?
  • Does the storyboard really help the team keep to its work-flow?
  • Are resources properly associated with the open stories?
  • How much will limited resources affect the team’s work?
  • What resource constraints are you experiencing?
  • Can these constraints be resolved with cross-training or are they something to live with?
  • Does the storyboard reflect constraints and help the team manage them?
  • What needs to be more visible to management?
  • How will you manage your dependencies
page 95f

The authors clearly are not satisfied with the amount of guidance provided by and the reality around Scrum. “Going beyond Scrum”, they present their own extended Scrum called “Scrum#” in two pages. They also introduce Kanban as a simpler framework. A key concept already mentioned in the last citation, even more relevant for Kanban, are “work in progress limits”. WIP limits was a known concept to me since some years, learned from former colleagues. The relationship to lean, however, was new to me and it makes total sense. Focus is soooo important, it cant be overrated. In my own experience it would say around 50% of all issues in software projects originate from lack of focus and too many things going on in parallel. Finally in its comparison of process frameworks, the authors do not forget about Extreme Programming.

Before you write a line of code, set up the following:

  • The product
  • The team
  • The environment
  • The architecture
page 109

This sounds simple, still I never experienced a software project in which more than one of those points was clear to a basic degree. Too often, organisations spawn projects with a very fuzzy project idea, an undefined team, unknown environment and a notion of “architecture will be clarified along the way”. The book goes on to provide guidance on how to set up each points before the first development iterations are started. On page 140 the authors present a template for how to draft a product vision statement.

The book also spends a chapter on “The Role of Quality Assurance in Lean-Agile Software Development”. It has, in essence, one key recommendation which is Test Driven Development (TDD). They claim “The role of testers must be one of preventing defects, not finding them”. While TDD has its merits, I think this statement is too simple. It is a bit far from reality to expect testers to have tests ready before every implementation, especially in projects in which even technological basics are not remotely clear. On the other hand, I am not saying TDD is not recommended wherever it can be applied.

If the customer cannot or will not confirm that you have delivered what they want, you should simply state that you believe that the customer does not value the feature; that it is not a high priority. If it were valuable, they would make the effort to specify the tests. Moreover, you should tell your management team that you recommend not building the functionality. If you are required to build it anyway, go ahead, but know that it could well turn out to be a waste of time.

page 164

This is quite some radical take, and it its a bit hard to match to a typical setup in which customers typically do not care about the test cases. However, I can imagine that pushing for a project setup in which tests have such a crucial position that both customers and developer give them utmost priority can nothing but benefit the efficiency of the resulting project. The authors recommend to always keep the question “How will I know I’ve done that?” in mind, which according to them is an ultimate tool tom avoid waste.

After focusing on the team-scale, the book goes on to widen the scope to full enterprises. Also here they provide some anti-patterns (excerpts):

  • Teams are not well formed.
  • Large batches of unprioritized requirements are pushed through the organization.
  • There is no mechanism to limit work to resource capacity.
  • Program managers and business sponsors compete for resources rather than working together to maximize the return on them.
  • Automated acceptance testing is not being done. Test-driven development also is not being done. Testing is initiated too late in the development cycle.
  • Code quality is left up to programmers’ personal beliefs.
  • Finding and removing the root causes of problems is not pursued aggressively. Bugs are tolerated as a way of life in the software world. In fact, many organizations utilize bug tracking as status for release readiness.
  • Continuous process improvement is not practiced or valued. Most companies are so busy trying to fix the latest crisis that there is no time to focus on process improvement to avoid causing the next one.
Page 171f

This is a good list, of course its rather examples than extensive or complete. The second item from the top is re-iterated on page 182 when the authors state “For example, it is common for management to track the number of unfixed bugs. It seems like natural approach to assess how a team is doing. Lean-Agile thinking uses a different approach: Instead of worrying about fixing the bugs, we should concern ourselves with what is causing them.”

In my opinion, all or most of the above points originate from a lack of discipline in the management team, leading to evasive activities with the above symptoms. For examples it is much simpler to track bug lists than to solve root causes in the organization. It is mentally simpler to run from fire to fire than to reflect about fundamental improvements for the process. It is simple to request new reports in every escalation meeting than to use existing ones continuously to create a sustainable frame for the development team. Who is to blame? I think its a management culture based on 100% meetings giving almost no time to reflect and short-sighted office politics. It speaks for the authors, when Alan Shalloway write:

Some people are natural managers; I am not one of them. Historically, I have always micromanaged. Because I am good in a crisis (often creating and then solving them), when one occurred I would tend to jump in and tell my team how fix it. I knew that this behavior was inhibiting the team’s growth, so I tried delegating—letting the team figure out how to do things on their own—often with very poor results.

I was really abdicating via delegation. I needed to find a way to let the team figure out the solution but remain involved enough to ensure that it would be a good one. Fortunately, Lean management provides a way to do this. With visual controls, I can see the team’s process—I can see how the team is doing at any time—and I can see the team’s outcomes.

If the team gets into trouble, I can actively coach them to improve their results without telling them what to do. Lean gives me a way to become a better manager without resorting to old habits.

page 190

Already earlier, we have touched on software architecture. In a separate chapter the authors dive more into the question how to find the sweet spot between too much and too less architectural work.

Build only what you need at the moment and build it in a way that allows for it to be changed readily as you discover new issues.

page 204


The purpose of software design is not to build a framework within which all things can fit nicely. It is to define the relationships between the major concepts of the system so that when they change or new requirements emerge, the impact of the change s required is limited to local modifications

page 208

Almost at the end the book comes to speak about the origins of lean at Toyota. Interestingly, they are highlighting that

One of the brilliant insights at Toyota was that Lean principles are implemented differently in manufacturing than they are in product development.

This gave rise to another great example of Lean: the Toyota Product Development System, which is a better example for us in software development.

page 215

Now this is one revelation. So far all trainings were about Toyota’s Production System, not the Product Development System. This makes me wonder if our sources are the right ones.

With that let me close this review. This book was a good read, it explained the existing frameworks, showed their flaws and issues both in theory and practice, made concrete recommendations. All in all this book is a recommendation if you want to read how agile and lean can be combined in state-of-the-art software development processes.

Book Culture

Thoughts on “Reinventing Organizations” by Frederic Laloux

Next in my reviews of work-related books we have “Reinventing Organizations” by Frederic Laloux (the illustrated, shortened version). This is going to be short, I essentially read the book in an afternoon.

After some introductions, Laloux gives a historical overview on organizational philosophies, based on Ken Wilber. It starts at the impulsive, pure power-based via hierarchical via modern performance-oriented philosophies to postmodern pluralist ones. One interesting aspect here is, that even the two “older” ones can still be experienced today, the first one e.g. in street gangs/mafia, the second one e.g. in the church or public administration – and of course many companies.

The latter is based on empowerment, values and integrates diverse interest groups, and probably is by most considered already quite new or in many organizations “still upcoming”. However, Laloux claims that we may experience aklready now the dawn of a new, next philosophy, a integral, evolutionary one. He gives names three breakthroughs, which require but also trigger this. First, the increasing complexity of the systems organizations need to cope with, and the need for self-organized, decentral organizations to be able to cope with that. I would say that this approach “extrapolates” the empowerment by “built-in” trust and top-down-ruling-inhibiting organization forms in small, completely autonomous entities.

The second breakthrough is “wholeness” (“Ganzheit” in my German version of the book). It is based on the view that at their workplace, many people show and use only a part of their personality. This leads to loosing many valuable aspects and creates an unhealthy mental tension. Laloux gives some examples how this can be reliefed, e.g. by medition, group-reflection and new meeting-moderation methodologies.

Third, there is a breakthrough called “evolutionary purpose”. Its probably a reference what more recent publications and consultants call “VUCA”(Volatility, Uncertainty, Complex, Ambiguous). According to that perspective–which I have some doubts on, but maybe more in another blog post–the world around is is increasingly “vuca”. Traditionally, we would try to fight this vuca-ness, trying to impose existing problem-solving techniques–and ultimately fail. The other mental approach is to accept, even embrace this situation and–how Laloux phrases is–“dance with it”. Here, Laloux makes an interesting claim: “Most of us react very cynical on mission statements”. Oh yes!

The book contains some interesting food for thought, and it certainly has some relevant observations. Where my personal doubts come in: If I look at an historic scale, the four “traditional” organizational philosophies have been in place 10.000s, 1.000s, 100s and 10s of years, respectively. As I wrote, especially the last one (postmodern, integral) is still a quite new kid on the block. What does it tell us that new even a new philosophy comes up, and if we extrapolate the logarithmic timescale even more, will it be replace in a few months? Just kidding, however I wonder if Laloux’ evolutionary model is just a sub-type of e.g. the postmodern one, or really a new philosophy. And of course – what comes beyond that. Will there be an even more “advanced”, “progressive” organizational philosophy soon? Will things break apart and we go back to the wild days? I dont know…

Book Culture

Thoughts on “How Google Tests Software”

During my recent years as Software Project Manager I learned to appreciate modern software testing beyond just a tool. In earlier blog posts I described my personal hands-on progress on the matter. Thanks to some teachers and mentors I gained knowledge in theory and practice and nowadays call myself a software testing enthusiast (which doesnt mean I am particularly good at it – I just enjoy the topic).

The Book “How Google Tests Software” from James Whittaker, Jason Arbon and Jeff Carollo is already some years old – a fact I didnt know about initially, but some aged concepts made me look into the publishing date. Also some references are not working anymore, including a link to Google’s own “Google Test Automation Conference”. However, its still a great summary on aspects of modern software testing, with a focus on organizational aspects. There is no mention of mutation testing, probabily-based testing and other testing approaches I would call “advanced”, instead there is a lot of focus on the specific roles in and around testing at Google.

In the forewords the authors’ boss Patrick Copeland introduces the term of “Engineering Productivity” and why he has chosen this as the department’s name. I find this term very appealing, especially in its connection to testing. In my domain, automotive, testing is often seen as something which slows projects down and its done in the very end after features have been completely implemented to “drive maturity”. For me, proper testing is a method to be faster after all, and doing it late is possible but is for from being efficient. The Engineering Productivity organization at Google gathers the lead on test activities, respective tooling around testing and the mentoring/training efforts to spread a testing mindset throughout the whole organization.

In the book, the saying “scarcity brings clarity” is coming up multiple times, and the authors say

The first piece of advice I give people when they ask for the keys to our success: Don’t hire too many testers.

How Google Tests Software, page 4

Googlers might be smart, but they are not plentiful. Every TEM we’ve ever hired from outside of Google makes the comment that their project is understaffed. Our response is a collective smile. We know and we’re not going to fix it. It’s through knowing your people and their skills well that a TEM can take a small team and make them perform like a larger team.

How Google Tests Software, page 188

As a young leader I dread the constant understaffing all around me. Such looks into highly regarded companies like Google make me believe its something inevitably in any fast-paced company. Being always short on good people compared to the tasks at hand must yield the most efficient use of people as possible. But how to balance this in a way which satisfies me is something I admit to not have found the key for yet.

In major parts of the book some key roles are described in detail with their relevance to testing: The (“normal”) Software Engineer (SWE), the Software Engineer in Test (SET) and the Test Engineer (TE). On page 7 the roles are summarized in a nutshell, while many many pages repeat and detail this again and again – a style I find very typical for American textbooks. In essence its clear and I assume my mentors must have either read the book years ago or used common root sources. The worksplit seems obvious to me now. SWEs are developing the code, but have to contribute automated testing on all levels to secure the functionality and quality of the system, even when no other test-focused roles are around. SETs are pushing the boundaries of test tooling, enabling new kinds or aspects of test automation wherever they can. TEs are driving testing especially in a “challenger” approach, to catch all the issues SWEs don’t see or dont want to see initially.

I already mentioned that the book doesnt go very deep into test technology. What I found very interesting is that Google separates their test activities simply by “small tests”, “medium tests” and “large tests”. According to the authors they chose to do this because other terms are overloaded and vague throughout the software industry do a degree that people are talking about different things and often didnt recognize.

On pages 40f some nowadays state-of-the-art concepts about how to include test automation into pipelines are brough up. E.g. having test code near to the functional code. Also the classification of small/medium/large tests is added with some concrete data: large tests have to run within 15 mins, small tests are required to execute in less than 100ms. This shows the enourmous compute power Google is able to leverage for their software development. Also there is a good note to quote:

Small tests lead to code quality. Medium and large tests lead to product quality.

How Google Tests Software, page 47

Only both in combination provide long-term value to a company. Having insufficient small tests strongly correlates to bad code quality which will slow down further development. Having insufficient medium and large tests (system tests) lead to many gaps in the functional and non-functional end-to-end testing, thus yielding bad product quality if there is not immense manual testing in place.

On page 48 two very basics rules for test cases are defined, which I find highly relatable: Tests have to be independent of each other so they can be executed in any order (and in parallel). Also tests must not have any persistent side effects.

Interestingly the book makes some references to a build and test execution system with sophisticated dependency graph analysis, without giving the tool’s name. Nowadays I know that they are referring to Bazel. The book must come from a time when it was not yet published.

On pages 54ff the authors describe a “test certified” program which they used to promote the testing inside Google, including some test certified levels for teams to achieve. In a very telling interview I learned a lot that even at Google the testing proponents had to overcome a lot of internal resistance.

The dream of a unified dashboard has haunted Googlers, much like it has at many other companies. Every year or so, a new effort tried to take root to build a centralized bug or project dashboard for all projects at Google.

How Google Tests Software, page 124

Its late and I want to finish this blog post, so I am closing with an inspiring quote from Joel Hynoski, answering why someone should pursue a career in test:

Test is the last frontier of engineering. We’ve solved a lot of the problems of how to develop software effectively, but we’ve still have a green field of opportunity to attack the really meaty problems of testing a product, from how to organize all the technical work that must get done to how we automate effectively, and responsively, and with agility without being too reactive. It’s the most interesting area of software engineering today, and the career opportunities are amazing. You’re not just banging on a piece of software any more, you’re testing the GPU acceleration of your HTML5 site, you’re making sure that you’re optimizing your CPU’s cores to get the best performance, and you’re ensuring that your sandbox is secure.

How Google Tests Software, page 206
Book Coding Culture

Thoughts on Refactoring from Martin Fowler

In my series of book reviews on classics its time to take on another evergreen: Refactoring from Martin Fowler. It was already referenced a multiple times in the earlier books and now it was time.

The concept of refacoring is well-established nowadays, and I would say that only a company culture which incentivizes and motivates a spirit of constant refactoring can be a state-of-the-art software company. In my experience, this is easier said than done, especially when immense release pressure and tight resources lead to the well-known vicious circle of crunching and task forces, which never end. When one release was successfully squeezed out “somehow” and the next one is already knocking on the doors, how could anyone expect refactoring of code which actually “works” (it made it into the last release after all!!!). I had such discussions many times, and until today I dont feel strong enough in my arguments to convince senior managers in typical situations. Arguing with sustainable code and continuous improvement when the other side virtually puts the existence of the whole organization at risk is an uphill battle. But its worth fighting.

In essence when you refactor you are improving the design of the code after it has been written. […] With refactoring you can take a bad design, chaos even, and rework it into well-designed code. Each step is simple, even simplistic. You move a field from one class to another, pull some code out of a method to make into its own method, and push some code up or down a hierarchy. Yet the cumulative effect of these small changes can radically improve the design. It is the exact reverse of the normal notion of software decay.

Martin Fowler: Refactoring, page 9

I like this simple but strong definition, which underlines the cumulative effect of small improvements. Refactoring is not the same as the “grand redesign” or “from scratch” approaches which are often taken and loved, especially when the staff (developers, managers) is changed. Refactoring can achieve the same goals with either the same or a different staff.

After some introductory words Fowler goes ahead with a concrete example. He emphasizes the repetitive nature of changing something a little bit, then running an extensive unit test suite and then committing the changes to a repository. This approach is something I immediately started to exercise. For that I had to setup some of my testing scripts to work locally (they were purely in my CI before), but the effort was totally worth it. With my unit test coverage being 98% nowadays plus mutation testing in place, I dont have to worry a lot to break existing functionality while refactoring, and as a lost resort I have all the extensive API and acceptance testing in my CI before I can merge on master.

The true test for good code is how easy it can be changed.

Martin Fowler: Refactoring, page 77 (translated back to English)

This quote may be controversial among people with low exposure to professional software development (juniors, managers who gave up on programming a while ago), but for professionals in my environment its fortunately common sense. Which doesnt mean that its applied constistently, though.

If someone tells you that their code during refactoring didnt work for a few days you can be quite sure that they didnt apply refactoring as such.

Martin Fowler: Refactoring, page 79 (translated back to English)

Such reworks or redesign of larger parts of a codebase of course can and may happen, too, but I appreciate Fowler’s take on a clear separation. Especially in arguments with upper management, clear terminology can help. If refactoring can mean everything happening to a codebase except adding new functionality, its probably to vague. Restricting its meaning to pure tiny and small changes reduces risks a lot and may help to increase acceptance.

Refactoring Helps You Program Faster

Martin Fowler, Refactoring, page 82

This was and is also my personal strong belief, and I think there is enough evidence that this is a fact. I mentioned earlier sustainability, and if a well-design and maintained code base based on clean code principles and with constant refactoring exists, its the best way to get a graph like the red one below.

Source: https://frontendwebblog.wordpress.com/2020/07/07/notes-from-refactoring/ (who presumably has taken it from Martin Fowler: Refactoring, English version)

Note also the short part in the lower left corner, where poor design for a while may have more functionality than good design. Naturally, with hacky proof of concepts hitting production, you can push out some functionality “fast”, but at the cost of low efficiency later on. On pages 90f (German version) declares this the major argument in favor of refactoring – in the end its about the business value it provides, and that one can be immense.

On pages 86f Fowler discusses if one should reserve dedicated time for refactoring. I heard before of models like “every third sprint is a refactoring sprint” and of course situations where “management decided the next sprint is for refactoring to fully focus on features after that again”. Fowler argues that reserved refactoring slots shouldnt be a thing. I tend to agree with him, by stating that refactoring should be considered in day-to-day work efforts and not be a sepereate activity in contrast to “normal development work” (which it should be a part of). However, as the value of refactorings is hard to measure and aforementioned “implicit” integration into the developer workflow easily gets it ignored or watered down, some accompanying activities and events encouraging refactoring may provide benefit.

What Do I Tell My Manager? […]

Of course, many people say they are driven by quality but are more driven by schedule. In these cases I give my more controversial advice: Don’t tell!

Martin Fowler: Refactoring, page 89

Similar to Bob Martin, Fowler argues with refactoring being an essential part of professional software development, thus its not something a manager should even have the change to interfere with in particular. I agree – mature develops should just do it and get the time needed. Asking management for approval for such a core activitiy in the actual coding, that it would be weird to explicitly ask for it. Do you ask for permission to apply certain design patterns, too? Of course, when asked, management will respond and the answer in probably 80% not what you would do. Management will probably realize at some point of refactoring gets too extensive (if that is even possible), but in the end refactoring is for everyone’s best (see the graph above).

On pages 93f Fowler has some interesting input on the interaction of (feature) branches and refactorings which I didnt see before. In essence its simple: Both are working in opposite directions. If there are multiple branches living in parallel and in one of them refactoring is happening, it may make the other branches unmergable (and vice-versa). So if a company or project wants to encourage refactoring, it should avoid active branches wherever possible.

There are more arguments for refactoring and a myriad of hints and tricks how to apply it, when to apply it and how to carry it out in an optimal way. The second part of the book contains detailled descriptions of all types of refactorings. I have skipped through them, but I know I wouldnt remember them exactly anyways. However, with the gist of things in mind I did a major refactoring of my current pet project, and it was really fun and productive. Now my codebase is even cleaner (I did already a few rounds after reading Clean Code) and I am looking forward repeat it while I continue adding functionality.

It was a very good book and a certain recommend, if you didnt read it already.

Book Coding Culture

Thoughts on Clean Agile from Robert C. Martin

My third book review leads me to the third book in a row from Robert C. Martin. You may think I am a fanboy or such, but its just that as written before all those books and more I got provided from my employer, and somehow I liked the flow of Martin’s books, so I read them one after the other. But fear not, the next book will be from another author.

So “Clean Agile”: Already the title is a bit triggering. I remember a online talk from another big agile representative (forget the name though) who was very clearly discouraging to use “Agile” as a noun for anything, but always use it in its grammatically correct way as an adjective. Thats what I thought about when holding the book in my hand. However, I dont really care for the difference. Grammar is the one side, the other is that agile/agility has been a standing term in the sw industry for a while now, so one may as well use it in its noun form. Nevermind.

The first thing noteworthy is the history lesson about the waterfall process. According to Martin, the 1970 paper from Winston Royce which according to popular belief introduced waterfall and initiated its establishment, was actually arguing against this model. If that is true, a lot of people must have read only the first part of that paper and then went on implementing that part – because it chimed well with the notions of Scientific Management which were cool back then.

Martin is also very explicit about the small vs big team problem. In his opinion the organization of big teams is a solved problem since ages. What was open was the organization of small teams developing software. To scale from there seems to be the easy part for him. In theory, he may be right, but looking at my project which we scaled from 1 to 65 teams in years working on one product, the organism and interactions between teams and the overall organization cannot be reduced that easily. That would assume an organizational maturity on project level and on each single team that is hard to reach under realistic circumstances. I am not saying its not feasible, just that its not that simple. But Martin has shown to simplify a lot to make his points.

Another new information for me was the term “Iron Cross” of project management:

The reason that these techniques fail so spectacularly is that the managers who use them do not understand the fundamental physics of software projects. This physics constrains all projects to obey an unassailable trade-off called the Iron Cross of project management. Good, fast, cheap, done: Pick any three you like. You can’t have the fourth.

Robert C. Martin, Clean Agile, page 15

I agree with this assessment (of course in a theoretic setup you may be able to achieve all four, but in this world this is unlikely). On the following pages, Martin describes quite typical project development in the waterfall world up until the deatch march phase.

Another example of Martin’s provocations is his statements that “The loss of hope is a major goal of Agile.” Sounds crazy, right? What he aims at by using the metrics (like velocity) in a proper way, arbitrary and unrealistic deadlines, which are based on hopes, can be overcome, and gradually more realistic estimates can come out. This is also were he makes this statement:

Some folks think that Agile is about going fast. It’s not. It’s never been about going fast. Agile is about knowing, as early as possible, just how screwed we are.

Robert C. Martin, Clean Agile, page 27

Some pages later, Martin once again talks about the “grand redesign” – the typical “cure” when a software organisation has maneuvered itself into a dead end. He repeats his example from his earlier books, where a Tiger Team to redesign the software from scratch is caugth between having to catch up with the legacy software’s development while generating no income.

On page 50 Martin makes a memorable insight:

Humans make things better with time. Painters improve their paintings, songwriters improve their songs, and homeowners improve their homes. The same should be true for software. The older the software system is, the better it should be.

Robert C. Martin, Clean Agile, page 50

I really liked this statement. Software is in many cases connotated with the stance that old software/code bases are mostly getting worse, seldom better. Of course we all know good cases, mostly Open Source projects going on since decades (Linux Kernel, gcc, Latex, …), but in the proprietary software industry its often different. I cannot count how often old software really got worse over time – no matter if it was disimproved by rushed hacks or not changed at all. Martin is stating the obvious, but in that clarity I never saw that.

Many people have claimed that up-front planning is not part of Agile development. […] Of course the business needs a plan. Of course that plan must include schedule and cost. And, of course that plan should be as accurate and precise as practical. […]

In short, we cannot agree to deliver fixed scopes on hard dates. Either the scopes or the dates must be soft.

Robert C. Martin, Clean Agile, page 57f

I like this statement, because in few sentences Martin clarifies a typical debate around agile approaches. Some nitpickers – often with the aim to show the unsuitability of agile practices for tough industry business – claim that agile is anti plans or discourages plans.

Rising Velocity

If we see a positive slope, it likely does not mean that the team is actually going faster. Rather, it probably means that the project manager is putting pressure on the team to go faster. As that pressure builds, the team will unconsciously shift the value of their estimates to make it appear that they are going faster.

This is simple inflation. The points are a currency, and the team is devaluing them under external pressure. Come back to thatteam next year, and they’ll be getting millions of points done per iteration. The lesson here is that velocity is a measurement not an objective. It’s control theory 101: don’t put pressure on the thing you are measuring. […]

One way to avoid inflation is to constantly compare story estimates back to the original Golden Story, the standard against which other stories will be measured. Remember that Login was our original Golden Story, and it was estimated as 3. If a new story such as Fix Spelling Error in Menu Item has an estimate of 10, you know that some inflationary force is at work.

Robert C. Martin, Clean Agile, page 81

This is a very important observation and I can confirm it from practice. As a project manager I am monitoring team’s velocity charts. However, I am not doing this to enforce rising velocity charts, but to identify trends in underachievements (commitment vs achievements). Speaking about the average mature team, I think the best to aim for is a stable sprint velocity over longer duration of times, leading to a sustainable pace (Martin writes more on that on page 103). The last paragraph above gives a nice approach how to determine velocity inflation.

Working overtime is not a way to show your dedication to your employer. What it shows is that you are a bad planner, that you agree to deadlines to which you shouldn’t agree, that you make promises you shouldn’t make, that you are a manipulable laborer and not a professional.

This is not to say that all overtime is bad, nor that you should never work overtime. There are extenuating circumstances for which the only option is to work overtime. But they should be extremely rare. And you must be very aware that the cost of that overtime will likely be greater than the time you save on the schedule.

Robert C. Martin, Clean Agile, page 103

I really like this quote, it fits to my “natural instincts”. My work and contributions are most sustainable when I can work more or less normal hours. That means I can rest, stay sane and – more relevant to my employer – reflect on what is going on at work. As a team and project manager my day at work is usually crammed with meetings, mails and nagging people. Very rarely I have time to think deeply about what goes on. Hence, the best ideas I have after work, during weekends and vacations. I usually make sure to take notes about those ideas, so I can followup during working times.

On pages 106f Martin shares another interesting anecdote about the “developer hierarchy” at a printer company. The baseline is, that the printer software developers enjoyed a lot of praise for their direct contributions to the company’s main business. However, that led to elitism and closing up of their code. Other software developer working in indirect matters did receive lower trust and couldnt contribute that much. I think this is something to observe also around my work. The ones directly developing fancy features which make it on magazines and news pages feel much more appreciated than the ones working in the machine room, keeping the whole machinery running. I think the boundaries between those groups must be as permissive as possible.

The point is that Continuous Integration only works if you integrate continuously.

Robert C. Martin, Clean Agile, page 108

Simple but powerful words. I cannot believe how often CI is preached and promised, while it boils down to integration of a full team’s work once a sprint or even less.

The continuous build should never break. A broken build is a Stop the Presses event. I want sirens going off. I want a big red light spinning in the CEO’s office. A broken build is a Big Effing Deal. I want all the programmers to stop what they are doing and rally around the build to get it passing again. The mantra of the team must be The Build Never Breaks.

The Cost of Cheating

There have been teams who, under the pressure of a deadline, have allowed the continuous build to remain in a failed state. This is a suicidal move. What happens is that everyone gets tired of the continuous barrage of failure emails from the continuous build server, so they remove the failing tests with the promise that they’ll go back and fix them “later.”

Robert C. Martin, Clean Agile, page 109

Not much more to say, except that this is hard in traditional environments to achieve, and very easy to loose once gained. Just recently I got the chance to focus more, which I really appreciate.

The belief that quality and discipline increase speed is a courageous belief because it will constantly be challenged by powerful but naive folks who are in a hurry.

Robert C. Martin, Clean Agile, page 134

I love that quote because it is just true. And I admit that sometimes I am one of those naive folks. However I work to be it less every day.

Towards the end of the book, Martin lays out the new movement of Software Craftmanship. I frankly didnt hear about it before, but I like the ideas about it.

As aspiring Software Craftsmen we are raising the bar of professional software development by practicing it and helping others learn the craft. Through this work we have come to value:

Not only working software, but also well-crafted software

Not only responding to change, but also steadily adding value

Not only individuals and interactions, but also a community of professionals

Not only customer collaboration, but also productive partnerships

That is, in pursuit of the items on the left we have found the items on the right to be indispensable.


Many will recall the format following the Agile Manifesto. However I see it a very valuable extension from the perspective of developers.

Let me end here. Probably long enough text for anyone ever to read here 🙂


Thougts on Clean Architecture from Robert C. Martin

After recently reading Clean Code from the same auther, now I followup on Clean Architecture from Robert C. Martin. While Clean Code focusses mostly on the smallest entities of software development and in a simplified manner can be called bottom-up, Clean Architecture naturally comes from top-down. Clean Code is much easier to apply in examples such as my current pet project, Clean Architecture is something more abstract for me, as I do not really have a huge software project at hand to apply.

Down the darkest path comes the idea that strong and stable architecture comes from authority and rigidity. The architect’s mandate is total and totalitarian, the architecture becoming a dystopia for its developers and a constant source of frustration for all.

Down another path comes a strong smell of speculative generality. A route filled with hardcoded guesswork, countless parameters, tombs of dead code, and more accidental complexity than you can shake a maintenance budget at.

The path we are most interested is the cleanest one. It plays more to our strengths than our weaknesses. We create things and we discover things. We ask questions and we run experiments. A good architecture comes from understanding it more as a journey than a destination, more an ongoing process of enquiry than a frozen artefact.

Kevlin Henney in the Foreword

Kevlin’s words chime very well with my personal understanding of good software architect mindset. In my career, I have seen all of the above, and I am glad in my current project all architects have the latter. However, its not a closed book. Many managers, suppliers and developers still demand for what I would call static enterprise architecture, with different rationale behind. While management sees architecture as a definition of tools, libraries and complete set of boxes to ultimately save money by “clarity”, suppliers want something fixed to make an offer towards and have something to blame when problems arise.

I’ve seen it happen. I’veworked in projects where the design and architecture of the system made it easy to writeand easy to maintain. I’ve experienced projects that required a fraction of theanticipated human resources. I’ve worked on systems that had extremely low defectrates. I’ve seen the extraordinary effect that good software architecture can have on asystem, a project, and a team. I’ve been to the promised land.


This is an important statement, easy to overlook and ignore. Most of us know reality well enough to have skepticism regarding the applicability of clean approaches to corporate software development. The constant stress by deadlines, tight budget, competing stakeholder’s requirements and technology fights often overrules the best intentions. But giving up is no option. Every new project, every new week its worth to keep trying to establish a bit better code, a bit better architecture. Our own future selfs, and our professional spirits will be thankful.

The goal of software architecture is to minimize the human resources required to build and maintain the required system.

What is design and architecture?

I am not sure if I would call the above “The” goal, but certainly its “a” central goal. Replacing “human resourcues” by “effort” for me would make this goal statement more generally applicable. Effort can include human resources, but also license cost, mental imbalance and turnover.

On pages 5ff, Martin gives a case study of a company sinking exponentially more engineering staff while creating exponentially less output. Interesting and highly relatable. I hope I find ways to calculate according metrics also for my private and professional projects.

And that’s the answer to the executive’s dilemma. The only way to reverse the decline in productivity and the increase in cost is to get the developers to stop thinking like the overconfident Hare and start taking responsibility for the mess that they’ve made.

The developers may think that the answer is to start over from scratch and redesign the whole system—but that’s just the Hare talking again. The same overconfidence that led to the mess is now telling them that they can build it better if only they can start the race over. The reality is less rosy:

Their overconfidence will drive the redesign into the same mess as the original project.

What is design and architecture?

Martin is focussing here on the software developers and provoking them to not take management influence as an excuse for their own failure to maintain sane software projects. He has done the same in Clean Code. Personally, I can see both sides. Of course management often overrules developers to gain what they call “speed” by effectively telling them to hack some dirty solution together. On the other hand I have seen enough developers either having a messy approach or giving up on management on the slightest push. So being somewhat in between I always try to balance and find tradeoffs. It sounds tempting to target only clean approaches and refuse deadlines completely, but its unrealistic and being too hard here my backfire. I know a developer who is so strict in his believes (and judging only by the technical content he is always right), that he is notorious to be ignored. Who wants to work with somebody who is not willing to compromise at all? (Me, yes, but many other managers take the short path and just go to the next developer)

Thus the final version of the SRP is:

A module should be responsible to one, and only one, actor.

SRP: The Single Responsibility Principle

Finally I got it. For a long time I really struggled to internalize the Single Responsibility Principle. All the other definitions or descriptions kindof made sense, but failed me to be applicable in a crisp way. With the above definition using the actor I got it 🙂 Apart from the SRP, Martin of course also introduces also the other SOLID principles.

While the SOLID principles apply to class interdependencies, he continues to establish three principles on component interdependencies. Naturally, they relate to the SOLID principles, so I wondered if there are ways to generalize a set of principles to apply them on any level of software, from systems via components to classes (and maybe even functions).

The Reuse/Release Equivalence Principle:

The granule of reuse is the granule of release

Component Cohesion

I really liked this principle, it sounds so obvious and clear, however I never thought about release cadence as architectural principle. Nice!

The Common Closure Principle and the Common Reuse Principle are very similar to the Single Responsibility Principle (see above) and the Interface Segregation Principle from SOLID, so both didn’t sound so new.

In the section on Component Coupling, Martin introduces two interesting metrics: Instability and Abstraction. After some reasoning, he defines the area around low instability and abstraction as the zone of pain and the area around high instability and abstraction as zone of uselessness. Classes on the “main sequence” in the diagonal from (0/1) to (1/0) instead are considered sane. Exceptions of course apply, e.g. for database schemes (stable and very concrete).

I checked around and really could find a tool which calculates and displays the metrics for my pet project code. I directly added it to my CI pipelines. See below the recent output. Seems I need to check some of my code.

The strategy […] is to leave as many options open as possible, for as long as possible.

What is architecture?

Only 135 pages in Martin asks the question “what is architecture?”, and his answers really surprised me, it was nothing I really expected. In the following pages he gives many examples (really many, probably too many) for this. For Martin, “details” such as external interfaces (IO, databases), frameworks, the web are not relevant for the core business rules. The latter are for him the core and to be untainted from all such details.

A good architect maximizes the number of decisions not made.

What is architecture?

I think again this is the provocative author, but in the context of his definition of “decision” and “details” this makes sense.

And then, thankfully, the Agile Software Development revolution arrived and put architects like me out of our misery. I’m a programmer. I like programming. And the best way I’ve found to have a positive impact on code is to write it.

The dinosaurs of Big Architecture—typically to be found wandering the primeval plains of Big Process—were wiped out by the asteroid of Extreme Programming. And it came as a blessed relief.

[…] The problem with leaving architecture to programmers is that programmers have to be able to think like architects. It turns out that not all of the stuff we learned during the Big Architecture era was of no value. The way that software is structured can have a profound impact on our ability to keep adapting and evolving it, even in the short term.

[…] Learn how you can incorporate design principles and Clean Architecture into your development processes […]. Perhaps, before you commit code, ask a colleague to review it with you. And look into the possibility of adding a code “quality gate” to your build pipeline as a last line of defense against unclean architecture. (And if you don’t have a build pipeline, maybe it’s time to create one?)

Most important of all is to talk about Clean Architecture. Talk about it with your team. Talk about it with the wider developer community. Quality is everybody’s business, and it’s important to reach a consensus about the difference between good and bad architecture.

Be mindful that most software developers are not very architecture-aware, just as I wasn’t 25 years ago. More experienced developers clued me into it. Once you’ve wrapped your head around Clean Architecture, take the time to wrap someone else’s head around it. Pay it forward.

[…] The real journey starts here.

Jason Gorman in the Afterword

I think thats perfect last words without any more need of commenting!


Thoughts on “Clean Code” from Robert C. Martin

I recently ordered some books from my project’s book recommendation list. This list is maintained by a few colleagues and contains many classics, but also some more hidden gems. Clean Code probably had to be on that list, and its not my first encounter. I skipped through the book already a few occasions before, but never really had the time and motivation to really digest its content. This time, this was different.

This is not a book review, but rather some personal notes and thoughts I gathered while reading it.

I particularly liked the first chapter, because it gathers quite some good statements and arguments why clean code really matters. Of course, everyone who ever development reasonably sized software themselves, know about the pros of clean code (as per whatever definition/guideline/philosophy). However, in daily close combat with management, customers and fellow developers, its not always easy to have the perfect argumentation at hand. So I always like to learn about new arguments objectively supporting subjective opinions.

The section “The Grand Redesign in the Sky” describes very picturesque who the “its such a mess, lets do it from scratch” often tends to (not) pan out in reality. That fits nicely to earlier thoughts and reads of mine regarding featuritis and the competition of legacy code maintainers and the new tiger team starting from scratch. Martin’s anecdote about such a competation taking more than 10 years was really eye-opening.

The next section talks about the attitude of developers needed to defend clean code:


[…] Why does good code rot so quickly into bad code? We have lots of explanations for it. We complain that the requirements changed in ways that thwart the original design. We bemoan the schedules that were too tight to do things right. We blather about stupid managers and intolerant customers and useless marketing types and telephone sanitizers. But the fault, dear Dilbert, is not in our stars, but in ourselves. We are unprofessional.

This may be a bitter pill to swallow. How could this mess be our fault? What about the requirements? What about the schedule? What about the stupid managers and the useless marketing types? Don’t they bear some of the blame?

No. The managers and marketers look to us for the information they need to make promises and commitments; and even when they don’t look to us, we should not be shy about telling them what we think. The users look to us to validate the way the requirements will fit into the system. The project managers look to us to help work out the schedule. We are deeply complicit in the planning of the project and share a great deal of the responsibility for any failures; especially if those failures have to do with bad code!

Robert C. Martin: Clean Code, Pearson Education, 2009

I can very much relate to that code and – as a engineering manager and project manager with developer history – support confident developers defending their ground. Of course, there are more shades of grey, and I have experienced many arguments with developers who went too far in unconcrete demands for more time because of better quality. In my perspective, a proper discussion based on facts, figures and clear arguments from both sides – management and developers – yields the most best results, preferably as a compromise. But I get why Martin is triggering developers here explicitly.

Following are some statements from famous developers, you can read the first chapter of the book for free here. Then the actual content begins, where Martin describes one technique/rule/guideline after another, better naming, function design, commenting, formatting, error handling, testing etc. What follows are some refactorings of different sized open source code snippets, piece by piece, and in the end a list of “smells and heuristics”, an enumeration of indicators for bad code.

I have tried to apply the knowledge I gathered from reading this book in two ways:

  1. I added phpmd – php mess detection – to my current pet project, with the clean code rules. Of course it can only identify a subset of bad code indicators, but its already a good start. There were some findings which I could quickly fix.
  2. More practically, I started to refactor the code. E.g. I removed duplicate code, reduced function sizes and improved the error handling. Some of those improvements I already merged on master, but I am definetly not yet finished.

The worst part of the book was the example for “improved” code given on pages 50ff. Maybe I didn’t get something essential, but this code is super-hard to read, understand, and totally cluttered with 1 to 3 line functions with awkward naming. I even suspected that this was kind of a trap/test to trigger the reader’s objection. Seems not. This made me also google for critique on the book, and this blog article also puts some good stances on it.

Can I recommend the book? Yes. I would take basically everything in it with a grain of salt, and there are some elements which I find “too much”. With that in mind, its a good motivation to critically reflect on own code and improve it iteratively.