During my recent years as Software Project Manager I learned to appreciate modern software testing beyond just a tool. In earlier blog posts I described my personal hands-on progress on the matter. Thanks to some teachers and mentors I gained knowledge in theory and practice and nowadays call myself a software testing enthusiast (which doesnt mean I am particularly good at it – I just enjoy the topic).
The Book „How Google Tests Software“ from James Whittaker, Jason Arbon and Jeff Carollo is already some years old – a fact I didnt know about initially, but some aged concepts made me look into the publishing date. Also some references are not working anymore, including a link to Google’s own „Google Test Automation Conference“. However, its still a great summary on aspects of modern software testing, with a focus on organizational aspects. There is no mention of mutation testing, probabily-based testing and other testing approaches I would call „advanced“, instead there is a lot of focus on the specific roles in and around testing at Google.
In the forewords the authors‘ boss Patrick Copeland introduces the term of „Engineering Productivity“ and why he has chosen this as the department’s name. I find this term very appealing, especially in its connection to testing. In my domain, automotive, testing is often seen as something which slows projects down and its done in the very end after features have been completely implemented to „drive maturity“. For me, proper testing is a method to be faster after all, and doing it late is possible but is for from being efficient. The Engineering Productivity organization at Google gathers the lead on test activities, respective tooling around testing and the mentoring/training efforts to spread a testing mindset throughout the whole organization.
In the book, the saying „scarcity brings clarity“ is coming up multiple times, and the authors say
The first piece of advice I give people when they ask for the keys to our success: Don’t hire too many testers.
How Google Tests Software, page 4
Googlers might be smart, but they are not plentiful. Every TEM we’ve ever hired from outside of Google makes the comment that their project is understaffed. Our response is a collective smile. We know and we’re not going to fix it. It’s through knowing your people and their skills well that a TEM can take a small team and make them perform like a larger team.
How Google Tests Software, page 188
As a young leader I dread the constant understaffing all around me. Such looks into highly regarded companies like Google make me believe its something inevitably in any fast-paced company. Being always short on good people compared to the tasks at hand must yield the most efficient use of people as possible. But how to balance this in a way which satisfies me is something I admit to not have found the key for yet.
In major parts of the book some key roles are described in detail with their relevance to testing: The („normal“) Software Engineer (SWE), the Software Engineer in Test (SET) and the Test Engineer (TE). On page 7 the roles are summarized in a nutshell, while many many pages repeat and detail this again and again – a style I find very typical for American textbooks. In essence its clear and I assume my mentors must have either read the book years ago or used common root sources. The worksplit seems obvious to me now. SWEs are developing the code, but have to contribute automated testing on all levels to secure the functionality and quality of the system, even when no other test-focused roles are around. SETs are pushing the boundaries of test tooling, enabling new kinds or aspects of test automation wherever they can. TEs are driving testing especially in a „challenger“ approach, to catch all the issues SWEs don’t see or dont want to see initially.
I already mentioned that the book doesnt go very deep into test technology. What I found very interesting is that Google separates their test activities simply by „small tests“, „medium tests“ and „large tests“. According to the authors they chose to do this because other terms are overloaded and vague throughout the software industry do a degree that people are talking about different things and often didnt recognize.
On pages 40f some nowadays state-of-the-art concepts about how to include test automation into pipelines are brough up. E.g. having test code near to the functional code. Also the classification of small/medium/large tests is added with some concrete data: large tests have to run within 15 mins, small tests are required to execute in less than 100ms. This shows the enourmous compute power Google is able to leverage for their software development. Also there is a good note to quote:
Small tests lead to code quality. Medium and large tests lead to product quality.
How Google Tests Software, page 47
Only both in combination provide long-term value to a company. Having insufficient small tests strongly correlates to bad code quality which will slow down further development. Having insufficient medium and large tests (system tests) lead to many gaps in the functional and non-functional end-to-end testing, thus yielding bad product quality if there is not immense manual testing in place.
On page 48 two very basics rules for test cases are defined, which I find highly relatable: Tests have to be independent of each other so they can be executed in any order (and in parallel). Also tests must not have any persistent side effects.
Interestingly the book makes some references to a build and test execution system with sophisticated dependency graph analysis, without giving the tool’s name. Nowadays I know that they are referring to Bazel. The book must come from a time when it was not yet published.
On pages 54ff the authors describe a „test certified“ program which they used to promote the testing inside Google, including some test certified levels for teams to achieve. In a very telling interview I learned a lot that even at Google the testing proponents had to overcome a lot of internal resistance.
The dream of a unified dashboard has haunted Googlers, much like it has at many other companies. Every year or so, a new effort tried to take root to build a centralized bug or project dashboard for all projects at Google.
How Google Tests Software, page 124
Its late and I want to finish this blog post, so I am closing with an inspiring quote from Joel Hynoski, answering why someone should pursue a career in test:
Test is the last frontier of engineering. We’ve solved a lot of the problems of how to develop software effectively, but we’ve still have a green field of opportunity to attack the really meaty problems of testing a product, from how to organize all the technical work that must get done to how we automate effectively, and responsively, and with agility without being too reactive. It’s the most interesting area of software engineering today, and the career opportunities are amazing. You’re not just banging on a piece of software any more, you’re testing the GPU acceleration of your HTML5 site, you’re making sure that you’re optimizing your CPU’s cores to get the best performance, and you’re ensuring that your sandbox is secure.
How Google Tests Software, page 206