Coding Culture

Technology Radar #27: Automotive SW perspective

As written before, I really like the regular updates provided by Thoughtworks in their Technology Radar. My focus is on the applicability of techniques, tools, platforms and languages for automotive software, with a further focus on embedded in-car software. Hence, I am ignoring pure web-development and machine learning/data analytics stuff which usually makes a huge portion of the whole report. Recently, its volume 27 has been published. Let’s have a look!

As usual, lets start with a dive in the “Technologies” sector and its “Adopt” perimeter. The first entry we can find is about “path-to-production mapping“. Its as familiar as it sounds – many of my readers will have heard about the Value Stream Mapping or similar process mapping approaches. Thoughtworks state by themselves that this one is so obvious, still they didnt cover it in their reports yet. Sometimes, the simple ideas are the powerful ones. I can confirm from my own experience that a value stream map laying out all the process steps and inefficiencies in an easy to digest manner is a good eye opener and can help to focus on the real problems instead of beating around the bush.

Something very interesting for all the operating systems and platform plans in Automotive is the notion of an “incremental developer platform“. The underlying observation that “teams shooting for too much of that platform vision too fast” is something I can confirm from own experience. Engineers love to develop sustainable platforms, but underestimate all the efforts required for it, and management with its impatience is further undermining platform plans. Following the book Team Topologies’ concept of a “thinnest viable platform” makes sense here. Not shooting too far in the first step, but also treating a platform product as an incremental endeavour.

Another one which strikes me is “observability in CI/CD pipelines“. With the increasing amount and complexity of CI/CD pipelines in one project, let alone a whole organization, many operational questions arise. And operations always benefit from clear data and overview. Recently, a then-student and now colleague and me designed and realized a tool which enables CI/CD monitoring for more than one repo, but for a graph of repos. I hope we can publish/open this project anytime soon.

In the platforms sector, Backstage entered the “adopt” perimeter. The project is actively developing forward, and indeed could be an interesting tool for building an internal sw engineering community.

Looking at the tools sector, I liked Hadolint for finding common issues in Dockerfiles.

Coding Culture

Technology Radar #26: Automotive SW perspective

As written before, I really like the regular updates provided by Thoughtworks in their Technology Radar. Since the new version #26 was released a few weeks back, I found now the time to put down my notes. My focus is on the applicability of techniques, tools, platforms and languages for automotive software, with a further focus on embedded in-car software. Hence, I am ignoring pure web-development and machine learning/data analytics stuff which usually makes a huge portion of the whole report. Let’s go!

In the techniques section in the “adopt” circle we initially have “single team remote wall”. In a nutshell I think they mean having a dashboard showing the essential data, kpis and tasks for a remote development team. I think the trick here is the “single” as I assume that most remote teams have dashboards, however usually multiple ones loosely coupled. In my current team, our Scrum Master has created a great Jira dashboard showing some essential data which could give hints at the team’s performance.

The second noteworthy technique is “documentation quadrants”. Referring to documentation.divio.com/ this provides a nice taxonomy of different documentation types. This is very relatable, as I very often experience a fuzzy mixture of all those types scattered in many places. Certainly this is something I will bring to my work network’s attention.

Third, we have “rethinking remote standups”. This follows a general observation that conducting remote daily standups in the same duration and content like the were recommended in former times (e.g. the typical 15 min Scrum daily) does not provide the same amount of alignment within a development team. This is not necessarily because of the the meeting itself, but because other casual sync occasions during the day are happening less in remote setups. In the radar, its recommended to try an extension to one hour, and of course the goal is to decrease the overall meeting load by this. I am thorn on this one, as I was always a fan of crisp daily meetings, avoiding random rambling on topics concerning only parts of the team. Blocking 1 hour for everyone every day sounds like an overshoot approach.

Next there is again the “software bill of materials” topic. This is currently a huge topic in the software industry, there have been very concrete examples recently (e.g. the Log4Shell or NPM package events you probably read about). Tool support to transparently and consistently managing the used software in a bigger project is really needed. While in the web and cloud business there is a growing number of tools, in the embedded world there are only some puzzle pieces. I can currently think of some Yocto support for this, however this covers only Linux parts in usually more complex multi-os automotive ECUs.

“Transitional Architecture” sounds like a promising thing, even though the radar’s description stays a bit vague. Luckily there is an extensive article by Thoughtworks’ Martin Fowler on this approach. In my opinion, managing legacy software in complex setups is one of the key challenges in the whole software industry, even more so in automotive embedded software, which is characterized by the co-existence of decades old technologies with state of the art approaches. Formalizing the transition from on older architecture to a newer makes sense, as usually this transition is often not architecturally covered as extensively as a the target architecture. This leads to misunderstandings, hacky workarounds and other unwanted side effects to a sustainable development.

Going one circle to the outside, in the “assess” perimeter, we first find CUPID. Aimed at replacing the SOLID rules with instead a set of properties of “joyful code”, it has some interesting observations and paradigms. Currently I only skipped over it, I think this deserves more time and maybe a dedicated article. However, I can recommend to check out the well written original blogpost by Dan North.

In the “hold” perimeter we see “miscellaneous platform teams”. In contrast to “platform engineering product teams” described earlier in the radar, this is kind of a degradation form. If a platform team fails to define a clear product goal and identify its customers, usually the scope becomes (or is) very fuzzy, leading to a unclear platform system. Hence, its strongly recommended to avoid this by achieving clarity of what actually is the scope of the team.

In the platforms sector, I could only identify one relevant blip “GitLab CI/CD”. Recently I see a lot of discouragement of using Jenkins, and of course if you use already Gitlab for its other elements (code hosting, code review, issue tracking) you may as well use it for CI/CD pipelines. For sure its better integrated in the overall Gitlab experience. However its just another vendor-specific DSL, so I wonder if there will be practical standardization on the pipeline definition soon.

Looking at the “Tools” sector, I found the reference to the two code search tools Comby and Sourcegraph. Besides offering code search and browsing using abstract syntax tree analysis, they are also offering semi-semantical batch changes, enabling “large scale changes”. Comby is an open source tool, while sourcegraph is commercial. I think I will try at least one of them soon.

Coding Tinkering

Simple CI/CD for embedded devices

After some years of private tinkering with CI/CD workflows for web development, and a good load of professional exposure to embedded projects working hard to get CI/CD in a scaling, fast and reliable setup, I wanted to combine both. Earlier, I did some trial-and-error-project leveraging the great NodeMCU boards, but it was without any automated testing and no ci pipelines used. So it was time to make a step further in my private endeavors and setup a CI/CD pipeline with automated flashing and testing of new embedded code. An important requirements was that the automated testing should test the complete embedded device consisting of its hardware and software in completeness (black box test). Hence, only the “official” outside interfaces like serial interface and physical output (LED!) should be used for automated tests. Of course, the used hardware, software and complexity in no way match what our projects’ engineering teams handle every day, and I don’t intend to compete with the engineers. Its an exercise for myself to learn.

Without further ado, lets have a look at the setup:

We see the following: My vserver-hosted Jenkins is the same as usual. However, as local node connected to the target device (NodeMCU), I am using a Raspberry Pi Zero 2 W. I connected it as a node (formerly called slave) to my Jenkins master. The NodeMCU target is connected via a USB cable to that Raspberry Pi. The Raspi is able to build the code, flash it to the NodeMCU and run some tests written with Python. The NodeMCU target has an LED (with resistor) connected, which is controlled by the embedded software. To close the circle, the Raspi also has a BH1750 light detector board connected. The idea is: Whenever a software change is built and flashed, the Raspi can automaticall test if the LED is correctly lit, and if not, fail the test, hence the overall pipeline.

Eventually, with some vacation-breaks I made this work. Yay! There were, however, some impediments to overcome. You can find them here in case you would ever have a similar endeavor 🙂

  • To build the software on the Raspi, I had to install on it the according framework. After great experience with it on the desktop VSCode extension, I learned in the docs that PlatformIO also has a headless cli client. How to get it onto the Raspi? First I thought using a docker container would be a good choice, to have a reproducible environment. However, it was really hard to find any working, up-to-date docker for that. Hence, I finally decided to got with PlatformIO’s super-handy installer script.
  • Getting the LED to work with some simple code was not as straightfoward as I hoped after studying the docs initially. I could not make the LED light up at 50% no matter what I did to the values or wire connections. It was always at full power (which would be generally fine but not with the “product” I had in mind, more on that in a later blog post I guess). I even used my simple osciloscope. Finally, it turned out that all the tutorials which explained this dead-simple setup had one snag: the provided value range had beed changed as a breaking change quite recently. After adapting to the new range it worked!
  • The BH1750 light sensor is connected with an I2C connection to the Raspi, which was my first own engineering exposure to I2C ever. In my first attempt I could connect it to the NodeMCU successfully, but after some travel and trying the same via the Raspi failed miserably. Again a lot of trial-and-error, until some random guys with the same issues on the interner hinted towards the pin connection on the BH1750. Indeed, it was extremely sensitive and the pins were always a bit off. I finally soldered it together and then it worked like a charm.
  • Using a Raspberry Pi Zero 2 to build the embedded software doesn’t sound very proper, and indeed a clean build takes a while and mostly blocks the complete machine for other things. The good news is that PlatformIO offers a simple out-of-the-box build cache solution, which requires not more than configuring a build cache directory, which is subsequently used. Of course, caches should always used with care and for reproducible build they should probably be turned off. Again, in my very simple setup it does its job.

So, as usual I spent most time debugging unexpected snags than on the actual concept or missing pieces. Still learned a lot.

I still have some open to-dos and followups:

  • Of course, my current “product” is dead simple. Having the basics in place I want to extend the functionality gradually. With this minimal CI setup I can add/modify functionality without introducing regressions on the existing functionality.
  • Improve the pipeline duration. The average pipeline takes about 1 min, of which the flashing consumes the majority (30-40 secs) of time. Also there is a significant waiting time in the beginning which Jenkins does not include in the pipeline duration. I need to investigate what that is about.
  • At the moment the software is updated via USB cable. The NodeMCU framework also offers software update over the air (OTA). Leveraging that would some more flexibility in placing the device, and of course enabling Continuous Deployment on “production targets”. A first step I am considering is adding OTA as a second flash procedure, followed by another test run.
  • Exploring more options of NodeMCU’s and PlatformIO’s frameworks/tools. While setting up the above, I saw a lot of interesting things worth to investigate further.

Here is the (shortened) Jenkins log for reference:

Push event to branch master
Looking up repository jakob/NodeCI
Querying the current revision of branch master...
[Pipeline] Start of Pipeline
[Pipeline] node
Running on rpizero in /home/jakob/jenkinsnode/workspace/Jakob_NodeCI_master
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Declarative: Checkout SCM)
[Pipeline] checkout
The recommended git tool is: NONE
using credential [...]
Fetching changes from the remote Git repository
Fetching without tags
Commit message: "added acceptace test for physical led"
 > /usr/bin/git config core.sparsecheckout # timeout=10
 > /usr/bin/git checkout -f
[Gitea] Notifying branch build status: PENDING Build started...
[Gitea] Notified
[Pipeline] }
[Pipeline] // stage
[Pipeline] withEnv
[Pipeline] {
[Pipeline] stage
[Pipeline] { (build)
[Pipeline] sh
+ /home/jakob/.platformio/penv/bin/pio run
Processing nodemcuv2 (platform: espressif8266; board: nodemcuv2; framework: arduino)
Verbose mode can be enabled via `-v, --verbose` option
CONFIGURATION: https://docs.platformio.org/page/boards/espressif8266/nodemcuv2.html
PLATFORM: Espressif 8266 (3.2.0) > NodeMCU 1.0 (ESP-12E Module)
HARDWARE: ESP8266 80MHz, 80KB RAM, 4MB Flash
 - framework-arduinoespressif8266 3.30002.0 (3.0.2) 
 - tool-esptool 1.413.0 (4.13) 
 - tool-esptoolpy 1.30000.201119 (3.0.0) 
 - toolchain-xtensa 2.100300.210717 (10.3.0)
LDF: Library Dependency Finder -> https://bit.ly/configure-pio-ldf
LDF Modes: Finder ~ chain, Compatibility ~ soft
Found 36 compatible libraries
Scanning dependencies...
No dependencies
Building in release mode
Retrieving maximum program size .pio/build/nodemcuv2/firmware.elf
Checking size .pio/build/nodemcuv2/firmware.elf
Advanced Memory Usage is available via "PlatformIO Home > Project Inspect"
RAM:   [===       ]  34.3% (used 28088 bytes from 81920 bytes)
Flash: [===       ]  25.7% (used 268009 bytes from 1044464 bytes)
========================= [SUCCESS] Took 7.07 seconds =========================
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (flash)
[Pipeline] sh
+ /home/jakob/.platformio/penv/bin/pio run --target upload
Processing nodemcuv2 (platform: espressif8266; board: nodemcuv2; framework: arduino)
Verbose mode can be enabled via `-v, --verbose` option
CONFIGURATION: https://docs.platformio.org/page/boards/espressif8266/nodemcuv2.html
PLATFORM: Espressif 8266 (3.2.0) > NodeMCU 1.0 (ESP-12E Module)
HARDWARE: ESP8266 80MHz, 80KB RAM, 4MB Flash
 - framework-arduinoespressif8266 3.30002.0 (3.0.2) 
 - tool-esptool 1.413.0 (4.13) 
 - tool-esptoolpy 1.30000.201119 (3.0.0) 
 - tool-mklittlefs 1.203.210628 (2.3) 
 - tool-mkspiffs 1.200.0 (2.0) 
 - toolchain-xtensa 2.100300.210717 (10.3.0)
LDF: Library Dependency Finder -> https://bit.ly/configure-pio-ldf
LDF Modes: Finder ~ chain, Compatibility ~ soft
Found 36 compatible libraries
Scanning dependencies...
No dependencies
Building in release mode
Retrieving maximum program size .pio/build/nodemcuv2/firmware.elf
Checking size .pio/build/nodemcuv2/firmware.elf
Advanced Memory Usage is available via "PlatformIO Home > Project Inspect"
RAM:   [===       ]  34.3% (used 28088 bytes from 81920 bytes)
Flash: [===       ]  25.7% (used 268009 bytes from 1044464 bytes)
Configuring upload protocol...
AVAILABLE: espota, esptool
CURRENT: upload_protocol = esptool
Looking for upload port...

Warning! Please install `99-platformio-udev.rules`. 
More details: https://docs.platformio.org/page/faq.html#platformio-udev-rules

Auto-detected: /dev/ttyUSB0
Uploading .pio/build/nodemcuv2/firmware.bin
esptool.py v3.0
Serial port /dev/ttyUSB0
Chip is ESP8266EX
Features: WiFi
Crystal is 26MHz
MAC: e0:98:06:85:d6:23
Uploading stub...
Running stub...
Stub running...
Configuring flash size...
Compressed 272160 bytes to 199933...
Writing at 0x00000000... (7 %)
Writing at 0x00004000... (15 %)
Writing at 0x00008000... (23 %)
Writing at 0x0000c000... (30 %)
Writing at 0x00010000... (38 %)
Writing at 0x00014000... (46 %)
Writing at 0x00018000... (53 %)
Writing at 0x0001c000... (61 %)
Writing at 0x00020000... (69 %)
Writing at 0x00024000... (76 %)
Writing at 0x00028000... (84 %)
Writing at 0x0002c000... (92 %)
Writing at 0x00030000... (100 %)
Wrote 272160 bytes (199933 compressed) at 0x00000000 in 18.0 seconds (effective 121.2 kbit/s)...
Hash of data verified.

Hard resetting via RTS pin...
========================= [SUCCESS] Took 27.74 seconds =========================
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (test)
[Pipeline] sh
+ python test/acceptance.py -v
testHelloWorld (__main__.AcceptanceTests) ... ok
testLEDIsTurnedOn (__main__.AcceptanceTests) ... ok

Ran 2 tests in 0.743s

[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
[Gitea] Notifying branch build status: SUCCESS This commit looks good
[Gitea] Notified
Finished: SUCCESS


3D Printing Periphery

Some of my reader already know: A few months ago my wife and me bought a 3d printer, the Prusa MK3S+ (yes, she was pushing this). We decided for the self assembly version, which did not only save us some €, but while we assembled the device we learned a lot about its construction. By the way, the self-assembly handbook were the best printed instructions I ever saw, much recommended.

While my wife focuses on designing impressive organic 3d models with Blender, I envisioned my main usage for tinkering and maker products. It turns out, the printer itself is also a great target for tinkering 🙂 Today, I reached a major milestone, hence I thought about making this short writeup, as inspiration for others and reminder for future self.

One wiring diagram says more than thousands words:

Some bullet points:

  • The Raspberry Pi Zero 2 W (just got available a few weeks back) is always connected to the power and wifi of course.
  • The Raspi runs current versions of OctoPi/OctoPrint with some fancy plugins, so the typical operation and monitoring can all happen from any device with a browser.
  • The Raspi is equipped with the RaspiZ HD Cam.
  • I had some issues with the Raspi file system getting corrupted when there was power loss, leading to unavailability from the network. Which was kind of a dilemma, because how could I turn off this headless device without being able to connect via SSH and shutdown? The solution is an extra hard button soldered to the Raspi’s GPIO pins. When this button gets clicked, the Raspi is running its proper shutdown sequence (which should not be required in many cases).
  • As the images/videos from the camera where too dark even with the room lights on, I added an LED stripe behind the camera so its illuminating the printer
  • The printer is in a room which can get quite cold in winter, and low temperature stops the printer from being available (some safety thresholds kick in), we use a fan heater to get the temperature to the required minimum in the room. (Yes I am aware that the printer should be stored at room temperature the whole year, but we dont have space for that. Something to fix later.)
  • The printer, heater and LEDs can all be turned on and off via a remote controllable Shelly Plug. The nice perk here is that there are plugins to make the power switching available in the OctoPrint UI (PSU Control and its Shelly sub-plugin)
  • OctoPrint is directly available from the LAN, and for internet accessibility I recently started to use OctoEverywhere.

I think its a nice setup so far, and still more ideas to improve. Its not recommended to use the 3d printer in absence, even with remote monitoring. I am thinking about adding some fire/smoke detection device to the setup.

Book Culture

Thoughts on “Reinventing Organizations” by Frederic Laloux

Next in my reviews of work-related books we have “Reinventing Organizations” by Frederic Laloux (the illustrated, shortened version). This is going to be short, I essentially read the book in an afternoon.

After some introductions, Laloux gives a historical overview on organizational philosophies, based on Ken Wilber. It starts at the impulsive, pure power-based via hierarchical via modern performance-oriented philosophies to postmodern pluralist ones. One interesting aspect here is, that even the two “older” ones can still be experienced today, the first one e.g. in street gangs/mafia, the second one e.g. in the church or public administration – and of course many companies.

The latter is based on empowerment, values and integrates diverse interest groups, and probably is by most considered already quite new or in many organizations “still upcoming”. However, Laloux claims that we may experience aklready now the dawn of a new, next philosophy, a integral, evolutionary one. He gives names three breakthroughs, which require but also trigger this. First, the increasing complexity of the systems organizations need to cope with, and the need for self-organized, decentral organizations to be able to cope with that. I would say that this approach “extrapolates” the empowerment by “built-in” trust and top-down-ruling-inhibiting organization forms in small, completely autonomous entities.

The second breakthrough is “wholeness” (“Ganzheit” in my German version of the book). It is based on the view that at their workplace, many people show and use only a part of their personality. This leads to loosing many valuable aspects and creates an unhealthy mental tension. Laloux gives some examples how this can be reliefed, e.g. by medition, group-reflection and new meeting-moderation methodologies.

Third, there is a breakthrough called “evolutionary purpose”. Its probably a reference what more recent publications and consultants call “VUCA”(Volatility, Uncertainty, Complex, Ambiguous). According to that perspective–which I have some doubts on, but maybe more in another blog post–the world around is is increasingly “vuca”. Traditionally, we would try to fight this vuca-ness, trying to impose existing problem-solving techniques–and ultimately fail. The other mental approach is to accept, even embrace this situation and–how Laloux phrases is–“dance with it”. Here, Laloux makes an interesting claim: “Most of us react very cynical on mission statements”. Oh yes!

The book contains some interesting food for thought, and it certainly has some relevant observations. Where my personal doubts come in: If I look at an historic scale, the four “traditional” organizational philosophies have been in place 10.000s, 1.000s, 100s and 10s of years, respectively. As I wrote, especially the last one (postmodern, integral) is still a quite new kid on the block. What does it tell us that new even a new philosophy comes up, and if we extrapolate the logarithmic timescale even more, will it be replace in a few months? Just kidding, however I wonder if Laloux’ evolutionary model is just a sub-type of e.g. the postmodern one, or really a new philosophy. And of course – what comes beyond that. Will there be an even more “advanced”, “progressive” organizational philosophy soon? Will things break apart and we go back to the wild days? I dont know…


Comparison of Python mutation testing modules

When I discovered the existence of Mutation Testing, it was a revelation. It showed me an almost effortless way to “test my tests” with an absolutely clever way. Recently I did a lot of coding in Python and when I looked for a mutation testing framework for that language I found many results, but no comparison. So I did my own 🙂

The following table is a very subjective and superficial evaluation, without investing extensive digging into each tool’s configuration options. I just took the most straightfoward way to run it on a current company project.

ModuleActively maintained?Ease of useRaw outputEvaluation

run & generate html
“Killed 640 out of 954 mutants”

HTML file
314 non killed mutants of which 280 were in a config.py file, so can be ignored.
MutPynoneeded local fix

all: 146
killed: 0 (0.0%)
survived: 144 (98.6%)
incompetent: 2 (1.4%)
timeout: 0 (0.0%)
The run is very fast, so I have the feeling that the unit tests for some reasons are not really executed (even though they are listed on startup)
mutatestnosimpleSURVIVED: 5
RUN DATETIME: 2021-10-10 13:46:20.181892
It seems to run only a random set of mutations
Cosmic Rayyescomplex

create config file, init, baseline, execute, generate html
HTML file
199 non-killed mutants

No summary/expand for each file, only long list of findings

From the above, the maintained mutmut and cosmic ray found the most surviving mutants in my example project, while both other tools found much less. MutPy left a dubious feeling if it actually worked as intended. Both MutPy and mutatest are not maintained anymore. In the future I will look closer into mutmut and cosmic ray and see if I can tweak them towards actually using the results, as the initial ones are still a bit “too raw” to directly act upon (which lies in the nature of my test project, too).


VS Code Extension: Save & Run

Visual Studio Code is my current IDE and I really enjoy it. Its quite simple and elegant in its vanilla version and the extension ecosystem is really great. In this short blogpost I want to promote a little extension which really helps my daily workflow a lot: Save & Run

What it does? Whenever a (code) file is saved, it launches a defined command.

What can it be used for? In my use case, I am using it to execute the unit tests for my project(s) whenever I save a code file. Of course I could execute unit tests and Static Code Analysis also via the GUI in VS Code or by a dedicated keyboard shortcut. But I am saving files quite often via Ctrl+S, so why not combine it together.

Lets walk through an example. After installing the extension, this is how the settings.json (either for the user or the workspace) needs to be adapted:

    "saveAndRun": {
        "commands": [
            "match": "\\.py$",
            "cmd": ".\\local_testing.ps1",
            "useShortcut": false,
            "silent": false

As you can see, you can define multiple commands, each triggered when the “match” clause applies. In this case, I am working on a Python project, so I want to execute the command whenever I change (and save) a .py file. The “cmd” is then the shell command to execute, here its a Powershell script executing local tests:

python.exe -m pytest --cov . --cov-report=html --cov-report xml:cov.xml --quiet --cov-fail-under=75
python.exe -m flake8 --ignore=E501 .

Thats it. Whenever I hit Ctrl+S in the Terminal view the following output is given:

As you can see, the tests are really fast, so its not executing forever. Of course thats because the project and its test cases are simple. For bigger projects this may get more beefy. However, in my current project it helps me a lot to not accidentally miss when I break the code.

Book Culture

Thoughts on “How Google Tests Software”

During my recent years as Software Project Manager I learned to appreciate modern software testing beyond just a tool. In earlier blog posts I described my personal hands-on progress on the matter. Thanks to some teachers and mentors I gained knowledge in theory and practice and nowadays call myself a software testing enthusiast (which doesnt mean I am particularly good at it – I just enjoy the topic).

The Book “How Google Tests Software” from James Whittaker, Jason Arbon and Jeff Carollo is already some years old – a fact I didnt know about initially, but some aged concepts made me look into the publishing date. Also some references are not working anymore, including a link to Google’s own “Google Test Automation Conference”. However, its still a great summary on aspects of modern software testing, with a focus on organizational aspects. There is no mention of mutation testing, probabily-based testing and other testing approaches I would call “advanced”, instead there is a lot of focus on the specific roles in and around testing at Google.

In the forewords the authors’ boss Patrick Copeland introduces the term of “Engineering Productivity” and why he has chosen this as the department’s name. I find this term very appealing, especially in its connection to testing. In my domain, automotive, testing is often seen as something which slows projects down and its done in the very end after features have been completely implemented to “drive maturity”. For me, proper testing is a method to be faster after all, and doing it late is possible but is for from being efficient. The Engineering Productivity organization at Google gathers the lead on test activities, respective tooling around testing and the mentoring/training efforts to spread a testing mindset throughout the whole organization.

In the book, the saying “scarcity brings clarity” is coming up multiple times, and the authors say

The first piece of advice I give people when they ask for the keys to our success: Don’t hire too many testers.

How Google Tests Software, page 4

Googlers might be smart, but they are not plentiful. Every TEM we’ve ever hired from outside of Google makes the comment that their project is understaffed. Our response is a collective smile. We know and we’re not going to fix it. It’s through knowing your people and their skills well that a TEM can take a small team and make them perform like a larger team.

How Google Tests Software, page 188

As a young leader I dread the constant understaffing all around me. Such looks into highly regarded companies like Google make me believe its something inevitably in any fast-paced company. Being always short on good people compared to the tasks at hand must yield the most efficient use of people as possible. But how to balance this in a way which satisfies me is something I admit to not have found the key for yet.

In major parts of the book some key roles are described in detail with their relevance to testing: The (“normal”) Software Engineer (SWE), the Software Engineer in Test (SET) and the Test Engineer (TE). On page 7 the roles are summarized in a nutshell, while many many pages repeat and detail this again and again – a style I find very typical for American textbooks. In essence its clear and I assume my mentors must have either read the book years ago or used common root sources. The worksplit seems obvious to me now. SWEs are developing the code, but have to contribute automated testing on all levels to secure the functionality and quality of the system, even when no other test-focused roles are around. SETs are pushing the boundaries of test tooling, enabling new kinds or aspects of test automation wherever they can. TEs are driving testing especially in a “challenger” approach, to catch all the issues SWEs don’t see or dont want to see initially.

I already mentioned that the book doesnt go very deep into test technology. What I found very interesting is that Google separates their test activities simply by “small tests”, “medium tests” and “large tests”. According to the authors they chose to do this because other terms are overloaded and vague throughout the software industry do a degree that people are talking about different things and often didnt recognize.

On pages 40f some nowadays state-of-the-art concepts about how to include test automation into pipelines are brough up. E.g. having test code near to the functional code. Also the classification of small/medium/large tests is added with some concrete data: large tests have to run within 15 mins, small tests are required to execute in less than 100ms. This shows the enourmous compute power Google is able to leverage for their software development. Also there is a good note to quote:

Small tests lead to code quality. Medium and large tests lead to product quality.

How Google Tests Software, page 47

Only both in combination provide long-term value to a company. Having insufficient small tests strongly correlates to bad code quality which will slow down further development. Having insufficient medium and large tests (system tests) lead to many gaps in the functional and non-functional end-to-end testing, thus yielding bad product quality if there is not immense manual testing in place.

On page 48 two very basics rules for test cases are defined, which I find highly relatable: Tests have to be independent of each other so they can be executed in any order (and in parallel). Also tests must not have any persistent side effects.

Interestingly the book makes some references to a build and test execution system with sophisticated dependency graph analysis, without giving the tool’s name. Nowadays I know that they are referring to Bazel. The book must come from a time when it was not yet published.

On pages 54ff the authors describe a “test certified” program which they used to promote the testing inside Google, including some test certified levels for teams to achieve. In a very telling interview I learned a lot that even at Google the testing proponents had to overcome a lot of internal resistance.

The dream of a unified dashboard has haunted Googlers, much like it has at many other companies. Every year or so, a new effort tried to take root to build a centralized bug or project dashboard for all projects at Google.

How Google Tests Software, page 124

Its late and I want to finish this blog post, so I am closing with an inspiring quote from Joel Hynoski, answering why someone should pursue a career in test:

Test is the last frontier of engineering. We’ve solved a lot of the problems of how to develop software effectively, but we’ve still have a green field of opportunity to attack the really meaty problems of testing a product, from how to organize all the technical work that must get done to how we automate effectively, and responsively, and with agility without being too reactive. It’s the most interesting area of software engineering today, and the career opportunities are amazing. You’re not just banging on a piece of software any more, you’re testing the GPU acceleration of your HTML5 site, you’re making sure that you’re optimizing your CPU’s cores to get the best performance, and you’re ensuring that your sandbox is secure.

How Google Tests Software, page 206
Coding Culture

Technology Radar #24: Automotive SW perspective

I am following the Thoughtworks Technology Radar since approximately five years. Since I first became aware about it (IIRC a colleague forwarded a version to me), I enjoyed its volumes as a condensed summary of current trends in the software industry. There is some bias towards cloud technology and machine learning which is out of my current professional interest, however there are enough other concepts/tools/aspects every time which make me investigate and followup to my best possibilities either at work or in my private projects. In this blog post I will try to list those parts which are roughly new and relevent from an automotive software perspective.

As usual, there are some selected topics on the top level they are presenting. Already the first one – Platform Teams Drive Speed To Market – is highly relevant. Its no secret that the automotive industry is widely moving towards in-house platforms, recent example from the news are vw.os (Volkswagen) and MBOS (Mercedes-Benz). The particular recommendation is to handle such platform activities explicitly as products, with all the implications. Only a platform with its own set of features, product vision, dedicated team and lifecycle can actually serve its purpose. A platform handled as a side-gig of an enduser product will neither serve the platform users well nor will it provide sustainable value, plus there would be a high chance of not surviving the first generation.

The second highlighted aspect – Consolidated Convenience over Best in Class – is also very relevant in the automotive SW sector. All of us love the regular discussion which code review tool is the best, and which CI tool is the newest hot shot we must adopt. Integrated workflow tools which integrate requirements/tickets to code, integration, tools and deployment such as gitlab and github (with github actions) promise so much more convenience in inter-tool integration and generally less setup hassle compared to having multiple tools each with its own authentication foo. However, there is also much higher risk of vendor lock-in. This can result in financial dead-ends and of course sooner or later lead to loss of flexibility on adopting new technology in the workflow area.

Jumping over the third highlighted topic and going straight ahead to the fourth – Discerning the Context for Architectural Coupling – this covers the architectural state of the union address. Baseline of the message here is “it depends”. Discussions with buzzwords like “monoliths” and “microservices” are seldom helpful, the real discussion needs to happen on the detailled requirements, constraints and principles.

Going deeper into the four sectors of the radar – Techniques, Tools, Platforms, Languages&Frameworks – lets start with Techniques. The new entry API expand and contract is directly an “adopt” recommendation. It means rather than breaking changes with new API versions too often its more about adding new interfaces while keeping but deprecating old ones, giving consumers a chance to adopt without breaking their usage. I can see immediate benefit of such patterns especially on highly used interfaces such as the vehicle communication interfaces. If “deprecation would be a thing (at scale)” it could relief many breaking integration challenges in the automotive domain.

In the trial circle we find Hypothesis-driven legacy renovation:

Rather than working toward a standard backlog, the team takes ownership of a measurable technical outcome and collectively establishes a set of hypotheses about the problem. They then conduct iterative, time-boxed experiments to verify or disprove each hypothesis in order of priority. The resulting workflow is optimized for reducing uncertainty rather than following a plan toward a predictable outcome.


Sounds quite interesting. I am, too, used to handling technical stories like user stories. The given approach may indeed yield more efficient and effective collaboration on technical improvements.

Another very relatable technique is Lightweight approach to RFCs. Its something which my personal workstyle already incorporated since long time. I strive to create very fast first drafts for any task and then seeking review comments from my peers. Hence, I can absolutely understand the point of this topic and clearly recommend this. Any organization in need of collaboration and frequent alignment on concepts can benefit from a very lean set of templates encouraging early feedback rather than a culture which expects “perfect first shots”.

Team cognitive load and its linked Inverse Conway Maneuver give also very interesting food for thought. I am well aware about some teams in my environment who struggle a lot with the complexity at hand, and the organzational environment adds to that complexity significantly. The Inverse Conway Maneuver – organizing the team organization around how the proper architecture of a system looks like rather than structuring the architecture around the semi-random existing organizational structure – makes a lot of sense. It chimes well with the notion of Feature Teams and team ownership.

Remote mob programming as an extension of local mob programming, which is by itself the advancement of pair programming is certainly a recommended practice in many situations. Recently I have tried to bring teams together in “remote debug sessions” for hard inter-team nuts (=bugs) with not so great success. There is a lot of hesitance of teams to just spend some focus time together, and time difference in international setups is only one external factor. While (local) pair programming is something I think every team manager should at least try with his team a few times to see if it works out, mob programming depends an order of magnitude more mature team dynamics.

Under “hold” we can find Peer review equals pull request. The baseline here is that pull requests are only one way to conduct peer review, and as such its a very valid claim and call for further action. Looking only at fragmented changes (which 99% of PRs usually are) and reviewing that alone doesnt give context how a codebase looks overall. Full code reviews or audits certainly give great insights for potential refactoring efforts and even re-designs, besides other learnings.

The scaled agile framework SAFe is a hold and a look in the history shows it was since the beginning. Thats interesting. I am no expert in SAFe, and only know it in theory.

Separate code and pipeline ownership is really triggering me while I write this. Thats because I recently gave a presentation about the exact same thing and why both shouldnt be separated. I am wondering if that thought appeared to me myself or if I read about it the radar earlier. Nevertheless, I think anything else than keeping both together cannot really be justified from a professional sw development context (neglecting some extreme theoretical circumstances). Having that said, its not yet an established truths, and organizational structures and other opinions enforce a separation more often than I would wish for.

The tools sector doesnt offer really relevant insights this time, its all about backend intra, web dev, ML and that kindof stuff.

In the platforms sector the first thing which appears intersting is Backstage. Its a platform developed by Spotify which in essence is a simple overview on projects going on within an organization (company). It can be used to advertise cool efforts in the developer community. Something which could be worth a try. However, as a separate tool it may lack synchronizity with the actual developer activities. Integrating such a frontend into gitlab could yield some interesting dynamics, because that could leverage existing information about activitiy of projects and use data directly from the individiual projects/repos in there.

In the languages & frameworks sector again nothing relatable.

In total the new tech radar volume give me some very valuable insights and food for more thought.


Dependency updates with Renovate

Recently I studied the Thoughtworks Technology Radar in its newest version and came upon the recommended activity of “Dependency Drift Fitness Function“, referring to the output of dependency analysis and update tools like Dependabot and Snyk. I knew Dependabot already since a while as handy tool to analyze a repo’s dependencies and create automatic pull request. The article motivated me to look into the topic for my own project. As usual I didnt want to use a SaaS solution for my personal pet projects, but use my own infrastructure. Thats where I found renovate, a handy tool for the same purpose with quite straightforward setup.

Nevertheless it took me a bit trial-and-erroring to make it work for my setup with self-hosted gitea. The renovate documentation – at least the parts I studied – didnt really describe clearly how to connect to a self-hosted tool until I found the –endpoint argument in the CLI help. In total its usage boiled down to two lines to get it running:

npm install renovate
GITHUB_COM_TOKEN=ghp_FOOBAR node .\node_modules\renovate\dist\renovate.js --platform gitea --token abcdef1234567890 --endpoint https://example.com/gitea/api/v1 --labels renovate project/git_repo

That resulted in a first pull request to setup a configuration file. I really liked the well written comment on this PR from the bot, giving details and even an outlook on next steps.

After merging the inital PR, the first “real” one was created upon executing the same script as above. As announced the PR was about lifting the version of the phpdotenv package from an old version 2.4+ to 5+. Really cool, I didnt earlier check this, it must be really old.

There was a little snag as this PR couldnt be merged by itself, as the version bump resulted in some necessary code adaptions. But that was due to syntax changes from the package itself, so nothing to blame on renovate.

As a little further step I added a renovate_runner repo with only the following Jenkinsfile to execute the script every midnight. I am looking forward to future PRs from this bot.

pipeline {
     agent any
     triggers {
     stages {
         stage("Run renovate") {
             steps {
                 sh "npm install renovate"
                 sh "GITHUB_COM_TOKEN=ghp_FOOBAR node .\node_modules\renovate\dist\renovate.js --platform gitea --token abcdef1234567890 --endpoint https://example.com/gitea/api/v1 --labels renovate project/git_repo"
     post {
         always {
             echo "One way or another, I have finished"