Developing a CI/CD maturity model for ECU SW engineering

Continuous Integration in Automotive is like teenage sex:

everyone talks about it, nobody really knows how to do it, everyone thinks everyone else is doing it, so everyone claims they are doing it…

(after Dan Anely)

When talking about software integration and its modern CI/CT/CD patterns, there is a lot of confusion and misunderstanding. I was already in multiple rounds in which projects showed their process maturity in shiny colors, however when observing the daily practice it was far away from the presented state. Claiming to do “CI/CD” and really living it is a huge difference. As we seek objectivity in comparisons, we went ahead and started to develop a CI/CD maturity model, applied to ECU software engineering.

As input we took our own experience and a lot of research papers. Of course we could hardly avoid the canonical work Accelerate from Forsgren et al. But even that one cannot be applied in a straightforward manner. For us, it is important to focus on one ECU project, and not the overall carline project. The latter is of course desirable, however far out of our scope and reach.

Without further ado, here is our current state for discussion, review and comments. Feel free to share constructive criticism. At the bottom you will find our references (the ones we already incporporated and the ones we still have on our bucket list).

Unfortunately the social intranet crops the table. You can still read it by clicking the middle mouse button and dragging it.

KPIDescriptionL1: InitialAd-hocL2: ManagedManaged by humansL3: DefinedExecuted by toolsL4: Quantitatively managedEnforced by toolsL5: OptimizingOptimized by tools
Lead time for changesadapted to ECU project:Product delivery lead time as the time it takes to go from code committed to code successfully running in release customers (carline, BROP orga)One month or longerBetween one week and one monthBetween one day and one weekless than one dayless than one hour
Deployment frequencyadapted to ECU project:The frequency of code deployment to internal customers (carline, BROP orga). This can include bug fixes, improved capabilities and new features.Fewer than once every six monthsBetween  once per month and once every six monthsBetween once per week and once per monthBetween once per day and once per weekOn demand, once per day or more often
Mean Time to Restore (MTTR)How quickly can a formerly working feature/capability be restored?(not: How quickly can a bug be fixed!)More than one monthLess than one monthLess than one weekLess than one dayLess than one hour
Change Fail PercentageApplicable ?
Pipeline knowledge bus factorHow many people are able to fix broken pipelines within reasonable time and/or add new pipeline features with significant complexity (doesnt include rocket science)?≤ 1233 + people in various teamsat least one in most project teams
DevOpsAre pipelines managed by the software developers or not?Pipelines are (almost) exclusively managed by dev-team-external staffPipelines are maintained by the developers; new capabilities are added by externalsPipelines are maintained and improved by the developers in almost all casesPipelines are maintained and improved by the developers in all cases
AccessibilityCan all contributors, also from suppliers, access whatever they need for their jobs?Developers can access only fragments of the system, and additional access is hardly possible (only with escalation/new paperwork)Developers can get access to most parts of the system, and for the other parts there is some replacement (libraries)Developers dont have full access, however they can trigger a system build with their changes and get full resultsDevelopers have access to the full project SW and are able to do a full system build.
Review CultureWhat (code) review culture does the project have?Optional reviews without guidelines and unclarity who is actually supposed to give reviews; regular flamewarsMandatory reviews with defined reviewers; defined guidelines; flamewars happen seldomTools conduct essential parts of review (code style, consistency); barely any flamewars
Release Cycle DurationTime needed from last functional contribution until release is officially deliveredmonthsweeksdayshoursminutes
Delayed deliveriesHow often do delayed contributions lead to special activities during a release cycleEvery timeOften (>50%)Seldom (<25%)Rarely (<10%)Never
Release timelinessHow often are releases not coming in timeOften (>50%)Seldom (<25%)Rarely (<10%)NeverNo release timelines needed
Release scopeAre given timelines determining the planned scope for the next release (excluding catastrophic surprise events)Planned scope is mostly considered unfeasible; priority discussions are ongoing at any time80% of planned scope seems feasible; priority discussions are coming up during the implementation cycle repeatedly100% of planned scope usually is feasiblePlanned scope doesnt fill available capacity, leaving room for refactoring, technical debt reductionThere is no scope from outside the dev team defined, things are delivered when they are done (team is trusted)
Release artifact collectionHow are release artifacts gathered, combined, put togetherEverything manualSome SW manual, some SW automaticall SW automatic, documentation manualSW + documentation automatic
TracabilityConsistency between configurations, requirements, architecture, code, test cases and deliveriesNo consistency/tracability targetedIncomplete manual activitiesMostly ensured by toolsUntracable/inconsistent elements are syntactically not possibleUntracable/inconsistent elements are semantically not possible
DeliveryHow is delivery happening?Ad-hoc without appropriate tooling (mail, usb stick, network drives)Systematic with appriopriate tooling (artifactory, MIC)Automatic delivery from development to customer with manual triggerAutomatic delivery for every release candidate
Ad-hoc customer/management demoability
Feature Toggles
Test automationNo or a bit of exploratory testingTest plan is executed by human testers for all test activitiesAll test activities except for special UX testing are automatedNo contributions can avoid passing the obligatory automated testsAautomatically deriving new test cases
Test activitiesA bit of exploratory testingManual, systematic black box testingStatic code analysisUnit TestingIntegration TestingSystem TestingE2E Acceptance TestingChaos engineering, Fuzzing and Mutation testing
Virtual Targets
Quality CriteriaSW Quality management does not exist/not known/not clearSW Quality management is a side activity and first standards are “in progress”SW quality is someone’s 100% occupation and a quality standard existsSW quality is measured every daySW quality measured gaps are actively closed with highest prio
RegressionsHow often do regressions occur (regression = loss of once working functionality)Regressions happen with every releaseRegressions happen often (>50%)Regressions happen seldom (<25%) and are known before the release is deliveredRegressions happen rarely (<10%) and are known before the released software is assembled
ReportingVideowall, dashboards, KPIsNo or ever changing KPIsDefined KPIs, manually measuredSome KPIs are automatically measured, but its meaningfulness is debated, therefore doesnt play a huge role in decision-makingAutomatically measured and live displayed KPI data, used as significant input by decision-makersLive data KPIs are main source of planning and priorization
A/B Testing
Release CandidatesHow often are release candidates made availableNo release candidates existingBefore a release, release candidates are regularly identified and sharedDaily release candidatesEvery change leads to a release candidate
Master branchWhen do developers contribute to master?Irregularly, feature branches are alive for weeks or moreRegularly, feature branches exist for daysChanges are usually merged to master every day
DevOpsAre pipelines managed by the software developers or not?Pipelines are (almost) exclusively managed by dev-team-external staffPipelines are maintained by the developers; new capabilities are added by externalsPipelines are maintained and improved by the developers in almost all casesPipelines are maintained and improved by the developers in all cases
SW Configuration ManagementHow are product variants and generations managed?No systematic variant management, some variants are just ignored yetsystematic manual management of variants, variants are partially covered by branchesAll variants are managed via configurations and pipelines on same branchSoftware is reused over generations and variants
Customer FeedbackCustomers can be internal, too; not only end-usersCustomer not known/existing, no customer feedback availableInternal proxy customer, providing feedback which doesn’t play a big roleExternal customer is available and his/her feedback plays a relevant roleEnd-users’ feedback is available to the developers and a relevant input for design, planning and priorizationEnd-users’ feedback is main input for design, planning and priorization
HeartbeatIs a regular heartbeat existing?No regular heartbeat (e.g. time between reelases)Regular heartbeat, but often exceptions happenRegular heartbeat without exceptions
IT reliabilitySW Factory is regularly breaking; developers have local alternatives in place to not get blockedSW Factory is often broken..
Developer FeedbackHow quickly do developers get (latest) feedback from respective test activitiesMonths after the respective implementationweeks after the respective implementationdays after the respective implementationhours after the respective implementationminutes after the respective implementation
DevSecOps Stuff
Security Scanning

Incorporated references: