Towards Better DevOps KPIs: A codeless change maturity model

DevOps Research and Assessment (DORA) and Google Cloud recently released the 2019 Accelerate State of DevOps Report. The report discusses four core DevOps key performance indicators (KPIs) for measuring change. For most of us in software engineering, especially in the era of “configuration as code,” change means “code change” and is about commits to some source control repository.

And yet, value can be delivered and systems changed without a single line of code being written, through the concept of codeless changes, which include any changes to a software product that don’t require a developer to write code. Generally, these changes are requested to enable a particular stakeholder to get more value out of a software product. When broadened to the general case, these changes may even result in an entirely new business model.

However, with the current code change-based KPIs, moving from code changes to codeless changes can depress KPI measures of performance by decreasing the measured rate of code change. Additionally, moving the easiest and most familiar code changes to codeless processes means that changes accomplished by altering code are trickier and more substantial, which could increase defects per change — and also depress KPIs.

Codeless Change Maturity Model

Non-code changes are less discrete and harder to measure than changes to code, especially since some codeless changes are fully automated. Therefore, instead of a KPI-based metric, I am proposing a codeless change maturity model. This model looks at business processes and evaluates whether software development or other processes are making changes that deliver value.

We’ll follow a particular business process, business intelligence (BI) metrics monitoring, to see how it takes shape through the maturity model.

Hands-On Change

This is the most basic maturity level, where business processes start. At this level, R&D knows little to nothing about the business process and the changes it requires. Every change is a new user requirement delivered by the business owner and must be evaluated and implemented using the “generic” software development life cycle (SDLC). In our example, this would be represented by the engineering team building the first BI dashboard for a stakeholder.

Hand-Crafting a Process

At this level, R&D has had multiple change requests from users to implement similar business processes. The requirement will still be delivered to the business owner, who has some experience in estimating and defining it. The change will be implemented by the R&D team, perhaps using tools and techniques they have adopted to make these specific sorts of changes without having to write lots of new code. When the engineering team builds the tenth BI dashboard, they may already have informal processes for building it.

To mature to this level, a company should be asking:

• Is this user request similar to the ones we’ve had before?
• Can we create any processes or tools to make this change faster?

R&D Hand-Off

At this level, the organization has defined a class of codeless changes as a routine part of doing business. There’s a defined use case to which this process should be applied, and a specific way to request the change. This change will often be supported using a formal set of tools and techniques, usually enabling changes to be performed outside of the SDLC. Having a formal process means that these changes can be carried out by technical personnel who are not part of the R&D team, such as a professional services team or integration partners. For our example, this could be BI dashboards that are built by an analyst rather than a developer, and relying on raw information exported to a data warehouse.

To mature to this level, a company should be asking:

Should we create a formal process for customers to request the change?
Are we prepared to spend the money and time to create tools and methods so that non-developers can make the change?
Given that more users will request the change once they know it’s possible, do we have the necessary staff time?

‘All Hands’ (DIY)

At this level of maturity, the organization has not only defined a class of changes but has implemented a process allowing end-users to make the changes by themselves. The change needs to be very rigidly defined so it can be easily distributed to a large number of people. The UX is central to this level of maturity. It must not only be possible to make the change but also easy for the end-user to do so.

A self-service BI tool that allows all stakeholders to create and share their own dashboards would be a good example of this type of end-user change. It’s also a good, real-world example of the importance of the UX; there are a lot of confusing and badly-designed dashboard tools out there, and they make it hard for end-users to make the changes they need, on their own!

• Not every change should be handed to end-users in this way. To mature to this level, a company should be asking:
• Is the change defined narrowly and simply enough to allow the end-user to make it?
• Can the end-user possibly break anything by making the change?
• Is the change requested often enough to justify the time and expense of developing user-facing software to handle it?
• Will giving users control of the change limit future development possibilities?

Hands-Free (automation)

At this level, the organization has built a system that’s capable of modifying its behavior to meet individual users’ needs. Machine learning is often used to provide this level of flexibility but it can also be achieved using simple rulesets

To mature to this level, a company should be asking:

• Is the change something that users would want to be automated?
• Will automation feel ‘creepy’ or intrusive?
• Are we prepared to provide ongoing product support for an automated feature?
• Do we want to provide users with the ability to manually override the automation?

Bottom Line

Measuring code change for code change’s own sake can sometimes incentivize developers to take on the ‘easy’ tasks that really should be taken out of code altogether. Businesses need to move beyond the standard DevOps KPIs and properly assess and value codeless changes, too. This means measuring how development teams are advancing along this maturity model alongside the classic KPIs.

Modern software engineering focuses on delivering value to the parent organization, so it’s important for software companies to understand the business needs that require changes to be made. If we can pinpoint where change is most required and ask ourselves the questions above, we can advance more quickly through the change maturity model, with less coding and faster, easier and safer delivery of value.

Illustrations by Ortal Avraham

This article was originally published on TheNewStack