Measuring the right things at the right time is fundamental to DevOps delivery. How you do it, and the actions you take afterward, greatly influence speed and success.
It is a very human trait to evaluate ourselves against others, be it in terms of intelligence, success, wealth, or attractiveness: rightly or wrongly, we often look to others as a comparative standard for how we should behave, think, and feel. According to Psychology Today, as much as 10% of our thoughts involve comparisons of some kind, so it should be no surprise that evaluation and comparison in the IT industry are hot topics.
The RIGHT metrics matter
Identifying the right metrics, those Key Performance Indicators (KPIs) that make the most sense in the context of the specific task at hand, is critical. By understanding how you are progressing against those metrics, you can make informed decisions, change course, or step up the acceleration. But what should you care about? What are the key influences on your business outcome? And what is everyone else doing?
For me, the most logical approach to addressing these questions is the adoption of a framework called DevOps maturity assessments.
Transformation ready
Modern organisations are driven by technology, at the heart of which is software development. By developing and delivering software quickly and reliably, organisations aim to make their customers happy, deliver features ahead of their competitors, accomplish mission goals and win in the market. Of course, that usually doesn’t come for free – it means hard work, and often enough, it also means undertaking a technology transformation.
A successful transformation means focussing on several key elements:
- How you measure software delivery performance
- How much and how often are you improving your capabilities – especially those critical capabilities that drive your business outcomes
- Creating a road map of the capabilities to demonstrate how you will improve quarter to quarter (year to year is in this case looking too far ahead).
Building on your capabilities
What, then, is a capability in this context? Capabilities are key processes and practices that predict software delivery performance. They usually fall into four categories:
- Technical – capture practices that are essential components of a continuous delivery practice
- Lean/ process – important pieces about how the work is done
- Metrics and monitoring – how you use metrics and monitoring tools to make business decisions
- Cultural – measures of culture that indicate high trust and information flow, the value of learning, job satisfaction, etc.
DevOps assessments
We define DevOps as a set of practices and cultural values that has been proven to help organisations of all sizes improve their software release cycles, software quality, security, and ability to get rapid feedback on product development.
To move forward in their DevOps journey, companies should outline, identify and map the areas they need to improve their capabilities. Having a clear idea of the maturity of your current IT performance, and not just infrastructure will help you greatly in this process. This is where DevOps assessments come into play: a set of questions that only your team, department, or company can answer.
How can a set of questions help you transform your business? The secret ingredients are WHAT are you trying to assess, and HOW does that information help you identify whether you are indeed moving in the right direction. You should ask yourself what elements can help you outline your current maturity levels and identify those critical items for inclusion in your backlog of capabilities improvements.
Where to start
Among all the elements you could focus on, looking at software delivery performance should be considered the baseline. The reference standard on what you should be assessing is the work that Nicole Forsgren, Jez Humble, and Gene Kim have done, outlined in their brilliant book, Accelerate: The Science of Lean Software and DevOps. It talks about four critical metrics, capturing two aspects of software delivery, Throughput, and Stability.
Throughput
- Change Lead time – How long it takes to go from code commit to code successfully running in production or a releasable state.
- Deployment frequency – How often is code deployed
Stability
- Mean time to restore (MTTR) – How long it generally takes to restore service when a service incident occurs (e.g., unplanned outage, service impairment, etc.)
- Change fail percentage – The percentage of changes that result in degraded service or subsequently require remediation (e.g., lead to service impairment, service outage, require a hotfix, rollback, fix forward, patch, etc.)
There is no trade-off between speed and stability. In fact, throughput and stability enable one another.
A real-world DevOps assessment approach
Our own experience of working with Tier 1 global banks led to the creation of a specific approach to assess DevOps maturity, the top-line of which is as follows:
- Definition – Define the scope of the assessment
- Preparation – Assessment planning and organisation
- Execution – Run interviews and consolidate the results
- Analysis – Analyse the data and capture improvement recommendations
- Results – Create a detailed report result and an execution plan
Our DevOps Assessment looks at 14 maturity parameters across Culture, Technical Practices, Risk Mitigation, Infrastructure, Metrics, and Release Management. Our five-point scale enables a customised target maturity plan.
A visualisation is essential to get everyone on board across the business, not just the engineers, so we find this kind of summary helps as part of the reporting:
Now what?
After completing a DevOps assessment, the expectation is that you will have a clear idea of any underperforming capabilities, those that are performing as expected and others that are overperforming. DevOps assessments aim to evaluate these capabilities and provide a visualisation of where you are in your transformation journey.
To make it practical, the next step is to define a way of prioritising which capabilities you want to enhance and then focus on them. There are various techniques to help with prioritisation. The most intuitive one is a strength/impact approach – it makes sense to look first at whatever elements fall under low capability strength and high impact on software delivery performance.
In conclusion
So, imagine you are at the end of the DevOps assessment journey. You’ve spent some time figuring out those capabilities where you are strong and also what needs to be improved to become better at your software delivery. Is that it?
My favourite analogy springs to mind. Five frogs are sitting on a log. Four decide to jump off, how many are left? You would be tempted to say one. But the answer is five. Why? Because there’s a difference between deciding and doing.
In my experience, knowledge is only half of the battle – the execution is where most companies fail. It is not enough to understand the need for change, you must also instigate that change. That responsibility lies on the shoulders of the leaders of the organisation and will be dependent on your overall organizational maturity, priorities, and the current stage of your business strategy. You can decide some capabilities are more important than others, but you must execute a plan to improve them, do the hard work of convincing people they need to change, get budgets in place, allocate teams.
Article written by Catalin Stoiovici, Managing Principal and Head of Engineering Delivery at Capco.
