This is the tenth in a series of blogs about various enterprise DevOps topics. The series is co-authored by Sacha Labourey, CEO, CloudBees and Nigel Willie, DevOps practitioner.
In our previous blog , we set out some principles on metrics and what and how to measure. In this blog, we'll add a few more insights into metrics and their measurement.
Specific considerations
As stated in our original article, we do not intend to define comprehensive metrics. We do want to take the opportunity to suggest a couple of areas we feel worthy of further consideration which are not regularly discussed. They are:
Lost opportunity time
Business value points
RAG status (red, amber, green)
Unexpected consequences of measurement
Lost opportunity time
Moving to a new way of working and a new culture is difficult. Indeed, it is often far more challenging than the technical challenges. In our experience, in large enterprises, rebuilding the bridge between the business and IT is a critical step. Sometimes IT commences the transformation without sufficient engagement with the business. The product manager is critical, ensuring that the correct initiatives are delivered at the correct time. To measure the success of business engagement, we recommend you consider measuring the time between any deliverable being available for release into production and the actual production release. Please note we are aware of the release challenges of approvals and sign-offs - this is not an attempt to measure this step.
In simple terms, "lost opportunity time" is an attempt to understand whether technologists are currently working on the correct initiatives for the business. If content is being developed which is not required for production release as soon as it is complete, one could argue that the individuals concerned would be more profitably engaged working on other initiatives. Using this metric, we believe business unit and portfolio managers could identify potential areas where effort could be better optimized.
Real-world example : The lost opportunity time metric was triggered by a conversation Nigel had with a team at a former employer, a few years ago. Everybody had been challenged to be more Agile, and a lot of teams were trying to do the right thing. Nigel was chatting to a team who advised they had moved to sprints and had now completed four sprints. He then asked if they had all made it to production, only to be told, “None have yet, as we can’t find anyone in the business to sign off on the release.”
Business value points
This is a common concept. The product team provides an indication of the effort involved in delivering each requirement in their backlog, by the same token business value points are a measure of the value to the product owner of each requirement. The rationale behind the two complementary measures is to enable a balance between effort and value to be achieved. This was adopted by the Agile community as an aide to backlog grooming. In our experience, you should not attempt to standardize the values of these points between product owners and teams; they have value within a team, but not between teams.
It is very easy to find yourself in a position where new, high priority deliverables are passed to the technologists regularly. It is less common for the business to volunteer that something that was a high priority has now assumed lower priority. The net result can be an increase in high priority items over time and the team spinning more and more plates. To an extent this can be inevitable, and increased throughput increases capacity, but it is a significant risk. Taking early steps to enable the enterprise to articulate value and demand can significantly enable the quality of decisions and agility.
Real-world example : A product team has been working through a backlog for some time. Each item on the backlog was provided with business value points by the product owner. We can show that 70% of the business value has been delivered from the backlog. From the effort points for the remaining requirements, we can see based on current cadence it will take six months to deliver the rest of the backlog. Further, no significant new requirements have been added to the backlog by the product owner for a couple of months.
Based on this information, technology management can have a conversation with the business about the likelihood that this product is moved toward BAU soon and members of the team potentially moved to new initiatives on the business backlog.
In short, using metrics of this type can enable the enterprise to better identify the point at which products move from active to BAU or, where resources are constrained, which current initiatives could be quiesced to enable new priorities to commence.
RAG status
In a large enterprise, particularly prior to a DevOps transformation, the technologists often end up under-represented at a senior level within a company. Timeliness and budget are always discussed at a senior level, technical health less regularly. One initiative to encourage a level of discussion on technical health is to incorporate this as a core measure on any executive dashboards. RAG status dashboards (red, amber or green) are common in large companies. Project or programme managers will provide a RAG status for timeliness and one for budget adherence.
We recommend an additional RAG signifier to those that are traditionally discussed. Each product should have a technical lead who is accountable for the technical health of the product. They will have access to metrics that indicate the quality, and quality trend, of the product. There are several products available that provide this type of metric. The information about defects, architectural consistency, maintainability, etc. is not something that can be easily discussed at a senior level. A more useful approach is for the technical lead to use their judgment to provide a RAG status for product quality. This can sit alongside the other two indicators in discussions.
The relationship between the product owner and the technical lead is critical. The technical lead should be in a position where they feel comfortable recommending remedial or evergreening work on the product to the product owner, rather than pure business value delivery. The product owner should be confident enough in the technical lead to represent the need to carry out this work to the business stakeholders.
Real-world example : The quality RAG status is something Nigel has experienced, and has seen it reduce pressure on technologists to just deliver, regardless of the degradation in technical quality.
Beware unexpected consequences
Remember the act of measuring something amends behavior. You should be aware of unexpected impacts on behavior of any metric collection. Using the examples above, you could see:
Soft product launches to artificially reduce lost opportunity time.
Business value inflation of requirements to retain your product team.
Pressure on technologists to change their quality RAG status to avoid difficult discussions. In the extreme, objectives could be written that state “product quality is not to reach red status” which, while admirable, in one way can lead to a suppression of proactive reporting of issues.
Follow the Enterprise DevOps blog series from Sacha and Nigel
Enterprise DevOps: I Wouldn't Start from Here: Understand Your DevOps Starting Point
Enterprise DevOps: The Peoples' Front of Judea, or Don't Sweat the Small Stuff
Enterprise DevOps: 13 Principles of Meaningful Metrics Measurement, Part 1
Enterprise DevOps: 4 Further Considerations for Metrics Measurement, Part 2 (this post)
About the Authors
Sacha Labourey
Sacha is a native of Switzerland and graduated in 1999 from EPFL. While at EPFL, he started his first consulting business - Cogito Informatique. In 2001, he joined Marc Fleury’s JBoss project as a core contributor, and implemented JBoss’ original clustering features. Sacha went on to become GM for JBoss Europe. In this role, he led the strategy and helped to recruit the partners that fueled the company’s growth in that region. In 2005, he was appointed CTO, overseeing all of JBoss engineering.
In June 2006, JBoss was acquired by Red Hat (NYSE:RHT). As CTO, Sacha played a crucial role in integrating and productizing the JBoss software with Red Hat offerings. In 2007, Sacha became co-General Manager of Red Hat’s middleware division. He left Red Hat in 2009 and founded CloudBees in March 2010.
Nigel Willie
With over 20 years’ experience working in IT for one of the world’s largest financial institutions, Nigel has experience managing and delivering global transformation programs. Starting his career as a developer, Nigel’s most recent role was to deliver cross-platform DevOps automation capabilities to the enterprise.
From his experience, he understands many of the challenges and mistakes involved; indeed, he claims to have made most of the mistakes himself. Nigel has also had the good fortune to work with a lot of highly skilled individuals, both as colleagues and across the industry. He is attempting to share some of his personal observations and thoughts in the hope they will be of value to others. Nigel is a great believer that every initiative is individual and any observations he makes are intended as principles or guidelines, not rules.