Agile Maturity and KPIs

The problem with Maturity Models

Maturity models can be useful to kick off a reflection on where your team or organization is right now on its agile path.

However, often times these models get confused with metrics. They provide a number which then suggests something got measured. In reality, though, it’s just a subjective value attached to a high level, abstract question like “On a scale from 1 to 5, how much is your HR a catalyst for agile transformation?”. Uh? 3…-ish?

Our approach is different.

First, in our understanding the word “agile” is only defined by the Agile Manifesto, i.e. its 4 value pairs and 12 principles.

Second, we ask our clients these quantitative questions, referencing the Agile Manifesto:

“Outcome (e.g. working & valuable software), delivered in short cycles”

1: Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.
3: Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.

  1. What is your project’s definition of “delivery”?
  2. When/how often has your project delivered working software in the last 3 months?
  3. Was it actually shipable at those times as well?
  4. What is the cycle time of a feature in your project (measured from “came up with idea” to “delivered”)? (average & variance)
  5. When/how often have you integrated (into) the full system in the last 3 months?
  6. How many pieces of actual user feedback have you received in the last 3 months?
  7. How many of these feedback items were turned into backlog items?

“Welcome Change”

2: Welcome changing requirements, even late in development. Agile processes harness change for the customer’s competitive advantage.

  1. How many changes have been proposed in the last 3 months?
  2. How many of them were implemented?
  3. Out of those that were NOT implemented, how many of them were rejected or postponed because
    1. other backlog items were deemed to yield higher value (for the users, for the business etc.) or lower (risk of) loss (e.g. urgent security fixes, avoiding penalties etc.),
    2. previously made commitments had to be kept (on less valuable items),
      the roadmap couldn’t be changed,
    3. established processes/policies didn’t allow for it (e.g. strict Change Management Process, definition that “an active sprint’s scope must not be changed” etc.),
    4. the architecture/code base wasn’t flexible enough to implement the change,  i.e. the effort for implementation was deemed disproportionate to the expected value,
    5. it was deemed too risky,
      i.e. undetected regression bugs were likely / the effort for testing the change would have been disproportionate to the expected value,
    6. the change couldn’t be broken down into a size that matches the release iteration length?

“Face-to-Face, Business & Development collaboration”

4: Business people and developers must work together daily throughout the project.
6: The most efficient and effective method of conveying information to and within a development team is face-to-face conversation.

  1. When/how often have developers and business people had face-to-face conversations in the last 4 weeks?
  2. How many business people are part of the team and at what is their individual FTE?
  3. When have developers collaborated with relevant people outside the team (stakeholders, customers, business people, domain experts etc.) in the last 3 months?
  4. Face-to-face vs. text communication:
    1. If you use Jira, Redmine or similar, how many comments do tickets have? (average & variance)
    2. How many of them have more than 5 comments?
    3. How many tickets have or refer to descriptions that are in contradiction to agreements in the comments?
  5. How many requirements have been implemented in the last 5 iterations?
    How many of them have been personally presented and explained by business people to the team (or parts of the team)?
    How many of them have been discussed / invited questions?
    How many of them have been modified as a result of the team’s questions and discussion points?

“Technical excellence”

9: Continuous attention to technical excellence and good design enhances agility.

  1. How do you assess the technical excellence in your project and what metrics/evidence is this based on?
  2. When you think of the last 3 changes (see above), how much time was spent on the actual implementation?
  3. Of that time spent, how much was spent on
    1. understanding the existing code base,
    2. refactoring the existing code,
    3. working around issues in the code base?
    4. fixing regression bugs?
  4. How many backlog items exist for technical improvements / resolving technical debt?
    What is their average Cycle Time (measured from “added to backlog” to “done”)?
  5. Out of the last 10 iterations, how often have the Product Owner and the Stakeholders been fully satisfied with the product’s quality, adaptability and implementation speed?

“Sustainability”

8: Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely.

  1. How many people have left the project in the last 6 months?
  2. How many new people have joined the project in the last 6 months?
  3. How long has the project been going on?
  4. How many different Product Owners, Scrum Masters and Architects have worked on the project so far?
  5. What has been the weekly work time for people over the last 3 months?
  6. Which people on the team CANNOT be replaced by a successor or compensated for by the remaining team within 4 weeks?

“Self-organization”

11: The best architectures, requirements, and designs emerge from self-organizing teams.

  1. How many decisions has the team made in the last 3 months?
  2. How many of them were made by individual people, e.g. Scrum Master, Product Owner, Project Manager,
    vs. decisions made by the dev team?
  3. How many of them had to be approved by someone outside the team before they became effective, e.g. the Line Manager, Director of XYZ, CEO etc?
  4. How many of them were opposing or enhancing company guidelines?
  5. How many of them were opposing the Product Owner’s / Project Manager’s, Line Manager’s etc. or any outside stakeholder’s opinion?

“Reflection”

12: At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.

  1. How often has the team met for reflection sessions in the last 3 months?
  2. Has the team or any outside stakeholder had the impression that the project was in a bad situation at least 1 time in the last 3 months?
    If yes, did the team meet for a reflection session / stop-the-line meeting outside their regular meeting schedule?
  3. How many and what changes to the way of working has the team tried in the last 3 months?
  4. How many proposed changes to the way of working were NOT tried in the last 3 months, e.g. they
    1. were turned down by the Product Owner or Scrum Master or an individual lead role on the team,
    2. were turned down by an outside stakeholder, e.g. Head of Department,
    3. were not pursued by the team because they feared rejection or wanted to avoid the need for outside approval,
    4. lacked the time, budget, resources?

Image source: Tumisu from Pixabay