When looking at the transparency of Agile and the granularity of team-based metrics, it is extremely important to be responsible in how you use your measurements. Recorded live on May 31, 2023, Applied Frameworks Joel Bancroft-Connors and Laura Caldie will join to discuss the five principles they use when dealing with Agile Metrics:
1. Measure the project, the teams, and the adoption separately
2. Start collecting metrics early and often
3. Stay focused
4. Be consistent
5. And, most importantly, measure responsibly.
Transcript
Laura Caldie 00:02
Hi there, Joel, how are you doing this morning?
Joel BC 00:16
I’m doing good. How are you?
Laura Caldie 00:18
I’m actually I’m great. It’s a beautiful sunny day in Chicago. And I’ve had my two cups of coffee. So I’m feeling pretty good. How about you?
Joel BC 00:27
Oh, looks like it’s sunny in the Pacific Northwest, which is once you always have to double check.
Laura Caldie 00:32
Exactly. And it could change in a minute. So who knows what it’ll be at the end of our time together.
Joel BC 00:37
Yes.
Laura Caldie 00:39
All right. Well, we’ve got people trickling in to the the Zoom platform. So I guess why don’t we take care of some housekeeping stuff while people do that. So first off, I just want to say thank you for everyone. For being here. My name is Laura Caldie. And I lead sales and marketing at Applied frameworks. And Joel and I’ve had the chance to work together on a number of engagements where he’s kind of the the expert on agility and Scrum for teams. And for scaling, and I’m his support supporting cast member and a lot of those engagements. So we’ve had the chance to talk about metrics frequently so I’m happy to be able to support Joel in this it’s I know it’s one of his areas of passion. And, yeah, I guess I’m based in Chicago and happy to be here with Joel. So let’s see, I want a little bit about Applied Frameworks. We are known for helping companies design and develop profitable software. So if you think about all the solutions out there that are driven by software, our goal is to help companies organize around that value and then to build and deliver a solution that sustainably profitable. So we have a new book out called Software profit streams. And we also have a lot of resources on profit-streams.com. And so go ahead and check that out as well. Of course, all that information will be in the follow up email. So we are recording this and we will send out a link to everyone who’s registered for the course and will include some new interesting information into that. The email that goes along with it. So for anything that we do mention, we’ll reference that in the follow up email. So the last thing I’ll say before I turn it over to Joel is please enter any questions that you have as we go forward into the q&a. We’ll do our best to save some time for Joel to answer those but if there are things that we only partially answer, if we don’t get to everything, then they will be found in that follow up email to everybody. So with that, let me turn it over to Joel to introduce yourself.
Joel BC 02:46
Ah, so I’m Joel Bancroft Connors. I’m the gorilla coach. I am a Certified Scrum trainer, certified team coach, just to have a second
Laura Caldie 02:58
and he also has the cutest dog and two cats and they’re probably messing around in the background right now. Just my guess.
Joel BC 03:06
Deal with the Oh yeah. Let’s not get it up to the middle middle of this. So yeah, I’m a Certified Scrum coach Certified Scrum trainer. It means I’ve got the dual dual skills have both been able to really help people to understand how to use Scrum well and then actually have done it and helped teams to implement it in person. You can find me on LinkedIn. I also have a new YouTube channel where I’m tackling simple little problems to help people to get better. As Laura said, I’ve got dogs. I’ve got cats. I live in the Pacific Northwest on six acres, so there’s always a lot going on around me. I am a recovering project manager. It has been over 13 years since I opened a Gantt chart I did once upon a time say you will pry waterfall from my cold dead hands. So when it comes to Agile resistors I was one of the originals. And my favorite agile principle is number five build projects around motivated individuals given the environment and support they need and trust them to get the job done. So I’m looking forward to being with you all here as we talk about this topic of the responsible use for agile metrics. It’s gonna go back a slide. This builds on a webinar Laura and I did previously where we talked about agile team metrics and what metrics would you use with a team? And in that one was very specifically talking about metrics. It’ll be used for a team today. Instead, what we want to talk about is how do we responsibly use map metrics. And we’ll start with the same quote we started with the last last one. If you cannot measure it, you cannot improve it. So that is a very important thing is we want to improve as organizations Agile is about inspection, adaption, transparency, inspection adaptation. If you don’t have transparency, you can’t improve transparency means that you usually have to find some way to measure it some way to track that work. So let’s summarize the the five principles and these are ones Laura and I’ve talked about a lot because it’s the it’s great to have metrics at the team level. If your organization doesn’t know how to responsibly use them. It’s like having
Laura Caldie 05:32
Oh, I can’t help you with the analogy sorry.
Joel BC 05:35
Well, no, I would find algae. What No, no, that was probably not a good analogy. Is this. It’s if you don’t have good principles around the metrics you use, you’re gonna have really, really great tools without the training, how to use them safely.
Joel BC 05:58
So here’s a question and I’m gonna ask it to learn I also have my own answer. It’s like what it’s a something I’m a really big fan of a marketplace on podcast called Make me smart. And one of the questions they always like to ask is What is something you thought you knew but later found out you were wrong about with metrics, Laura, what’s something you thought you knew about metrics that you later found out? You are walking about?
Laura Caldie 06:19
Yeah, so it’s gonna give you a general answer. But I originally thought that there were three metrics that you had to track all the time and there weren’t anything else and as long as you did those three they were You were set for life. What I learned later, is that the collection and use of metrics and also how they support decision making, they should evolve over time and they may change and what you’re trying to improve or measure is obviously not going to stay the same. So metrics is a fluid thing. And originally I was thinking that it was like permanent rules.
Joel BC 06:54
Yeah, that’s That’s it. That’s a great one for me. It was measuring is not about getting to a goal. It’s about knowing where you are. And I like to think about good hearts law, and good arts law is a British economic economist. There’s a very fancy economist speak version of his law, but the English version of it is when a measure becomes a target, it ceases to be a good measure. And I always thought metrics are about getting to that goal and instead really the realization that metrics are about helping you to make that adaption transparency, inspection adaptation, the three pillars of Scrum. If you have transparency of what’s going on good metrics. It allows you to then inspect where you are and make good decisions about where do you want to go. So let’s talk about these, these principles in more detail. And again, as a reminder, we’ve got the q&a button on your zoom bar. If you click that, you can type questions and we will try and answer them. Feel free to say hello in zoom chat if we I don’t know if we have zoom chat enabled for people to like introduce themselves or not. Laura, do we?
Laura Caldie 08:07
Ah, that is a super good question. I didn’t double check. It looks like it looks like it would work for everybody but I’m not sure.
Joel BC 08:16
Okay. Well, hello, everyone. Let’s go ahead and dive in. But yeah, if you have questions, please, please go. Drop them in. We will try and answer them as we go. So the first part of this is measure separately. You mentioned the project, the teams and the adoption separately, as Laura was just saying she used to think there were just three metrics. Well, can’t measure everything that goes on an organization you really can’t measure all with just three metrics. You have a project, you have teams and you have adoption, really when you think about what you’re doing. So what is the project and how do you measure that? There you’re really wanting to start to look at things that tie into how do you measure value or more importantly, how do you measure profitability? At the teams you want to be measuring of are the teams sustainable? Are they healthy? Are they growing? And then you also want to look at with teams, what is the work they’re doing, what’s the outcome that they’re generating? And then you want to measure your adoption or your your your change to better ways of working completely separately from the teams as well. You got to remember just because your team is doing great doesn’t mean that the project is on track or that the Agile adoption is sustainable. I have dealt with organizations where look, the teams were happy they were they were they were cheerful they were they were really enjoying themselves. They were getting a lot of work done. And yet the project was completely failing because they weren’t actually connecting the work they were doing to profitability.
Joel BC 09:48
So what do you want to measure? With the team? Things you want to measure are things like cycle time, escape defect planted, done and happy. So we talked all about these in detail in a previous webinar that we did. And there’s also an article on applied frameworks.com If you go to search, it’s actually the very first thing that appears when you in the search bar is agile metrics. For for metrics for team team use on the project or product you want to be what’s your revenue, what your cost savings, what’s your business value plan to done your net promoter or your fit to purpose and these days and Laura will talk about it more broadly at the end and you’ll see it in the email the we have the book software, profit streams, right here and software profit streams really has a lot of stuff around making sure that your project or your product is more successful. Adoption. How do you measure if you’re an agile adoption, your the work you’re doing to improve your ways of working is it accessible, large scale retrospectives, we have a webinar. We have a article on that we do a webinar on lunch go introspective or account
Laura Caldie 10:58
I think we did ages ago. I’ll link to it when we send out the email
Joel BC 11:04
Yeah. And then organizational assessment tools. Tools, like comparative agility offer great organizational assessments that can help you to judge whether or not your adoption of new techniques are working. Things you don’t want to measure. You don’t want to measure velocity. Velocity is individual to a team. It’s a planning tool. And really the only thing velocity answers well is Are we done yet? No, sorry. It’s really those a trick question. It’s really only answering the question, How much work do we think we can take on in the next sprint? Also be aware of vanity metrics, lines of code bugs reported during a sprint, new users pages? These are all horrible things to measure. And it goes back to that good hearts law. Whenever a measure becomes a goal. It is no longer a good measure. There’s this wonderful story from a really great book that if you’ve never read I highly recommend called Freakonomics. And in Freakonomics, there’s this story about the turn of the century turn at the turn of the 20th century in India and Delhi. They were having a huge problem with Cobras and the British government which ruled India at the time, had a brilliant idea and they offered a bounty on COBRA heads. Every time you turned into Cobra head, you would get you would get a cash bounty. Well, that was a great incentive to get rid of Cobras. It was also a great incentive to create Cobra farms and start farming cobras well, so And suddenly, the British government was putting out a lot of money paying for these Cobra heads when they discovered though that it was all because of these Cobra farms. And it wasn’t actually dealing with the Cobras in the wild. They canceled the program which then meant all of these farmers with cobras no longer how to use for them and so they just let them go. And so the Cobra population in and around Delhi exploded, didn’t go down. So you have to be very, very careful with vanity metrics because they are very easy to become goals and from goals. It’s very easy to get abused. Any other thoughts here, Laura, any questions?
Laura Caldie 13:13
No. It’s it’s a good story though.
Joel BC 13:17
Yeah. All right. Next, or responsible use of metrics is start early. Collect your agile metrics early and often. It’s really hard to know where you are if you don’t know where you started. A solid trend can help answer the question of when will we be done. Years ago, I was a full time coach at AOL. And when I came on board, the teams have been doing Agile for a while they have metrics and everything. And I started by observing and the first thing I did was I created measurements around the teams, those format measurements that we talked about in the other in the other webinar in the article. And I did that for three sprints before I ever talked to any teams. So we had a trend already before I talked to teams and then we continued to measure them by measuring early and often you start you get a better idea of where are you and where are you going. If you’re trying to get to New York, and you don’t know if you are in Miami or in Los Angeles, it’s going to be really hard to get to New York. You don’t know where you started from. So you create those baseline metrics. Figure out where you are. A lot of people don’t want to do that right away. Like oh, well, we don’t want to know how bad we are. How do you ever know how much you’re going to improve if you don’t know how, where you were when you started? So, again, we go into more details in the other webinar. Here’s examples of these measuring and be able to tell a story because we’ve got that that data. So here we’re looking at those team metrics that we’ve talked about cycle time, planted on escape effect, and happiness. And these are all this is all real data back from my days at AOL. And the first thing you can see is when this team started at a cycle time of almost 13 days, their two week sprints that meant every story on every story on average was taking more than a sprint to complete by the time they have when we’re measuring now which is only about six Sprint’s later they’ve reduced their cycle time to just under four days. At the same time that predictability went from around 50% meaning only 50% of the work they said they would do in sprint planning did they actually get done to over 90% and because they have the vil that happened because they literally have the visibility, they were able to start seeing that they weren’t making making good on their commitments. And we can see also similar stuff with quality and with the happiness or health metric. So by measuring early and often you can start to get these trends you can start to see what’s going on and you can start to ask questions that can help you to be better. And again, this goes back to that transparency, inspection adaptation, the three pillars of Scrum. If you don’t have transparency, you’re never going to be able to say, well, how are we actually doing? You’re gonna go back into the battle days of stoplight project management or as I used to call them watermelon projects. green on the outside red in the middle.
Laura Caldie 16:18
Schedule it could you talk just a little bit more about measuring velocity. So exactly what is it and you know, fill us in a little bit more and there’s a question that’s in the q&a about whether you know, is it the same as increments and so if you want to talk to us a little bit more about velocity before we go on?
Joel BC 16:37
Yeah, so I like to talk about these what I call the team health metrics are the metric the team can use to improve themselves as a team velocity on the other hand, is what I like to think of as a planning metric. Velocity the idea of velocity is what what have you done before or yesterday’s whether three Sprint’s ago we we complete the 10 stories two Sprint’s ago we completed 12 stories? last sprint we complete the nine stories, when we average that out, the average is about 10. So we say okay, we’ve got a velocity of about 10 stories per sprint. That is a useful planning measure, measure because then when you go into sprint planning, you can ask yourself, you can say okay, how much work do we think we can get done in this sprint? Well based on the last three, Sprint’s we averaged around 10. So maybe we start strong start by looking around nine to 10 stories to take into the sprint. It’s a good guideline for a team to be able to be able to start to planning. One of the important things though is remember your team is self managed, and they’re really smart. Sometimes they’re gonna say okay, great. Yeah, we we’ve got a velocity of 10 stories. These ones though, are a lot more complex. We’re only going to take on six this sprint. It gives you a guideline for the team to ask how much work can we put take on sometimes it is used for predictability? There, it’s the Okay, great. We know that there are 440 stories in the product backlog, and it takes the team the team on average is getting five stories done per sprint, they’re going to probably take about eight stories. It’s not the greatest way of doing things and we’ll actually talk about that in an upcoming webinar. We’re going to talk about better ways to predict capacity through estimation and better techniques for how to say when will we be done? Hopefully that answers the
Laura Caldie 18:30
question. You know, I think it probably does. And before we move on, I can’t remember specifically if we talk about this in the slides. So there are two questions that just came into chat that I think are related. How are these measured? Typically, and you know, I was going to mention the team happiness one which is it really is a self reported. answer to a question and and David says, you know, in what way are you thinking about happiness? Is it just, I’m paid well, or is it something more profound, like I understand what I’m working on and how it’s connected to the bigger mission, and I’m feeling like I’m contributing to something that’s bigger than myself, right? It’s almost like where does happiness come from? So my my field and I’ve been you know, I’ve seen you do this before, but how you ask the question will get you the answer. You’re looking for it so, but it is a self reported method for the team reports back in mostly an anonymous way how they’re feeling about whatever question you posed to them. Right. So the question you asked is pretty important, and then I’ll let you comment on the rest of them as well.
Joel BC 19:36
No, so I don’t one thing is I don’t want to go too deep down this rabbit hole, because again, we have a whole article on just this section of this webinar. We have a whole article and a whole webinar on it, but you’ll see somebody’s asking for a link. I’m gonna take that up right now and stick into Yeah, Laurel, dig that up, and it’ll also go out in the email. At a high level though. Team Admin is one of the things here is context. Context matters. And with I say this with retrospectives, and also with Team happiness, you need to ask the right question. Generally, the question that I have found to be successful is during a retrospective actually, the way I like to kick off retrospectives is just to just to get that happiness metric. And you ask, well, how are you feeling about the team, the sprint, or how are you feeling about the work that got done? And here, this is really about the team itself. And so the team might not be generating great value. And that’s okay, because what you’re you’re going is how is this team doing as a cohesion? So how do you feel about working in this team team, this sprint and so that’s what you what you ask. I do like the the whole how does this tie to purpose and autonomy and mastery and purpose, which goes to Dan Pink, we actually have another webinar that just happened a few weeks ago where myself and a colleague Kim Prensky, were talking about how to bring more purpose to the team and that’s really the secret there. is you’ve got to help them to understand how what they’re building today connects to the overall big picture vision. That’s not what happiness is really intended to address. It’s more of how is the team as as an entity working together are they able to make any commitments together and solve problems together? Or is there a lot of fraction fractures and conflict in the team? So really, Team happiness is more around what we think of as the Tuchman model forming, storming, norming performing or the Satir change curve where you have a team that’s going through change and then gets better. Team happiness is one of those ways of saying, okay, is this team going to be able to continue to be productive, continue to be predictable? One of the reasons I’m a real big fan about happiness is it’s one of the few leading indicators. You can actually see that in the this, this diagram here is the team was at 4.6 happiness and it dropped to 3.5. They had a team member removed from their team without consultation that was moved off somewhere else. And you actually saw a small spike in cycle time, and you saw a dip in, planted done. Now they were fairly small, but they weren’t they were noticeable considering the fact that this team had been in a very steady, steady direction. And so you were able to see hey, this team is starting to starting to starting to dip here and sprint five as one and you started to see the that impact start to happen. A sprint or two later. And again, Laura has put the links to that webinar, and I definitely recommend watching that recording if you want to learn more about the team based metrics let’s go on this just an example. The these kinds of metrics also exist in Scaled Agile, and so you can look at the overall multiple teams with predictability, Manager Program predictability and also business outcome measures. Alright, so number three, stay focused. We’re going to limit your measures to three to five items. Now again, this is based on where you are on project team or adoption, you want to limit to three to five measures. For example, again, in in the teams we just have those four metrics that we recommend because they self-supporting If you gain cycle time, it usually will affect quality and such So stay focused. You’re measuring too many things at once. You’re, you’re you’re you’re doing multitasking and nothing’s gonna get measured well. So figure out what is that key indicator that helps you to know that your product is doing your product is on track? What are the key measures that are going to help your team and then what are the key measures for measuring adoption and this is a place where a lot of times we’ll see these huge dashboards for adoption and most of the those adoption measures are pure vanity metrics, or worked with one organization where the primary measure of success was how many teams had been trained, and it’s like, okay, great. And so they were going oh, we’re doing wonderful. Within three months, that agile Center of Excellence hadn’t been disbanded because it was not actually providing any impact to the organization because they weren’t actually generating any better outcome. Next, be consistent. Keep your agile metrics consistent in terms of teams working on the same product need to be using the same measures or you’re literally have apples and oranges. If you’ve got five teams working together, they should all be using the same team metrics. Again, I recommend cycle time. Plan to Dawn escape defect and happiness. But if you have different teams measuring different things that can be a problem. This is you can do this, even if you’ve got teams doing Scrum. And Kanban. With that, the way you do that is you create a common common sight cycle of measurement. When I worked at AOL, we have teams working in two weeks sprints, and we have teams running in Kanban. What we did was everybody was in JIRA and we structured the teams and Kanban they actually created instead of using a Kanban board in in JIRA, they used a sprint board and they set it to one month increments and once a month, they would basically go okay, let’s close this sprint and use that to pull metrics and because they were pulling their metrics every four weeks, and the other teams were pulling their metrics every two weeks, it was very easy for us to start to look at organizationally, why? How are we doing things and we reported metrics up to leadership on a one month basis, aggregating all of the team’s data together. And so leadership didn’t actually see how Team A was doing or Team D was doing. Leadership just saw how was the sales side teams doing and how were the buy side teams doing? So I was on I worked in the AOL video advertising division. And so it allowed you to aggregate up to a larger thing. This does mean you need some consistencies. Everything needs to be divisible or running on the same cycle. So I recommend Scrum teams running on the same cadence as every two weeks and Kanban teams find a regular cadence for when you want to check check them their metrics
Joel BC 26:27
so I saw a question coming back to the happiness and the anonymity part of it is where are you in the health of your team? And when you’re when you’re first doing this, yeah out allowing them to report anonymously writing a post it note and folding it or I worked with a really brilliant scrum master when I was doing work with BMW, and he actually, he would send out an anonymous Google survey before the retrospective. And so then they were able to have the conversation there. The next thing though, is you then need to have be able to have conversations like Okay, so, we had some people that put in twos, does anybody want to talk about why we might have seen TOS or what might have caused those tos? And literally just that barrier of that anonymity of nobody knowing exactly who put in the TOS, somebody could say, well, could it be because of XY and Z? And so you start to create some safety there. The obvious goal is you want to create as much safety as you can in those retrospectives. No managers in the retrospective what happens in the retrospective stays in the retrospective and over time, you’re going to get to the point where if you’ve got a well functioning team, they’re going to be able to justify their happiness at the beginning of the meeting because they feel safe enough. This really goes back to that do we have courage in our organization, one of the five values of Scrum, you think about courage? A lot of really, do we have psychological safety? So there it’s the how do you promote psychological safety in the team? Yeah, Laura.
Laura Caldie 27:48
I was just gonna say I was typing an answer in chat to that and my answer was actually pretty similar to what you were talking about, which is, like, you got to kind of dig sometimes it gives you a topic for your retrospective potentially. And sometimes the reasons are pretty obvious. You lose a really productive well loved team member and yeah, people feel sad they might give it to and it’s not something that you can truly act on. It’s just acknowledging that we lost a person that is pretty important to our team versus, hey, our quality is dipping because there’s some tooling issue or something more similar to that. So
Joel BC 28:26
I want to be real Chris Yeah. So the webinar we did earlier was very specifically how to create team based metrics and how to make them successful. We actually have a spreadsheet template on our website that’s free to download that you can use that will allow you to use to do this kind of metrics tracking this metric. This webinar is more about the target. The target audience really is leaders and how do you use metrics responsibly? Once you’ve got measures how do you use them responsibly so you’re not using them for evil?
Laura Caldie 29:01
You know, that actually gets into Andrews question, which is if the you know a good practice is to not use metrics as a target. The reality is the business and leaders actually appreciate knowing what the predictability measures are because that creates this long running confidence that the teams can deliver what and when they say they can deliver it. So having a high predictability metric is a is an ingredient to a super healthy relationship between the technology teams and business teams.
Joel BC 29:33
Yeah. All the time. When I work with teams, they go but leadership’s doesn’t trust us, and they’re always micromanaging. And I tell them look, the number one way to get leadership to start trusting you is to make ends meet your commitments. And that is reflected in your plan to done ratio. If you say you’re going to do 10 things in a sprint, do those 10 things. Don’t say you’re gonna do 30 things and only do 10 Don’t say you’re gonna do 10 things and then we’ll swap out those 10 things for a brand new things in the middle of the sprint. Yeah,
Laura Caldie 30:05
yeah, I think maybe the reason why you wouldn’t want to make it a hard target is that you know, it’s it’s easy to gain that one right where if making immediate commitments is the thing and you always have to hit 90% or you’re somehow a team in trouble, then you just plan for a whole lot less than your sprint and you’re always hitting it and that’s not necessarily meeting the needs of the business. Right. So I think the idea is, you want the measure to be something that helps guide business and the teams to make the environment such where it is possible to make and meet your commitments. If it becomes a tool for I don’t know, punishment of some kind, then it no longer becomes effective for the business or the teams to be measuring in the first place. So it’s, you know, at target as in how can we make sure we’re making it possible for teams to make and meet commitments? What’s my responsibility as a leader to make that true is how it can be used. And if you switch it and say, and if you don’t hit 90%, you are losing your job, it becomes a completely different game. That’s not really helpful. For anybody.
Joel BC 31:12
Yep. And that actually brings us to the fifth one measure responsibility, measure with care or there will be nothing to measure. And this is a really big thing for leaders and it goes directly into the question that that question is, I love these two quotes. One is tell me how you how you measure me and I will tell you how I will behave by Eli Goldratt. He’s the author of the goal, really great book really talks on there’s a lot in there about measurement and everything and what how to look at it responsibly. And the Winston Churchill quote, no Uncle Ben from Spider Man did not say this. First, where there is great power, there is great responsibility. Metrics are incredibly powerful as leaders, you’ve got to be very careful with how you use them if you’re set using them to set goals. That’s exactly what you’re going to get people are going to meet the goal. Is that goal healthy anymore? If you say look, every you have to have 100 100% predictability. You know, teams probably that are going to be under committing they’re never going to be pushing the envelope. One of the things I like to think about is when you’re looking at, let’s go back up here with the team ones for example, I tend to see anything over about 85% in predictability, I will start to look at to go Are they gaming the system? And so I would look there, if your cycle time in if your cycle time just is completely crashed. Then I’m going to be looking at going okay great is your quality good? And that’s one of the reasons why you have self supporting metrics. The advice I like to give to leaders on the use of metrics is metrics allow you to ask questions, and the number one question a leader should ask when they see a metric that doesn’t make sense. Is what can I do to help? So if the leader sees that the happiness has dropped? That leader could be that that manager could be asking the scrum team Hey, the Scrum Master, what can I do to help? What is there anything I can do to help Help Help help your team? And that’s where the scrum master game can go. Look, the teams upset because I had this person move and there was no explanation maybe if you came in and gave a better explanation of the big picture what what was the big picture decision for for making this change? That might help them better?
Joel BC 33:41
So one of the things I’m realizing is I didn’t add anything about profit stream into this. I’m the again the book profit stream just came out in April. Software, profit profit streams, you’ll there’ll be more in the email that’ll follow up here. That’s where you want to be again. What is your How are you measuring your product? How are you measuring your team? How are you measuring your adoption? When you’re measuring your product? The big thing here is a couple of great quotes that have been out recently one comes from the fields the former chief digital officer, I think it was McDonald’s and a couple of other companies. And he said basically, the days of blank check innovation are over. And so we can’t just be innovating for innovation safe, and then look home and when he talks about the books off our profit streams, says it is no longer enough to generate value we have to be generating profitability. And so that’s a place where leaders really should be focusing more on is are we getting profitability out of the product. We’re getting profitability out of the product. That’s our goal. We can set goals there. It’s up to the teams to help to figure out how they can get beam deliver more healthily, to get to that profitability. And it goes back to that question somebody had of that purpose in and then the webinar that Kim and I gave, which I like. I marketing, I think gave it a much nicer name for the website. But the the actual slide slide deck is called My sprint team doesn’t give a What should I do? And that’s the look of your team doesn’t know what they’re building and why. Then they’re never they’re never going to build their two metrics are always going to start with so a lot of this starts with it. They don’t understand the purpose, then they’re not going to know what they should be trying to help the company to work towards. And then they can’t answer the question. themselves, how are we going to get better? So those are the five, five measures so responsible measuring a couple of do’s some do’s and don’ts. Do look forward metrics are mostly lagging, so don’t use them to analyze past performance. Use them to plan for the future. If you’re if your cycle time was horrible the last three Sprint’s you go okay, what is it we need to do to improve our cycle time? Not why was a cycle time so horrible? It’s like how do we learn from this? Supporting metrics interlocking metrics revealed better data and are harder to influence. That’s the reason why I’m a proponent of the four metrics that I T metrics I have. If you crash your cycle time and you’ve got a cycle time of a half a half a day. If you’re actually crashing it and you’re doing your gaming, your escape defect rate is going to rise and you might have great predictability. But cycle time might be high because it’s like oh, well, we got 100% predictability but our cycle times 10 days because we’re taking we’re taking very little stuff taking forever. And overall your happiness metric is that that one leading indicator that can help you to go ahead, look, the team’s doing great everywhere else but their happiness to trashing what’s going on and how do we prevent everything else from falling apart? And then customer focused and this goes back to profit profit streams, measure the product or product using value based measures. And honestly, what I need to do here is I need to change that to be profit these metrics. One big thing here is sustainability. Sustainability is an important part of our global global environment today, no matter how you feel about it. Everybody agrees that if we don’t have enough resources, and we don’t have enough to keep going, we’re going to run we’re going to run out and that’s going to be a problem. In business. There are three P’s to sustainability, people process and profitability. The people do you have good team measures, the process? Is the adoption working or do you actually know how to make immediate commitments and then profitability if a business is not profitable, it can’t be sustainable. And if it’s not sustainable, profitable, profitable wise, it’s not going to be able to contribute to the global economy, for sustainable environment and sustainable people and sustainable practices. So profitability is a core concept that you have to always be thinking about. As a leader with measurements Don’t Don’t worry about is the team doing have a cycle time of five or three days. Ask yourself, Does the solution does the cost to build this product cost more than it does to sell it? Than we get when we sell this product? And then don’t don’t measure team against team. They’re apples and oranges. I worked with two teams that both have the same exact same product or work from the same product backlog. One of those teams, their average story size when they estimated was 13 points. The other team the average story size was five points. You looked at the number of points that got done one of these teams was getting like 100 points of Sprint done and the other one was like getting 40 points of Sprint done. That’s not what you measure because that’s their team’s specific. The next one is don’t react too quickly. one data point does not a trend make at least three Sprint’s or at least three data points before you start to make any decision. All too often I see organizations just starting into a scrum, Scrum adoption. And the first two one or two sprints go horribly and they just go and they throw in the towel. It’s like look, no until we get three sprints. You don’t even have to have a data trend much less know whether or not you’re actually going in the right direction. And also don’t forecast with estimates. Again in July. We’re going to be doing a webinar with Troy McGinnis we come in and talking and we’ll talk more about stochastic and estimation we’re using empirical data to forecast use Cycle Time to forecast not estimates. It’s like okay, the team’s got an average cycle time of four days. And so now we know how much work we think they can get done over the next three sprints as opposed to well, let’s look at their estimates because remember, at the end of the day, an estimate is really nothing more than an educated guess. So those are responsible measuring do’s and don’ts. Did you have any comments or questions for
Laura Caldie 39:59
No, but I do you know the to an untrained metric user it is almost it seems appealing to say okay, if one team has, you know, they can do I don’t know twice as many story points as they as another team to think about what’s wrong with the team that can’t produce as much and I think it’s a you know, it doesn’t take much to understand how that initial metric is even created and how teams because they’re doing it as a single unit. They’re not comparable, right? It’s the it’s so it’s just interesting that on the surface, it may look like something you need to think about and you just dig into it for you know, with your eyes open and realize that it’s actually a false set, right. You don’t want to use as I just think it’s important to remember that about comparing teams. It’s that’s not the right metric to ever be using.
Joel BC 40:56
Yeah, and even I mean, some people will go oh, well let’s let’s get everybody on on outcome based metrics and and so Okay, we’re gonna look at what’s the business value generated by teams and even if you were to tie that directly to dollar value, even that’s hard, because maybe this team is getting handed the really gnarly, hard to solve problems for the cutting edge part of the technology. And this team over here is doing a lot of the let’s just add the new functions, the simple new functionalities these new plug and play things. And so this team over here seems to be generating a lot of business value. And this team over here seems to be generating none. However, they’re they’re solving problems for the future. The real thing you want to be looking at and the way to use that is you look at an individual team and you look at whether the first thing is really are they stable? If one sprint they did 100 points and next sprint they did 20 points and then they do 40 points and then they do 150 That’s not a stable team. So the first thing you want to be looking at is, is there low variation from Sprint to sprint on them on the metrics? And then just are they generally improving and improving to a higher higher level? Again, be careful because you don’t want to improve too much. If your team is regularly got a 100% predictability, you might start to ask Are they not pushing themselves any further, but it’s metrics are to allow the team to ask the question, how can we be more effective? How can we improve our quality which are the two questions that you’re supposed to ask in every retrospective?
Joel BC 42:31
Alright, so that is the five responsible uses. Let me go back just to some measure the project and the teams and the adoption separately. Start collecting metrics early and often. Stay focused, be consistent, and most importantly, measure responsibly. Don’t don’t use metrics as a weapon use metrics. As a way to ask questions.
Laura Caldie 42:59
All right, Joel. So what’s the question that you hope somebody asks you if
Joel BC 43:09
you’re good at asking those questions that make me think was the question I hope somebody’s gonna ask me.
Laura Caldie 43:13
Well, you’re, I’ll ask you one. So if I’m on a team, and you mentioned a couple of times during the webinar that thinking about sustainable profitability is certainly important, especially when your value stream is really a profit stream. Right? There’s a so if you’re a team member, and maybe not even the product owner, how should I in that team be thinking about profitability is a measure of my contribution to the organization and to a successful product?
Joel BC 43:45
Um, so that’s a great question. I think part of this is it really comes to traceability and it’s not so much a metric as the how do we trace the work that an individual is doing to the to the greater goal because I don’t necessarily need to know how much what I’m specifically doing is gender is contributing to the bottom line. Overall, you will you’re talking about the solution ROI. What does it cost to build the solution? The cost of your Scrum teams, the cost of your Kanban teams, the cost of third party licenses, that you have the cost of your data center, that’s all your solution ROI. The big thing that a developer should be asking is, is there any way that I can do this more effectively? And can they connect the work that they’re doing up to that big picture and that’s that traceability, it’s like okay, here’s the product. Do I know how my work is doing to contribute to to the delivery of that product? If you think about this in story hierarchy, is to we’ve got an initiative that’s made, their epics and their features, their stories and their tasks. If you can show how your task directly connects to the initiative, then then you’re being successful. Yeah, I agree.
Laura Caldie 45:06
That one of the questions that somebody asked earlier on as well, which is, you know, understanding the big vision for a solution and both from a leadership perspective, having confidence because of that transparency, that I can tell that what the teams are working on, contributes to the vision of the solution that we’re building, and vice versa, as a member of a team. I want to know that what I’m doing matters. And, you know, in one sense, I could be building a I don’t know like, the next rev of an API structure and it may not it may not be fun work for me to be working on on the API’s. Unless I realized that once these are completed, we’ve just unlocked a new set of customers or a new set of of, I don’t know revenue into an in creating a more successful solution. So on one hand, a set of tasks might not inspire me very much unless I realize how valuable it is. If I can see the connection back up again, that might change how I feel about my role.
Joel BC 46:05
Yeah, Luke Hohmann, founder of applied frameworks has a great story of working with a company that was building software, or solar panels on people’s ropes. And it was all about connecting back to the grid, and helping this API developer to understand it’s like, Look, I’m just doing this API stuff down here that makes this stuff possible in this plugin. It’s like, why did why does that matter? Why don’t why do I care? And he said, Look, your API is allowing the creation of some universal connectors to be able to allow your solar your solar solar cell company to be able to connect with any any power grid in the United States. By being able to do that. It’s going to make any power plant, any power company more able to get power back from people use excess solar solar power from people. Because of that, they’re going to have to build less new power plants to handle rising demand demand, because they have to build less new power plants. They’re not going to be contributing as many carbon emissions to the environment, which means that they’re potentially going to help the environment to be more sustainable. You are helping to prevent global warming. And that person is like, oh, okay, I get that’s literally like that’s an ultimate purpose connection. Sometimes you’re not going to have that great, huge purpose, but creating our purpose. David asks, In the question, what’s a metric you found to inspire teams to be engaged? Honestly, I’m not even sure it’s a metric. It’s the the feedback from the stakeholders the feedback from from the market all too often, we treat our developers like mushrooms. They’re in a cave. We feed them, and we never show them the light. And we never show them what people are saying they never get to see how is what they do connect to the user. How does the Connect? Is the user using this? They don’t they’re not participating in sprint reviews with stakeholders. They’re not hearing from the customers. directly. So really, that’s honestly the metric is the What’s that feedback? So in a lot of ways it’s going to be your net promoter score is one of the best things that’s a net promoter or a similar score that can help you to judge how is the customer the user receiving the work that you’ve done?
Laura Caldie 48:17
And then there’s another one and chat about capturing and calculating the metric of fit, fit for purpose.
Joel BC 48:26
I will admit I am not as familiar with fit for purpose. We had a colleague at Applied frameworks two years ago who was an expert on different purpose. He went off and I became a VP of Engineering at some company. So I would, I would defer. I would defer to chat but GPT on this one. Because it’s while I while I have great respect for the person recommended I never personally have put it in fit for purpose. Myself, I think, really, I would probably start with more of a fundamental mental Foundation, like the software, profit stream, canvas of are you building a product that at the end of the day, the idea is, are you able to sell the product for more than it costs? You to make an end is the customer getting more value than what they feel like they have paid for it? And that I think really is probably going to be a better area for us to start looking at profitability and that I’m product success more than NPS because I mean, everybody knows NPS Now and they know well if I give if I if I give an eight I can give a really high score without actually committing anything but if I give a nine then that’s going to count for something. So really, I mean, at the end of the day, are you is the company profitable is one of the best ways to measure whether or not your product is successful. If you’re an internal, a lot of internal teams that will how does that matter for it for me? They’re your internal teams, you should understand what your solution ROI is, how much does it cost you to to support this product and what’s the difference between that and what you were you were able to sell the product for? And so there your real metric is can we reduce our solution ROI? Because if you can reduce your solution ROI without sacrificing quality, the company can make more money without even charging more.
Laura Caldie 50:22
Yeah, I mean, I actually think that’s a good question for us to noodle on a little bit more and include some more thoughts in our follow up email. So
Joel BC 50:32
yeah, maybe that’s I mean, I mean, it might be a whole nother conversation. Yeah.
Laura Caldie 50:35
Yeah. It’s I think there’s I think it’s a I think there’s it’s a system right in thinking about fit, fit for purpose, measuring that and then what do you do with the results like how do you let’s just say you, you get a simple measure, it’s a 10. What do you how do you take action against that knowledge? Are you looking at trends are you looking at, you know, is that more of a customer oriented question? So we’ll think about it a little bit more. But I think what you’re getting at is, is a systems approach to that question, and how do you want to look at the different dimensions of answering it and then what action as a leader do you want to take with that data? So I’m saying all this out loud, because we’ll remember that that’s what we wanted. To think about after the webinar is over. And then I will take a crack at answering that and the email that we send out.
Joel BC 51:25
And then there is a question about NPS surveys, just if so an NPS stands for Net Promoter Score. If you’ve ever seen been given a question of, would you recommend this product to somebody and you see a scale of one to 10 That’s a net promoter score, Net Promoter Score. The way it works is if you give a nine or a 10 you’re considered a promoter. If you give a six or less you’re considered a detractor and or I think it’s a five or less and then six, seven and eight, is you’re just neutral. And the idea is with net promoter score, and the thing that really boggles people’s brains is a zero is actually a good net promoter score because it just means people are completely neutral a one a one means there are more people that like like your product than don’t like your product. And so that’s net promoter. And the problem is, it’s become well established enough that it’s, it kind of becomes easy to game. And so it’s not as easy to get there. Generally, you’re sending this out as some kind of customer survey, and you’re gonna get a very straightforward question like, Would you recommend this to another recommend this product to somebody else is the most common Net Promoter Score question you get, you can also there is the idea of employee net promoter and they’re usually the question is, would you recommend our company as a place to work? And it’s the anything over A zero is good. Most companies want to have generally 25 and above is considered considered good. And anything over about 60 is considered phenomenal. Net Promoter Score.
Laura Caldie 53:03
Yeah. What’s interesting about that is I kind of go back to that my comment I just made before, which is and what do you plan on doing with the information that you get? How do you want to take action against it? And so for me, you know, in that sales and marketing hat that I wear, it’s not so much the number I care about, it’s the why after the number and if I can’t drill into the wire, what’s going on and the numbers pretty useless to me because I can’t take action against it. So that would be you know, my my feeling about it is it’s not that it’s that you shouldn’t do it, but when you do and you start to get data back, do it in a way that allows for the deeper answering the question, what what why is it the number that it is and what can we do about making people happier or recommend us more or be a place that people want to work?
Joel BC 53:56
Yeah, and then a follow up on the employee. Net Promoter. So there’s employee net promoter, which the organization can use to judge the overall organization. However, if we’re stakeholders, that’s more like a customer next letter, and so there, you could as a development team or a development organization, you could send out a net promoter survey to your internal customers and their again, you want to carefully craft your question something along the lines of would you recommend this product to someone else or would you recommend working with with with our team to another another group in the company would be a kind of question you could use as an internal net promoter. So when you really think about it, you’ve got external customer facing net promoter, you’ve got employee net promoter, which is a way to judge how employees feel about the company itself. And then you’ve got internal customer net promoter, which is are your internal customers happy with the work you’re doing?
Laura Caldie 54:55
So Excellent. Any other questions got about a minute to give them to us. While we do that, like I said, Any everything that we’ve talked about today, I know Joel has mentioned some books. We’ve both mentioned some webinars. We’ve got websites, web addresses that we’ve given out so we will definitely put them into the follow up email. I think one that you shared earlier, and we’ve mentioned it a couple of times. Joel is the profit streams book, which is the newest book to be delivered by our founders of a company Luke Holman and Jason Tanner. You learn more about that there. We’ve shared the webinar, pest metrics, webinars a couple of times in here, we’ll make sure that those are in the email as well. Yeah, I can’t think of anything else that we should be talking about Joel. So I just want to say thank you so much for pairing up with me and being the kind of expert with lots of years of bruises and successes and trying to figure out in what way can metrics really help drive effectiveness and happiness and high quality solutions and happy customers and what do we want to avoid to make them tools for, you know, retribution? Or even if it’s usually unintentional, right? How can we make sure that they remain effective so that leaders can kind of create the environment and the solutions they want for their customers?
Joel BC 56:21
Yep, so I would say my closing thought is four metrics. The purpose of metrics is to allow us allow you to ask the question, how can we get better? That is, what you use metrics for is to ask that question and then try and find the answer.
Laura Caldie 56:38
Perfect. Well, thank you very much for sharing the recording and we’ll see you in our next webinar, which will be in the next probably six weeks.
Joel BC 56:46
All right,
Laura Caldie 56:47
all right. Take care. Bye. Bye. Thank you.
