Evaluating AI Opportunities: Applying Range-Based Estimates

A partner recently asked a great question after reading our recent post about range-based thinking so we thought others might find the question and answer that our Chief Innovation Officer, Luke Hohmann, offered in response.

Question: 

I’m currently working with a municipal CFO to evaluate several finance-related AI opportunities, and range-based estimates fit the conversation well.

In the public sector, returns extend beyond revenue to include recovered staff capacity, faster reporting, stronger compliance, avoided losses, and greater public trust.

We’re pairing cost and public-value ranges with bounded 90-day tests, measurable quality thresholds, and explicit criteria to scale, revise, pause, or stop. This gives us a useful foundation for discussions with the CFO leadership team and staff about investment decisions.

One question: Is the expected ROI within the range probability-weighted based on the likelihood of the underlying scenarios?

Response: 

We agree that identifying a broad range of tangible and intangible benefits is the right approach. Both can be be modeled economically (e..g, the value of greater public trust can be financially modeled - the work of Douglas Hubbard and his book ‘How to Measure Anything’ informs our approach when modeling intangible benefits).

Your specific question on probability-weighted modeling is based on the specific function used in modeling your confidence internal. A range-based estimate is based on a ‘90% confidence interval’: We believe the expected outcome will occur between these two values. The most likely value is this’. So, ROI could be a range of $2.4M - $3.1M with the most likely outcome being $2.7M.

Modeling this in Monte Carlo is then based on your expected distribution of these ranges. In most portfolio decisions, a triangular distribution is sufficient. However, you can have other distributions. 

You can have other distributions depending on the nature of the variable. For instance, when modeling public sector risks like "avoided losses" from catastrophic system failures or compliance breaches, traditional bell curves drop the ball. We often have to map these using fat-tailed distributions.

You can also model correlated values via Markov Simulations. And there are even more sophisticated approaches to modeling, where you might want to combine Real Options with investments (your use of 90-day bounded tests appears to be a form of real options).

Your specific question appears to be grounded in a broader question “What level of sophistication in our financial modeling enables us to to create the highest impact decisions?”

I suspect our general approach is aligned on creating the minimum set of financial models that enable the portfolio to make the best possible decisions with the information they have at hand.

Are you using Monte Carlo analysis in your evaluation of potential investments?  That's a good place to start.

If this resonates with you as well, please set up time to talk with us.  We will also be at the SAFe Summit in San Diego from 9/15 - 9/18.  Luke will be giving a talk about AI and ROI and happy to meet up in person.  Keep abreast on what is going on at the Summit by following us on LinkedIn or via our Summit update page.