Valuate the Model, Don’t Just Evaluate It

Data
Written By Christina Ellwood

Episode Summary

A data scientist finishes a model and reports an area under the curve of 0.83. Eric Siegel points out that those numbers are, in his words, entirely arcane to the business, that the person responsible for the operation cares about profit, savings and KPIs, and that the translation between the two is generally not done. The missing step is small: take the same test data already used to evaluate the model, and valuate it as well, expressing its performance in business terms. That pays off twice: it steers development toward value, and it gives the data scientist a way afterwards to say what the model is worth, which is what gets an organization to deploy it. This, he says, is why predictive AI is failing. Not the mathematics, but a no man’s land between business and technology where both sides point at the other and the hose never connects to the faucet.

Key takeaways

  • Predictive AI is most of what AI meant until about three years ago. Fraud detection, credit risk scoring, marketing targeting and predictive maintenance: learning from data to put odds on a per-case outcome, then using those odds to drive the operational decision

  • A data scientist is trained to report the wrong thing, and their tools support it. Area under the curve, precision, recall and accuracy are what the training and the tooling produce, and he calls them entirely arcane to the business stakeholder, who cares about profit, savings and KPIs

  • The fix is one more step on data you already have. Take the same test data used to evaluate the model and valuate it as well, expressing its performance in business metrics rather than technical ones

  • Valuating pays off twice. During development it lets the data scientist navigate the train and test iteration toward actual value, and afterwards it gives them a way to convey the model’s potential value in business terms, which he calls the carrot at the end of the stick that gets an organization to deploy

  • Both sides point at the other. He describes a no man’s land between business and technology in which responsibility is always somebody else’s, so the hose never connects to the faucet

  • Most predictive AI projects stall before deployment. He says the field is failing like crazy and calls it a crisis, while adding that this does not mean the value is unproven: if only 15 percent succeed, that 15 percent of many projects is a lot of success, but overall the track record is dismal

  • He is careful that this is not just his opinion. He has run Machine Learning Week since 2009, has been involved in rounds of industry research, and cites surveys of data scientists and an IBM study of executives, before saying plainly that the models are not deploying

  • If you do not measure business value, you cannot be pursuing business value. He states it as a principle rather than a checklist, and it is why the first of the two problems he names is a problem in the development of the model itself

  • Predictive AI can be a guardrail on a generative one. A system that performs at 95 percent cannot simply be unleashed, but a predictive layer can learn which cases are most likely to go wrong

  • Route the risky slice to a human and keep most of the promise. His worked example sends the top 15 percent of riskiest cases for human review, which is more expensive per case, and still realizes 85 percent of the autonomy, which he sets against the zero percent you get from a system that never deploys

  • The executive has to go one level in, not become technical. His guidance is that a business leader assess the model’s potential value in business terms before accepting it, because the alternative is having the model handed to you and having no basis to act on it. His framing is to dive in a little, which he says is not super technical

About Eric Siegel

Eric Siegel is CEO of Gooder AI, a product built to close the gap between what a predictive model does technically and what it is worth to a business, which he describes as a de facto business console for predictive AI projects. He has run the Machine Learning Week conference since 2009 and has been involved in rounds of industry research, including surveys of data scientists. He is the author of two books written for both sides but, as he puts it, first and foremost for the business reader: The AI Playbook, which sets out the six-step practice he calls BizML, and Predictive Analytics, his first. On air he is direct that predictive AI is failing to deploy at scale, that this is a crisis rather than a quibble, and that the cause is a biz-tech gap nobody is incentivized to bridge.

 

In this episode

00:00 Welcome
00:21 Guest introduction, Gooder AI and the two books
00:47 What predictive analytics is, for an executive who has not met it
01:03 Predictive AI was most of what AI meant until three years ago
01:20 Fraud detection, credit risk, marketing targeting, predictive maintenance
01:39 Playing the odds better, and how it relates to generative AI
02:12 Odds on the outcome, driving the per-case decision
02:32 Not generally used together yet, and why they need each other
02:49 Why a language model is harder to use well for concrete value
03:34 No magic crystal ball, only probabilities
03:48 Decades of positive track record, and how little is realized
04:11 Less magic-seeming, and the semi-technical understanding it needs
04:30 How each resolves the other’s weaknesses
05:28 A specialized chatbot built on Claude
06:52 Autonomy as the ideal, and why it is out of reach
07:13 A system that is right 95 percent of the time
07:31 Using predictive AI to find the cases most likely to go wrong
08:12 Sending the riskiest 15 percent to a person
08:30 85 percent of the promise, against zero for a system that never ships
08:44 The executive considering an agentic solution
09:19 Justifying the build with an expected outcome
11:06 Deployment means operations change
11:30 Area under the curve, precision, recall, accuracy
12:00 What the business side cares about instead
12:51 Where the financial support usually already is
13:04 Where the project stalls
13:24 Post-POC, pre-deployment
13:40 Predictive AI as a field is failing like crazy
14:04 The biz-tech gap, still not widely bridged
14:10 Adil Ajmal at Fandom, and the same combination in use
16:06 No man’s land between biz and tech
16:32 What pure predictive performance does and does not tell you
16:58 Performance in business terms instead
17:26 If you do not measure business value you cannot pursue it
17:50 Operations do not improve unless they change
18:26 Why he wrote them and who they are for
18:32 For both sides, but business first
19:32 BizML, and six steps across six chapters
20:13 Why deployment is the whole point
20:35 Running Machine Learning Week since 2009
21:06 They just do not deploy, and the failures get swept under the rug
21:24 Viable models that could be delivering value
21:50 More excited about the rocket science than the launch
21:55 Adoption lagging enthusiasm, and 3 percent at the first summit
22:14 The BARC research, and 19 percent today
22:32 Friction on both sides, and what still needs solving
23:33 His writing and his course at machinelearning.courses
23:53 James Taylor, from the business vantage
24:34 What Gooder AI was built to do
24:37 What executives are delegating without realizing
25:01 Diving in one level, without becoming technical
25:35 The leadership skill that matters most
26:42 Making the haystack smaller
27:03 The arithmetic that turns performance into a financial win
27:50 Hands dirty, or feet cold
28:17 Assessing potential value before you use it
28:36 What the business side has to be given
29:06 Wrap-up

In Guest’s words

“Let’s take this same test data that we’re using to evaluate the model, but not just evaluate the model, let’s valuate the model.”

— Eric Siegel   (16:32)

“There’s this sort of no man’s land between biz and tech, and both sides point to the other to say it’s their responsibility. So the hose isn’t connecting to the faucet.”

— Eric Siegel   (16:06)

“The translation from technical performance, pure predictive performance, to potential business value is generally not done.”

— Eric Siegel   (12:00)

“Predictive AI as a field is failing like crazy. It’s really a crisis.”

— Eric Siegel   (13:40)

“Because if you don’t measure business value, you can’t be pursuing business value.”

— Eric Siegel   (17:26)

“It’s like being more excited about the rocket science than the launch of a rocket.”

— Eric Siegel   (21:50)

“If you don’t get your hands dirty, then your feet will get cold.”

— Eric Siegel   (27:50)

“Operations don’t improve unless they change, and in this case we’re talking about changing them with probabilities.”

— Eric Siegel   (17:50)

 

Resources

Eric Siegel, Gooder AI and the books

Ideas and frameworks discussed

  • Predictive AI: His term for what most people called AI until about three years ago. Learning from data to put odds on a per-case outcome, then using those odds to drive an operational decision. Fraud detection, credit risk scoring, marketing targeting and predictive maintenance are his examples

  • BizML: The six-step business practice for running machine learning projects that The AI Playbook is built around, described across six chapters and aimed at both sides but at the business reader first

  • Evaluate versus valuate: His wordplay, and the practical core of the episode. Evaluating a model asks whether it beats guessing; valuating it asks what it is worth, on the same test data, expressed in business metrics. He puts it as one more step right after evaluation

  • The biz-tech gap: His diagnosis of why projects stall. Technical metrics on one side, business metrics on the other, a no man’s land in between where both sides point at the other, and no incentive system for anyone to bridge it

  • Predictive AI as a guardrail: Layering a predictive model over a generative or agentic system to score which cases are most likely to produce a bad outcome, so the riskiest slice can be routed to a person while the rest runs

  • The 15 and 85 arithmetic: His worked illustration. Send the riskiest 15 percent for human review, keep 85 percent of the promise of autonomy, and compare that to the zero percent delivered by a system nobody is willing to deploy

  • Making the haystack smaller: How he describes what a model does for fraud detection. Not finding the needle, narrowing where you look, and then doing the arithmetic that turns that into a financial win

Named on air

  • Claude, Anthropic: The model behind the specialized chatbot Gooder AI built into the product, which answers questions about the predictive AI project, the product and what the levers and sliders do. The what-if scenarios are the interface itself rather than the chatbot

  • IBM: Named for a study of executives which he says found these projects usually come out even on ROI. The clause is garbled in the transcript, so nothing beyond that is claimed here

  • James Taylor: Named as someone else working on the same gap, coming at it from the business and operationalization side

  • BARC: The research Christina cites for a 19 percent figure on generative AI deployment, which she says three or four other studies support

  • AI Business Value Is Not an Oxymoron: The title of the talk he says he will give at AI Realized Summit, which took place on 5 November 2025

Related AI Realized episodes and events

 

Frequently Asked Questions

  • Predictive AI puts odds on a specific outcome for a specific case, and generative AI produces content. Eric Siegel of Gooder AI points out that predictive AI is most of what the word AI meant until about three years ago, covering fraud detection, credit risk scoring, marketing targeting and predictive maintenance, and that both are built on machine learning. The practical difference he draws is that generative AI is by definition much easier to use, while predictive AI is used on cases that are robust against error, is less user-friendly, and is less magic-seeming because it deals in probabilities rather than answers. He is emphatic that the two need each other and address one another’s weaknesses.

  • Machine learning projects fail to reach production because a model that performs well technically has still not been expressed as a business case, and that translation is generally not done. Eric Siegel of Gooder AI describes a data scientist trained, and tooled, to report area under the curve, precision, recall and accuracy, and a business stakeholder who needs profit, savings and KPIs. His image for the result is a no man’s land where both sides point at the other, so the hose never connects to the faucet. He calls the field a crisis on this point and grounds it in more than opinion, citing the conference he has run since 2009 and industry surveys of data scientists. His framing is that operations do not improve unless they change, so a project that never changes an operation has delivered nothing.

  • You measure the business value of a machine learning model on the test data you already have. Eric Siegel of Gooder AI describes taking the same test set used to evaluate a model and valuating it as well, which means expressing its performance in profit, savings or whatever KPI the organization manages rather than in the technical metrics alone. He is blunt that if you do not measure business value you cannot be pursuing business value. His worked example is fraud detection: a model narrows which transactions are worth investigating, and the question is the exact arithmetic that turns that narrowing into a financial win. Technical scores only tell you that the model beats guessing and the data is sound.

  • Yes: predictive AI lowers the risk of a generative or agentic system by scoring which cases are most likely to go wrong, so those can be handled differently. Eric Siegel of Gooder AI starts from the problem that a system performing at 95 percent is impressive and still cannot be unleashed, then adds a predictive layer trained on which cases carry the risk. His illustration routes the riskiest 15 percent to a human, which costs more per case, and keeps roughly 85 percent of the promise of autonomy. He sets that against the zero percent delivered by a system nobody is willing to deploy at all.

  • Yes, though only enough to judge a model in business terms rather than technical ones. Eric Siegel of Gooder AI is explicit that this is not about becoming technical, and calls what is needed a semi-technical and accessible understanding of what it means to improve a large-scale operation. His argument is that an executive who does not assess a model’s potential business value before accepting it has delegated something crucial without meaning to, and has no basis on which to act when the model arrives. His phrase for it is that if you do not get your hands dirty, your feet will get cold.

  • BizML is a six-step business practice for running machine learning projects, set out across six chapters of The AI Playbook. Eric Siegel of Gooder AI wrote it for the business reader first, on the reasoning that anything written only for data scientists will not bridge the gap it is meant to bridge. The book’s purpose, as he describes it, is to bring a business reader up to the semi-technical understanding needed to participate in those six steps and collaborate properly with the data scientists, including on what the model is worth in business terms.

  • The business case belongs in two places: before anyone commits to deploying, and during development, where measuring value is what lets a data scientist steer the train and test iteration toward it. Christina Ellwood puts that to Eric Siegel of Gooder AI as post-POC and pre-deployment, and he accepts it with a qualification: POC means the number-crunching part to some people and at least a pilot deployment to others. By that stage, he says, there is usually already some financial support and the analysis is sound, and what stalls the project is that nothing has bridged it. Additional resources may be needed for the technical integration.

 

  • [00:00] Christina Ellwood: Welcome to AI Realized, the podcast for enterprise executives leading AI deployments. From tackling security, data, and operational challenges to navigating organizational transformation, AI deployment offers a unique opportunity to redesign our organizations from the inside out. I'm Christina Ellwood, your host for today's episode. We're talking today with Eric Siegel, the founder and CEO of Gooder AI, and the author of "The AI Playbook" and "Predictive Analytics: Predicting Who Will Click, Buy, Die, or Lie." Eric, welcome to the show today.

    [00:35] Eric Siegel: Thanks, Christina. Thanks for having me.

    [00:37] Christina Ellwood: It's a pleasure to have this conversation. I'm a big fan of the other types of AI besides generative AI, predictive machine learning, other types of AI. And today we're gonna zero in on predictive analytics. And so for our executives who are not familiar with predictive analytics and how it relates to the work that they may be doing in generative, can you please help us understand the connection and understand the basis of predictive analytics?

    [01:02] Eric Siegel: Yeah, sure. Predictive analytics or predictive AI, same thing, is everything-- was most everything that was referred to as AI up until three years ago with the advent of generative AI. So it's older but not old school. It's learning from data to predict in order to target, uh, any and all of your existing large-scale operations. So for fraud detection, credit risk scoring, targeting marketing, predictive maintenance, all of these use cases, any of your, all of your largest enterprise operations consist of millions of micro-decisions which are best informed by the holy grail for improving them is prediction. That is a probability. So learning from data to put the odds, so we can play those odds games better. Now, in, like generative AI, it's built on machine learning. In fact, you could, to clarify, these are really two categories of use cases of machine learning. But instead of learning from data to predict what the next word would be for a large language model to be writing text, it'd be on the enterprise level, an enterprise unit or element, the customer, the healthcare patient, the item f- rolling off the assembly line, the satellite that might run out of battery, any and all of that level of organizational, on that level of granularity. Put the odds and the outcome or behavior, that's what's gonna inform the per case operational decision.

    [02:18] Christina Ellwood: So it sounds like the answer really to the, some of the problems that plague generative AI, which is hallucinations and so forth, so it sounds like they're a nice complement to each other. How are they typically used together?

    [02:32] Eric Siegel: Um, they're generally not used together, but I very much see that as on the horizon and absolutely necessary. They really need each other. They address one another's weaknesses. And the weaknesses are that generative AI is by definition much easier to use. You can use it with English or other human languages. That's why they call it a language model. It's built on that. But arguably harder to use well in terms of reliably delivering or capturing, realizing concrete enterprise value. So making sure that the potentially aggressive or optimistic use case is feasible, that it's actually viable to be deployed despite the fact that the system's gonna be wrong, right? That it's, it's gonna hallucinate and there's other things it does wrong, right? It could be jailbroken, it could be offensive. It's like comparing cats and dogs, right? Cats are much easier to manage, but much harder to train, right? Predictive AI, though, is used on use cases that are robust against error. You don't expect it to be perfect. There is no such thing as a magic crystal ball. Rather, we have the next best thing, which is putting probabilities and odds of those outcomes or behavior. So the problem with predictive AI is not that there's not a concrete use case. There is a concrete use case. There's decades of experience of positive track record of enterprise outcomes. But it's still only a small potential that's being realized. Most of those projects actually fail to deploy because it's harder to use. It's, it deals with probabilities. Probability's just a number between zero and 100, and again, we don't have absolute confidence, so we have to use that. We have to use probabilities. But, you know, it's a little less magic-seeming. It, right, is a little bit less user-friendly. It requires a bit of very accessible semi-technical understanding to make use of it. So I'd say if the weakness of generative AI is that it makes these mistakes, the weakness of predictive is that it's harder to use. So how do they resolve one another's weaknesses? If you're running a predictive AI project, there's lots of ways in which generative AI can make it easier and accessible to use predictive AI. So for example, in our product at Gooder AI, which establishes the business value of a predictive AI project so that you can maximize that value and communicate it credibly to stakeholders and get authorization and green light for it to be deployed, which is usually actually where the project stalls. We have an interface, you can interact with it and try what if scenarios for deployment. What if I use this model to target marketing in this way, or this fraud detection model to target my auditing team, or blocking credit card transactions in this way? Whatever it is It's not the rocket science part, it just has to do with steering the rocket and establishing its potential behavior, its performance in terms of the KPIs that matter. But it's a relatively new endeavor, so it could use a little kinda hand-holding and guidance. So we actually put a specialized chatbot, it's built on An- Anthropic's Claude model, into the interface, and it allows you to ask any questions about the predictive AI project in general, about the product, Gooder AI product, Gooder AI, about the levers and sliders and th- things you can move on the screen, what they all mean, why does the cur- profit curve go up and down? We've got a demo of that right on the top of our webpage at gooder.ai. So that's great, right? All of a sudden you have this sort of well-caffeinated, well-informed expert data sci- virtual data scientist available to ask any and all questions ad infinitum. There's no such thing as a stupid question. You're certainly not gonna worry about that knowing that you're not bothering a human expert. So there's all sorts of upside and advantages to bridging the gap to what's really just a semi-technical understanding that's necessary in order to drive these predictive AI projects by way of a chatbot. So that's one way in which generative helps predictive. And how about the other way around? If generative has this reliability issue, it has that potential to hallucinate and create other problems, that means that you might have a generative AI system, say it does customer service to end consumers, and you're supposed to be able to make changes to a reservation or ask questions about the product or make an order or change to your order, whatever the customer service agent is supposed to do. The ideal is autonomy, right? We want the machine to be able to operate reliably enough that it could be unleashed, but the problem is that for most of those types of projects, you very quickly conceive of a scope of intentions of how the chatbot's supposed to be used. That's too audacious. It's too optimistic and ambitious. It's gonna have errors. It may be correct, perform well, let's say, 95% of the time, which is incredibly impressive technolo- technology-wise, something we never would've imagined possible just five years ago. And yet still potentially completely not viable, won't get deployed because that's too high of an error rate. But if you use predictive AI as another layer on top of that and ha- use it to learn about the cases that are most potential to have that negative outcome, to have that risk of poor behavior that would be unacceptable in a consumer-facing applic- application. And by doing that, you can flag those most risky cases and potentially divert them to a human in the loop. Now, the human in the loop obviously is much more expensive than an automatic system, but there's a time and place they're needed. This system could be learning to target that accordingly, to be triaging and only using that expensive human in the loop for those more expensive cases, and you-- the numbers may pan out really well. Where, for example, if you sent, let's say, the f- top 15% of most risky cases to have a human look at it and potentially intervene, which is more costly, maybe the net would be only 1% of cases would be problematic, and for, for some projects that may be viable, and the whole project can now potentially deploy. What does that mean? That means we've realized 85% of the otherwise audacious promise of autonomy Which is a lot better than 0% if you can't deploy the thing in the first place.

    [08:44] Christina Ellwood: So that's a great example. Thank you for making it so vivid. So if I'm a business executive of a f- functional group in the enterprise, and I'm considering an agentic solution for a customer use case, could I go to the Gooder AI website and ask it to help me understand what machine learning I would u- or what predictive model I would use in my agent for that 15% that needs to be shunt- shunted off to, to the human? So what do I need to use, and then what is the likelihood that it would provide me a certain outcome so that then I can bring that to justify, "I wanna build this agent. I wanna have this predictive analytics built into it to minimize the number of errors or, or problems that it causes. And I wanna f- and I believe this will be the outcome based on the modeling that I have done, and I want funding to be able to go do that project." Is that how I would use Gooder

    [09:49] Eric Siegel: AI? Almost. It's a great question you're asking, so let me try to clarify. So our product, Gooder, is to help support an existing predictive AI project. And a- any and all type of predictive AI project, including the one I just mentioned where you're layering it on top of Gooder, um, excuse me, where you're layering it on top of a generative AI system, that's just another predictive project like any other. And the nature of these projects, of any of these all, any and all of these use cases for predictive, is that you're, you're, as I said, you're putting these odds on it. You're, in other words, you're able to rank and triage the cases according to most likely down to least likely, highest risk down to least likely risk, or highest opportunity for targeting marketing. Either way, you're basically able to order and draw a line. What I'm describing is the nature of most any predictive AI project. If you have an existing project, any of the above types of projects Gooder will help you actually establish, maximize the business value so that you can actually then sell the project. That is to say, pitch it g- to get it green lit so that it actually deploys, gets integrated into operations, right? If you don't deploy, you don't realize the value. The number crunchings only creates potential value, but you don't capture or realize that value until you act. The operations don't improve if you don't change them. They need to be changed by way of the predictions. That's deployment. So any and all predictive AI project needs a data scientist. Data scientists will definitely not go away. You're not gonna be fully automating that, those types of projects. But a data scientist has been trained to communicate in technical terms, "Hey, look, this predictive model predicts a lot better than guessing. It has an area under the curve of .83. It has this and that and the other pred- precision, recall, even accuracy is just a technical metric." They just are trained in general, and their tools support them in reporting those types of metrics that really are entirely arcane to the business. They-- So there's a gap that greatly needs to be bridged between that and the business stakeholders, the person in charge of the operation meant to be improved by a predictive model, who, of course, is representing the business and cares about what? They care about business metrics like profit and savings, right? Monetary performance or any and all other kind of KPIs. The translation from technical performance, pure predictive performance, to potential business value is generally not done. That gap is generally not bridged, and that's what Gooder AI does.

    [12:23] Christina Ellwood: Okay, so going back to my scenario, uh, if I'm a business executive with a potential use case, I would work with my data scientist to find, to get whatever model they're recommending that we use for our use case. Then I could bring that to Gooder AI, or my data scientist could bring that to Gooder AI, to frame the br- business value for communicating it to the, my peers and my board to get the financial support. Is that correct?

    [12:51] Eric Siegel: Yeah, that's exactly correct. Although- Okay ... actually, at that stage of the project, there's already usually s- at least some financial support. Typically, the pr- what happens is the projects get greenlit, and the resources are put towards the project, and then the number crunching takes place. It's sound, and then the project stalls 'cause it hasn't been bridged. There's, there may be additional financial re- resources needed to actually do the technical integration. That is to say, operationalization, deployment, whatever you wanna call it. And then in, in that case, it would do exactly what you described, right? It would-

    [13:24] Christina Ellwood: Oh, yeah ...

    [13:24] Eric Siegel: a-

    [13:24] Christina Ellwood: actually serve- So, so really, it's post-POC, pre-deployment. It fits in that space- Yeah ... between I've piloted- Well, yeah ... but I haven't deployed. Okay.

    [13:32] Eric Siegel: Yeah. If you, if by POC you mean just the number crunching part. Sometimes people won't say, will say POC actually means at least some kind of pilot deployment. But the fact is, most projects actually stall. That is to say predictive AI as a field is failing like crazy. It's really a crisis. That doesn't mean it's a, doesn't have proven value. If only 15% succeed, that 15% of many projects is a lot of success, but overall, the track record is dismal. That field, predictive AI, is still not professionalized in this way. There is a biz-tech gap Uh, that is still not widely bridged

    [14:10] Christina Ellwood: Okay, so this helps to solve, uh, close that gap. What we were speaking recently with Adil Ajmal from Fandom, and he was talking about this very type of use case where they have a- an ad placement use case, they have customer support use cases and so forth, where they're using this very combination. And it does put the, uh, data scientists front and center in their, in their approach to how they develop the products and how they evaluate them before deployment, and then how they monitor them and adjust them when they're in deployment. So what you're describing, I think he vividly made clear the value to the business of having this combination in their particular case. As we prepare to get ready for AI Realized on November the 5th, the summit on November the 5th in San Francisco, where you will be one of our speakers, as will Adil, we should put the two of you together to talk about how you can combine your, your guidance for f- people into a clear action plan that people can take away with them and go back and use immediately. Because I think our listeners are gonna readily understand the value of both of them. I think they're gonna readily understand the high level of how you bring the data scientists together with the use case to be able to close this gap. I don't know that they will, will feel like they can fill in all the blanks on what that workflow looks like. So it might be worthwhile doing that, unless you feel like you can articulate that today. I think that might be a good goal for us to set for sharing that with, at AI Realized Summit on November the 5th.

    [15:47] Eric Siegel: Sure, absolutely. As far as the workflow, the workflow right now, to this date After decades of mach- enterprise machine learning, predictive AI, whatever you want to call it, has been applied for targeting for credit score, for targeting marketing, and fraud detection, such an endless range of applications. To this date, the typical workflow is so close to complete and yet so far. There is, there, there's this sort of no man's land between biz and tech, and both sides point to the other to say it's their responsibility. So the hose isn't connecting to the faucet. But it really, what it comes down to at the core, during the development of the model, before its deployment, they're just looking at these technical metrics. Pure predictive performance, it tells you that the m- it's very helpful. It tells you the model's better than guessing. It tells you that the data is sound, the analytics are sound, that the model is potentially viable, but tell you little to nothing about the potential business value. So it's basically one more step right after that, "Hey look, let's take this same test data that we're using to evaluate the model, but not just evaluate the model, let's valuate the model." That is to say, let's look at its performance in business terms of business metrics or KPIs, like profit savings or whatever applies for the organization. So it's just that gap which helps, which solves two problems. One is in the development of the model, you're ensuring that what's always a semi-automatic iteration between devel- train the model, test it, train, test, train, test, that the data scientist conducting, that you're navigating the development towards actual value. Because if you don't measure business value, you can't be pursuing business value. The second business problem is bridging that gap, and so that the data scientist now has a way to convey in meaningful business terms the potential value of the model to motivate, serve as the carrot at the end of the stick so that the organization gets itself together and realizes, "Okay, we actually need to deploy this, which means we need to manage change." Operations don't improve unless they change, and in this case we're talking about changing them with probabilities.

    [17:56] Christina Ellwood: Gotcha. Now I wanna shift our gears a little bit here and talk about the books that you've written. I've really been enjoying Predictive Analytics. Love the title by the way, I think it's just delightful. But you are a very good writer. You make these, these subjects easy and entertaining to understand. It actually, I have to say, is a brisk read. And for the topic, I think that's a pretty, pretty big accomplishment. That's a difficult thing to make a brisk read, but you do that very well. You wrote the Predictive Analytics first, and then you wrote The AI Playbook. Tell me a little bit about why you wrote them, who they're for, and who you recommend read them today.

    [18:32] Eric Siegel: They're for both sides, both tech and biz, but first and foremost, they're for biz because you c- anything written just for data scientists is generally not gonna be accessible, and I really appreciate your positive feedback. I did bend over backwards to, to offer the content here in a way that's meant to be relevant, relatively entertaining, and totally accessible. Which, uh, thematically, that aligns with what we're doing at Gooder AI. We're trying to bridge that gap. And the theme of my second book, "The AI Playbook," is that we can't get these things deployed without bridging that gap, without fostering a very particular deep collaboration between tech and biz, which means that the decision-makers, the stakeholders, the people who are usually not data scientists and are not doing the hands-on number crunching, do need to ramp up on a very accessible semi-technical understanding so that they can collaborate end-to-end across the lifestyle, excuse me, the life cycle of the project from its inception to its hopefully successful deployment. So the point of the book is to offer, uh, the titular playbook paradigm framework that I call BizML, business practice for running machine learning projects, a six-step practice described across six chapters. But first and foremost, that book, "The AI Playbook," is meant to ramp up that business reader on that semi-technical accessible understanding so that they can then participate in the execution of those six steps and actually collaborate deeply with the data scientists, get everyone on the same page, including the potential performance in business terms of the model, which is what we're handling technically with Gooder AI, and get those darn models actually deployed.

    [20:13] Christina Ellwood: Great. Deploying AI to production is what AI Realized as a community is all about. Yeah. So it's a perfect fit, and I believe that this complementary nature is, is demonstrated. Actually, I'm surprised that it, that you find so much impedance in the market because I see it everywhere when I'm talking to enterprises. They've been using machine learning long before they were using generative AI.

    [20:35] Eric Siegel: Yeah. So- Can I actually back that up a bit? I want to be clear, this is not just anecdo- anecdotal is overwhelming, I might say, and I've been running the mach- the machine learning project conference called Machine Learning Week since 2009. But I was also involved with a couple rounds of industry research, and I'm not the only one conducting that kind of industry research. We do surveys of data scientists. IBM did one of the executives showing that the, those types of projects usually come out even in terms of their ROI rather than actually on average. And we bel- and all evidence points to the fact that just this comes down to the fact they just don't deploy. They stall, stall. The industry has gotten good at sweeping those failures under the rug, but it's not sustainable. There's no reason for it to, that track record not to improve. But this is real. This isn't just my opinion. The models are not deploying, and many viable models that could be delivering value, the project needs to reor- re- reorient, reframe, present itself as a business project meant to improve operations that happens to necessarily use machine learning Rather than being a machine learning project where people are fetishizing and, and focusing on the core number crunching itself rather than the use thereof. It's like being more excited about the rocket science than the launch of a rocket.

    [21:55] Christina Ellwood: Yeah. You see this in, in, in generative AI and in other areas of technology evolution where the adoption is not instantaneous. It takes time, and it lags behind the enthusiasm. And in the case of generative AI, in, in October of last year when we held the first AI Realized Summit, there was 3% deployment in enterprises. The most recent research that we have presented f- from BARC, which w- was supported by three or four other studies as well, is more like 19% today. So we still have plenty of places where it's not getting to prime time on the generative side, too. So we don't wanna repeat the mistakes that we've made in the past. If we've had friction associated with deploying machine learning projects, and we're having friction in supporting generative AI projects deploying, we need to solve the problem of deployment. That's the problem to solve, not the-- We have plenty of technology. We need to get it deployed. And there's, besides the technology, or excuse me, so beside the issues that you've raised about the technology valuing the technology deployment, there is also the organizational issues. There are s- security and governance issues. So there are multiple issues at play here. We pull on this particular thread, we still have the other ones to solve. So I love that you have these books as resources for people to go to, and I will make sure that we include links to them in the show notes. Do you have other resources that you recommend our listeners use to either come up to speed on predictive analytics on this gap between the business value and the, um, the predictive value and the business value, or links that would help them to take their first steps?

    [23:33] Eric Siegel: Um, that's a great question. I have to say that I've taken that on in terms of writing and the courses that I teach, that bridging that gap more than most people tend to be on one side or the other and stick to that side, which is fundamental to the problem. So in addition to my books, I do have a course which is available at machinelearning.courses. That encompasses the content of both of the books plus a bit more. And but there are other books. There are books by, for example, James Taylor, and he comes, he's a consultant, he comes really more from the business vantage and operationalization, business value side orientation, and yet he does a lot more than most, I would say, to bridge that gap. So there's, I would say there's a general understanding that the gap is unbridged and needs to be. But somehow there's a lack of incentive system to be bridging the gap. Everyone's getting their salaries.

    [24:28] Christina Ellwood: So what's your guidance for executives who are early in their adoption journey?

    [24:34] Eric Siegel: We build our product, Gooder AI, to help bridge the gap. So we are always pitching to executives, "Hey, look at what you're expecting your data scientists to do. You know what? You're delegating a little bit, a little crucial bit too much to be on their side of the fence, and you need to, you or, or your VP or somebody needs to take a bit of a concrete look at the potential business value in concrete data-driven estimations of business value." Just dive in there a little bit. It's not super technical, but it has to do with what it means to improve a large scale operation. And to facilitate that, your data scientists can very easily take their model, put it into Gooder AI, look at its performance through Gooder's value-oriented lens, and then they can share that with you. You can move the levers on the screen. They are all business levers. So that's the purpose. You can think of this as de facto business console for predictive AI projects.

    [25:35] Christina Ellwood: Oh, that's a great way to describe it. I like that. So in your work in working with executives to understand and benefit from AI, what's the leadership skill you find most useful?

    [25:55] Eric Siegel: Oh, great question. I feel like my main thing, and I think that's probably come across by now, is a clarification thing of, look, you need to look a bit into detail, right? The kind of detail I'm describing, it would be something like this. You've got a fraud investigation team. They could be... Right now, they're investigating some random 7% of the transactions, 'cause that's how big the team is. They could be investigating a more finely targeted group of transactions that might be fraudulent by way of flagging those transactions with a model. So the model could flag the top 3% most likely to be fraud, or it could flag the top 7%, right, which is a bigger group, which means it's not gonna be as rich in fraud, right? Fraud detection's a needle in a haystack. We're talking about making the haystack smaller. But what's the exact arithmetic that translates that type of performance? It can identify, you know, 3% of the transactions that are seven times more likely than average to be fraud, or the 6% of transactions that are three times more likely than average to be fraud. What- whatever those particular numbers are, how do you do the right math to translate that into the financial win of improved fraud detection and management, right? And how do we take that particular arithmetic for business value and put it into a interactive visuals you can try different potential deployment scenarios, right? So a- again, this is not the rocket science part. It's the part that's missing, which is steering the rocket so that you can be a- assessing and directing its use and performance to maximize business value. It's-- So my, sort of my speech here is, look, the business side, either the executives or one or two level below them, need to actually get their hands a bit dirty. If you don't get your hands dirty, then your feet will get cold, 'cause that's the scenario. The model is passed to you from the data scientist, and then you balk at its deployment because you... How can you get it without that visibility into potential business value? How can you change massive, ongoing, critical large-scale operations according to numbers if there's been no... You can't use something unless you've assessed its potential value from the business standpoint, not just the sort of arcane technical performance beforehand, so that there can be this credibility, this trust, this understanding of what value you expect to gain with deployment. So It's just a clarification. There's a certain amount of semi-technical insight and accessibility that needs to be made available to the business side, and that the business stakeholders need to actually immerse themselves in.

    [28:47] Christina Ellwood: Great. Any last thoughts?

    [28:50] Eric Siegel: No, I'm really looking forward to the summit. I'll read off the title of my talk, "AI Business Value is Not an Oxymoron." I think that's what we've been talking about. It's certainly a theme that applies for both predictive and generative in very similar, although also differentiated ways.

    [29:06] Christina Ellwood: I love it, and we're really looking forward to having you, and thank you so much for your time today, Eric.

    [29:11] Eric Siegel: Yeah, my pleasure, Christina. Thank you.

    [29:12] Christina Ellwood: Eric Siegel, co-founder and CEO of Gooder AI. Bye-bye. Bye.

    [29:19] Eric Siegel: Thank you.

 
Listen to other episodes
 
 
 
Join AI Realized
 
 
 
Subscribe to the newsletter
 
 
 
Christina Ellwood
Previous

Zero to Campaign With Everyday AI, in Four Steps
Next

More Podcast Episodes