Judge a Model on Cost and Latency, Not Just Accuracy
Episode Summary
Ivan Lee, founder and chief executive of Datasaur, spends this episode on what an executive should measure before putting a model into production. His starting point is that this is the first generation of software with real unit costs. The old assumption of a one-time build followed by free scaling is gone, because GPUs and compute make every query cost something, and methods like chain of thought reasoning multiply that cost again. So the first test is whether the unit cost survives the business case at all. After it come latency, where an answer that takes many seconds is a long way from what people expect of a search box, and quality, with compliance and the question of what data may be sent to a third party sitting behind all three. For getting started he wants crawl, walk, run, with success criteria agreed before anyone builds. His closing point is that no single model is best for a whole company.
Key takeaways
His product analogy for keeping a live model honest comes from software rather than from data science. Prompt unit testing borrows the practice that stops something like the CrowdStrike issue reaching production: a set of prompts and expected answers, run daily or weekly, that flags the moment the model diverges from what you expected
His central claim is that the economics of software have changed. This is the first time in our generation, he says, that we have seen software with actual unit costs, against the old assumption of a one-time build cost and then free scaling forever
His reason is physical rather than commercial. The expensive GPUs and the sheer amount of compute give every query a real cost, and the popular research methods make it worse: chain of thought reasoning and calling several models to get one answer each multiply that core cost again
He turns that into a first test an executive can apply before anything else. In early 2025 he was seeing pilots that cost a dollar or five dollars a query, and his question is whether you would pay a dollar fifty for a single automated customer support answer
The second dimension is one he says people are only starting to think through. There is a lag in getting answers, and in early 2025 he says some replies were taking twenty seconds. The wait is masked by the text appearing as it is typed, but it is still a long way from a search box that returns results in a fraction of a second
He names the three criteria in one sentence, and then names what sits behind them. Cost, latency and then finally the quality itself are what people look at for ROI, and underneath are compliance and regulatory questions about which models you may deploy at all and what data you may send to a third party
His evidence that the industry had turned toward price is a product launch. When OpenAI announced GPT-4o Mini it was the first announcement not built on a bigger model or new use cases but purely on being cheaper than its own predecessor, with open source models like Llama applying downward pressure from below and Google Gemini advertising its efficiency against competitors
On running models privately he expects consumer hardware to move the conversation. As Apple put Apple Intelligence on the iPhone, the mainstream assumption that a query has to travel to a large cloud model starts to break
His technical point underneath that is about size rather than location. An array of smaller language models can run on the device and do a serviceable job, so the question stops being about the largest models on a cloud and becomes what fits in your pocket
The consequence he is most interested in is regulatory rather than technical. Industries that were never allowed to send data to a model provider, healthcare among them, change position entirely once the model can be self-hosted on an air-gapped machine
His advice for an executive early in the design is the oldest one in the book, and he says so. Crawl, walk, run, which is what he calls the traditional advice, and what it means in practice is starting with a pilot that demonstrates the technology works and can solve the problem
His fix for the failure he has watched for over a decade is a conversation held before anyone builds. Agree very clear success criteria ahead of time, the response times and the costs that count as success, because otherwise a solution reaches high accuracy with good recall and precision and an executive tries one query that fails and calls for the whole thing to be redone
His closing point is aimed squarely at the search for a winner. He wants it understood, and says he wants to ingrain it with the audience, that there is no single best model
His worked example splits one company into different answers. Marketing and HR at a business like Starbucks might find an off-the-shelf solution works fine, while the divisions holding the secret sauce decide they need to build in-house for strategic or proprietary reasons
The conclusion is a change in how organizations will describe themselves. It stops being like today, where a company is an AWS house or a GCP house, and he thinks it moves to a multi-model landscape where you can be much more agnostic as to the proper solution
About Ivan Lee
Ivan Lee is the founder and chief executive of Datasaur, which he describes on this episode as a five-year-old startup founded to democratize access to natural language processing. Its data labeling platform has been used by Netflix, Spotify, Qualtrics and the FBI to annotate the data behind their models, and in the two years before this February 2025 conversation the company added LLM Labs, for comparing and selecting models and gathering the data to fine-tune private ones. He says he has worked in AI for over a decade. His argument here is that a model in production is judged on three things rather than one: the unit cost of a query, the latency of the answer and the quality, with compliance and the limits on what data may leave the organization behind all three. He is also firm that no single model is best for a whole company.
In this episode
| 00:42 | Welcome, and who is Ivan Lee |
| 01:12 | What Datasaur is, and democratizing access to NLP |
| 01:29 | The data labeling platform, and who used it |
| 02:20 | The question: using the product in production |
| 03:10 | Prompt unit testing, borrowed from software |
| 03:40 | The question: what goes into the ROI |
| 04:23 | The first generation of software with real unit costs |
| 04:47 | GPUs, compute, and what chain of thought multiplies |
| 05:12 | Pilots at a dollar a query, and the dollar fifty test |
| 05:40 | Latency, and answers that take twenty seconds |
| 05:58 | Cost, latency, and then the quality itself |
| 06:16 | Those are the three criteria, and what sits behind them |
| 06:38 | The question: what drives cost down |
| 06:47 | GPT-4o Mini, announced on price rather than size |
| 07:47 | The question: LLMs behind the firewall |
| 07:58 | Apple Intelligence, and the mainstream conversation |
| 08:19 | Small language models on device |
| 08:47 | Air-gapped self-hosting, and what it opens up |
| 09:09 | The question: enough for smaller companies |
| 10:13 | Crawl, walk, run, and what a pilot has to prove |
| 11:24 | Agreeing the success criteria before anyone builds |
| 12:22 | The question: what listeners should take away |
| 12:42 | There is no single best model |
| 13:02 | Starbucks, and why departments diverge |
| 13:28 | Not an AWS house or a GCP house |
| 13:54 | A multi-model landscape |
In Ivan’s words
“this is the first time in our generation that we’ve seen software with actual unit costs”
Ivan Lee (04:23)
“If you want to automate customer support for an organization like Hyatt Hotels, would you be willing to pay $1.50 per answer?”
Ivan Lee (05:12)
“Those are the three main criteria that people are looking at when it comes to ROI”
Ivan Lee (06:16)
“I want to really ingrain with the audience that there is no single best model”
Ivan Lee (12:42)
Resources
Ivan Lee on LinkedIn: His LinkedIn profile
Datasaur: The company he leads, which he describes at 01:12 as a five-year-old startup founded to democratize access to natural language processing
The AI Ownership Threshold: One of two articles he has written for the AI Realized Now Substack
Outcome-Based Pricing: Nirvana, AI Necessity… or a Trap?: The second of his two articles for the AI Realized Now Substack
Named on air
LLM Labs: The Datasaur product he describes at 02:01, for comparing and selecting models and gathering the data to fine-tune a private one
Prompt unit testing: His term, introduced at 02:43 and explained at 03:10. A fixed set of prompts and expected answers run against a live model on a schedule, so divergence is caught rather than reported
GPT-4o Mini: The OpenAI model he names at 06:47 as the first announcement he had seen built on price rather than on capability
Llama and Google Gemini: The two he names at 07:12. Open source models like Llama apply downward pressure because they carry no proprietary fee; Google Gemini is where he says that pressure lands, and it advertises its efficiency against competitors
Apple Intelligence: Named at 07:58 as the thing he expected to change the mainstream assumption that a query has to reach a cloud model
CrowdStrike: The incident he reaches for at 03:10 as the kind of thing unit testing exists to prevent
Netflix, Spotify, Qualtrics and the FBI: The customers he names at 01:29 for the data annotation platform
Ideas and terms discussed
Unit cost: Every query against a model consumes GPU time, so software now carries a per-use cost rather than a one-time build cost
Cost, latency and quality: His three criteria at 06:16 for judging a model in production, set against the accuracy and precision a data scientist would reach for first
Small language models: His answer at 08:19 to running privately. An array of smaller models on the device rather than the largest model in a cloud
Crawl, walk, run: His phrase at 10:13 for the sequence. A pilot that proves the technology works on a real workflow, then responsible building with agreed timelines and criteria
A multi-model landscape: His conclusion. Different models for different jobs inside one organization, chosen on the merits rather than by standardizing on one provider
Related AI Realized episodes and events
Smaller Models, Bigger Wins: Verify Before You Answer: Jason Williamson on doing more with less compute, which is the same economics argued from the small-model side.
From Firefighting to Fire Prevention in IT Operations: Karthik SJ on where a predictive approach beats a generative one on cost, which is the unit-cost test applied to one workload.
Analytics as Code: Why AI Stops Guessing With Data: Chris Parmer on making an answer verifiable rather than plausible, which is the quality half of the three criteria.
Frequently Asked Questions
-
Measure three things before a model goes into production, and know the constraints sitting behind them: the unit cost of a query, the latency of the answer, the quality, and then compliance and regulatory limits on which models may be used and what data may leave the organization. Ivan Lee of Datasaur puts cost first because it is the one that is genuinely new. Software used to carry a one-time build cost and then scale for nothing; a model carries a real cost every time it answers, and methods that call several models multiply it. His practical test is whether that per-answer cost survives the business case at all.
Transcript 03:40 to 06:16
-
An AI query costs money every time because the expensive GPUs and the volume of compute behind it are consumed on each run, which is what makes this the first generation of software with real unit costs. Ivan Lee of Datasaur contrasts it with the older assumption that software is built once and then scales out in production for free. He adds that the popular research methods make it worse rather than better: chain of thought reasoning, and calling several models to arrive at one answer, each multiply the underlying cost.
Transcript 04:23 to 04:47
-
Inference costs were falling because model providers had begun competing on price rather than on capability, as of this February 2025 conversation. Ivan Lee of Datasaur pointed to OpenAI announcing GPT-4o Mini on how much cheaper it was than its own predecessor rather than on how much bigger it was, which he treated as a first. Open source models with no proprietary fee attached were applying pressure from below, and Google Gemini was advertising its efficiency against competitors. He expected costs to keep falling and thought that would unlock business cases that did not previously make sense.
Transcript 06:38 to 07:47
-
Yes, and an array of them can do a serviceable job of answering questions without a query ever reaching a large cloud model. Ivan Lee of Datasaur points to consumer devices to explain why the assumption was shifting: once Apple Intelligence runs on a phone, the idea that every query must travel to a large cloud model starts to break. His main interest is what that opens up, and it is regulatory rather than technical: industries that were never permitted to send data to a model provider, healthcare among them, change position entirely when a model can be self-hosted on an air-gapped machine.
Transcript 07:47 to 09:09
-
No. Different parts of one company will land on different models, and Ivan Lee of Datasaur wants that understood rather than argued. His example splits a single business: marketing and human resources might evaluate the options and find an off-the-shelf solution from a well-known provider works fine, while the divisions holding the proprietary work decide they need to build in-house for strategic reasons. His conclusion is that the era of a company being an AWS house or a GCP house is ending, and what replaces it is a multi-model landscape where an organization can be much more agnostic about which solution fits which job.
Transcript 12:22 to 13:54
Item description
-
[00:42] Christina Ellwood: Welcome to AI Realized, the podcast for enterprise executives leading AI deployments. From addressing security, data, and operations challenges, to managing the organizational and management changes, AI deployment presents the opportunity to redesign our organizations from the inside out. I’m Christina Ellwood, your host for today’s episode. We’re talking today with Ivan Lee, founder and CEO of Datasaur. Ivan, welcome to AI Realized.
[01:12] Ivan Lee: Thanks, Christina. Thanks for having me on.
[01:12] Christina Ellwood: Just by way of a little background, tell us a little bit about Datasaur.
[01:12] Ivan Lee: Of course. So Datasaur is a five-year-old startup. We were founded with the mission of democratizing access to the world of natural language processing, or NLP. So that’s the subset of AI related to text, documents, and audio We started by introducing a data labeling platform. A lot of these machine learning models out there require a lot of data in order to train and understand and be customized to your specific workflows, and our data annotation platform helped, uh, organizations, Fortune 500 companies like Netflix, Spotify, Qualtrics, and even the FBI in annotating this data and training their NLP models. In the last two years, we’ve launched a new product, LLM Labs, that helps people with doing exactly the same, but on the LLM side, comparing, selecting your models, and then gathering the data to fine-tune and customize your own private LLMs.
[02:01] Christina Ellwood: That’s fascinating, and it seems so important that people have tools like yours. Once they’ve selected, labeled their data and selected their model, how do they use your, uh, products or services once they’re in production?
[02:20] Ivan Lee: Yeah. We are built to, to support people in production, right? A lot of folks will, will train their models for the very first time, but there is an evaluation part of the cycle. Once you have something in production, how do you ensure that not only, um, are you maintaining that level of quality, but also identifying the areas where it’s faltering and improving upon that? So we’re a core part of that cycle. Take LLMs for a moment. As you’ve deployed your model, you have it live, we are intr- we’ve introduced this notion of prompt unit testing. Unit testing is a well-known, beloved practice in software. It just has all these mini tests that software has to go through before it’s deployed live to prevent something like the CrowdStrike issue. We’re doing the same thing for LLMs, right? You can have all these unit tests, all these prompts and correct answers out there, and you can run this daily, weekly just to make sure that your model, whatever is out there, is answering exactly as you expected. As soon as something diverges from what you’d expect, we will notify you, and you can review what might be going on.
[03:40] Christina Ellwood: That seems very commonsensical and practical. Um, when calculating the return on investment for a production AI use case, what factors do you recommend people consider?
[03:40] Ivan Lee: This is a really important conversation that I’m seeing become increasingly important, particularly as data scientists who generally are more interested in kind of research metrics like F1 score and accuracy, they need to also learn to speak the language on, of the business side of things and understanding how to explain the ROI for, for these AI solutions. So There’s a couple of things to note here. Um, first of all, this is the first time in our generation that we’ve seen software with actual unit costs. So this is not-- In the past, we’ve always assumed there’s a one c- one-time cost to build the software, and then after that, it’s forever free to scale out in production. LLMs, because of these very expensive GPUs, because the, the sheer amount of, uh, compute required to power these, they have real costs. Now, a lot of the research methods that are very popular out there, things like chain of thought reasoning and, and calling multiple models to, to get the right answer, you can s- imagine that each of those multiplies out that existing core cost. So I’m seeing a lot of pilots that may cost a dollar, $5 per query, and it doesn’t make sense in a business environment And so the very first thing is, okay, does the unit cost make sense on an ongoing basis for this business application? If you want to automate customer support for an organization like Hyatt Hotels, would you be willing to pay $1.50 per answer? Probably not. So you have to make the unit cost make sense first, first of all. But beyond this, there’s a lot of other dimensions that people are just starting to think through. There’s also a lag time in getting these answers. So when you call ChatGPT, some of these answers are taking you 20 seconds. It’s masked a little bit because it’s typing it out in front of you, and you feel like it’s making progress, but it’s a far cry from Google, which delivers 100 million results in .07 seconds. So here we’re looking at cost, we’re looking at latency, and then finally, of course, there’s the quality itself. Those are the three main criteria that people are looking at when it comes to ROI, and then there are additional behind-the-scenes factors, things like compliance and regulatory measures and what kind of models are you allowed to deploy and what kind of data can you send to third parties. So there’s a lot that goes into this equation beyond just accuracy and precision.
[06:38] Christina Ellwood: The costs are changing considerably, aren’t they, Ivan? What do you see as the principal drivers for cost reduction in the next year or two?
[06:47] Ivan Lee: Yeah. So I am very optimistic about a lot of the changes that we’re seeing. Uh, for example, when OpenAI launched their GPT-4o Mini, it was their first, their first announcement that was not as focused on how much larger the model was or the new use cases they’d had unlocked, but it was solely focused on how much cheaper it was compared to their own precessing-- predecessor product. We’re also seeing downward pressure because of open source models like Llama, which won’t have that proprietary fee attached to it. We’re seeing downward pressure from models like this onto other models like Google Gemini, which also is advertising how efficient its model is relative to its competitors. So I do think that, uh, costs, inference costs will go down over the next 12 months, and that’s just great news for the industry because it’s going to unlock that many more business use cases that will now make sense where it may not have made sense before.
[07:47] Christina Ellwood: Many companies are considering moving their or running their LLMs behind the firewall rather than in a public service. How is that gonna affect the cost model?
[07:58] Ivan Lee: I think that as Apple launches their Apple Intelligence on their iPhone 16s, that’s going to change the mainstream conversation a lot. Because we always assumed that these queries would need to go to a large, uh, cloud server somewhere, uh, and that a large model had to answer the queries. But as people are realizing that an array of smaller language models, or SLMs, can operate on device and do a serviceable job at answering these questions, uh, that will change what w- how we think about gen AI as a whole, right? It’s no longer about 405 billion parameter models, the, the most powerful models operating on a cloud, but being able to bring this on device, uh, to our pockets. And so I’m excited for the new use cases this is going to unlock. There’s a lot of industries where they weren’t allowed to send their data to a company like OpenAI or Anthropic previously, things like healthcare, where there’s a lot of private patient information, but if you can self-host on an air-gapped computer, that’s going to change the equation entirely.
[09:09] Christina Ellwood: Do you think the costs will go down, and will they go down substantially enough to open up use cases for smaller companies?
[09:16] Ivan Lee: I think that there’s an incredible amount of, of research and development and innovation coming in from researchers and academics through to hardware companies and the Nvidias of the world, through to startups building high performance inferencing capabilities. So I think all of this will actually decrease by an order of magnitude or larger, and that’s really going to supercharge adoption in this space.
[09:46] Christina Ellwood: So when you’re talking to enterprise executives who are in the early stages of designing their, uh, production AI use case infrastructure What do you advise them to do? Do you advise them to start by looking at small language models behind the firewall, or start by using a public model and then migrating, assuming that it is not specifically driven by compliance and regulatory requirements?
[10:13] Ivan Lee: So this is something that I have seen this journey many times at a lot of different organizations, and the best performing ones are ones that the traditional advice of crawl, walk, run. And so for a lot of organizations, they start with a pilot, a POC, just to demonstrate that the technology does work and it can solve their solution. There’s a lot of concerns out there about things like hallucination and cost and runtime, right? So being able to take a promising workflow and demonstrate to a business committee, a board of directors, and say, “Look, the technology GenAI absolutely does work for this workflow. Here are the first three use cases we would recommend.” Once you have demonstrated the potential in a handful of use cases, then it’s about being able to build responsibly, again, with reasonable timelines, being upfront about the costs of what it’s going to take. For example, it is not easy to hire data scientists and ex-- and engineers with expertise in this space just because it’s so new. So being able to provide, again, reasonable timelines for when this can be delivered and demonstrating very clear success criteria, agreeing ahead of time to say, “If we’re able to develop a solution that is able to provide an answer with these SLAs and under these costs, then can we consider this to be a success?” This is something I have worked for over a decade in the AI world, and time and again, we will get to 96% accuracy with very s- good recall and precision metrics, and inevitably an executive will come in and be like, “I tried this query and it doesn’t work. So the whole thing we need to redo it,” right? But being able to have those conversations upfront and understand that like any software, there will be some failure component, but if we can achieve these metrics, is that good enough on the broader spectrum, right? Having those conversations upfront goes a long way.
[12:22] Christina Ellwood: So we’ve covered a lot today in this episode. As we wrap up, what would you like our listeners to take away from our conversation today?
[12:30] Ivan Lee: Yeah. I think The-- there’s a couple of things that I’m really trying to drive home here. First is there’s a lot of chatter out there about what is the best model. I want to really ingrain with the audience that there is no single best model. What we will see in the coming years is that, let’s just take an organization, let’s just use Starbucks for today, right? An organization like Starbucks, each of their departments will actually have different gen AI solutions. For example, their marketing and their HR departments might evaluate all the options and decide that certain solutions from well-known providers off the shelf will work just fine for them. But on the other hand, their business analyst division and maybe their recipe generation division, more of the secret sauce to Starbucks, they realize, hey, off the shelf isn’t going to work for us. Maybe for strategic reasons, maybe for proprietary reasons, we need to develop our own in-house solution here. So a large organization like this, they won’t be-- it’s not like today where we say a company is an AWS house or a GCP house. We will see an organization have some of their solutions utilizing Google Gemini, others utilizing OpenAI, others still using Llama 3.1 on Bedrock. So I think we’re going to move to a multi-model, uh, landscape where we can be much more agnostic as to the proper solution.
[14:05] Christina Ellwood: Ivan Lee, thank you so much for talking with me today on AI Realized.
[14:11] Ivan Lee: Thank you.