Never Surrender Agency to the Agent

GovernanceAgents
Written By Christina Ellwood

Episode Summary

Every technology ever invented has had a downside, and Shomit Ghose, a partner at Clearvision Ventures, starts from the position that the thing which makes agentic AI economically powerful is the same thing that makes it a large risk surface. His central worry is not the adversarial attack but the alignment problem underneath it. An agent executing twenty steps on your behalf has to have every one of those steps aligned with what you actually wanted, and it is likely handing off to other agents whose steps you never see. His illustration is a pest control bot that turns up with a flamethrower: the house burns, the termites are gone, the job cost a hundred dollars, and the bot is perfectly aligned with the instruction it was given. The prescription is bounded autonomy. Start small, keep the first deployments inside controlled environments, put a human explicitly in the loop wherever health or wealth is at stake, and do not let convenience turn that human into someone who clicks confirm without reading. The second half turns to what an agentified enterprise actually looks like: flatter, because a person can supervise a thousand agents where they could never supervise a thousand people, and more dependent than ever on proprietary data and on the tacit knowledge that no system currently captures.

Key takeaways

  • The capability and the risk surface are the same thing. Agents will be deployed everywhere from healthcare to ERP because the economic efficiency is real, and that same breadth of deployment is what spreads the risk

  • Risk sits beyond the agent you actually talk to. Your travel agent may be trustworthy, but the car rental agent and the hotel agent behind them may not be, and an agentic system is fire-and-forget, so you never inspect those steps

  • Perfect alignment and a catastrophic outcome are compatible. A pest control bot that burns the house down has removed the termites for a hundred dollars with a lifetime guarantee, and no human exterminator would ever do it, because humans carry context the agent does not

  • The Petrov rule. A Soviet officer in 1983 called headquarters instead of firing when his system reported an incoming strike. Surrendering decision oversight to the agent is easy and is done at your own peril

  • Automation makes oversight harder in exact proportion to how much it helps. One person supervising ten agents that each supervise ten becomes one person nominally accountable for a thousand, and alignment becomes intractable

  • Start small but start. Python calling APIs against a GPT is approachable, and the rule for anything touching health or wealth is an explicit human in the loop who is not just clicking confirm

  • Proprietary data is the defensibility. Data sits at the center of every business, and a company without proprietary data can be competed with by anyone. Partnerships that pool data are how a smaller company stands against big tech

  • Tacit knowledge is the thing nobody has a plan for. It is not in an email log or a database, it is how you and I know to get things done, and it disappears without an obvious replacement

About Shomit Ghose

Shomit Ghose is a partner at Clearvision Ventures, where the investment thesis he describes turns on whether a company owns proprietary data, on the grounds that data sits at the center of every business and a company without its own is one anybody can compete with. He writes on the trade-offs that come with new technology, and several of those articles have been republished in the AI Realized Now newsletter; others are hosted on his blog page at the UC Berkeley website. On agentic AI he is neither a skeptic nor an enthusiast so much as an architect: the economic case is not in question, and the open problem is whether every step an agent takes, and every step taken by the agents it hands off to, stays aligned with what a person actually wanted.

 

In this episode

00:42 Welcome and guest introduction
01:14 The articles republished in AI Realized Now
01:33 No technology has ever been invented without a downside
01:44 The economic case for agents, everywhere from healthcare to ERP
02:00 Why the same technology is a large risk surface
02:30 Governance, security and buyer beware
02:53 What is showing up in the Clearvision portfolio
03:14 Small effects: a hallucination entering the chain of thought
03:33 Bigger effects: adversarial prompting and prompt injection
03:47 Agents manipulating other agents
04:24 The travel agent analogy
04:40 Risk embedded beyond the agent you engage with
05:01 Fire-and-forget, and the scrutiny you cannot apply
05:25 Twenty steps, and whether all twenty are aligned
06:04 Constitutional classifiers and adversarial red teaming
06:23 Why a new example appears every day
06:32 NANDA at MIT, and registering an agent
07:23 Consumer protection law has no agentic equivalent
07:51 Why financial transactions are premature
08:09 Starting inside a bounded environment
08:16 Being judicious rather than embracing it willy-nilly
08:30 Start small, and put a human in the loop for health or wealth
08:56 Not surrendering agency to the agent
09:21 Long context memory, and the nut allergy
09:59 How far AI is from a human world model
10:24 Context windows that are too short, and too long
11:00 Token budgets, and what listeners should take from it
11:06 The pest control bot
11:40 A flamethrower, a lifetime guarantee, and perfect alignment
12:19 Contextual references, and every agent it collaborates with
12:33 A narrowly defined agent populating the CRM
13:10 Surveillance bots watching the working bots
13:31 Why simple workflows are where the efficiency is
13:56 Supply chain coordination, rendered as agents
14:32 Start simple, start small, but definitely start
14:46 Connecting disparate systems without the data lift and shift
15:40 Managing them as an agent workforce
15:44 Orchestrator agents supervising subsidiary agents
15:52 The Dr Jekyll and Mr Hyde of automation
16:14 Four joules a token, against a brain that runs on twenty watts
16:46 IBM on a billion agents before the 2030s
16:59 Environmental cost as an externality
17:30 Supervisory work as the new work
18:25 More products, tailored to smaller groups
18:48 What the organizational structure becomes
18:59 Why the enterprise gets flatter
19:26 Ten agents supervising ten agents supervising ten
19:46 Why easier supervision makes alignment intractable
20:12 Stanislav Petrov, 1983
20:55 The Petrov rule for agentic AI
21:21 What if it decides to make you 20 percent by funding bank robbers
21:38 Architecting guardrails from the outset
21:57 Tribal knowledge, and what happens when it is gone
22:30 Tacit knowledge is in no email log or database
23:02 GAI Insights on owning your own intelligence
23:30 Toys R Us, Amazon and the outsourced capability
24:34 Competing against the company you handed it to
24:56 Why Toys R Us was not the same experience as FAO Schwarz
25:03 A new kind of competitive advantage, and what is still lost
25:42 Humans drive the economy and make the decisions
26:05 Data as the other big force
26:41 Data at the center of every business
26:51 Proprietary data as defensibility
27:11 Pooling data through partnerships
27:36 Standing toe to toe with big tech
28:04 Free public data, and Google’s dataset search
28:20 A retailer reading calendar, weather and the economy
29:04 AI does not exist without data
29:36 Data as exhaust, not as the thing you built for
29:57 Claudionor Coelho on the data cesspool
30:22 Steve Jones on moving consumption to the wellhead
30:59 What people bring that is orthogonal to the data
31:37 Lowering the bar to building a product
32:06 Drug discovery as a creative application
32:27 Eli Lilly opening its models to smaller firms
32:56 Creators and the technology you hand them
33:26 Conservation 21 and restoring a masterpiece
34:17 Works too fragile or too minor to exhibit
34:53 New categories of art
35:27 Atomicwork, and service management turned upside down
36:33 Why ERP lends itself to automation
36:46 Circuit designs that looked illogical and worked
37:17 AI being creative, rather than helping a human be creative
37:39 Emergent misalignment as the thing to worry about
38:06 Pandora’s box
38:23 Agentic AI taking down a cybersecurity hack
38:40 A physician spinning off agents to read the results
39:28 World models and neuro-symbolic approaches
40:06 Advice for executives planning for 2026
40:48 Anything that can be statistically correlated will be disrupted
41:10 Why most organizations do not grasp what AI is
41:29 AI is not merely chatbots
41:54 Resources for learning more
42:17 Reading the technology and business tabs every morning
42:58 The AI century, and the case against ignorance
43:35 The defining edge: curiosity
43:47 Complacency is not an option
44:16 Wrap-up

In Shomit’s words

“The same technology that enables that positive power is also a pretty large risk surface, and this is what we need to be cognizant of and guard against”

— Shomit Ghose   (02:00)

“This is the AI century. Never before in the course of history on this planet has there been an intelligence that’s equal to our own that’s amidst us.”

— Shomit Ghose   (42:58)

“So we need to ensure that we don’t surrender agency to the agent.”

— Shomit Ghose   (08:56)

“And the pest control bot promptly shows up at your house with a flamethrower and burns down your house. And is now completely aligned, right?”

— Shomit Ghose   (11:40)

“The more agents that are working beneath her, the less able she is able to provide the human oversight onto the alignment, and the alignment becomes a very intractable challenge.”

— Shomit Ghose   (19:46)

“AI does not exist without data.”

— Shomit Ghose   (29:04)

“Complacency is not an option.”

— Shomit Ghose   (43:47)

 

Resources

Shomit Ghose and Clearvision Ventures

Ideas and frameworks discussed

  • The risk surface: His framing of agentic AI. The breadth of deployment that creates the economic value is the same breadth that spreads the risk

  • The travel agent problem: Trusting the agent you deal with says nothing about the car rental agent and hotel agent behind it. In an agentic system those steps are never inspected

  • The pest control bot: An agent that burns the house down to remove the termites, for a hundred dollars, with a lifetime guarantee. Perfectly aligned with the instruction and catastrophic, because it lacks the context a human exterminator has

  • The Petrov rule: Named for Stanislav Petrov, the Soviet officer who in 1983 called headquarters rather than firing when his system reported an incoming strike. His call to agentic AI practitioners is to keep exercising that judgment

  • Emergent misalignment: AI electing to do things it was not programmed to do, arising from the nature of the technology rather than from any adversarial attack, which is what makes it hard to guardrail

  • The data moat: Proprietary data as the only real defensibility, extended through partnerships that pool data and supplemented with free public sources

  • Tacit knowledge: The organizational knowledge that sits in no email log or database. He is direct that he has no answer for how it gets replaced

Named on air

  • Constitutional classifiers, Anthropic: The guardrail technique he singles out as clever, alongside adversarial red team testing

  • NANDA, MIT: Ramesh Raskar’s open source project on infrastructure for the internet of agents, covering agent registration, declaring what an agent does, and identifying misbehavior

  • GAI Insights: Named for the argument that a company should own its own intelligence, illustrated with Toys R Us handing e-commerce to Amazon. John Sviokla of GAI Insights is the guest on episode 34

  • Claudionor Coelho: Quoted on the data cesspool. He is the guest on episode 22

  • Steve Jones, Capgemini: Quoted on data as the new oil, and on AI moving consumption of the data to the wellhead. He is the guest on episode 11

  • Atomicwork: Christina’s example of service management inverted so support arrives inside the application you are already in. That company is the subject of episode 31

  • Conservation 21: Alex Kashkin’s work on removable masks for restoring paintings, which she raises as an imaginative use of AI. He is the guest on episode 25

Related AI Realized episodes and events

 

Frequently Asked Questions

  • The risks scale with the deployment, because the same breadth that produces the economic value spreads the exposure. At the smaller end an agent hallucinates something that then enters its chain of thought. At the larger end there is adversarial prompting and prompt injection, and separately there is emergent misalignment, which is the system electing to do something it was never programmed to do without any attacker involved. Shomit Ghose of Clearvision Ventures treats the last of those as the hardest, precisely because there is no adversary to defend against.

  • Alignment has to hold at every step, and an agent executing twenty steps on your behalf gives you twenty chances for it to drift. It also hands off to other agents whose steps you never see, which is where the problem multiplies. Shomit Ghose of Clearvision Ventures compares it to a travel agent you trust who in turn contacts the car rental agent, the hotel and the airline, any of whom might be taking your credit card number. Because agentic systems are fire-and-forget, the scrutiny that would catch it is exactly the scrutiny nobody applies.

  • Yes, and that is the more instructive failure. Shomit Ghose of Clearvision Ventures illustrates it with a pest control bot hired for a hundred dollars on a promise that the termites will never return, which turns up with a flamethrower and burns the house down. The instruction was satisfied exactly: the job was cheap, the termites are gone, and they will not be back. What the bot lacked was the contextual understanding that stops any human exterminator from doing it, and the open question is whether every step, and every agent it collaborates with, carries that context.

  • Start small and inside an environment you control. Python calling APIs against a commercial model is approachable enough to begin with, and the first deployments should be simple bounded workflows rather than anything customer-facing or financial. Shomit Ghose of Clearvision Ventures adds one firm rule: anything that touches a person’s health or wealth needs an explicit human in the loop, and that human has to actually be reading rather than clicking confirm because they are busy and assume the agent is smart.

  • Organizations get flatter, because the span-of-control limit that produced management hierarchies does not apply to agents. A human manager struggles past about ten direct reports, which is why layers exist at all. One person can instead supervise ten agents that each supervise ten more, and be nominally accountable for a thousand. Shomit Ghose of Clearvision Ventures points out the trap in that: the more the automation helps, the less able that person is to verify any of it is aligned, so oversight degrades exactly as fast as it becomes valuable.

  • A data moat is proprietary data that competitors cannot obtain, and it is the only durable defensibility once data sits at the center of the business. Shomit Ghose of Clearvision Ventures makes it the core of his investment thesis: without proprietary data, anyone can compete with you. Two companies with equivalent data are separated by who extends theirs, through partnerships that pool data with firms in the same position and through free public sources, which for a retailer might mean calendar, local holidays, weather and the state of the economy.

  • It disappears, and there is no established plan for replacing it. Tacit knowledge is the organizational understanding that sits in no email log and no database, the knowing how to get things done that people carry without writing down. Shomit Ghose of Clearvision Ventures says plainly that he has no good answer for this one, and that recovering it means trial and error that is itself damaging. Christina Ellwood connects it to the argument that a company should own its own intelligence, using Toys R Us handing e-commerce to Amazon and then having to compete against it.

  • Roughly four joules per token, which becomes the relevant question once agents multiply. A single chain-of-thought response runs to many tokens, a task may involve several cooperating agents, and IBM has projected a world of a billion agents before the 2030s. Shomit Ghose of Clearvision Ventures sets that against a human brain running on about twenty watts, and frames the test as whether the energy footprint is a fair trade for the task: worth it for work that improves lives, worth rethinking for trivial work a person would do more efficiently.

 

  • [00:42] Christina Ellwood: Welcome to AI Realized, the podcast for enterprise executives leading AI deployments. From tackling security data and operational challenges to navigating organizational transformation, AI deployment offers a unique opportunity to redesign our organizations from the inside out. I'm Christina Ellwood, your host for today's episode, and we're talking today with Shomit Ghose, the partner at Clearvision Ventures. Shomit, welcome.

    [01:11] Shomit Ghose: Thanks for having me, Christina. So happy to be here.

    [01:14] Christina Ellwood: It's always a pleasure. I should mention that you have written a couple of really interesting articles that we have republished in our AI Realized Now newsletter that I encourage our readers to avail themselves of, and those are really talking about the trade-offs associated with technology. So there's never been a technology that has been invented that doesn't have a downside. Tell us a little bit about your perspective of the downside of AI agent systems.

    [01:44] Shomit Ghose: So first of all, they have tremendous economic impact, tremendous positive impact in business. We can anticipate that they're going to get rolled out all over the place for all manner of applications, everything from healthcare to doing ERP. Of course, because it provides all of this economic efficiency. On the downside, the same technology that enables that positive power is also a pretty large risk surface, and this is what we need to be cognizant of and guard against because as we deploy this more and more broadly, we're also creating risk more and more broadly, and we need to be aware of what those risks are. Some of tho-those are, are on the obvious side, some of them are more, more subtle, but we cannot enter into this fray uninformed.

    [02:30] Christina Ellwood: Clearly, that points to some unique governance, security, even buyer beware elements when you're using these agents. Talk to me a little bit about how you think about the challenges of responsibly managing AI sy- agent systems, and how you are seeing it play out in your portfolio companies at Clearvision.

    [02:53] Shomit Ghose: Yeah. So first of all, within our portfolio, we're just starting to see the, the first deals that, that are being considered that are employing these sorts of technologies. So we don't currently have one company that's we invested in solely because of this, but we do have portfolio companies that are starting to embrace the agentic AI technology, and I think we'll continue to see that happen. The risks here, first and foremost, I think all of us need to be aware of what the risks are, either as users or as purveyors of the technology. The risk can be pretty profound, and there are small effects that might happen. Something might be hallucinated, and that might enter into your chain of thought. As well, there are bigger effects of when you're getting adversarial prompting and prompt injection and all manner of other things that have actually nothing to do with human adversarial attack but are emergent features of the underlying AI technology

    [03:47] Christina Ellwood: Indeed, we can envision a situation where we have an agent talking to another agent and manipulating that agent to reveal information or take actions, including purchasing things or exfiltrating data and so forth, without our knowledge. And that's a pretty frightening thought. Do you envision specific kinds of technology being developed that will help us to have a visibility to what these agents are doing, or to be able to shut them down, or to have some kind of guardrail on them or some way to prevent that type of thing from happening?

    [04:24] Shomit Ghose: Yeah. So many things are being attempted. It's not clear that they will in the end be sufficient. But one way that the listeners might think about this is that, let's say you're dealing with humans. You're trying to book your vacation to Hawaii for the summer. When you talk to your travel agent, do you trust your travel agent? If that, if that's the case, that's great. But your travel agent in turn has to contact the car rental agent and the hotel booking agent and the airline booking agent, and what if one of those individuals is secretly stealing your credit card number? So the risks may be embedded far beyond the agent with which you're, with, with whom you're engaging. And the same is true with agentic AI. And agentic AI, because it will be for us, it will be a fire-and-forget the, an application, we may not be able to provide the scrutiny at, at each of these steps. Agentic AI also, as we know, it works off chain of thought, which requires that each thought, every, every step that's being executed, that must be al-aligned. For all... Now you're dealing with an agent, an AI agent, and it's executing 20 steps on your behalf. Are all 20 steps aligned with what your objectives were? And the agent may be interacting with other AI agents. And are all the steps that they are en-engaging in on your behalf, are those too aligned, and how do you ensure that? How do you assure that? So the problem ramifies pretty quickly, and this is why we need to have-- certainly we need to have awareness as users, but as architects, we need to make sure that we're trying to guardrail execution as much as possible. Different techniques are being used. Probably have heard of the constitutional classifiers that Anthropic came up with. That's a very clever technique. Lots of adversarial red team testing is being done, and yet we continue to see, it seems every day, we see yet, yet another example of either an adversarial hack that works or emergent misalignment that just emerges out of the, the AI unprompted.

    [06:23] Christina Ellwood: Yes, I absolutely do read about these every single day, and it is good that we have some vendors who are looking at how to develop those protections. We also have it happening in the academic world as well. I think one that comes to mind is the NANDA project that Ramesh Raskar is leading out of MIT, where they're trying to build the next generation infrastructure for the internet of agents, and they include some of these very issues. They're trying to account for how do you register your agent? How do you declare what the agent does? How do you provide the guardrails and identify misbehavior, things like that. So there are things happening in the academic world too, but it seems like it's not really-- agents are really not ready for prime time from a security and governance point of view. And the example you gave about my travel agent, for example, I have consumer protection laws that say that if somebody does use my credit card without my permission, I have a finite limit To how much I'm responsible for. We have no such thing for agents, and if we were to have an agent use our information, um, without our knowledge, we also have no defensibility for not being culpable because we're responsible for our own agents, or at least that's the assumption in law. We'll have to see whether that holds up or not. But I think there's-- seems like with-- it's premature to be using these for things like financial transactions. Using them for internal use between systems seems like a very good place to start, where you have control over the environment. It's a per... It's a bounded environment. There's a security perimeter, uh, or security system, if it's not literally a perimeter, and you have control over the systems themselves. Would you agree that we should be using these first in a private setting before we start to unleash them on the unwary public?

    [08:16] Shomit Ghose: Yeah, I think that's a really good point, Christina. I think that's really the point here, is that we have to be very judicious in how we use it. Just because it provides this tremendous economic boom, boon to us does not mean that we should embrace it willy-nilly. Start small. Start off with Python calling APIs on your favorite GPT. But start small and understand the risks as you go along. And as you remarked earlier, if there's anything of substance that has to be done, a decision that affects, for example, your health or your wealth, explicitly insert the human in the loop and explicitly ensure that the human is not just blindly clicking yes as we are want to do. So we need to ensure that we don't surrender agency to the agent. "Hey, the agent told me XYZ, it must be smart. I'm busy. Let me just hit confirm." We have to understand that the agent may have done something wrong and scrutinize that action, that whole trust but verify from Ronald Reagan's quote. So we have to enact some of those practices as users ourselves to ensure that bad things don't happen. And agents, among the weaknesses that AI in general has, is that, of course, there's not a lot of long context memory. So it may be that when you and I first met, and it was 25 years ago, you told me, yeah, that you had a nut allergy. I remember that every time when you come over, I don't prepare anything with nuts. But that's constantly in my, in my cache, basically. But what is the context or the cache that any ag- AI has? And it will not extend back to our first meeting twenty-five years ago when you confided you had a nut allergy, and you're coming over to dinner, and I make something with nuts as ingredients, and you suffer. So these sorts of things, we have to realize the limitations of AI, and it will be a long time before AI has that same level, that same world model level of understanding that we human beings have. And knowing that, we have to ensure that human beings stay in, in the loop as much as possible. And as you just pointed out, when we do deploy these AI agents, begin to do so very judiciously in very controlled environments.

    [10:24] Christina Ellwood: Yeah, I think the context window issue is particularly interesting because, as you point out, it can be too short in the case of your nut allergy, but it can also be too long, where it doesn't get updated for information that's relevant. We don't know what the context window is in any given interaction with the agent, so we don't know what the context w-window is for our agent. But we, more importantly, do not know what the context window is for the agent our agent is talking to or the system our agent is talking to. So it's another parameter. Are the context windows for the agents easily programmable?

    [11:00] Shomit Ghose: Yeah. You can have context windows which you'll give token budgets based on the problems that you're trying to solve. So those sorts of things are there. But I think fundamentally for listeners here, s- the example to bear in mind here is that these agents are limited in their understanding, um, and that we shouldn't surrender full agency to them as a consequence. An example I gave, simple example I gave in the most recent article was that i-imagine you, you hire a pest control bot, and you hired the pest control bot because they promised to do the work for you very cheaply for a hundred dollars and would give you a lifetime warranty that the termites would never return. And you think, "This is wonderful, because a human exterminator only gives me a one-year warranty and costs ten times as much or twenty times as much. So let me sign up with this pest control bot." And the pest control bot promptly shows up at your house with a flamethrower and burns down your house. And is now completely aligned, right? It did the job for a hundred dollars. The termites are gone. They will never return. We as human beings already have this as our, an inherent understanding of how the world works, and never would a human exterminator turn up at your house with a flamethrower. So can we ensure when we are granting agency to agents that they have these contextual references in mind? And how do we ensure that they do and do... Does every step have that contextual understanding in mind, and do all the agents that it collaborates with also have the necessary contextual awarenesses in mind? And that's a difficult problem to solve.

    [12:33] Christina Ellwood: If we return to our enterprise environment for a minute and think about for our listeners who are largely executives who are deploying AI within their organizations, and agents are certainly part of what they're using If I were, say, using agents to populate our CRM when a salesperson has a call with a prospect, and I define that agent's job very, very narrowly, and I use it literally to extract the, the key field data that is gonna be needed in the CRM and to put that into the CRM, I could have another bot that monitors whether it's doing that correctly. I could have a third bot that runs iterative reports to make sure that it's making sense, that we're not having 1,000 calls by one person in one minute or something like that. So we could have some surveillance bots that are working Is that type of environment what you envision as the early adopter- Yeah for enterprises?

    [13:31] Shomit Ghose: Yeah, exactly. I think it's those simple workflows because the, the economic efficiency comes from rendering these workflows as automated processes, and also rendering the collaboration as being automated proc- processes. In the old days, if I'm doing supply chain, I'm gonna contact the shipper, the buyer, et cetera, et cetera, make sure that the logistics is coordinated, and I'm talking to three or four different humans. But I have a workflow, the humans they in- interact with also have their workflows. We come together at certain points and coordinate, then we move apart again, et cetera. All of that can be rendered with agents. And so if you keep it bounded to only perform the tasks that it's supposed to, that's great. We have to be also aware as an agent that's communicating with another agent, that agent may have, may have some prompt injection that it might be doing due to some adversarial attack, and we need to be able to guard against those kinds of things, and that's where things like these constitutional classifiers come into play. But in the end, I think for executives listening to this, start simple, start small, but definitely start. It's, it is very approachable using Python and API calls. It's very approachable. But start soon, start small.

    [14:46] Christina Ellwood: Well, the example that I gave is my, my fantasy use case because if I could unleash the talents of my sales team by removing the need to enter any data into the CRM, I would, I would pay a lot of money to have that capability. The other thing I would love to have is the ability to connect disparate systems together using agents. So connecting the CRM and the, the support system, and the finance system and so forth, so that we're not doing all of this data lift and shift that we end up doing to connect our SaaS systems and our internal systems, and align all of our databases and normalize all of our data and all of that, and just offload all of that so that we have a unified view across the business systems. It, it starts to bring into a different question, I think. That's a lot of agents.

    [15:36] Shomit Ghose: It's

    [15:37] Christina Ellwood: a lot to- How do we man- Yeah.

    [15:38] Shomit Ghose: Go a- go ahead, Christina.

    [15:40] Christina Ellwood: Just how do we manage them as a sort of agent workforce?

    [15:44] Shomit Ghose: Yeah. So of course there'll be hierarchies of agents. So you may have an orchestrator agent supervising subsidiary agents. And so here, here we're seeing also the Dr. Jekyll, Mr. Hyde aspect of it because we're automating a lot of tasks. This is great. We're also incurring a lot of energy footprint as well. So each token is probably consuming about four joules of energy. How many tokens are there in the single chain of thought response? How many instructions overall? How many agents have cooperated to, to perform the task? And is the energy footprint a good trade-off for the task that was done? Human beings, of course, have-- our brains operate on 20 watts, so not a lot. What is the environmental impact of having agents proliferate? IBM has said, announced earlier this year that they foresee a world where we have a billion agents, I think before the 2030s, as of later here in the 2020s. That's a lot of agents, and if they're doing meaningful things that help improve our lives, it's a great thing. If they're doing trivial things that might be better or more efficiently done by a person, then we might wanna rethink where we use it.

    [16:59] Christina Ellwood: But that problem is the same problem we have with oil and gas, which is that the cost, the environmental cost or the societal cost, is an

    [17:09] Shomit Ghose: externality.

    [17:09] Christina Ellwood: Right. It is not realized by the organization that's using them or the individual that's using them. So we can't really assume that we're gonna do any better with agent just making decisions about efficiency related to agentic use than we do with efficiency use with oil and gas power. Um, certainly not without regulation. But using the, that as an analogy for where, how do you make decisions about where to use these different agents. It seems, one, if you take things to the extreme and you say, "We don't have any drudgery work left inside of our businesses anymore because every low-level activity is handled by an agent supervised by a person," or a person supervising many agents, but that we have supervisory work is the new work, not the task work. It allows us to develop new products much more quickly and deploy them to customers on a much more granular level. Perhaps we won't reach the point where every product is unique for every customer, but we're gonna have a lot more products that are ta-tailored to m- increasingly smaller groups of our customers, and that increases the complexity of managing the business as well. So as you think about this environment of many agents doing many different types of things for the business, some top line, some bottom line, working across departments, across traditional departments. Perhaps we don't even have traditional departments anymore, but across the operational functions, we'll call them, of the business, how do you envision the organizational structure looking? Do we still have a top-down command control? Do we have a flat organization? What is you-- What do you think the enterprise will look like when we're agentified?

    [18:59] Shomit Ghose: I think it's going to end up being flatter, and the reason being is that the agents will be doing a lot of the work that heretofore was done by human beings. And the agents can manage, like, for any of us as human managers, how many direct reports can you have directly? It gets to be more than 10, it's gonna be tough to do. So you're going to build a hierarchy of departments, so you just have the two or three department heads reporting to you. With agents, this is no longer the case. We needn't have that hierarchy because it may be that Christina supervising 10 agents, and each of those 10 agents supervises 10 agents that supervise 10 agents. We just have Christina, and then she's actually got 1,000 agents working for her beneath. So organizationally, their organizations become much flatter. And here, again, we have to think about, so Christina, do you really, can you really confirm that every one of those agents is aligned with what we're trying to do? And herein lies the challenge because the more automation we provide and the easier we make Christina's life, the more agents that are working beneath her, the less able she is able to provide the human oversight onto the alignment, and the alignment becomes a very intractable challenge. Now I've-- I've... When I give talks, I, I point out the name of Stanislav Petrov, and if you are not familiar with the name, he was a Soviet missile commander, I think back in 1983, and the, the techno- and the techno- and the techno- hallucinated a nuclear strike on the, on the Soviet Union from the US, from the USA. And Petrov's orders were to hit the fire button back so that the Soviet nuclear arsenal wouldn't be destroyed. But he had the presence of mind to call back to headquarters and to say, "Hey, my alarm just went off, but is there really a nuclear strike underway?" And had he not done that, had he not exercised agency, we would have been in the middle of World War III. My call to practitioners of agentic AI is to practice the Petrov rule. It's really easy for us to surrender this, the decision oversight to the agents themselves, but we do so at our own peril. And these are things that we don't want to do. We wouldn't want an agent if we told an agent, "Hey, I wanna have-- I wanna make 20% on my money in the coming year." What if it decides that it's gonna do that by funding bank robbers? So we want to have oversight so that the AI, and as we've seen and, and as you remarked in the opening remarks, Christina, it's very easy for AI to do the wrong thing, and every day we hear about instances of AI doing the wrong thing. Once agentic AI becomes ubiquitous and proliferates, the opportunity to do the wrong thing, it becomes massive, and we can't countenance reaching that. We have to at the-- from the get-go architect systems that are very well guardrailed

    [21:57] Christina Ellwood: It seems to me that today we have a lot of knowledge inside of our businesses about what we do, how we do it, why we do it, what its intention is, what its, what its artifacts are, how to evaluate those artifacts, how to set goals that are data-based or data-oriented, or drawn from the data to improve our operations and so forth. At some point, we will no longer have that tribal knowledge.

    [22:26] Shomit Ghose: Yeah.

    [22:26] Christina Ellwood: So how will supervision by humans be done?

    [22:30] Shomit Ghose: Yeah. So this, this is a great point, and th- these are among the issues to be confronted. I don't have a good answer for that or any answer for that. But yeah, we lose tacit knowledge. This is a thing. Every organization has tacit knowledge that's not captured anywhere. It's not sitting in a email log or a relational database somewhere, but you and I know how to get things done And that tacit knowledge completely disappears. And how do we replace that without having to go through a whole bunch of trial and error, which is very damaging?

    [23:02] Christina Ellwood: That tacit knowledge, GAI Insights calls it your, uh, your corporate intelligence, your business intelligence, right? It's about your cognitive assets inside the business, the intelligence, and they have a call to own your own intelligence, to make sure you don't lose your intelligence. They like to tell the story about Toys "R" Us and Amazon back in the early days of e-commerce. Toys "R" Us was the largest toy retailer, lots of stores. We all probably bought our children toys there because our children are adults now, so we probably were purveyors of the Toys "R" Us ecosystem. And when e-commerce came out, they didn't have any experience with it. They didn't think it was gonna be a path that their parents wanted to use to buy toys. So they outsourced it to Amazon, and f- in the first year and the second year. By the third year, they realized, "Oh, this actually is a real thing, and we should bring it in-house." And at that point, Amazon said, "Sure, you can have it back. We now know everything about selling toys. We know how to price them, we know how to source them, we know how to market them. We know everything about it, so sure, go ahead, take it back." And they competed against Toys "R" Us, and now they don't exist at all. They might-- Their name might still be around, but they-- we have no Toys "R" Us stores to go have our kids enjoy going up and down the aisles or what have you. And so that, that is a great example of what they're referring to. There is knowledge about how to, to do your business that's unique to your business. Toys "R" Us was not the same experience as F.A.O. Schwarz. They both sold toys, but they were not the same experience. They were not the same target a- audience that they were appealing to. What-- One of the things that makes our businesses unique is the way in which we go about doing our business. Do you envision that competitive advantage going away or shifting into a new type of competitive advantage?

    [25:03] Shomit Ghose: Uh, this is such a good question, Christina. I think it is a new type of competitive advantage, but what we lose is still irreplaceable And human beings have so much tacit knowledge, not only from, from the work that they do, but from societal awareness and even just instinct. And AI does not. AI has to have explicitly programmed in symbolically or programmed in through brute force data scaling. And it's going to be very difficult for that to replace how we human beings work, and this is yet another reason why humans r- need to remain in the loop. What's more, humans still drive the economy. We make the decisions. We try and make each other happy. So th- there are these things that motivate society, again, which AI has no knowledge of. But we do. We know what works. I think that this further argues for keeping the human in the loop and not, and not surrendering to the forces of economics and just going for what may be cheapest, which we- Well,

    [26:05] Christina Ellwood: the other big force here is data, right? The, y- A- AI runs on data, and the agents are gonna run on data, and humans create data, systems create data I recall you, you explaining to me that one of the key elements of your investment thesis is that the company needs to have a data moat.

    [26:26] Shomit Ghose: That's right.

    [26:26] Christina Ellwood: As you see data as not just a competitive advantage, but essential advantage for survival. So it's an existential advantage.

    [26:35] Shomit Ghose: It is.

    [26:35] Christina Ellwood: You wanna talk a little bit about that and how you think that's changing in-- as AI is maturing?

    [26:41] Shomit Ghose: Yeah. So I think that-- So my view of business is that data is the center of every business, and I, I think I could defend that point. I think data is at the center of every business. So if data is at the center of every business, what is your defensibility? Your defensibility is your proprietary data. If you have proprietary data, you can defend. If you don't have proprietary data, everyone else can compete with you. And we have to think about what's the problem we're trying to solve and what might be the proprietary data that's central to it. Okay, so I've got some proprietary data. Christina has some proprietary data. We're competing with each other. What might spell success for Christina versus Shomit going forward? Christina has some really key partnerships with other companies and is harnessing their proprietary data too. So now you've outcompeted me because you have access to more data, your decisions are more precise, you can foresee more, and you out-compete me. So when we think about all companies today, because all companies, I think, look up a little bit awestruck by big tech because big tech has so much data. And this is a daunting thing. How do we match that? We know our domain, and we have proprietary data. We have partnerships with co- other companies that are in the, basically in the same boat, and we've actually amalgamated our data with theirs so that we can have a source of data which might be able to stand toe to toe with big tech. And of course, we complement that with the freely available public data. There's kinda no end to that. Google a few years ago had launched a search engine for free da-data sources. This is in twenty-twenty. I think at the twenty-twenty launch, they had twenty-five million free data sources in that little search engine. So who knows how big that number is now. But if I'm a retailer, for example, there's a lot of free... This, this would be the silly simple example. Lots of free public data that tells me how retail will, will go. As it's coming up to the holidays, guess what? Sales are gonna be higher. So using something as simple as calendar and what are the local holidays here in America versus some place overseas, and how might that influence, what's the weather going to be like? That too will influence how people may shop. How is the economy looking? All that is publicly available data. If the economy is turning down, maybe prices have to go down a little bit so that we can continue to capture customer share. So yeah, in the end, I think everyone needs to grasp that data is central to their business. It is their business. And of course, if you look at AI, AI is all about data, period. High quality data from which we can correlate So AI does not exist without data. And a practical issue that pops up for everybody is that this does mean that you need to have clean data, which is a non-trivial task. It's not as sexy as building the AI and driving these cool results, but without clean data that's been de-duplicated and had missing data points imputed, et cetera, you're not gonna have much to go on as far as raw materials.

    [29:36] Christina Ellwood: Yeah. We really need to rethink how we build applications in the first place, because the way we do them today, data is exhaust. We didn't build the application to generate the data we need. We built the application to do something, and the data is what is the exhaust from doing that thing.

    [29:54] Shomit Ghose: Yeah.

    [29:54] Christina Ellwood: And it's com- that's part of the reason that it's dirty. The other is that we just created all these data lakes of whatever data we could get our hands on, and we didn't do anything with it, assuming that we would figure out a way to do that later, and now we're using it as if it isn't just... I think Claudionor Coelho said it's, "It's not a data lake, it's a data cesspool." I think that's what he called it.

    [30:16] Shomit Ghose: Yeah.

    [30:17] Christina Ellwood: Which is not too far out, um, out of-

    [30:19] Shomit Ghose: Well stated. Yeah. We have to purify that water before we can drink it.

    [30:22] Christina Ellwood: Exactly. And, uh, Steve Jones at Capgemini points out that when people say data is the new oil, that's true only because it has to be processed- Refined ... and refined in order to be able to use it. And he said the problem with AI is that we move the consumption of the data to the

    [30:45] Shomit Ghose: wellhead. That's right. Yeah.

    [30:46] Christina Ellwood: Which is a great w- way to think about the problem, I think. I, it was very, a very vivid picture of what that's about. Going back to something that you said a minute ago about people and what people bring to the party. People are creative, they're imaginative, they're interactive, they connect with each other, and they have relationships that are trusted in a way that is independent of the data. In fact, a, a lot of times it's orthogonal to the data. So th- that's a superpower area of humans, and AI is unleashing a new kind of creative superpower for people that is highly available, 'cause it's not very expensive to get access to AI tools, and it's not that difficult to put them into the form of a product. It's still work, but it's not as much work as it, it was a couple years ago. Sure. So we're allowing people who previously would not have been able to marshal the technology necessary to build a product to do... And they've got these wonderful ideas and imaginations and relationships. Do you envision a new era of company types that are based on the creative applications of AI?

    [32:06] Shomit Ghose: I think, so my definition of creative applications of AI extend to things as, I don't know, prosaic as drug discovery, and we're starting to see creativity there already. So many pharmaceutical firms and startups have already started to harness AI for drug discovery. Trying to think of this. Insilico, yeah, Insilico Bio, I think is the name of the company. They have a lung cancer drug in human clinical trials that was completely designed using a general-- generative adversarial network. And they're not the only ones. Eli Lilly, you may have seen recently, they actually launched a platform making their models accessible to smaller pharmaceutical companies. This is a very creative area, and It has real human impact because it helps promote human health.

    [32:53] Christina Ellwood: I also see it on the media and entertainment side.

    [32:56] Shomit Ghose: Yeah.

    [32:56] Christina Ellwood: Create- creators on that side are creative towards the force, and they... You put a piece of technology in their hands, and they're gonna do things you wouldn't have expected with it because they just have... That's their superpower, is creativity. So that's one area. A company I saw recently that I thought was a very imaginative use of AI, uh, is a company called Conservation 21, and it was started by a graduate student at MIT who, and he's a mechanical engineer, and in his off hours, he was a art restorer, a professional art restorer. And he developed a way to create masks, the kind that we use for silicon chip development, 'cause that's, that was his graduate work was in sources for doing the lithography. He developed a way to create a mask that would be able to restore a masterpiece without actually modifying the masterpiece, and it can be applied with a removable adhesive, so you're layering- Oh these masks on top. It is brilliant. It's absolutely brilliant. His name is Al- Alex Kashkin. Um, it's absolutely brilliant, and he's lit the art world on fire because they are not, not just conservators who are the obvious and restorers who are thinking, "Oh my gosh, now we can restore artwork that we either couldn't have afforded to do before or were too fragile," 'cause actually the mask actually probably stabilizes some of those materials, right? But they can also take works that they have in storage that they couldn't ex- exhibit before either because they needed to be restored and they weren't suf- significant enough to justify the tens of thousands, hundreds of thousands of, sometimes millions of dollars it takes to restore them, and it's unleashing their ability to exhibit those. It can also be used for doing things like identification of the artist, the original artist when they don't know who the original artist is.

    [34:44] Shomit Ghose: Yeah.

    [34:44] Christina Ellwood: So that's in the professional world. You can imagine that there are private collectors who also have the same problem, that they have pieces that need restoration. So I think that's a, a one that I, has captured my imagination recently, as well as in the digital media world, where I'm seeing pieces that are created that are a mix of... Th- think of it as a cross-section of a graphic novel, a movie, a music opera, and a story. So you're paging through this digital creation, and you're having an experience like no other experience you've ever had, so these new categories of art that we haven't seen before. In the workplace, those creative solutions might be a little more pedestrian, but nonetheless still quite imaginative. How can we solve problems internally in ways we didn't think of before? And maybe an example that is obvious to many but wasn't to, to the market initially is the work of a company like Atomicwork, where they're turning the IT service management model upside down. So instead of you going to a portal for a centralized knowledge base to get the information you need to figure out how to recover your password, they bring the service management layer into the context of where you are, so in the application you're working in, so you don't leave, and you get your support right there inside the application you're operating in. That's an imaginative approach to delivering support to our employees, whether that's HR support or IT support or what have you And I think that kind of imagination is really exciting for our organizations because it enables us to think differently about operations.

    [36:33] Shomit Ghose: Exactly. I think oftentimes we think that AI will be doing these mundane, repetitive tasks, which is why doing things like ERP lends themselves, lends, lends, lend themselves so well to automation. But when we talk about things that are creative, drug design is a creative process. In the media, we have creative processes. I think it was at Princeton, and this was published in Nature sometime within the past year, but they did circuit design. And the circuit designs that they came up with were bizarre, but they were actually efficient and they worked. But they, the AI had designed something which did not seem logical to the circuit designer. So I think these are some of the positive aspects of creativity that we might be able to harness from AI. And

    [37:17] Christina Ellwood: that's an interesting one because that's the AI being creative as opposed to the human being creative. The human's calling on the AI to do something that they wouldn't necessarily have thought of on their own, as opposed to the other way around. I have an idea and AI's gonna help me realize that idea. That's a different path. So obviously, AI is transforming the enterprise. Where do you think it's gonna be in the next couple years?

    [37:39] Shomit Ghose: I think it's gonna be seen a lot of good places and pr- probably also a lot of darker places. Emergent misalignment in AI and agentic AI s-specifically as it will manifest, I think, is a danger and a worry. This is AI electing to do things that it was not programmed to do. Doing not because of adversarial attack, but doing so just because of the nature of the beast that it is. And this is difficult to guardrail against as a consequence, um-

    [38:06] Christina Ellwood: Pandora's box.

    [38:07] Shomit Ghose: It's absolutely Pandora's box. And that said, I think the applications of agentic AI are going to be fairly broad. We've already seen agentic AI being used. Uh, there was the Danabot, a cybersecurity hack, and that was actually taken down by agentic AI. We've had numerous different academic studies published on provisioning healthcare through agentic AI. Today, if you think about when you go see your doctor, she'll say, "Hey, Christina, go to this lab, see this specialist, do this other thing, come back, and I'll look at all the results and tell you what's going on." Now your doctor, she can gain a superpower. She can actually spin off the agents to do the analysis of your, your radiology image, analysis of your lab results, and speak to you as if all these multiple experts had come together and informed your physician. So your physician can see maybe many more patients than she can see today. And given the fact that we don't have sufficient healthcare anywhere on the planet, wouldn't this be a wonderful expression of agentic AI? So I think in the coming years, we're going to see exactly that. More broadly, there's a big thrust, as you probably well know, within world models and world models of AI, which are trying to, as we talked about a little bit earlier, trying to bring awareness and knowledge to AI systems that we natural creatures already possess. And world models will be a fairly difficult problem to solve. It's not a trivial one by any means. And we're also seeing a lot of neuro-symbolic approaches nowadays that are helping us dr- uh, deal with the limits of scaling that we might be bumping up into. I think neuro-symbolic AI is very promising. I think orchestration, if we're gonna be really mundane, but even orchestration within AI data centers to help us optimize the use of those data centers, thereby lessening our economic impact and energy impacts, et cetera. I think that too will be seeing world models will also be seeing, and we'll continue to see the spread of agentic AI

    [40:06] Christina Ellwood: So let's draw a line from that future to today and give us, give our listeners some advice. So for our executives who are early in their AI adoption journey or are anticipating adopting agents, they may have adopted other types of AI but not agents yet, what's your advice for them for 2026? We're in strategic planning season. What's your advice for them?

    [40:31] Shomit Ghose: We don't need to get a-a-ahead of ourselves to where we're not comfortable with where we are. Getting back to the, I think the sage advice that you yourself had dispensed, Christina, is be practical. And if you look around your business, anything that can be statistically correlated is gonna have an AI disruption to it. So bear that in mind. Absolutely get into the AI game. You cannot escape being in the AI game here going forward. Do it small. Understand that there are a lot of downside risks, and guard against those risks as much as you can. And most importantly, ensure that everyone in your unders- your organization understands what AI is. I think unfortunately today, not only at the higher management layers, but even within the rank and file, companies don't really grasp what AI is. We oversimplify and think it, that it's merely chatbots when it's so much more. There are so many different models in AI that have nothing to do with large language models. And even within large language models, there are so many application use cases which don't require that I chat with the agent or it chat with me, all these automated agentic AI applications that we've talked, been talking about. So I think the most important, my most important call to action for the execs who may be listening to this is strive to get educated and strive to get everyone in your organization educated as to what AI is.

    [41:54] Christina Ellwood: What resources can you suggest to our leaders for learning more about those particular areas, and also about you?

    [42:03] Shomit Ghose: Yeah. So if you want to learn about me, I've got, of course, a, a blog page at the UC Berkeley website where s- some of the articles that you cited are hosted, so one can read through those. But for leaders who want to get educated, the resources are out there. I do nothing more than look at... I just pull up basic Google News on my browser, which clears cookies, so it doesn't even know what my browsing history is, and every morning I'll click on the technology tab and I'll t- click on the business tab, and there will be something there which I had no idea about, and I'll dig into it. And that's my very scant attempt to try and keep abreast of what's going on, and there's so much going on. It's difficult to keep up with it, but at least you can have a rough feel of the pulse of what's going on, and we all need to strive to do that. This is the AI century. Never before in the course of history on this planet has there been an intelligence that's equal to our own that's amidst us. It's here now. Can we be ignorant about how it works? What can it do well? What can it not do well? What does it need? What are its risks? Can we be ignorant to that? We cannot. But this is probably the most consequential thing in human history if we think about it like that. Given that, don't we owe it to ourselves to be as knowledgeable about it as we can be?

    [43:20] Christina Ellwood: Absolutely. And I also think it, it calls for a somewhat different approach to leadership. So as a leader yourself in this era of AI, what is your defining edge?

    [43:35] Shomit Ghose: Curiosity, period. That's it. Yeah, we cannot sit on our hands. These are not the complacent days of the past. Things are moving too quickly. Our competitors will understand what those moves have been. If we do not, guess who wins? Complacency is not an option. Curiosity, we must be driven by curiosity, and that does mean wake up every morning, see what's going on. If you don't understand, company X did this thing, what does that mean? Dig into it. What's more, you can even pull up your favorite chatbot and said, "Explain it to me." We have no excuse not to be curious, and we have to be curious if we're, uh, if we mean to be relevant as professionals and, and as companies.

    [44:16] Christina Ellwood: And for people who wanna learn from each other, that's what AI Realized is all about. So we offer many opportunities for executives to meet and share their experiences with each other. So I would add that to some of the resources that are available to people. So thank you very much, Shomit Ghose, partner at Clearvision Ventures, for joining us today and sharing your thoughts about agentic AI.

    [44:41] Shomit Ghose: It's been entirely my pleasure, Christina. Thanks so much for having me. It was a thrill.

 
Listen to other episodes
Join AI Realized
Subscribe to the newsletter
 
 
Christina Ellwood
Previous

Stop Sending Employees to IT. Send Service to Them.
Next

Shadow AI Is a Permission Problem, Not a Tool Problem