Threat Intelligence Is a Board Question, Not an IT One
Episode Summary
Most executives now understand that deploying AI creates new threat vectors. Threat intelligence is a different discipline from knowing what those threats are. Staffan Truvé, co-founder and CTO of Recorded Future, defines it as the outside-in view: what an attacker can learn about your organization by observing it from the outside, set against what that attacker is both capable of and motivated to do. His company tracks roughly 5,000 threat actors on exactly those two axes, and the ones worth your attention sit where high capability meets high intent. The conversation moves from who owns this inside a company, which is usually the CISO but arguably the board, to what it costs, which is 5 to 10 percent of the IT budget, to the two kinds of AI-enabled attack now emerging: the ones aimed at our systems, and the ones aimed at our brains.
Key takeaways
Threat intelligence is the outside-in view. Not what your monitoring sees, but what an attacker can find out about your mail and web servers, your tech stack, your personnel and your locations just by watching from outside
Relevance is capability multiplied by intent. Capability is readable from the malware an actor uses and what it has attacked before. Intent is estimated by how closely you resemble its previous targets. Recorded Future tracks about 5,000 actors on both axes
The buyer is usually the CISO, but the question belongs to the board. Risk reduction is the board’s job, and what finally got it onto board agendas was watching peer companies get hit by ransomware
Budget roughly 5 to 10 percent of IT spend against this, adjusted for risk appetite. It is sized in relation to the IT budget rather than as a dollar figure, the same way a CISO budget is
The single contract term that matters with a foundation model provider is that they may not train on your data. Information can be extracted back out of trained models in practice, whatever the assurances say
Vibe coding moves the work from writing code to inspecting it. Chaining ten probabilistic black boxes compounds the randomness, which is what breaks conventional regression testing
AI-enabled attacks come in two kinds: attacks on our systems, and attacks on our brains. The second, meaning influence operations, phishing and deep fakes, is where attackers are moving fastest right now
About Staffan Truvé
Staffan Truvé is co-founder and Chief Technology Officer of Recorded Future, the threat intelligence company he and his co-founder began planning in 2007 around the idea of organizing the internet for analytics rather than for search. He has worked in AI in one form or another since the mid-1980s, starting with expert systems, then a PhD in computer vision, then years on information visualization and analytics. The company began in geopolitical risk analysis and moved into cyber threat intelligence about five years later, when customers started raising it, and it now tracks roughly 5,000 threat actors ranging from nation states to ransomware gangs. Alongside the commercial work he has taught at university and runs academic collaborations, including projects with the Royal Institute of Technology in Stockholm on AI-based red teaming that feeds real threat intelligence into automated attack simulation.
In this episode
| 00:42 | Welcome and guest introduction |
| 01:13 | Where AI meets threat in the cybersecurity stack |
| 01:36 | Seventeen years of using AI on both sides of the problem |
| 01:56 | Expert systems in the mid-1980s, then a PhD in computer vision |
| 02:17 | Why 2007: the iPhone, AWS and the first deep learning papers |
| 03:00 | Organizing the internet for analytics, not research |
| 03:25 | From geopolitical risk to cyber threat intelligence |
| 03:51 | The internet of agents, and Ramesh Raskar’s NANDA project at MIT |
| 04:19 | Three phases of AI at Recorded Future |
| 04:49 | Managed autonomy as the phase now beginning |
| 05:20 | Why threat intelligence is more than knowing the threats |
| 05:51 | The outside-in view: what an attacker can see |
| 06:26 | Your vulnerabilities on one side, actor intent on the other |
| 06:52 | Tracking about 5,000 threat actors |
| 07:11 | Reading capability from the malware an actor uses |
| 07:31 | Opportunistic gangs against actors with a defined intent |
| 07:59 | Capability and intent together, and what to patch first |
| 08:40 | Who owns this: CISO, CIO, CEO or board |
| 09:24 | The case for putting it in front of the board |
| 09:55 | Why boards are a hard audience to reach |
| 10:17 | Ransomware as the thing that got board attention |
| 10:38 | A preventive measure, and reducing the blast radius |
| 10:57 | Why banks are the most mature, and why regulation is the reason |
| 11:29 | Healthcare as a primary ransomware target |
| 11:46 | The two-stage ransomware model: lock, then steal and expose |
| 12:35 | What percentage of budget this needs |
| 13:08 | Five to ten percent of the IT budget |
| 13:40 | What executives outside security can actually do |
| 14:10 | Caution about what you share with external AI services |
| 14:31 | The contract term: no training on your data |
| 14:59 | Extracting information back out of a trained model |
| 15:02 | Information security hygiene and upload policy |
| 15:41 | Guidance for people doing vibe coding |
| 16:11 | From writing code to inspecting the AI’s code |
| 16:31 | Automating the security checking, and testing probabilistic systems |
| 16:53 | Ten black boxes in a chain, and compounding randomness |
| 17:19 | What happens to regression testing |
| 17:28 | The framework session at the AI Realized Summit |
| 17:46 | An OpenAI tool for finding vulnerabilities in code |
| 18:06 | Collaborating outside the company |
| 18:16 | AI-based red teaming with the Royal Institute of Technology |
| 18:37 | Feeding threat intelligence into an attack simulation |
| 19:22 | Why corporate collaboration beats standards bodies |
| 19:44 | The problem with standards: once they are set it is too late |
| 19:50 | MCP as the example, and what it left out |
| 20:23 | Why security keeps arriving last |
| 20:47 | Two kinds of AI-enabled attack |
| 21:04 | Attacks on our brains: influence operations, phishing, deep fakes |
| 21:24 | Detecting an AI-generated face on a video call |
| 21:43 | The evil hackathon |
| 22:05 | Reading a factory’s control systems off its own brochure photos |
| 22:50 | Not yet seen in the wild |
| 23:02 | AI-generated polymorphic malware |
| 23:34 | Could behavioral detection catch it |
| 24:30 | The arms race, and bots that learn your typing pattern |
| 25:00 | Voice patterns |
| 25:06 | The defender’s dilemma |
| 25:35 | Rethinking authentication |
| 26:08 | How long before today’s defenses stop working |
| 26:30 | Why attackers have not needed the next generation yet |
| 27:05 | The OpenAI Atlas browser |
| 27:24 | A browser that sees every single thing you do |
| 27:42 | Why running it in a container does not solve it |
| 28:17 | What individuals bring into a company inadvertently |
| 28:51 | Start with training, especially on information leakage |
| 29:16 | Being careful about which tools you allow |
| 29:54 | Key takeaways |
| 30:06 | Tech optimism with a devil’s advocate attached |
| 30:31 | Resources |
| 30:39 | The Record, and the Click Here podcast |
| 31:20 | Leadership: curiousness with a skeptical mind |
| 31:57 | Wrap-up |
In Staffan’s words
“Essentially, if you’re concerned about someone attacking you in some way, what do they know about you? What can they find out by just observing you from the outside?”
— Staffan Truvé (05:51)
“If you see enough of your peers being attacked and suffering tremendously, then all of a sudden it does become a board level question.”
— Staffan Truvé (10:30)
“It’s really to ensure that you have your contract set up so with whatever, let’s say, foundation AI model provider you’re using, that they are not allowed to train on your data in any way.”
— Staffan Truvé (14:31)
“You move from writing code to actually inspecting the AI’s code.”
— Staffan Truvé (16:11)
“There are two kinds of AI-enabled attacks. I sometimes think about them as the attacks on our brains and the attacks on our systems.”
— Staffan Truvé (20:47)
“People like to talk about the defender’s dilemma, the fact that if you’re a defender, you need to succeed every time, whereas if you’re an attacker, you only have to succeed once.”
— Staffan Truvé (25:06)
“Given that it’s very easy to become enthusiastic, I think trying to play the devil’s advocate and carefully think about what risks come with every opportunity is a good mindset to have.”
— Staffan Truvé (30:16)
Resources
Staffan Truvé and Recorded Future
• Staffan Truvé on LinkedIn: linkedin.com/in/staffan-truv%C3%A9-51539
• Recorded Future: recordedfuture.com. Threat intelligence, tracking roughly 5,000 threat actors across nation states, ransomware gangs and smaller operators
What he points listeners to
• The Record: therecord.media. Recorded Future’s news publication, covering both geopolitical and cyber threats
• Click Here: therecord.media/podcast. The podcast he recommends above everything else, on the grounds that it gets the attackers themselves to talk about how they operate
Ideas and frameworks discussed
• The outside-in view: His definition of threat intelligence: what an attacker can learn about you by observing from outside, covering externally visible servers, your tech stack, your personnel and your locations
• Capability and intent: The two axes he scores threat actors on. Capability comes from the malware they use and what they have attacked before, intent from how closely you resemble their previous targets. The intersection is your real threat landscape
• Managed autonomy: His name for the phase after summarization and report writing, where the tasks get automated at scale and the open problem becomes control mechanisms for large networks of agents
• The defender’s dilemma: A defender has to succeed every time, an attacker only once
• Attacks on our brains and attacks on our systems: His split of AI-enabled attacks. Influence operations, phishing and deep fakes on one side, reconnaissance and polymorphic malware on the other
• The evil hackathon: A week his team spent trying to use AI for bad. The standout result was identifying a factory’s control systems from photographs the company had published in its own brochures
Named on air
• Ramesh Raskar, MIT: The NANDA project on the internet of agents, which Christina raises as a parallel to organizing the internet for analytics
• Maher Hanafi, Betterworks: Named as presenting a framework at the AI Realized Summit. He is also the guest on episode 12
• Royal Institute of Technology, Stockholm: The academic collaboration on AI-based red teaming, combining automated attack simulation with threat intelligence
• Model Context Protocol: His example of a standard developed fast and adopted widely, with authentication and security added afterward rather than designed in
Related AI Realized episodes and events
• Your AI Agent Is Not the Risk. Its Authority Is.: Yogita Parulekar on delegated authority, which is the control problem Staffan raises when he talks about managing large networks of agents.
• AI Governance as Code: From PDF Policies to Pipelines: Ken Johnston and Bob Rapp on controls that run in the pipeline, the same move as automating the security checking rather than writing it down.
• Connecting AI Agents to Live Enterprise Data: Deepti Srivastava on the gap between probabilistic models and dependable data, which is the testing problem from the data side.
Frequently Asked Questions
-
Threat intelligence is the outside-in view of an organization: what an attacker can learn about you by observing you from the outside, and what that attacker is capable of and motivated to do with it. That includes externally visible resources such as mail and web servers and the flaws in them, who you connect to, and whatever is publicly discoverable about your tech stack, your personnel and your locations. Staffan Truvé of Recorded Future describes it as two halves that have to be held together: understanding your own vulnerabilities, and understanding the opportunities and intent of the actors who might come after you.
-
The actors likely to attack you sit at the intersection of capability and intent. Capability is readable from an actor’s history, the malware it uses and the vulnerabilities it is known for exploiting, and laying that over the weaknesses in your own systems shows what it could do to you. Intent is estimated by comparing your organization against the other targets that actor has gone after. Recorded Future tracks roughly 5,000 threat actors on both axes, from nation states to ransomware gangs to script kiddies, and the ones that matter are those scoring high on capability and high on intent at the same time.
-
Responsibility usually sits with the CISO, and in larger or more mature organizations with a head of threat intelligence reporting into the CISO. Where neither role exists it tends to fall to the CIO. Staffan Truvé of Recorded Future argues the underlying risk assessment belongs higher than any of them, with the CEO or the board, because risk reduction is what a board is for. Banks, insurers and some government agencies are the most likely to have a dedicated threat intelligence function already.
-
Roughly 5 to 10 percent of the IT budget. Staffan Truvé of Recorded Future gives that as the range commonly cited, with the caveat that the right number depends on the organization’s risk appetite. It is framed as a proportion of IT spend rather than as a dollar figure, on the same logic that a CISO budget is separate from the IT budget but sized in relation to it.
-
The contract is the first control: make sure the foundation model provider you use is not permitted to train on your data in any way. Staffan Truvé of Recorded Future calls that the key ingredient, because information can in practice be extracted back out of a trained model even where providers say it cannot be reverse engineered. Everything after that is ordinary information security hygiene, meaning a stated policy on which kinds of files can be sent off for AI analysis at all, and on what comes back.
-
Vibe coding is only as secure as the inspection that follows it, and taking it seriously means moving the effort from writing code to reviewing what the model wrote. Staffan Truvé of Recorded Future names two changes the software development process needs: automatic checking for weaknesses, with AI used to automate the security review itself, and a way to test systems whose components are probabilistic rather than deterministic. Chain ten black-box models together and that variability compounds along the chain, which is what conventional regression testing was never built for.
-
Attackers are using AI in two broad ways: attacks on people and attacks on systems. The attacks on people are influence operations, phishing and deep fakes used in scams, and that is where attackers are moving fastest today. The attacks on systems are things like AI-driven reconnaissance and, so far in research rather than in the wild, polymorphic malware that changes shape to evade detection. Staffan Truvé of Recorded Future describes an internal exercise in which his team identified a factory’s control systems from photographs the company had published in its own marketing, and says no verified case of a threat actor doing the same has been observed yet.
-
AI-generated images can be detected today, but only up to a point, because detection and generation are in an arms race. Recorded Future’s platform includes detection for AI-generated images, so a warning can fire if the executive apparently on a video call is synthetic. Staffan Truvé is direct that generation models keep improving and detection has to be retrained constantly, and that malicious bots will get better at imitating human behavior as keyloggers feed them real typing and mouse patterns to learn from.
-
[00:42] Christina Ellwood: Welcome to AI Realized, the podcast for enterprise executives leading AI deployments. From tackling security, data, and operational challenges to navigating organizational transformation, AI deployment offers a unique opportunity to redesign organizations from the inside out. I'm Christina Ellwood, your host for today's episode, and today we are talking with Staffan Truvé, the CTO of Recorded Future. Welcome to the show.
[01:10] Staffan Truvé: Thank you. Thanks for having me.
[01:13] Christina Ellwood: So I've been interested in learning more about this threat intelligence work that you're doing and understanding how AI is creating new risks associated with threat and how it, the pieces of the AI puzzle fit together in the cybersecurity stack. Can you help me with that?
[01:36] Staffan Truvé: Uh, I should be able to do that. We've been in that business for the last 17 years, actually, so it's been a long story for us, actually, in using AI in several stages. And of course, as you say, today not only to work with it on the positive side, but actually to figure out how to counter malicious use of AI.
[01:53] Christina Ellwood: What got you interested in this area in the first place?
[01:56] Staffan Truvé: If you go back a long time, I've been doing AI in one shape or form since the mid-'80s. So I was old enough to have been writing some expert systems back in the mid-'80s. Then I did a PhD in computer vision, which of course was very different in those days compared to modern-day deep learning-based image analysis. And then I spent many years working on information visualization and analytics in different kinds. And then going all the way forward to 2007, actually, we started, my co-founder and I, started talking about doing something new together. And if you jump back then to 2007, that's the year the iPhone was launched, so that's when all humans pulled, went full into being sensors feeding the machines with data. It's also the year AWS was launched, so cloud computing, meaning that we could actually do scalable computing in the way we needed. And it's also the year when the, the first articles about, first pa- research papers about deep learning, the birth of modern-day AI, was published. So at that point when we said that we think there's a point in organizing all of the internet, very ambitiously you might say, for analytics, not just research, and we saw AI as a possible and necessary component to do that. Now, back in those days, we were actually more focused on geopolitical risk analysis. Took us another five years until we saw or actually heard from our customers that cyber threats were an emerging area, and that's when we slightly refocused. We're still doing geopolitical risk, but cyber risk and cyber threat intelligence has became, become our main focus since then.
[03:42] Christina Ellwood: I think it, I think the whole geopolitical risk is emerging as a-
[03:46] Staffan Truvé: I was gonna say it's, there's been a big- Yeah there's been a big comeback for geopolitical risk, you might say.
[03:51] Christina Ellwood: Yeah. You're on two trend lines from the sounds of it. Yeah. Your story about the two of you thinking about the internet for analytics as opposed to the internet for messaging reminds me of some of the work that Ramesh Raskar is doing right now at MIT with his NANDA project, where they're thinking about the internet of agents, and what do we need to do in the internet infrastructure to support agents. Is that a good parallel?
[04:19] Staffan Truvé: It's very relevant. I'm sometimes saying that we've been using AI for, in three phases at Recorded Future. So the first one, which goes way back, was using AI to organize the information on the internet, to make it available for analytics and search. The second phase, which sort of coincides when, with ChatGPT, you might say, modern large language models, is really to use AI for, you know, first summarization, and then as a next step to, to write reports and so on. And the third step is clearly to start automating these tasks at a larger scale. So then it, it all becomes agents, whether they are LLM-based or otherwise. I think automation or managed autonomous, I like to call it, is really the next phase we're now entering. So thinking about how both what that enables and what's lacking in terms of control mechanisms, the ability to understand and manage huge networks of agents is clearly among the more relevant things you could do research about.
[05:20] Christina Ellwood: The threats maybe are somewhat intuitively understood by our audience of executives because they're deploying AI in their own organization, so they have some sense that there are some threat vectors maybe that are new or unique. But the idea of threat intelligence is a different concept than just knowing what the threats are. Can you talk to us a bit about w- how you define threat intelligence and how you see its span of influence or, or responsibility?
[05:51] Staffan Truvé: So I like to talk about it as threat intelligence being the outside in view on a, let's say a company or an organization. Essentially, if you're concerned about someone attacking you in some way, what do they know about you? What can they find out by just observing you from the outside? And that can be anything from externally visible resources, your mail and web servers, and what flaws there might be in them, but also who are you connecting to, what other kind of information is available online about your tech stack, about your personnel, your locations. It spans everything. So threat intelligence is in one part to understand your vulnerabilities. The other part is to understand the opportunities and intents of the threat actors. So we're trying to assess or help our clients assess both how capable a threat actor is of attacking you in particular, and also how interested they are in it. And the way we do that is that we map out all the big relevant threat actors. I think we have about 5,000 threat actors we're tracking today. Everything from nation states to ransomware gangs to script kiddies. And essentially, we say that their capabilities are possible to see by the way, which malwares they use. What have they been historically attacking? Are they known for attacking vulnerabilities in your Oracle web servers or whatever? And essentially, if you look at their toolbox, their set of capabilities, and the weaknesses you have in your system, the intersection of that, if you like, gives them the opportunity to attack. But then it's also a question of intent. So whereas some groups like ransomware gangs are completely opportunistic, they will go after anyone if they can make money out of it, some are much more, have a much more clear, clearly defined intent. They might be doing this for political reasons, for ideological reasons, and others. So by comparing your company with, uh, other targets they've gone after, you can see how similar we are, and that gives you an estimate of the intent. And of course, what you want to do then is to find the most relevant threats to you is really the, the, the people who are both willing and able, so they have high capability and high intent. Threat intelligence can help you map out that threat landscape and tell you what to focus on. And of course, it's important that it's not only a question of understanding who is the highest risk for you, it's also what you can do to reduce that risk. Now if, as, again, an example that a certain gang who are probably interesting going after you are good at utilizing a certain set of weaknesses, then you can prioritize, "I should be patching these systems. I should shut these doors so they can't get in," essentially.
[08:40] Christina Ellwood: So who you, who is the executive responsible for this particular category? 'Cause it strikes me as very different than cybersecurity
[08:51] Staffan Truvé: It's a good question, and maybe honestly, sometimes that's a challenge for us. So in some organizations, I would say the typical buyer here is the CISO, the chief information security officer, if a company has one, which of course not everyone does. Sometimes if the company's big and, I would say, mature enough, there might actually be a head of threat intelligence reporting into the CISO. If you go to a mature organization, which could typically maybe be a bank, some government agencies, or an insurance company, they would probably have that function. In some cases, they don't have either a threat intelligence responsible person or a CISO, so then it might be the CIO who's responsible for this. You could argue, though, that at some level, the, this kind of risk assessment, which is what we're doing in one, in, as one part, is really a question which is mostly applicable to the CEO or even the board. Like, you know-
[09:43] Christina Ellwood: That's what I was thinking. That's part of the job of the board in many ways.
[09:47] Staffan Truvé: Exactly. Their job is to do risk reduction, right?
[09:49] Christina Ellwood: Absolutely. Do you talk to the boards much? Are they aware of this technology even existing?
[09:55] Staffan Truvé: Not as much as we would like to. No. They just don't do that, yeah. Yeah. So sometimes...
[10:02] Christina Ellwood: Yeah, go ahead. You say they're not an easy audience to reach, and they have, they have many competing priorities on their plate, and I think m- like many things related to risk, they wait till the horse is out of the barn.
[10:17] Staffan Truvé: Yeah. No, exactly. Exactly. No, but I think some- sometimes, I'm, I'm half-jokingly saying that the one good thing about the surge we've seen in ransomware attacks is actually that this has been brought up to the attention level of the boards. If you see enough of your peers being attacked and suffering tremendously, then all of a sudden it does become a board level question.
[10:38] Christina Ellwood: Yeah. It, it make- it makes sense, so it's a preventive, it's a risk assessment and preventive measure to avoid an attack or to minimize the blast radius of an attack. Is that-
[10:48] Staffan Truvé: Yeah, the impact. Exactly ...
[10:50] Christina Ellwood: a good summary? Yeah. Yeah. What are the hallmarks of organizations that are vulnerable?
[10:57] Staffan Truvé: That's an interesting question. I think we see... Let me turn that around and say that the most mature, the most sophisticated companies here are, I would say, the financial industry and banks in particular. I think it's for two reasons. One is that, first of all, they have very valuable assets to protect, of course. They have a long tradition of thinking about security, but maybe most importantly, they are the most regulated. So there has actually been a legislative pressure on them to install systems and have processes and procedures for thinking about these things. Uh-
[11:28] Christina Ellwood: Strikes
[11:29] Staffan Truvé: me that
[11:29] Christina Ellwood: healthcare is a particularly vulnerable area because of the criticality of being shut down. It's not, it, it-
[11:37] Staffan Truvé: It's, you know, you're making a, you're making a great point there, actually, because... And we've also seen that healthcare has become one of the primary targets of ransomware attacks. You know, not only what, b- because of what you said, but also because, of course, they have highly sensitive data. And if you look at the, how the ransomware gangs work, the first generation, you could say, was they attacked your systems, they locked down your files, and they, yeah, and you had to pay to have them unlocked. Now, as companies got better at protecting themselves by having backups so they could actually restart without paying ransom, the ransomware guys took this inventive step and added the second phase, which is when they steal your data and threaten to expose it publicly if you don't pay ransom. So they have a sort of two-stage approach, and in particular for healthcare, because of the sensitivity of the data, they are quite likely, unfortunately, to pay ransom to avoid having the patient data published, for example.
[12:35] Christina Ellwood: Yeah, I can see that. What kind of budget is needed for a company, just not in dollar numbers, but the percentages? If we think about typical mature corporations spend as few percentage points, 3% to 5% on marketing, and we've got all these numbers that have blown up in AI of what people are spending on inference and other AI costs. What percentage of budget would you estimate needs to go against this issue?
[13:08] Staffan Truvé: It's a great question, and again, of course it depends on your risk appetite, if you like. Numbers I've heard being tossed around are that people should think of spending maybe on the order of 5% to 10% of their IT budget on this.
[13:24] Christina Ellwood: Yeah, that's a really good way to think of it, right? 'Cause it, it is likely to be related to the IT budget in the same way that CISOs' budgets are related to the I... They're separate, but they're related to each other in terms of the balance of the distribution of the resources. So that's a good way to think about it. So for the executives who are not the CISO, but are deploying AI in their departments or helping to deploy it across multiple departments in their organizations, what can they do to address or reduce the risk associated with these threats? As they think about designing their infrastructure or using the infrastructure and creating their systems, what can they do to lower the threat posture?
[14:10] Staffan Truvé: Yeah. It's interesting. So when you look at threats to AI, there, there are two aspects of r- looking at this. We'll come back maybe later to the threats, how the threat actors are using AI. But one aspect is, of course, to be very cautious. Top of the line, I would say, to be cautious about what information you are sharing with external AI services. And if I would say any key ingredient here, it's really to ensure that you have your contract set up so with whatever, let's say, foundation AI model provider you're using, that they are not allowed to train on your data in any way. 'Cause one of the most worrying sources for information leakage is that even though they say that it cannot be reversed engineered, very often that you can actually extract information from training data out of the trained models, you know, which could be-
[14:59] Christina Ellwood: It's been proven, yeah, it's been proven many times
[15:02] Staffan Truvé: it can be done. Exactly. So that kind of leakage, I think, is one thing. The other aspects are, I would say, general information security hygiene, what kind of information in general do you upload? What are your policies about which kind of files you can send off for AI analysis? And on the counterpart, of course, the results you get back there. We have a, we, um, ourselves are now, of course, trying to come up with good li- guidelines for AI based, when you do code development using AI, for example, how do you, what kind of procedures do you have in place to ensure that there are no security holes in the code you, you get if you're vibe coding or something like that?
[15:41] Christina Ellwood: That's a very good point. What are, what guidance do you have for people who are doing vibe coding?
[15:46] Staffan Truvé: First of all, I think it's, we're still very early in, like everyone else in this process. I think it's very easy to do small-scale experiments and get blown away by how efficiently you can write code. So this is actually, it's funny because for myself, I'm noting this is a case where I have tons of views from having spent some years at university teaching there and having seen a lot of bad code written by students. So I think the, essentially what happens to a lot of us when we do vibe coding, if we want to be serious about security, we have to spend much more time on code inspection. So you, you move from writing code to actually inspecting the AI's code. But I think the, we need a whole set-- I think in general, the whole software development process has to be changed. One aspect is, of course, how do you do code inspection and automatic checking for weaknesses? I think that is a key area to auto- use AI to actually automate the security checking. Another aspect is when you build these AI-based systems, is how do you take into account that what used to be deterministic processes now are actually probabilistic? You know, we, we all know that LLMs can, can give, have a fair amount of variation in the results they send back to you. How do you test a system which has components like that? And even worse, if you have, if each AI is a black box, when we start building more complex systems, you could have a sequence of, let's say, 10 black boxes sending stuff into each other, and, and the stochasticness of that sort of compounds in that chain. So I think old aspects of software testing and regression tests and so on are lying slate now, right? Now how are we gonna do that in a good, reliable way?
[17:28] Christina Ellwood: Clearly, we need some new approaches and frameworks and so forth. That's actually one of the topics that we're gonna have presented at AI Realized Summit on November 5th, is some of the app foundation framework, so you might find that, that interesting. Maher Hanafi at Betterworks is gonna be presenting a framework.
[17:46] Staffan Truvé: Nice. And- And as I mentioned, AI, AI, I would say AI mentioned AI best te- testing. So one example is, so OpenAI, for example, have developed a, a tool for finding vulnerabilities in code. I think there's, that's another area where we're, both they and the community in general have to find new ways of automating that testing.
[18:06] Christina Ellwood: Are you collaborating in the community in a formal way? Have a community of your own or are participating maybe in a standards group or some other kind of formal group?
[18:16] Staffan Truvé: Not in a formal group. We of course have very close collaboration with our customers. Also collaborating on the academic side. So one example is we're, we have, have had a couple of projects with Royal Institute of Technology in Stockholm, where we're looking at attack simulations, or I think we, we should start call it AI-based, uh, red teaming instead maybe. But essentially we're doing something there. So we're combining- Their frameworks for automated attack simulations with our threat intelligence. So what happens if you take the information we have, which as I said, is exactly what a criminal could get, you know, if he's good enough, and then you use that in an attack simulation and see how much more efficient are they if they get access to that extra information.
[19:01] Christina Ellwood: Yeah, that's really interesting. And then will that lead to a paper, or what's the output that an executive could-
[19:06] Staffan Truvé: We have a couple of academic papers published on that. We should probably do some more popular science work around that. We're also now start, we're actually now form, more practically, we're also starting to collaborate with some, some companies in this space to see how we can jointly sort of raise the bar.
[19:22] Christina Ellwood: In the technology sector, that's where a lot of the best practices come from, is through collaboration between companies that are stakeholders in the same space. So I think that's probably a good outcome. We do it through standards bodies too, but it's more in, in these cutting edge areas where we need to move quickly, corporate collaborations seem to be the more effective approach.
[19:44] Staffan Truvé: I was gonna say, and that's the problem with standards, is that when they're set, it's already way too late.
[19:49] Christina Ellwood: Yeah, for sure.
[19:50] Staffan Truvé: There, there is actually a great example of that right now. One of the hottest trends, I guess, in AI now is building MCP tools, you know, the Model Context Protocol for giving AIs access to other external sources. And that's a great example of a standard which was developed quite quickly, originally with, by Anthropic, but now being adopted by many others. And it's clearly not up to snuff when it comes to questions like authentication and security and so on. So again, sadly, it's the standard where that has to be thrown in a, a bit as an afterthought.
[20:23] Christina Ellwood: Yeah. Un- unfortunately, we seem to be very resistant of hum- as humans to build security in first. It's, uh, somehow anathema to the way the human brain works.
[20:33] Staffan Truvé: Yeah, that, that new shiny feature is so much more interesting to get out there. So
[20:37] Christina Ellwood: much more, yeah. For sure. You were gonna talk a little bit about the AI risks and threats and so forth, and we've deviated. Maybe you wanna get to that?
[20:45] Staffan Truvé: Yeah, yeah, I was gonna ... No, exactly. So we are, of course, tracking very closely what the threat actors are doing in that space, and I think it's, a good way to think of it is that they are, there are two kinds of AI-enabled attacks. I sometimes think about them as the attacks on our brains and the attacks on our systems. So the attacks on our brains are all the kinds of influence operations, phishing attacks, deep fakes being used in scams and things like that. And, of course, no doubt that's where they are at full speed now, using AI to, to, in a very clever and devious way, come up with new attack metrics and new attack vectors. So we're seeing that. We're developing on our side the countermeasures. There are things like we now have in our platform means of detecting AI-generated images. So if you see, if you detect your CEO being on a video call or something like that, you want to have a little red light start flashing if it's an AI-generated image. And similarly, we have other means for detecting those kinds of attacks. Uh, on the technical side, that's things like, for example, doing reconnaissance, so trying to find the weaknesses using AI. We did a fun thing. We did a, an evil hackathon here a while back in the company, where we let our best people spend a week trying to figure out how to use AI for bad. And one, I think, pretty intriguing example was we... Some guys, they took essentially the brochures from a company. They had built a new fancy factory and of course they wanted to brag about it and posted pictures of it, and we just sent those pictures to an AI and asked it to identify what systems were visible in this. So this was a factory, so there was various kinds of control electronics in there, and it gave us a great listing here. Here's the network controller, these other things. Here's all the PLCs, here's all the, the PLCs that are controlling the assembly line. It, oh, wow. Ex- ex- exactly. And then from there it's easy to find what weaknesses are existing in those.
[22:44] Christina Ellwood: Interesting.
[22:46] Staffan Truvé: That's something we are-
[22:47] Christina Ellwood: Back to your computer vision days, right? Where-
[22:50] Staffan Truvé: Yeah, exact- ex- exactly. And fortunately, we have not seen that in the wild. Uh, we don't, I don't think we, we have any verified case where a threat actor has done that. Now I told them, so now, now I have to rush.
[23:00] Christina Ellwood: Yeah, I was gonna say, you just caused the next problem.
[23:02] Staffan Truvé: Yeah. The other aspect, which is, has been a very hot topic for a while in academic research, is really to have AI-generated malware. You know, malware which can adapt to its surroundings, which can be polymorphic, so it changes shape so that it becomes undetectable by classic, uh, malware detection or virus detection software. Again, there's some very good research on that, but we have not seen any, so far, any validated cases where it's ha- where it has been used in the wild. It might be- Yeah.
[23:34] Christina Ellwood: Would you think that would be detectable using behavioral m- methods? Like, you can detect whether I'm really the person who logged in by looking at my biometrics and my historical patterns and so forth, that you say, "Agents don't, pr- proper agents don't behave this way. This is an agent behaving in an anomalous fashion, so we're gonna treat it as a, as an actor"? You do that in DNS, for example- Yeah ... where looking to detect a nefarious attack on the net, uh, uh, i- inside the perimeter is they look at what they're asking to access, and they can tell from all the other people who have ever been on their, on their network using those resources, they can tell this one is doing something no one else has ever done or doing in a way that's really not normal, and they're able to then shut it down. Is that an ap- approach that would be used for these type of poly agents?
[24:30] Staffan Truvé: It's could be an approach, but it's an interesting, again, it's an interesting arms race question because of course the bots, the malicious bots will get better at imitating human behavior. But you can imagine, so they will be, of course, think of all the keyloggers which the criminals have installed worldwide, which are now recording your typing pattern. It's gonna learn exactly how you use your mouse and keyboard. And of course, they will be building models which can mimic human behavior in that sense. So we have to then be-
[25:00] Christina Ellwood: Voice patterns, right? They can mimic your voice patterns to the point where you can't even tell the difference between your own voice.
[25:06] Staffan Truvé: Yeah, it's the same with what we discussed before, you know, about we can now detect some AI-generated images, but of course, the AI, AI generation models are getting better, so th- we have to train constantly, find new ways of detecting it. So clearly an arms race. I think, sadly, if you look at it longer term, people like to talk about the defender's dilemma, the fact that if you're a defender, you need to succeed every time, whereas if you're an attacker, you only have to succeed once. So they are against us on the good side there. I think in the long run, I think we need to rethink some things in terms of security. I think in terms of having better means for authentication content, of course, we all know that we're beyond the age when passwords are a useful means of protection, and you have to have multi-factor, maybe triple, four-factor authentication. So, like, I think we're gonna have to raise the bar in those kind of ways to have a chance to stand up against the clever strategic attackers.
[26:08] Christina Ellwood: What kind of timeline do you think we're on here, that today's threats that you're able to detect and help companies to thwart today, how long do you think it'll take before you're needing to do a different, take a different approach because of the morphing that's going on in the attacks or the vulnerabilities?
[26:30] Staffan Truvé: We probably should be doing it right now, judging from how successful they are. No, I think it's quite urgent. I think figuring out the next generation of protection is very important. Totally. And, yeah, I think my personal feeling is that the fact that the threat actors have not moved on to, let's call it the next generation of AI-supported attacks, is that they are successful enough using the, today's methods, so they haven't been forced to do that. But the technology is, of course, available to them just as much to u- as to us, so there's nothing else than motivation stopping them, really.
[27:05] Christina Ellwood: As I think about agents, I see that as a whole new level of risk. Have you tried this new OpenAI Atlas browser, for example?
[27:15] Staffan Truvé: I have. Uh-
[27:18] Christina Ellwood: Scary piece of software.
[27:21] Staffan Truvé: It's, it's fascinating scary when you use it, right? The- Yeah ... of course, it's extremely handy in many ways, but to think- thinking of the power which gets into that browser now that it sees every single thing you do. Again, back to the point about learning how to do. Just wait until there's, there's a malicious version of that which gets spread around. That's gonna know everything about you.
[27:42] Christina Ellwood: Yeah, I agree. And in fact, it was interesting because I, I immediately recognized the vulnerability and started to ask what people were doing to, to try and isolate the vulnerability, and someone had suggested running it in a docker. Mm. And that's a great idea in terms of keeping it from looking at your data on your laptop or whatever, but the moment you log into something, you've opened the door to whatever you just logged into. So I'm not sure there's a safe way to use it.
[28:11] Staffan Truvé: No, I was gonna say, I, I will definitely not be logging into my bank using Atlas, for example. Yeah.
[28:17] Christina Ellwood: Yeah, for sure. For sure. Yeah. Uh, so that, that... I just, I wanna sort of surface some things that individually we should be thinking about, because companies are made up of people, and a lot of the risks that get brought into a company are brought in inadvertently by individuals, right? Either because they click on the phishing email or because they bring in a piece of unsecured technology or whatever. What can we as individuals and executives do f- to protect our companies and to communicate to our teams how to protect our companies?
[28:51] Staffan Truvé: Um, to be boring, I can say what every security person says, is to start by training. To give people training on how to think about risks and, I would say, particularly how to think about information leakage. It's, since that can be the door opener to so many things, not only direct access to systems, but also a means for creating very devious phishing emails, spear phishing directed towards a certain individual. Mm-hmm. So I think that, that-
[29:15] Christina Ellwood: Fair ...
[29:16] Staffan Truvé: yeah, I, I think that's, that is the most important thing actually. And then of course, the second step then re- related to that is to be even more careful about what tools to allow. I'm a very libertarian person. I liked people to be able to run whatever they want on their laptops, but clearly on a corporate laptop today you have to be much more careful about which tools you let people use because, you know, every single tool is gonna be connecting out to some server sitting on the internet, and the whole supply chain there with vulnerabilities really opens up very troubling opportunities for the threat actors.
[29:54] Christina Ellwood: Totally. This has been a very f- interesting conversation. For the listeners, what key takeaways do you want them to leave our conversation with today?
[30:06] Staffan Truvé: Yeah, that's always a difficult question. No, I think I'm a very tech optimist person in general. I think there's unprecedented opportunities to use AI in, in new ways now. But I think given that it's very easy to become enthusiastic, I think trying to play the devil's advocate and carefully think about what risks come with every opportunity is a good mindset to have.
[30:31] Christina Ellwood: Okay. And what resources do you recommend to listeners who wanna learn more?
[30:35] Staffan Truvé: I have to promote a couple of resources which are actually related to us at Recorded Future. We have two, I would say. There is... We publish an online, uh, newsletter called The Record, which is a great source both for geopolitical and cyber threats. Even better, I would say, is the podcast called Click Here, with, which is, I would, which is very high up on the ranks now, but it's the best place to get interesting aspects. And what we do there is that we not only talk to the defenders, we are actually quite good at getting the dark side to open up and come and talk to us about how they're operating, how they're thinking, and so on. So if you wanna know both sides, that's a great resource, actually.
[31:13] Christina Ellwood: Okay, and it's called again?
[31:15] Staffan Truvé: Click Here, yeah. Okay. Downloadable where podcasts are available.
[31:20] Christina Ellwood: All right, great. I love it. So in the AI revolution, what is the leadership skill that you find most valuable in your role?
[31:33] Staffan Truvé: And I think, uh, curiousness, if I think about it. To be curious to explore these new opportunities, but then also, as I said before, with a little bit of a skeptic mind as well. But I think curious is the most important, to be willing and daring to try out the new opportunities. Maybe be careful and do it in a protected, sandboxed environment, but don't be scared to try things.
[31:57] Christina Ellwood: Okay. Staffan Truvé, it has been such a pleasure to talk with you today. Thank you for joining us on the AI Realized Podcast. Thank you.