Bring the AI to Your Data, Not Your Data to the Cloud

DataSecurity
Written By Christina Ellwood

Episode Summary

Speaking in November 2024, Mark Heynen, chief product officer and co-founder of Knapsack, argues that the data problem in enterprise AI is architectural rather than procedural. Regulated firms found the best way to use large language models was to upload their data into an LLM cloud, and their regulatory status would not let them, so the useful move is to reverse the direction: bring the AI to the data instead of the data to the AI. He says there is already enough compute on a desktop or a smartphone to run smaller language models locally, which makes that reversal practical rather than theoretical. From there he treats compliance as a by-product, because software that runs on your own device inherits the security policies already on it. He is equally direct about what he thinks goes wrong: one of the biggest risks he sees is a company pulled into an expensive deployment whose return nobody can demonstrate.

Key takeaways

  • The specific bind he watched those firms get into is the one the whole episode answers. The best way to use the technology was to upload a lot of data into an LLM cloud, whether ChatGPT or Gemini or Claude, and their regulatory status simply did not allow it

  • His second reason for the bind is uncertainty rather than rules, and he says it has not resolved. It was unclear and still remains unclear to most people what happens to data once it is uploaded: there is always the risk of it being used for training even when people say they will not, and the risk of leakage

  • The number that made it urgent for them was about behavior, not about technology. He said in November 2024 that a Microsoft survey he names as the Work Trend Survey put 78 percent of people using consumer tools at work, from a pool of thirty-one thousand respondents across thirty-one countries

  • What he draws from that number is a mismatch, not an adoption win. The majority of those people are not using a tool fit for purpose, and in finance and healthcare he says they are putting their jobs at risk by doing it

  • He is candid that the rule loses to the tool, which is why he thinks a policy is not enough. People are often not allowed to upload data to a tool under their own AI policies, and the temptation is so great considering how powerful the technology is

  • A solution already existed when he started, and his objection to it was the price. He said in November 2024 that Microsoft offered it through OpenAI Azure service agreements at about one million dollars a year on average, so there was an expensive answer but not a widely accessible one

  • His diagnosis is the sentence the rest of the episode hangs on, and it is about architecture rather than policy. We should not be bringing the data to the AI, we should be bringing the AI to the data, because there is no reason people should round trip their data somewhere to get insights on it back

  • What makes that practical rather than theoretical is where the compute already is. There is plenty of compute on the edge, on your desktop and smartphone, smaller language models are driving more efficiency, and he says that talking to Qualcomm and looking at projections pointed to more GPU capability arriving on phones and laptops

  • The product shape follows from the diagnosis, and it starts small on purpose. You download an app to your computer and it runs across your data; cloud data still works because you download and embed it locally; the enterprise version is a server behind the firewall, without having to trust anyone, including us

  • He extends that to a claim about his own company, which is the strongest version of the argument. In their architecture they do not have access to the data at all, and he takes the principle from crypto: there is no reason why the software should have access to the value

  • Asked to lay out the landscape, he answers chronologically rather than by category. Large language models came first and cloud services made them accessible; small language models followed through advances in quantization and pruning, and he credits those same advances for the Apple Intelligence announcements

  • On agents he is the opposite of bullish, and he puts a number on it. He said in November 2024 that agents were maybe two or three years out from being very useful for mass market, because running autonomously without a human in the loop raises questions about accountability and about confidence that the job gets done well

  • What he offers instead is a smaller step he thinks is available now. Automations are the same capability with a human kept in the loop: the computer reads the data, produces insights and can even take some actions, and a person reviews before anything goes out

  • His worked example is a physician doing a pre-authorization, and the claim he makes for it is modest. The work runs and is produced automatically, the physician reviews it and sends it off, and he says it could hopefully shave a day off the work week

  • Asked which model the product runs, he says open source, and gives a commercial reason rather than a technical one. Fine-tuning open source models is the most cost-effective way to get to market, and he wants to be the prime beneficiary of the innovation happening in that community

  • He names the obligations rather than gesturing at them. Confidentiality commitments to partners, certifications including CCPA, GDPR, HIPAA and SOC 2, and in finance a books and records requirement under the SEC and the FTC that means everything has to be logged

  • On compliance his argument is that local execution makes it a by-product rather than a project. Running everything locally on your device leaves you in full control of where the data goes, and because the software adopts the security policies already on the device, you are immediately compliant with the obligations you already carry

  • The risk he puts first is commercial, not technical, and he names who is selling it. One of the biggest risks companies are grappling with, he says, is being pulled into a high-cost deployment that does not have ROI, and he points at large deployments being pitched by Accenture and Microsoft

  • His evidence for that is one report and he hedges the attribution twice. He said in November 2024 that a Morgan Stanley report carried an interview with a pharmaceutical executive, he thinks the CTO of Bristol Myers Squibb, who said a Microsoft deployment they had paid for delivered extremely limited ROI and produced PowerPoints on the level of a middle schooler

  • Asked what instant private automations actually means, he answers it as three separate claims. Instant is that it works as soon as you connect it to your data; private is that the embedding and the RAG pipeline both run on the device; and the application itself runs there, so there are no files hosted with them

  • His argument for automations over chat is about friction rather than capability. The chat box is the dominant UI and a genuinely accessible one, but you have to know how to write something and know what to write, and that is a block on using a technology that is available whenever your computer is running

  • His closing ask is narrower than it sounds and it is aimed at permission. We are at the very beginning of this AI world, there is a way to use AI at work without violating whatever internal or external agreements you have, and he says we should work together to find out how

About Mark Heynen

Mark Heynen is chief product officer and co-founder of Knapsack, which builds automations that run on your own device rather than in a vendor cloud. The host puts its tagline to him on this episode as instant private automations, and he takes the three words apart one at a time. He co-founded PayJoy in 2015, where he worked on access to smartphones and smartphone finance, and before that he worked on Android at Google. He describes access problems as his bread and butter, and traces Knapsack to a problem he and his co-founder each ran into separately: regulated finance firms that could see what large language models could do for them and were not permitted to upload the data those models needed. Everything he says here about Knapsack, its architecture and its certifications describes the company as it was in November 2024, when the product was in private beta.

 

In this episode

00:28 Welcome, and who Mark Heynen is
01:04 The question: what inspired Knapsack
01:08 A lived experience both founders had
01:47 The bind: upload the data, or comply
02:11 What happens to data once it is uploaded
02:34 What CISOs told him about AI and data security
02:54 The 78 percent using consumer tools at work
03:15 Thirty-one thousand respondents, and a tool not fit for purpose
03:31 The policy loses to the temptation
03:48 A million dollars a year for the expensive answer
04:15 The diagnosis: bring the AI to the data
04:32 Compute on the edge, and what the projections showed
04:52 Download an app, then upgrade to a server behind the firewall
05:22 Software that has no access to the value
06:19 The question: how the pieces fit together
06:49 Large models, cloud services, then quantization and pruning
07:13 Small models catch up, and Apple Intelligence
07:37 Agents, accountability, and the step before them
08:13 Automations with a human in the loop
08:30 The physician who reviews and sends
08:57 The question: which model does it run
09:17 Fine-tuning open source, and why
09:57 The question: what else does compliance cover
11:02 The compliance problem stated plainly
11:28 CCPA, GDPR, HIPAA and SOC 2
11:51 Books and records, and running it locally
12:18 Inheriting the policies already on the device
12:53 The question: what other risks
12:58 A high-cost deployment with no ROI
13:24 The Morgan Stanley report, and the PowerPoints
14:09 The setup cost before any value appears
14:33 The question: what instant private automations means
14:45 What instant, private and automations each mean
15:40 The friction in the chat box
16:49 Monitoring on a schedule, and what it needs from you
17:22 The question: what to read and try
17:44 The white paper, and knapsack.ai
19:38 The question: the closing takeaway
19:44 Using AI at work without breaking your agreements

In Mark’s words

“the majority of those people are actually not using a tool fit for purpose”

Mark Heynen   (03:15)

“We should not be bringing the data to the AI. We should be bringing the AI to the data.”

Mark Heynen   (04:15)

“There’s no reason why the software should have access to the value.”

Mark Heynen   (05:22)

“there’s probably a step before we get to agents, which is what we call automations”

Mark Heynen   (07:37)

“having everything run locally on your device, so you have full control of the data and where it goes”

Mark Heynen   (11:51)

“I think there’s an inherent friction in the chat box.”

Mark Heynen   (15:40)

“there is a way to actually use AI at work without violating whatever internal agreements or external agreements you have”

Mark Heynen   (19:44)

 

Resources

Named on air

  • PayJoy: The company he co-founded in 2015, named at 01:14 and again at 03:48 as where he worked on access to smartphones and smartphone finance

  • Android at Google: Named at 03:48 as the earlier access problem he worked on, which he describes as fundamentally access to smartphone technology

  • Lambda Labs: Where his co-founder was working with data and LLMs before they started the company, named at 01:14

  • Qualcomm: Named at 04:32, as the company they talked to while looking at projections for GPU capability on smartphones and laptops

  • Groq: Named at 15:13 as where the optional cloud model runs when ephemeral computing is used

  • LM Studio: Named at 18:06 as a local app he suggests trying to understand how local models work, with the caveat that it does not generally connect to your data

  • The Microsoft Work Trend Survey: The name he gives at 02:54 for the source of the 78 percent figure, from a pool of thirty-one thousand respondents across thirty-one countries

  • A Verizon data breach report: His attribution at 10:45 for the figure that 36 percent of breaches stem from user error with software tools

Ideas and terms discussed

  • Bringing the AI to the data: His own phrase at 04:15 for reversing the usual direction, so the model travels to the data instead of the data traveling to the model

  • The edge: His word at 04:32 for the compute already sitting on a desktop or a smartphone, as opposed to a cloud server

  • Small language models: Models made small enough to run on a device, which he says reached similar functionality to large ones through quantization and pruning

  • Automations: His term at 07:37 for the step before agents: the same capability with a person reviewing before anything goes out

  • Ephemeral computing: The optional cloud path he describes at 15:13, where an answer is produced and not saved

  • RAG pipeline: Named at 14:50 as something they built themselves, to select the right data on the device for the model to work on

Related AI Realized episodes and events

 

Frequently Asked Questions

  • Regulated companies cannot upload their data to a hosted model because their regulatory status does not permit it, whatever the tool would do for them. Mark Heynen of Knapsack, speaking in November 2024, describes watching finance firms hit exactly that wall: the best way to use the technology was to load a lot of data into an LLM cloud, and they were not allowed to. His second reason is uncertainty rather than rules, and he says it has not been resolved: it remains unclear to most people what happens to data once it is uploaded, including whether it is used for training after an assurance that it will not be.

    Transcript 01:47 to 02:34


  • Bringing the AI to the data means running the model where the data already sits instead of uploading the data to where the model runs. Mark Heynen of Knapsack calls the alternative a round trip and treats it as an architectural mistake rather than a procedural one: there is no reason people should have to send data somewhere to get insights on it back. What makes the reversal practical is that there is already enough compute on a desktop or a smartphone to run a smaller language model there, so the model can travel instead of the data.

    Transcript 04:15 to 05:22


  • Yes, and the reason is a combination of hardware that is already in the device and models that have been made smaller. Mark Heynen of Knapsack said in November 2024 that there was plenty of compute on the edge, on a desktop and a smartphone, and that advances in quantization and pruning had got small language models close to the functionality of large ones. He credits those same advances for the Apple Intelligence announcements of that period, and says a conversation with Qualcomm pointed to more GPU capability arriving on phones and laptops.

    Transcript 04:32 to 07:13


  • An automation keeps a person in the loop and an agent does not, and that is the whole distinction. Mark Heynen of Knapsack describes an automation as a computer reading data, producing insights and even taking some actions, with a human reviewing before anything goes out, and an agent as an AI trusted to run autonomously without that review. He said in November 2024 that agents were maybe two or three years out from being useful for mass market, because autonomy raises questions about accountability and about confidence that the work is any good.

    Transcript 07:13 to 08:30


  • Running AI locally helps with compliance because software on your own device inherits the security policies already applied to that device. Mark Heynen of Knapsack frames the underlying problem as accidental rather than deliberate: the question a company has to answer is how to be sure the data on its employees’ computers is not uploaded by mistake to somewhere that is not compliant. He points at confidentiality obligations to partners, at certifications including CCPA, GDPR, HIPAA and SOC 2, and at the books and records requirement finance firms carry under the SEC and the FTC.

    Transcript 11:02 to 12:18


  • About 78 percent, on the survey cited in this November 2024 conversation. That is the figure Mark Heynen of Knapsack gives for people using consumer tools at work, and he attributes it to a Microsoft survey he names as the Work Trend Survey, from a pool of thirty-one thousand respondents across thirty-one countries. His point in raising it is not adoption but mismatch: the majority of those people are not using a tool fit for purpose, and in finance and healthcare he says they are putting their jobs at risk by doing it.

    Transcript 02:54 to 03:31


  • The risk named first here is spending heavily on a deployment whose return nobody can demonstrate afterwards. Mark Heynen of Knapsack calls it one of the biggest risks companies are grappling with, a risk of sucking the entire company into a high-cost deployment that does not have ROI, and he names Accenture and Microsoft as sources of the deployments being pitched. His supporting example is a Morgan Stanley report he describes from November 2024, carrying an interview with a pharmaceutical executive he believes was a chief technology officer, who said the deployment they had paid for produced extremely limited return.

    Transcript 12:58 to 14:09


  • People underuse AI because of friction in the chat box rather than any limit in the technology. Mark Heynen of Knapsack calls the chat box the dominant UI and a genuinely accessible one that everyone understands intuitively, and then names its cost: you have to know how to write something and know what to write. His alternative is an automation running in the background on work you already know needs doing, such as watching an inbox for a message that needs an answer today.

    Transcript 15:40 to 16:49


 

  • [00:28] Christina Ellwood: Welcome to the next episode of AI Realized, the podcast for enterprise executives leading AI deployments. From addressing security, data, and operations challenges to managing organizational and management changes, AI deployments present the opportunity to redesign our organizations from the inside out. In this episode, we’ll talk about what’s new and now. I’m your host, Christina Ellwood, and we’re talking today with Mark Heynen, chief product officer and co-founder of Knapsack. Mark, welcome to AI Realized.

    [01:02] Mark Heynen: Happy to be here.

    [01:04] Christina Ellwood: You’re a serial entrepreneur. What inspired you to start Knapsack?

    [01:08] Mark Heynen: So, Knapsack was really created out of a lived experience that me and my co-founder both had. You know, we had been working in startups. Both of us met each other at PayJoy, which I co-founded in twenty fifteen. He was one of the first four engineers at PayJoy. And he was at Lambda Labs and interacting with a lot of data with LLMs and also seeing how companies were adopting AI. I was actually at a, you know, within the fintech environment, seeing how regulated finance companies were reacting to ChatGPT being available to this very powerful LLM technology. And seeing them really struggle because the best way to use this technology for them was to upload a lot of data into LLM clouds, whether it was ChatGPT or Gemini or Claude or other things. But ultimately, they were not allowed to do that. Their regulatory status does not allow them to do that. They were very concerned about private information, sensitive information leaking. It’s still-- It was unclear and still remains unclear to most people what might happen to that data once it’s uploaded. You know, ultimately, there’s always the risk of it being used for training, even when people say they won’t use it for training. There is the risk of data leakage. We’ve recently had a couple of high-profile leaks, AT&T and other leaks, where very sensitive information was leaked out. And different CISOs I’ve talked to have indicated that ultimately, data security is now becoming a really challenging area because of AI, because now people are uploading a lot of data up into AI systems. So we had this intuition that this was a significant problem that was not being addressed by any of the people starting companies. A lot of people starting companies were just sidestepping this problem and starting on top of ChatGPT or on top of other APIs. And then we saw the Microsoft Work Trend Survey, which said that seventy-eight percent of people were actually using consumer tools at work. And that really was a wake-up call for us because that’s a big number. I mean, this is actually a pool of thirty-one thousand respondents, thirty-one countries. You know, the number of people using it at work is very significant. And so the majority of those people are actually not using a tool fit for purpose. And in many cases, in the case of, you know, finance or, as we learned, healthcare, they’re actually putting their jobs at risk. They’re often not allowed to be uploading data to a tool in their AI policies But the temptation to use them is so great considering how powerful the technology is. So we realized this is a very significant issue, and no one’s found an elegant solution to it. I mean, there’s an expensive solution, which is you do a fully on-prem private cloud deployment. Microsoft offers that with OpenAI Azure service agreements. Those are one million dollars per year on average. So yes, there is an expensive solution, but there’s not a widespread accessible solution. And I love those access problems. I mean, those access problems are my bread and butter. That’s what I did at PayJoy, where we provided access to smartphones and smartphone finance, and, you know, I loved even back at Google when I worked on Android, which is really fundamentally access to smartphone technology. So Cooper and I thought about it, and we realized, well, actually, the fundamental problem is in the architecture. We should not be bringing the data to the AI. We should be bringing the AI to the data. There’s no reason in the modern day and age people should round trip and have to upload their data somewhere for it to be processed to get insights on the data back. There’s actually plenty of compute available on the edge, as we call it, on your desktop and smartphone. There’s also more efficiency being driven by these new smaller language models, so it’s possible to run it. And looking forward, as we talked to Qualcomm and looked at projections, more and more GPU capability will be available on smartphones and laptops going forward. So we realized really the best solution to this is bringing AI to the data and allowing people to begin with download an app to their computer and have that run across their data. That means you can still use cloud data. You just download the cloud data and embed it on your computer. And then eventually people can upgrade to a server-side solution for the enterprise and allow people to access and embed their corporate data, their enterprise data, but again, behind the firewall without having to trust anyone, including us. So in our architecture, we do not have access to the data. You know, we’re just providing tools that allow you to have access. So, you know, I spent a little bit of time in crypto. One really interesting part of that is this concept of decentralization. There’s no reason why the software should have access to the value. You can write software that can process the value without having access to it, and that’s really what we’re doing here. Similar to the way WhatsApp works, where they don’t have access to the messages. They provide the software to allow you to transmit the messages. So it’s a similar concept, and something we’ve heard was really interesting for the people we interviewed. In all my companies, I interview, you know, hundreds of people before we write a single line of code, and we try to follow the same philosophy here. Did a lot of interviews and realized this is something that people really wanted, and so we are launching an application to allow people to do that.

    [06:14] Christina Ellwood: Wow, there’s so much to pick up on in what you just said, Mark. I’m not sure where I want to start. But let’s go back to a more basic question. There’s a landscape of AI technologies changing very rapidly. You’re building on the edge. There are solutions in the cloud. There are large language models, small language models, agents, and compound systems. Can you give us a very simple description of how all those pieces fit today, and then where you are, uh, focused on with your technology?

    [06:49] Mark Heynen: Yeah, I would look at this in a chronological sequence. So we started with large language models and, you know, through recent developments, we’ve been able to make those large language models accessible to people through cloud services. Now we’ve seen small language models be available through advances in quantization and advances in pruning and other advances. We’ve been able to actually get similar functionality to the large language models in small language models, and that’s very exciting. A lot of that-- those advances are what’s behind Apple Intelligence announcements we’ve heard recently. Now, agents, I think, are a next iteration. I would say agents are maybe two or three years out in terms of being very useful for mass market. With agents, you’re really trusting an AI to run autonomously to do things for you without a human in the loop, and I think that raises a lot of questions in terms of accountability and also in terms of, you know, the confidence level that they’re gonna do a great job. And so one thing we realized is that there’s probably a step before we get to agents, which is what we call automations, which is taking this experience we have that we can have a computer actually think about data, read data, produce interesting insights on the data, and even take some actions. But we can do that with a human in the loop to begin with. We can actually, you know, share insights automatically, for example, on your next upcoming meeting, and you can then act on those insights yourself with a human in the loop. You can imagine a physician doing a pre-authorization for a procedure. They automatically have that run and be produced, but ultimately they’re the ones reviewing it and then sending it off. It makes them a lot more productive. We can hopefully shave a day off of the work week. But ultimately, the human’s in the loop. I think agents will come after that, and we’re really excited about that future where we then go to the next step and have people actually do what we call write automations, where the automations are doing things on their own as well.

    [08:57] Christina Ellwood: Well, physicians are very familiar with that these days too, because they do a lot of their note-taking using AI, and then they have to check that note-taking.

    [08:57] Mark Heynen: That’s right.

    [08:57] Christina Ellwood: So that model seems well-suited to the way they’re operating even today. So is Knapsack using a open source small language model? Or what, what kind of a model is your-- or the-- does the user get to choose?

    [09:17] Mark Heynen: So right now we’re fine-tuning open source models. We find that’s the most cost-effective way to get to market, and we want to be the prime beneficiary of a lot of the innovation happening in the open source community. We’re seeing so many new things come out every day, and that’s where we’re starting. I think in the near future, we expect this proliferation of small language models, and we expect us to be able to choose the best small language model for different prompts or different automations. And we really expect to be the switch or, you know, the best suited software platform to figure out which small language model to use for which task with which data.

    [09:57] Christina Ellwood: That makes a lot of sense to me. So besides the problems that you’ve already addressed related to data security and management, you’re hinting at compliance issues. Are there other compliance issues that companies need to deal with related to AI that you help with?

    [10:16] Mark Heynen: There are. There are quite a few. And so we’re releasing a white paper shortly that’s going to be talking through these compliance issues and what we are helping address. You know, I think if I think about it at the highest possible level, really we have this situation where data is the new oil, and we have data everywhere, and we’re now going through a stage in humanity really where we’re trying to wrap our hands around all the data that’s out there and what kind of rights people should have in relation to their data. And for businesses, this really comes down to the fact that if someone doesn’t trust the business with their data, they’re gonna stop doing business with that data. You know, we’ve seen, you know, civic reports come out about this, and then Verizon did a data breach report that shows that 36% of breaches actually stem from user errors with using software tools. So it’s not always intentional. Sometimes you can upload data somewhere and some cis admin or someone at the company actually doesn’t do something right, or even the person doesn’t know what they’re doing with their tools. And so really what we are doing is solving this fundamental compliance issue about how do I make sure that as a company, the data on my employees’ computers is not being uploaded by mistake to someplace that is not compliant. And that ensures that, A, they’re compliant with their obligations to other partners where they have confidentiality obligations. It also makes sure they’re compliant with the things they’ve signed up for in terms of certification, whether it’s the CCPA or GDPR or HIPAA or, you know, SOC 2. There are all sorts of other policies a lot of these companies have signed up for. And then in case of finance, they go one step further where the SEC and the FTC regulates them, so they may have a books and records requirement where they need everything they do logged, and that also requires that, you know, everything is done in a certain way. And so ultimately what we are doing is making sure people are compliant by having everything run locally on your device, so you have full control of the data and where it goes. And because it actually adopts whatever security policies you already have on the device, you are immediately compliant with all those things. And then on top of that, we are compliant. So we are undergoing and just receiving HIPAA certification, so the application is HIPAA compliant, and we’re going through the process of getting other certifications as well. And so you don’t have to worry about it. Now, I think with other startups, they often have clouds where they claim some level of compliance. But in the end, do you really wanna take the gamble? Something might happen if that data leaks, and that could actually be disastrous for your business

    [12:53] Christina Ellwood: What other risks do enterprises face when adopting AI?

    [12:58] Mark Heynen: I think one of the biggest risks that companies are grappling with right now is a risk of sucking the entire company into a high-cost deployment that does not have ROI. And I think we’ve seen this more recently with some of these large deployments that are being pitched, whether it’s by Accenture or Microsoft or even other newer companies like Hebia. These are extremely expensive deployments, and ultimately, there’s a learning process that organizations have to go through. So the ROI is unclear to begin with, to be honest. And then we’ve seen... Actually, recently, there was this great report. Morgan Stanley had a report with an interview with a pharmaceutical executive, I think it was the CTO of Bristol Myers Squibb, who was claiming that the Microsoft deployment that they had paid for had extremely limited ROI and produced PowerPoints that were the equivalent of what a middle schooler might put together. And so no one wants to have that be their legacy in a company. And then on top of that, if you’re talking about uploading a whole bunch of data up into a new cloud, you often have to, A, gather that data from multiple clouds, and then you have to have a lawyer get involved to figure out what they can and cannot share. Now, that is extremely high risk that you’re gonna be sucking a lot of time and energy and resources without getting incremental benefit. And so one thing we really focused on is allowing people to download an app instantly and instantly start using it without having to consult with a lot of people because it actually is running completely turnkey and locally. And I think that is probably the biggest sort of meta risk that people are gonna be running into right now.

    [14:33] Christina Ellwood: Great. So your tagline is instant private automations. What does that really mean, and what are the things that companies are automating with your product?

    [14:45] Mark Heynen: So what it means specifically is that people can use it instantly when they connect it to their data. It’s happening privately, so we have a embedding that happens on the device itself, on the computer. We have a RAG pipeline we built ourselves that automatically grabs the right data to enable the LLM to process it, and then we have the application running on the device so that you don’t necessarily need to worry about us hosting any files. So that’s the instant and private component of it. We do have the option to use ephemeral computing, where you can have a cloud LLM on Groq providing you advice, but... And answers, but that is not saved, so it flushes out immediately once you actually upload the data. So that’s the instant and private, and then the automation piece is really the observation that people are really not using AI as much as they probably could and should. I think there’s an inherent friction in the chat box. The chat box is the dominant UI. It’s what ChatGPT has been able to produce, and ultimately other people have copied that, and I think it’s a great accessible UI. Everyone understands it intuitively. At the same time, there’s high friction. You have to know how to write something. You have to know what to write. In the meantime, this technology is available always, and if you’re using your local compute, it’s available whenever your computer’s running. And so there’s no reason why you should be blocked by having to write something in, that you should actually have an automation running in the background doing all the things you might need to do, whether it’s monitoring customer service chats for some kind of issue, monitoring your inbox for an email. One great automation idea someone shared with me is monitoring your inbox for an email with emotional content, so you’re immediately notified if there’s some kind of a landmine in your inbox that you need to get to as soon as possible. We all just think in messaging. We all deal with tons of different inboxes at this point, including Discord and, you know, Slack and your email inboxes, m- sometimes multiple email inboxes. Just having something monitoring that on a regular basis and telling you if there’s something you need to deal with is an excellent automation. That’s just a starting point. And of course, you need to be comfortable connecting all that data for that to work. And so the, the idea of automations is that you can have things running regularly that are making you a lot more productive and giving you a lot more impact in your job.

    [17:10] Christina Ellwood: That’s a great example, and I imagine you could also do that if it was on a server and you had all your employees connected to the server. You could automate things between your employees too. Is that correct?

    [17:21] Mark Heynen: That’s correct, yes.

    [17:22] Christina Ellwood: Great. So what resources do you recommend to listeners who want to learn more about your company, but also about the points that you made around open source and small language models and having your model behind the firewall and some of these really important principles that you’ve covered?

    [17:44] Mark Heynen: So I think one thing I would recommend is to review our white paper. It’ll probably be out by the time this podcast is out, and I think that gives everyone a sense of where we’re coming from. So you can go to knapsack.ai and download that. The second thing I would recommend is to actually just start experimenting with using local LLMs and see, you know, how you like it. Ultimately, we are in private beta, and so do-- go to knapsack.ai and sign up for our private beta and you, you will be able to see what we can do. But we definitely encourage people to download LM Studio or other local apps to understand how those might work, and those do not generally connect with your data, but it gives you a sense of what’s possible. I also really encourage people to start thinking about automation ideas without worrying about where their data might go, and thinking about what they might do with that. I think ultimately there’s a whole new sphere of innovation that will open up if people start, you know, experimenting a little bit with that, and I would encourage people to do that. And I, I imagine that especially if we’re talking on-device, the fact of the matter is we have a lot of, you know, new functionality that opens up to us. We can actually use all of the capabilities of the device, including screen recording, the microphone, et cetera. Because it’s on-device, there’s no latency, so you can actually have in real-time information coming at you. For example, while you’re in a meeting, you can have suggestions coming at you on things you should be bringing up because it’s all happening locally. You don’t have a round trip. You don’t have to worry about latency. You have situations where on-device, while you’re on the go, you can have things happening in the background without having to share your location or other information. So I think thinking through those opportunities is really what the main thing I would recommend people do, because ultimately with our application, you’ll be able to build your own automations and then have a pretty dramatic impact on your life.

    [19:38] Christina Ellwood: Great. So as we wrap up, what would you like our listeners to take away from our conversation today?

    [19:44] Mark Heynen: The first thing to take away is that we’re at the very beginning of this AI world, and that there is a way to actually use AI at work without violating whatever internal agreements or external agreements you have, and that we should work together to find out how to do that. And so that, that’s the main thing I would take away.

    [20:03] Christina Ellwood: Mark Heynen, Chief Product Officer and Co-founder of Knapsack, thank you so much for talking with me today and sharing your really fabulous insights about where we’re going and how we can use AI at work. Thank you.

    [20:19] Mark Heynen: Thank you very much.


 
Listen to other episodes
 
 
 
Join AI Realized
 
 
 
Subscribe to the newsletter
 
 
 
Christina Ellwood
Previous

An AI Committee Needs Every Department and Real Authority
Next

Own Your Own Intelligence Before Your Vendor Learns It