When Restoring the Painting Costs More Than the Painting

Adoption
Written By Christina Ellwood

Episode Summary

Art restoration has been technologically the same for 500 years: damage is filled in by hand, with a brush. Alex Kashkin, then a mechanical engineering graduate student at MIT, brought the first technology to it. Generative neural networks construct the paint that is missing, the result goes onto the original on a reversible ultra-thin polymer film, and he puts the process at almost two orders of magnitude faster than by hand. What that changes is the economics. Restoration is priced by labor, so conservators who contacted him say they reject almost 80 percent of potential clients on cost, and for a lot of artworks the repair costs more than the artwork itself. The rest of the conversation is about restraint. Asked whether he uses machine learning besides the generative step, he says not for making masks at this point, because the science of human color perception is unfinished. Conservators, he says, have really appreciated exactly that.

Key takeaways

  • Art restoration has been technologically the same for the past 500 years. It is the first thing he says about his own field, and he means it literally rather than as a figure of speech

  • The method is a mask, and it goes onto the painting rather than into it. Generative neural networks construct the fill for the damaged areas, and the result is physically applied to the original with a reversible ultra-thin polymer film. He describes it as the first step in bringing any technology into a field where damage has always been restored by hand, with a brush

  • He puts the gain at almost two orders of magnitude. The restoration process, he says, is almost that much faster than doing the same work by hand, and that is what makes it transformational for conservators and for institutions holding damaged art in need of repair

  • The technique came out of semiconductors. Masks for silicon are made with extremely precise electron lithography and placed precisely enough to align many layers over each other, and he says the same precision constraints, the same tolerance modeling and the same analysis are directly applicable to a painting, differing in spatial scale

  • The idea itself came from the hobby. He had been restoring paintings by hand for around a decade, started because he wanted some cool art, and noticed while traveling up the East Coast that so many museums hold most of their collections in storage where nobody gets to see them

  • Some of what sits in storage is damaged beyond the institution’s means. He says a fair portion of those collections, in some cases up to 10 percent, is damaged art the museum has no means to restore, and that the biggest issue in restoring paintings is the time it takes to inpaint the losses

  • Cost, not capability, is what leaves art unrestored. He says conservators have told him they reject almost 80 percent of potential clients, and that the quotes are high for one reason: the labor it takes to do the restoration

  • For a lot of artworks, restoring it costs more than the artwork itself. That, he says, is why a lot of people, families and institutions are just never going to restore what they hold, and opening up that swath of opportunity is what he hopes commercializing the technique will do

  • Some work is too fragile to be restored at all. He describes paintings sitting in storage, damaged in a way that leaves them beyond restoring, where moving them around to different places is, in his words, a huge bureaucratic nightmare in its own right

  • There is huge demand for facsimiles instead. Reproductions of art otherwise locked in storage let institutions that are based on visitors exhibit something, and let owners have access to their works in multiple places at once. He calls it a really fun opportunity, and was working with a couple of groups in Geneva to make it possible with his masking technique

  • You cannot get this by prompting. He is direct that passing a damaged painting to a conventional image model and typing a prompt asking it to restore the painting will not produce anything near what a conservator would do, because the considerations there are really not intuitive

  • A different art problem sits closer to what AI is already good at: attribution. He describes digital fingerprinting of artworks, taking an image, even from a phone camera, and identifying the textural aspects of a work that really imply authorship

  • Being able to parse that authorship, he says, is a huge technical challenge, but one where AI tools are actually really adept at noticing patterns that the human eye is not really capable of noticing

  • Asked whether machine learning is used anywhere besides the generative step, he says not for making masks at this point. He has tested a number of different strategies and describes what he published as methodical instead

  • The mask process is deliberately hard-coded. He says there is a lot of rigor and basically hard-coded design in the process of generating the mask, because he wants a more thorough understanding and a rigorous application of the decision-making that goes into constructing one

  • The reason is that the science underneath is unfinished. There are candidate metrics for how different two colors look to a person, he says none of them are fully correct, and the ones that are most accurate take an enormous amount of computing to actually realize

  • He is not against training a model for the judgment, but it has to be tied to human vision. On how bad a scratch or a discoloration has to be before it warrants repair, he says yes, we can train models to do that, and then that in the end it is really important to tie it back to the way humans perceive color

About Alex Kashkin

Alex Kashkin is CTO of Luminato, the company commercializing the first new technology to reach art restoration in 500 years. A mechanical engineering graduate student at MIT, whose research is in electron sources for building semiconductors, he had also been restoring paintings by hand for around a decade as a hobby. He put the two together: generative neural networks construct a painting’s missing paint, and the result is applied to the original with a reversible ultra-thin polymer film rather than altering the work itself. The paper made the cover of Nature in June 2025, under his legal name Alex Kachkine, which is the spelling MIT, Nature and the New York Times use. At the time of this recording he was days from leaving MIT to commercialize the work, and the company had not yet been named Luminato.

 

In this episode

00:00 Welcome
00:28 Guest introduction, MIT and electron sources for semiconductors
01:18 Art restoration, technologically the same for 500 years
01:28 Restored by hand with a brush, and generative masks on a removable film
02:03 Almost two orders of magnitude faster than by hand
02:19 The connection between the mask and the work in silicon
02:30 Micro and nano scale patterns embedded in silicon wafers
02:46 Electron lithography, and the same precision constraints on a painting
03:17 Bespoke, niche, and nobody had thought of doing it
03:44 A decade of restoring by hand, and collections kept in storage
04:10 Up to 10 percent damaged, with no means to restore it
04:44 Why can’t we do that to paintings
05:18 Publishing the paper, and the cover of Nature
05:34 Conservators asking to use it in their own practice
05:53 Rejecting almost 80 percent of clients, because of labor
06:16 Buddhist textiles, cave drawings, Italian frescoes
06:59 Leaving MIT on Monday, and raising funds
07:23 Most restored art is high value, and the rest dwarfs it
07:46 When restoration costs more than the artwork itself
08:42 The talk to the San Francisco museums
09:09 Modernizing restoration, and what else conservators want to try
09:45 Work too fragile to be restored, and the trouble with moving it
10:13 Huge demand for facsimiles, and the groups in Geneva
10:42 A field lagging on technology, and ripe for it
11:12 Free, high quality images from cataloging institutions
11:32 Training a model to emulate restoration, and where a prompt falls short
12:01 The dataset that does not exist yet
12:38 Adobe’s research branch and patched inpainting
13:29 Digital fingerprinting, and the textures that imply authorship
14:01 A corpus of art with no known author
14:22 Patterns the human eye is not capable of noticing
14:46 The proof of concept paper, and the flak it drew
15:27 Whether machine learning is used besides the generative step
15:35 Not for making masks at this point
15:52 Hard-coded design, and the unfinished science of human color vision
16:23 Two colors, and no fully correct metric for the difference
17:05 The threshold at which a damage counts
17:38 The algorithm, and why conservators appreciate the rigor
18:14 The New York Times piece
18:24 The Spectrum analysis, and the paper on Nature
18:46 Conservation 21 as the candidate name
19:19 Agreeing with people, and resolving conflict through positivity
19:56 Technology like this takes a bit of a crazy mind
20:08 Wrap-up

In Alex’s words

“For a lot of artworks, the conventional cost of restoration is much larger than the cost of the artwork itself.”

— Alex Kashkin   (07:46)

“Art restoration has been technologically the same for the past 500 years.”

— Alex Kashkin   (01:18)

“They have to reject almost 80% of potential clients because the clients can’t afford the enormous quotes that are offered. And those quotes are high just because of the labor it takes to do the restoration.”

— Alex Kashkin   (05:53)

“There’s a lot of rigor and basically hard-coded design of that process of generating the mask.”

— Alex Kashkin   (15:52)

“Conservators have really appreciated that part of the work because it is so rigorous.”

— Alex Kashkin   (17:38)

“You’re not going to get anything near what a conservator would do.”

— Alex Kashkin   (11:32)

“Art as a field has been so far lagging behind the adoption of tech and everything, that it’s just ripe for having a lot of tech being used to solve challenges that have been there for centuries.”

— Alex Kashkin   (10:42)

“AI tools are actually really adept at noticing patterns that the human eye is not really capable of noticing.”

— Alex Kashkin   (14:22)

 

Resources

Alex Kashkin and the research

The press coverage he names on air

Ideas and terms discussed

  • The digitally constructed mask: The core of the method as he describes it on air. A generative model constructs the paint that is missing, and the result is physically applied to the painting with a reversible ultra-thin polymer film, so the original is covered rather than altered. He puts the process at almost two orders of magnitude faster than restoring by hand. How the mask is physically produced, printed in two aligned layers on a conservation-grade polymer film, is described in the paper and the coverage rather than in this conversation

  • Restoration priced by labor: His economic argument, and the reason the technique matters beyond speed. Quotes are high because restoration is hand work; conservators who contacted him say they reject almost 80 percent of potential clients on cost; and for a lot of artworks the restoration costs more than the artwork

  • Facsimiles: Reproductions of works too fragile to be restored, so that a museum that depends on visitors can exhibit something and an owner can have access in more than one place. He calls moving such pieces between venues a huge bureaucratic nightmare in its own right. An application he did not design for, and one he was pursuing with groups in Geneva

  • Digital fingerprinting: Reading the textural aspects of a work, potentially from a phone camera, to say something about who painted it. He describes a corpus of art whose authorship is unknown, sometimes because several hands worked on one painting, and calls parsing it a huge technical challenge

  • Human color vision as the limit: Why the mask process is hard-coded rather than learned. There are candidate metrics for the perceived difference between two colors, none fully correct, and the most accurate are computationally enormous, so what counts as damage is tied back to human perception by design

  • Conservation 21: The candidate name he gives at 18:46 for the company he was leaving MIT to start. He is explicit that it may not stick, and it did not: the company is now Luminato, where he is CTO

Named on air

  • DALL-E: He names it at 04:10 as the generative image model that was current when he had the idea, and as the example of image models already being applied in patched ways to repair images. Christina picks it up at 12:17 and asks at 12:19 which other models he uses

  • Adobe Research: He names the research teams funded by Adobe’s research branch as doing the best job in the space of patched inpainting, and says their work has been instrumental in making results possible that were not before

  • StreamCore: Christina names it at 15:05 as a startup combining provenance for musical works with detection of playout on the internet, and suggests the ideas may be complementary. The name is hers rather than his, and no company matching it could be confirmed, so nothing is claimed for it here

  • The San Francisco museums: He describes being invited a few weeks before recording to speak to conservators and curators across the Bay Area in one room, and says the modernization of art restoration is very much in vogue among them

 

Frequently Asked Questions

  • Yes. AI can restore a damaged painting by generating the paint that is missing and applying that reconstruction physically to the original. In the method Alex Kashkin published at MIT, generative neural networks build the fill for each damaged area, and the result goes onto the painting as a mask on a reversible ultra-thin polymer film, so the original is covered rather than altered. He puts the process at almost two orders of magnitude faster than filling the same damage in by hand with a brush, which is how restoration has been done for the past 500 years. The paper describing the method made the cover of Nature.

  • AI-assisted painting restoration works in two operations: a generative model constructs the paint that is missing, and that reconstruction is applied physically to the original as a mask on a thin removable film. Alex Kashkin explains the parallel with semiconductors, where masks are made with extremely precise electron lithography and applied precisely enough to align multiple layers over each other. He says the same precision constraints, the same tolerance modeling and the same analysis are directly applicable to a painting, differing in spatial scale, and calls the transfer fairly bespoke and niche, something nobody had really thought of doing before. He had been restoring paintings by hand for around a decade, which is where the problem came from.

  • Restoring a damaged painting is expensive because it is hand work, so the cost is almost entirely labor. Alex Kashkin says conservators have told him they reject almost 80 percent of potential clients, not because the work cannot be done but because those clients cannot afford the quotes, and the quotes are high for that single reason. The consequence is economic: for a lot of artworks the conventional cost of restoration is larger than the cost of the artwork itself, so a lot of people, families and institutions never restore them. He also describes so many museums holding most of their collections in storage, with a fair portion of those holdings, in some cases up to 10 percent, being damaged art the institution has no means to repair.

  • No. Prompting a general image model to restore a damaged painting will not get you anything close to what a conservator would do, and Alex Kashkin is direct about that. His explanation is that restoration carries a lot of very niche considerations and details that you might not think of, and that the dataset needed to train a fully competent restoration model does not yet exist. He expects one within a few years, helped by institutions cataloging their holdings and making high quality images available for free with no licensing required.

  • In the decision-making around the generative step, on the evidence of how Alex Kashkin built his. Asked whether he uses machine learning or other algorithmic mechanisms besides the generative AI, he answers that he does not, not for making masks at this point, and says the process of generating a mask carries a lot of rigor and largely hard-coded design instead. His reason is that there are candidate metrics for how different two colors look to a person, none of them fully correct, and the most accurate take an enormous amount of computing. On deciding how bad a scratch or a discoloration has to be before it warrants repair, he says models can be trained to do that, but that in the end it is really important to tie it back to the way humans perceive color. He leaves the door open, saying there might be ways of applying machine learning there, while adding that it is harder for him to see how those would account for the biological side of human vision. Conservators, he says, have really appreciated the work because it is that rigorous.

  • Not reliably yet, but the technique exists and has a name: digital fingerprinting. Alex Kashkin describes taking an image of a work, potentially from a phone camera, and identifying the textural aspects that really imply authorship. He points to a whole corpus of art whose author is genuinely unknown, sometimes because several artists worked on the same canvas with one painting figures and another backgrounds. He calls parsing that authorship a huge technical challenge, and an opportunity that has not been explored much lately, while noting that AI tools are adept at noticing patterns the human eye is not capable of noticing. An earlier proof of concept paper, he says, drew a lot of criticism for not being as rigorous as conventional attribution.

  • Art that is too fragile to restore can be reproduced as a facsimile, so it can be seen without being repaired or moved. Alex Kashkin describes paintings sitting in storage, damaged in a way that leaves them too fragile to be restored, where transporting them between places is a bureaucratic nightmare in its own right, and says there is huge demand for making facsimiles of them. A facsimile lets an institution that is based on visitors exhibit something, and lets an owner have access to a work in multiple places at once. He calls it a really fun opportunity, and was working with a couple of groups in Geneva on making it possible with the masking technique.

  • Not on the evidence in this conversation, where the demand runs the other way. Alex Kashkin says it was conservators themselves who reached out after the paper was published, wanting to see the technique applied in their own workplaces and their own practice, and he describes the benefit to them as saving time and being able to access a much larger portion of the market. His framing is that work currently priced out of restoration becomes reachable, rather than that existing work changes hands. He does not take up the question of displacement directly on air.

 

  • [00:00] Christina Ellwood: Welcome to AI Realized, the podcast for enterprise executives leading AI deployments. From tackling security, data, and operational challenges to navigating organizational transformation, AI deployment offers a unique opportunity to redesign organizations from the inside out. I'm Christina Ellwood, your host for today's podcast, and we're talking today with Alex Kashkin. Alex is a unique voice for us on the "AI Realized" podcast. He is actually a graduate student at MIT, and he has in- invented a new application for AI that does art restoration. I'm gonna let him tell you all about that, but I should also mention that his graduate research is in electron sources because he works with materials for coming up with electron sources for building semiconductors. And so s- there's some connection between those two that he's gonna share with us today, and we will explore the technology that he has, uh, developed and its application, and then talk about the startup that he's spinning out. So with that, let me welcome Alex to the conversation.

    [01:04] Alex Kashkin: Thank you for having me. Very excited to be here. I thought I could give you a brief overview of what I've done with this new art technology that's come out, and just we can dive in from there if that sounds good.

    [01:16] Christina Ellwood: Sounds wonderful. Go

    [01:18] Alex Kashkin: for it. Awesome. Yeah, so put simply, art restoration has been technologically the same for the past 500 years. Damages on paintings have always been restored by hand with brush. And I've done the first step in bringing in any technology into the space, and trying to alleviate some of the difficulties that conservators and organizations face with the labor cost of needing to restore artworks. And what I've done is applied generative neural networks to the creation of masks that fill in damages present on an artwork, and then physically apply those masks with a reversible ultra-thin polymer film to an actual painting. And that restoration process is almost two orders of magnitude faster than doing so by hand, and can lead to a lot of transformational changes for conservators and institutions that hold damaged art in need of repair.

    [02:19] Christina Ellwood: So I can understand the connection between the mask and the work that you do in silicon, but some of our m- listeners may not. Can you just explain how you make masks for silicon production fabrication?

    [02:30] Alex Kashkin: Yeah. For a more general audience, in the semiconductor world, chips that go into your phone, those have very micro scale, often at the nano scale, patterns that are embedded into silicon wafers that are covered with different types of materials. Those masks are made with extremely precise electron lithography tools and other forms of pattern creation, and are applied extremely precisely because you need to be able to align multiple layers over each other. And the interesting connection with that and a mask restoration of an artwork is that the same precision constraints, the same tolerance modeling, the same analysis that happens just on a spatial scale is actually directly applicable. And that transforma- A- applying those techniques from the semiconductor world to the art world is something that is, is fairly bespoke and niche, and no one's really thought of doing before. But it turns out it is pretty important and critical if we're trying to apply some of these new image-based technologies to the real world in affecting artworks.

    [03:40] Christina Ellwood: Did you come up with this idea because you're an, a restorer yourself?

    [03:44] Alex Kashkin: Yeah. I've been restoring paintings for around a decade at this point, and it started off as a hobby of sorts. I just got into it because I wanted to have some cool art. And a few years back, as I was making my way up the East Coast, I was realizing that so many museums, in the United States at least, and really across the world, have most of their collections in storage, and you're not going to get to see those. And a fair portion of those collections, in some cases up to 10%, is just damaged art, and they don't have the means to restore it. And the biggest issue with restoring paintings is the time it takes to inpaint losses and really reconstruct the aesthetic losses that are present. So at that point, this was the era when we already had DALL-E and other image, generative image models out, and they were applied as well in patched ways, so it was possible to do patched application of these models to repair images. And I had the idea, why can't we do that to paintings? And conceptually, it's a pretty simple idea, and it just turns out that to actually apply it in a way that's satisfactory to human vision, th- that's where it takes a lot of engineering, and perhaps much more so than I originally expected.

    [05:02] Christina Ellwood: A story many entrepreneurs can tell you, and inventors and entrepreneurs, right? Absolutely. It, the simple idea is often much more complicated than it, than meets the eye.

    [05:11] Alex Kashkin: Yeah.

    [05:11] Christina Ellwood: So you came up with the idea, you did the application, you worked out the challenges. Then what happened?

    [05:18] Alex Kashkin: So once I published the paper a couple months back, which fortunately made the cover of Nature of all places, a lot of people started reaching out to me, both art owners with a ton of damaged work saying, "Hey, we can't-- we haven't been able to restore this. Can we use your technique to restore it?" And m- much more interestingly, actual conservators and people in the art business world reaching out with interest in actually seeing this technology applied in their workplaces and applied in their practice. And that's especially the case with conservators. Some reached out to me complaining that they have to reject almost 80% of potential clients because the clients can't afford the enormous quotes that are offered. And those quotes are high just because of the labor it takes to do the restoration. So there was a huge outpouring of demand for trying to apply these techniques, not just to paintings, but other types of media as well. I've had conservators across the world reach out with interest in restoring Buddhist works of art on textiles, with artworks on cave drawings, with artworks on Italian frescoes. And there's been so much interest from so many places, and I'm trying to answer everyone's questions and make sure that I'm reaching out to everyone that I can. But it is a lot, and I think there's clearly a lot of excitement, and I'm hoping that more than just me will be contributing to developing this tech forward.

    [06:49] Christina Ellwood: I think you, you may find that you get a- attraction from entrepreneurs and investors as well. It sounds like there's real business to be made here. So you wanna talk a bit about that?

    [06:59] Alex Kashkin: Yeah. So I am leaving MIT effective Monday, and will be commercializing this new technology, and we're actively raising funds. There's a lot of interest behind it. People have been reaching out basically with the... There's the obvious use case of this technology of saving conservators time and being able to access a much larger portion of the market. And I think what people might not readily understand there is the vast majority of artworks that are restored nowadays are very high value artworks. We're talking names of artists that you'll recognize. But the amount of paintings that exist that are damaged by perhaps artists that are lesser known, that dwarfs the number of paintings that exist by artists that we do know of. And one of the consequences of that is that for a lot of artworks, the conventional cost of restoration is much larger than the cost of the artwork itself. And for a lot of people, a lot of families, and a lot of institutions, that means they're just not going to ever restore them. And my hope is that in commercializing this technique, we'll be able to open up that swath of opportunity for conservators both in the US and abroad who conventionally are just unable to help out in those cases.

    [08:18] Christina Ellwood: And- And of course, there's also the expanded opportunity beyond paintings, right? You mentioned frescoes and you mentioned- Oh,

    [08:23] Alex Kashkin: absolutely ...

    [08:23] Christina Ellwood: textiles, and so there are many others. And I imagine textiles, um, including paper, are, um, areas that are particularly sensitive and could benefit because the film would actually give them, uh, uh, more of a structural s- um, uh, integrity than they would-

    [08:38] Alex Kashkin: We're diving into actual conservation talk here. I'll tell you, I was fortunate enough to be invited to give a talk to the San Francisco Museums of Art a few weeks back, and got to talk with basically every conservator and curator in the Bay Area in one place. And they-- we had a lot of exchange of ideas and dialogue, and it turns out that on, at least in, on the western side of the planet, the modernization of art restoration is very much in vogue. The conservators in San Francisco are very interested in applying new techniques, not just to paperworks and oil works, but really trying to see what other ways are there to apply generative AI and other transformational computing tools to their goals. And a lot of those are very interesting, a lot of those are behind the scenes that I can't really share, but there's so much excitement building around this, and there's a lot of use cases that I hadn't thought of before that are actually extremely interesting opportunities that I expect will be quite meaningful. One of the ones I want to highlight is in the reproduction of damaged artworks. It's the creating facsimiles of art. And you might encounter this sometimes for sale, artworks that are replicas of famed works. But as it turns out, there's a lot of paintings out there that are in storage and damaged in a way that it-- they're too fragile to be restored, and transporting them around to different places just, it is a huge bureaucratic nightmare. So there's huge demand for making facsimiles of art that is otherwise locked in storage, and making those facsimiles available to institutions like museums that are based on visitors, and also making it available to art owners who would like to have access to their works in multiple places at once. So I'm-- That's a really fun opportunity, and I'm working with a couple groups in Geneva actually to try and make that possible given the new masking techniques that I have. But it's a very different application, very different than what I originally developed and had in mind. But is one of those translational opportunities that I think shows how art as a field has been so far lagging behind the adoption of tech and everything, that it's just ripe for having a lot of tech being used to solve challenges that have been there for centuries.

    [11:07] Christina Ellwood: Yeah, I think it's also an interesting source of new images for training models, don't you think, Alex?

    [11:12] Alex Kashkin: Oh, yeah. There's, I think the actual dataset collection that is happening in various places around the planet, it's really ramped up in recent years. You can now access a lot of really good quality images for free with no licensing required from major institutions that are cataloging the works that they have. And those images are really useful, not just for training and painting models, but also for a more specific niche case of training models that can at least emulate the process of restoration or conservation. And the considerations there are really not intuitive. And so if you pass in an image of a damaged painting to a conventional model nowadays, and you type in a prompt saying, "Restore this painting for me," you're not going to get anything near what a conservator would do. And part of the reason for that is there's a lot of very niche cons- considerations and the details that you might not think of, and we just don't yet have the dataset to train a fully competent model for that. But I think it's coming, and in a few years, we should definitely be able to have one.

    [12:17] Christina Ellwood: You mentioned DALL-E. Are there other models that you use, and if so, which ones?

    [12:24] Alex Kashkin: Oh, there's so many models. They all have their bespoke names right now 'cause all the, every new iteration of every new GitHub will have its fun little name, and everyone keeps updating theirs. I might be able to pull up somewhere which ones I've tried out using. There's a lot of development happening both in the academic world and also in the private world, and I think the folks who are doing the best job at actually contributing in the space of patched inpainting is the research teams that are being funded by Adobe's research branch. And their work has been pretty instrumental in making new, making new results possible in this space and demonstrating abilities that we didn't have before. And I think that's tied in as well in AI is such a broad term for algorithms, basically. And in a lot of ways, the new tech that's becoming really applicable in the arts is based on more than just trying to generate images, but trying to incorporate some logical reasoning in various aspects of doing so. And some of the folks that I know are doing really interesting things in the space are doing digital fingerprinting of artworks. Basically taking an image, even if you do it with a phone camera, and being able to identify the textural aspects of a work that really imply authorship. And that's one of those areas where there's been some interest directed at me in trying to incorporate some of the techniques that I've developed, and understanding and analyzing, basically, authorship of an artwork. And I think that's a very interesting opportunity that really hasn't been explored that much lately. There's a whole corpus of art that exists where we just don't really know who painted it because it might have been multiple people painting the same work. Some people might have been-- Some artists might have been working on figures, some artists might have been working on backgrounds. And being able to parse that authorship is a huge technical challenge, but one where AI tools are actually really adept at noticing patterns that the human eye is not really capable of noticing. And I think that's one of those really interesting tangential opportunities that really I'm expecting it to become bigger in the next few years. There was a paper that came out, I think, a couple years back trying to do, and it was a proof of concept paper, and got a lot of flak for not being as rigorous as conventional authorship goes. But that's one area that I definitely think we'll go forward in and we'll see a lot of interesting results come out.

    [15:05] Christina Ellwood: There's similar work that's been done in music, and in fact, there's a startup called StreamCore that is doing a combination of provenance for musical works and the detection of playout on, on the internet. So that, that, those ideas might be complementary and they're, it may be valuable for you to talk to some of the folks from there as well. Are you using any kind of machine learning or other algorithmic mech- mechanisms besides the generative AI using other types of AI?

    [15:35] Alex Kashkin: Not for making masks at this point. So one of the reasons-- I, I've tested out a number of different strategies, and one of the reasons that what I've published is so methodical and, you know, I'm a mechanical engineer, so I wanna use the term mechanical, but it really isn't. Um, but the, there's a lot of rigor and basically hard-coded design of that process of generating the mask, and the reason for that is really because we want to have a more thorough understanding and just rigorous application of decision-making in constructing such a mask. So one of the interesting areas of science that we're really not that far in at this point is in understanding human color vision. So we don't-- If I show you a color that looks red and a color that looks blue and I tell you how different are these two colors, you might-- There's a metric that can be assigned to that. We have way, we have some candidate metrics for evaluating that difference, but none of them are fully correct, and the ones that are most accurate take an enormous amount of computing to actually realize. And that's connected with art restoration in a very direct sense because when you look at a painting, the parts of a painting that might be damaged stand out. That's the whole point of correcting damages in the first place, is making sure that when you view an artwork, you reintegrate surviving areas in a cohesive way. But identifying what those damages are and what is the threshold at which a damage count, how bad does a scratch or discoloration need to be in order to make it candidate, a c- a good candidate for restoration, that's something where, you know, yes, we can train models to do that, but in the end, it's really important to tie it back to the way that humans perceive color and understanding what is the relationship, what is the science between the way that we, our trichromatic vision perceives these colors and damages, and how can we use that in creating a mask? And I address that in the algorithm that I use for making the mask in an effort to be very rigorous, and that's been-- Conservators have really appreciated that part of the work because it is so rigorous. But I think there might be ways of applying machine learning there. Uh, it is more difficult for me to see how those would be able to account for the biological aspects of art perception and human vision.

    [18:01] Christina Ellwood: Yeah, for sure. This has been a fascinating conversation. Thank you so much, Alex. What, what resources would you point listeners to if they want to learn more or they want to follow along as on your journey?

    [18:14] Alex Kashkin: Yeah, I think there's been a lot of publications in the news about this work. You can read the New York Times piece that highlights the Italian collaboration I have going on. The Spectrum magazine did a very thorough analysis and brought in some folks to comment on this work. If you want to read the paper itself, it's on Nature. There is a free access link that I can share too. I'm sure you will hear about the way that we commercialize this technology and make it accessible to conservators and institutions across the world. The candidate name for this company is Conservation 21. Whether that will stay or not, we will see. But if you-- I'm sure there will be coverage of what we end up doing and the way that, that our techniques will be applied.

    [18:59] Christina Ellwood: Well, if you provide us those links, we'll put them in the show notes. And I just have one last question for you. I know it's early in your journey, but for every entrepreneur who's leading the charge like this, there are leadership skills that they find are particularly critical. So for you thus far, what is the leadership skill that you have found to be most valuable?

    [19:19] Alex Kashkin: I think it's being able to agree with people and trying to resolve conflicts through positivity. It's become really important. I've been talking with literally hundreds of people in the past couple of months, a lot of them, you know, disagreeing with me on certain issues, a lot of them wanting to contribute to what I'm doing in interesting ways. And being able to take in that feedback, view it positively, and work with people towards new ends and trusting that people have good intentions, that's become really important towards getting things forward. Because technology like this takes a bit of a crazy mind to do, and if you're not willing to just go along with things and see how things end up going, you're not gonna get to interesting places. Well

    [20:08] Christina Ellwood: said. Alex Kashkin, thank you so much for spending the time with us today. It's been a delight to talk with you.

    [20:15] Alex Kashkin: Thank you so much.

 
Listen to other episodes
 
 
 
Join AI Realized
 
 
 
Subscribe to the newsletter
 
 
 
Christina Ellwood
Previous

More Podcast Episodes
Next

The Data Itself Is the First Prompt in Vibe Analytics