Use AI for the Questions Nobody in the Room Will Ask
Episode Summary
Decision-making is where Chris Butler, then staff product operations manager at GitHub, puts the value of an LLM, and his reason is about power rather than intelligence. Teams struggle as they get larger, he says, because decision-making calcifies, and he reaches for the dominant logic idea: the people reusing the mental models that made them successful are usually higher up, which filters out variance in ideas and in ways of working until a bad outcome gets likelier. A model in that conversation has no stake in it. It is not emotionally involved, it is not part of the hierarchy and it does not care about power, so it can ask what the room will not, or give examples of other cases worth reconsidering. He is candid about the yield, putting it in February 2025 at about 80 percent not helpful, and the value in the few that come from somewhere else.
Key takeaways
He defines his own role in a way that explains everything after it. He calls himself a PM for the PM experience, there to understand how the product management organization works together and how its product managers build healthy tensions with engineers, designers, project managers, legal, privacy, security and responsible AI
His personal brand is a Dungeons and Dragons alignment and he means it as a working method. Chaotic good is how he approaches a world where teams fall into patterns or into undue certainty, and his job is to mix that up through questions, workshops and even meeting design
His analogy for where AI sits right now is deliberately unflattering to everyone including himself. It is the move from web to mobile, when the apps were fake lighters and slot machines and nobody had gotten to TikTok yet
His statement of the problem is aimed at his own side. He thinks teams that consider themselves very certain about the future are wrong, and his personal motto is that he is wrong today, he just does not know how yet
He borrows a framework to explain why one team’s practices do not fit another, and names its author. Kent Beck’s three-X model puts technology adoption on an S curve in three phases, explore at the bottom with cheap rapid experiments, expand at the hockey stick where everything breaks and bottlenecks appear, and extract or exploit at the top where you stop making new things and optimize
The diagnosis he draws from the three-X model is about mismatch rather than skill. Teams are somewhere on the curve for a given product or feature, they do not always understand where, and so they create dynamics that are not appropriate for the stage they are actually in
He puts optimizing on the same footing as building. The idea of optimization is also a creative act, because taking something hard and making it simple, or reducing a system’s complexity so it is more resilient and robust, is itself creative work
His account of why he studies decisions starts with a specific frustration rather than a theory. People wanted to do the right thing and kept getting stuck in hierarchies and power structures, so they made the safe decision rather than the one that would push things in a better direction
He is careful about what a good decision even means, and the correction matters for everything after it. Better or bad decisions is maybe not good terminology, he says, because it focuses on the outcome, and in an uncertain and complex world a decision will sometimes be wrong, meaning it did not have the impact you wanted; what you can improve is the process that produces it
The mechanism he blames for decisions getting worse as a company grows has a name and a citation. Dominant logic, from a paper he places around 1985 or 1986: people who have been successful reuse their past mental models in current situations, and those people are usually higher in the organization
The damage dominant logic does is specific. It filters out variance of ideas and variance of ways of doing work, and eventually produces the wrong decision, or one with a bad outcome, more often than not
His limit on what he would hand a model comes before any of the benefits. He would not want a system making a decision on something that is not heuristic based, because given a brand new situation he does not think a model can yet decide from new information and circumstance rather than falling back on what it was trained on
The reason he wants a model in the room is about position rather than capability. It is not emotionally involved, it is not part of the hierarchy, and it does not care about power inside the system, so it can ask questions or raise other cases that make people reconsider their current thinking
The prompts he actually uses are ordinary enough to copy, and he lists them. Here is a process I am thinking about implementing, what will be confusing about it; play out how someone would go through it and the problems they would have; what am I not thinking of; what best practices should I consider
He puts a number on how much of it is worth having and it is not a flattering one. About 80 percent of what comes back is not helpful and is something he had already thought of, and the value is the few things from another domain he had not considered
Research he was part of at a design consultancy produced two answers, and his own reframing of them years later is the useful part. Users were asked whether they understood what the system was doing and whether they liked it; looking back, he reads the liked-it group as describing work being taken off them that was not the creative part, and the did-not-like-it group as describing things about human connection and finding meaning in the work
The distinction gives him a test for which work moves, and he applies it to specific roles. Summarization is a real part of what an operations person, a technical program manager or an executive assistant does, it is daunting when you are trying to get into the mindset of whoever wrote the thing, and he would say that type of work will start to go away because it will be done automatically
What he says is left for people is not the usual list. Organizational dynamics, creativity, making exceptions, and one he offers half seriously, that humans are really good at phoning it in when appropriate, so part of the job is telling the system that something is just not worth doing right now
The future he describes for GitHub is translation rather than automation, and he frames it as a world he could imagine. A product manager works with an agent to turn a concept into boilerplate, a designer’s mockups in a tool like Figma update it, and then the trade-offs an engineer makes get illustrated back in a form the product manager, the designer and even the customer would understand
His closing ask has two halves and the second is aimed at IT, HR, legal and responsible AI. Try the tools and understand where they are not working, and then get IT, HR, legal and responsible AI to build environments where experimenting is safe, rather than publishing a page that says you cannot use this
About Chris Butler
Chris Butler was staff product operations manager at GitHub at the time of this conversation, and is now director of product operations at the same company. He describes the role as being a product manager for the product management experience: making the people who build products more effective, and watching how those product managers build healthy tensions with engineering, design, legal, privacy, security and responsible AI. Chaotic good product manager is the label he explains on this episode when the host raises it as his personal brand: a Dungeons and Dragons alignment used as a working method, aimed at the undue certainty teams fall into. Before GitHub he worked at Waze and KAYAK and at a boutique design consultancy called Philosophie, all of which he names on this episode, and he dates his turn toward product work to a Microsoft interview that asked him how he would design a washer and dryer combination for someone who is sight impaired. His interest in decision quality came out of wanting to make the right decisions and finding that hierarchies and power structures produced the safe ones instead.
In this episode
| 00:41 | Welcome, and who Chris Butler is |
| 01:17 | Product manager against product operations manager |
| 02:09 | A PM for the PM experience |
| 02:34 | Understanding how the PM organization works together |
| 02:53 | Legal, privacy, security, safety, responsible AI |
| 03:09 | What is a chaotic good product manager? |
| 03:25 | An Easter egg for the Dungeons and Dragons players |
| 04:50 | Web to mobile: fake lighters before anyone got to TikTok |
| 05:12 | Certain teams are wrong. I am wrong today, I just do not know how yet |
| 06:03 | The iterative trap, and where Kent Beck comes in |
| 06:24 | The three-X model: explore, expand, extract |
| 07:05 | Teams are somewhere on the curve and do not always know where |
| 07:45 | Optimization is also a creative act |
| 08:44 | An apprenticeship with an art director, and X-Acto knives |
| 09:16 | Job offers in secure radio, and the thing he actually loved |
| 09:36 | The Microsoft interview question about a washer and dryer |
| 10:22 | Product, evangelism, and hybrid roles at Waze and KAYAK |
| 10:36 | Stuck in hierarchies, making the safe decision |
| 11:29 | Better decisions, and why outcome is the wrong test |
| 12:26 | Including uncertainty rather than shutting it down |
| 13:11 | Decision-making calcifies as teams get larger |
| 13:44 | Dominant logic, and the variance it filters out |
| 14:07 | Reconsidering the dogmas and norms of an organization |
| 14:30 | What he would not hand a model: anything not heuristic based |
| 14:58 | No stake, no hierarchy, no interest in power |
| 15:20 | Summarization, and the status reporting every org asks for |
| 15:42 | Abductive thinking, and taking a leap from evidence |
| 16:03 | An LLM taking on personas to ask the hard questions |
| 16:23 | The prompts he actually uses |
| 16:39 | About 80 percent is not helpful, and why he keeps asking |
| 17:00 | LLMs as thought partners, and AI as a provocateur |
| 17:41 | The Philosophie research with field service dispatchers |
| 18:21 | The reframing: work that is not the creative part |
| 18:43 | The human connection group, and whose job summarization is |
| 19:03 | Daunting work that will start to go away |
| 19:15 | What is left for people, and phoning it in when appropriate |
| 19:44 | The future of AI for development teams at GitHub |
| 20:03 | A world he could imagine: concept to boilerplate to trade-offs |
| 20:32 | Mockups in Figma, and the trade-offs made visible |
| 21:23 | What listeners should take away |
| 21:34 | Try them, and understand where they are not working |
| 21:49 | Make it safe to experiment, rather than publishing a ban |
| 22:27 | Close |
In Chris’s words
“my personal motto is that I’m wrong today, I just don’t know how yet”
Chris Butler (05:12)
“I think of myself as actually a PM for the PM experience”
Chris Butler (02:09)
“the idea of optimization is also a creative act”
Chris Butler (07:45)
“The problem is that filters out variance of ideas, variance of ways of doing work”
Chris Butler (13:44)
“And I think I would not want a system to make a decision on something that is not heuristic based”
Chris Butler (14:30)
“it’s not part of the hierarchy, it doesn’t care about power inside of the system”
Chris Butler (14:58)
“about like 80% of the stuff is not helpful. It’s things I’ve already thought of”
Chris Butler (16:39)
“how do we make it so that it’s safe for people to experiment with these things”
Chris Butler (21:49)
Resources
Chris Butler on LinkedIn: His LinkedIn profile
GitHub: Where he was staff product operations manager at the time of this conversation, and is now director of product operations
Liminal Practice: His Substack, where he writes now
Referenced on air
Fast, slow in 3X: Explore, Expand, Extract: Kent Beck’s own account of the model Chris Butler describes at 06:24. Beck was one of the original signatories of the Agile Manifesto and worked at Facebook, both of which Chris Butler notes on air
The dominant logic: A new linkage between diversity and performance: The paper he reaches for at 13:11 and dates to about 1985 or 1986. It is C. K. Prahalad and Richard Bettis, Strategic Management Journal, volume 7 issue 6, 1986
Thinking, Fast and Slow: The source of the system one and system two framing he uses at 10:54. He names the book at 10:54 and does not name Daniel Kahneman
Named on air
Philosophie: The boutique design consultancy where he did the multi-agent research at 17:41, with field service dispatchers. It was acquired in 2019 and has traded as InfoBeans Accelerate since December 2022, so there is no current site under the name he says
Waze and KAYAK: The hybrid business development and product roles he describes at 10:22, which he calls a different type of product management
Microsoft: Where he interviewed at 09:36, and the source of the washer and dryer question, on designing one for someone who is sight impaired, that he says turned him toward product work
Motorola and General Dynamics: The two job offers he mentions at 09:16, in secure radio development and secure networking, at the point where he realized programming itself was not the thing he loved
Figma: Named at 20:32 as an example of the kind of tool where a designer’s mockups would live in the workflow he describes for GitHub
ChatGPT: Named at 21:49 in his bet that a lot of people are already using these tools on personal accounts, which is his argument for making internal experimentation safe
Ideas and terms discussed
Chaotic good product manager: His own public description of how he works, taken from the Dungeons and Dragons alignment system. The method underneath it is unsettling undue certainty through questions, workshops and meeting design
The three-X model: Explore, expand, extract. Kent Beck’s three phases of technology adoption on an S curve. Chris Butler uses it to explain why a practice that fits one stage may not be appropriate for another
Dominant logic: Reusing the mental models that made you successful when the situation is new. His explanation for why decision quality falls as an organization grows, and for why what an organization loses shows up as filtered-out variance
Abductive thinking: Taking a leap based on evidence, which he says a lot of product managers do, as against deduction moving back and forth between evidence and rules
The model with no stake: A model in a decision is not emotionally involved, is not in the hierarchy and does not care about power, so it can ask what the room will not
I am wrong today, I just do not know how yet: His stated personal motto, and the reason he treats certainty as the thing to design against
Translation across practices: The future he describes for GitHub from 20:03, and the word he uses for it at 20:59, where a trade-off an engineer makes is rendered back in terms a product manager, a designer or a customer would understand. Episode 42 is where he takes this further
Frequently Asked Questions
-
AI improves a team decision by asking the questions the people in the room will not ask, which works because the model has no position to protect. Chris Butler of GitHub puts it in terms of power rather than intelligence: a model is not emotionally involved, is not part of the hierarchy and does not care about power inside the system, so it can raise a question or an example from another domain that makes people reconsider what they already think. He is candid about the yield, saying about 80 percent of what comes back is something he had already thought of, and the value is the remainder.
Transcript 14:30 to 17:00
-
Dominant logic is the tendency of people who have been successful to reuse the mental models that worked before when they face a new situation. Chris Butler of GitHub attributes it to a paper he places around 1985 or 1986, and the reason he raises it is what it does inside a company: the people applying those old models are usually higher up the hierarchy, so their reuse filters out variance in ideas and variance in ways of doing work, and eventually produces a decision with a bad outcome more often than not.
Transcript 13:11 to 14:30
-
Decisions get worse as a company grows because decision-making calcifies once choices have to be passed up a hierarchy. Chris Butler of GitHub says teams will always struggle that way as they get larger and larger, and his contrast is with teams he has worked with that stay dynamic. Those teams move fast in a highly volatile environment and keep trying a lot of things, and he treats that as only one of the things, because they also reconsider what have become the dogmas and norms of the organization.
Transcript 13:11 to 14:07
-
A product manager pushes innovation and customer value, while a product operations manager works on the way the product organization does its work. Chris Butler of GitHub calls the second role a product manager for the product management experience: understanding how leaders work with their teams, how the teams work with each other, and how product managers build what he calls healthy tensions with engineers, designers, project managers, legal, privacy, security and responsible AI. He places project and program managers at the other end of a spectrum, where the job is to reduce risk in how software gets delivered.
Transcript 01:17 to 03:09
-
Kent Beck’s three-X model describes technology adoption in three phases along an S curve: explore, expand, and extract or exploit. Chris Butler of GitHub uses it to explain why one team’s practices do not transfer to another. Explore sits at the bottom of the curve and is about cheap, rapid experiments to find what creates value. Expand is the hockey stick, where everything breaks and new bottlenecks keep appearing. Extract is the top, where you stop building brand new things and optimize what is there. His point is that teams are somewhere on that curve without always knowing where, and then may end up creating dynamics that do not fit the stage they are in.
Transcript 06:03 to 07:45
-
Kent Beck’s three-X model describes technology adoption in three phases along an S curve: explore, expand, and extract or exploit. Chris Butler of GitHub uses it to explain why one team’s practices do not transfer to another. Explore sits at the bottom of the curve and is about cheap, rapid experiments to find what creates value. Expand is the hockey stick, where everything breaks and new bottlenecks keep appearing. Extract is the top, where you stop building brand new things and optimize what is there. His point is that teams are somewhere on that curve without always knowing where, and then may end up creating dynamics that do not fit the stage they are in.
Transcript 06:03 to 07:45
-
The parts worth handing over first are the ones people do not find meaningful, and summarization is the worked example. Chris Butler of GitHub reaches that test from research he was part of at the design consultancy Philosophie, which asked people whether they understood what an AI system was doing and whether they liked it, and he later reframed the two answers: the work people were glad to lose was the busy work they did not enjoy, and the work they did not want taken was about human connection and finding meaning. Summarization is a real part of an operations person’s, a program manager’s or an executive assistant’s day, it is daunting when you are reconstructing someone else’s intent, and he says that type of work will start to go away.
Transcript 17:41 to 19:44
-
IT and legal should build an environment people can safely use, rather than publish a ban. That is the ask Chris Butler of GitHub makes of IT, HR, legal and responsible AI together. He said in February 2025 that he was betting a lot of the people using these tools were doing it on a personal ChatGPT account rather than through anything internally confidential and secure. His ask is that those functions create the environments that make experimenting safe, instead of putting out a page that says you cannot use this.
Transcript 21:23 to 22:27
-
An AI should not be making the decision where the decision is not heuristic based. That is the line Chris Butler of GitHub draws for himself. His reason is about novelty rather than accuracy: models can produce interesting content and do a kind of pseudo-analysis, but given a brand new situation he does not think one can yet decide from new information and circumstance, and it will fall back on what it was trained on. What he wants instead is a participant that questions the decision rather than one that makes it. He puts it as what he would not want rather than as a rule for anyone else.
Transcript 14:30 to 15:20
-
[00:41] Christina Ellwood: Welcome to AI Realized, the podcast for enterprise executives leading AI deployments. From addressing security, data, and operations challenges to managing the organizational and management changes, AI deployment presents the opportunity to redesign our organizations from the inside out. I’m Christina Ellwood, your host for today’s episode. We’re talking today with Chris Butler, the staff product operations manager for GitHub. Chris, welcome to AI Realized.
[01:10] Chris Butler: Great. Thank you for having me, and I’m excited to talk about the way that we, we leverage AI not only for our products, but also for the way we do our work.
[01:17] Christina Ellwood: That sounds great. But just before we get into AI, I’d love to just have you give us a distinction between a product manager and a product operations manager.
[01:25] Chris Butler: Absolutely. So a product manager, I see them as someone that’s really trying to push innovation within the organization. They’re trying to make sure that we’re still-- we’re addressing kind of customer needs and providing customer value, but we’re also creating things that, that in the end create business value so that we can continue to exist as an organization. Um, I would say that there’s an interesting spectrum between them and project managers, program managers, TPMs, where those types of roles are really trying to reduce risk within the organization and how we deliver software, usually through a software development life cycle. But they’re important extremes on a spectrum, and we need to have them basically pulling in different directions to be able to build something that is not only like innovative, but also something that is deliverable. And so every industry requires a different place on the spectrum because medical work, for example, requires a lot more de-risking than something like a very early stage, like B2C startup, where you’re just doing lots of experimentation. I would say product operations, what I do is maybe-- I, I would call it maybe because it, it is focused on operations, like the way we do work, I think of myself as actually a PM for the PM experience. And so what that means is that I’m there to understand the way that the PM organization works together, and that’s usually focused on the way that leaders work with their teams, but also the way the teams work with each other, and then the way those product managers end up kind of building the healthy tensions with other groups like project managers or engineers, designers. But also the idea of like legal, privacy, security, safety, responsible AI. Those are all different groups that end up coming together to create a product. And so my role is to really make sure that those PMs are being as effective as possible.
[03:04] Christina Ellwood: That sounds like a fascinating role and one that is cross-functional in many ways. And I understand that you have an area of specialty that perhaps other people have not heard of before. You have a unique personal brand, chaotic good product manager. What is a chaotic good product manager? I know what a good product manager is, but what’s a chaotic one?
[03:25] Chris Butler: Yeah. A chaotic good product manager, this is a bit of an Easter egg for the Dungeons & Dragons nerds that are out there, which I used to be one. I still play some role-playing games, by the way. But, like, chaotic good is an alignment within the Dungeons & Dragons, like, character system. And what it really says is how do you approach the world? And the way that I mean chaotic good product manager is that I approach the world in such a way that I, I think a lot about that the rules that are there, especially when it comes to the way that software’s developed, the way that we consider our customers’ needs, that sometimes we fall into patterns or certainty that we sh- that is undue certainty. And so it’s my job to try to mix that up a bit, and that can be through a lot of different techniques. Sometimes it’s really asking just important questions. Other times it’s helping create a workshop that allows for the right type of information to come out. Even meeting design is something I think a lot about. And just organizationally, like, how do we build the dynamics inside the team so that we’re always thinking about the most important thing? And so that’s where chaotic good comes in. And it means I might be a little mischievous as well sometimes when it comes to the way that I build these systems. But that’s, yeah, that’s what it means.
[04:32] Christina Ellwood: I think that’s probably a good thing because most people associate chaos and randomness with badness, not with goodness. So I think that’s a good thing that you’re more mischievous about it than maybe serious.
[04:43] Chris Butler: And let me be clear too, the world is a very uncertain place, especially in the AI kind of environment and world, and everybody building stuff. We d- no one knows very much right now about the end kind of place that these tools are gonna be in. We’re, like, at a place where we were moving from, like, web to mobile, where we didn’t know... It was a lot of, like, apps for, like, fake lighters and, like, slot machines and stuff like that. We didn’t really get to something like TikTok yet, which is a brand new way for people to interact with media in a very mobile approach. And so we’re at that place right now with AI. And so I think teams that do consider themselves very certain about the future, they’re wrong. And I would say that my personal motto is that I’m wrong today, I just don’t know how yet. And so for me, that chaos actually allows us to at least understand that we cannot always control everything, and then it allows us to build systems and organizations and even products that can be resilient inside of this kind of highly complex and uncertain world, essentially.
[05:42] Christina Ellwood: Besides accounting for that fact of uncertainty and imagining all of the things, ways things could go wrong or be used at the extremes. Are you also trying to help the teams to be more innovative and maybe create products that otherwise they might not consider because they aren’t within some very narrow confines?
[06:03] Chris Butler: Yeah, that’s right. I th- I think a lot of the time product teams fall into this, like, highly iterative pattern where once they see something that is successful, they continue to just keep working on that towards, like, fixing problems. And there’s, like, a method or I guess a framework that Kent Beck, who was one of the original signers of the Agile Manifesto, he worked at Facebook for a while really on, like, team dynamics. But he talks about this, like, three X model on an S curve, which is that when we talk about, like, technology adoption in general, we go through three very distinct phases. One at the very bottom of the S curve is explore, where we’re creating lots of experiments, ideally very cheap experiments, very rapid experiments to understand what would be valuable, how we create value, how we deliver value. At the kind of, like, hockey stick part where things start to go up, that is called expand. And inside of that realm, everything is breaking, you’re finding new bottlenecks, people continue to want to have more and more. And then finally there’s the kind of extract or exploit at the very top, which is that you’re gonna get off this curve of creating brand-new stuff, and you’re gonna optimize everything. And so I think whenever we talk to teams, they’re somewhere on this curve for even for a product or a feature. They don’t always understand, and so they end up maybe creating dynamics inside of their world where it’s not appropriate for the stage that they’re in. And so my role, and at least this understanding of, like, how do we do a better job of kind of pushing for innovation, is that in the explore area, again, it’s how are we experimenting and learning as rapidly as we can. And so setting up systems to allow for that learning is really important. And then same thing for, like, in, in the expand, like detecting the bottlenecks and addressing the most important ones that are going to stop people from using our technology. So I think that’s the thing. Again, it is all about innovation inside of each one of those different areas. I tend to specialize in the explore or expand type of products, but I would say that there’s... There is absolutely in that exploit or extract, the idea of optimization is also a creative act, right? Because you’re taking something that’s very hard to do maybe, and making it very simple or reducing the complexity of a system so that it’s more resilient and robust. And so I would just say that is my job as this type of product officer person, is to help set up the environment by which those people can make those types of innovative decisions, of course, with tension to the other teams that are asking for other types of op- optimizations or things that they wanna get done, basically, or the goals they have. And so that’s really how I see my role as helping le- like leverage innovation or create systems that allow for innovation within a team.
[08:27] Christina Ellwood: I think it’s fair to say that many product managers are attracted to the role because of the structure and the predictability and the, the sort of certainty element of it. So how did you come to specialize in chaos, bias, uncertainty, and randomization?
[08:44] Chris Butler: Yeah. I think... So I grew up as, like, an apprentice to my dad, who was an art director, and this was during a changeover from X-Acto knives, spray adhesive, and typesetting, and so it was like a very antiquated world, at least of kind of the way that we end up doing design of a print ad, for example. And I helped him come into the new world of digital layout and QuarkXPress and Adobe Illustrator and all this stuff, right? And then I end up going into engineering school because I had an affinity for computers. I would say that I was gray hat-ish when it came to computer security stuff when I was in high school. And then I was interviewing at a bunch of different places, and I got job offers from, like, Motorola in, like, secure radio development or General Dynamics in, like, secure networking. And I could’ve become an engineer that was really focused on that, but I realized, I guess internally, that doing programming was not the thing that I loved. It was actually like figuring out the problems that help... That, that then we build the, the thing to do. And I interviewed at Microsoft, and they, for some reason, actually put me in the pro- the program manager track, which at the time was a weird in-between a product and project manager. And the interview question they asked me was: How would you design a washer/dryer combo for someone that is sight impaired? And this really sparked a lot of interesting questions for me, and it made me consider that, like, actually the thing that I wanna do is continue to solve these problems and to continue to help build things for people. And so that was like, that was maybe the first transition, was like going from this idea of just a very interesting, like creative kind of backgrounds into this world of like, how do we now build products that are for people? As I continued on that path and I did product management, tech evangelism, I did some like hybrid roles at like Waze and Kayak that were like BD and product, which I think business development is just like a different type of product management, honestly. It just ends up using other companies’ resources together. I started to see that we wanted to do the right thing. We wanted to make decisions that were right for ourselves, but we kept on being stuck in hierarchies, in power structures. In the way that we ended up making decisions, we would just make the safe decision rather than the decision that would push us into a better direction. And so all of that really made me start to study and understand not only human bias, right? So like the idea of system one and two thinking from the idea of thinking fast and slow. That was maybe like an entry point into that, like biases are things that, that are there. After a little while though, I started to, to realize that the reason why... And this, and again, this is not any... There’s lots of researchers that have done this. I just, it was like a personal journey that, yeah, there are biases. They exist for a reason, and it’s because we live in a world of like limited information, limited time, limited resources. And so that’s why we have biases, is to help us guide us through this world of like mult- multiple optionality. And then I started to think how might I actually leverage the way that we make decisions to, to more often make better decisions than bad decisions. And I think like better or bad decisions is maybe not a, a good terminology either because it’s really focused on the outcome, when the reality is like, because the world is uncertain and complex, you will make a decision and it will be wrong sometimes. And, and when I say wrong, it will just not have the impact that you wanted. And so from that perspective, we want to then leverage the best decision-making process that we can because that decision-making process will hopefully give us better outcomes overall rather than worse outcomes overall. But we don’t know for sure. That’s the uncertainty. And so helping people adjust and adapt to that, help- helping them be resilient inside teams and as a product in the environment of the marketplace, I think that’s where it, this kind of like journey took me to, was that kind of considering more, I guess, like meta or more abstract or cross-domain types of things that we can adjust to be able to help in those cases.
[12:26] Christina Ellwood: You’re helping them to rather than shut down the uncertainty, to in- include it in the dynamic. And of course, r- research has shown that the companies that outperform their competitors are companies that make good decisions. And those decisions are defined not just by what we initially think of when we hear good decision, which is the decision itself, go, no-go type of thing. It actually includes the process of executing the decision, evaluating the decision, and then, uh, you know, feeding back and, um, reconsidering whether or not the decision was good. Is that your experience as well? You’ve worked with a lot of different companies who have been very successful, and you’ve joined them at different points in their journey. Is that your observation as well?
[13:11] Chris Butler: Yeah, I think that teams will always struggle as they get larger and larger in the way that decision-making ends up being calcified, I guess you could say, right? Especially when you start to have structures where you have to pass a decision up the hierarchy. What ends up happening is there’s this paper from, I think it’s like 1985 or '6 or something like that, about this concept of dominant logic, which is that people that are successful, they end up using their past mental models for decision-making in current situations because they were successful, and that’s usually people that are higher up in the org. The problem is that filters out variance of ideas, variance of ways of doing work, and it actually ends up eventually creating a circumstance by which they’ll make the wrong decision or a decision that has a bad outcome more often than not because of those past experiences. And so I agree. I think that like teams that I’ve worked with that are very dynamic and, and adjust to the environment, they not only are moving fast, right? That’s one of the things. In a very highly volatile environment, you wanna keep trying a lot of things. But it’s also that we reconsider what are maybe considered to be like dogmas or norms or kind of ways of doing things inside of the organization. I think what’s really interesting and a little side project I’m probably not ready to announce, but I’d like to talk about maybe the discussion around what does it mean to basically use these types of LLMs inside of decision-making. And I think I would not want a system to make a decision on something that is not heuristic based. And what I mean by that is that, yes, LLMs can come up with a lot of very interesting content. They can do very interesting things when it comes to pseudo-analysis of what’s going on, right? But when they’re given a brand new situation, I don’t think that they can quite yet make a new decision based on new information and circumstance. They’ll fall back to the things that they were trained on or it was trained on, right? And so I think there’s this opportunity for how do we actually have something that is not emotionally involved, right? It, it’s not part of the hierarchy, it doesn’t care about power inside of the system, and it can actually ask questions or give examples of other cases that we should then cons- reconsider our current thinking. And so that’s why I think there’s some really interesting things around not only the idea of, say, summarization. As a product ops person, every new organization I, I go to wants a new status reporting infrastructure. And so there’s things there just about like how do we do a better job of like how that information passes not only within the team, but also to other people up the hierarchy and across the hierarchy, like how does summarization happen. I think there’s a lot of interest, like LLMs can be a huge benefit of that tool as a tool to be able to do that going forward. But then I would also say as a PM, being able to maybe not come up with the idea, because I think abductive thinking is, is something that a lot of product managers do, which is like taking a leap based on evidence rather than assuming some type of deduction back and forth between evidence and rules, for example, which is deduc- deductive reasoning. And so I think there’s like this opportunity that once we want to try to make this leap, right, as a product manager, we talk to a lot of customers, we see a market environment that we wanna now do something different. I think this idea of an LLM being available to take on different personas maybe to ask really hard questions and to ask things that maybe you wouldn’t have considered previously. And this is something that I’ve found over and over again, like I’ve used LLMs to be able to like, here’s a process that I’m thinking about implementing. What do you think is gonna be confusing for people about this process? Or what are-- can you please play out how someone might go through this process and the problems they would have, right? Or what are things here that I’m not thinking of? Or what are best practices that I should consider that are here? And again, you know, right now I would say about like 80% of the stuff is not helpful. It’s things I’ve already thought of. But there’s like a few things that are probably from some other domain that I had not considered that if I ask that question, it will make the, the decision-making I’m doing better in some way.
[17:00] Christina Ellwood: Yeah. I think LLMs as thought partners is a fabulous use case for them really, for the reason that you just articulated. But I think there’s another really interesting aspect of this whole AI generation for your work, which is that it’s changed all the rules or it’s removed a lot of rules, created uncertainty for people in a way that they don’t normally have in their operating life. So you have a provocateur in this you don’t know, right? We don’t know what it’s gonna do, we don’t know how it’s gonna fit in or whatever, which maybe creates the opportunity for you to have more elasticity in people’s thinking. Are you finding that?
[17:37] Chris Butler: Yeah. Yeah, I think that’s, I think that’s good because it pushes you in different directions. But I would also say that... So I did, um, some work at a company called Philosophie that was like a boutique design consultancy, where we were building basically like a multi-agent system, uh, through a conversational agent to help out these like field service operations dispatchers, which are people that are like taking in like a request for service and then dispatching service workers, but also trying to understand do we know what the solution to this problem is just based on the explanation, for example. We did a bunch of research. We were saying like, “Do you understand what this system is doing and do you like it or not?” And there are two domains of the, “I understand what it’s doing and I like it,” and there’s the, “I understand what it, it’s doing and I don’t like it.” And at first we thought of like the, “I understand what it’s doing and I like it,” as we built the right thing. And then the other one is, “I don’t like it,” we thought of as like job security issues. After thinking about it over a couple years and like giving that talk about that user research, I think I’ve reframed it in a way that like the things that I know what it’s doing and I like what it’s doing, it’s actually getting rid of the bullshit parts of my job, and it’s doing the busy work for me, and it’s doing the things that I really just don’t enjoy about my job because it’s not the creative part. And the other side, which is I don’t like that it’s doing this stuff, really were all things that are about human connection, and they’re about the idea of I’m finding meaning in my work to do this. And so I think there’s like a distinction that we should be making is that for all of these tools, a- an operations person, a TPM, an executive assistant, some of their job is to do summarization, for example. And the fact that they have to do that, and it, it is daunting work sometimes if you’re not the person that wrote something and then summarize it, you’re trying to get into the mindset of what that person wants. And so I would say like that type of work will start to go away because it will be done automatically. And so I think, yes, we have to start to then consider the organizational dynamics. We have to consider the creativity. Those are the human things that we’re really good at, or making exceptions. I would also say that humans are really good at phoning it in when appropriate. Like, we are lazy people. We are trying to optimize our energy at all times. And I think, like, our job is to also tell the system, like, “Hey, this is just not worth for you to do right now.” And so I think there’s-- I think that distinction between what is good human stuff versus machine stuff, I think, will become more and more clear as time goes on.
[19:44] Christina Ellwood: So what do you see as the future of AI for development teams at GitHub?
[19:49] Chris Butler: I think, like, we’re really focused right now on the developer experience. How do we make it so that as they’re thinking about algorithms in their head or the way to build something, that they can just make it happen and do it in a way that is very low cost to them and helps them, right, in a lot of ways. I think longer term, though, the reason why I joined GitHub is, and, and mostly because I’m focused on my role as, like, an operations person, is I think there’s, like, a real opportunity for the fact that we can start to do this for all of the teams that work together, and it’s not just engineers. So I could imagine a world where product manager comes up with a concept, does some time, kind of work with an agent to create kind of an idea, a conceptual idea that can automatically be built at least in a boilerplate type of level. A designer can come in and create mockups in Figma or something like that type of tool, and that updates the boilerplate. But then as the engineer is making changes to this work, there are trade-offs that they’re making. And so those trade-offs then being illustrated in a way that a product manager can understand inside of whatever they wrote, written or inside of, like, a design as a designer can understand, or even as the idea of what is a, what would a cust- like all, all the customer feedback we got, what would that potential customer think about this trade-off? It starts to then be like translation across all of these different practices in a way that, that, that ends up becoming this kind of like in between to be able to make sure that everybody understands each other, and I think that’s the future of, like, software development and, and that’s what I’m really excited about.
[21:14] Christina Ellwood: Mm-hmm. Yeah. That sounds great, actually.
[21:19] Chris Butler: Every doc that’s written is suddenly out of date all the time, and so that’s just the reality of the world today.
[21:23] Christina Ellwood: For sure. So as we wrap up, what would you like our listeners to take away from our conversation today?
[21:29] Chris Butler: Yeah, I, I think, like, for a lot of people that are trying out these tools for internal use cases, there’s two things. There’s, first, you should just try them out, right? Like, you should understand where they’re not working. Like, you should really keep an open mind, but you should try them. And then second of all, I think there’s, like, a lot of work that needs to be done inside of organizations from an IT, HR, and, like, responsible AI standpoint. Like, how do we make it so that it’s safe for people to experiment with these things? Because I’m betting you that every-- a lot of people that are do- using these tools today are doing so with ChatGPT on a personal account rather than some type of, like, internally confidential, secure type of system. And so I think I wanna push those teams that are, again, IT, HR, responsible AI, legal, to do a better job of actually creating those environments that are helpful inside of these organizations rather than just putting out a page that says, “You cannot use this.” Because they then cover their ass when it comes to legal demands, right? Yeah. So I think that’s the thing that I really want to maybe push on, is that we need to have people trying this stuff out, and we need to make it safe for them to do so.
[22:27] Christina Ellwood: Yeah, for sure. Chris Butler, Staff Product Operations Manager for GitHub, thank you for talking with me today on AI Realized.
[22:36] Chris Butler: Thank you so much for having me. Um, I’m excited for the conference, so, uh, thank you again.