Artifact-Scoped Agents: Stop Mimicking Job Titles
Most enterprises design AI agents to imitate roles: an agent that is a marketer, an agent that is an analyst. Chris Butler, Director of Product Operations at GitHub, argues that framing loses the plot. Agents are combinations of deterministic and non-deterministic instructions, not coworkers, and they should be scoped to the specific artifact they produce and run inside governed workflows with clear human handoffs. He walks through what his team built at GitHub, the trust boundary he recommends, why he automates dissent, and why the compounding advantage in enterprise AI is institutional context rather than the model.
Key takeaways
Agents are a combination of deterministic and non-deterministic instructions. Treating one as a full coworker is the wrong mental model, and it leads to scoping agents to job titles instead of to outputs
The artifact is the unit that matters. If no human interacts with what an agent produces, that agent is probably not valuable
Read wide, write narrow. Give an agent permissive read access, because context helps, then restrict its write actions to one narrow declared output channel. GitHub Next calls this safe outputs
Scoping each agent to one artifact means many agents can run in the same environment without stepping on each other\
Automating dissent works because there is no reputational risk in an agent being blunt. Chris runs an adversarial PM and a chaos monkey for process
The model is not the advantage, because frontier models diffuse fast. The enterprise context inside the harness is what differentiates you, which is why durable artifacts matter more than polished ones
About Chris Butler
Chris Butler is Director of Product Operations at GitHub, where he leads Synapse, a team building AI-powered tools that automate how cross-functional teams work. He has held product leadership roles at Microsoft, Facebook, KAYAK, and Waze, and he writes and speaks on practical methods for reducing bias and improving outcomes in product development. He also runs Liminal Practice, a consulting and learning-and-development practice focused on helping organizations implement agentic workflows.
Chris was a previous guest on this podcast. Click here to listen to his earlier interview.
In this episode
| 00:00 | Welcome, and why Chris is back |
| 01:55 | Why personifying agents loses the plot |
| 02:44 | Bloom, and decomposing the product manager role |
| 03:56 | Agents as the embodiment of an artifact |
| 04:53 | Keeping documents current as scope changes |
| 05:51 | Building a network between artifacts |
| 06:41 | ADRs, and why coding agents need decision records |
| 07:31 | Living documents and the how we work retrospective |
| 08:37 | Detecting strategy gaps from escalation themes |
| 09:44 | Status reporting, and where the human shapes it |
| 11:06 | How to adopt Agentics Beyond Code |
| 12:26 | Read wide, write narrow |
| 13:19 | Prompt injection and token scoping |
| 13:55 | Safe outputs, and one agent per artifact |
| 15:20 | A/B testing instead of evals |
| 16:05 | The adversarial PM |
| 16:36 | Rubber ducking 2.0 |
| 17:53 | A chaos monkey for process |
| 19:00 | Why a blunt agent carries no reputational risk |
| 19:40 | Team of Tomorrow, and agents in meetings |
| 20:13 | Room Clarity |
| 21:52 | Provocation versus adversarialness |
| 22:45 | Different meetings need different stances |
| 23:22 | Why the model is not the advantage |
| 24:01 | Model, harness, app |
| 24:53 | Why durable artifacts compound |
| 26:00 | What leaders should build today |
| 27:20 | Mapping where artifacts actually live |
| 29:00 | Resources |
| 30:14 | Leadership skill: asking good questions |
| 31:16 | The one thing to remember |
| 31:45 | Wrap-up |
In Chris’ words
“These agents are really just a combination of deterministic and non-deterministic instructions that are operating. The idea of personifying these agents so much to be like a full coworker is the wrong thing.”
— Chris Butler (01:55)
“If a human is not interacting with something that comes out of the agent, that agent is actually probably not that valuable.”
— Chris Butler (04:14)
“Agents can read a lot of context but then only take one action.”
— Chris Butler (15:20)
“There’s no reputational risk for these agents to be mean, because I don’t respect the agent. The agent is just a process automaton. But if I get this feedback, it really does help me a lot.”
— Chris Butler (19:00)
“Agents should not be coworkers, and the artifacts that they create, and the way that the humans actually work with those artifacts, are really the most important things.”
— Chris Butler (31:16)
Resources
Chris Butler
• Chris Butler on LinkedIn: linkedin.com/in/chrisbu
• Liminal Practice: liminalpractice.com/contact. His consulting and learning-and-development practice, offering workshops and implementation help
• Liminal Practice on Substack: liminalpractice.substack.com
• Agents Should Produce Artifacts, Not Impersonate Roles: His article for the AI Realized Now newsletter at airealizednow.substack.com, which this conversation builds on
• Podcast episode 43: AI Governance as Code
Projects and tools
• Agentics Beyond Code: github.com/chrizbo/agentics-beyond-code. Chris’s open source collection of agentic workflows for non-engineering roles: launch readiness, compliance review, status reporting, transcript parsing, process review, the adversarial PM, and the chaos monkey for process
• GitHub Agentic Workflows: githubnext.com/projects/agentic-workflows. GitHub Next’s framework, with the token-scoped permission model Chris builds on
• Safe Outputs documentation: github.github.io/gh-aw/reference/safe-outputs. The mechanism that restricts an agent’s write actions to one declared channel
• Automate repository tasks with GitHub Agentic Workflows: github.blog. The GitHub blog post introducing the framework
• GitHub Next sample pack: github.com/githubnext/agentics. Related but distinct from Chris’s repo. Worth linking separately rather than in place of his
Research and reading
• McKinsey Podcast, Trust in the Age of Agents. Nearly two-thirds of respondents cite security and risk as the top barrier to scaling agentic AI, and only about a third of organizations have reached governance maturity adequate for autonomous agents
• McKinsey, The State of AI Trust in 2026
• Why Your IT Project May Be Riskier Than You Think: Bent Flyvbjerg and Alexander Budzier, Harvard Business Review. Average IT project cost overrun of 27 percent, with one in six exceeding 200 percent
• Gary Klein’s pre-mortem: Prospective hindsight, and the grounding for the adversarial PM concept
• Vending-Bench: A benchmark for long-term coherence of autonomous agents, showing how agents decohere over longer tasks
• Red Teaming, Bryce G. Hoffman: Crown Business, 2017. Background reading on structured dissent
Also referenced
• Chaos Monkey and chaos engineering: The Netflix practice Chris adapts for process rather than infrastructure
• Room Clarity: His side project experimenting with agents that listen to meetings and surface decisions, assumptions, and risks
• Cognizer: Raised in passing as solving document discovery on the legal side. Confirm the spelling before publishing
Related AI Realized episodes and events
Agents Should Produce Artifacts, Not Impersonate Roles: Chris Butler makes the full argument in the AI Realized Now newsletter.
AI Governance as Code: From PDF Policies to Pipelines: Ken Johnston and Bob Rapp on making governance executable inside the deployment pipeline.
Governing Agents at Scale roundtable readout: The executive roundtable on governing agents at scale.
Agentic AI and Revenue Work: What Actually Pays Off: Christopher Penn of Trust Insights on agentic AI, measurement, and treating agents as untrusted contractors.
Frequently Asked Questions
-
Chris Butler’s argument is that agents should be scoped to the specific artifact they produce, not to a job title they imitate. A product manager’s work decomposes into tasks, and only some of those can be automated, so building one agent that is a product manager scopes the problem wrong. What matters is the artifact the agent produces, whether that is a compliance draft, a roadmap item, or a notification, and the human review that turns a draft into something official.
-
Because they are not people. Butler describes agents as a combination of deterministic and non-deterministic instructions, and says the early framing of an agent joining the team and being invited to meetings loses the plot. People have values and make decisions. An agent produces output, and if no human interacts with that output, the agent is probably not valuable in the first place.
-
It is Butler’s design rule for agent permissions: let an agent read broadly, because more context usually helps, but let it write only through one narrow declared output channel. He bases it on GitHub Next’s agentic workflow framework, which scopes access by token and uses a mechanism called safe outputs to restrict what an agent can act on. His status reporting agent, for example, can create exactly one discussion in one category with a fixed title prefix, and nothing else.
-
It limits the blast radius and stops agents colliding. Butler runs many agents in the same enterprise environment and says the reason that feels safe is that each is scoped to the single artifact it should be touching. He also flags the live risk: any repo that accepts outside issues is a prompt injection surface, and over-permissive tokens are the hardest part of this to get right.
-
It is an agent that surfaces counterarguments to decisions a team is making, effectively automating dissent. Butler is a proponent of red teaming in product work, and runs several variants including one that plays a blunt senior engineer. His observation is that it works precisely because there is no reputational risk: people do not resent an agent being harsh, but they still recognize the objections as ones real colleagues are probably thinking.
-
By annotating and updating the documents that go stale fastest. Butler describes agents that read meeting transcripts and update the relevant issue, generate decision records, and run a weekly retrospective against the how we work document to flag both automation opportunities and processes the team has quietly stopped following. Previously this needed a program manager riding herd on it full time.
-
From the harness around the model, not the model itself. Chris Butler notes that frontier models diffuse quickly, and frames the stack as model, harness, and app. The context of the enterprise sits inside the harness, and that is what differentiates your use of a model from anyone else’s. Most of that context is currently trapped in leaders’ heads or in a strategy document last updated six months ago and too polished to help anyone make hard choices.
-
Start by mapping where your artifacts actually live, which IT usually knows because it pays for the platforms. Butler points to the practical friction of drafting in one graph, say M365, while the content sits in another, say GitHub, and to the security and privacy questions that automating between them raises. His view is that organizations that identify their most important documents and have agents updating and annotating them will end up far ahead of organizations with content scattered everywhere. description
-
[00:00] Christina Ellwood: Welcome to AI Realized, the podcast for enterprise executives leading AI adoption. From tackling security, data, and operational challenges to navigating organizational transformation, AI deployments offer a unique opportunity to redesign our organizations from the inside out. I'm Christina Ellwood, your host for today's episode, and today we are talking with Chris Butler, the Director of Product Operations at GitHub. Chris has been with us once before, so we're gonna be catching up with him on what he's been working on lately. He leads a team called Synapse that builds AI-powered tools to automate how cross-functional teams work. This is internal uses of AI, and he's always got very interesting cases of implementation, adoption, and friction that, that he will be sharing with us today. And he's pushing a kind of provocative argument right now, and that's that enterprises are designing their AI agents wrong, which I think is gonna be really fun to talk about. So his premise is that instead of building agents that impersonate roles oh, here's an agent that's like a product manager, they should be building agents that produce specific artifacts And he's recently writ- written an article for us about this for AI Realized Now you can follow up and read a little bit more about it in that article. But today we're gonna dig into what that means in practice for enterprise leaders deploying agentic AI, and I wanna take a minute to just welcome you, Chris, back to the show.
[01:23] Chris Butler: Yeah. Thanks for having me again. I'm excited to chat with you.
[01:26] Christina Ellwood: It's always fun to have you on. You're full of interesting information and experiences because you're a very prolific user of AI, and you are working with people who are prolific users of AI. So you've got really two perspectives. You recently wrote about how the most common mistake in enterprise agentic AI is scoping agents to titles- Yeah ... instead of to outputs. Let's dig in on that a little bit.
[01:51] Chris Butler: Yeah,
[01:51] Christina Ellwood: I think- Talk a bit about that and why you think that's a, an important distinction.
[01:55] Chris Butler: Yeah. I think it's really important that when we talk about the way agents work, like I, I think at first there was a lot of speculative kind of thought about that they would join the team as another teammate, that th- that would be someone that you would like invite into meetings in a, as you would like another coworker. And I think that loses the plot a lot on the way that actually people are, individualized experiences. We have values, we make decisions. But the reality is these agents are really just a combination of deterministic and non-deterministic instructions that are operating. And so to me, I think the idea of personifying these agents so much to be like a full coworker is the wrong thing. And I think this is related to the way that jobs will change within organizations based on AI in the sense that, like the product manager, for example is a bunch of different tasks, decision-making alignment, like conversations. Those are all things that a product manager would do. But if we try to decompose the things that it does, that that the person does actually, there are some things that actually can be automated. So for example, one of the early things we did with Synapse is this project called Bloom, which was really about giving early feedback to product managers about the initiatives that they're building. And so that feed- early feedback would be in the form of a bunch of different types of we almost call them skills n- we call them skills now because it's really more about the idea of like we wanna give feedback on how aligned with the strategy is this, or are there gaps in this plan in some way. And the other part of it was that we wanted to start to give them a way to automatically generate drafts of the 10 to 15 different documents that they have to create when they're actually bringing something to market, right? And that includes go-to-market, like public roadmap item content or blog post announcements. But also like compliance. So how is this when it comes to responsible AI or privacy or security or something like that. And so- Or
[03:37] Christina Ellwood: governance
[03:38] Chris Butler: Yeah, or governance. Any of these things really when it comes to all of the different teams that a product manager has to work with. And what I started to see when we would use these different drafts of documents was that it was actually this like combination of the draft and then the human review to then turn in or push to other teams so that they could then do their work. And so from that perspective, agents for me really started to become that they were almost like an embodiment of an artifact to a certain extent. And what we really wanted to do is we wanted to be able to provide that this document is actually a draft, right? It's not actually an official thing until a human has reviewed it or someone has actually used it in some way. And honestly, like when an agent is going off and doing all of its research it's checking a bunch of different kind of data sources, and then it's generating something, it really doesn't matter that much until that artifact actually exists so a human can interact with it. And in fact if like a human is not interacting with something that comes out of the agent, that agent is actually probably not that valuable, right? Even if It's only occasionally that the agent will actually push an artifact. It's that artifact of a-- even something as simple as a notification is actually the thing that matters in this case. And so it's how am I using this information that was generated in part of my workflow? And so that is the future of the way that I think we should consider agents, and we should not consider them full coworkers. I think that is a really important aspect of this.
[04:53] Christina Ellwood: One thing that-- I used to be a product manager, so I'm always interested in your use cases because I lived the pain. It's been a long time since I did it, and I know a lot of the models in terms of how we go about doing it have changed, but- one of the things that hasn't changed is that what is in and what is out changes all the time. Yeah. And one of your early cases, as I recall when we were at the executive roundtable, was related to triaging- ... feature requests.
[05:18] Chris Butler: Yeah.
[05:18] Christina Ellwood: So if I've got an asset that's been drafted and I have a feature request that comes in that gets approved to, to be included in that particular, having a way in which the agent is smart enough to update the right documents- Yeah with the right information and create some kind of indicator of when it was most recently updated is super valuable because the recipient of it has-- needs to identify whether or not there's a big gap or a little gap between the draft and the version that I'm looking at. Is that a fair,
[05:50] Chris Butler: Yeah I think that's very fair. And I would also say that what we're actually trying to do, what we're using agents to do, is to not only help us build these artifacts in some way, because that helps us do our job, but also to build a better network between the different artifacts. And I think a good example of maybe where I would draw the line is transcripts coming from meetings actually hold a lot of context for what is being decided within the team, right? And it's not always-- It doesn't always make it into the issues or the other artifacts that are durable in some way, right? And so some of the automations-- And so I, I created this open source repo called Agentics Beyond Code, which is really a combination of all of the different non-coder workflows that I've been building internally and some that are more speculative that I think are interesting. We can talk about those. But that example of the transcript parser it really isn't the transcript that matters so much, it's more the fact that the transcript, when it's parsed, it can actually update a particular issue with a comment about what was discussed or decided at that point. Or there should be a PR. There's a lot of people, when they end up using GitHub as kind of their basis for all of this content, that they may create a bunch of different markdown files that are different ADRs or basically decision documents. And so being able to auto-generate drafts of those things, because that ADR is actually used not only by humans later, so like, why did we make this decision? What were the other options we could have considered? But when you start to include coding agents in here, that decision set is actually very important to provide context to the coding agent to then go and do the right thing. And I would argue that we're talking about documents like a strategy document things like how does this team do its work, like the how we work type of documents. Those are really valuable. And to your point they get out of date very quickly. And so how do we actually then use these agents to automatically annotate these different documents? So some of the workflows that I think are really interesting is like The how we work kind of-- there's a retrospective that an agent can do. It looks at all the transcripts, it looks at all the automations, it looks at the how we work document, and it says, "Hey, actually, here's one opportunity of something we could automate," right? And then two, "You actually don't do this process anymore, even though you say you do. And so we should remove it," right? We should say that this is now deprecated in some way. And so I call these living documents. And I think agents really help us do this in a way that before it required a program manager or a TPM to be really on top of it to constantly be editing those things and auditing them themselves, which is a huge time suck for those people, right? And so anyways, that, that's where I think it gets really interesting for agents is that if we consider them as like these agents are helping upkeep the how we work document, that actually is-- the how we work document is the most important part. We may add and remove agents that are doing different things on that, but it's the how we work document that we really care about.
[08:14] Christina Ellwood: Yeah. I-- and I'm really glad you brought up this sort of interstitial element, right? There is this kind of fabric that exists between all of the systems that connects them together but also acts as the place where we do the the decision-making about updates. And so I'm really glad you brought that up. And so do you have an agent in your Agentics Beyond Code GitHub repo that is specifically for that interstitial component?
[08:37] Chris Butler: Yeah. And so being able to-- There's a couple different ones that are in there w- using things like transcripts to then update the how we work document or looking at decision documents that are generated from a meeting to then say, like, how many decisions are we making that are in alignment with our strategy document, and which ones are in misalignment with our strategy document? And I don't mean misalignment in a bad way necessarily, because a lot of the times teams that are on the ground they're seeing much more than their executives are because they're so close to the work. They may be making decisions that show that there's a gap in our actual strategic understanding of the world at that point. It's almost like the idea of, when we do escalations up in a hierarchy to leaders within the team. If you start to look at the themes of those escalations, which is another thing that agents could start to pull out of these different escalation documents that come out, we can start to say, "Hey, actually, there's a gap in the way that people do execution because there's nothing in the strategy that says how they should make this decision." And so that's the reason why this theme of escalation keeps on coming up, is because we need to either better communicate the strategy or refine the strategy in some way. And so I think those are really important. And so I would point to yeah, any of these ones that are like the transcript to issues is a really important one for sewing all that stuff together. The things where it's doing kind of the process review and tran- and like how we work reviews on a weekly basis. These are all things that are on different cadences as well. ... And so I think that's really valuable. I mean- Yeah ... I think one that I just made some updates earlier this week on was really this how we do status reporting is something that requires a lot of legwork by the agents. You wanna go and look at all of the different issues and PRs that have been updated in the last week. And we wanna then match that to our strategy to say, "Hey, here's a summary draft of what happened in the last week," so we can send it to a leader. But there's a human shaping component. So the idea of is our status red, yellow, green, right? Is it really red or is it just yellow? Because I'm shaping this for the leader in, in, in whether I want them to actually get involved or not. And so what this does is this actually triggers the creation of a Google Doc that is the draft so that humans can collaborate, because that is a better place to do that than maybe in an issue. And it cross-links those two different documents that when you resolve the comment on the Google Doc, it tells the agent that now it can publish the actual status that has been, like, drafted by the agent and then reviewed by the and formed by the human. And then it automatically updates the people that need to be updated on Slack. And so- Wow ... that's an example of where- ... what we're doing is we're trying to make sure that the agents are doing the parts they should, and we have, want the humans to do the parts they're doing. And the human gets the final kind of sign-off of, "Yes, this is ready to now talk to everybody." And we just take away the toil of them having to copy and paste stuff to different channels and do all that work.
[11:06] Christina Ellwood: Wow that sounds amazing and super valuable. And I'm excited to mention to the audience that you've open sourced- Yeah ... this Agentics Beyond Code at, in GitHub, right? Yeah. So how would someone like myself- Yeah ... take advantage of the Agentics Beyond Code repo?
[11:24] Chris Butler: I think the first thing is that this is fairly GitHub-centric, at least right now. And so it does look at the fact that you're keeping key documents mostly in GitHub repos, that you're using things like projects and issues that you're using kind of PRs. And so that is one thing that it does assume right now. But I actually ended up creating skills for almost any one of these, like IDE or Claude Cowork or Codex where you can basically have a conversation with the skill to say, "Here's the type of problem I'm trying to solve right now," or, "Here's the type of team I have." And it will help you basically pull all of those workflows over, compile them, and put them in the repo that you need them to be in. And so I've tried to make it as simple as possible so that you can piecemeal pull the different processes that are most helpful. And actually, the skill will also tell you probably first thing to do is these three skills for the problems you have, and then here's the next, like... Or sorry, agents for the problems you have, and then here's five more agents that once you start to have data running through it, here's the next ones to do. And and also just by the way, I'm always happy to help people if they do need help implementing this inside of their organizations. It's something that I do through a, both a consulting and learning development practice standpoint.
[12:26] Christina Ellwood: That's actually really good to know, and we should add that to the show notes of how- Yeah ... someone would get in touch with you if they wanted to do such a thing. Absolutely. You also make an argument, a design argument actually, that the- agent's trust brough- boundary- ... should be that it can read widely, but write only through a narrow declared output channel. Now, governance is a huge topic right now. Yeah. And in fact, we're doing an executive roundtable on the 17th of June on this very topic of governing agents at scale, and this strikes me as one of the pieces of advice that would be helpful for the Yeah attendees and the participants to discuss. So-
[13:04] Chris Butler: Yeah, and I'll be there on that as well, so I'm excited to be part of that conversation. Yeah, and I think The hypothesis here is that when we start allowing agents to use tools or to edit artifacts inside the enterprise environment, there's a real possibility that it will do something wrong. And that's because it's a very non-deterministic system, right? Even though it's following instructions because of context and because of also security concerns. So if you have, for example, an open source project that allows people to submit issues, there is actual a possibility of prompt injection for any of the agents you have on that particular repo, right? So what I would argue is that we want agents to be able to go out and look at things. And actually this is all based off of a security model that GitHub Next, they created this agentic workflow framework. And part of it is that it is scoped by tokens. And so that is something that you say, "Yes, you probably want to have access to these different repos in this different way." And you want it to be more permissive because more context in this case from those artifacts is helpful a lot of the time. And then they have something called safe outputs, which really restricts down the type of thing that can-- it can take action on inside the environment. And I think that's really important because if I have this status reporting agent, for example there's really never a time that the status reporting agent should change the status of different items because it's not appropriate. Like it's-- even if the transcript one will update these with comments, that is helpful, but this status reporting one should not do that. And so in this par-particular case, that, that agentic workflow is actually scoped down to creating one issue, one-- sorry, one discussion in one category with-- that starts with this title of basically weekly status. And the reason why that it makes me feel much safer is that even though I have many different agents that are out there doing stuff inside of this enterprise environment, I know that the likelihood of them stepping on each other is much less because they're all scoped down to the one artifact that they should be dealing with. Now, there are possibilities for agents that pick up off of that discussion that was just posted but that then do other work, for example. And so this is another job for another type of agent, which is like trying to look for the way that agents may interact and look for actually bad interactions. I would argue that this comes from like the GitHub Next team, that it is actually less about evals, especially when we're talking about internal tooling, and it's more about this idea of kind of A/B testing your way towards the best set of instructions, the best set of models and for your environment. Because your environment also changes over time. Your information environment is constantly evolving inside of an enterprise organization. So anyways I think it is really powerful to say that agents can read a lot of context but then only take one action, and that's one of the reasons why I think the model that agentic workflows from GitHub Next is using is going to be a model that a lot of people use in the future. Now, it's still really hard to deal with like tokens. That is probably the biggest pain to actually deal with for these things, and I think there is the possibility in-- for people being too permissive with the tokens that they actually provide to these agents. I think there's a lot of work over the next couple years where we're gonna dial that in more. But for now, I think as long as we're restricting the impact on the artifact environment, I think it's much safer than it would've been just having like random tool use for an agent.
[16:05] Christina Ellwood: Okay. Good. Thank you. That's a very clear description too. One of your other provocative ideas is something you call the adversarial PM.
[16:14] Chris Butler: Oh.
[16:15] Christina Ellwood: And this is an agent that finds two or three of the most consequential decisions of the week for the product management team, and then posts structured counterarguments. Yeah. So basically it is automating dissent.
[16:29] Chris Butler: Yes.
[16:30] Christina Ellwood: So talk about the inspiration- Yeah ... behind building that and the impact it's had.
[16:36] Chris Butler: So I've been doing a lot of workshops with different teams, both inside, basically helping them understand how PMs should utilize AI. And I think there's this loop that I've started to discover, which is that I will start with maybe an idea, a very messy collection of thoughts that I'll maybe like just, talk out through and then put into a, some type of tool. That beginning requires actually the LLM to... It's basically like almost like rubber ducking 2.0 is what I'm calling it. So this idea that like I'm talking to something and then it's giving... Because I'm externalizing my knowledge, it can also then do intelligent critique in some ways. And so adversarial stances are really helpful for people to hone their ideas. And so I output something, I get critique, and then based on all of the kind of things that I respond to or ignore, right? Like it really matters most what the human's doing there. I then have the system to create an artifact, and that could be a document, that could be a prototype, and then I critique that. And so this idea of like back and forth creates this like cycle of context collection that I think is really helpful for PMs. And so the adversarial PM is one. I'm a huge proponent of like adversarial thinking and red teaming and stuff like that within the world of product management because there's a lot of the time that even ourselves or our team may be not thinking through or thinking about the possibilities that we should be. And so the adversarial PM is really to push people's thinking and make them think harder about this. And I have lots of different versions of this. I have one which is like a, the like grouchy, mean senior engineer and telling you your idea is a bad idea. I have another workflow that's in that Agentics Beyond Code, which is I call a chaos monkey for process. And what it does is it looks at all of your kind of things that are happening inside the information environment- And then it starts to try to detect if there's too much like stasis or groupthink happening, and then tries to stir up the process in some random way. And so this comes from Chaos Monkey from Netflix and chaos engineering, which is like Google SREs really talk about this stuff a lot. It really is the fact that there are plenty of times where we start to fall into this kind of like groove, and that, that's helpful when we have like good thinking or thinking that is ideal. But there are times where we don't know that we're in this groove, and it's actually not helpful to us. And so all of these different types of processes I think are just trying to enrich the environment by saying, "Hey, your idea is not always perfect, and so let's try to help you explore what are the possibilities." An adversarial PM as like an agent in your environment I think is really interesting and pretty helpful. The-- even the rude like Q&A one, which is the kind of mean engineer that we had inside of GitHub, people would like laugh at it because it is really mean. And- ... but at the same time... And they would say "Actually, these are probably things that people are thinking, and so I do need to have a good answer for this." And the thing that's so interesting about this is that there's no reputational risk for- ... these agents to be mean. Because I, I- ... don't respect the agent. The agent is just a process automaton. But if I get this feedback, it really does help me a lot.
[19:24] Christina Ellwood: Yeah. It makes good sense. And is the adversarial PM part of the Agentics Beyond Code? Yep, that's right.
[19:29] Chris Butler: It's a, it's
[19:30] Christina Ellwood: another one of the workflows that's in there That's very cool. Yeah, I think that's very cool and really one that's super helpful. At the adversarial model has I think been one of the most valuable unlocks for teams.
[19:40] Chris Butler: Yeah.
[19:40] Christina Ellwood: That's right. So in your Team of Tomorrow talk, which was a talk that you... W- people can find that Team of Tomorrow on YouTube. I'll put it in the show notes as well Yeah ... you showed a future where agents sit in the kickoff meeting alongside product managers. So this was something hearkening back to the beginning of this discussion today- Yeah where you're basically saying, "Let's not personify the agents. Let's focus on the outputs." But in this case, you actually have an agent that participates in meetings. Talk about that, and what value is it bringing, and what advice do you have for people who are thinking about using such a thing?
[20:13] Chris Butler: So I have a different pro- kind of side project called Room Clarity, which is really about experimenting with this and what it means to bring an agent into the meeting is not that the agent is sitting there as another attendee that is automatically part of the conversation. It's listening in on the conversation, and it can actually, based on... And I f- I think what's really interesting here is that there's actually multiple types of agents you might wanna have in the meeting. You might wanna have a facilitator that's helping keep the meeting on track. You might want to actually have an assumptions agent that is like, "Hey, there's an assumption here that we should probably challenge." Or even the idea of a y- maybe even the adversarial thinking one, like a pre-mortem type of thing. And w- the way that I think is the ideal model right now is that actually we want the humans to be able to have the conversation that they need to have, right? But we want them, if they notice that one of these agents is And the way it works is a queue of things that are popping up as it's monitoring the meeting, and this is something that you would use as a dashboard for your meeting basically. And it starts to collect decisions that are starting to form. It starts to look for a- action items and risks and things like that. But the agents can also then say, "Hey, I think this is a potential problem." What's really interesting is that if the humans then talk about it, then it is actually valuable. But if it's not talked about, then it should just be discarded. And so I think that's what I'm saying is that in the end what we're trying to do is the artifact in this case is actually probably the set of decisions and the things that the human talked about those decisions, but they're provoked in some ways to talk about things maybe more broadly or more directly or more de- in a detailed way based on these agents trying to push them in a different direction based on what that... basically it's like a skill is the way to think about it, and these skills get to exist within this meeting. And so-
[21:52] Christina Ellwood: So it's a provocational a- agent as opposed to an adversarial agent
[21:55] Chris Butler: Y- yeah, I would say provocation is something... And provocation adversarial, I think it just depends on the tone a little bit.
[22:01] Christina Ellwood: It also depends on the context. Yeah, that's true too. Yeah. In this case, if you're asking it to argue with you, you obviously are looking for adversarial. Yeah. But in the context of the meeting, it's provoking- That's right ... the opportunity- That's right ... to have a discussion about something that it's discerning that may- That's right or may not be significant. So-
[22:13] Chris Butler: and again, the end goal is not necessarily what the agent says, it's how the humans react to it, and then the log of what decisions are made or what next steps need to take place that the humans identify as i- as valuable. And so again, I think that's why it's trying to join these two ideas of the artifact is what mattered. In this case it's the decision record, right? And the conversation around the decision. But then it's also trying to bring them in as a kind of provocational or adversarial, and I think that different types of meetings require different stances from these agents, right? One of the things I'm starting to think about more is in the... if we do a user interview, I think that is a very different thing. You might wanna have agents that are helping you, not gloss over something that you should've asked a better question about. Or a sales meeting where, could you ask this question to get more information, right? Or if you're in an architecture meeting, are we considering the actual architecture implications? So I think there's something there about the type of meeting and the types of provocations or adversarialness that you need, or help in general, right? That these agents could start to provide. But I think the model is gonna be really key, the u- the user experience model. 'Cause again, they should not be, like, speaking up randomly in the meeting, but if someone notices it and they say they wanna talk about that actually is good signal that agent brought up something that was valuable.
[23:22] Christina Ellwood: Yeah. It seems like there's probably more refinement that will happen in your- Absolutely ... thinking as it gets used more in context too. So y- you've also talked about how frontier AI models diffuse very quickly. But task libraries, rubric sets, and decision archives don't. And you're saying, I think, that the compounding advantage that we gain with AI itself is about that institutional memory and the artifacts- Yeah that they create rather than about the model. You wanna talk a little bit about that?
[23:50] Chris Butler: Yeah. I guess I-- with all of the churn of new model releases, even the Mythos model that just came out recently from Anthropic there's constant improvement in the way that these models can be interacted with. And I think a lot of people are talking about this as three components, which is the model itself, the harness around it- ... which includes context, and then the user experience, which they call app, right? And so I think what needs to go into that harness is actually the context of the enterprise, and that is really the part that differentiates your use of the model from anybody else's use of that model. Now, there are definitely aspects of the harness that are like, how are we prompting it inside of this? Or what skills are being used? Or what-- how are these agents being configured and instructed in some way? And so I see that as part of the harness. And then the app part of it is really how do I then engage with this artifact? And it should ideally be in a way that makes sense to me. That example from the status reporting, the Google Doc is a much better way to actually do collaborative editing of something across multiple people rather than, say, like in a PR or something else. And so that's why I think that context though within that harness is really your differentiator when it comes to the way you do your business. And so those durable artifacts become more and more important over time especially because if they're mostly caught in people's heads right now, like a leader's head, like usually there is a strategy document, but it hasn't been updated for six months. It's highly polished, so it actually doesn't m- it doesn't help you make hard choices. But every single time that product leader goes in and does like a product town hall or does a prioritization meeting or, an escalation meeting, they are actually conveying their model for that. And so what we need to do so that agents can actually do the right work inside of these enterprise environments, we need that context written down somewhere, and not just like ephemeral in someone's head or stuck in a transcript recap somewhere. And so that, that to me is why these these core artifacts are just so important.
[25:33] Christina Ellwood: It goes back to this idea that there's corporate intelligence in the way in which we operate, interact, make decisions, and communicate, and that's part of the reason that companies are increasingly using private models multiple models, private- Sure behind the firewall as a part of that sort of corporate intelligence. And you're just referencing that in the context of the the harness section, and I think there's a lot of refinement that we're gonna see. Yeah. And in fact, I'm wondering what you think the the leaders that are listening today, what should they start building today, especially since we just had some new models come out from Anthropic yesterday, what should they start building today so that they have that compounding advantage a year from now?
[26:12] Chris Butler: I think one is- Identifying the way, like where different types of artifacts exist today. And they probably IT knows this because they have to pay the bill for those platforms, right? Inside of a place like GitHub, we will have a huge graph of information inside of GitHub, right? We'll also have a huge graph of information that's in M365. And there's probably like a couple other places like, for example, any type of sales tools or customer support tools, right? And so the question becomes how do you start to map these things to understand where is it best for a human to actually do the work on this, and where is it best for an agent to take action and create those artifacts in some way? And so I think that is where, with status reporting, we constantly see this weirdness between M365, which is where we wanna do our document drafting and shaping, but the content is all in GitHub. And so being able to then offer automation capabilities between those two different graphs is actually a huge problem for enterprise. And the reason why is because of the fact that there are security and privacy issues that will come up for that. Now, choosing a platform that does allow you to guardrail the agents, I think helps improve that in a lot of ways and makes it a lot less likely to actually do something bad. But that to me is the most important thing, is that we-- where are these key documents that we think are gonna be like, help inform the context of everything, right? And that could be even the employee manual potentially, right? So we need to identify what do we think are these durable things. Now I am also like working on another weird side project, which is about like, how do you map, say your Google Drive, your M365 to understand what documents are actually starting to rise in importance, which ones are hub documents that everybody references, which documents are stale. And maybe from an ontology standpoint, what's the difference between all these documents that are out there? Are they in alignment or are they not in alignment, right? And so I think there's a lot that, you know, providers like Google Drive or M365, and I think a lot of maybe homegrown solutions that, like the one I'm talking about right now, are gonna start trying to do that idea of understanding the actual ecosystem. Because I would say that like the SharePoint kind of unless you highly curate your SharePoint or your Google Drive like folder, like it's not very, it's not very helpful to be able to just look at it because you don't see the activity on these documents. You don't see how these documents are relating to each other. And so I think that's something that's gonna be worked on a lot. And I think actually if enterprises start to focus on what are the most important documents and they track them in a way that is, is actually helpful and have agents updating them or annotating them, I think that gets them a l- like really much further along than a lot of people that just have scattered content everywhere inside their ecosystems.
[28:43] Christina Ellwood: Yeah. You might remember Randy Friedman, who's with Cognizer, and that's one of the problems that they solve on the legal side, right? Is where are all the documents in the enterprise, right? Definitely. And ca- categorizing them and doing the various work associated with making them available to use properly by an agent. So moving on to our closing section here, I'm very interested in what resources that you would recommend today for listeners who want to explore artifact scoped agents, your Agentics Beyond Code project, or other things that you are finding really valuable today in your work in implementing AI.
[29:20] Chris Butler: I would say th- definitely go to the Agentics Beyond Code repo. There's a lot of kind of examples in there of ways that, that I've tried to create valuable a- actual agents within our e- our environments, and so I would say go check that out. I think it is really interesting to be able to adopt some of these patterns to other places as well. And if you are building with a different platform than, say, GitHub for your for where you store a lot of your context, I think that these are patterns that people will use. They just have to figure out, and they can use these instructions and the way that we're, like, splitting out instruction and policy into helpful ways. I think those are patterns that they should be able to reuse in almost any kind of technology environment. So that would be my main pointer is basically to go and check that out. And then if they do need help, a- again, like I said, I'm always happy to to help out if they need to get some guidance on how to do this inside their organization.
[30:09] Christina Ellwood: Sounds like that could be a really good workshop for us to offer to our community as well. Yeah. So we should talk about that. And in the, in this phase of the AI revolution, you've been a leader a long time. I'm curious, what leadership skill do you find most valuable today?
[30:24] Chris Butler: I think right now there's a huge rush towards being able to just build a bunch of stuff. I think the thing that we are missing a lot of the time is the ability to ask really good questions. And I mean that both in the sense of like of your teams asking good questions of them, but also when we talk about like understanding whether this thing is actually more valuable or not. What are the questions we should be asking the people that are using these tools? And I am a big believer in like user research, especially from a qualitative standpoint, to understand what's going on. And so I think being really good at pulling out unbiased kind of... or in the-- from the point of view of that person, like their true opinion about something, I think is really important because there's gonna be lots and lots of change inside the environment over the next couple years. And that just means that people are gonna be way more heads down just like trying to just do execution, but we're not actually gonna understand each other more. And so that to me is a really important leadership skill.
[31:16] Christina Ellwood: I like that. Yes. And it's a, it's not one I've heard mentioned before, but I think it's absolutely true. If our listeners remember just one thing from today's conversation, what would you like that to be?
[31:27] Chris Butler: That agents should not be coworkers, and that the artifacts that they create and the way that the humans actually work with those artifacts are really the most important things. And so you should always be thinking about that stuff first rather than, "Hey, let's try to create this random thing that we think might be good," but not really think about how the human deals with it. That's what I'd say.
[31:45] Christina Ellwood: Okay. Great. Chris Butler, director of product operations at GitHub, thank you so much for joining us once again on AI Realized. We really appreciate your time.
[31:53] Chris Butler: Thank you so much for having me, and great talking with you.