Name: From Pilots to Scale: Operationalizing Conversational and Agentic AI for Real Business Impact
Uploaded: 2025-11-06T18:16:21.844Z
Duration: 58 min 15 s
Description: From Pilots to Scale: Operationalizing Conversational and Agentic AI for Real Business Impact

Transcript for "From Pilots to Scale: Operationalizing Conversational and Agentic AI for Real Business Impact": So today, we're gonna jump on in and start talking about the reality of pilots. They are exciting, but scaling is where real business impact happens. And it's not just the technology. It's about leadership, strategy, and execution. We're going to be covering some topics today. Why executive sponsorship and change management are critical for success. I've been on a few web webinars, over the past week or two, and everyone's talking change management. It's a hot topic. A fascinating harmony between voice and AI and why it matters. Brandon has some great thoughts on that, so I'm excited to hear from him. And real world insights, innovation outpacing execution and how leaders can overcome fear to move forward. We're gonna talk about what really moves the needle, dashboards, data pipelines, governance frameworks, and aligning AI with your broader transformation goals. Before we dive in, I'm excited to introduce today's speakers. First off, we have Ian Jacobs, vice president and research director at Opus Research. Ian's recognized as a thought leader in conversational technologies and customer engagement strategies. He brings deep insights into how enterprises can operationalize AI while staying true to their brand promise and delivering measurable business outcomes. Ian, welcome. Thanks, Paula, and thanks everyone for taking time out of your busy day to join us. Wonderful. And I know it's early for Ian, so I appreciate you, jumping on in early in the morning for you out out there on the West Coast. And then we also have Brandon Knott. He's our chief product officer here at IntelePeer. He leads product innovation for the company, driving advancements in conversational and agentic AI that help organizations accelerate digital transformation, reduce operational costs, and deliver exceptional customer experiences. Brandon, great to have you with us today. Thank you so much. Really excited to be here and excited with Ian to explore some of these topics. Wonderful. We're gonna jump on in and we're gonna talk about the topic du jour. Pilots, they're exciting. They prove the power of they they prove the concept, show the potential, and generate buzz. But the hard truth is many pilots never make it to full scale deployment, and that's because technical success doesn't guarantee operational success. I think we've probably have all seen that. Scaling requires organizational readiness, cross functional buy in, and a clear road map for change management. So, Brandon, from a provider perspective, what should buyers be asking upfront to avoid these pitfalls? And how do dashboards, data pipelines, and ROI tracking help move from pilot to the critical scale stage? Yeah. Great question. I think it's an extremely timely topic because everybody that I talk to has some sort of engagement with generative AI right now. Everything from personal to, POCs to broad implementations at their organizations. And so so it it's this is touching everybody's life. And I think as it's doing it, there are some misconceptions that that are helpful to call out. I think a lot of people are under the the impression that this is so easy. And it's easy because you can go get your OpenAI API key. You can start writing prompts. You can test it, and you can immediately see the results. And so you go from zero to happy path in in a matter of hours. And and it feels great because usually, you you have to spend a lot of time learning a coding language, understanding architecture, finding these deployment, methodologies that you're gonna use. But but here, you can get started so quick. And so while that's simultaneously a benefit in terms of velocity and momentum, It can also be a hindrance because all the things that you are talking about are are part of a reliable, robust enterprise grade system. Right? So so when we think about our POCs, we have to think about what we're trying to do and what and where we ultimately wanna be in order to do that. We have to prioritize. Right? And and because it's so easy to, adjust prompts, let let's say, and change the behavior, I've noticed a real tendency on POCs to to have scope creep in in kind of an exponential way compared to classic POCs. Right? Because I don't have to communicate to my product manager or my project manager who's gonna communicate to an engineer who's gonna do a change that's gonna go through QA, that's gonna come back around. No. I can just get in there. I can I can adjust the prompt? Right? Because it's easy. So what happens is the the the feedback, rate increases, and we stop prioritizing, and we start just continually to continuing to change different aspects of that. So prioritizing the things that matter. What are the changes that are gonna have a material impact on the outcomes that we're trying to validate versus what is a curiosity? What is a personal preference? I know we'll we'll talk a little bit about this later on. The these are how you keep, a generative AI POC on the rails. The other thing is you you mentioned the dashboards, measurement. The this is no different than any classic SaaS based or technology based POC or implementation. We need data. GenAI is kinda notoriously a black box. When you put in your prompt, you get a response with very little information about what's going on behind the scenes. But every turn that we have, every process that we do, especially if we're using different models, that that are, you know, more focused around chain of thought, we can measure the steps in between the uses of generative AI. And so what I what I highly recommend, anytime somebody's stepping into this space, is throw off as much data as you can and capture that in a way that you can go back and compare two ident you know, seemingly identical runs of a process and and understand what about the the first one was different than the second that netted you out this this different result. That's something that we spend a tremendous amount of time on. But because AI is such an exciting term and data has been around for a little while, we tend to really focus on this AI concept and and just kind of ignore the the the data element of the conversation. So these two should really sit hand in hand. And and finally, there there's a MIT study out recently about, generative AI pilots and what the challenges were. And one of their strong recommendations, was having external partnerships. And and the reason for that well, we'll we'll get into the reasons that he and I think can talk to him a lot better than I can. But people who engage with external, partners had a had a twice as likely chance of being successful. So if if you think about that and unpack it for a minute, just go back through the last couple of minutes of what I said, it's helping you to to narrow the focus. It's helping you to measure what matters. It's helping you to do these things that are classic and we know, but because this is so novel and it's so easy, quote, unquote, we we we we're kind of victims of of our own curiosity. So I'll I'll pause there, but, yeah, I think this is a great topic. I love it. And I I will admit I've fallen victim to the happy path, which usually leads to frustration. So, not all happy paths are good. You know, what role does aligning AI your AI initiatives with broader digital transformation and business goals play in ensuring that a pilot does install? Brandon, I'm asking that of you. Sorry. I'm Yeah. Great great question. You know, I would say that it's flexible, but it should be very intentional. Right? There are some scenarios in which you want disruptive thinking to occur within your organization. In fact, right now, earlier today, I got off a call. We have two different teams working on the same project with no, kind of boundaries on what technology. But one has to use one set of technology, one can use anything else. So the point is we're trying to understand the difference. We're we're forcing disruption on ourselves in order to to, explore blind spots in our own process. That's an example where where we are intentionally being open minded and disruptive. There are other scenarios in which there is no world in which we don't take something to market that does not use our own data pipelines. Like, it it must. They are so robust. You have so much additional analysis you can do on top of it. So there's another class of POCs that have to use those data pipelines. That is a hard requirement. So getting clear on what your intention is. Am I trying to disrupt myself? Am I trying to incrementally build? That's gonna tell me kind of how I wanna size and where I wanna pull from, from a technology standpoint as I do that that POC. I love it. I love it. So, Ian, from your research, what are the most common pitfalls that cause AI pilots to stall when you're moving to enterprise scale? Well so Brandon hit on almost all of the the key topics. And, Paul, you mentioned one that we're gonna talk about a bit more because it really is critical, and that is that the organizations aren't ready. We're not talking about the technology itself, but cross functional ownership. Right? CX, operations, risk, IT, legal, are not working together to figure out what this thing needs to do to satisfy all of their requirements. And all of those requirements are going to be in place when this gets operationalized at scale. So if you're not doing it upfront in the POC, you're never gonna have a chance to make that POC work at scale unless you've checked all the boxes and work with everybody. There's also not necessarily an enablement plan for frontline teams. Right here, we're talking about pilots around conversational AI doing service y and sales types of functions. They are either automating the work that human beings do, doing things that the human beings had never done, or working in concert and augmenting those human beings. In any of those use cases, they have a huge impact on those frontline teams. If you're automating the work that frontline teams used to do, they're doing something different. If you're not ready upfront with what that's going to be and how that's going to impact them, Again, your pilot kinda stalls at the pilot phase because you don't know the operational piece, the organizational piece. And then finally, this is something Brandon talked about, but we we've been I, maybe, me, maybe more than the rest of Opus. Been kind of thinking about it as evaluation as policy. Right? It needs to be upfront. Leaders need to encode the success criteria upfront. Otherwise, you're not gonna get a durable ROI. You need to know the things that you're trying to do as Brandon was saying. Like, what is the actual goal? Not we need to automate 40% of inbound interactions, but are we trying to disrupt the business and do things we hadn't done before? Are we trying to save money on this part of operations so that we can reinvest and make change over here? Right? That doesn't happen unless you have those success criteria encoded upfront. So those are the things that I'm seeing that that are causing pilots to stall, but I really wanna, again, just underscore that cross functional ownership. That tends to be the biggest one that I see over and over again. This comes out of a technology house with inside the organization. And, sure, it's cool technology, and it even does what it's supposed to do, but that doesn't help the organization because it runs afoul all these other parts of the organization's needs to present the unified brand to the out to the world and the individual customers. So, Ian, let me ask you a question. Have you I'm I'm inclined to say you've seen projects that have stalled or have just completely failed either because there there was misalignment. And we'd love to hear an example, me selfishly, because I kinda thrive on things like that. Or if you have one where maybe things were askew at first and then kinda people woke up and realized, hey. We gotta get all get in this boat and row together, in order for this to succeed. Could could you tell our listeners either, you know, kind of an example full of other I'm really glad you put in that second case because, yeah, what this is a brand that that eventually realized what was going on. It's a retailer, a big retailer, and they were piloting a voice bot for fairly simple tasks. They were looking for things like order status and returns. That's really all they needed, but that's a huge chunk of volume for them. So if they got this to work, it would be a big deal. And, technologically, you know, the early numbers were coming back great. They had containment. They had reduced AHT. CSAT was neutral, which in this case is good enough, probably. But, there was some surprise. The the the voice bot was capturing the call recordings and then transcribing them to improve the models. But to the customer, it was only playing a generic, this call may be recorded message. As you guys may have seen in the news, there's been some kerfuffle about, like, two party consent states where the wording should probably cover automated analysis and secondary use for that recording, and they didn't put that in there. So that was problem one. Problem two was this was doing order status and returns that the voice bot offering the text a return label to customers. Sounds great. Right? Complete automation. The problem is that the voice bot was treating prior purchase SMS opt in as sufficient. Like, I had already said, yes. You can SMS me because I bought something. But they didn't have a double opt in. Essentially, another one to say service messages also have my permission. Compliance flagged all this during, a pre rollout audit, but they weren't part of it upfront, so none of this was built in, which means they had to pause the program, go back and change all of the things that were wrong, and they lost I don't know exactly how long, but I know it was more than a month that they lost that they could have had if they just brought legal compliance and brand in upfront. So that's the kind of example that I'm talking about where you need all of these people in the room upfront. Again, the technology worked. It was doing what it was supposed to do in terms of, like, function. It was helping with orders or order status and automating returns. It just they couldn't roll it out because they hadn't brought everybody into the room up front. Yeah. That's that's a great example. And I actually believe, and it was probably some of these two consent states, that there may have been a lawsuit or two floating around out there. Yeah. Exactly. And that's why So yeah. They're so sensitive to this. Yeah. Yeah. So very interesting and a great example. Thank you. So it it's not just about the technology. It it's also a cultural shift, taking bringing AI onboard into any organization. Without top down leadership, both bottom up adoption and bottom up adoption, even the most the best technology can stall. Stall is the word of the day here, and we're gonna help you not stall. So, Ian, why is executive sponsorship critical for scaling AI initiatives, and how do you balance top down leadership and empowering teams at the ground level? Or conversely, how do people on the ground level who are like, this technology is really great. We need to get the c suite bought into it. How do how do people navigate those scenarios? Yeah. Well, I it's kinda obvious, I think, why executive sponsorship is important in that this is just what Brandon said. This is moving from pilot to POC just like in any technology. The thing that's slightly different here is once executives start to see some success, Right? You published an executive dashboard of the CX improvements and the operational KPIs tied to the AI interventions. They're gonna want more. So understanding where that thinking is going to come from upfront and having a sponsor there to say, okay. First, here are, like, the non delegable success metrics. Here's the things that we absolutely cannot mess with, and these things need to improve. Whatever that is, customer outcomes or the things that you can't mess with, like in my previous example, our risk limits. Right? Like, we we no matter what. If you've got the executive sponsor upfront, you can see how what you're gonna do, in this case, using my old example and just automating returns, the ROI you're gonna get, how that's gonna trigger thinking across the whole brand. But there's also, frankly, the power of the purse. I mean, like, let's be serious here. Having executive sponsorship means you're gonna be able to go from pilot to full on deployment because they're willing to say, okay. We're gonna throw in the money for the change management, for the training, and all the things that are required there. When you think about the top down versus bottom up question, I find that one really fascinating, Paula. I think that we can think about it, though, fairly simply. The top leadership have kind of two main jobs. They fund and they unblock. They're there to pull roadblocks out of the way. That's what they do. The line of business teams and the tech teams, right, they're the ones that own the feedback loops and the continuous evaluation, the thing that you can think of as a runbook. Like, how do we run this thing? That's their role. As long as each of those groups understand where they fit and how they complement the other, I think top down and bottom up is the answer. It can't be either or. It's an and. You need both. I think that, this is going back to Brandon's comment about needing partners. This is an area where partners actually help because they've worked with vendors before, and they have kind of proof of change enablement. Things like training kits or governance templates or or even just, like, evaluation artifacts. Things that aren't just the demo of the technology, but all the experience that they've had in working with brands like yours. How do we get that top executive who's somewhat shy about making AI customer facing? How do we convince them? What are the KPIs that that sort of flip the switch for those guys? I think that's really, you know, one of there are a lot of roles for partners, but in getting executive sponsorship and change management programs off the ground and running, having people who've done it before over and over again in different kinds of brands, they're gonna be an invaluable resource here. Yeah. I I really appreciate that, and I see Brandon nodding along with us. One of if so it's it definitely sounds like as you're talking to different providers, you should make sure that they have experience and they're willing to sort of kinda help you work that change management magic, shall we say. Are there any other things that you should be asking your providers or your partners as you're embarking upon this journey to make sure that the journey will actually align with your organizational goals. Yeah. So, I was sounding a little dismissive about an executive dashboard, but I'm not. Right? This is one of the things where I think provider tech providers can really help. I was saying, oh, you know, what are the KPIs that unlock the magic key in the executive's brain? It's not just that. It's how do you actually present that data so it's consumable by those executives, but also all the other organize parts of the organization that I was talking about that are required for the pilot. Great. If you're a technology team, you wanna see the data this way. If you're a customer service organization, you're looking for the outcomes that make your life easier. If you're a top exec, you're looking for the impact on the business overall on the brand. So things like sentiment and customer perception of their experience with this automation, and so on. So you need to be asking your potential provider, your prospective provider, how they can help you build dashboards to present almost the exact same kind of core data in ways that are consumable by different critical parts of the organization. It's not something that necessarily comes natural to a team that's out there developing conversational AI or a voice bot, but it is something, again, that's critical to the success. So if you can get your tech partner to help with that, you're, like, five steps ahead when the starting gun goes off. Oh, yeah. I absolutely love that. And I'm definitely one of those people who, I work the word magic. The math magic is a little sketchy for me. And I reached out to our implementation team at one point, and they basically threw a bunch of charts back at me. And I was like, oh, this is all very interesting. So I I spent some time, and I'm like, I think this is what this is saying. Let me confirm. So I I really appreciate that you brought that up, Ian. Brandon, it let's tie this back to ROI. How do governance frameworks and ROI dashboards help secure cross departmental buy in and keep leaders engaged? Very important. Yeah. I I I really agree with everything that Ian said there. Particularly, I liked that top down plus bottom up, narrative that that you two were talking about. And I think if we unpack it a little bit from a technology standpoint, why well, how is this similar to every other type of project, and how is it different? It's different because Gen AI is moving so unbelievably fast that it's a perishable skill, not because you're going to forget about it, but because what you know today is not relevant, a quarter from now. Right? If you had taken a break, last you know, this year even, you would have come back and said, okay. I'm ready to do my API integration, and somebody would have somebody would have said, oh, no. We're using MCP now. You know, we're just we're we're in a different pattern altogether than what you thought we needed to do. So there's there's no way you know, leadership in in an organizational structure have have too broad a remit to keep up with the pace of change. Therefore, you need an exchange format between leadership and those hands on engineers. What's a great exchange format? Dashboards. Right? Metrics, value, ROI. How are we achieving these goals that have been set out by the the hands on keyboard work that's happening? So I I see that as as the, you know, requirements and dashboards and metrics as really being the communication medium between these two parties that are worlds apart in terms of what their day to day focus are. The other thing you mentioned was governance. Right? And and agreed with with Ian, you know, wholeheartedly. But let's let's put this in the GenAI, context. Right? Models come out weekly now at this point, and and even major models are are revved, you know, on a quarterly basis. So so if you started your project in GPT three five, and it worked great, and you had it all dialed in, and you had your prompt sorted, And but it's been a while. And and now someone says, hey. We really need all the the capabilities of GBT $4.01. He said, no problem. And you just copy, paste, drop it in, and all of a sudden, it's very strange. It's strange responses are coming back. Maybe tool calling improved, but, the the the the format of the responses is somehow so worse. These models don't work the same way even in the the same lineage. Right? So what we this is not bad, by the way. This is not a weakness. This is very natural. It's just that the pace of change is so quick. It feels like it should be somehow more cohesive. So governance is is what gives us our test plans, our release processes. It's going that that is what we need to rely on to to push back. If someone says, oh, four one is so much better. Get off that t p t three five stuff immediately. You need you need ammunition. You need to be able to say, here's what we're we're going to be limited on. And finally, I think I was taking notes. The last thing I wanted to respond to with this top down, bottom up concept is, an example and, Paul, I know you love examples, so I'm I'm probably gonna try and go there. Let I'll look internally at IntelePeer, and our journey into using Cursor and use of generative a using generative AI to to code, has been awesome, and it has been a top down and a bottom up approach. We've had our early adopters that that were getting started before anyone really knew it was a thing. And then we had our CEO come in and say, you all are going to use Cursor. Show me the enablement plan. Show me utilization. How are we gonna measure this? Is it gonna be lines of code? Is it individual contribution? Let's talk about it. And out of that, we came out with a with a really good structure about how we are going to capitalize that momentum, but give that support and have our exchange format of those goals and metrics that then people know where they are in that process and and how to lean in more. I'm so happy you said that. I actually have a few cursor projects in my head that I'll be reaching out to you about. So thank you. I really appreciate that. Look. I'd like to talk you know, we're conversational AI company. I I really would like to talk about voice, which is probably one of the most natural interfaces we have, but it's so incredibly subjective. So when you combine voice with AI, it creates a powerful experience if it's done right. And it it I think it can go sideways quite quickly and probably more often than not. Brandon, my question for you. Why is the combination of voice and AI so compelling for the enterprise, and what challenges does voice introduce when you when you compare it to something like text based AI? Yeah. Great. This is a topic that's very near and dear to my heart because voice, at its simplest form, opens up a tremendous amount of opportunity when it comes to our ability to communicate. Right? When we have to text, we need a screen. That screen has to be a certain size. We have to have a keyboard. That keyboard has to work for whoever is using it. There there are inherent limitations there. Voice is the easiest kind of modality that we have or channel that we have in order to to communicate. But and there's some there's a big dot here. We have very different expectations about a conversation than we do a chat, and not the least of which is latency. Even as we're talking back and forth, our brains are super good at picking up when we're finished talking and then allow, you know, when I should start talking. And it have all happens very comfortably. But some of us are from the East Coast and, prefer a a quicker gap in between our communication. Some are from the Midwest, and I'm I'm perfectly happy with the longer silence. Right? I look. I'm I'm over, over indexing here on my example. My point is we all have very personal preferences that have been built up over decades of talking with each other. But when we type, it's very sterile. They're right. You get just a couple sentences. What's on the page is what on the page. There's no additional expressive interactions, no sarcasm other than what you can infer from the the words. Right? It is such a limited, medium, and we have no expectation of latency. I mean, when you're typing on a, you know, website that you just bought something from, you kind of expect that person's chatting with three other five other people, and your return message could be anywhere from ten seconds to ninety seconds. That's easy for generative AI to keep up with. It's harder to keep up with voice. So first, latency. Second, you know, just this notion of expressiveness. So how how do we bring that together? If I play a group of people of voice and I say, this is what we're going to do for your, virtual agent. I am 100% guaranteed that somebody in that room is going to say, excuse me. Would you you know, I'd like that to be a little bit more human like. I would like that to be which is totally subjective to their experiences, not wrong, just subjective. And somebody else is going to say, yeah. It should, yeah, it should be a little faster. And and another person's gonna say, little should be have some more umms and ahs in it. Right? And so we get kinda distracted by the characteristics of the voice with because it's being delivered without context. So what do you do? You work with someone who's an SME in voice tuning. Right? Or you you pursue this. And that person is probably gonna start off and they're gonna say, okay. What's the use case? Is it medical? If so, and someone's calling about a condition, maybe we the the the voice shouldn't sound too enthusiastic. Right? It should probably be a little bit more serious. Right? What's next? What's the speed that we want? What's the how verbose do we want a phrase to be or how tight? And we're gonna we're gonna set these requirements out according to the use case and according to the expectations and the brand of the organization, and and that's gonna be our first our first shot. But then I'm gonna go to that same room of people and I'm gonna say, alright. I'm gonna play you two voices. Here's what we're optimizing for. This is a serious situation. We're talking about people's health. We're gonna have a serious but upbeat tone. We're going to be very short and succinct because we don't we don't think this person has much time, and we're gonna go through these criteria. Then I'm gonna play the voice for you. In the context of those requirements, you're gonna shift from, hey. I'm an expert in voice because I've been speaking for forty five years or, you know, fifty years. You're gonna switch from that context to, I understand this person that we're trying to reach, and this voice, I believe, is going to be helpful or counterproductive to this persona that's calling in. So sorry. I'll pause there. I could talk about this for literally an hour. But it it's so important how we we tune voices to the audience, to the brand of the organization, and to the use case that that we're presenting. You know, I love that. And being in marketing marketing, I have had numerous conversations with executives about company's brand. Ian, I'm gonna ask this of you. You know, in addition to the AI and the voice element, it's really about making a system sound natural. And as Brandon said, making the systems represent and reflect the brand, the org the brand of a particular organization. Why is leadership alignment on branding critical for success when you're scaling voice AI? Well, I think Brandon touched on a whole a lot of those reasons, but, you know, I have a couple of different points here. One, voice in this case is your brand if you're presenting this. So, obviously, having the stewards of the brand involved is critical. And that includes the kind of stuff that Brandon was talking about, the prosody of of it. Right? All of the characteristics of how fast it's talking, whether it does the up speak at the end, whatever it is that you're you're trying to make it do. That has to encode the brand tone. But there's also a few other ways that executives looking at things like voice are going to have a feel for and and a say for, and also some things that they can dictate that the brand should do. And I wanna touch on some of those. So let's think about not the medical use case that Brandon was mentioning necessarily. Or even even in that, there's the concierge medicine where there's some desirable difficulty as part of the brand. And you're an elite member because you're paying however much it is to be part of this executive group who gets to see this medical provider. Right? That that part of it also might need to be encoded in the brand. But, think about the brands outside of the medical world where, the Birkin bag where you have to, like, get a referral to even buy a purse. They're instilling some kind of desirable difficulty into the brand itself. And in a weird way, that needs to be reflected as well in in your voice deployments, your voice plus AI deployments. And who's gonna stand up for that piece of maintaining the brand differentiation if not the executives. But the other thing that I think that executive buy in here is important for is you need to maintain kind of a living brand voice spec, if it is, like, golden audio clips that you have already created that define what is good, what is the right thing. You know, let's go back to Brandon's use case here where you're you don't expect or how do I say this? I don't believe that voice AI can provide empathy. Brands can provide empathy through voice AI if they set it up properly. If they understand, as Brandon said, that this person is in a sensitive situation and they don't have much time, and you've designed the voice that way to to meet those needs, you as a brand are being empathetic through the voice AI. And I think that's another area where executives can say, let's keep the customer in mind. Let's keep the the actual situate the lived experiences that they're having while they're interacting with with us in mind and keep that. And, again, have like, a library of the golden audio clips. We were talking about the happy path before, Paula. This is the equivalent. What is the happy example that we have here? Right? And that golden set of golden audio clips, they can be used to evaluate all of the future outputs as well. Now it's living, and it will change. As Brandon said, new technology comes in, but also brands offer new products and services. Government shutdowns change people's financial situations. Whatever it is, the world changes. And so it can't be set in amber. But executive sponsors can also then be the ones to say, hey. Government is shut down. Here's the things that we're going to be facing. Let's be sure that our voice is reflecting that reality back to our customers. Those are some examples. I love that. I'm gonna go off script a little bit. So and and I was thinking as you said government shutdown. I'm like, so that, like, for a financial institution, that really has impact upon their customers. Is it relatively easy on the fly to adjust someone's AI agent to kind of maybe have that little bit more of an empathetic tone in times of crisis, shall we say? Brandon, I'm asking that of you being the kind of the product guru over here. So I have two answers. The first is, yes. It is very easy. It could be as easy as just depending an extra 10 words to your prompt and your system prompt. Right? You are an agent helping people with refunds. At any chance you have, help assure them that they will get this as soon as possible. You know, something you know, I'm just making it up. Now we talked about earlier in our session about unintended consequences. So when I start changing parts of the prompt, I may have an impact or a conflict later on in the prompt or in the experience. So as soon as I say, you know, assure someone it's gonna happen as soon as possible, and then I make the API call, I see it's back ordered for the replacement, and I come back, it's gonna be two months. And the voice says, that's gonna be two months, and we'll get it to you as soon as possible. Now now now we have a conflict of of instructions. So short answer, yes. Long answer, let's not fall into that trap we said earlier where it's so easy and because it's just some some pros that we can make a change, we still have to think about it in terms of system thinking and make sure that how we tweak that personality is going to be congruent with the experience that comes through. Okay. Lots to think about. I love it. Well, I'm gonna kinda switch gears a little bit, but it's along the same topic, obviously. You know, a few weeks ago, gosh, maybe a month, month and a half ago now, it seems like it was yesterday. Time flies. But at Dreamforce, there was a theme that stood out, and I've seen a couple of articles since then. And it's basically that innovation is racing ahead, but execution is lagging. And many and while many leaders are excited about what's possible, they're hesitant to act. I I would say maybe they're they're risk averse. So, Dian, let's unpack that. Why do you think adoption is trailing innovation? And and do you perhaps in the research you've done, do you have any stats to back this up? And are there any fears or misconceptions that are holding organizations back? So let let's start with the tech side of what's holding things back. AI hype is cheap. Integration is hard. That that's, like, a big issue. Right? The data plumbing, the policy encoding, the workforce, the workflow orchestration that goes around your deployments, that that's hard stuff to deal with. So it's easy to do the innovation, hard to do the execution. You know, Brandon's example talked about just the one he just gave, hitting a back end system. Well, that data had to come from a back end system, and it had to be the right data, and it had to be within all the brand specs. And as Brandon then mentioned, the front facing piece, the the communication to the customer had to be right, but the underlying data connection had to be there. And somebody had to either give an instruction to an AI saying, here are the systems you have access to, and here's the things they know. Now go figure out when a customer asks this. Or they have the hard code and integration because in this one case, they actually did not want, you know, a a completely free flowing set of interactions. They wanted a deterministic flow behind the generative conversation. Right? Whatever the case is, that that integration piece is really one thing that holds people back. But you mentioned fear, Paula. That that's definitely true. We've talked about a lot of the reasons why brand risk. Right? Again, I just go back to Brandon's example. If you have voice AI that says we're gonna get it to you as soon as possible, two and a half months from now, There's some reputational risk that that sits there. But there's also the compliance stuff that I tough started with. Right? And those fears are holding people back. I think what we need to do is normalize experimentation here. Right? And one way to overcome a lot of those fears is getting the idea into everybody's heads that small frequent releases with proper guardrails beat the big bang transformations every day. Right? And that's an area where you can take that innovation that you want and are seeing in the technology and translate that into execution. This again goes back to executive leadership. If the exec top executives are saying, if you can't show me a 197% ROI in three weeks, If they're saying that, we got some work to do. If they're saying, show me a plan to get to that kind of transformation, we can start to say, okay. Here are the small kind of releases we're going to do, but they're gonna be frequent, and you're just gonna see constant improvement. I think that kind of chips away at the fear because the big ROI usually requires a big bet. The big ROI that is built out of a lot of small bits of ROI requires lots of little small bets, and so the risk tolerance is not exceeded inside of brands. So that that's not only why it's happening, but my prescription for for how to fix it, if that makes sense. That totally makes sense. And I I super appreciate that. I came from a b to c background, business to consumer background, completely different than b to b and and SaaS and tech. And when I first started in tech, I was like, what are all these releases? Can't they just, like, create the product and and get it out there to the market? And why do they have to keep updating? But then you take a step back, and after, like, a quarter of updating, it's like, oh, all those small changes led to this great solution. And and it it was very eye opening for me. So I I really appreciate that you that you brought that up. Brandon, I'm gonna toss this to you. What steps can organizations take to bridge that gap between vision and execution? Yeah. Great question. I'm I'm kind of reminded of that that phrase, you know, let let your reach exceed your grasp, you know, and and certainly vision should be out there. Right? I mean, come up with something big and unattainable by some year end. We're going to have launched a new product with a gross margin of, you know, x, using generative AI and and do this. Probably whatever you set now, it's a year or two years out, you'll over you know, you'll exceed it. But remember that that things don't happen in straight lines. Right? It's it's not just, like, step by step process to get there. You'll make progress. You'll have a limitation of the model. You'll handle it natively. And then the next version of the model says, hey. Look. We do it all now. Then you then you would have to go back, rip out that other stuff that you don't need anymore, bake it into the the core model. Right? And it's it's gonna be this, like, two steps forward, one step back, two steps forward, one step back, and that's okay. It's still going to be happening at a pace that we have never experienced before in terms of digital transformation and SaaS or or on prem or, you know, whatever the the the case is, improvements. Right? What I will say is do your best from an execution standpoint to pick an overall provider or architecture and and stick with it to the extent that it serves your needs. It's easy with all these models coming out to have this, like, swivel chair. Oh, no. We have to go use Croc now. Oh, no. That that had some negative press. We now we're gonna go GBT 5. Oh, that turns out that's super expensive. What can we do in house? We let's let's just so if you allow yourself to be whipped around in the process of of that experimentation and execution, you'll find that you're, like, one step forward, three steps back. You're you're constantly reinventing, and it's important to understand what's good enough for this objective that you're trying to get to and also have some comfort that things are gonna be moving forward so much anyway. Let's not try and suboptimize. Let's not try and get just a little bit more performance by switching a model today. Let's really learn everything that we can and know that three months from now, we're likely to to to do a model upgrade. Excellent. I love that. And I definitely have been whipped around in the beginning, jumping from model to model. So I really appreciate you bringing that up, Brandon. We are beginning to wind down, but before we wrap, I'd love to give the audience some actionable takeaways. So if, like, if you're evaluating providers or planning your next phase of your implementation, what's on your checklist? Governance, ROI, things we've talked about today pretty in-depth. But, Brandon, give me your top three steps companies should take when moving from pilot to scale and one piece of advice you'd give a buyer on this journey. Sure. You know, I think my first one is is gonna be a page right out of Ian's book. I think you you said earlier, you know, make sure you know what success looks like. And that is hard. You know, pick your favorite, analogy, metaphor, you know, start with the end in mind. You know, there there are all these things out that we have been inundated with, like, defining success, and, yeah, we don't. We just we just plow forward. It's it's so exciting, and we get that dopamine hit of starting the journey. But, really, what what does success look like? What are we trying to achieve? And it's okay if success is simply disrupting our current way of doing things. Right? It doesn't have to be some big, huge outcome, but just be clear on it and remind people what it is. The next is, I think, you know, that this notion of data we keep on talking about data. The quality of data is important. Ensuring that we are are are we have our milestones. We have our custom analytics. We have our telemetry. We we understand what data these systems should be throwing off, and we've got an appropriate mechanism. And there was a a great question in the chat, that I responded to around, you know, what are some examples of, what we should, be be feeding up in in terms of, dashboards and and analytics? So so get that understood because it is that exchange format. And the the third is probably, an offshoot of what a success look like. We've come into contact with scenarios in which we thought, like, a a containment rate was most important. And then after a while talking to the customer, they said, no. It's sheer number of things. And and these are somehow different. And can we what's more important, getting to the end in kind of a messy way and you may have to go back and clean it up later, but you got there? Or is it per is perfection required? Right? And so understanding kind of the, I don't know what the right word is, but under understanding just just how robust or quick or thorough or just binary, right, these results need to be makes all the difference as you're as you're getting through there. And and finally, you know, what sort of advice it's just data. You know? AI is awesome. That's what we're here to do. Don't forget about that other half to the coin, data. Make sure that you're you're you're focusing on it, tracking your outcomes. And guess what? If you do that well enough, you're gonna be able to start using that to inform the AI in that next chapter of your journey. I love it. I love it. Ian, how about you? Top three steps and one piece of advice. I think Brandon actually took basically all three of my three steps because they really are the top three. Right? You you need to know your outcomes and guardrails before you start. You need to stand up data and governance, whatever that is. Right? If it's unifying all your interaction data, if you're looking at the pipeline from sales, whatever the data is that you need to be pulling in, you need to figure out what that's gonna look like and how you're gonna offer governance. And then, you know, well, you don't know. Brandon used a critical word, which is telemetry. So if you wanna scale, you need production grade telemetry, I guess we would call it, to make it a production grade deployment, as well as the rollback paths. If you do go a step too far, how far are you gonna roll back to? Whether that's going back to the old model because you know it worked or or you knew its parameters anyway. Right? You knew how far it went. Right? Then the example that Brandon used of moving from GPT three five to four, It's it's the devil you know. You know how to work with three five, so keep roll back to there. Whatever it is, you need a plan for that. My advice, though, is is a little bit different than than Brandon's. I tend to think about customer experience as the three e's. There's ease, effectiveness, and we I used to say emotion, but empathy seems to be the word of the day, over overriding emotion. I'm not really sure why, but that that change happened. Whatever. It starts with an e. We're good. So what I would say as advice is you need at least one metric per e that you're gonna provide in your executive dashboard. Right? What is the impact on the ease of our customers doing the thing, whatever the thing is that we want our voice AI deployment to do? Are they effective at doing it? It could be easy but still ineffective. Again, go back to the sure. We'll get it to you as soon as possible, two and a half months. It's it's easy but not effective. So you need a metric there. And then around how the customers feel about interacting with this. Is it the kind of interaction that is creating in their mind the impression that you want as the brand takeaway? Right? Like, I think you need at least one high level metric for executives in each of those categories before you're ever gonna hope to get funding the next time and the next time and the next time. So that that's my takeaway piece of advice. Think about the three e's and how you're gonna measure that and distill it for top executives. Excellent. We have about two minutes left. I think you answered Brandon went in and answered questions on the fly. Thank you, Brandon. I appreciate your, skills. But and I think, Ian, you just addressed one of the questions, which was, you know, what what information to to display on a a dashboard. But we do have another question here that Brandon responded to, and I'd love to get some insights from Ian. And I hope I I get this correctly, but AI code generation is making it much more efficient to build proof of concepts. How can we apply that to feed an experimental mindset similar to Google's 10%, but it's a little different now where current POCs can take a month or two? If we can get that down to a week or two, then we could set up policy to allocate experimental time. Experimentation is where real innovation happens. So that's a little bit of a statement slash question. But, Ian, last minute thoughts, anything to say to this? Yeah. I think I was kind of hinting at the same idea when I was saying lots of small wins adds up to the big, big win. I think the same is true when we're not talking about the deployment, but in POCs. Right? We we have the opportunity, as the the attendee said, through AI cogeneration to create experiences really rapid fire. And some of those are gonna be great, and some of those are gonna be suboptimal. But having a flood of them and the opportunity to have a flood of them means you're gonna get to the excellent experiences faster. I think if you have the deployment mindset where lots of little wins gives you the big win, that then will translate into the preproduction thinking as well, because how are you gonna get those lots of little releases if not through lots of efficiently built POC kind of nuggets. Right? So I think start with the thinking on the the the customer facing deployed end, and then it will translate back inside the enterprise into that kind of innovation engine that that the attendee is in envisioning, which I love. I think that's a great idea. Excellent. Well, we're gonna end it on that. Scaling Conversation AI, it's a journey, and the insights you heard today can help you move from pilots to real business impact. Thank you, Ian and Brandon. Really appreciated all of your insights. I think this was a great conversation. And to those who joined us today, thank you. If you'd like to learn more or or schedule a strategic consultation, drop us a line@intellipier.ai. Thanks again, and have a great day. Thanks a lot. Thank you, gentlemen. Thank you.