{"type":"rich","version":"1.0","provider_name":"Transistor","provider_url":"https://transistor.fm","author_name":"The Experimentation Edge","title":"ServiceNow's Customer Zero approach to AI experimentation","html":"","width":"100%","height":180,"duration":1666,"description":"Summary\nServiceNow runs its own platform on itself. Ashraf Karim, Senior Vice President, Connected Customer Experience, owns everything that happens after a customer buys, and her team is Customer Zero, testing every product and capability internally before it ever reaches a customer.\nShe joins Ashley Stirrup to unpack what that vantage point teaches you about experimenting with AI. The headline lesson is not a technical one. ServiceNow built agentic skills that worked, and users abandoned them anyway, because a task that ran five to fifteen minutes broke every expectation people had about what a machine should feel like. The fix was not a faster model. It was telling the user what the AI was doing while it did it.\nAshraf also gets into the harder measurement problem underneath all of it: how do you evaluate a nondeterministic system? ServiceNow's answer is an internal LLM judge scored against a golden data set, plus a customer effort score that catches what CSAT reports too late.\n\nChapters\n00:00 What customers actually care about\n01:14 Meet Ashraf Karim of ServiceNow\n01:39 Owning the entire post-sale experience\n02:08 Why ServiceNow runs as its own Customer Zero\n03:02 Lessons from Google, PayPal and Verizon\n07:11 The agentic skills users kept abandoning\n09:29 Latency, expectations and the feedback loop fix\n12:29 Users wanted the outcome, not the process\n14:49 Designing experiments that can afford to lose\n18:13 Using an LLM judge on nondeterministic models\n21:48 Customer effort score as a guardrail metric\n25:03 Where experimentation at ServiceNow goes next\n\nTakeaways\n- ServiceNow is its own Customer Zero, running the platform on itself and testing every feature internally before customers see it.\n- The agentic skills worked. Users still abandoned them, because a five to fifteen minute wait broke their expectation of what AI should feel like.\n- The fix was a feedback loop that narrates what the AI is doing in the background, which buys the patience a long task...","thumbnail_url":"https://img.transistorcdn.com/D9kLs0HSsqR4ttk_5ESEdC1jX-wmD76GK-OHmb3a9B8/rs:fill:0:0:1/w:400/h:400/q:60/mb:500000/aHR0cHM6Ly9pbWct/dXBsb2FkLXByb2R1/Y3Rpb24udHJhbnNp/c3Rvci5mbS80YTFk/MGU1MjJlODhlNjJh/MTdlZTZkN2Q1ODY5/OTdjYy5wbmc.webp","thumbnail_width":300,"thumbnail_height":300}