{"type":"rich","version":"1.0","provider_name":"Transistor","provider_url":"https://transistor.fm","author_name":"Chain of Thought | AI Agents, Infrastructure & Engineering","title":"Practical Lessons for GenAI Evals | Chip Huyen & Vivienne Zhang","html":"<iframe width=\"100%\" height=\"180\" frameborder=\"no\" scrolling=\"no\" seamless src=\"https://share.transistor.fm/e/c5db81b4\"></iframe>","width":"100%","height":180,"duration":2883,"description":"As AI agents and multimodal models become more prevalent, understanding how to evaluate GenAI is no longer optional – it's essential. Generative AI introduces new complexities in assessment compared to traditional software, and this week on Chain of Thought we’re joined by Chip Huyen (Storyteller, Tép Studio), Vivienne Zhang (Senior Product Manager, Generative AI Software, Nvidia) for a discussion on AI evaluation best practices. Before we hear from our guests, Vikram Chatterji (CEO, Galileo) and Conor Bronsdon (Developer Awareness, Galileo) give their takes on the complexities of AI evals and how to overcome them through the use of objective criteria in evaluating open-ended tasks, the role of hallucinations in AI models, and the importance of human-in-the-loop systems.Afterwards, Chip and Vivienne sit down with Atin Sanyal (Co-Founder & CTO, Galileo) to explore common evaluation approaches, best practices for building frameworks, and implementation lessons. They also discuss the nuances of evaluating AI coding assistants and agentic systems.Chapters:00:00 Challenges in Evaluating Generative AI05:45 Evaluating AI Agents13:08 Are Hallucinations Bad?17:12 Human in the Loop Systems20:49 Panel discussion begins22:57 Challenges in Evaluating Intelligent Systems24:37 User Feedback and Iterative Improvement26:47 Post-Deployment Evaluations and Common Mistakes28:52 Hallucinations in AI: Definitions and Challenges34:17 Evaluating AI Coding Assistants38:15 Agentic Systems: Use Cases and Evaluations43:00 Trends in AI Models and Hardware45:42 Future of AI in Enterprises47:16 Conclusion and Final ThoughtsFollow:Vikram Chatterji: https://www.linkedin.com/in/vikram-chatterji/Atin Sanyal: ⁠⁠https://www.linkedin.com/in/atinsanyal/Conor Bronsdon: https://www.linkedin.com/in/conorbronsdon/Chip Huyen: ⁠https://www.linkedin.com/in/chiphuyen/⁠Vivienne Zhang: ⁠⁠https://www.linkedin.com/in/viviennejiaozhang/Show notes:Watch all of Productionize 2.0:...","thumbnail_url":"https://img.transistorcdn.com/opYb_ogIEF0JJPpA1L13NhLPore7cUzGtcL8Okdgfg4/rs:fill:0:0:1/w:400/h:400/q:60/mb:500000/aHR0cHM6Ly9pbWct/dXBsb2FkLXByb2R1/Y3Rpb24udHJhbnNp/c3Rvci5mbS81YjBh/ODMzMTY3ZjQ0MjBj/YTE1ODMwYTZlNDgx/Mjc2Mi5qcGc.webp","thumbnail_width":300,"thumbnail_height":300}