As generative AI systems move from novelty to infrastructure, questions of safety, trust, and governance are becoming urgent. In this episode of TechSurge, host Sriram Viswanathan is joined by Dr. Rumman Chowdhury, CEO of Humane Intelligence PBC and responsible AI Pioneer, about what AI safety really means and why the industry may be focusing on the wrong problems.
Rumman argues that the most overlooked lever in AI development is evaluation. While companies emphasize model training and capabilities, far less attention is paid to how systems are assessed in real-world contexts, who defines “good,” what risks are measured, and how societal impacts are accounted for at scale. She distinguishes between technical assurance and broader sociotechnical risk, from misinformation and bias to over-reliance and erosion of institutional trust.
Drawing on her experience at Twitter (X) and in global policy circles, Rumman highlights a fundamental governance gap: unlike finance, aviation, or healthcare, AI lacks a mature, independent ecosystem of auditors and evaluators. Today, the same companies building AI systems often define what counts as harm. She also challenges the belief that stronger guardrails alone will solve the problem, noting that cultural context, language differences, and human behavior complicate any notion of “neutral” or fully objective AI.
Rather than focusing solely on speculative existential threats, Rumman urges attention to the harms already visible from AI-enabled misinformation to mental health risks and shifts in how younger generations relate to knowledge and authority. The future of AI, she suggests, will be determined not just by technological breakthroughs, but by whether we build credible systems of accountability, evaluation, and global cooperation around them.
If you enjoy this episode, please subscribe and leave us a review on your favorite podcast platform.
Sign up for our newsletter at
techsurgepodcast.com for updates on upcoming TechSurge Live Summits and future Season 2 episodes.
Episode Links
Timestamps
- 02:50 Why AI Evaluations Matter: Defining “Good” Models in Context
- 04:25 What Is AI Safety? From Product Performance to Societal Harm
- 11:30 Regulation Reality Check: EU AI Act, Conformance Assessments & Checklists
- 15:25 Building the AI Evaluation Profession: Audits, Red Teaming & Legal Protections
- 23:00 When It’s OK to Outsource Judgment and When It’s Dangerous
- 39:38Who’s Responsible When AI Outcomes Go Wrong?
- 52:37 Design vs Governance: Complex Systems, System-Level Evaluation, and Regulating Horizontally
- 44:11 AI Psychosis, Youth Harm, and What’s Already Here
- 47:27 What Keeps Rumman Up at Night: Kids, Algorithms, and Hope from Global Governance
- 54:00 Bringing Sci-Fi to the Real World?
What is TechSurge: Deep Tech Podcast?
The TechSurge: Deep Tech VC Podcast explores the frontiers of emerging tech, geopolitics, and business, with conversations tailored for entrepreneurs, technologists, and investment professionals. Presented by Celesta Capital, and hosted by Founding Partners Nic Brathwaite, Michael Marks, and Sriram Viswanathan. Send feedback and show ideas to techsurge@celesta.vc.
Each discussion delves into the intersection of technology advancement, market dynamics, and the founder journey, offering insights into the vast opportunities and complex challenges ahead. Episode topics include AI, data center transformation, blockchain, cyber security, healthcare innovation, VC investment trends, tips for first-time founders, and more.
Tune in to hear directly from Silicon Valley leaders, daring new founders, and visionary thinkers. Past guests include investor Vinod Khosla, former PepsiCo CEO Indra Nooyi, the Global Head of McKinsey, and executive leaders from Microsoft, OpenAI, and other leading tech companies.
New episodes release every two weeks. Visit techsurgepodcast.com for more details and to sign up for our newsletter and other content!