{"type":"rich","version":"1.0","provider_name":"Transistor","provider_url":"https://transistor.fm","author_name":"The Experimentation Edge","title":"The auto researcher: How FanDuel finds a winning model in 300 attempts","html":"","width":"100%","height":180,"duration":2455,"description":"Michael Mathews, Director of Data and AI Product at FanDuel, runs the teams behind the auto researcher, an agent that takes a data set, an objective function and the production model, then proposes, writes and scores about 300 strategies in an hour for roughly $300 in tokens, handing back the winner as readable code. Host Ashley Stirrup gets the three things that surprised FanDuel as it scaled: the agent builds its own features (756 down to 20), the holdout is enforced so it cannot cheat, and it reframes problems when the scoring rules tell it to.\nThe conversation covers the churn run that lifted F1 by 6% and is worth about a million dollars over baseline, the two popular parlays tests that became penalty weights and floors in the next experiment, North Star versus OEC in a luck-contaminated business, responsible gaming as a guardrail a machine has to be taught, and Michael's six-level ladder of experiment autonomy, including the shadow mode rung where FanDuel buys trust with six months of evidence. For product, data and growth leaders deciding how much of an experiment a machine should be allowed to run.\n \n \nChapters\n00:00 Cold open\n01:07 FanDuel, prediction markets and the data and AI product team\n03:57 The auto researcher: 300 strategies, an hour, $300 in tokens\n06:22 Three surprises: it builds features, enforces the holdout, reframes the problem\n09:33 Two popular parlays tests and the penalty weight they became\n14:39 What humans still own: the hypothesis and the goal\n18:27 North Star vs OEC and luck-adjusted margin\n23:16 The churn run: 750 features to 20 and a million dollars over baseline\n28:25 The six levels of experiment autonomy and shadow mode\n42:37 Always measure baseline, and when to stop iterating\n \n \nTakeaways\n- FanDuel's auto researcher reads an objective function, proposes a strategy, writes the model, scores it and repeats about 300 times in an hour for around $300, returning the winner as readable code.\n- The agent builds its own features (756...","thumbnail_url":"https://img.transistorcdn.com/D9kLs0HSsqR4ttk_5ESEdC1jX-wmD76GK-OHmb3a9B8/rs:fill:0:0:1/w:400/h:400/q:60/mb:500000/aHR0cHM6Ly9pbWct/dXBsb2FkLXByb2R1/Y3Rpb24udHJhbnNp/c3Rvci5mbS80YTFk/MGU1MjJlODhlNjJh/MTdlZTZkN2Q1ODY5/OTdjYy5wbmc.webp","thumbnail_width":300,"thumbnail_height":300}