{"type":"rich","version":"1.0","provider_name":"Transistor","provider_url":"https://transistor.fm","author_name":"IntelliJAMS","title":"IntelliJAMS EP 058: Outputs vs. Outcomes: A Better Way to Think About A/B Testing","html":"<iframe width=\"100%\" height=\"180\" frameborder=\"no\" scrolling=\"no\" seamless src=\"https://share.transistor.fm/e/ca4f3578\"></iframe>","width":"100%","height":180,"duration":838,"description":"Running 10 tests a month sounds productive, but if none of them are tied to a real business question, you're just taking swings for the sake of swinging. In this episode Adam and Alex dig into why \"how many tests should I run?\" is often the wrong question, and what to ask instead when you're building an experimentation program on Shopify.\nIn this episode of IntelliJAMS, Adam and Alex explore the difference between test output and test outcomes, how to run concurrent tests without confounding your results, and why elevating the conversation from \"number of tests\" to \"strategic priorities\" changes everything.\nTimestamps:\n0:00 - Intro\n0:27 - The question everyone's asking right now\n0:52 - Why \"how many tests\" is the wrong question\n2:23 - Outputs vs. outcomes: reframing productivity\n3:10 - Test the plan, don't plan the tests\n3:51 - You can't predict how many tests it'll take\n4:54 - Running concurrent tests without confounding results\n5:53 - Scoping metrics to the right part of the funnel\n6:48 - Every test needs a hypothesis\n7:30 - One test often leads to five new questions\n8:15 - Real example: ending a checkout upsell test early\n9:11 - How to push back on \"I need 10 tests this month\"\n11:49 - Elevating the conversation from output to strategy\n13:28 - Final thoughts: see the forest for the trees\nTopics covered:\nWhy counting tests is an output metric, not an outcomes metric\nHow to reframe testing around strategic business goals\nRunning concurrent A/B tests on Shopify without confounding data\nSegmenting tests by funnel stage and visitor type\nHow one test can lead to five new questions\nPushing back when stakeholders demand a test quota\nReal-world example: killing a checkout upsell test 24 hours in\nThe \"test the plan, don't plan the tests\" framework\nWant to sharpen your experimentation skills? Join GEM Academy for free courses and a community of brands sharing what works: https://www.skool.com/intelligems-academy-1535\nWebsite: https://intelligems.io\nBlog:...","thumbnail_url":"https://img.transistorcdn.com/GPyAni4HeVbzD7e9GxLnjZGKmsG6Zq9pxBZ8oU9uQlE/rs:fill:0:0:1/w:400/h:400/q:60/mb:500000/aHR0cHM6Ly9pbWct/dXBsb2FkLXByb2R1/Y3Rpb24udHJhbnNp/c3Rvci5mbS8zMWUy/YzNkYjdmOGU0YTc5/NDU0NjYzMjA3ZDhm/Yjg5My5qcGc.webp","thumbnail_width":300,"thumbnail_height":300}