{"type":"rich","version":"1.0","provider_name":"Transistor","provider_url":"https://transistor.fm","author_name":"IntelliJAMS","title":" IntelliJAMS EP 057: Calling out the anxieties of A/B testing","html":"","width":"100%","height":180,"duration":1633,"description":"A/B testing can feel high-stakes, especially in e-commerce. You launch a test, check it an hour later, and either think you've broken your store or discovered a goldmine. Neither is true yet. This episode is a therapy session for anyone who's felt the anxiety of running A/B tests on their Shopify store.\nIn this episode of IntelliJAMS, Alex McEachern and experimentation expert Ally Petretti walk through the most common anxieties of A/B testing and how to manage them with better process instead of more stress.\nShould you check your A/B test results every day?\nChecking your test daily is like weighing yourself multiple times a day on a diet. The number isn't wrong, but it's not ready to be meaningful yet. Check to make sure the test is collecting data and firing correctly, but don't read into the directional results until you've hit your planned runtime and traffic thresholds.\nWhat is statistical significance and when should you end an A/B test?\nStatistical significance is often treated as a finish line, but reaching stat sig on day one doesn't mean you have a winner. Stat sig only looks at the math. You also need time (to account for day-of-week behavior changes, promotions, and the novelty effect) and volume (small sample sizes are fragile and can skew results dramatically). A test typically needs to run for at least one to two full weeks to account for these variables.\nWhat is the novelty effect in A/B testing?\nThe novelty effect happens when returning visitors see something different on your site and react to the change itself rather than the actual experience. In price testing, this can look like a customer treating a different price as a discount and converting out of urgency. The novelty effect fades after a week or two as your sample includes more new visitors and repeat visitors adjust.\nWhat is metric shopping in CRO?\nMetric shopping is when you pick a winning metric after a test is already running, instead of sticking to the success metric defined in your...","thumbnail_url":"https://img.transistorcdn.com/GPyAni4HeVbzD7e9GxLnjZGKmsG6Zq9pxBZ8oU9uQlE/rs:fill:0:0:1/w:400/h:400/q:60/mb:500000/aHR0cHM6Ly9pbWct/dXBsb2FkLXByb2R1/Y3Rpb24udHJhbnNp/c3Rvci5mbS8zMWUy/YzNkYjdmOGU0YTc5/NDU0NjYzMjA3ZDhm/Yjg5My5qcGc.webp","thumbnail_width":300,"thumbnail_height":300}