{"type":"rich","version":"1.0","provider_name":"Transistor","provider_url":"https://transistor.fm","author_name":"Data Science Tech Brief By HackerNoon","title":"Sampling Ad Exchange Events at Scale: What We Kept, What We Dropped, and Why","html":"","width":"100%","height":180,"duration":619,"description":"\n This story was originally published on HackerNoon at: https://hackernoon.com/sampling-ad-exchange-events-at-scale-what-we-kept-what-we-dropped-and-why.\nHow we reduced Kafka stream volume by 80%, cut Spark workers from 240 to 30, and lowered reporting costs through sampling and executor tuning at Unity.\nCheck more stories related to data-science at: https://hackernoon.com/c/data-science.\n You can also check exclusive content about #data-engineering, #apache-kafka, #apache-spark, #big-data, #data-pipelines, #adtech, #cloud-cost-optimization, #data-sampling-techniques, and more.\nThis story was written by: @vikrantbachhav. Learn more about this writer by checking @vikrantbachhav's about page,\n and for more stories, please visit hackernoon.com.\nAt Unity, our ad exchange was processing approximately 10 GB of Kafka event data per second, much of which wasn't needed for downstream reporting. By introducing auction-level sampling, retaining critical events at full fidelity, and optimizing Spark executor configurations, we reduced stream volume by 80%, cut Spark workers from 240 to 30, and lowered DSP reporting transform costs by approximately 41%. Along the way, we uncovered longstanding reporting inaccuracies and learned why understanding data usage is just as important as optimizing infrastructure.","thumbnail_url":"https://img.transistorcdn.com/8VxAgS1Ll3FJEERcAdhFdqqXJMnE7OfD2RUvrjauLt0/rs:fill:0:0:1/w:400/h:400/q:60/mb:500000/aHR0cHM6Ly9pbWct/dXBsb2FkLXByb2R1/Y3Rpb24udHJhbnNp/c3Rvci5mbS9zaG93/LzQxMjY4LzE2ODM1/ODI1ODUtYXJ0d29y/ay5qcGc.webp","thumbnail_width":300,"thumbnail_height":300}