{"type":"rich","version":"1.0","provider_name":"Transistor","provider_url":"https://transistor.fm","author_name":"Data Science Tech Brief By HackerNoon","title":"How I Fixed a Data Ingestion Bottleneck With Date Partitioning","html":"<iframe width=\"100%\" height=\"180\" frameborder=\"no\" scrolling=\"no\" seamless src=\"https://share.transistor.fm/e/97bfa91f\"></iframe>","width":"100%","height":180,"duration":552,"description":"\n        This story was originally published on HackerNoon at: https://hackernoon.com/how-i-fixed-a-data-ingestion-bottleneck-with-date-partitioning.\nI fixed a recurring data ingestion bottleneck by working with a vendor to introduce date-partitioned exports and reduce unnecessary file discovery.\nCheck more stories related to data-science at: https://hackernoon.com/c/data-science.\n            You can also check exclusive content about #data-engineering, #distributed-systems, #data-ingestion-pipelines, #data-partitioning, #object-storage, #incremental-data-ingestion, #idempotent-data-processing, #data-pipeline-reliability,  and more.\nThis story was written by: @rahuln. Learn more about this writer by checking @rahuln's about page,\n            and for more stories, please visit hackernoon.com.\nOur vendor-data ingestion jobs were repeatedly enumerating a growing history of export files. I worked with the vendor's engineers on date-partitioned delivery and prepared our consumer in parallel. This case study explains how partitioning bounds file discovery, why event-date partitions need late-arrival and replay policies, and how to coordinate the producer-consumer cutover.","thumbnail_url":"https://img.transistorcdn.com/8VxAgS1Ll3FJEERcAdhFdqqXJMnE7OfD2RUvrjauLt0/rs:fill:0:0:1/w:400/h:400/q:60/mb:500000/aHR0cHM6Ly9pbWct/dXBsb2FkLXByb2R1/Y3Rpb24udHJhbnNp/c3Rvci5mbS9zaG93/LzQxMjY4LzE2ODM1/ODI1ODUtYXJ0d29y/ay5qcGc.webp","thumbnail_width":300,"thumbnail_height":300}