This story was originally published on HackerNoon at:
https://hackernoon.com/how-i-fixed-a-data-ingestion-bottleneck-with-date-partitioning.
I fixed a recurring data ingestion bottleneck by working with a vendor to introduce date-partitioned exports and reduce unnecessary file discovery.
Check more stories related to data-science at:
https://hackernoon.com/c/data-science.
You can also check exclusive content about
#data-engineering,
#distributed-systems,
#data-ingestion-pipelines,
#data-partitioning,
#object-storage,
#incremental-data-ingestion,
#idempotent-data-processing,
#data-pipeline-reliability, and more.
This story was written by:
@rahuln. Learn more about this writer by checking
@rahuln's about page,
and for more stories, please visit
hackernoon.com.
Our vendor-data ingestion jobs were repeatedly enumerating a growing history of export files. I worked with the vendor's engineers on date-partitioned delivery and prepared our consumer in parallel. This case study explains how partitioning bounds file discovery, why event-date partitions need late-arrival and replay policies, and how to coordinate the producer-consumer cutover.