{"type":"rich","version":"1.0","provider_name":"Transistor","provider_url":"https://transistor.fm","author_name":"Screaming in the Cloud","title":"Engineering Around Extreme S3 Scale with R. Tyler Croy","html":"<iframe width=\"100%\" height=\"180\" frameborder=\"no\" scrolling=\"no\" seamless src=\"https://share.transistor.fm/e/c1aea350\"></iframe>","width":"100%","height":180,"duration":2019,"description":"R. Tyler Croy, a principal engineer at Scribd, joins Corey Quinn to explain what happens when simple tasks cost $100,000. Checking if files are damaged? $100K. Using newer S3 tools? Way too expensive. Normal solutions don't work anymore. Tyler shares how with this much data, you can't just throw money at the problem, but rather you have to engineer your way out.\nAbout R. Tyler: \nR. Tyler Croy leads infrastructure architecture at Scribd and has been an open source developer for over 14 years. His work spans the FreeBSD, Python, Ruby, Puppet, Jenkins, and Delta Lake communities. Under his leadership, Scribd’s Infrastructure Engineering team built Delta Lake for Rust to support a wide variety of high performance data processing systems. That experience led to Tyler developing the next big iteration of storage architecture to power large-scale fulltext compute challenges facing the organization.\nShow Highlights:\n01:48 Scribd's 18-Year History\n04:00 One Document Becomes Billions of Files\n05:47 When Normal Physics Stop Working\n08:02 Why S3 Metadata Costs Too Much\n10:50 How AI Made Old Documents Valuable\n13:30 From 100 Billion to 100 Million Objects\n15:05 The Curse of Retail Pricing \n19:17 How Data Scientists Create Growth\n21:18 De-Normalizing Data Problems\n25:29 Evolving Old Systems\n27:45 Billions Added Since Summer\n29:29 Underused S3 Features\n31:48 Where to Find Tyler\n\nLinks: \nScribd: https://tech.scribd.com\nMastodon:  https://hacky.town/@rtyler\nGitHub: https://github.com/rtyler\nSponsored by: \nduckbillhq.com","thumbnail_url":"https://img.transistorcdn.com/vgV98ushAFvdAFQSQazvt5GYVniCaXSWIjs5eI3r0gQ/rs:fill:0:0:1/w:400/h:400/q:60/mb:500000/aHR0cHM6Ly9pbWct/dXBsb2FkLXByb2R1/Y3Rpb24udHJhbnNp/c3Rvci5mbS9zaG93/LzE0OTQvMTU4Mzg2/OTQ4My1hcnR3b3Jr/LmpwZw.webp","thumbnail_width":300,"thumbnail_height":300}