I have led teams of data engineers and software engineers at Airbnb, Facebook, and Netflix. My next goal is to upskill as many data knowledge workers ...
Compressing a 100 TB data lake into 5 TBs with Iceberg and Parquet
Locations: SydneyPerth
In this workshop, we explore the fundamental concepts essential for effective data modeling and database management. We dive into understanding your data consumer, the distinctions between OLTP and OLAP data modeling, the principles of Cumulative Table design, and the tradeoff between compactness and usability. We also address the challenges of temporal cardinality explosion and the potential pitfalls of run-length encoding compression. By the end of this lecture, you will have a strong foundation in the key concepts of data modeling and be prepared to apply these principles in practice.