Implementing the medallion architecture with Starburst
Description
The focus of this workshop is to implement tables in the bronze, silver, and gold layers of the medallion architecture using the Starburst platform. The hands-on exercises will be completed on the Trino-based Starburst Galaxy platform, so just bring your laptop and be ready to learn.
Abstract
The focus of this workshop is to implement tables in the bronze, silver, and gold layers of the medallion architecture using the Starburst platform. The hands-on exercises will be completed on the Trino-based Starburst Galaxy platform, so just bring your laptop and be ready to learn. In this hands-on workshop, you will complete these activities: Beginning with the bronze layer, see how Starburst ingestion tools make it easy to source data from files on the object store, as well as from a Kafka topic. Then you will perform transformation logic to move the raw table in the bronze layer to a business-ready structured & validated table in the silver layer. The gold layer will include views that you will construct which are aligned to consumption-oriented business requirements. You will learn how Starburst auto-tagging features use AI to identify PII data and apply appropriate tags that you can review. Next, you'll create a Starburst Data Product with your gold and silver tables and views. Then you will apply RBAC and ABAC security policies to ensure different roles can only access the information they should be able to see. For the speedy kids, an additional exercise will be present allowing them to use the MERGE command to populate a CDC Type 2 table based on change deltas being applied to an existing table.
Key Takeaways
- Learn how to implement bronze, silver, and gold layers using Starburst platform
- Master data ingestion from object stores and Kafka topics
- Understand transformation logic for moving data between medallion layers
- Create business-aligned views in the gold layer
- Apply AI-powered PII auto-tagging and security policies
- Build Starburst Data Products with proper RBAC and ABAC controls
Prerequisites
- Basic understanding of SQL
- Familiarity with data lake concepts
- Laptop with web browser for accessing Starburst Galaxy
Speakers
Lester Martin
Lead Developer Advocate, Starburst Data
Lester Martin leads the DevRel function at Starburst. He is a seasoned developer advocate, trainer, blogger, and data engineer focused on data pipelines & data lake analytics using Starburst, Trino, Iceberg, Hive, Spark, Flink, Kafka, NiFi, NoSQL databases, and, of course, classical RDBMSs. Check out Lester's blog at https://lestermartin.blog.
Michael Markieta
Lead Solution Architect, Starburst Data
Michael leads the Solution Architecture practice in the APJ region for Starburst. He brings over a decade of experience ranging from roles as a OpenShift Black Belt at Red Hat, bootstrapping startup co-founder and CTO, and Customer Engineer at Google.