Spark Declarative Pipelines in Action: Live Avionics Streaming from 40,000 Aircraft Overhead
Frank transforms complex technical concepts into compelling demos and hands-on experiences where he bridges developer experience and product. With a Ph.D. in Computer Science (summa cum laude, TU Munich) and over 25 years of expertise in data, AI, and cloud computing, he brings both deep technical understanding and a passion for storytelling to every session. He's presented at tier-1 conferences across six continents—including many Devoxxs, KubeCon, Java One and re:Invent. No stranger to Australia, he spent two years in Sydney as a Technical Advisor for BEA Systems, specializing in Java Enterprise and Middleware solutions. At AWS, he founded developer relations for Central EMEA and tripled the team's reach. A published author of 17 scientific articles and 3 textbooks used in US universities, he was recognized as Cloud Technologist of the Year by Oracle. His approach: pair rigorous technical depth with narrative clarity to make complex systems feel approachable. Expect a session where you'll leave ready to build!
Apache Spark 4.1.0 is here—released just weeks ago in January 2026—and it introduces a game-changing capability: Spark Declarative Pipelines (SDP). This declarative framework brings battle-tested ETL patterns to the open-source ecosystem, simplifying the development and maintenance of data pipelines by shifting from an imperative to a declarative mindset. Instead of telling Spark how to transform your data in seemingly endless lines of DataFrame code, you define what datasets should exist in your pipeline and Spark handles the heavy lifting. In this hands-on session, I'll provide a critical review of SDP's design decisions and demonstrate how declarative pipelines replace traditional imperative Spark code. Next, we'll build a streaming pipeline processing real avionics telemetry from tens of thousands of aircraft—from tiny Cessnas to massive Airbus A380s—flying overhead right now. The backbone of this demo is an open-source PySpark data source I developed in collaboration with OpenSky founders from Oxford and ETH Zurich. This custom but standard data plugin enables direct access to live ADS-B flight telemetry and showcases SDP's power: streaming tables for ingestion, materialized views for transformations - all defined declaratively. No orchestration glue code. Just define what your pipeline should accomplish, and let Spark handle execution. I'll live-code a challenging streaming data pipeline with real-world data using nothing but Visual Studio Code, PySpark, and open-source Apache Spark. No proprietary tools, no vendor lock-in, just you and the code. Alternatively, attendees can replicate the demo on a forever-free Lakehouse account. Either way, you can follow along at no cost. You'll leave with practical SDP knowledge and best practices for pipeline modernization. WARNING: After this session, you may catch the virus of tracking live aircraft data and getting deeply proficient with Apache Spark—don't worry, the data pipeline won't be your bottleneck!
