Spark Declarative Pipelines in Action: Live Avionics Streaming from 40,000 Aircraft Overhead

Description

Apache Spark 4.1.0 is here—released just weeks ago in January 2026—and it introduces a game-changing capability: Spark Declarative Pipelines (SDP). This declarative framework brings battle-tested ETL patterns to the open-source ecosystem, simplifying the development and maintenance of data pipelines by shifting from an imperative to a declarative mindset. Instead of telling Spark how to transform your data in seemingly endless lines of DataFrame code, you define what datasets should exist in your pipeline and Spark handles the heavy lifting. In this hands-on session, I'll provide a critical review of SDP's design decisions and demonstrate how declarative pipelines replace traditional imperative Spark code. Next, we'll build a streaming pipeline processing real avionics telemetry from tens of thousands of aircraft—from tiny Cessnas to massive Airbus A380s—flying overhead right now. The backbone of this demo is an open-source PySpark data source I developed in collaboration with OpenSky founders from Oxford and ETH Zurich. This custom but standard data plugin enables direct access to live ADS-B flight telemetry and showcases SDP's power: streaming tables for ingestion, materialized views for transformations - all defined declaratively. No orchestration glue code. Just define what your pipeline should accomplish, and let Spark handle execution. I'll live-code a challenging streaming data pipeline with real-world data using nothing but Visual Studio Code, PySpark, and open-source Apache Spark. No proprietary tools, no vendor lock-in, just you and the code. Alternatively, attendees can replicate the demo on a forever-free Lakehouse account. Either way, you can follow along at no cost. You'll leave with practical SDP knowledge and best practices for pipeline modernization. WARNING: After this session, you may catch the virus of tracking live aircraft data and getting deeply proficient with Apache Spark—don't worry, the data pipeline won't be your bottleneck!

Abstract

Apache Spark 4.1.0 is here—released just weeks ago in January 2026—and it introduces a game-changing capability: Spark Declarative Pipelines (SDP). This declarative framework brings battle-tested ETL patterns to the open-source ecosystem, simplifying the development and maintenance of data pipelines by shifting from an imperative to a declarative mindset. Instead of telling Spark how to transform your data in seemingly endless lines of DataFrame code, you define what datasets should exist in your pipeline and Spark handles the heavy lifting. In this hands-on session, I'll provide a critical review of SDP's design decisions and demonstrate how declarative pipelines replace traditional imperative Spark code. Next, we'll build a streaming pipeline processing real avionics telemetry from tens of thousands of aircraft—from tiny Cessnas to massive Airbus A380s—flying overhead right now. The backbone of this demo is an open-source PySpark data source I developed in collaboration with OpenSky founders from Oxford and ETH Zurich. This custom but standard data plugin enables direct access to live ADS-B flight telemetry and showcases SDP's power: streaming tables for ingestion, materialized views for transformations - all defined declaratively. No orchestration glue code. Just define what your pipeline should accomplish, and let Spark handle execution. I'll live-code a challenging streaming data pipeline with real-world data using nothing but Visual Studio Code, PySpark, and open-source Apache Spark. No proprietary tools, no vendor lock-in, just you and the code. Alternatively, attendees can replicate the demo on a forever-free Lakehouse account. Either way, you can follow along at no cost. You'll leave with practical SDP knowledge and best practices for pipeline modernization. WARNING: After this session, you may catch the virus of tracking live aircraft data and getting deeply proficient with Apache Spark—don't worry, the data pipeline won't be your bottleneck!