Data downtime costs companies millions in bad decisions, lost revenue, and broken trust. Unlike application outages that trigger immediate alerts, data issues are silent. Pipelines would run successfully while producing incorrect results and by the time someone notices, damage is done and large clean up job is required. This talk will show you how Open Telemetry can assist data teams, much like how it helps application teams. We will discuss current state of Otel standards for data teams, what it takes to instrument data pipelines, and level of support in various common open source data projects. We'll cover how to: - Connect Otel traces across your entire pipeline from orchestration through processing to querying - Instrument quality checks at each transformation stage - Reduce investigation time with AI-powered root cause analysis We'll walk through instrumenting a real pipeline spanning Airflow, Spark, Dbt and Trino to show how data engineers can stop context-switching between tools and get the unified investigation experience that application teams already have. We'll also demonstrate how AI can accelerate investigations by querying traces, logs, and quality metrics through natural conversation, turning hours of manual troubleshooting into minutes of guided analysis.
Data downtime costs companies millions in bad decisions, lost revenue, and broken trust. Unlike application outages that trigger immediate alerts, data issues are silent. Pipelines would run successfully while producing incorrect results and by the time someone notices, damage is done and large clean up job is required. This talk will show you how Open Telemetry can assist data teams, much like how it helps application teams. We will discuss current state of Otel standards for data teams, what it takes to instrument data pipelines, and level of support in various common open source data projects. We'll cover how to: - Connect Otel traces across your entire pipeline from orchestration through processing to querying - Instrument quality checks at each transformation stage - Reduce investigation time with AI-powered root cause analysis We'll walk through instrumenting a real pipeline spanning Airflow, Spark, Dbt and Trino to show how data engineers can stop context-switching between tools and get the unified investigation experience that application teams already have. We'll also demonstrate how AI can accelerate investigations by querying traces, logs, and quality metrics through natural conversation, turning hours of manual troubleshooting into minutes of guided analysis.
Solutions Architect
Johnny Mirza is a Solution Architect with ClickHouse, working with users across APAC. With over 20 years of background in solutions engineering, he's experienced in architecting and enabling solutions for enterprise clients in the telecommunications, media, insurance, and financial services sectors. Johnny has a high level of expertise of integration between both public cloud and on-premise infrastructure, while focussing on service assurance, monitoring platforms, and open-source technologies. Prior to ClickHouse, Johnny was part of the solution engineering teams at Confluent, Splunk, and Optus, to name a few.
