← Back to Sessions

From Adaptive Pipelines to Autonomous Agents

40 minBeginnerBrisbaneAuckland
Open SourceApache NiFiData EngineeringETL and Data PipelinesApache IcebergSchema EvolutionLLMs in ProductionData Platform ArchitectureSnowflakeAI agentsdata agentsRAGsemantic layers

Description

Data and AI engineers are trapped in a cycle of fixing brittle ETL and managing unpredictable AI demos. This two-part session provides a blueprint for breaking that cycle by building a data stack that doesn't just move data, but understands it. Part 1: I'm Tired of Rebuilding ETL So I Built Pipelines That Adapt Every data engineer knows the pain—schema drift, broken pipelines, endless custom mappings. Kamesh has spent years doing work that shouldn't exist, and he's done with it. In this live, code-driven session, he'll demonstrate how to use AI to understand data semantically rather than just generate it. Using Apache NiFi, LLMs, and Apache Iceberg, he'll build an end-to-end pipeline that transforms messy, inconsistent data into analytics-ready tables—without hand-written mappings. Part 2: Data Agents in Production: Patterns for Building AI That Understands Your Business Once your data pipeline is clean, the next challenge is making it actionable. The gap between an impressive AI demo and a trusted enterprise tool is enormous — and it's filled with questions about grounding, governance, evaluation, and orchestration. We'll break down the anatomy of a production-grade data agent: one that interprets intent, reasons over governed enterprise data, calls the right tools, and explains its work. Drawing on real implementation patterns, we'll explore how semantic layers, retrieval-augmented generation, and platform-native AI capabilities come together to create agents that are accurate, auditable, and genuinely useful.

Abstract

Data and AI engineers are trapped in a cycle of fixing brittle ETL and managing unpredictable AI demos. This two-part session provides a blueprint for breaking that cycle by building a data stack that doesn't just move data, but understands it. Part 1: I'm Tired of Rebuilding ETL So I Built Pipelines That Adapt Every data engineer knows the pain—schema drift, broken pipelines, endless custom mappings. Kamesh has spent years doing work that shouldn't exist, and he's done with it. In this live, code-driven session, he'll demonstrate how to use AI to understand data semantically rather than just generate it. Using Apache NiFi, LLMs, and Apache Iceberg, he'll build an end-to-end pipeline that transforms messy, inconsistent data into analytics-ready tables—without hand-written mappings. Part 2: Data Agents in Production: Patterns for Building AI That Understands Your Business Once your data pipeline is clean, the next challenge is making it actionable. The gap between an impressive AI demo and a trusted enterprise tool is enormous — and it's filled with questions about grounding, governance, evaluation, and orchestration. We'll break down the anatomy of a production-grade data agent: one that interprets intent, reasons over governed enterprise data, calls the right tools, and explains its work. Drawing on real implementation patterns, we'll explore how semantic layers, retrieval-augmented generation, and platform-native AI capabilities come together to create agents that are accurate, auditable, and genuinely useful.

Speaker

Kamesh Sampath

Kamesh Sampath

Lead Developer Advocate

Kamesh Sampath is Lead Developer Advocate at Snowflake. He focuses on learning more, doing more, and sharing more with the data community.