Breaking silos, one stream at a time
he/him
Irfan Elahi is a full-stack customer-focused big data and analytics specialist with a proven combination of diverse consulting and technical competencies. He has delivered substantial impact across telecom, retail, energy, and healthcare in multiple capacities. An analytics evangelist, Irfan is the author of a published book, Udemy courses, blog posts, trainings, and presentations with global reach. He helps organisations turn Kafka-heavy integration backbones into governed, multi-contract streaming platforms without the sprawl.
In telecom, like in plenty of other industries, Kafka is the integration backbone: alarms, probe telemetry and outages all move through it. The payloads are mostly JSON, still the most common serialization format on Kafka, and ingesting them is typically the easy part. Republishing them, Kafka to Kafka, at scale, is not. That part is rarely passthrough: it means joins, lookup and real transformation to turn drift-prone raw events into operational-ready ones for a dozen downstream systems, without breaking the consumers still depending on the old shape. And there is never just one consumer. Operations teams want enrichment, routing and audit metadata. Service assurance teams want a fully standardized contract with derived metrics. Analysts just want a clean table they can query without unpicking nested JSON. Publish all of that from one source stream and you accumulate tangled jobs, duplicated transformation logic and schema drift, the kind of tech debt that compounds with every consumer you add. This session is a battle-tested blueprint for doing it without the sprawl: one raw stream, fanned into several governed outputs from a single place where the logic lives. Two go straight back to Kafka, the primary hot path: - An augmentation-only topic that preserves the original shape and adds operational metadata and enrichment - A curated contract topic with standardized, enriched payloads and derived metrics Two land in the lakehouse: - A Bronze evidence store that keeps the raw payload for lineage and optimal reprocessing - A Silver table for BI and self-serve analytics Telecom is the reference story, but the pattern holds anywhere Kafka is the integration backbone and schemas drift e.g., banking, energy and logistics to name a few. You'll leave with the patterns and best practices, the trade-offs and lessons learned and the broader integration opportunities this opens up downstream.
