records/day via Kafka + Airflow on
Accelerated ETL via predicate pushdown and
Feature 01
Scaled throughput past 2M/day
I owned the Databricks + Kafka + Airflow ETL on AWS and pushed daily throughput from a stuck 1.6M to over 2M records without growing the cluster.
Feature 02
Killed the skew bottleneck
I diagnosed and fixed the stage-skew bottleneck using broadcast joins, key salting on skewed publishers, and multi-threaded I/O on the slow steps.
Feature 03
30% faster ETL, same budget
I accelerated end-to-end ETL by 30% through predicate pushdown, materialised Parquet snapshots, and a careful cluster rightsizing pass.
What I shipped
Big-data ETL on Databricks for TV audience measurement.
2M+ records/day via Kafka + Airflow on AWS.
Resolved bottleneck via broadcast joins, key salting, multi-threading.
Accelerated ETL 30% via predicate pushdown and rightsizing.