Jesse Pepple Portfolio

This project showcases a complete end-to-end Azure Data Engineering solution, handling data from ingestion through transformation to delivery using modern, cloud-native best practices. The architecture leverages Azure Data Factory, Azure Databricks, and Delta Live Tables (DLT) to create a scalable, reliable, and production-ready data pipeline. The pipeline is designed to manage incremental data ingestion, support schema evolution, implement change data capture (CDC), and enab

This project delivers a fully integrated Azure Data Engineering solution that manages the entire data lifecycle—from ingestion to transformation and final delivery. Built using modern cloud-native best practices, the architecture combines Azure SQL, Azure Data Factory, Azure Databricks, Logic App and Delta Live Tables (DLT) to create a highly scalable, reliable, and production-grade data pipeline designed for real-world analytics and business intelligence workloads. Resources

Resources Azure Services Azure Databricks Spark Structured Streaming Delta Live Tables Delta Lake & Delta Tables Databricks SQL Warehouse Unity Catalog Languages & Tools: Python (PySpark / notebooks) & SQL Project Overview This project is built entirely on Azure Databricks , leveraging Spark Structured Streaming  for real-time data ingestion and PySpark  for large-scale data transformations. I implemented Delta Live Tables  to automate Slowly Changing Dimensions (SCDs)  and