Jesse Pepple Portfolio

Resources Azure Services Azure Databricks Spark Structured Streaming Delta Live Tables Delta Lake & Delta Tables Databricks SQL Warehouse Unity Catalog Languages & Tools: Python (PySpark / notebooks) & SQL Project Overview This project is built entirely on Azure Databricks , leveraging Spark Structured Streaming  for real-time data ingestion and PySpark  for large-scale data transformations. I implemented Delta Live Tables  to automate Slowly Changing Dimensions (SCDs)  and

This Data Engineering project demonstrates the full potential of Azure Databricks , Azure Data Factory (ADF) , and Azure Data Lake , showcasing how modern data solutions can be built with scalability and automation in mind. The project incorporates key engineering practices such as Slowly Changing Dimensions (SCD) Type 1  — implemented manually — and Type 2 , automated through Delta Live Tables (DLT) , alongside Star Schema modeling  and incremental data loading . Using Spark

This project focused on building an end-to-end Azure & Databricks data pipeline  using the Olympics 2024 dataset . I began by provisioning the required Azure resources and designing the solution around a Medallion Architecture (Bronze → Silver → Gold)  to ensure scalability, data quality, and reliability. Data was ingested using Azure Data Factory (ADF)  and stored in Azure Data Lake Storage , establishing the raw Bronze layer. From there, I applied data governance through Un