Jesse Pepple Portfolio

This project showcases end-to-end data engineering in Microsoft Fabric  using Airbnb datasets. Data was ingested via Fabric Data Factory  and stored in a Lakehouse following the Medallion Architecture  (Bronze, Silver, Gold). Transformations were performed in Fabric Notebooks , with a dynamic SCD Type 2  applied in the Gold layer, and the final Gold layer was modelled as a Star Schema  to support efficient analytics and reporting.  Key Takeaways / Project Impact Manual vs Aut

This project showcases a complete end-to-end Azure Data Engineering solution, handling data from ingestion through transformation to delivery using modern, cloud-native best practices. The architecture leverages Azure Data Factory, Azure Databricks, and Delta Live Tables (DLT) to create a scalable, reliable, and production-ready data pipeline. The pipeline is designed to manage incremental data ingestion, support schema evolution, implement change data capture (CDC), and enab

This project delivers a fully integrated Azure Data Engineering solution that manages the entire data lifecycle—from ingestion to transformation and final delivery. Built using modern cloud-native best practices, the architecture combines Azure SQL, Azure Data Factory, Azure Databricks, Logic App and Delta Live Tables (DLT) to create a highly scalable, reliable, and production-grade data pipeline designed for real-world analytics and business intelligence workloads. Resources

Resources Azure Services Azure Databricks Spark Structured Streaming Delta Live Tables Delta Lake & Delta Tables Databricks SQL Warehouse Unity Catalog Languages & Tools: Python (PySpark / notebooks) & SQL  Overall Project Impact End-to-End Automation:  Bronze → Silver → Gold pipeline fully automated with streaming, DLT, and UPSERTs Incremental Data Handling:   100% incremental load success → reduced processing time and cost Data Quality & Reliability:  All DLT expectations

This Data Engineering project demonstrates the full potential of Azure Databricks , Azure Data Factory (ADF) , and Azure Data Lake , showcasing how modern data solutions can be built with scalability and automation in mind. The project incorporates key engineering practices such as Slowly Changing Dimensions (SCD) Type 1  — implemented manually — and Type 2 , automated through Delta Live Tables (DLT) , alongside Star Schema modeling  and incremental data loading . Using Spark

This project focused on building an end-to-end Azure & Databricks data pipeline  using the Olympics 2024 dataset . I began by provisioning the required Azure resources and designing the solution around a Medallion Architecture (Bronze → Silver → Gold)  to ensure scalability, data quality, and reliability. Data was ingested using Azure Data Factory (ADF)  and stored in Azure Data Lake Storage , establishing the raw Bronze layer. From there, I applied data governance through Un

In this project, I implemented Unity Catalog in Databricks  to establish secure, centralized data governance , ensuring controlled access and compliance across all datasets. The solution leveraged Delta Lake and Delta Tables  to provide scalable, reliable, and efficient storage , while the pipeline followed the Medallion Architecture (Bronze → Silver → Gold)  to maintain data quality, reliability, and usability  at every stage. The end-to-end workflow utilized Azure Data Fact

This project implemented Unity Catalog in Databricks to provide secure, centralized data governance, while leveraging Delta Lake and Delta Tables for scalable and efficient data storage. The data pipeline followed the Medallion Architecture (Bronze → Silver → Gold), ensuring data quality, reliability, and usability at every stage. The solution was built using Azure Data Factory, Azure SQL Database, Azure Data Lake Storage, and Azure Databricks, with Python and SQL as the prim