top of page


Microsoft Fabric Airbnb Data Engineering Project
This project showcases end-to-end data engineering in Microsoft Fabric using Airbnb datasets. Data was ingested via Fabric Data Factory and stored in a Lakehouse following the Medallion Architecture (Bronze, Silver, Gold). Transformations were performed in Fabric Notebooks , with a dynamic SCD Type 2 applied in the Gold layer, and the final Gold layer was modelled as a Star Schema to support efficient analytics and reporting. Key Takeaways / Project Impact Manual vs Aut
Jesse Pepple
5 min read


End-to-End-Sales-Azure-Data-Engineering-Project-With-Databricks-AssetsBundle-CI-CD
This project showcases a complete end-to-end Azure Data Engineering solution, handling data from ingestion through transformation to delivery using modern, cloud-native best practices. The architecture leverages Azure Data Factory, Azure Databricks, and Delta Live Tables (DLT) to create a scalable, reliable, and production-ready data pipeline. The pipeline is designed to manage incremental data ingestion, support schema evolution, implement change data capture (CDC), and enab
Jesse Pepple
5 min read


MusicStreaming Azure Data Engineering Project With CI/CD And Databricks Asset Bundles
This project delivers a fully integrated Azure Data Engineering solution that manages the entire data lifecycle—from ingestion to transformation and final delivery. Built using modern cloud-native best practices, the architecture combines Azure SQL, Azure Data Factory, Azure Databricks, Logic App and Delta Live Tables (DLT) to create a highly scalable, reliable, and production-grade data pipeline designed for real-world analytics and business intelligence workloads. Resources
Jesse Pepple
8 min read


Flights Azure Databricks End To End DataEngineering Project
Resources Azure Services Azure Databricks Spark Structured Streaming Delta Live Tables Delta Lake & Delta Tables Databricks SQL Warehouse Unity Catalog Languages & Tools: Python (PySpark / notebooks) & SQL Overall Project Impact End-to-End Automation: Bronze → Silver → Gold pipeline fully automated with streaming, DLT, and UPSERTs Incremental Data Handling: 100% incremental load success → reduced processing time and cost Data Quality & Reliability: All DLT expectations
Jesse Pepple
6 min read


Azure-Databricks-End-to-End-DataEngineering-Project-With-Azure-Devops
This Data Engineering project demonstrates the full potential of Azure Databricks , Azure Data Factory (ADF) , and Azure Data Lake , showcasing how modern data solutions can be built with scalability and automation in mind. The project incorporates key engineering practices such as Slowly Changing Dimensions (SCD) Type 1 — implemented manually — and Type 2 , automated through Delta Live Tables (DLT) , alongside Star Schema modeling and incremental data loading . Using Spark
Jesse Pepple
8 min read


Olympics Data Engineering Project With Azure DevOps
This project focused on building an end-to-end Azure & Databricks data pipeline using the Olympics 2024 dataset . I began by provisioning the required Azure resources and designing the solution around a Medallion Architecture (Bronze → Silver → Gold) to ensure scalability, data quality, and reliability. Data was ingested using Azure Data Factory (ADF) and stored in Azure Data Lake Storage , establishing the raw Bronze layer. From there, I applied data governance through Un
Jesse Pepple
8 min read


Azure Netflix Data Engineering Project
In this project, I implemented Unity Catalog in Databricks to establish secure, centralized data governance , ensuring controlled access and compliance across all datasets. The solution leveraged Delta Lake and Delta Tables to provide scalable, reliable, and efficient storage , while the pipeline followed the Medallion Architecture (Bronze → Silver → Gold) to maintain data quality, reliability, and usability at every stage. The end-to-end workflow utilized Azure Data Fact
Jesse Pepple
6 min read


Cars Data Engineering Project
This project implemented Unity Catalog in Databricks to provide secure, centralized data governance, while leveraging Delta Lake and Delta Tables for scalable and efficient data storage. The data pipeline followed the Medallion Architecture (Bronze → Silver → Gold), ensuring data quality, reliability, and usability at every stage. The solution was built using Azure Data Factory, Azure SQL Database, Azure Data Lake Storage, and Azure Databricks, with Python and SQL as the prim
Jesse Pepple
5 min read
bottom of page