scd2

Open Source data engineering demo project using dbt, DuckDB, dlt, Dagster and Metabase.

Slowly Changing Dimension Type 2 (scd2) custom materialization

SCD2 implementation using pyspark

snap_join is a dbt macro that joins multiple source dbt snapshots or scd2 tables into a single new table.

SCD2 implementation using pyspark

A modern banking data pipeline built with Dagster and DBT!

Fortune-500-grade banking analytics platform: OLTP -> medallion lakehouse -> Kimball star schema -> semantic layer -> 9-tab executive dashboard + 5 ML models (churn, fraud, segmentation, forecasting). Production-ready, governed, fully tested.

Incremental EPIC Clarity ingestion into Databricks using Epic CSA tables: Bronze staging, AUTO CDC to Silver SCD Type 2 (deletes included), Lakeflow Jobs, a Control Plane app, and Unity Catalog governance end to end.

Governed ARR, retention, and AI-adoption metrics with SCD2 modeling, role-based access, and safe SQL compilation.

Legacy HR and payroll data migrated into a Workday-shaped target: staged, cleansed, validated, loaded and reconciled to the cent, with every rejection and change traced to a numbered rule. Pure Python + SQLite, zero dependencies, with an interactive replay of the run.

PostgreSQL service-desk DB for a residential complex: SCD2 dimension, validation triggers, partitioning, EXPLAIN demos and a seeded Faker data generator

This is a data engineering pipeline built on Databricks + Delta Lake + PySpark that ingests travel booking and customer master data, applies SCD Type 2 logic, and delivers analytics-ready tables. It includes data quality enforcement, dimension versioning, fact aggregation, and performance tuning.

ELT pipeline with dbt & Snowflake on Olist dataset. Medallion architecture, dimensional modeling, SCD2, RFM and Cortex AI sentiment analysis.

Pipeline ETL MySQL en 3 couches - staging, modele en etoile avec SCD Type 2, marts analytiques. Orchestrateur Python, 18 tests de coherence inter-couches

Analytics data warehouse for a production livestock ERP — dbt + BigQuery, 5-layer architecture, DAMA-DMBOK governance implemented as executable tests.

Pipeline 100% Open Source

PostgreSQL data warehouse for government citizen service analytics: Medallion layers, star schema with five SCD2 dimensions, and explicit handling of five source data-quality defects

Convention-aware integrity auditor for SCD Type 2 dimensions. 16 SQL checks covering overlaps, gaps, current-flag drift, sentinel drift and point-in-time referential breaks, with findings ranked by the fact rows and revenue at risk and idempotent repair SQL in DuckDB and Snowflake via sqlglot. 1.28M version rows audited in 3.95s.

Modern data stack reference: dbt + BigQuery + Airflow (Cloud Composer) with medallion layering, SCD2 snapshots, exposures, freshness SLAs, and 45× cost reduction via partition + cluster + incremental tuning.

Read-only temporal-integrity validator for SCD Type 2 dimensions. Runs seven DuckDB window-function checks (overlaps, gaps, multiple current rows, and more) and reports each violation's blast radius in keys, fact rows, and dollars.