data-lake

data load tool (dlt) is an open source Python library that makes data loading easy 🛠️

lakeFS - Data version control for your data lake | Git for data

Apache Kyuubi is a distributed and multi-tenant gateway to provide serverless SQL on data warehouses and lakehouses.

Few projects related to Data Engineering including Data Modeling, Infrastructure setup on cloud, Data Warehousing and Data Lake development.

BitSail is a distributed high-performance data integration engine which supports batch, streaming and incremental scenarios. BitSail is widely used to synchronize hundreds of trillions of data every day.

An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.

Apache Iceberg REST Catalog in Rust — access control, credential vending and audit for every engine and AI agent. Apache 2.0.

Apache Amoro(incubating) is a Lakehouse management system built on open data lake formats.

Kylo is a data lake management software platform and framework for enabling scalable enterprise-class data lakes on big data technologies such as Teradata, Apache Spark and/or Hadoop. Kylo is licensed under Apache 2.0. Contributed by Teradata Inc.

Personal Data Engineering Projects

开源中国股市（A股）本地数据湖：沪深股票行情、A股股票数据与金融数据，面向量化研究、量化交易与回测，含财务数据、期货数据、资金流向、公告、指数行业与宏观。55 个日更数据集，复权与历史成分内置，行级溯源、PIT，MCP 原生。免注册、免 Token，自托管。Open-source China stock market data lake for quant research, backtesting and AI agents.

An efficient storage and compute engine for both on-prem and cloud-native data analytics.

Data API Framework for AI Agents and Data Apps

This repository has been merged into Canner/WrenAI under the core/ directory

Enterprise-grade, production-hardened, serverless data lake on AWS

Generic Data Ingestion & Dispersal Library for Hadoop

Real Time Big Data / IoT Machine Learning (Model Training and Inference) with HiveMQ (MQTT), TensorFlow IO and Apache Kafka - no additional data store like S3, HDFS or Spark required

Use SQL to build ELT pipelines on a data lakehouse.

BtrBlocks: Efficient Columnar Compression for Data Lakes (SIGMOD 2023 Paper)

Amazon S3 Find and Forget is a solution to handle data erasure requests from data lakes stored on Amazon S3, for example, pursuant to the European General Data Protection Regulation (GDPR)