Awesome Data Engineering

A curated list of awesome things related to Data Engineering.

Contents

• Databases
• Data Comparison
• Data Ingestion
• File System
• Serialization format
• Stream Processing
• Batch Processing
• Charts and Dashboards
• Workflow
• Data Lake Management
• ELK Elastic Logstash Kibana
• Docker
• Datasets
  • Realtime
  • Data Dumps
• Monitoring
  • Prometheus
• Profiling
  • Data Profiler
• Schema
• Testing
• Community
  • Forums
  • Conferences
  • Podcasts
  • Books

Databases

• Relational
  • RQLite - Replicated SQLite using the Raft consensus protocol.
  • MySQL - The world's most popular open source database.
    • TiDB - A distributed NewSQL database compatible with MySQL protocol.
    • Percona XtraBackup - A free, open source, complete online backup solution for all versions of Percona Server, MySQL® and MariaDB®.
    • mysql_utils - Pinterest MySQL Management Tools.
  • MariaDB - An enhanced, drop-in replacement for MySQL.
  • PostgreSQL - The world's most advanced open source database.
  • Rivestack - Managed PostgreSQL with pgvector for AI workloads. HNSW indexing, sub-4ms latency, and built-in SQL editor with automatic embedding generation.
  • Amazon RDS - Makes it easy to set up, operate, and scale a relational database in the cloud.
  • Crate.IO - Scalable SQL database with the NOSQL goodies.
• Key-Value
  • Redis - An open source, BSD licensed, advanced key-value cache and store.
  • Riak - A distributed database designed to deliver maximum data availability by distributing data across multiple servers.
  • AWS DynamoDB - A fast and flexible NoSQL database service for all applications that need consistent, single-digit millisecond latency at any scale.

Data Comparison

• datacompy - A Python library that facilitates the comparison of two DataFrames in Pandas, Polars, Spark and more. The library goes beyond basic equality checks by providing detailed insights into discrepancies at both row and column levels.
• dvt - Data Validation Tool compares data from source and target tables to ensure that they match. It provides column validation, row validation, schema validation, custom query validation, and ad hoc SQL exploration.
• koala-diff - A high-performance Python library for comparing large datasets (CSV, Parquet) locally using Rust and Polars. It features zero-copy streaming to prevent OOM errors and generates interactive HTML data quality reports.
• FutureSearch SDK - Python SDK that dispatches parallel web-research agents across table rows, synthesizing multi-agent findings into structured columns.

Data Ingestion

• DataSpoc Pipe - Data ingestion engine that connects 400+ Singer taps to Parquet files in cloud buckets (S3, GCS, Azure). Streaming, incremental, with auto-catalog.
• Enrich.sh - Managed event ingestion service that converts JSON sent to a REST API into Hive-partitioned Parquet on Cloudflare R2, queryable from DuckDB, ClickHouse, BigQuery, Snowflake, and Python.
• enrich-companies - CLI tool to enrich CSV files with company data (financials, contacts, metadata) from 250M+ company records. Available on npm.
• ingestr - CLI tool to copy data between databases with a single command. Supports 50+ sources including PostgreSQL, MySQL, MongoDB, Salesforce, Shopify to any data warehouse.
• Kafka - Publish-subscribe messaging rethought as a distributed commit log.
  • BottledWater - Change data capture from PostgreSQL into Kafka. Deprecated.
  • kafkat - Simplified command-line administration for Kafka brokers.
  • kafkacat - Generic command line non-JVM Apache Kafka producer and consumer.
  • pg-kafka - A PostgreSQL extension to produce messages to Apache Kafka.
  • librdkafka - The Apache Kafka C/C++ library.
  • kafka-docker - Kafka in Docker.
  • kafka-manager - A tool for managing Apache Kafka.

File System

• HDFS - A distributed file system designed to run on commodity hardware.
  • Snakebite - A pure python HDFS client.
• AWS S3 - Object storage built to retrieve any amount of data from anywhere.
  • smart_open - Utils for streaming large files (S3, HDFS, gzip, bz2).
• Alluxio - A memory-centric distributed storage system enabling reliable data sharing at memory-speed across cluster frameworks, such as Spark and MapReduce.
• CEPH - A unified, distributed storage system designed for excellent performance, reliability, and scalability.
• JuiceFS - A high-performance Cloud-Native file system driven by object storage for large-scale data storage.
• OrangeFS - Orange File System is a branch of the Parallel Virtual File System.
• SnackFS - A bite-sized, lightweight HDFS compatible file system built over Cassandra.
• GlusterFS - Gluster Filesystem.
• XtreemFS - Fault-tolerant distributed file system for all storage needs.
• SeaweedFS - Seaweed-FS is a simple and highly scalable distributed file system. There are two objectives: to store billions of files! to serve the files fast! Instead of supporting full POSIX file system semantics, Seaweed-FS choose to implement only a key~file mapping. Similar to the word "NoSQL", you can call it as "NoFS".

Serialization format

• AKF - The AI native file format. Trust scores, source provenance, and compliance metadata that embed into 20+ formats (DOCX, PDF, images, code). EXIF for AI.
• Apache Avro - Apache Avro™ is a data serialization system.
• Apache Parquet - A columnar storage format available to any project in the Hadoop ecosystem, regardless of the choice of data processing framework, data model or programming language.
  • Snappy - A fast compressor/decompressor. Used with Parquet.
  • PigZ - A parallel implementation of gzip for modern multi-processor, multi-core machines.
• Apache ORC - The smallest, fastest columnar storage for Hadoop workloads.
• Apache Thrift - The Apache Thrift software framework, for scalable cross-language services development.
• ProtoBuf - Protocol Buffers - Google's data interchange format.
• SequenceFile - A flat file consisting of binary key/value pairs. It is extensively used in MapReduce as input/output formats.
• Kryo - A fast and efficient object graph serialization framework for Java.
• PFC-JSONL - Specialized JSONL log compressor with block-level timestamp indexing and DuckDB integration. Achieves ~9% compression ratio (better than gzip) with time-range random access queries.
• ParquetKit - Browser-based viewer, SQL workbench and converter for Parquet files powered by DuckDB-WASM. Fully client-side, no upload.

Stream Processing

• Apache Beam - A unified programming model that implements both batch and streaming data processing jobs that run on many execution engines.
• Spark Streaming - Makes it easy to build scalable fault-tolerant streaming applications.
• Apache Flink - A streaming dataflow engine that provides data distribution, communication, and fault tolerance for distributed computations over data streams.
• Apache Storm - A free and open source distributed realtime computation system.
• Apache Samza - A distributed stream processing framework.
• Apache NiFi - An easy to use, powerful, and reliable system to process and distribute data.
• Apache Hudi - An open source framework for managing storage for real time processing, one of the most interesting feature is the Upsert.
• CocoIndex - An open source ETL framework to build fresh index for AI.
• VoltDB - An ACID-compliant RDBMS which uses a shared nothing architecture.
• PipelineDB - The Streaming SQL Database.
• Spring Cloud Dataflow - Streaming and tasks execution between Spring Boot apps.
• Bonobo - A data-processing toolkit for python 3.5+.
• Robinhood's Faust - Forever scalable event processing & in-memory durable K/V store as a library with asyncio & static typing.

Batch Processing

• Hadoop MapReduce - A software framework for easily writing applications which process vast amounts of data (multi-terabyte data-sets) - in-parallel on large clusters (thousands of nodes) - of commodity hardware in a reliable, fault-tolerant manner.
• Spark - A multi-language engine for executing data engineering, data science, and machine learning on single-node machines or clusters.
  • Spark Packages - A community index of packages for Apache Spark.
  • Deep Spark - Connecting Apache Spark with different data stores. Deprecated.
  • Spark RDD API Examples - Examples by Zhen He.
  • Livy - The REST Spark Server.
  • Delight - A free & cross platform monitoring tool (Spark UI / Spark History Server alternative).
• AWS EMR - A web service that makes it easy to quickly and cost-effectively process vast amounts of data.
• Data Mechanics - A cloud-based platform deployed on Kubernetes making Apache Spark more developer-friendly and cost-effective.
• Tez - An application framework which allows for a complex directed-acyclic-graph of tasks for processing data.
• Bistro - A light-weight engine for general-purpose data processing including both batch and stream analytics. It is based on a novel unique data model, which represents data via functions and processes data via columns operations as opposed to having only set operations in conventional approaches like MapReduce or SQL.

Charts and Dashboards

• Highcharts - A charting library written in pure JavaScript, offering an easy way of adding interactive charts to your web site or web application.
• ZingChart - Fast JavaScript charts for any data set.
• C3.js - D3-based reusable chart library.
• D3.js - A JavaScript library for manipulating documents based on data.
  • D3Plus - D3's simpler, easier to use cousin. Mostly predefined templates that you can just plug data in.
• SmoothieCharts - A JavaScript Charting Library for Streaming Data.
• PyXley - Python helpers for building dashboards using Flask and React.
• Plotly - Flask, JS, and CSS boilerplate for interactive, web-based visualization apps in Python.
• Apache Superset - A modern, enterprise-ready business intelligence web application.
• Redash - Make Your Company Data Driven. Connect to any data source, easily visualize and share your data.
• Metabase - The easy, open source way for everyone in your company to ask questions and learn from data.
• stratif.io - Open-source, self-hosted, warehouse-native product analytics. Runs funnels, retention, and paths directly on DuckDB, Postgres, Snowflake, or ClickHouse.
• PyQtGraph - A pure-python graphics and GUI library built on PyQt4 / PySide and numpy. It is intended for use in mathematics / scientific / engineering applications.

Workflow

• Bonnard - Governed, multi-tenant MCP access to your customers' data. Turn your warehouse, dbt, or semantic layer into a secure, per-customer MCP for AI agents.
• Nika - Intent-as-code workflow engine for AI data pipelines: reviewable YAML DAGs statically checked (schema, permits, cost floor) before execution, with tamper-evident run traces.
• OrionBelt Semantic Layer - Open-source semantic sidecar that compiles YAML-defined dimensions, measures, and metrics into optimized SQL across 8 engines (BigQuery, ClickHouse, Databricks, Dremio, DuckDB, MySQL, PostgreSQL, Snowflake). Unified REST, MCP, and Postgres wire protocol; one model powers AI agents, analytics, DQ rules, and KPIs.
• Bruin - End-to-end data pipeline tool that combines ingestion, transformation (SQL + Python), and data quality in a single CLI. Connects to BigQuery, Snowflake, PostgreSQL, Redshift, and more. Includes VS Code extension with live previews.
• DataFlow - Open-source platform for data preparation, synthetic data generation, and AI/data pipelines. Includes reusable skills for automating workflow steps across data and AI tasks.
• Luigi - A Python module that helps you build complex pipelines of batch jobs.
• CronQ - An application cron-like system. Used w/Luigi. Deprecated.
• Cascading - Java based application development platform.
• Airflow - A system to programmatically author, schedule, and monitor data pipelines.
• Azkaban - A batch workflow job scheduler created at LinkedIn to run Hadoop jobs. Azkaban resolves the ordering through job dependencies and provides an easy-to-use web user interface to maintain and track your workflows.

Data Lake Management

• lakeFS - An open source platform that delivers resilience and manageability to object-storage based data lakes.
• Project Nessie - A Transactional Catalog for Data Lakes with Git-like semantics. Works with Apache Iceberg tables.
• Ilum - A modular Data Lakehouse platform that simplifies the management and monitoring of Apache Spark clusters across Kubernetes and Hadoop environments.
• Gravitino - An open-source, unified metadata management for data lakes, data warehouses, and external catalogs.
• FlightPath Data - FlightPath is a gateway to a data lake's bronze layer, protecting it from invalid external data file feeds as a trusted publisher.
• rawquery - Managed lakehouse platform on Apache Iceberg with DuckDB query compute, S3 storage, Postgres wire protocol, and SQL transforms.

ELK Elastic Logstash Kibana

• docker-logstash - A highly configurable Logstash (1.4.4) - Docker image running Elasticsearch (1.7.0) - and Kibana (3.1.2).
• elasticsearch-jdbc - JDBC importer for Elasticsearch.
• ZomboDB - PostgreSQL Extension that allows creating an index backed by Elasticsearch.

Docker

• Gockerize - Package golang service into minimal Docker containers.
• Flocker - Easily manage Docker containers & their data.
• Rancher - RancherOS is a 20mb Linux distro that runs the entire OS as Docker containers.
• Kontena - Application Containers for Masses.
• Weave - Weaving Docker containers into applications.
• Zodiac - A lightweight tool for easy deployment and rollback of dockerized applications.
• cAdvisor - Analyzes resource usage and performance characteristics of running containers.
• Micro S3 persistence - Docker microservice for saving/restoring volume data to S3.
• Rocker-compose - Docker composition tool with idempotency features for deploying apps composed of multiple containers. Deprecated.
• Nomad - A cluster manager, designed for both long-lived services and short-lived batch processing workloads.
• ImageLayers - Visualize Docker images and the layers that compose them.

Datasets

Realtime

• DexPaprika - DEX data via SSE streaming across 36 blockchains. 36M+ pools, 33M+ tokens. Metered free tier, no API key required, data delayed up to 15s; real-time on the paid tier. Docs
• Helium MCP - Remote MCP server for real-time financial data, 3.2M+ news articles, ML options pricing, and news bias analysis. Free, no API key. MCP
• Twitter Realtime - The Streaming APIs give developers low latency access to Twitter's global stream of Tweet data.
• Sorsa API - Real-time X (Twitter) data API providing tweets, profiles, search, communities and engagement metrics. Up to 50x cheaper than the official X API with 20 req/sec rate limit, JSON output.
• Eventsim - Event data simulator. Generates a stream of pseudo-random events from a set of users, designed to simulate web traffic.
• Eventum - Data generation platform for producing synthetic event streams with complex correlations.
• Reddit - Real-time data is available including comments, submissions and links posted to reddit.

Data Dumps

• GitHub Archive - GitHub's public timeline since 2011, updated every hour.
• Common Crawl - Open source repository of web crawl data.
• Wikipedia - Wikipedia's complete copy of all wikis, in the form of Wikitext source and metadata embedded in XML. A number of raw database tables in SQL form are also available.
• The Quiet-Broke Index - A 30-metro composite of US household cost burdens (housing, taxes, childcare, healthcare, transport) aggregated from Census ACS, BLS Consumer Expenditure Survey, and HUD Fair Market Rents. Open methodology, free, no email gate.
• FirstData - The world's most comprehensive authoritative data source knowledge base. 160+ curated sources from governments, international organizations, and research institutions with MCP integration.
• Mindweave Synthetic Business Data - 42-table synthetic SME dataset with double-entry accounting, tax compliance (AU/US/UK), and temporal realism. CSV, SQL, Parquet, SQLite. Ideal for ETL pipeline testing.
• LatAm Synth - Synthetic financial savings behavior generator for Latin America: users, savings goals, and transactions calibrated on 506K real records (2015–2024). Reproducible by seed, 100% synthetic.

Monitoring

Prometheus

• Prometheus.io - An open-source service monitoring system and time series database.
• HAProxy Exporter - Simple server that scrapes HAProxy stats and exports them via HTTP for Prometheus consumption.
• Signals CLI - Intent signal monitoring CLI. Track LinkedIn engagers, keyword posters, job changers, funding events. JSON output for data pipelines.

Profiling

Data Profiler

• Data Profiler - The DataProfiler is a Python library designed to make data analysis, monitoring, and sensitive data detection easy.
• YData Profiling - A general-purpose open-source data profiler for high-level analysis of a dataset.
• Desbordante - An open-source data profiler specifically focused on discovery and validation of complex patterns in data.

Schema

• SchemaCrawler - Open-source and free relational database schema discovery and comprehension tool. Documents and diagrams relational database schemas from your Java programs, build tools and the command line. Find database design issues with lint, and write scripts against the database. Includes an MCP Server for use by AI agents.

Testing

• Aegis DQ - Open-source agentic data quality framework with LLM-powered diagnosis, root-cause analysis, SQL auto-fix proposals, and 31 rule types — DuckDB, Postgres, BigQuery, Databricks, Athena, Snowflake.
• Grai - A data catalog tool that integrates into your CI system exposing downstream impact testing of data changes. These tests prevent data changes which might break data pipelines or BI dashboards from making it to production.
• DQOps - An open-source data quality platform for the whole data platform lifecycle from profiling new data sources to applying full automation of data quality monitoring.
• DataKitchen - Open Source Data Observability for end-to-end Data Journey Observability, data profiling, anomaly detection, and auto-created data quality validation tests.
• GreatExpectation - Open Source data validation framework to manage data quality. Users can define and document “expectations” rules about how data should look and behave.
• Provero - A vendor-neutral, declarative data quality engine. Define checks in YAML, run anywhere. Includes 16 built-in check types, SQL batch optimizer, anomaly detection, and data contracts.
• Scherlok - Zero-config data quality CLI. Profiles every table on first run, then auto-detects anomalies (volume drops, schema drift, freshness misses, distribution shifts) on subsequent runs. No YAML, no rules to write. Works with Postgres, BigQuery, Snowflake, and dbt.
• RunSQL - Free online SQL playground for MySQL, PostgreSQL, and SQL Server. Create database structures, run queries, and share results instantly.
• Spark Playground - Write, run, and test PySpark code on Spark Playground's online compiler. Access real-world sample datasets & solve interview questions to enhance your PySpark skills for data engineering roles.

Community

Forums

• /r/dataengineering - News, tips, and background on Data Engineering.
• /r/etl - Subreddit focused on ETL.
• AI Dev Jobs - Job board focused on AI, ML, and data engineering roles with 7,400+ listings, salary data, and a free REST API.

Conferences

• Data Council - The first technical conference that bridges the gap between data scientists, data engineers and data analysts.

Podcasts

• Chain of Thought - Interviews with AI and data infrastructure leaders on building production systems.
• Data Engineering Podcast - The show about modern data infrastructure.
• Latent Space - Technical deep dives on AI engineering, from model training to deployment.
• Practical AI - Making AI practical, productive, and accessible to everyone.
• Software Engineering Daily - Daily interviews about technical software topics, including data infrastructure.
• The Analytics Engineering Podcast - How analytics engineers build and maintain data pipelines at scale.
• The Data Stack Show - A show where they talk to data engineers, analysts, and data scientists about their experience around building and maintaining data infrastructure, delivering data and data products, and driving better outcomes across their businesses with data.

Books

• Snowflake Data Engineering - A practical introduction to data engineering on the Snowflake cloud data platform.
• Best Data Science Books - This blog offers a curated list of top data science books, categorized by topics and learning stages, to aid readers in building foundational knowledge and staying updated with industry trends.
• Architecting an Apache Iceberg Lakehouse - A guide to designing an Apache Iceberg lakehouse from scratch.
• Learn AI Data Engineering in a Month of Lunches - A fast, friendly guide to integrating large language models into your data workflows.