Category: Data Engineering

2026-05-31 • Alex Merced

Data Platform Native AI Agent Tooling in 2026

A comprehensive comparison of AI agent tooling across Dremio, Snowflake, Databricks, Microsoft Fabric, AWS, Google Cloud...

2026-05-24 • Alex Merced

How dbt Fusion Reshapes Analytics Engineering

dbt Fusion entered public beta in May 2025 with a Rust-powered runtime that changes how analytics engineers develop, val...

2026-05-24 • Alex Merced

Using DuckDB and Polars to Query Iceberg Tables

DuckDB 1.4 LTS and Polars streaming engine now both support reading and writing Apache Iceberg tables. Learn how to use ...

2026-05-24 • Alex Merced

Kafka 4.0 Changes Streaming Platform Operations

Kafka 4.0 removes ZooKeeper and ships KRaft and KIP-848 by default. Learn what those changes mean for platform operation...

2026-05-24 • Alex Merced

OpenLineage as the Spine of Data Observability

OpenLineage provides a standard API for collecting pipeline lineage across Airflow, Spark, Flink, and dbt. Learn how it ...

2026-05-23 • Alex Merced

Single-Node Data Engineering: DuckDB, DataFusion, Polars, and LakeSail

Optimize single-node data engineering with DuckDB, DataFusion, Polars, and LakeSail. Compare architectures and learn whe...

2026-04-29 • Alex Merced

How Query Engines Think: The Tradeoffs Behind Every Data System

Every database is a collection of engineering tradeoffs. Learn the 9 design decisions that shape how query engines store...

2026-04-29 • Alex Merced

Row vs. Column: How Storage Layout Shapes Everything

Row stores keep records together for fast transactions. Column stores keep field values together for fast analytics. Her...

2026-04-29 • Alex Merced

How Databases Organize Data on Disk: Pages, Blocks, and File Formats

Databases structure data on disk as heap files, sorted files, or LSM trees, then wrap it in formats like Parquet with me...

2026-04-29 • Alex Merced

B-Trees, LSM Trees, and the Indexing Tradeoff Spectrum

B-trees balance reads and writes for OLTP. LSM trees maximize write throughput. Bitmap indexes accelerate OLAP filtering...

2026-04-29 • Alex Merced

Inside the Query Optimizer: How Engines Pick a Plan

Query optimizers transform SQL into execution plans using rule-based rewrites, cost-based search, and adaptive runtime a...

2026-04-29 • Alex Merced

Volcano, Vectorized, Compiled: How Engines Execute Your Query

The Volcano model processes one row at a time. Vectorized execution processes batches with SIMD. Code generation fuses o...

2026-04-29 • Alex Merced

Buffer Pools, Caches, and the Memory Hierarchy

Databases use buffer pools, column caches, and result caches to keep hot data in RAM. Here is how each caching strategy ...

2026-04-29 • Alex Merced

Partitioning, Sharding, and Data Distribution Strategies

Hash partitioning distributes data evenly. Range partitioning enables fast range scans. Both create tradeoffs. Here is h...

2026-04-29 • Alex Merced

Hash, Sort-Merge, Broadcast: How Distributed Joins Work

Distributed joins move data across the network using shuffle, broadcast, or co-location strategies. Here is how each wor...

2026-04-29 • Alex Merced

Concurrency, Isolation, and MVCC: How Engines Handle Contention

Databases handle concurrent access using locks, MVCC, or optimistic concurrency control. Here is how each approach works...

2026-04-13 • Alex Merced

Apache Software Foundation: History, Purpose, and Process

Part of the Open Source and the Lakehouse series...

2026-04-13 • Alex Merced

What is Apache Parquet? Columns, Encoding, and Performance

Part of the Open Source and the Lakehouse series...

2026-04-13 • Alex Merced

What is Apache Iceberg? The Table Format Revolution

Part of the Open Source and the Lakehouse series...

2026-04-13 • Alex Merced

What is Apache Polaris? Unifying the Iceberg Ecosystem

Part of the Open Source and the Lakehouse series...