Data engineering · science · architecture

IngestThis

Working notes for people who move data for a living. Pipelines, lakehouses, table formats, and the architecture underneath them.

Read the archiveJoin the Slack
Articles
500+
Topics
Iceberg, pipelines, AI
Submissions
Open
Price
Free

Latest

All articles
  1. 01Agent-Driven Storage Tiering for Apache Iceberg: Moving Cold Data Without Breaking QueriesA background agent can move cold Iceberg partitions to cheaper tiers without breaking live queries. Heatmaps, path-safe moves, and restore paths.2026-08-25
  2. 02Securing the Agentic Lakehouse Gateway: Preventing Prompt Injection and Data ExfiltrationAgentic lakehouse gateways face prompt injection and exfiltration through query results. A threat model and defenses for the layer in front of data.2026-08-25
  3. 03Apache Ossie and Apache Polaris: Putting Semantic Models in the Open CatalogApache Ossie and Polaris put metric definitions in the open catalog. What the spec covers, what Polaris stores, and what is still unfinished.2026-08-25
  4. 04Arrow Flight SQL and ADBC: Why the Database Driver Is the Slowest Part of Your QueryJDBC and ODBC often dominate large-result time. Flight SQL and ADBC keep data columnar from server to client, with Python, Go, and Rust examples.2026-08-25
  5. 05DataFusion Comet 1.0 and What Native Rust Scans Change for Spark on IcebergDataFusion Comet 1.0 replaces Spark Iceberg scans with native Rust. What speeds up, what still falls back to the JVM, and how to deploy it.2026-08-25
  6. 06FSST and ALP: The Two Encodings Fixing Parquet's Weakest Compression CasesALP and FSST target Parquet's worst cases: floats and high-cardinality strings. How they work and what they change for Iceberg tables.2026-08-25
  7. 07Governance-as-Code for the Lakehouse: Managing REST Catalog RBAC and Masking in GitPut REST catalog RBAC and masking in Git. How to review grants, apply them safely, and keep lakehouse access from drifting.2026-08-25
  8. 08Metric Contracts in Code: Testing, Versioning, and Serving Business Logic to Multi-Agent SystemsMetric contracts in code let teams test, version, and serve business logic to multi-agent systems without each agent inventing its own SQL.2026-08-25

Reference shelf

Long reads from around the web
Semantic Layer

The Semantic Layer: Definitive Guide

A comprehensive guide to the Semantic Layer — how it creates a single source of truth for metrics, powers headless BI, and makes AI agents answer business questions accurately.

ReadApache Polaris

Apache Polaris: The Catalog Standard for Lakehouses and AI

How Apache Polaris is emerging as the universal Iceberg catalog standard, enabling multi-engine interoperability and governed AI access across the lakehouse ecosystem.

ReadTable Formats

What Are Table Formats and Why Were They Needed?

The origin story of open table formats — the problems with Hive, why Apache Iceberg, Delta Lake, and Hudi were created, and what they unlock for modern data platforms.

ReadDremio

What Is Dremio?

A clear-eyed breakdown of what Dremio is, how its semantic layer, query federation, Reflections, and Apache Arrow Flight power the Intelligent Lakehouse Platform.

ReadApache Iceberg

What Apache Iceberg Native Actually Means

Not all 'Iceberg support' is equal. This piece breaks down what it means to be genuinely Apache Iceberg native versus bolt-on, and why it matters for your lakehouse.

ReadOpen Source

Open Source and the Data Lakehouse

How the Apache Software Foundation's open-source projects — Iceberg, Arrow, Parquet, Polaris — form the modular foundation of the modern open data lakehouse.

ReadAgentic AI

What Is Agentic Analytics?

Agentic AI is reshaping how organizations interact with data. This guide explains agentic analytics, the role of the semantic layer, and why query performance matters for AI agents.

ReadData Lakehouse

Definitive Guide to the Data Lakehouse

The complete, authoritative guide to the Data Lakehouse architecture — what it is, why it supersedes the data warehouse + data lake combination, and how to build one.

ReadAI & Performance

How Dremio Keeps Agentic Analytics Fast Without Manual Tuning

How Dremio's layered autonomous performance architecture — Reflections, caching, vectorized execution — handles unpredictable AI agent query patterns at interactive speed.

Read

Elsewhere

Pitch an idea: alex@ingestthis.com