# IngestThis

> Articles, tutorials, and resources for Data Engineers, Scientists, Analysts, and Architects.

## About

IngestThis is a technical blog focused on the modern data ecosystem — covering Data Engineering fundamentals, Apache Iceberg, Data Lakehouses, AI-ready data infrastructure, and developer tools. It is authored primarily by Alex Merced (Head of Developer Relations at Dremio), with guest contributions from the data community.

- **Author:** Alex Merced
- **Author Site:** https://www.alexmercedcoder.dev
- **Author LinkedIn:** https://www.linkedin.com/in/alexmerced
- **Author Twitter:** https://www.twitter.com/alexmercedcoder
- **Contact:** alex@ingestthis.com
- **Community:** https://join.slack.com/t/thedatalakehousehub/shared_invite/zt-274yc8sza-mI2zhCW8LGkOh1uxuf8T5Q
- **RSS Feed:** https://ingestthis.com/feed.xml
- **Sitemap:** https://ingestthis.com/sitemap.xml

## Topics Covered

- AI
- AI & Agents
- AI & Data
- AI & Machine Learning
- AI & Security
- AI & Society
- AI Tools & Software Development
- Agentic AI
- Agentic Analytics
- Agentic Lakehouse
- Apache Arrow
- Apache Iceberg
- Apache Parquet
- Apache Polaris
- Artificial Intelligence
- Data Analytics
- Data Architecture
- Data Engineering
- Data Governance
- Data Lakehouse
- Data Modeling
- Data Platforms
- Data Quality
- DevOps
- Devops
- Governance
- Hardware
- Javascript
- Lakehouse
- MCP
- Observability
- Open Source
- Security
- Security & Governance
- Semantic Layer
- Software Development
- Technology & Culture
- data
- data engineering
- data lakehouse
- database
- frontend
- oltp
- python
- rust
- sql

## Technical Articles & Tutorials

- [Data Quality Tooling Compared: Great Expectations, Soda, dbt Tests, and Anomaly Detection](https://ingestthis.com/posts/2026/2026-09-02-data-quality-tooling-compared): A comparison of Great Expectations, Soda, dbt tests, and anomaly detection, and a layered design that uses each where it fits.
- [The Data Team of the Agentic Era: Generalists Owning End-to-End Workflows](https://ingestthis.com/posts/2026/2026-09-02-data-team-of-the-agentic-era): The case for generalists owning end-to-end data workflows with agents, the counterargument, and how to make the transition work.
- [dbt on Iceberg: Incremental Models on Open Tables](https://ingestthis.com/posts/2026/2026-09-02-dbt-on-iceberg-incremental-models): How dbt incremental materializations map to Iceberg operations, and the configuration, predicates, and maintenance that keep them healthy.
- [Disaster Recovery for Iceberg Tables: Replication, Backup, and Restore](https://ingestthis.com/posts/2026/2026-09-02-disaster-recovery-for-iceberg-tables): Disaster recovery for Iceberg across four tiers: snapshots, object versioning, catalog backup, and cross-region replication.
- [Deleting User Data From an Immutable Lakehouse: GDPR Hard Deletes on Iceberg](https://ingestthis.com/posts/2026/2026-09-02-gdpr-hard-deletes-on-iceberg): How to turn a logical delete on immutable Iceberg into a physical erasure across snapshots, versions, replicas, and downstream copies.
- [Geospatial Data in Apache Iceberg: Geometry, Geography, and GeoParquet](https://ingestthis.com/posts/2026/2026-09-02-geospatial-data-in-apache-iceberg): How Iceberg v3 geometry and geography types, bounding boxes, and native Parquet types give spatial data first-class standing.
- [Default Column Values and Field IDs: How Iceberg Schema Evolution Works at the Spec Level](https://ingestthis.com/posts/2026/2026-09-02-iceberg-default-values-and-field-ids): How field IDs and initial and write defaults let Iceberg change schemas on large tables without rewriting data, at the spec level.
- [The Iceberg Table Properties That Actually Matter](https://ingestthis.com/posts/2026/2026-09-02-iceberg-table-properties-that-matter): The Iceberg table properties that decide file count, pruning, write amplification, retention, and metadata growth, by workload.
- [Inside the Puffin File Format](https://ingestthis.com/posts/2026/2026-09-02-inside-the-puffin-file-format): The Puffin file format inside out, byte by byte, covering Theta sketches for distinct values and deletion vectors.
- [The Lakehouse Ingestion Tool Landscape: Fivetran, Airbyte, dlt, and CDC vs Batch](https://ingestthis.com/posts/2026/2026-09-02-lakehouse-ingestion-tools): How Fivetran, Airbyte, dlt, and CDC and streaming tools land well-behaved Apache Iceberg tables, and how to choose and maintain them.
- [Local Iceberg Development Environments: Docker, MinIO, and In-Memory Catalogs for CI](https://ingestthis.com/posts/2026/2026-09-02-local-iceberg-development-environments): Local Iceberg development environments: in-process catalogs, a Docker Compose stack with MinIO, and CI configurations that run either.
- [Metadata Platforms in 2026: DataHub, OpenMetadata, Atlan, and Catalog Convergence](https://ingestthis.com/posts/2026/2026-09-02-metadata-platforms-in-2026): How the technical catalog and the metadata platform are converging in 2026, and how to arrange the two layers for a lakehouse.
- [Moving Iceberg Tables Between Catalogs Without Rewriting Data](https://ingestthis.com/posts/2026/2026-09-02-moving-iceberg-tables-between-catalogs): Why moving Iceberg tables between catalogs is a pointer copy, and the protocol that makes a cutover safe for one table or thousands.
- [Logs, Traces, and Metrics as Tables: Building an OpenTelemetry Data Lake on Iceberg](https://ingestthis.com/posts/2026/2026-09-02-opentelemetry-data-lake-on-iceberg): Building an OpenTelemetry data lake on Iceberg: schemas for spans, logs, and metrics, ingestion, query patterns, and retention.
- [Orchestration in 2026: Airflow 3 vs Dagster vs Prefect vs Event-Driven](https://ingestthis.com/posts/2026/2026-09-02-orchestration-in-2026): Where Airflow 3, Dagster, Prefect, and event-driven triggering stand for lakehouse pipelines in 2026, after the Prefect acquisition of Dagster.
- [Postgres Meets the Lakehouse: pg_lake, pg_duckdb, and When Postgres Is Enough](https://ingestthis.com/posts/2026/2026-09-02-postgres-meets-the-lakehouse): What pg_lake, pg_duckdb, and pg_mooncake do at the Iceberg level, and honest thresholds for when Postgres is enough.
- [Schema Registries and Event Schemas: Avro, Protobuf, and JSON Schema on the Way Into the Lakehouse](https://ingestthis.com/posts/2026/2026-09-02-schema-registries-and-event-schemas): How Avro, Protobuf, and JSON Schema evolve through a registry, and how that maps to the schema evolution rules of Iceberg.
- [Storage-Partitioned Joins and the Bucket Transform](https://ingestthis.com/posts/2026/2026-09-02-storage-partitioned-joins-and-the-bucket-transform): How the spec-defined bucket transform lets engines skip the shuffle in joins, and how to set it up and keep it engaged in Spark.
- [Synthetic Data in the Lakehouse: Generation, Governance, and Testing](https://ingestthis.com/posts/2026/2026-09-02-synthetic-data-in-the-lakehouse): What synthetic data in a lakehouse is for, the generation methods, how to preserve fidelity, and where synthetic tables belong.
- [Will AI Replace Data Engineers?](https://ingestthis.com/posts/2026/2026-09-02-will-ai-replace-data-engineers): What the evidence shows about whether AI replaces data engineers, which parts of the job compress, and which parts do not.
- [Agent-Driven Storage Tiering for Apache Iceberg: Moving Cold Data Without Breaking Queries](https://ingestthis.com/posts/2026/2026-08-25-agent-driven-iceberg-storage-tiering): A background agent can move cold Iceberg partitions to cheaper tiers without breaking live queries. Heatmaps, path-safe moves, and restore paths.
- [Securing the Agentic Lakehouse Gateway: Preventing Prompt Injection and Data Exfiltration](https://ingestthis.com/posts/2026/2026-08-25-agentic-lakehouse-gateway-prompt-injection): Agentic lakehouse gateways face prompt injection and exfiltration through query results. A threat model and defenses for the layer in front of data.
- [Apache Ossie and Apache Polaris: Putting Semantic Models in the Open Catalog](https://ingestthis.com/posts/2026/2026-08-25-apache-ossie-polaris-semantic-models): Apache Ossie and Polaris put metric definitions in the open catalog. What the spec covers, what Polaris stores, and what is still unfinished.
- [Arrow Flight SQL and ADBC: Why the Database Driver Is the Slowest Part of Your Query](https://ingestthis.com/posts/2026/2026-08-25-arrow-flight-sql-adbc-connectivity): JDBC and ODBC often dominate large-result time. Flight SQL and ADBC keep data columnar from server to client, with Python, Go, and Rust examples.
- [DataFusion Comet 1.0 and What Native Rust Scans Change for Spark on Iceberg](https://ingestthis.com/posts/2026/2026-08-25-datafusion-comet-1-spark-iceberg): DataFusion Comet 1.0 replaces Spark Iceberg scans with native Rust. What speeds up, what still falls back to the JVM, and how to deploy it.
- [FSST and ALP: The Two Encodings Fixing Parquet's Weakest Compression Cases](https://ingestthis.com/posts/2026/2026-08-25-fsst-alp-parquet-encodings): ALP and FSST target Parquet's worst cases: floats and high-cardinality strings. How they work and what they change for Iceberg tables.
- [Governance-as-Code for the Lakehouse: Managing REST Catalog RBAC and Masking in Git](https://ingestthis.com/posts/2026/2026-08-25-governance-as-code-lakehouse-rest-catalog): Put REST catalog RBAC and masking in Git. How to review grants, apply them safely, and keep lakehouse access from drifting.
- [Metric Contracts in Code: Testing, Versioning, and Serving Business Logic to Multi-Agent Systems](https://ingestthis.com/posts/2026/2026-08-25-metric-contracts-code-multi-agent): Metric contracts in code let teams test, version, and serve business logic to multi-agent systems without each agent inventing its own SQL.
- [High-Throughput Branch Merging: Automating Concurrency and Conflict Resolution in Multi-Branch Iceberg Pipelines](https://ingestthis.com/posts/2026/2026-08-25-multi-branch-iceberg-merge-automation): High-throughput Iceberg branch merges need conflict detection and automation. How to reconcile concurrent writes without stalling pipelines.
- [Multi-Cloud REST Catalog Topologies: Running Apache Polaris Across AWS, Azure, and GCP](https://ingestthis.com/posts/2026/2026-08-25-multi-cloud-polaris-rest-catalog-topologies): Polaris can catalog Iceberg tables across AWS, Azure, and GCP. Four topologies, credential vending, and the tradeoffs of each design.
- [Parquet-Only Manifests in Iceberg v4: Why the Metadata Layer Is Going Columnar](https://ingestthis.com/posts/2026/2026-08-25-parquet-manifests-iceberg-v4): Iceberg v4 is moving manifests from Avro to Parquet so planners can read only the stats they need. Why the metadata layer is going columnar.
- [Query Routing at Machine Scale: Dynamic Workload Distribution Across Lakehouse Engines](https://ingestthis.com/posts/2026/2026-08-25-query-routing-lakehouse-engines): Route each lakehouse query by shape, not by sender. Signals, rules, and how to keep dashboards, batch jobs, and agents from sharing one engine.
- [Semantic Layer Federation: One Logical Model Over Data on Three Clouds](https://ingestthis.com/posts/2026/2026-08-25-semantic-layer-federation-three-clouds): One logical model over Iceberg and databases on three clouds. Pushdown, egress, Reflections, and where semantic federation still breaks.
- [Serverless Iceberg Ingestion with PyIceberg and DuckDB: Micro-Batches Without a Spark Cluster](https://ingestthis.com/posts/2026/2026-08-25-serverless-iceberg-microbatch-pyiceberg-duckdb): Land small Iceberg micro-batches with PyIceberg and DuckDB in a serverless function. Commits, concurrency, and why Spark is the wrong default.
- [Zero-Copy Warehouse Modernization: Migrating Legacy Databases to Apache Iceberg Without Downtime](https://ingestthis.com/posts/2026/2026-08-25-zero-copy-legacy-db-to-iceberg): Move a legacy warehouse to Iceberg without downtime by virtualizing first. Consumer cutover, parity checks, and background copy without double-ETL.
- [The Agent Is Now a Named Coworker, and It Needs a File Format](https://ingestthis.com/posts/2026/2026-08-24-agents-with-personalities): Named, persistent agents need a file format. Open Agent Profile, Buzz, Grok Bot, and Hermes Bot Mode show why a portable agent identity matters.
- [Your Agent Should Answer the Phone: A Field Guide to AI Gateways on Slack, Discord, Telegram, Signal, and Teams](https://ingestthis.com/posts/2026/2026-08-24-ai-gateways-field-guide): A field guide to AI gateways on Slack, Discord, Telegram, Signal, and Teams: architecture, auth, cost, and the failure modes that matter.
- [Graphs in AI Engineering Have Solved Three Problems. The Fourth Is the Plan.](https://ingestthis.com/posts/2026/2026-08-24-graphs-in-ai-engineering): Knowledge graphs, GraphRAG, and LangGraph solved three problems. The fourth is the work itself: a reviewable graph of bounded agentic loops.
- [The Hidden Cost of Tiny Iceberg Commits](https://ingestthis.com/posts/2026/2026-08-24-hidden-cost-of-tiny-iceberg-commits): Trace what one tiny Iceberg commit writes, then model hourly, per-minute, and per-second cadences so streaming costs become arithmetic, not adjectives.
- [Deletion Vectors vs Position Deletes vs Equality Deletes: The Iceberg Delete Story in 2026](https://ingestthis.com/posts/2026/2026-08-24-iceberg-delete-story-2026): Position deletes, equality deletes, and deletion vectors compared from the Iceberg spec: what each writes, how readers apply it, and when to use which.
- [Iceberg Is Becoming a Library, Not Just a Table Format](https://ingestthis.com/posts/2026/2026-08-24-iceberg-is-becoming-a-library): Iceberg is turning from a JVM table format into a library other systems embed. What that shift changes for engines, catalogs, and the spec itself.
- [Iceberg Is Escaping the JVM: Why Rust, Go, Python and C++ Implementations Matter](https://ingestthis.com/posts/2026/2026-08-24-iceberg-is-escaping-the-jvm): Rust, Go, Python, and C++ Iceberg implementations change who can write the format. Why multi-language clients matter more than another JVM engine.
- [The Iceberg REST Catalog Compatibility Test: One Suite of Operations Every Platform Should Pass](https://ingestthis.com/posts/2026/2026-08-24-iceberg-rest-catalog-compatibility-test): One suite of REST catalog operations every Iceberg platform should pass. What sameness means, where implementations diverge, and how to test it.
- [Iceberg REST Remote Scan Planning Changes More Than Query Performance](https://ingestthis.com/posts/2026/2026-08-24-iceberg-rest-remote-scan-planning): Remote scan planning moves Iceberg file selection into the catalog. What that changes for engines, governance, and operational cost beyond query speed.
- [Iceberg Row Lineage: The Feature AI and CDC Workloads Will Eventually Depend On](https://ingestthis.com/posts/2026/2026-08-24-iceberg-row-lineage): Iceberg row lineage gives rows a durable identity across rewrites. Why CDC pipelines and AI workloads will eventually depend on it.
- [Iceberg v4's Adaptive Metadata Tree, Explained From First Principles](https://ingestthis.com/posts/2026/2026-08-24-iceberg-v4-adaptive-metadata-tree): Iceberg v4's adaptive metadata tree, explained from first principles: why commits rewrite too much today and how the tree makes change cheaper.
- [Why Iceberg v4 Is Really About Making the Cost of Change Proportional to the Change](https://ingestthis.com/posts/2026/2026-08-24-iceberg-v4-cost-of-change-principle): Iceberg v4 is really about making the cost of a change proportional to the change. The principle, the current tax, and what the redesign pays down.
- [The Catalog Can Now Plan Your Iceberg Query: Inside REST Scan Planning](https://ingestthis.com/posts/2026/2026-08-24-inside-iceberg-rest-scan-planning): A mechanics walkthrough of Iceberg REST scan planning: client-side planning, remote endpoints, pagination, and where engine support stands in 2026.
- [Can Seven Different Iceberg REST Catalogs Really Run the Same DuckDB Code?](https://ingestthis.com/posts/2026/2026-08-24-seven-rest-catalogs-one-duckdb-script): Can seven Iceberg REST catalogs run the same DuckDB script? What the protocol makes portable, what still differs, and a test matrix you can rerun.
- [Stop Flattening Your JSON: How Iceberg Variant Changes Semi-Structured Analytics](https://ingestthis.com/posts/2026/2026-08-24-stop-flattening-your-json-iceberg-variant): Iceberg Variant stores JSON as navigable binary with shredding for columnar filters. Why flattening wide tables is no longer the only performance path.
- [What Actually Happens When Two Engines Write the Same Iceberg Table at Once?](https://ingestthis.com/posts/2026/2026-08-24-two-engines-one-iceberg-table): What happens when two engines write the same Iceberg table at once: snapshot isolation, optimistic commits, conflict detection, and when retries fail.
- [Variant Shredding Explained: How Iceberg Gets Columnar Performance From Messy JSON](https://ingestthis.com/posts/2026/2026-08-24-variant-shredding-explained): Variant shredding turns messy JSON into Parquet columns with statistics. How the layout works, how readers reassemble values, and why some queries prune.
- [Who Actually Owns an Iceberg Table? Managed, External, and the New Vocabulary of Lakehouse Control](https://ingestthis.com/posts/2026/2026-08-24-who-owns-an-iceberg-table): Managed and external Iceberg tables mean different things on every platform. Five ownership dimensions and a translation method for vendor vocabulary.
- [Mastering Apache Iceberg v3 Deletion Vectors for High-Throughput Streaming Ingest](https://ingestthis.com/posts/2026/2026-08-19-apache-iceberg-v3-deletion-vectors-streaming-ingest): Apache Iceberg v3 deletion vectors for high-throughput streaming ingest: how bitmaps and Puffin files fix CDC write amplification and read decay.
- [The Decoupled Data Lakehouse: Multi-Engine Freedom with Open REST Catalogs](https://ingestthis.com/posts/2026/2026-08-19-decoupled-data-lakehouse-multi-engine-rest-catalogs): The decoupled data lakehouse: multi-engine freedom with open REST catalogs, credential vending, and an estate that outlives its tools.
- [The Five Layers of an Agentic Lakehouse](https://ingestthis.com/posts/2026/2026-08-19-five-layers-agentic-lakehouse): The five layers of an agentic lakehouse: Storage, Catalog, Semantic, Gateway, and Agent Surface, and how one question travels through all of them.
- [Goal-Directed Data Quality Agents: Anomaly Quarantine on Apache Iceberg](https://ingestthis.com/posts/2026/2026-08-19-goal-directed-data-quality-agents-iceberg): Goal-directed data quality agents that watch Apache Iceberg tables, detect anomalies, and quarantine suspect data safely with snapshot isolation and branches.
- [Managing the TCO of Agentic Analytics: Token Budgets, Query Throttles, and the Economics of Autonomy](https://ingestthis.com/posts/2026/2026-08-19-managing-tco-agentic-analytics): Managing the total cost of agentic analytics: token budgets, query throttles, unit economics, and the FinOps discipline that keeps AI spend under control.
- [Metric Contracts in 2026: Standardizing Business Logic Across Multi-Agent Frameworks](https://ingestthis.com/posts/2026/2026-08-19-metric-contracts-2026): Metric contracts in 2026: versioned, testable definitions of business logic that let multi-agent frameworks compute revenue identically, with OSI interchange.
- [Policy-Aware Lakehouse Telemetry: Building Auditable AI Records on Apache Iceberg for the EU AI Act Era](https://ingestthis.com/posts/2026/2026-08-19-policy-aware-lakehouse-telemetry-eu-ai-act): Policy-aware lakehouse telemetry for the EU AI Act era: build auditable, immutable, governed AI decision records on Apache Iceberg.
- [Query Routing at Machine Scale: Multi-Engine Workload Distribution for the Agentic Lakehouse](https://ingestthis.com/posts/2026/2026-08-19-query-routing-machine-scale): Query routing at machine scale for the agentic lakehouse: engine selection, acceleration substitution, admission control, and placement across multi-engine estates.
- [Securing the Agentic Lakehouse Gateway: A Threat Model for Prompt Injection, Exfiltration, and the Firewall That Reads Sentences](https://ingestthis.com/posts/2026/2026-08-19-securing-agentic-lakehouse-gateway): A threat model for the agentic lakehouse gateway covering prompt injection, exfiltration, and the firewall that reads sentences.
- [Semantic Layer Federation: One Meaning for Data That Lives Everywhere](https://ingestthis.com/posts/2026/2026-08-19-semantic-layer-federation-multi-cloud): Build a federated semantic layer across multi-cloud data so one set of governed metric definitions serves BI tools, dashboards, and AI agents identically.
- [Building Lightweight Serverless Ingestion to Apache Iceberg with PyIceberg and DuckDB](https://ingestthis.com/posts/2026/2026-08-19-serverless-iceberg-ingestion-pyiceberg-duckdb): Build lightweight serverless ingestion to Apache Iceberg with PyIceberg and DuckDB, running small feeds in functions that bill for seconds.
- [Building Stateless AI Tool Gateways with FastMCP, the 2026 MCP Spec, and Kubernetes](https://ingestthis.com/posts/2026/2026-08-19-stateless-mcp-tool-gateways-fastmcp): How to build stateless AI tool gateways with FastMCP, the 2026 MCP specification, and Kubernetes, and why statelessness finally makes MCP scale.
- [Write-Audit-Publish with Apache Iceberg Branches: CI/CD for Your Data](https://ingestthis.com/posts/2026/2026-08-19-write-audit-publish-iceberg-branches): Write-Audit-Publish with Apache Iceberg branches brings CI/CD to your data: stage, audit, then fast-forward to main so consumers never see unvalidated bytes.
- [Zero-Copy Warehouse Modernization: Moving to Apache Iceberg Without Downtime](https://ingestthis.com/posts/2026/2026-08-19-zero-copy-warehouse-modernization-iceberg): A practical guide to modernizing a data warehouse to Apache Iceberg without downtime, using federation first, then redirecting new data, then materializing what earns a migration.
- [The Plan and the Worker: Two Open Specifications for Agent Harnesses](https://ingestthis.com/posts/2026/2026-08-10-agentic-graph-open-agent-profile-two-open-specs-agent-harnesses): Two open specifications, the Agentic Graph Specification and the Open Agent Profile, turn agent plans and agent identity into portable, reviewable files.
- [Apache Arrow Flight and ADBC, and Why Database Connectivity Finally Went Columnar](https://ingestthis.com/posts/2026/2026-08-06-arrow-flight-adbc-explained): Article about Apache Arrow Flight and ADBC, and Why Database Connectivity Finally Went Columnar
- [Budgeting for Agentic Analytics When Every Question Costs Something Different](https://ingestthis.com/posts/2026/2026-08-04-agentic-analytics-tco-token-budgets): Budgeting for agentic analytics when every question costs something different: token economics, query economics, instrumentation, and the cost controls that actually return.
- [The Five Layers of an Agentic Lakehouse and Where the MCP Server Sits](https://ingestthis.com/posts/2026/2026-08-04-agentic-lakehouse-mcp-architecture): The five layers of an agentic lakehouse and where the MCP server sits: storage, catalog, semantic layer, MCP gateway, and agent surface, plus identity, session isolation, and budgets.
- [Autonomous Table Optimization When Your Query Workload Stops Being Predictable](https://ingestthis.com/posts/2026/2026-08-04-autonomous-table-optimization): Autonomous table optimization when query workloads stop being predictable: observing file layout and query patterns, scoring compaction work, adaptive sort order, and cost discipline.
- [Building Apache Iceberg Lakehouses That Run Without an Internet Connection](https://ingestthis.com/posts/2026/2026-08-04-building-air-gapped-iceberg-lakehouse): How to build an Apache Iceberg lakehouse that runs fully offline: storage, catalog, compute, cross-zone transfer, compliance, and the failure modes that bite.
- [Wiring Analytical Queries to Transactional APIs in Closed-Loop Decision Agents](https://ingestthis.com/posts/2026/2026-08-04-closed-loop-decision-agents): Wiring analytical queries to transactional APIs in closed-loop decision agents: conditional writes, sagas with compensations, decision records, and blast radius controls.
- [Surviving Optimistic Commit Collisions When Hundreds of Agents Write to Iceberg](https://ingestthis.com/posts/2026/2026-08-04-high-concurrency-agent-writes-iceberg): Surviving optimistic commit collisions when hundreds of agents write to Iceberg: which conflicts are real, commit buffers, partitioning, and the patterns that prevent commit storms.
- [How Iceberg V3 Deletion Vectors Fixed Merge-on-Read for Streaming Tables](https://ingestthis.com/posts/2026/2026-08-04-iceberg-v3-deletion-vectors-merge-on-read): How Iceberg V3 deletion vectors replaced accumulating positional delete files and made merge-on-read viable for streaming and CDC tables.
- [How Iceberg V3 Variant Shredding Changed Semi-Structured Data on S3 Tables](https://ingestthis.com/posts/2026/2026-08-04-iceberg-v3-variant-shredding-s3-tables): How Iceberg V3's Variant type and Parquet shredding turn JSON columns into prunable typed columns, with real benchmark tradeoffs and a migration path.
- [Reading the Apache Iceberg V4 Proposals Before They Land](https://ingestthis.com/posts/2026/2026-08-04-iceberg-v4-roadmap): A field guide to the Apache Iceberg V4 proposals: adaptive metadata trees, single-file commits, typed statistics, column families, and what is safe to build on today.
- [Building an Honest TCO Model for Open Lakehouses and Proprietary Warehouses](https://ingestthis.com/posts/2026/2026-08-04-lakehouse-vs-warehouse-tco): An honest TCO framework for open lakehouses versus proprietary warehouses: five cost categories, measured numbers, sensitivity analysis, and where each side still wins.
- [Why Agentic AI Needs a Governed Semantic Layer Behind the Model Context Protocol](https://ingestthis.com/posts/2026/2026-08-04-mcp-governed-semantic-layer): Why agentic AI needs a governed semantic layer behind the Model Context Protocol: metric consistency, access control, Apache Ossie for portable definitions, and Apache Polaris for enforcement.
- [Moving From Supply Chain Dashboards to Decision Loops With the Model Context Protocol](https://ingestthis.com/posts/2026/2026-08-04-mcp-supply-chain-decision-loops): Moving from supply chain dashboards to decision loops with MCP: sense, decide, act, and verify, with typed action tools, idempotency keys, and graduated human approval.
- [Metric Contracts as the Interface AI Agents Actually Need](https://ingestthis.com/posts/2026/2026-08-04-metric-contracts-for-ai-agents): Metric contracts as the interface AI agents need: calculation, inclusion rules, grain, temporal semantics, ownership, semantic versioning, and testing metrics in CI.
- [Cross-Cloud Credential Vending in Apache Polaris and the End of Permanent Storage Keys](https://ingestthis.com/posts/2026/2026-08-04-polaris-cross-cloud-credential-vending): How Apache Polaris vends short-lived, prefix-scoped storage credentials across AWS, Azure, and GCP, and how to retire permanent storage keys for good.
- [Designing Policy-Aware Telemetry Tables for AI Systems in Apache Iceberg](https://ingestthis.com/posts/2026/2026-08-04-policy-aware-ai-telemetry-iceberg): Designing policy-aware AI telemetry tables in Apache Iceberg: what to log, tamper evidence, retention against conflicting deletion requirements, and tracing agent decisions.
- [Defending the Lakehouse Gateway Against Prompt Injection and Data Exfiltration](https://ingestthis.com/posts/2026/2026-08-04-preventing-prompt-injection-lakehouse-gateway): Defending the lakehouse gateway against prompt injection and data exfiltration: per-user identity, no-SQL tool surfaces, volume bounds, and detection in query behavior.
- [How the Iceberg REST Catalog Turned Into the Lakehouse Control Plane](https://ingestthis.com/posts/2026/2026-08-04-rest-catalog-v2-multi-table-commits): How the Iceberg REST catalog became the lakehouse control plane: multi-table atomic commits, credential vending, capability negotiation, and what still breaks.
- [A Migration Playbook for Moving Legacy Warehouses onto Apache Iceberg](https://ingestthis.com/posts/2026/2026-08-04-warehouse-to-iceberg-migration-playbook): A dependency-first playbook for migrating legacy warehouses onto Apache Iceberg: snapshot vs migrate vs add_files, four-level parity validation, and federation-based cutover.
- [What Zero-Copy Data Sharing Actually Does Between Salesforce, Snowflake, and Databricks](https://ingestthis.com/posts/2026/2026-08-04-zero-copy-data-sharing-iceberg): What zero-copy data sharing actually does across Salesforce, Snowflake, and Databricks: query federation, file federation, catalog federation, and when copying still wins.
- [Apache Polaris 1.7.0 and the Quiet Work of Making a Catalog Trustworthy](https://ingestthis.com/posts/2026/2026-08-02-apache-polaris-1-7-0): Apache Polaris 1.7.0 deep dive: idempotent writes, semantic models, stricter credential vending, orphan cleanup, and what the upgrade asks of you.
- [Designing Batch Pipelines That Write Well Into Apache Iceberg](https://ingestthis.com/posts/2026/2026-08-02-batch-pipelines-into-apache-iceberg): How to design batch pipelines that write well into Apache Iceberg: commit strategy, partitioning, sort order, write-audit-publish, and maintenance done right.
- [Apache Iceberg Support Across the Major Hyperscalers](https://ingestthis.com/posts/2026/2026-08-02-iceberg-across-the-hyperscalers): How AWS, Google Cloud, and Microsoft Azure actually support Apache Iceberg: storage, catalogs, maintenance, governance, and interoperability, layer by layer.
- [Guardrails for Analytics Agents That Do More Than Answer Questions](https://ingestthis.com/posts/2026/2026-07-28-agent-guardrails-human-in-the-loop): The risk isn't agents going rogue, it's agents acting correctly on bad input at machine speed. Here's how to classify actions by consequence, gate capability, and design approval steps people actually use.
- [Building Agent Telemetry Tables in Iceberg That Survive an Audit](https://ingestthis.com/posts/2026/2026-07-28-agent-telemetry-iceberg-audit): A practical guide to building agent decision traces in Apache Iceberg that support audit reconstruction, governance review, and cost attribution across sessions.
- [What Agentic Analytics Actually Costs, and How to Keep It Bounded](https://ingestthis.com/posts/2026/2026-07-28-agentic-analytics-tco): Agent analytics generates two cost streams that scale on different variables. Here's the arithmetic, the levers that actually move the number, and how to build attribution before you need it.
- [Running an Apache Iceberg Lakehouse With No Internet Connection](https://ingestthis.com/posts/2026/2026-07-28-air-gapped-iceberg-lakehouse): A practical guide to deploying an Iceberg lakehouse in air-gapped environments: component choices, artifact pipelines, identity without a cloud, and the operational realities that surprise teams.
- [When the Query Optimizer Starts Managing Its Own Materializations](https://ingestthis.com/posts/2026/2026-07-28-autonomous-reflections-agentic-lakehouse): Autonomous materialized view management replaces quarterly review meetings with workload-driven scoring, and it's essential when AI agents generate unpredictable query patterns.
- [Why AI Agents Fail on Raw Data, and What to Give Them Instead](https://ingestthis.com/posts/2026/2026-07-28-data-products-for-ai-agents): Agents fail on raw lake data because business rules live in people's heads. Data products with semantic contracts fix this at the source.
- [Why Iceberg V4 Wants to Retire Equality Deletes, and What Streaming Teams Should Do About It](https://ingestthis.com/posts/2026/2026-07-28-equality-deletes-iceberg-v4): Equality deletes made streaming upserts into Iceberg practical at the cost of read performance. V4 proposes retiring them in favor of deletion vectors with an async conversion path.
- [The Five Layers Between Your Lakehouse and a Trustworthy Agent](https://ingestthis.com/posts/2026/2026-07-28-five-layer-agentic-lakehouse): Agent reliability is a property of the stack the model sits on. Five layers with distinct owners and failure modes turn the agent is unreliable into a specific diagnosis.
- [Apache Fluss and Kafka Solve Different Problems in an Iceberg Pipeline](https://ingestthis.com/posts/2026/2026-07-28-fluss-kafka-iceberg-streaming): Fluss puts a columnar, indexed hot tier between Kafka and Iceberg. Here's what it changes structurally, what Kafka still does better, and how to benchmark the comparison yourself.
- [Serving Sub-Second Queries Over an Iceberg Lakehouse With a Hot Tier](https://ingestthis.com/posts/2026/2026-07-28-hot-tier-iceberg-serving): A lakehouse cannot serve sub-second queries over seconds-old data. A hot tier in front solves it, with consequences for consistency, governance, and operational surface.
- [Surviving Commit Conflicts When Dozens of Writers Hit the Same Iceberg Table](https://ingestthis.com/posts/2026/2026-07-28-iceberg-concurrent-commits-agents): Commit conflicts multiply with writer count, and AI agents introduce unpredictable write patterns. Here's how to diagnose, tune, and architect around Iceberg's optimistic concurrency.
- [The Jackson 3 Problem in Apache Iceberg, and What It Means for Your Code](https://ingestthis.com/posts/2026/2026-07-28-iceberg-jackson-3-migration): Jackson 3 changes everything: package names, unchecked exceptions, flipped defaults. Here's what breaks, why the engines are fine and your service isn't, and how to migrate safely.
- [Wiring an AI Agent to Apache Polaris with the Model Context Protocol](https://ingestthis.com/posts/2026/2026-07-28-mcp-apache-polaris): The catalog is the right attachment point for AI agents working against a lakehouse. Here's how to wire the official Polaris MCP Server and add the read path it deliberately leaves out.
- [Governing Iceberg Tables Across Regions Without Three Sets of Permissions](https://ingestthis.com/posts/2026/2026-07-28-multi-region-catalog-federation): Catalog federation gives you one authorization model and one audit point across regions. Here's what it solves, what it doesn't, and how to build a topology you can actually govern.
- [Federating Oracle With an Open Lakehouse Instead of Migrating It](https://ingestthis.com/posts/2026/2026-07-28-oracle-lakehouse-federation): Federate first so analytics work now, migrate what benefits from migrating, and leave the rest where it is indefinitely. Here's how pushdown and view layers make it work.
- [The Parquet Versioning Problem, and Why Iceberg Cares About It](https://ingestthis.com/posts/2026/2026-07-28-parquet-versioning-iceberg-v4): Parquet files have a version field that doesn't reliably signal feature requirements. A new versioning discipline is coming, borrowing from Iceberg's format version model.
- [Building Iceberg Pipelines in Python Without Standing Up Spark](https://ingestthis.com/posts/2026/2026-07-28-python-native-iceberg-pipelines): A large share of production transformations fit comfortably on one machine. PyIceberg, DuckDB, and branch isolation give you a production path that debugs in an IDE.
- [Governing What Agents Cost You](https://ingestthis.com/posts/2026/2026-07-agent-cost-governance): Agents break the four assumptions analytics platforms were built on. A practical guide to identity, budgets, semantic layers, caching, and instrumentation for agent workloads.
- [Every AI Model Family That Matters in Mid-2026](https://ingestthis.com/posts/2026/2026-07-ai-model-families-2026): A full survey of the AI model landscape in mid-2026: frontier families, open-weight labs, local inference, specialists, and how to build a routing layer instead of a dependency.
- [Freshness Is a Contract, Not a Note on a Dashboard](https://ingestthis.com/posts/2026/2026-07-freshness-as-a-contract): Data freshness needs to become an engineering contract with a measurable value, an owner, and consequences. How to decompose lag, make freshness queryable, and keep agents honest.
- [The Apache Iceberg Market in the Middle of 2026](https://ingestthis.com/posts/2026/2026-07-iceberg-market-2026): A survey of the Apache Iceberg market in July 2026: the state of the specification, platform support, the acquisition wave, the catalog contest, and how to evaluate real Iceberg support.
- [The Whole Lakehouse Fits on Your Laptop Now](https://ingestthis.com/posts/2026/2026-07-laptop-scale-lakehouse): Consumer hardware, columnar formats, single-node engines, and the Iceberg REST catalog crossed a threshold: a large share of cluster work now runs locally against the same governed tables.
- [Iceberg's Next Version Depends on Decisions Being Made in Parquet](https://ingestthis.com/posts/2026/2026-07-parquet-constrains-iceberg): The Iceberg and Parquet specifications are co-evolving. Several of the most consequential Iceberg v4 proposals are waiting on Parquet work, and practitioners should read both dev lists.
- [Building a Lakehouse That Stays Inside the Border](https://ingestthis.com/posts/2026/2026-07-sovereign-lakehouse): Residency is a storage location. Sovereignty is who can compel access, who operates the systems, and whether you can leave. A practical guide to sovereign lakehouse architecture in 2026.
- [Table Maintenance Stopped Being a Product](https://ingestthis.com/posts/2026/2026-07-table-maintenance-economics): Iceberg table maintenance commoditized when every platform started shipping it. What the six operations are, what they cost, and the observability you should keep even when the work is managed.
- [The Model Class Enterprise Data Teams Are Not Using Yet](https://ingestthis.com/posts/2026/2026-07-tabular-foundation-models): Tabular foundation models perform supervised learning in a single forward pass with no training run. What they are, where they win, and what they change in the data architecture.
- [Three Vendors Are Rebuilding the Path From Transaction to Agent](https://ingestthis.com/posts/2026/2026-07-transaction-to-agent-race): Databricks, Snowflake, and SAP are closing the gap between operational databases and analytical platforms through acquisition, betting on different layers of the same five-part architecture.
- [What Zero-Copy Actually Costs](https://ingestthis.com/posts/2026/2026-07-zero-copy-actually-costs): Six architectures share the phrase zero-copy, and they have different costs, failure modes, and governance stories. A walk through federation, virtualization, sharing protocols, and materialization.
- [The Filters We Build: Scams, Slop, and the Search for Signal, From Radio Ads to AI](https://ingestthis.com/posts/2026/2026-07-24-cognitive-filters-media-evolution): How every media era from radio to AI forces humanity to build new cognitive filters -- the shield against scams and the sieve for finding signal in an ocean of noise.
- [The Breakdown of the Open Lakehouse in 2026: Iceberg, Arrow, Polaris, Parquet, and Ossie, and How to Actually Build One](https://ingestthis.com/posts/2026/2026-07-24-open-lakehouse-breakdown-2026): A detailed breakdown of the five Apache projects powering the open lakehouse in 2026 -- Parquet, Iceberg, Polaris, Arrow, and Ossie -- and how to build one yourself or use a managed platform.
- [Understanding the RAMpocalypse: An Economic Autopsy of the Great Memory Shortage and What Comes Next](https://ingestthis.com/posts/2026/2026-07-24-rampocalypse-economics): An economic autopsy of the 2026 memory shortage -- why DRAM prices surged 90% in a quarter, and what supply, demand, and market structure mean for the future of computing hardware.
- [The Own Goal: What the Hugging Face Breach Reveals About Why Open Models Are a Security Necessity](https://ingestthis.com/posts/2026/2026-07-24-value-of-open-models-security): How the 2026 Hugging Face breach proved that open models are a security necessity -- closed models failed to defend, an open Chinese model succeeded, and the lessons for AI policy.
- [Active Analytics Loops for AI Action Agents](https://ingestthis.com/posts/2026/2026-07-13-active-analytics-loop-goal-directed-action-agents): An in-depth exploration of active analytics loops for ai action agents
- [Apache Polaris and Multi-Engine Iceberg Catalogs](https://ingestthis.com/posts/2026/2026-07-13-apache-polaris-asf-top-level-multi-engine-catalog-interoperability): An in-depth exploration of apache polaris and multi-engine iceberg catalogs
- [Automated Materialized Views in Dremio](https://ingestthis.com/posts/2026/2026-07-13-automated-materialized-views-autonomous-table-optimization-dremio-agentic-lakehouse): An in-depth exploration of automated materialized views in dremio
- [The Chase-Catch Gap in Enterprise AI Agents](https://ingestthis.com/posts/2026/2026-07-13-chase-catch-gap-agentic-ai-enterprise-scale-2026): An in-depth exploration of the chase-catch gap in enterprise ai agents
- [Data Lakehouse Open Standards for AI Agents](https://ingestthis.com/posts/2026/2026-07-13-data-lakehouse-open-standards-agentic-era): An in-depth exploration of data lakehouse open standards for ai agents
- [Dremio Lakehouse AI Report: Agentic Lessons](https://ingestthis.com/posts/2026/2026-07-13-dremio-state-data-lakehouse-ai-report-agentic-lakehouse): An in-depth exploration of dremio lakehouse ai report: agentic lessons
- [Five-Layer Agentic Lakehouse Architecture](https://ingestthis.com/posts/2026/2026-07-13-five-layer-agentic-lakehouse-architecture-enterprise-pattern): An in-depth exploration of five-layer agentic lakehouse architecture
- [GSA MCP Servers for Open Data AI Agents](https://ingestthis.com/posts/2026/2026-07-13-gsa-model-context-protocol-server-hackathon-open-data-ai-agents): An in-depth exploration of gsa mcp servers for open data ai agents
- [Hybrid Lakehouse Design for Regulated Markets](https://ingestthis.com/posts/2026/2026-07-13-hybrid-lakehouse-regulated-markets-on-prem-public-metadata): An in-depth exploration of hybrid lakehouse design for regulated markets
- [Iceberg v3 Deletion Vectors and Faster DML](https://ingestthis.com/posts/2026/2026-07-13-iceberg-v3-deletion-vectors-merge-on-read-dml-performance): An in-depth exploration of iceberg v3 deletion vectors and faster dml
- [Iceberg Variant Type for AI JSON Data](https://ingestthis.com/posts/2026/2026-07-13-iceberg-variant-type-semi-structured-ai-datasets): An in-depth exploration of iceberg variant type for ai json data
- [Iceberg Concurrency for AI Agent Writes](https://ingestthis.com/posts/2026/2026-07-13-llm-agency-concurrency-isolation-high-frequency-agent-writes-iceberg): An in-depth exploration of iceberg concurrency for ai agent writes
- [MCP Stateless Gateways for Data Agents](https://ingestthis.com/posts/2026/2026-07-13-mcp-stateless-specifications-scalable-data-gateways-ai-agents): An in-depth exploration of mcp stateless gateways for data agents
- [Policy Layer for Autonomous AI Data Agents](https://ingestthis.com/posts/2026/2026-07-13-policy-layer-security-policy-enforcement-autonomous-multi-agent-systems): An in-depth exploration of policy layer for autonomous ai data agents
- [REST Catalog Credential Vending for Iceberg](https://ingestthis.com/posts/2026/2026-07-13-rest-catalog-credential-vending-zero-copy-security): An in-depth exploration of rest catalog credential vending for iceberg
- [REST Catalog V2: Fixing Iceberg Protocol Debt](https://ingestthis.com/posts/2026/2026-07-13-rest-catalog-v2-standard-iceberg-scaling-protocol-debt): An in-depth exploration of rest catalog v2: fixing iceberg protocol debt
- [Semantic View Autopilot for AI Governance](https://ingestthis.com/posts/2026/2026-07-13-semantic-view-autopilot-dynamic-metric-definition-governance): An in-depth exploration of semantic view autopilot for ai governance
- [Zero-Copy Mirroring to Open Iceberg Tables](https://ingestthis.com/posts/2026/2026-07-13-zero-copy-mirroring-migration-proprietary-warehouses-open-iceberg): An in-depth exploration of zero-copy mirroring to open iceberg tables
- [High-Performance Columnar Transfers: Combining Apache Arrow Flight and Iceberg REST Catalogs](https://ingestthis.com/posts/2026/2026-07-06-arrow-flight-iceberg-rest-catalog-columnar-transfers): Article about High-Performance Columnar Transfers: Combining Apache Arrow Flight and Iceberg REST Catalogs
- [Block vs. Object Storage: A Deep Dive Into the Foundation of Modern Data, and How the Lakehouse Made the Slow Option Fast](https://ingestthis.com/posts/2026/2026-07-06-block-vs-object-storage-deep-dive): Article about Block vs. Object Storage: A Deep Dive Into the Foundation of Modern Data, and How the Lakehouse Made the Slow Option Fast
- [The Buyer's Scorecard for Agentic Analytics: Evaluating Tooling in the Enterprise AI Era](https://ingestthis.com/posts/2026/2026-07-06-buyers-scorecard-agentic-analytics-enterprise-ai-era): Article about The Buyer's Scorecard for Agentic Analytics: Evaluating Tooling in the Enterprise AI Era
- [Building Closed-Loop Decision Agents: Moving from Passive BI Dashboards to Active Goal-Directed Workflows](https://ingestthis.com/posts/2026/2026-07-06-closed-loop-decision-agents-passive-bi-active-workflows): Article about Building Closed-Loop Decision Agents: Moving from Passive BI Dashboards to Active Goal-Directed Workflows
- [Conversational AI on Managed Iceberg: Exposing Amazon S3 Tables through MCP](https://ingestthis.com/posts/2026/2026-07-06-conversational-ai-managed-iceberg-s3-tables-mcp-server): Article about Conversational AI on Managed Iceberg: Exposing Amazon S3 Tables through MCP
- [Decoupled Catalogs vs. Managed Tables: Architectural Freedom in the Age of Table Format Convergence](https://ingestthis.com/posts/2026/2026-07-06-decoupled-catalogs-vs-managed-tables-table-format-convergence): Article about Decoupled Catalogs vs. Managed Tables: Architectural Freedom in the Age of Table Format Convergence
- [Designing Your Own AI Harness: A Deep Dive Into the Architecture of Agent Loops, Tools, Context, and Control](https://ingestthis.com/posts/2026/2026-07-06-designing-your-own-ai-harness): A deep dive into custom AI harness architecture: model layers, tool design, context management, permissions, control budgets, persistence, orchestration, and evaluation systems.
- [Deterministic Data Engineering With AI Harnesses: Using Claude Code, Codex, Antigravity, and OpenCode for Data Work You Can Actually Trust](https://ingestthis.com/posts/2026/2026-07-06-deterministic-data-engineering-with-ai-harnesses): How to use AI agent harnesses for data engineering without losing determinism, reproducibility, and trust in your data pipelines and analytics.
- [Preparing Your Data Lakehouse for the EU AI Act: Auditable Lineage and Data Provenance](https://ingestthis.com/posts/2026/2026-07-06-eu-ai-act-data-lakehouse-lineage-provenance-compliance): Article about Preparing Your Data Lakehouse for the EU AI Act: Auditable Lineage and Data Provenance
- [Federation and the Lakehouse: Two Roads to Unified Data Access, and How to Know Which One to Take](https://ingestthis.com/posts/2026/2026-07-06-federation-and-lakehouse): Article about Federation and the Lakehouse: Two Roads to Unified Data Access, and How to Know Which One to Take
- [A Deep Dive Into File Compression: How Data Gets Smaller, Why Codecs Differ, and What to Actually Use in the Lakehouse](https://ingestthis.com/posts/2026/2026-07-06-file-compression-deep-dive): Article about A Deep Dive Into File Compression: How Data Gets Smaller, Why Codecs Differ, and What to Actually Use in the Lakehouse
- [The File Format Renaissance: Parquet, Lance, Vortex, Nimble, BtrBlocks, and the New Physics of Columnar Storage](https://ingestthis.com/posts/2026/2026-07-06-file-formats-renaissance): Article about The File Format Renaissance: Parquet, Lance, Vortex, Nimble, BtrBlocks, and the New Physics of Columnar Storage
- [Enforcing Fine-Grained Security at Machine Speed: Dynamic Access Control for High-Frequency AI Agents](https://ingestthis.com/posts/2026/2026-07-06-fine-grained-security-machine-speed-ai-agents): Article about Enforcing Fine-Grained Security at Machine Speed: Dynamic Access Control for High-Frequency AI Agents
- [Implementing Positional Deletes in Iceberg v3: Streamlining Merge-on-Read for Fast-Inbound Event Lakes](https://ingestthis.com/posts/2026/2026-07-06-iceberg-v3-positional-deletes-deletion-vectors-event-lakes): Article about Implementing Positional Deletes in Iceberg v3: Streamlining Merge-on-Read for Fast-Inbound Event Lakes
- [Mapping the Variant Type in Iceberg v3: Standardizing Semi-Structured AI JSON Payloads](https://ingestthis.com/posts/2026/2026-07-06-iceberg-v3-variant-type-ai-json-payloads): Article about Mapping the Variant Type in Iceberg v3: Standardizing Semi-Structured AI JSON Payloads
- [The State of Apache Iceberg v4 in July 2026: What the Dev List Tells Us About the Format's Next Chapter](https://ingestthis.com/posts/2026/2026-07-06-iceberg-v4-state-july-2026): What the Iceberg v4 dev list tells us about adaptive metadata trees, single-file commits, column updates, and the format's next chapter in mid-2026.
- [Designing Idempotent Pipelines in the Agentic Lakehouse: Eliminating Double-Write Anomalies](https://ingestthis.com/posts/2026/2026-07-06-idempotent-pipelines-agentic-lakehouse-double-write-anomalies): Article about Designing Idempotent Pipelines in the Agentic Lakehouse: Eliminating Double-Write Anomalies
- [File Encryption for the Lakehouse: The Terminology, the Machinery, and the Hard Problem of Interoperable Encrypted Tables](https://ingestthis.com/posts/2026/2026-07-06-lakehouse-encryption-deep-dive): Article about File Encryption for the Lakehouse: The Terminology, the Machinery, and the Hard Problem of Interoperable Encrypted Tables
- [The 2026-07-28 Model Context Protocol Release Candidate: What the Stateless Spec Means for Data Platforms](https://ingestthis.com/posts/2026/2026-07-06-mcp-2026-07-28-stateless-spec-data-platforms): Article about The 2026-07-28 Model Context Protocol Release Candidate: What the Stateless Spec Means for Data Platforms
- [The Metric Contract Mandate: Standardizing Semantic Layers Before AI Agent Access](https://ingestthis.com/posts/2026/2026-07-06-metric-contract-mandate-semantic-layers-ai-agent-access): Article about The Metric Contract Mandate: Standardizing Semantic Layers Before AI Agent Access
- [Multi-Engine Catalog Federation with Apache Polaris: Syncing Google Cloud, AWS, and Azure Metadata](https://ingestthis.com/posts/2026/2026-07-06-multi-engine-catalog-federation-apache-polaris-multicloud): Article about Multi-Engine Catalog Federation with Apache Polaris: Syncing Google Cloud, AWS, and Azure Metadata
- [Open Source Foundations, Explained: What Apache, Linux, Eclipse, and Their Peers Actually Do, and Why Governance Differences Matter](https://ingestthis.com/posts/2026/2026-07-06-open-source-foundations-explained): Article about Open Source Foundations, Explained: What Apache, Linux, Eclipse, and Their Peers Actually Do, and Why Governance Differences Matter
- [Operational vs. Analytical Systems: Why the Oldest Divide in Data Exists, What Physics Enforces It, and the Honest Truth About Hybrid Systems](https://ingestthis.com/posts/2026/2026-07-06-operational-vs-analytical-systems): Article about Operational vs. Analytical Systems: Why the Oldest Divide in Data Exists, What Physics Enforces It, and the Honest Truth About Hybrid Systems
- [Personal Context vs. Shared Context: A Deep Dive Into How Humans and Organizations Should Feed Their AI Agents](https://ingestthis.com/posts/2026/2026-07-06-personal-vs-shared-context): Article about Personal Context vs. Shared Context: A Deep Dive Into How Humans and Organizations Should Feed Their AI Agents
- [Designing Private, Air-Gapped Data Lakehouses: Scaling Iceberg in Highly Secure, On-Premises Clouds](https://ingestthis.com/posts/2026/2026-07-06-private-air-gapped-data-lakehouses-iceberg-secure-clouds): Article about Designing Private, Air-Gapped Data Lakehouses: Scaling Iceberg in Highly Secure, On-Premises Clouds
- [Migrating Proprietary Warehouses to Open Lakehouses: The 2026 Playbook for Zero-Copy Metadata Translation](https://ingestthis.com/posts/2026/2026-07-06-proprietary-warehouse-open-lakehouse-zero-copy-metadata-translation): Article about Migrating Proprietary Warehouses to Open Lakehouses: The 2026 Playbook for Zero-Copy Metadata Translation
- [The State of Agentic AI Standards in 2026: MCP, A2A, WebMCP, OSI, and the Protocol Stack Taking Shape](https://ingestthis.com/posts/2026/2026-07-06-state-of-agentic-ai-standards-2026): The agentic AI protocol stack is solidifying in 2026 — MCP for tools, A2A for agents, WebMCP for the web, OSI for semantics, payments, identity, and security.
- [The State of Apache Arrow in 2026: Ten Years In, the Invisible Standard Is Everywhere](https://ingestthis.com/posts/2026/2026-07-06-state-of-apache-arrow-2026): Apache Arrow at 10 — ADBC, Flight SQL, nanoarrow, the AI reinterpretation, and how an in-memory standard eliminated the copy tax across the data stack.
- [The State of Apache Parquet in 2026: The Quiet Format Enters Its Loudest Decade](https://ingestthis.com/posts/2026/2026-07-06-state-of-apache-parquet-2026): Apache Parquet in 2026 — variant types, geospatial, ALP encoding, footer redesign, the versioning debate, and how the decade-old format is renovating for AI workloads.
- [The State of Apache Polaris in July 2026: From Incubating Catalog to the Governance Layer of the Open Lakehouse](https://ingestthis.com/posts/2026/2026-07-06-state-of-apache-polaris-july-2026): Apache Polaris as a TLP — federation, credential vending, semantic layers, lineage, and how the open catalog became the governance plane of the multi-engine lakehouse.
- [The State of Streaming to Apache Iceberg in July 2026: Every Path, Its Latency, and What to Do When Seconds Are Not Fast Enough](https://ingestthis.com/posts/2026/2026-07-06-streaming-to-iceberg-july-2026): Every path for streaming data into Iceberg in 2026 — Flink, Spark, Kafka Connect, broker-native, managed pipelines — with honest latency numbers and sub-second hybrid architectures.
- [Lakehouse Table Formats in 2026: Iceberg, Delta Lake, Hudi, Paimon, and DuckLake, How They Work, Where They Stand, and Where They're Going](https://ingestthis.com/posts/2026/2026-07-06-table-formats-2026-breakdown): Article about Lakehouse Table Formats in 2026: Iceberg, Delta Lake, Hudi, Paimon, and DuckLake, How They Work, Where They Stand, and Where They're Going
- [Trustworthy Concurrency in the Agentic Lakehouse: Reconciling Academic Proofs with High-Frequency Production Writes](https://ingestthis.com/posts/2026/2026-07-06-trustworthy-concurrency-agentic-lakehouse-production-writes): Article about Trustworthy Concurrency in the Agentic Lakehouse: Reconciling Academic Proofs with High-Frequency Production Writes
- [When Gatekeepers Panic: The Encyclopédie, Open AI Models, and the Politics of Accessible Knowledge](https://ingestthis.com/posts/2026/2026-07-06-when-gatekeepers-panic): The fight over open AI models mirrors the 18th-century suppression of Diderot's Encyclopédie, revealing the same pattern of institutional fear of accessible knowledge.
- [The Who, What, and Why of Semantic Layers: The Layer That Decides Whether Your Numbers Can Be Trusted](https://ingestthis.com/posts/2026/2026-07-06-who-what-why-semantic-layers): Article about The Who, What, and Why of Semantic Layers: The Layer That Decides Whether Your Numbers Can Be Trusted
- [AI-Ready Metadata Prevents Query Failures](https://ingestthis.com/posts/2026/2026-06-22-ai-ready-metadata-lineage-quality-llm-query-failures): AI-ready metadata reduces query failures by making ownership, freshness, lineage, quality, and policy visible at execution time.
- [Autonomous Materialization for Agentic Analytics](https://ingestthis.com/posts/2026/2026-06-22-autonomous-materialization-ai-agents-table-performance): Autonomous materialization accelerates agentic analytics by letting AI agents create and manage materialized views without manual DBA intervention.
- [Composable Semantic Layers for Analytical Agents](https://ingestthis.com/posts/2026/2026-06-22-beyond-metrics-lists-composable-semantic-layers-agents): Composable semantic layers for analytical agents go beyond metrics lists to provide rich, reusable business context for AI-driven analytics.
- [Built for Agents and Managed by Agents](https://ingestthis.com/posts/2026/2026-06-22-built-for-agents-managed-by-agents-dremio-agentic-lakehouse): The Dremio Agentic Lakehouse is built for agents and managed by agents, creating an autonomous data platform for AI-driven analytics.
- [ClickHouse in the Loop for Active Agents](https://ingestthis.com/posts/2026/2026-06-22-clickhouse-loop-real-time-event-streams-active-agents): ClickHouse in the loop for active agents enables real-time event streams and low-latency analytics for agent-driven decision making.
- [The Context Layer for AI Agents](https://ingestthis.com/posts/2026/2026-06-22-context-layer-semantic-lineage-quality-ai-agents): The context layer for AI agents provides semantic lineage, quality, and governance metadata so agents understand the data they query.
- [Lakehouse as the Operating Layer for Agentic AI](https://ingestthis.com/posts/2026/2026-06-22-databricks-summit-2026-lakehouse-agentic-ai-operating-layer): The lakehouse as the operating layer for agentic AI provides governed, semantic-aware data infrastructure for autonomous analytical agents.
- [Event-Driven Table Compaction with Agents](https://ingestthis.com/posts/2026/2026-06-22-event-driven-table-compaction-agentic-coordination): Event-driven table compaction with agents automates Iceberg maintenance using agentic coordination for optimal table performance.
- [Fabric Agentic Analytics and Lakehouse Schema Design](https://ingestthis.com/posts/2026/2026-06-22-fabric-build-2026-lakehouse-schema-agentic-analytics): Fabric agentic analytics and lakehouse schema design explores how Microsoft Fabric architecture enables AI-driven analytics on the lakehouse.
- [Fine-Grained Security for AI Agents](https://ingestthis.com/posts/2026/2026-06-22-fine-grained-security-machine-speed-ai-agents): Fine-grained security for AI agents requires agent identities, token exchange, and column-level guardrails to protect sensitive data in the lakehouse.
- [Iceberg v4 Performance: Root Manifests and Calls](https://ingestthis.com/posts/2026/2026-06-22-iceberg-v4-performance-root-manifests-combined-calls): Iceberg v4 performance improvements through root manifests reduce metadata round trips and optimize object storage planning for faster queries.
- [What Is LTAP in the Lakehouse?](https://ingestthis.com/posts/2026/2026-06-22-ltap-lakehouse-transactional-analytical-processing-2026): LTAP in the lakehouse combines transactional and analytical processing for real-time Iceberg updates with analytical freshness.
- [The Model Is Not the Moat](https://ingestthis.com/posts/2026/2026-06-22-model-not-moat-semantic-lakehouse-layer-enterprise-ai): The model is not the moat — the semantic lakehouse layer and governed enterprise AI infrastructure provide the real competitive advantage.
- [PyIceberg at Scale Without Apache Spark](https://ingestthis.com/posts/2026/2026-06-22-pyiceberg-scale-high-concurrency-no-spark): PyIceberg at scale without Apache Spark demonstrates Python-native Iceberg pipelines using REST catalog concurrency for high-performance data processing.
- [The Real-Time Lakehouse with Streaming and Iceberg](https://ingestthis.com/posts/2026/2026-06-22-real-time-lakehouse-streaming-sql-cold-iceberg-storage): The real-time lakehouse with streaming SQL and Iceberg storage enables event-driven lakehouse architectures for low-latency analytics.
- [REST Catalog V2 LoadTable and Client Capability](https://ingestthis.com/posts/2026/2026-06-22-rest-catalog-v2-loadtable-protocol-design): REST Catalog V2 LoadTable and client capability protocols define open catalog contracts for interoperable Iceberg catalog access.
- [Rust vs C++ in Native Iceberg Scan Operators](https://ingestthis.com/posts/2026/2026-06-22-rust-vs-cpp-table-layer-native-scan-operators): Rust vs C++ in native Iceberg scan operators compares performance and safety tradeoffs for vectorized scans in modern data systems.
- [Server-Side Commit Deconflicting in REST Catalogs](https://ingestthis.com/posts/2026/2026-06-22-server-side-commit-deconflicting-rest-catalogs): Server-side commit deconflicting in REST catalogs resolves Iceberg commit conflicts with optimistic concurrency control for multi-engine environments.
- [Snowflake Interoperable Lakehouse Lessons](https://ingestthis.com/posts/2026/2026-06-22-snowflake-interoperable-lakehouse-production-lessons): Snowflake interoperable lakehouse lessons explore Iceberg interoperability and open table formats for multi-engine lakehouse architectures.
- [Unity AI Gateway vs Apache Polaris Control Planes](https://ingestthis.com/posts/2026/2026-06-22-unity-ai-gateway-vs-apache-polaris-governance-control-plane): Unity AI Gateway vs Apache Polaris control planes compares governance control plane architectures for modern data and AI platforms.
- [The Complete Guide to Agentic Coding Tools in 2026](https://ingestthis.com/posts/2026/2026-06-08-agentic-coding-tools): A deep dive into the four categories of agentic coding tools dominating 2026: CLI agents, desktop IDEs, 24/7 autonomous agents, and model routers. 40+ tools compared.
- [Agentic Lakehouse Concurrency and Isolation](https://ingestthis.com/posts/2026/2026-06-08-agentic-lakehouse-concurrency-isolation-contracts): How Iceberg optimistic concurrency control, partition-level isolation, and idempotency keys enable safe concurrent writes from multiple AI agents to the same lakehouse tables.
- [What AI Is and Isnt: A Laypersons Guide to How LLMs Actually Work](https://ingestthis.com/posts/2026/2026-06-08-ai-for-all-levels-1-what-ai-is-and-isnt): AI is not magic or sentient. Learn what large language models actually do, how vectors and embeddings work, and why todays AI is pattern matching not thinking. 155 chars
- [Getting Started with AI for Free: Every Tool Google Gives You at No Cost](https://ingestthis.com/posts/2026/2026-06-08-ai-for-all-levels-2-getting-started-for-free): You already have access to powerful AI tools through your Google account and no subscription needed. Here is everything Google offers for free and how to use it today. 157 chars
- [ChatGPT and Claude: Which AI Service Should You Pay For](https://ingestthis.com/posts/2026/2026-06-08-ai-for-all-levels-3-chatgpt-and-claude-deep-dive): ChatGPT and Claude are the two leading AI assistants. Here is what each offers at every price tier, how to use their desktop apps and advanced features like Dispatch and Clips, and which one fits your workflow. 162 chars
- [A Tour of Specialized AI Tools: Music, Video, Images, and More](https://ingestthis.com/posts/2026/2026-06-08-ai-for-all-levels-4-specialized-ai-tools): Beyond chatbots lie specialized AI tools for creating music, generating videos, and editing images. Here is a tour of the best tools in 2026 and how they fit into daily productivity. 157 chars
- [Going Advanced: Open Source Models, Hermes Agent, and Local AI](https://ingestthis.com/posts/2026/2026-06-08-ai-for-all-levels-5-going-advanced): Tired of subscriptions and privacy concerns? Open source models and agent frameworks let you run AI on your own hardware. Here is how to get started with Hermes Agent, OpenCode, DeepSeek, and local Llama models. 166 chars
- [Anatomy of an Agentic Lakehouse](https://ingestthis.com/posts/2026/2026-06-08-anatomy-agentic-lakehouse-four-layers): The four-layer architecture of the agentic lakehouse: object storage, Apache Iceberg table format, Apache Polaris catalog, and the semantic/agent layer. How each layer provides guarantees for AI agent access.
- [Apache Iceberg v4 Roadmap and Metadata Trees](https://ingestthis.com/posts/2026/2026-06-08-apache-iceberg-v4-roadmap-adaptive-metadata-delta-convergence): A deep technical breakdown of Apache Iceberg v4's proposed architecture: adaptive metadata trees, one-file commits, relative paths, column families, and what the Delta 5.0 convergence actually means for your data platform.
- [Lakehouse Context Layers with Atlan and Iceberg v3](https://ingestthis.com/posts/2026/2026-06-08-atlan-snowflake-iceberg-v3-context-layer): Lakehouse context layers bridge the gap between raw Iceberg tables and AI agents that need business meaning. Atlan and Snowflake Horizon each take different approaches to the same problem.
- [Real-Time Agentic Analytics with ClickHouse](https://ingestthis.com/posts/2026/2026-06-08-clickhouse-real-time-agentic-analytics-event-loops): ClickHouse has become the leading real-time analytics engine for AI agent workloads, with event-loop architectures that let agents query and act at sub-second speeds.
- [Composable Analytics Beats Metric Catalogs](https://ingestthis.com/posts/2026/2026-06-08-composable-analytics-semantic-layers-expressiveness): Metric catalogs define what terms mean. Composable analytics defines how terms combine, transform, and relate. For AI agents, composability is what turns definitions into reasoning.
- [Goal-Directed Analytics Agents on Apache Iceberg](https://ingestthis.com/posts/2026/2026-06-08-goal-directed-analytics-agents-apache-iceberg-action-loops): How goal-directed analytics agents decompose business questions into sub-tasks, execute action loops over Apache Iceberg tables, and use the lakehouse as both a data source and a state store for agent action logs.
- [Iceberg Remote Signing for Regulated Datasets](https://ingestthis.com/posts/2026/2026-06-08-iceberg-remote-signing-regulated-datasets): Iceberg REST catalog remote signing provides per-file pre-signed URL access for regulated datasets. How it differs from credential vending, audit trail capabilities, and Snowflake implementation for PII/compliance workloads.
- [Apache Iceberg v3 Deletion Vectors on Snowflake](https://ingestthis.com/posts/2026/2026-06-08-iceberg-v3-deletion-vectors-snowflake-dml): Apache Iceberg v3 deletion vectors replace positional delete files with binary bitmaps in Puffin files, delivering up to 10x faster DML on Snowflake. Deep dive into architecture, benchmarks, and migration.
- [CDC Without Complexity Using Iceberg v3 Row Lineage](https://ingestthis.com/posts/2026/2026-06-08-iceberg-v3-row-lineage-cdc): Iceberg v3 row lineage adds _row_id and _last_updated_sequence_number to every table, enabling native change data capture without Debezium or Kafka. Technical deep dive with Snowflake and Databricks examples.
- [The 2026 Guide to Iceberg View Federation](https://ingestthis.com/posts/2026/2026-06-08-iceberg-view-federation-portable-sql-2026): Iceberg views standardize SQL view definitions across engines, enabling view federation across Polaris, Nessie, and Gravitino catalogs. How Snowflake Horizon, Databricks Unity Catalog, and open source catalogs handle portable SQL.
- [Implementing MCP in the Lakehouse](https://ingestthis.com/posts/2026/2026-06-08-mcp-lakehouse-semantic-data-layer-python): How to build a Model Context Protocol (MCP) server that exposes lakehouse tables and semantic views as AI-accessible tools, with Python implementation patterns and authentication.
- [Microsoft Fabric Build 2026 Agentic Analytics Stack](https://ingestthis.com/posts/2026/2026-06-08-microsoft-fabric-build-2026-agentic-analytics-stack): Microsoft Build 2026 revealed an agentic analytics stack built on Fabric IQ, OneLake Iceberg support, and semantic models. The architecture shows how Microsoft competes with open lakehouse platforms.
- [Modern Python Tooling for Apache Iceberg](https://ingestthis.com/posts/2026/2026-06-08-python-tooling-apache-iceberg-pyiceberg-iceframe-iceberg-cli): PyIceberg, IceFrame, and the Iceberg CLI form a complete Python toolchain for Iceberg table management. Each tool targets a different workflow from metadata inspection to data engineering.
- [REST Catalog Credential Vending for Lakehouse Security](https://ingestthis.com/posts/2026/2026-06-08-rest-catalog-credential-vending-secure-lakehouse-storage): Iceberg REST catalog credential vending issues short-lived, table-scoped storage tokens for S3, ADLS, and GCS. How Polaris, Snowflake Horizon, Databricks Unity Catalog, and Dremio implement credential vending for secure lakehouse storage access.
- [SaaS Buyers Now Inspect Your Semantic Layer](https://ingestthis.com/posts/2026/2026-06-08-saas-procurement-semantic-layer-over-dashboards): Enterprise SaaS procurement in 2026 evaluates how platforms expose data to AI agents. Semantic layers have become a decision criterion alongside dashboards and APIs.
- [Securing Agent Identities in the Lakehouse](https://ingestthis.com/posts/2026/2026-06-08-securing-agent-identities-lakehouse-token-exchange): How OAuth 2.0 token exchange, OAuth 2.1 device authorization grant, credential vending, and fine-grained access control secure AI agent identities in the Iceberg lakehouse.
- [Bidirectional Iceberg Writes with Horizon Catalog](https://ingestthis.com/posts/2026/2026-06-08-snowflake-horizon-catalog-bidirectional-iceberg-writes): Snowflake Horizon Catalog enables bidirectional Iceberg writes from external engines like Spark and Trino, powered by Apache Polaris. Deep dive into REST Scan Plan API, governance, and Snowflake Summit 2026 announcements.
- [Semantic View Autopilot in Snowflake Semantic Studio](https://ingestthis.com/posts/2026/2026-06-08-snowflake-semantic-view-autopilot-business-logic): Snowflake Semantic View Autopilot automates semantic view creation from query history and BI assets. But production semantics still need human review, testing, and governance.
- [Zero-Copy Mirroring for Modern Lakehouse Migration](https://ingestthis.com/posts/2026/2026-06-08-zero-copy-mirroring-modern-lakehouse-migration): Zero-copy mirroring gives teams a safer migration path because they can expose a lakehouse surface before they duplicate every byte or rewrite every workload.
- [What AI Is and Isnt: A Laypersons Guide to How LLMs Actually Work](https://ingestthis.com/posts/2026/2026-06-01-ai-for-all-levels-1-what-ai-is-and-isnt): AI is not magic or sentient. Learn what large language models actually do, how vectors and embeddings work, and why todays AI is pattern matching not thinking.
- [Getting Started with AI for Free: Every Tool Google Gives You at No Cost](https://ingestthis.com/posts/2026/2026-06-01-ai-for-all-levels-2-getting-started-for-free): You already have access to powerful AI tools through your Google account and no subscription needed. Here is everything Google offers for free and how to use it today.
- [ChatGPT and Claude: Which AI Service Should You Pay For](https://ingestthis.com/posts/2026/2026-06-01-ai-for-all-levels-3-chatgpt-and-claude-deep-dive): ChatGPT and Claude are the two leading AI assistants. Here is what each offers at every price tier, how to use their desktop apps and advanced features like Dispatch and Clips, and which one fits your workflow.
- [A Tour of Specialized AI Tools: Music, Video, Images, and More](https://ingestthis.com/posts/2026/2026-06-01-ai-for-all-levels-4-specialized-ai-tools): Beyond chatbots lie specialized AI tools for creating music, generating videos, and editing images. Here is a tour of the best tools in 2026 and how they fit into daily productivity.
- [Going Advanced: Open Source Models, Hermes Agent, and Local AI](https://ingestthis.com/posts/2026/2026-06-01-ai-for-all-levels-5-going-advanced): Tired of subscriptions and privacy concerns? Open source models and agent frameworks let you run AI on your own hardware. Here is how to get started with Hermes Agent, OpenCode, DeepSeek, and local Llama models.
- [The Complete Guide to Agentic Coding Tools in 2026](https://ingestthis.com/posts/2026/2026-05-31-agentic-coding-tools): A deep dive into the four categories of agentic coding tools dominating 2026 -- CLI agents, desktop IDEs, 24/7 autonomous agents, and model routers. 40+ tools compared.
- [Data Platform Native AI Agent Tooling in 2026](https://ingestthis.com/posts/2026/2026-05-31-data-platform-ai-agent-tooling): A comprehensive comparison of AI agent tooling across Dremio, Snowflake, Databricks, Microsoft Fabric, AWS, Google Cloud, ClickHouse, VeloDB, SpiceAI, Bauplan, and Qlik.
- [Active Monitoring: How Agentic AI Auto-Heals and Protects Enterprise Data Pipelines](https://ingestthis.com/posts/2026/2026-05-28-active-monitoring-agentic-ai-pipelines): Static alerts miss cascading pipeline failures. Learn how agentic AI monitors, traces root causes, and automatically rolls back broken enterprise data pipelines.
- [Anatomy of an Agentic Analytics System: Inside the Multi-Step Reasoning Loop](https://ingestthis.com/posts/2026/2026-05-28-anatomy-agentic-analytics-system): How does an agentic analytics system actually work? Inside the ReAct loop, tool calling, schema exploration, and self-correction that power autonomous data agents.
- [Mastering Apache Iceberg v3: What's New and How to Plan Your Upgrade](https://ingestthis.com/posts/2026/2026-05-28-apache-iceberg-v3-upgrade): Apache Iceberg v3 adds deletion vectors, VARIANT type, row lineage, and table encryption. Here's what changed and how to plan your upgrade.
- [Building a Custom Agentic Analytics System: Python, LangChain, and SQL Data Lakes](https://ingestthis.com/posts/2026/2026-05-28-building-custom-agentic-analytics-python): Build a custom agentic analytics system using Python, LangChain, and Dremio. A developer tutorial covering SQL tool binding, prompt design, and secure execution.
- [The Death of the Data Swamp: Establishing Governance in Your 2026 Data Lakehouse](https://ingestthis.com/posts/2026/2026-05-28-data-governance-lakehouse-2026): Data lakehouses become data swamps without active governance. Learn how schema enforcement, catalog stewardship, and drift detection keep your lakehouse clean in 2026.
- [How Apache Iceberg Resolves the Hybrid-Cloud Challenge in Heavily Regulated Markets](https://ingestthis.com/posts/2026/2026-05-28-iceberg-hybrid-cloud-regulated-markets): Apache Iceberg gives regulated enterprises data sovereignty with hybrid-cloud deployments. Learn how open catalogs and Iceberg handle compliance requirements.
- [Securing Apache Iceberg Tables with Fine-Grained Row and Column Level Access Control](https://ingestthis.com/posts/2026/2026-05-28-iceberg-row-column-access-control): Apache Iceberg doesn't enforce row and column security on its own. Learn how catalog-level governance with Apache Polaris and Dremio delivers enterprise FGAC.
- [Designing an Immutable Data Lakehouse: Best Practices for Iceberg Snapshot Expiration](https://ingestthis.com/posts/2026/2026-05-28-iceberg-snapshot-expiration): Iceberg snapshot bloat silently kills query performance. Learn the best practices for snapshot expiration, manifest cleanup, and automated table maintenance.
- [Decoupling Storage and Compute in Apache Iceberg: A Deep Dive into Cost Optimization](https://ingestthis.com/posts/2026/2026-05-28-iceberg-storage-compute-decoupling): Learn how Apache Iceberg decouples storage from compute to cut data platform costs, enable multi-engine routing, and eliminate proprietary lock-in.
- [Legacy Warehouses to Open Lakehouses: A Step-by-Step Migration Playbook](https://ingestthis.com/posts/2026/2026-05-28-legacy-warehouse-to-lakehouse-migration): Migrating from a legacy data warehouse to an open lakehouse? This step-by-step playbook covers assessment, phased migration, validation, and avoiding common pitfalls.
- [Building the Brain of the Agentic Lakehouse: Designing an Open Catalog Architecture](https://ingestthis.com/posts/2026/2026-05-28-open-catalog-architecture-agentic-lakehouse): The open catalog is the brain of the agentic lakehouse. Learn how Apache Polaris, Dremio's Open Catalog, and catalog-native governance enable reliable AI data access.
- [Evaluating the TCO of an Open Lakehouse vs. Proprietary Data Warehouses](https://ingestthis.com/posts/2026/2026-05-28-open-lakehouse-vs-proprietary-warehouse-tco): Open lakehouse vs proprietary warehouse: a comprehensive TCO breakdown covering storage, compute, engineering, and hidden costs to help you make the right decision.
- [Real-Time BI: Enabling Sub-Second Queries on Apache Iceberg Data Lakehouses](https://ingestthis.com/posts/2026/2026-05-28-real-time-bi-iceberg-lakehouse): Sub-second queries on Apache Iceberg are achievable with the right architecture. Learn how Reflections, C3 cache, and query acceleration close the BI latency gap.
- [The Rise of Agentic Analytics: Shifting BI from Passive Dashboards to Goal-Directed Action](https://ingestthis.com/posts/2026/2026-05-28-rise-of-agentic-analytics): Agentic analytics replaces static dashboards with AI agents that pursue business goals autonomously. Learn what changed, why it matters, and what the shift requires.
- [The Semantic Layer as a Translation Engine: Bridging Natural Language and SQL](https://ingestthis.com/posts/2026/2026-05-28-semantic-layer-translation-engine): The semantic layer translates business language into accurate SQL for AI agents. Learn how virtual datasets, metric definitions, and wikis power agentic analytics accuracy.
- [Comparing the Top 2026 Agentic Analytics Tools: ThoughtSpot, Databricks, and Tableau](https://ingestthis.com/posts/2026/2026-05-28-top-agentic-analytics-tools-2026): How do ThoughtSpot, Databricks, and Tableau compare as agentic analytics platforms in 2026? A breakdown of semantic depth, agent capabilities, and where Dremio fits.
- [Trustworthy AI in the Agentic Lakehouse: Reconciling Concurrency and Isolation Contracts](https://ingestthis.com/posts/2026/2026-05-28-trustworthy-ai-concurrency-isolation): Hundreds of AI agents querying simultaneously create concurrency and isolation problems. Learn how Iceberg OCC, Dremio FGAC, and guardrail policies ensure trustworthy AI in the lakehouse.
- [The 2026 Unified Data Architecture: Reconciling Multi-Cloud Data Lakehouses](https://ingestthis.com/posts/2026/2026-05-28-unified-data-architecture-2026): Multi-cloud data lakehouses in 2026 run on Apache Iceberg, open catalogs, and zero-ETL federation. Here's what a composable, unified architecture looks like.
- [Why Traditional Lakehouses Fail AI Agents: The Mathematical Case for the Agentic Lakehouse](https://ingestthis.com/posts/2026/2026-05-28-why-lakehouses-fail-ai-agents): Traditional lakehouses expose raw directories and ambiguous schemas to AI agents, causing hallucination. Here's the mathematical case for why this fails and what fixes it.
- [The Era of Zero-ETL Federation: Fueling AI Agents with Real-Time Cross-Enterprise Data](https://ingestthis.com/posts/2026/2026-05-28-zero-etl-federation-ai-agents): Zero-ETL federation lets AI agents join real-time CRM data with historical lakehouse tables instantly. Learn the architecture, tradeoffs, and how Dremio enables it.
- [Use Hermes Agent for Free With DeepSeek V4 and Slack](https://ingestthis.com/posts/2026/2026-05-hermes-agent-free-deepseek-setup): Hermes Agent is a free, open-source AI agent from Nous Research. Connect it to DeepSeek V4 for zero-cost inference and Slack for anywhere access. Here is how to set it up in 10 minutes.
- [Automating Table Maintenance Before Small Files Accumulate](https://ingestthis.com/posts/2026/2026-05-24-automating-table-maintenance): Learn how Databricks Predictive Optimization, AWS S3 Tables, and Iceberg native actions automate compaction and snapshot management before small files degrade performance.
- [Choosing the Right Iceberg Control Plane: Polaris vs. Unity Catalog vs. Cloud REST](https://ingestthis.com/posts/2026/2026-05-24-choosing-iceberg-control-plane): Choosing an Apache Iceberg catalog? Compare open-source Apache Polaris, open Unity Catalog, and managed cloud REST control planes to unify your lakehouse.
- [Clean Rooms for Privacy-Preserving Analytics](https://ingestthis.com/posts/2026/2026-05-24-clean-rooms-privacy): Data clean rooms enable secure multi-party analytics without sharing raw data. Learn how Databricks Clean Rooms, AWS Clean Rooms, and BigQuery differential privacy work.
- [Building Composable Query Engines with Rust Runtimes](https://ingestthis.com/posts/2026/2026-05-24-composable-query-engines): Apache DataFusion, Velox, and Substrait form the foundation of modern composable query engine stacks. Learn how these components fit together and when to use each.
- [Data Mesh After the Hype: What Actually Works](https://ingestthis.com/posts/2026/2026-05-24-data-mesh-after-hype): Three years after Zhamak Dehghani's original papers, data mesh has proven valuable in specific organizational contexts and impractical in others. Here's what the practical implementations look like.
- [How dbt Fusion Reshapes Analytics Engineering](https://ingestthis.com/posts/2026/2026-05-24-dbt-fusion-analytics-engineering): dbt Fusion entered public beta in May 2025 with a Rust-powered runtime that changes how analytics engineers develop, validate, and deploy SQL models. Here's what changed.
- [Using DuckDB and Polars to Query Iceberg Tables](https://ingestthis.com/posts/2026/2026-05-24-duckdb-polars-iceberg): DuckDB 1.4 LTS and Polars streaming engine now both support reading and writing Apache Iceberg tables. Learn how to use them for lakehouse analytics in 2025.
- [FinOps for Data Warehouses with Open Billing Data](https://ingestthis.com/posts/2026/2026-05-24-finops-warehouse-cost): The FOCUS 1.3 specification and native warehouse cost views make real-time cost attribution practical. Learn how to build a FinOps pipeline for Snowflake, BigQuery, and multi-cloud environments.
- [Designing Governed RAG on Data Products](https://ingestthis.com/posts/2026/2026-05-24-governed-rag-data-products): Enterprise RAG architecture that trusts its own data requires governance at the retrieval layer. Learn how to build governed RAG using data products, access policies, and semantic layer routing.
- [What Iceberg V3 Advances Mean for CDC Pipelines](https://ingestthis.com/posts/2026/2026-05-24-iceberg-cdc-pipelines): Apache Iceberg V3 brings deletion vectors and row lineage that reshape CDC pipeline design. Learn what these features mean for your streaming data architecture.
- [Kafka 4.0 Changes Streaming Platform Operations](https://ingestthis.com/posts/2026/2026-05-24-kafka-streaming-operations): Kafka 4.0 removes ZooKeeper and ships KRaft and KIP-848 by default. Learn what those changes mean for platform operations, upgrades, and client configurations.
- [Lance and Iceberg for Multimodal AI Data](https://ingestthis.com/posts/2026/2026-05-24-lance-iceberg-multimodal): LanceDB and Apache Iceberg serve complementary roles in a multimodal AI lakehouse. Learn when to use Lance for embeddings and random access, and Iceberg for structured metadata and SQL analytics.
- [Bringing MLflow and Data Pipelines Closer Together](https://ingestthis.com/posts/2026/2026-05-24-mlflow-data-pipelines): MLflow 3 extends observability from classic ML experiments to GenAI tracing and data pipeline lineage. Learn how to connect data quality monitoring with model performance tracking.
- [Modern Feature Stores Beyond Batch Pipelines](https://ingestthis.com/posts/2026/2026-05-24-modern-feature-stores): Feature stores like Feast now support streaming feature views from Kafka and Kinesis alongside batch pipelines. Learn how to build real-time features that maintain training-serving consistency.
- [OpenLineage as the Spine of Data Observability](https://ingestthis.com/posts/2026/2026-05-24-openlineage-observability): OpenLineage provides a standard API for collecting pipeline lineage across Airflow, Spark, Flink, and dbt. Learn how it powers blast radius analysis and incident triage.
- [When Paimon Beats Iceberg for Mutable Streams](https://ingestthis.com/posts/2026/2026-05-24-paimon-vs-iceberg-mutable-streams): Apache Paimon uses LSM-Tree storage for native CDC upserts without restart. Learn when Paimon outperforms Iceberg for high-churn mutable streaming workloads.
- [Policy as Code for Lakehouse Governance](https://ingestthis.com/posts/2026/2026-05-24-policy-as-code-governance): OPA, ABAC, row filters, and column masks make lakehouse governance programmable and scalable. Learn how Databricks, Snowflake Horizon, and BigQuery implement policy-as-code.
- [Real-Time Lakehouse Patterns with Apache Flink and Iceberg](https://ingestthis.com/posts/2026/2026-05-24-real-time-lakehouse-flink): Learn how to build a real-time lakehouse with Apache Flink 2.1 and the Dynamic Iceberg Sink, covering schema evolution, exactly-once delivery, and compaction.
- [Why Semantic Layers Make Enterprise Text-to-SQL Safer](https://ingestthis.com/posts/2026/2026-05-24-semantic-layers-text-to-sql): Text-to-SQL accuracy jumps from 40% to 85-95% when grounded in a semantic layer. Learn how Dremio, Snowflake Cortex Analyst, and dbt Semantic Layer improve AI analytics reliability.
- [Choosing Vector Stores for Retrieval Workloads](https://ingestthis.com/posts/2026/2026-05-24-vector-stores-retrieval): pgvector, Milvus, Weaviate, and LanceDB each make different tradeoffs on index type, hybrid search, scale, and operational complexity. Learn which fits your retrieval workload.
- [An In-Depth Overview of the Apache Iceberg 1.11.0 Release](https://ingestthis.com/posts/2026/2026-05-23-apache-iceberg-1-11-0-deep-dive): Apache Iceberg 1.11.0 delivers manifest list encryption, the new pluggable File Format API, credential lifecycle refreshes, and Spark/Flink improvements.
- [Single-Node Data Engineering: DuckDB, DataFusion, Polars, and LakeSail](https://ingestthis.com/posts/2026/2026-05-23-single-node-laptop-data-engineering): Optimize single-node data engineering with DuckDB, DataFusion, Polars, and LakeSail. Compare architectures and learn when to transition to Dremio MPP.
- [What Are Table Formats and Why Were They Needed?](https://ingestthis.com/posts/2026/2026-04-29-apache-iceberg-masterclass-01-table-formats): Table formats like Apache Iceberg solved the ACID, schema, and performance problems that turned data lakes into data swamps. Here is how each one works.
- [The Metadata Structure of Modern Table Formats](https://ingestthis.com/posts/2026/2026-04-29-apache-iceberg-masterclass-02-metadata-structures): Iceberg uses a metadata tree, Delta Lake uses a transaction log, Hudi uses a timeline. Here is exactly how each format organizes metadata and why it matters.
- [Performance and Apache Iceberg's Metadata](https://ingestthis.com/posts/2026/2026-04-29-apache-iceberg-masterclass-03-iceberg-metadata-performance): Iceberg's three-layer metadata tree eliminates directory listing and enables multi-level data skipping. Here is how scan planning actually works.
- [Partition Evolution: Change Your Partitioning Without Rewriting Data](https://ingestthis.com/posts/2026/2026-04-29-apache-iceberg-masterclass-04-partition-evolution): Iceberg lets you change partition schemes without rewriting data. Here is how partition evolution works internally and why Hive-style partitioning could not do this.
- [Hidden Partitioning: How Iceberg Eliminates Accidental Full Table Scans](https://ingestthis.com/posts/2026/2026-04-29-apache-iceberg-masterclass-05-hidden-partitioning): Iceberg's hidden partitioning separates physical layout from user queries using transform functions. Here is how it works and why it eliminates accidental full scans.
- [Writing to an Apache Iceberg Table: How Commits and ACID Actually Work](https://ingestthis.com/posts/2026/2026-04-29-apache-iceberg-masterclass-06-writing-to-iceberg): Here is exactly how an engine writes to an Iceberg table, step by step, from data files through the atomic commit that makes ACID guarantees possible.
- [What Are Lakehouse Catalogs? The Role of Catalogs in Apache Iceberg](https://ingestthis.com/posts/2026/2026-04-29-apache-iceberg-masterclass-07-lakehouse-catalogs): Lakehouse catalogs store metadata pointers, manage namespaces, and enforce access control. Here is the complete catalog landscape from Polaris to Glue.
- [When Catalogs Are Embedded in Storage](https://ingestthis.com/posts/2026/2026-04-29-apache-iceberg-masterclass-08-embedded-catalogs): S3 Tables and MinIO AI Stor embed the Iceberg catalog directly in the storage layer. Here is when embedded catalogs make sense and when they do not.
- [How Data Lake Table Storage Degrades Over Time](https://ingestthis.com/posts/2026/2026-04-29-apache-iceberg-masterclass-09-storage-degradation): Iceberg tables degrade through small files, orphan files, metadata bloat, sort order decay, and partition skew. Here is how to diagnose each problem.
- [Maintaining Apache Iceberg Tables: Compaction, Expiry, and Cleanup](https://ingestthis.com/posts/2026/2026-04-29-apache-iceberg-masterclass-10-maintaining-iceberg): Keep Iceberg tables fast with compaction, snapshot expiry, orphan cleanup, and manifest rewriting. Here is when and how to run each operation.
- [Apache Iceberg Metadata Tables: Querying the Internals](https://ingestthis.com/posts/2026/2026-04-29-apache-iceberg-masterclass-11-metadata-tables): Iceberg metadata tables let you query snapshots, files, manifests, and partitions using SQL. Here is every metadata table and how to use them.
- [Using Apache Iceberg with Python and MPP Query Engines](https://ingestthis.com/posts/2026/2026-04-29-apache-iceberg-masterclass-12-python-and-mpp): Access Iceberg tables from Python with PyIceberg, DuckDB, and Polars, or through MPP engines like Dremio, Spark, and Trino. Here is how each approach works.
- [Approaches to Streaming Data into Apache Iceberg Tables](https://ingestthis.com/posts/2026/2026-04-29-apache-iceberg-masterclass-13-streaming-to-iceberg): Stream data into Iceberg with Spark Structured Streaming, Flink, or Kafka Connect. Here is how each works and the trade-offs between latency and maintenance.
- [Hands-On with Apache Iceberg Using Dremio Cloud](https://ingestthis.com/posts/2026/2026-04-29-apache-iceberg-masterclass-14-hands-on-dremio-cloud): A practical walkthrough of creating, querying, and optimizing Iceberg tables on Dremio Cloud, from account setup to AI-powered analytics.
- [Migrating to Apache Iceberg: Strategies for Every Source System](https://ingestthis.com/posts/2026/2026-04-29-apache-iceberg-masterclass-15-migrating-to-iceberg): Migrate to Iceberg from Hive, data warehouses, or raw files using in-place migration, full rewrite, or the zero-downtime view swap pattern.
- [How Query Engines Think: The Tradeoffs Behind Every Data System](https://ingestthis.com/posts/2026/2026-04-29-query-engine-optimization-01-overview): Every database is a collection of engineering tradeoffs. Learn the 9 design decisions that shape how query engines store, index, and process your data.
- [Row vs. Column: How Storage Layout Shapes Everything](https://ingestthis.com/posts/2026/2026-04-29-query-engine-optimization-02-row-vs-column-storage): Row stores keep records together for fast transactions. Column stores keep field values together for fast analytics. Here is how each layout works and when to use it.
- [How Databases Organize Data on Disk: Pages, Blocks, and File Formats](https://ingestthis.com/posts/2026/2026-04-29-query-engine-optimization-03-data-organization-on-disk): Databases structure data on disk as heap files, sorted files, or LSM trees, then wrap it in formats like Parquet with metadata that lets engines skip irrelevant blocks.
- [B-Trees, LSM Trees, and the Indexing Tradeoff Spectrum](https://ingestthis.com/posts/2026/2026-04-29-query-engine-optimization-04-indexing-strategies): B-trees balance reads and writes for OLTP. LSM trees maximize write throughput. Bitmap indexes accelerate OLAP filtering. Here is when to use each.
- [Inside the Query Optimizer: How Engines Pick a Plan](https://ingestthis.com/posts/2026/2026-04-29-query-engine-optimization-05-query-optimizer): Query optimizers transform SQL into execution plans using rule-based rewrites, cost-based search, and adaptive runtime adjustments. Here is how each approach works.
- [Volcano, Vectorized, Compiled: How Engines Execute Your Query](https://ingestthis.com/posts/2026/2026-04-29-query-engine-optimization-06-execution-models): The Volcano model processes one row at a time. Vectorized execution processes batches with SIMD. Code generation fuses operators into compiled code. Here is how each works.
- [Buffer Pools, Caches, and the Memory Hierarchy](https://ingestthis.com/posts/2026/2026-04-29-query-engine-optimization-07-memory-and-caching): Databases use buffer pools, column caches, and result caches to keep hot data in RAM. Here is how each caching strategy works and what happens when data does not fit.
- [Partitioning, Sharding, and Data Distribution Strategies](https://ingestthis.com/posts/2026/2026-04-29-query-engine-optimization-08-partitioning): Hash partitioning distributes data evenly. Range partitioning enables fast range scans. Both create tradeoffs. Here is how databases divide data across storage and nodes.
- [Hash, Sort-Merge, Broadcast: How Distributed Joins Work](https://ingestthis.com/posts/2026/2026-04-29-query-engine-optimization-09-distributed-joins): Distributed joins move data across the network using shuffle, broadcast, or co-location strategies. Here is how each works and when engines choose which.
- [Concurrency, Isolation, and MVCC: How Engines Handle Contention](https://ingestthis.com/posts/2026/2026-04-29-query-engine-optimization-10-concurrency-control): Databases handle concurrent access using locks, MVCC, or optimistic concurrency control. Here is how each approach works and what tradeoffs each creates.
- [Apache Software Foundation: History, Purpose, and Process](https://ingestthis.com/posts/2026/2026-04-13-apache-lakehouse-01-apache-software-foundation): Part of the Open Source and the Lakehouse series
- [What is Apache Parquet? Columns, Encoding, and Performance](https://ingestthis.com/posts/2026/2026-04-13-apache-lakehouse-02-apache-parquet): Part of the Open Source and the Lakehouse series
- [What is Apache Iceberg? The Table Format Revolution](https://ingestthis.com/posts/2026/2026-04-13-apache-lakehouse-03-apache-iceberg): Part of the Open Source and the Lakehouse series
- [What is Apache Polaris? Unifying the Iceberg Ecosystem](https://ingestthis.com/posts/2026/2026-04-13-apache-lakehouse-04-apache-polaris): Part of the Open Source and the Lakehouse series
- [What is Apache Arrow? Erasing the Serialization Tax](https://ingestthis.com/posts/2026/2026-04-13-apache-lakehouse-05-apache-arrow): Part of the Open Source and the Lakehouse series
- [Assembling the Apache Lakehouse: The Modular Architecture](https://ingestthis.com/posts/2026/2026-04-13-apache-lakehouse-06-assembling-apache-lakehouse): Part of the Open Source and the Lakehouse series
- [Agentic Analytics on the Apache Lakehouse](https://ingestthis.com/posts/2026/2026-04-13-apache-lakehouse-07-agentic-analytics): Part of the Open Source and the Lakehouse series
- [Context Management Strategies for ChatGPT: A Complete Guide to Getting Better Results](https://ingestthis.com/posts/2026/2026-03-07-context-management-blogs-01-chatgpt): Getting consistently useful results from ChatGPT requires more than writing good prompts. The real differentiator is how you manage context: the background information, instructions, documents, and accumulated knowledge that shapes every response ChatGPT generates. Without deliberate context management, you end up repeating yourself, getting generic answers, and wasting time course-correcting the AI.
- [Context Management Strategies for OpenAI Codex: A Complete Guide Across Browser, CLI, and App](https://ingestthis.com/posts/2026/2026-03-07-context-management-blogs-02-openai-codex): OpenAI Codex is not a chatbot. It is an autonomous software engineering agent that runs tasks in isolated cloud sandboxes, operates across a browser interface, a command-line tool, and a dedicated macOS app, and can work on multiple tasks in parallel. Because of this architecture, context management in Codex works fundamentally differently from ChatGPT or traditional coding assistants. Instead of conversational context windows, you manage context through persistent configuration files, skill definitions, and project-level instructions that shape how the agent approaches your codebase.
- [Context Management Strategies for Claude Web: A Complete Guide to Projects, Artifacts, and Intelligent Context](https://ingestthis.com/posts/2026/2026-03-07-context-management-blogs-03-claude-web): Claude's web interface at claude.ai combines one of the largest context windows in the industry with a structured Project system that makes it genuinely useful for sustained, complex work. While many AI chat interfaces are limited to one-off conversations, Claude Web is designed for ongoing engagement where the AI accumulates understanding of your work over time. The key to unlocking that potential is managing context deliberately rather than treating each conversation as a blank slate.
- [Context Management Strategies for Claude Desktop: A Complete Guide to MCP, Computer Use, and Local File Access](https://ingestthis.com/posts/2026/2026-03-07-context-management-blogs-04-claude-desktop): Claude Desktop takes everything available in Claude Web and adds three capabilities that fundamentally change how you manage context: MCP server connections that link Claude to external tools and data sources, direct local file access that eliminates the upload-download cycle, and Computer Use that lets Claude interact with your desktop environment. These additions make Claude Desktop the right choice when your work requires live data, local file system access, or integration with tools that Claude Web cannot reach.
- [Context Management Strategies for Claude CoWork: A Complete Guide for Knowledge Workers](https://ingestthis.com/posts/2026/2026-03-07-context-management-blogs-05-claude-cowork): Claude CoWork represents a fundamentally different approach to AI context management. Unlike chat interfaces where you send messages and receive responses, CoWork is an autonomous agent that works on your local machine, reads and writes files directly, and executes multi-step tasks with minimal supervision. For knowledge workers who spend their days in documents, spreadsheets, and presentations, CoWork replaces the constant back-and-forth of copy-paste workflows with direct delegation.
- [Context Management Strategies for Claude Code: A Complete Guide for Developers](https://ingestthis.com/posts/2026/2026-03-07-context-management-blogs-06-claude-code): Claude Code is a terminal-native agentic coding assistant that lives in your command line and operates directly on your codebase. Unlike chat-based interfaces where you copy and paste code snippets, Claude Code reads your files, explores your project structure, runs commands, executes tests, and commits changes. Context management in Claude Code is about configuring the agent's persistent knowledge of your project so it can operate effectively without constant direction.
- [Context Management Strategies for Gemini Web and NotebookLM: A Complete Guide to Google's AI Knowledge Ecosystem](https://ingestthis.com/posts/2026/2026-03-07-context-management-blogs-07-gemini-web-notebooklm): Google's AI ecosystem for knowledge work consists of two deeply integrated tools: Gemini (the conversational AI at gemini.google.com) and NotebookLM (the research-focused assistant at notebooklm.google.com). In early 2026, these two platforms became interoperable, allowing Gemini to access information stored in NotebookLM notebooks. This integration creates something unique in the AI landscape: a persistent knowledge infrastructure where documents you upload once become available across both conversational and research interfaces.
- [Context Management Strategies for Gemini CLI: A Complete Guide to Terminal-Native AI Development](https://ingestthis.com/posts/2026/2026-03-07-context-management-blogs-08-gemini-cli): Gemini CLI is an open-source terminal agent powered by Gemini models that operates directly in your command line. It brings Google's AI capabilities into the environment where many developers already live, with a context management system built around hierarchical configuration files, persistent memory, MCP server integration, and direct codebase interaction. Unlike web-based tools where context is managed through uploads and conversation, Gemini CLI assembles its context from your project structure, your instruction files, and the tools you connect to it.
- [Context Management Strategies for Google Antigravity: A Complete Guide to the Agent-First IDE](https://ingestthis.com/posts/2026/2026-03-07-context-management-blogs-09-google-antigravity): Google Antigravity is an agent-first IDE built by Google DeepMind's Advanced Agentic Coding team. It approaches context management differently from other AI coding tools because it is designed from the ground up around agentic workflows, where the AI is not just an assistant responding to prompts, but an autonomous agent that plans, executes, tracks progress, and retains knowledge across sessions. Its context management system centers on three pillars: Skills for reusable capability, Knowledge Items for persistent memory, and Artifacts for transparent documentation of its work.
- [Context Management Strategies for OpenCode: A Complete Guide to the Open-Source Terminal AI Agent](https://ingestthis.com/posts/2026/2026-03-07-context-management-blogs-10-opencode): OpenCode is an open-source terminal-based AI coding agent that prioritizes privacy, local-first operation, and broad model provider support. Built as a TUI (terminal user interface) application, it runs entirely in your terminal and supports dozens of LLM providers from OpenAI and Anthropic to local models through Ollama. Its context management system is built around configuration files, session persistence, MCP integration, and a dual-agent architecture that separates planning from code generation.
- [Context Management Strategies for OpenWork: A Complete Guide to the Desktop AI Agent Framework](https://ingestthis.com/posts/2026/2026-03-07-context-management-blogs-11-openwork): OpenWork is a desktop-native AI agent framework designed for local, multi-step task execution on your computer. Unlike browser-based AI tools or terminal agents, OpenWork operates as a desktop application that can interact with your file system, manage long-running sessions, and execute complex workflows autonomously. Its context management centers on Skills, session persistence, direct file system access, and a plugin architecture that extends its capabilities.
- [Context Management Strategies for Cursor: A Complete Guide to the AI-Native Code Editor](https://ingestthis.com/posts/2026/2026-03-07-context-management-blogs-12-cursor): Cursor is an AI-native code editor built on the VS Code foundation that integrates AI deeply into every aspect of the development workflow. Its context management system is one of the most sophisticated among coding tools, combining workspace-level indexing, granular rules files, documentation integration, MCP server support, and intelligent context assembly that automatically determines which files and symbols are relevant to your current task.
- [Context Management Strategies for Perplexity AI: A Complete Guide to Research-First AI Conversations](https://ingestthis.com/posts/2026/2026-03-07-context-management-blogs-13-perplexity): Perplexity AI occupies a unique position in the AI landscape: it is a research-first tool that combines conversational AI with real-time web search to produce answers grounded in current sources. Unlike coding-focused tools or general chatbots, Perplexity is built for information retrieval, analysis, and synthesis. Its context management is designed around Spaces (persistent research workspaces), Focus Modes (search scope control), and an elastic context window that adapts to the complexity of your query.
- [Context Management Strategies for Windsurf: A Complete Guide to the AI Flow IDE](https://ingestthis.com/posts/2026/2026-03-07-context-management-blogs-14-windsurf): Windsurf is an AI-powered IDE built on the VS Code foundation that introduces the concept of "Flows," a paradigm where the AI maintains deep awareness of your actions, codebase, and development patterns over time. Its context management differentiates from other editors through Cascade (its agentic coding assistant), persistent Rules files, Memories, and a sophisticated context engine that tracks not just what files you are editing, but how you work.
- [Context Management Strategies for Zed: A Complete Guide to the High-Performance AI Code Editor](https://ingestthis.com/posts/2026/2026-03-07-context-management-blogs-15-zed): Zed is a high-performance code editor built in Rust that prioritizes speed, simplicity, and real-time collaboration. Its AI integration is designed to be fast and unobtrusive, with context management built around an assistant panel, inline transformations, slash commands, and a flexible provider system that supports multiple AI services. What sets Zed apart from other AI editors is its focus on performance (everything runs natively, not in Electron) and its built-in multiplayer editing that extends to AI interactions.
- [Context Management Strategies for T3 Chat: A Complete Guide to the Unified Multi-Model AI Interface](https://ingestthis.com/posts/2026/2026-03-07-context-management-blogs-16-t3-chat): T3 Chat is a modern web-based AI chat interface that gives you access to multiple AI models through a single unified platform. Its primary value proposition is model flexibility: instead of being locked into one provider, you can switch between Claude, GPT, Gemini, Llama, and other models within the same interface. This makes T3 Chat unique from a context management perspective because the same context strategies must work across fundamentally different model families with different capabilities, context window sizes, and strengths.
- [Context Management Strategies for VS Code with LLM Plugins: A Complete Guide to Building Your Own AI-Powered IDE](https://ingestthis.com/posts/2026/2026-03-07-context-management-blogs-17-vscode-llm-plugins): Visual Studio Code is the most widely used code editor in the world, and its extensibility means you can integrate AI capabilities through a growing ecosystem of LLM plugins. Unlike purpose-built AI editors (Cursor, Windsurf, Zed), VS Code gives you the freedom to choose and combine AI extensions, configure them to your preferences, and even switch between providers without changing editors. The tradeoff is that context management is not as seamlessly integrated as in dedicated AI editors. It requires more deliberate configuration.
- [The Model Context Protocol (MCP) Explained: A Complete Guide to How Every Major AI Tool Connects to External Data](https://ingestthis.com/posts/2026/2026-03-07-context-management-blogs-18-mcp-deep-dive): The Model Context Protocol (MCP) has become the universal standard for connecting AI models to external tools, data sources, and services. Originally open-sourced by Anthropic in November 2024 and now managed by the Linux Foundation, MCP solves one of the biggest frustrations in working with AI: getting models to interact with the systems where your actual work lives. Instead of copying and pasting data into chat windows or uploading files manually, MCP lets AI tools query databases, read documentation, interact with APIs, manage files, and perform actions across your entire tool ecosystem through a standardized protocol.
- [How to Use Dremio with Amazon Kiro: Connect, Query, and Build Data Apps](https://ingestthis.com/posts/2026/2026-03-05-aitoolblogs-amazon-kiro): Amazon Kiro is an agentic AI IDE from AWS that introduces spec-driven development to the coding workflow. Instead of jumping straight to code, Kiro helps you define structured specifications : requirements, technical designs, and task breakdowns,  before writing a single line. It then generates code that follows those specs and keeps everything in sync as the project evolves. Dremio is a unified lakehouse platform that provides business context through its semantic layer, universal data access through query federation, and interactive speed through Reflections and Apache Arrow.
- [How to Use Dremio with Claude Code: Connect, Query, and Build Data Apps](https://ingestthis.com/posts/2026/2026-03-05-aitoolblogs-claude-code): Claude Code is Anthropic's terminal-based coding agent. It reads your files, writes code, runs commands, and maintains context across a session. Dremio is a unified lakehouse platform that gives AI agents three things they need to answer business questions accurately: deep business context through its semantic layer, universal data access through query federation, and interactive speed through Reflections and Apache Arrow.
- [How to Use Dremio with Claude CoWork: Connect, Query, and Build Data Apps](https://ingestthis.com/posts/2026/2026-03-05-aitoolblogs-claude-cowork): Claude CoWork is Anthropic's desktop agentic assistant. Unlike Claude Code (a terminal coding agent), CoWork operates as a general-purpose autonomous agent that reads and writes files, browses the web, manages tasks, and generates complete project artifacts. Dremio is a unified lakehouse platform that provides business context through its semantic layer, universal data access through query federation, and interactive speed through Reflections.
- [How to Use Dremio with Cursor: Connect, Query, and Build Data Apps](https://ingestthis.com/posts/2026/2026-03-05-aitoolblogs-cursor): Cursor is an AI-native code editor built as a fork of VS Code. It integrates AI directly into the editing experience with features like Chat, Composer (multi-file editing), and inline code generation. Dremio is a unified lakehouse platform that provides business context through its semantic layer, universal data access through query federation, and interactive speed through Reflections and Apache Arrow.
- [How to Use Dremio with Gemini CLI: Connect, Query, and Build Data Apps](https://ingestthis.com/posts/2026/2026-03-05-aitoolblogs-gemini-cli): Gemini CLI is Google's open-source terminal-based AI agent. It runs directly in your terminal, powered by Gemini models with a 1-million token context window. Dremio is a unified lakehouse platform that provides business context through its semantic layer, universal data access through query federation, and interactive speed through Reflections and Apache Arrow.
- [How to Use Dremio with GitHub Copilot: Connect, Query, and Build Data Apps](https://ingestthis.com/posts/2026/2026-03-05-aitoolblogs-github-copilot): GitHub Copilot is the most widely adopted AI coding assistant, integrated into VS Code, JetBrains IDEs, and the GitHub platform. Its agent mode allows Copilot to plan and execute multi-step coding tasks, run terminal commands, and interact with external tools through MCP. The Copilot CLI extends agentic development to the terminal. Dremio is a unified lakehouse platform that provides business context through its semantic layer, universal data access through query federation, and interactive speed through Reflections and Apache Arrow.
- [How to Use Dremio with Google Antigravity: Connect, Query, and Build Data Apps](https://ingestthis.com/posts/2026/2026-03-05-aitoolblogs-google-antigravity): Google Antigravity is an agent-first IDE built by Google DeepMind. Its autonomous agents plan multi-step tasks, write code, browse documentation, and iterate without constant hand-holding. Dremio is a unified lakehouse platform that provides the business context, universal data access, and interactive query speed that AI agents need to produce accurate analytics.
- [How to Use Dremio with JetBrains AI Assistant: Connect, Query, and Build Data Apps](https://ingestthis.com/posts/2026/2026-03-05-aitoolblogs-jetbrains-ai): JetBrains AI Assistant is built into IntelliJ IDEA, PyCharm, DataGrip, and every JetBrains IDE. It provides AI chat, inline code generation, multi-file refactoring, and agentic background workers that can autonomously execute multi-step tasks. Dremio is a unified lakehouse platform that provides business context through its semantic layer, universal data access through query federation, and interactive speed through Reflections and Apache Arrow.
- [How to Use Dremio with OpenAI Codex CLI: Connect, Query, and Build Data Apps](https://ingestthis.com/posts/2026/2026-03-05-aitoolblogs-openai-codex): OpenAI Codex CLI is a terminal-based coding agent built in Rust. It reads your codebase, writes files, executes commands, and supports MCP for connecting to external data services. Dremio is a unified lakehouse platform that provides the business context, universal data access, and query speed that coding agents need to produce accurate, working analytics code.
- [How to Use Dremio with OpenCode: Connect, Query, and Build Data Apps](https://ingestthis.com/posts/2026/2026-03-05-aitoolblogs-opencode): OpenCode is an open-source, terminal-based AI coding agent released under the MIT license. It provides a TUI with split panes, uses the Language Server Protocol (LSP) for deep codebase understanding, and maintains persistent project context through file-based memory. Dremio is a unified lakehouse platform built on open standards like Apache Iceberg, Apache Arrow, and Apache Polaris.
- [How to Use Dremio with OpenWork: Connect, Query, and Build Data Apps](https://ingestthis.com/posts/2026/2026-03-05-aitoolblogs-openwork): OpenWork is an open-source desktop AI agent built on the OpenCode engine. It runs entirely on your machine with your own API keys, giving you full control over your data and your AI costs. Dremio is a unified lakehouse platform built on open standards like Apache Iceberg, Apache Arrow, and Apache Polaris.
- [How to Use Dremio with Windsurf: Connect, Query, and Build Data Apps](https://ingestthis.com/posts/2026/2026-03-05-aitoolblogs-windsurf): Windsurf is an AI-native code editor built as a fork of VS Code. Its standout feature is Cascade, an agentic AI system that plans and executes multi-step coding tasks autonomously. Cascade understands your entire codebase, can chain together multiple file edits, terminal commands, and tool calls in a single flow. Dremio is a unified lakehouse platform that provides business context through its semantic layer, universal data access through query federation, and interactive speed through Reflections and Apache Arrow.
- [How to Use Dremio with Zed: Connect, Query, and Build Data Apps](https://ingestthis.com/posts/2026/2026-03-05-aitoolblogs-zed): Zed is an open-source, GPU-accelerated code editor written in Rust. It is designed for speed and collaboration, with a built-in AI assistant that supports multiple LLM providers and an agent mode for autonomous multi-step development. Dremio is a unified lakehouse platform that provides business context through its semantic layer, universal data access through query federation, and interactive speed through Reflections and Apache Arrow.
- [Classify Your Data with SQL: A Hands-On Guide to Dremio's AI_CLASSIFY Function](https://ingestthis.com/posts/2026/2026-03-01-AI_FEATURE_BLOGS-01-ai-classify): Most classification workflows require exporting data to Python, running a model, and importing results back into your warehouse. Dremio's `AI_CLASSIFY` function eliminates that entire pipeline. You write a SELECT statement, pass in your text and your categories, and the LLM assigns a label. The classified data stays in your lakehouse, governed and queryable immediately.
- [Generate Summaries and Insights with Dremio's AI_COMPLETE Function](https://ingestthis.com/posts/2026/2026-03-01-AI_FEATURE_BLOGS-02-ai-complete): Every data team has a version of this problem: a table full of raw data that needs human-readable summaries, translations, or narrative descriptions. Product descriptions that need rewriting for a new market. Customer records that need one-sentence executive summaries. Support interactions that need post-call notes.
- [Extract Structured Data from Text with Dremio's AI_GENERATE Function](https://ingestthis.com/posts/2026/2026-03-01-AI_FEATURE_BLOGS-03-ai-generate): Unstructured text is the most underused data in most organizations. Customer emails sit in inboxes. Contract notes live in text fields. Meeting summaries exist as free-text columns in CRM systems. The information is there, but it's locked inside prose that SQL can't filter, join, or aggregate.
- [Connect PostgreSQL to Dremio Cloud: Query, Federate, and Accelerate Your Data](https://ingestthis.com/posts/2026/2026-03-01-connector-blogs-01-postgresql): PostgreSQL powers more production applications than almost any other open-source database. It's where your customer records, transaction logs, product catalogs, and operational data live. But running analytics directly against PostgreSQL creates problems: heavy analytical queries compete with transactional workloads, cross-database joins require custom ETL, and your data team can't access PostgreSQL data alongside data in S3, Snowflake, or other systems without building pipelines.
- [Connect MySQL to Dremio Cloud: Federated Analytics Without ETL](https://ingestthis.com/posts/2026/2026-03-01-connector-blogs-02-mysql): MySQL runs more web applications, SaaS platforms, and e-commerce backends than any other database. It's fast for transactional reads and writes, but it becomes a bottleneck when your data team needs to run analytical queries, join MySQL data with other sources, or build dashboards that don't compete with application traffic.
- [Connect Oracle Database to Dremio Cloud: Enterprise Analytics Without Data Movement](https://ingestthis.com/posts/2026/2026-03-01-connector-blogs-03-oracle): Oracle Database runs the most critical enterprise applications in the world : ERP systems, financial ledgers, supply chain management, and HR platforms. These systems generate massive volumes of data that business teams want to analyze, but running analytical queries directly against Oracle is expensive (license costs scale with CPU usage), complex (Oracle-specific SQL dialects and tooling), and risky (heavy queries can impact transactional performance).
- [Connect Microsoft SQL Server to Dremio Cloud: Federate Enterprise Data Without ETL](https://ingestthis.com/posts/2026/2026-03-01-connector-blogs-04-microsoft-sql-server): Microsoft SQL Server is one of the most widely deployed enterprise databases in the world. ERP systems, CRM platforms, financial applications, and custom business applications run on SQL Server across on-premises data centers and Azure cloud deployments. But connecting SQL Server data to a modern analytics platform typically requires building ETL pipelines, managing SSIS packages, or purchasing additional SQL Server Enterprise licenses for analytics workloads.
- [Connect IBM Db2 to Dremio Cloud: Modernize Mainframe Analytics with Federation and AI](https://ingestthis.com/posts/2026/2026-03-01-connector-blogs-05-ibm-db2): IBM Db2 is the relational database that powers critical applications across banking, insurance, government, healthcare, and manufacturing. For organizations running Db2 : particularly on IBM Z (mainframes) or IBM i,  the database holds decades of transactional data: account balances, policy records, claim histories, manufacturing workflows, and government records. This data is enormously valuable for analytics but notoriously difficult to access outside the Db2/IBM ecosystem.
- [Connect SAP HANA to Dremio Cloud: Unlock Analytics Beyond the SAP Ecosystem](https://ingestthis.com/posts/2026/2026-03-01-connector-blogs-06-sap-hana): SAP HANA is the in-memory database platform that powers SAP S/4HANA, SAP BW/4HANA, and custom enterprise applications across finance, manufacturing, logistics, and supply chain. It's fast for SAP-native analytics : real-time financial reporting, material requirements planning, and production analytics run directly on HANA's in-memory columnar engine. But SAP HANA exists in a walled garden.
- [Connect Amazon S3 to Dremio Cloud: Query Your Data Lake with SQL, Federation, and AI](https://ingestthis.com/posts/2026/2026-03-01-connector-blogs-07-amazon-s3): Amazon S3 is the default landing zone for data in the cloud. Log files, Parquet datasets, CSV exports, JSON events, IoT telemetry, and raw data dumps : it all ends up in S3 buckets. But S3 is storage, not an analytics engine. You can't run SQL against S3 natively. To query it, you need Amazon Athena (per-TB pricing), AWS Glue ETL jobs (cluster management), or a data warehouse that imports the data. All add cost, complexity, and latency.
- [Connect Azure Storage to Dremio Cloud: Query Your Microsoft Data Lake with SQL and AI](https://ingestthis.com/posts/2026/2026-03-01-connector-blogs-08-azure-storage): Azure Storage is Microsoft's cloud storage platform, spanning Blob Storage, Azure Data Lake Storage Gen2 (ADLS Gen2), and Azure Files. If your organization uses Microsoft Azure, your data lake almost certainly lives in Azure Storage : Parquet files from Azure Data Factory pipelines, CSV exports from Azure SQL Database, JSON event streams from Azure Event Hubs, and raw data from Azure IoT Hub all land in Azure Storage containers.
- [Connect Amazon Redshift to Dremio Cloud: Extend Your Warehouse with Federation and AI Analytics](https://ingestthis.com/posts/2026/2026-03-01-connector-blogs-09-amazon-redshift): Amazon Redshift is AWS's managed data warehouse, designed for petabyte-scale analytics. If your organization chose Redshift for analytical workloads, you've built data pipelines, ETL jobs, and dashboards around it. But as data ecosystems grow, Redshift's limitations become painfully clear: connecting data outside Redshift requires ETL or Redshift Spectrum (additional cost per TB scanned), sharing Redshift data with non-AWS tools means exporting to S3, and Redshift's concurrency limits constrain how many dashboards and users can query simultaneously.
- [Connect Google BigQuery to Dremio Cloud: Cross-Cloud Analytics Without Data Movement](https://ingestthis.com/posts/2026/2026-03-01-connector-blogs-10-google-bigquery): Google BigQuery is Google Cloud's serverless data warehouse. If your organization uses Google Cloud Platform, BigQuery is where your analytics data, marketing attribution, Google Analytics exports, and machine learning model outputs live. BigQuery is powerful within Google's ecosystem, but it creates challenges when your data spans multiple clouds or when costs grow with usage.
- [Connect Snowflake to Dremio Cloud: Federate, Govern, and Accelerate Beyond Snowflake](https://ingestthis.com/posts/2026/2026-03-01-connector-blogs-11-snowflake): Snowflake is a popular cloud data warehouse known for its separation of storage and compute, near-zero maintenance, and broad ecosystem. Many organizations have made Snowflake their primary analytics platform. But as data ecosystems mature, limitations emerge: Snowflake credits are consumed on every query, connecting Snowflake data to non-Snowflake sources requires data sharing agreements or ETL, and running all workloads in Snowflake means paying Snowflake prices for everything : including repetitive dashboard queries and ad-hoc exploration.
- [Connect Azure Synapse Analytics to Dremio Cloud: Multi-Cloud Data Warehouse Federation](https://ingestthis.com/posts/2026/2026-03-01-connector-blogs-12-azure-synapse): Microsoft Azure Synapse Analytics combines big data analytics and enterprise data warehousing into a single Azure-integrated platform. If your organization has chosen the Microsoft cloud ecosystem, your cleaned and modeled analytical data likely lives in Synapse dedicated SQL pools or serverless SQL pools. Synapse works well within Azure, but it creates challenges when you need to connect that data with AWS, Google Cloud, or on-premises databases. Azure Data Factory pipelines handle some of this, but they add cost, latency, and engineering complexity.
- [Connect Vertica to Dremio Cloud: Federation for Analytics-Optimized Data](https://ingestthis.com/posts/2026/2026-03-01-connector-blogs-13-vertica): Vertica is a columnar analytics database engineered for fast aggregate queries on large datasets. It was built from the ground up for analytical workloads : column-oriented storage, massively parallel processing, and automatic database design optimization. Organizations running Vertica typically have years of investment in analytics infrastructure: curated schemas, optimized projections, and sophisticated workloads that depend on Vertica's high-performance query engine.
- [Connect MongoDB to Dremio Cloud: SQL Analytics on Document Data](https://ingestthis.com/posts/2026/2026-03-01-connector-blogs-14-mongodb): MongoDB is the most popular NoSQL document database. It stores data in flexible JSON-like documents, making it ideal for applications with evolving schemas : user profiles, product catalogs, IoT sensor data, and content management systems. But MongoDB's document model creates analytics challenges: you can't run SQL joins natively, aggregation pipelines are complex, and connecting MongoDB data to relational sources requires custom application code or ETL.
- [Connect Apache Druid to Dremio Cloud: Add SQL Joins, AI, and Governance to Your Real-Time Analytics](https://ingestthis.com/posts/2026/2026-03-01-connector-blogs-15-apache-druid): Apache Druid is a real-time analytics database designed for sub-second queries on high-ingestion-rate event data. Clickstream analytics, application monitoring, IoT telemetry, and ad-tech workloads rely on Druid's columnar storage and inverted indexes for instantaneous queries.
- [Connect AWS Glue Data Catalog to Dremio Cloud: Query and Manage Your AWS Iceberg Tables](https://ingestthis.com/posts/2026/2026-03-01-connector-blogs-16-aws-glue): AWS Glue Data Catalog is AWS's managed metadata service for data lakes. It stores table definitions, schemas, partition information, and statistics for data stored in Amazon S3. If you've built your data lake on AWS using Apache Spark (on EMR), AWS Glue ETL jobs, or Amazon Athena, your table metadata lives in Glue. But Glue is just a catalog : a registry of what's where. To actually query the data, you need Athena (per-TB pricing), EMR clusters (infrastructure management), or Redshift Spectrum (additional cost).
- [Connect Snowflake Open Catalog to Dremio Cloud: Multi-Engine Iceberg Analytics](https://ingestthis.com/posts/2026/2026-03-01-connector-blogs-17-snowflake-open-catalog): Snowflake Open Catalog is Snowflake's managed implementation of the Apache Iceberg REST catalog specification, based on the open-source Apache Polaris project. It serves as a centralized metadata catalog for Apache Iceberg tables, enabling multiple compute engines : including Dremio, Spark, Trino, and Flink,  to read from and write to the same Iceberg tables without metadata conflicts.
- [Connect Databricks Unity Catalog to Dremio Cloud: Query Delta Lake Tables with Federation and AI](https://ingestthis.com/posts/2026/2026-03-01-connector-blogs-18-unity-catalog): Databricks Unity Catalog is Databricks' governance layer for data and AI assets. It manages Delta Lake tables, machine learning models, feature stores, and other data objects across Databricks workspaces. If your data engineering team uses Databricks for ETL and ML, your curated analytical datasets likely live in Unity Catalog as Delta Lake tables.
- [Connect Any Iceberg REST Catalog to Dremio Cloud: Universal Lakehouse Access](https://ingestthis.com/posts/2026/2026-03-01-connector-blogs-19-iceberg-rest-catalog): The Apache Iceberg REST Catalog specification defines a standard HTTP API for managing Iceberg table metadata. Any catalog implementation that conforms to this specification : Apache Polaris, Amazon S3 Tables, Confluent Tableflow, Tabular, Apache Gravitino, and custom-built services,  can connect to Dremio Cloud through a single connector type.
- [Dremio's Built-in Open Catalog: Your Zero-Configuration Apache Iceberg Lakehouse](https://ingestthis.com/posts/2026/2026-03-01-connector-blogs-20-dremio-open-catalog): Every Dremio Cloud account starts with a built-in Open Catalog : a fully managed Apache Iceberg catalog with integrated storage. When you create a Dremio Cloud project, you immediately have a catalog where you can create namespaces (folders), tables, and views without connecting any external sources, configuring storage, or setting up credentials.
- [Connect Dremio Software to Dremio Cloud: Hybrid Federation Across Deployments](https://ingestthis.com/posts/2026/2026-03-01-connector-blogs-21-dremio-to-dremio): Dremio Cloud can connect to Dremio Software (self-managed) instances as a federated data source. This creates a hybrid deployment where Dremio Cloud serves as the primary query interface while accessing datasets managed by Dremio Software instances running in your own data centers or private cloud.
- [What Is Data Modeling? A Complete Guide](https://ingestthis.com/posts/2026/2026-02-19-data_modeling-01-what-is-data-modeling): Every database, data warehouse, and data lakehouse starts with the same question: how should this data be organized? Data modeling answers that question by creating a structured blueprint of your data : what it contains, how it relates, and what it means.
- [Conceptual, Logical, and Physical Data Models Explained](https://ingestthis.com/posts/2026/2026-02-19-data_modeling-02-types-of-data-models): Most data teams jump straight from a stakeholder request to creating database tables. They skip the planning steps that prevent misalignment, redundancy, and rework. The result: tables that make sense to the engineer who built them but confuse everyone else.
- [Star Schema vs. Snowflake Schema: When to Use Each](https://ingestthis.com/posts/2026/2026-02-19-data_modeling-03-star-schema-vs-snowflake): Both star schemas and snowflake schemas are dimensional models. They both organize data into fact tables (measurable events) and dimension tables (context about those events). The difference is how they structure the dimensions.
- [Data Modeling for the Lakehouse: What Changes](https://ingestthis.com/posts/2026/2026-02-19-data_modeling-04-data-modeling-lakehouse): Traditional data modeling assumed you controlled the database. You defined schemas up front, enforced foreign keys at write time, and optimized with indexes. The lakehouse changes every one of those assumptions.
- [Dimensional Modeling: Facts, Dimensions, and Grains](https://ingestthis.com/posts/2026/2026-02-19-data_modeling-05-dimensional-modeling): Dimensional modeling is the most widely used approach for organizing analytics data. Developed by Ralph Kimball, it structures data into two types of tables: facts (what happened) and dimensions (the context around what happened). The technique optimizes for query speed and business readability, not for storage efficiency or transactional integrity.
- [Slowly Changing Dimensions: Types 1-3 with Examples](https://ingestthis.com/posts/2026/2026-02-19-data_modeling-06-slowly-changing-dimensions): Dimensions change. A customer moves cities. A product gets reclassified. An employee changes departments. How your data model handles these changes determines whether your historical reports are accurate or misleading.
- [Data Modeling for Analytics: Optimize for Queries, Not Transactions](https://ingestthis.com/posts/2026/2026-02-19-data_modeling-07-data-modeling-for-analytics): The data model that runs your production application is almost never the right model for analytics. Transactional systems are designed for fast writes : inserting orders, updating inventory, processing payments. Analytics systems are designed for fast reads,  scanning millions of rows, aggregating across dimensions, filtering by date ranges.
- [Denormalization: When and Why to Flatten Your Data](https://ingestthis.com/posts/2026/2026-02-19-data_modeling-08-denormalization-when-why): Normalization is the first rule taught in database design. Eliminate redundancy. Store each fact once. Use foreign keys. It's the right rule for transactional systems. And it's the wrong rule for most analytics workloads.
- [Data Vault Modeling: Hubs, Links, and Satellites](https://ingestthis.com/posts/2026/2026-02-19-data_modeling-09-data-vault-modeling): Dimensional modeling works well when your source systems are stable and your business questions are predictable. But what happens when sources change constantly, new systems get added every quarter, and regulatory requirements demand a full audit trail of every attribute change?
- [Data Modeling Best Practices: 7 Mistakes to Avoid](https://ingestthis.com/posts/2026/2026-02-19-data_modeling-10-data-modeling-best-practices): A bad data model doesn't announce itself. It hides behind slow dashboards, conflicting numbers, confused analysts, and AI agents that generate wrong SQL. By the time someone identifies the model as the root cause, the team has already built dozens of reports on top of it.
- [How to Think Like a Data Engineer](https://ingestthis.com/posts/2026/2026-02-19-debp-01-think-like-data-engineer): The median lifespan of a popular data tool is about three years. The tool you master today may be deprecated or replaced by the time your next project ships. What doesn't change are the principles underneath: how data flows, how systems fail, how contracts between producers and consumers work, and how to decompose messy requirements into clean, maintainable pipelines.
- [How to Design Reliable Data Pipelines](https://ingestthis.com/posts/2026/2026-02-19-debp-02-design-data-pipelines): Most pipeline failures aren't caused by bad code. They're caused by no architecture. A script that reads from an API, transforms JSON, and writes to a database works fine on day one. On day ninety it fails at 3 AM because the API changed its response format, and the only way to recover is to rerun the entire pipeline from scratch : hoping that reprocessing three months of data doesn't create duplicates.
- [Data Quality Is a Pipeline Problem, Not a Dashboard Problem](https://ingestthis.com/posts/2026/2026-02-19-debp-03-data-quality-first): When an analyst finds null values in a revenue column, the typical response is to add a calculated field in the BI tool: `IF revenue IS NULL THEN 0`. That "fix" doesn't fix anything. It masks a problem at the source : and every downstream consumer has to independently discover and patch the same issue.
- [Idempotent Pipelines: Build Once, Run Safely Forever](https://ingestthis.com/posts/2026/2026-02-19-debp-04-idempotent-pipelines): A pipeline runs, processes 100,000 records, and loads them into the target table. Then it fails on a downstream step. The orchestrator retries the entire job. Now the table has 200,000 records : 100,000 of them duplicates. Revenue reports double. Dashboards misfire. Someone spends the next four hours manually deduplicating records and explaining to stakeholders why the numbers were wrong.
- [Schema Evolution Without Breaking Consumers](https://ingestthis.com/posts/2026/2026-02-19-debp-05-schema-evolution): A source team renames a column from `user_id` to `customer_id`. Twelve hours later, five dashboards show blank values, two ML pipelines fail, and the data engineering team spends the morning tracing a problem that could have been prevented with one rule: treat your schema like an API.
- [Batch vs. Streaming: Choose the Right Processing Model](https://ingestthis.com/posts/2026/2026-02-19-debp-06-batch-vs-streaming): "We need real-time data." This is one of the most expensive sentences in data engineering : because it's rarely true, and implementing it when it's not needed multiplies complexity, cost, and operational burden.
- [Partition and Organize Data for Performance](https://ingestthis.com/posts/2026/2026-02-19-debp-07-partition-and-organize): A table with 500 million rows takes 45 seconds to query. After partitioning it by date, the same query : filtering on a single day,  returns in 2 seconds. The SQL didn't change. The data didn't change. The only thing that changed was how the data was organized on disk.
- [Testing Data Pipelines: What to Validate and When](https://ingestthis.com/posts/2026/2026-02-19-debp-08-testing-data-pipelines): Ask an application developer how they test their code and they'll describe unit tests, integration tests, CI/CD pipelines, and coverage metrics. Ask a data engineer the same question and the most common answer is: "we check the dashboard."
- [Pipeline Observability: Know When Things Break](https://ingestthis.com/posts/2026/2026-02-19-debp-09-observability-monitoring): An analyst messages you on Slack: "The revenue numbers look wrong. Is the pipeline broken?" You check the orchestrator : all green. You check the target table,  data loaded this morning. You check the row count : looks normal. Forty-five minutes later, you discover that a source API returned empty responses for one region, and the pipeline happily loaded zero rows for that region without alerting anyone.
- [Data Engineering Best Practices: The Complete Checklist](https://ingestthis.com/posts/2026/2026-02-19-debp-10-de-best-practices-checklist): Best practices documents are easy to write and hard to use. They list principles without context, advice without prioritization, and rules without explaining when to break them. This one is different. It's a practical, tool-agnostic checklist organized by the categories that matter most : with each item tied to a specific outcome.
- [What Is a Semantic Layer? A Complete Guide](https://ingestthis.com/posts/2026/2026-02-19-semantic_layer_seo-01-what-is-a-semantic-layer): Ask three teams in your company how they calculate "revenue" and you'll get three answers. Sales counts bookings. Finance counts recognized revenue. Marketing counts pipeline value. All three call it "revenue." All three get different numbers. Nobody knows which one is right.
- [How to Build a Semantic Layer: A Step-by-Step Guide](https://ingestthis.com/posts/2026/2026-02-19-semantic_layer_seo-02-how-to-build-semantic-layer): Most teams start building a semantic layer the wrong way: they open their BI tool, create a few calculated fields, and call it done. Six months later, three dashboards define "churn" differently, nobody trusts the numbers, and the data team is debugging metric discrepancies instead of building new features.
- [Semantic Layer vs. Metrics Layer: What's the Difference?](https://ingestthis.com/posts/2026/2026-02-19-semantic_layer_seo-03-semantic-layer-vs-metrics-layer): Both terms appear in every modern data architecture diagram. They're used interchangeably in conference talks, Slack threads, and vendor marketing. And almost nobody defines them precisely.
- [Semantic Layer vs. Data Catalog: Complementary, Not Competing](https://ingestthis.com/posts/2026/2026-02-19-semantic_layer_seo-04-semantic-layer-vs-data-catalog): "We already have a data catalog, so we don't need a semantic layer." This is one of the most common misconceptions in modern data architecture. Catalogs and semantic layers both deal with metadata. They both improve data accessibility. But they solve fundamentally different problems.
- [Why Your AI Initiatives Fail Without a Semantic Layer](https://ingestthis.com/posts/2026/2026-02-19-semantic_layer_seo-05-why-ai-fails-without-semantic-layer): Your team builds an AI agent. It connects to your data warehouse. A product manager types "What was revenue last quarter?" and gets a number. The number is wrong. Nobody knows it's wrong until Finance runs the same query manually and gets a different result.
- [The Role of the Semantic Layer in Data Governance](https://ingestthis.com/posts/2026/2026-02-19-semantic_layer_seo-06-semantic-layer-data-governance): Most organizations have a data governance policy. It lives in a Confluence page. It defines who owns what data, what terms mean, and who should have access. And almost nobody follows it, because it's not enforced where queries actually run.
- [Data Virtualization and the Semantic Layer: Query Without Copying](https://ingestthis.com/posts/2026/2026-02-19-semantic_layer_seo-07-data-virtualization-semantic-layer): Every data pipeline you build to move data from one system to another costs you three things: time to build it, money to run it, and freshness you lose while waiting for the next sync. Most analytics architectures accept this cost as unavoidable. It isn't.
- [Headless BI: How a Universal Semantic Layer Replaces Tool-Specific Models](https://ingestthis.com/posts/2026/2026-02-19-semantic_layer_seo-08-headless-bi-semantic-layer): Your organization uses Tableau for executive dashboards, Power BI for operational reports, and Python notebooks for data science. Revenue is defined in Tableau's calculated field, Power BI's DAX measure, and a SQL query inside a Jupyter notebook. Three tools. Three definitions. None of them match.
- [How a Self-Documenting Semantic Layer Reduces Data Team Toil](https://ingestthis.com/posts/2026/2026-02-19-semantic_layer_seo-09-self-documenting-semantic-layer): Every data team knows documentation is important. And almost every data team has a backlog of undocumented tables, unlabeled columns, and outdated descriptions that nobody has time to fix. The problem isn't motivation. It's that manual documentation doesn't scale.
- [Semantic Layer Best Practices: 7 Mistakes to Avoid](https://ingestthis.com/posts/2026/2026-02-19-semantic_layer_seo-10-semantic-layer-best-practices): Semantic layers don't fail because the technology is wrong. They fail because of design decisions made in the first two weeks : choices that seem reasonable at the time and create compounding problems for months afterward.
- [A 2026 Introduction to Apache Iceberg](https://ingestthis.com/posts/2026/2026-02-2026-Intro-to-Apache-Iceberg): An updated introduction to Apache Iceberg
- [A Practical Guide to AI-Assisted Coding Tools](https://ingestthis.com/posts/2026/2026-01-A-Practical-Guide-to-AI-Assisted-Coding): An in-depth guide to understanding, choosing, and using AI-assisted coding tools effectively.
- [What Are Recursive Language Models?](https://ingestthis.com/posts/2026/2026-01-recursive-langauge-models): Recursive Language Models (RLMs) are a new class of language models that can call themselves to break down complex tasks into manageable parts. This article explores how RLMs work, the problems they solve, and why they represent a significant shift in language model capabilities.
- [2025 Year in Review Apache Iceberg, Polaris, Parquet, and Arrow](https://ingestthis.com/posts/2025/2025-12-2025-year-in-review-iceberg-arrow-polaris-parquet): A look back at key developments in Apache Iceberg, Polaris, Parquet, and Arrow in 2025.
- [dremioframe & iceberg - Pythonic interfaces for Dremio and Apache Iceberg](https://ingestthis.com/posts/2025/2025-12-dremioframe-and-iceframe): Discover DremioFrame and IceFrame, two new Python libraries that simplify working with Dremio and Apache Iceberg. Learn how these tools streamline data management and enhance productivity for modern data teams.
- [Introducing dremioframe - A Pythonic DataFrame Interface for Dremio](https://ingestthis.com/posts/2025/2025-11-introducing-dremioframe-dataframe-python-library): Discover dremioframe, a new Python library that offers a DataFrame-like experience for interacting with Dremio's data lakehouse platform. Learn how to leverage its intuitive API to streamline your data engineering workflows.
- [Comprehensive Hands-on Walk Through of Dremio Cloud Next Gen (Hands-on with Free Trial)](https://ingestthis.com/posts/2025/2025-11-Dremio-next-gen-tutorial): Walkthrough with the new trial of the Dremio Cloud Platform
- [2025-2026 Guide to Learning about Apache Iceberg, Data Lakehouse & Agentic AI](https://ingestthis.com/posts/2025/2025-10-2026-guide-to-learning-lakehouse-iceberg-agentic-ai): A curated guide to mastering Apache Iceberg, data lakehouse architectures, and the emerging field of Agentic AI for data professionals.
- [An Exploration of the Commercial Iceberg Catalog Ecosystem](https://ingestthis.com/posts/2025/2025-10-exploring-commerical-iceberg-catalogs): Dive into the world of commercial Iceberg catalogs and discover how they enhance data lakehouse architectures for modern data engineering.
- [Building a Universal Lakehouse Catalog - Beyond Iceberg Tables](https://ingestthis.com/posts/2025/2025-10-Building-Universal-Lakehouse-Catalog): Exploring paths to a universal lakehouse catalog that supports multiple data formats and engines, building on Apache Iceberg's success.
- [Intro to Apache Iceberg with Apache Polaris and Apache Spark](https://ingestthis.com/posts/2025/2025-10-Intro-to-Apache-Iceberg-with-Apache-Polaris-and-Apache-Spark): Learn how to leverage Apache Iceberg with Apache Polaris and Apache Spark to build scalable and efficient data lakehouses.
- [The State of Apache Iceberg v4 - October 2025 Edition](https://ingestthis.com/posts/2025/2025-10-apache-iceberg-v4): What's Coming in Apache Iceberg v4: A Deep Dive into the Future of Open Table Formats
- [The Ultimate Guide to Open Table Formats - Iceberg, Delta Lake, Hudi, Paimon, and DuckLake](https://ingestthis.com/posts/2025/2025-09-ultimate-guide-to-open-table-formats): Understanding Iceberg, Delta Lake, Hudi, Paimon, and DuckLake
- [The 2025 & 2026 Ultimate Guide to the Data Lakehouse and the Data Lakehouse Ecosystem](https://ingestthis.com/posts/2025/2025-09-2026-guide-to-data-lakehouses): What is the Data Lakehouse and the Data Lakehouse Ecosystem? This comprehensive guide covers everything you need to know about the Data Lakehouse architecture, open table formats like Apache Iceberg, Delta Lake, Apache Hudi, and Apache Paimon, and the modern data ecosystem that supports them.
- [The Endgame  – Building an Autonomous Optimization Pipeline for Apache Iceberg](https://ingestthis.com/posts/2025/2025-07-10-iceberg-autonomous-optimization-pipeline): Learn how to automate compaction, snapshot expiration, and layout optimization in Apache Iceberg using metadata-driven triggers and orchestration tools for a self-healing lakehouse.
- [Managing Large-Scale Optimizations  – Parallelism, Checkpointing, and Fail Recovery](https://ingestthis.com/posts/2025/2025-07-09-iceberg-large-scale-optimization): Learn how to scale Apache Iceberg table optimizations across large datasets using parallelism, checkpointing, and fail recovery to ensure reliability and performance.
- [Unlocking the Power of Agentic AI with Apache Iceberg and Dremio](https://ingestthis.com/posts/2025/2025-09-agentic-ai-dremio-apache-iceberg): Unlocking the Power of Agentic AI with Apache Iceberg and Dremio
- [Hidden Pitfalls  – Compaction and Partition Evolution in Apache Iceberg](https://ingestthis.com/posts/2025/2025-07-08-iceberg-partition-evolution-compaction): Partition evolution in Apache Iceberg is a powerful feature, but if not managed carefully, it can introduce fragmentation and impact compaction performance. Learn how to handle it effectively.
- [Using Iceberg Metadata Tables to Determine When Compaction Is Needed](https://ingestthis.com/posts/2025/2025-07-07-iceberg-metadata-triggered-compaction): Discover how to use Apache Iceberg's metadata tables to proactively detect small files, bloated manifests, and table fragmentation - so you can trigger compaction only when it's needed.
- [Designing the Ideal Cadence for Compaction and Snapshot Expiration](https://ingestthis.com/posts/2025/2025-07-06-iceberg-optimization-cadence): Learn how to design an effective schedule for compaction and snapshot expiration in Apache Iceberg to balance cost, performance, and data freshness.
- [Avoiding Metadata Bloat with Snapshot Expiration and Rewriting Manifests](https://ingestthis.com/posts/2025/2025-07-05-iceberg-metadata-bloat-cleanup): Learn how to prevent and clean up metadata bloat in Apache Iceberg by expiring snapshots and rewriting manifests for better performance and manageability.
- [Smarter Data Layout  – Sorting and Clustering Iceberg Tables](https://ingestthis.com/posts/2025/2025-07-04-iceberg-clustering-sorting-zorder): Improve query performance in Apache Iceberg by organizing your data layout with sorting and Z-order clustering. Learn how to reduce scan cost and improve filter effectiveness.
- [Optimizing Compaction for Streaming Workloads in Apache Iceberg](https://ingestthis.com/posts/2025/2025-07-03-iceberg-streaming-compaction): Learn how to design fast, incremental compaction strategies in Apache Iceberg to support high-throughput streaming pipelines without disrupting freshness or performance.
- [The Basics of Compaction  – Bin Packing Your Data for Efficiency](https://ingestthis.com/posts/2025/2025-07-02-iceberg-optimization-compaction-basics): Learn how standard compaction works in Apache Iceberg and why bin packing your data files is essential for maintaining query performance and cost efficiency.
- [The Cost of Neglect  – How Apache Iceberg Tables Degrade Without Optimization](https://ingestthis.com/posts/2025/2025-07-01-iceberg-optimization-degradation): Learn how Apache Iceberg tables can degrade over time without optimization and what issues this causes for performance, cost, and governance.
- [How to Discover or Organize Lakehouse & Apache Iceberg Meetups](https://ingestthis.com/posts/2025/2025-07-discovering-or-organizing-lakehouse-iceberg-meetups): Guide on How to Be Part of the Lakehouse Community
- [Introduction to Data Engineering Concepts | What is Data Engineering?](https://ingestthis.com/posts/2025/2025-05-intro-to-data-engineering-concepts-01): Introduction to the terms in data engineering
- [Introduction to Data Engineering Concepts | Understanding Data Sources and Ingestion](https://ingestthis.com/posts/2025/2025-05-intro-to-data-engineering-concepts-02): Introduction to the terms in data engineering
- [Introduction to Data Engineering Concepts | ETL vs ELT – Understanding Data Pipelines](https://ingestthis.com/posts/2025/2025-05-intro-to-data-engineering-concepts-03): Introduction to the terms in data engineering
- [Introduction to Data Engineering Concepts | Batch Processing Fundamentals](https://ingestthis.com/posts/2025/2025-05-intro-to-data-engineering-concepts-04): Introduction to the terms in data engineering
- [Introduction to Data Engineering Concepts | Streaming Data Fundamentals](https://ingestthis.com/posts/2025/2025-05-intro-to-data-engineering-concepts-05): Introduction to the terms in data engineering
- [Introduction to Data Engineering Concepts | Data Modeling Basics](https://ingestthis.com/posts/2025/2025-05-intro-to-data-engineering-concepts-06): Introduction to the terms in data engineering
- [Introduction to Data Engineering Concepts | Data Warehousing Fundamentals](https://ingestthis.com/posts/2025/2025-05-intro-to-data-engineering-concepts-07): Introduction to the terms in data engineering
- [Introduction to Data Engineering Concepts | Data Lakes Explained](https://ingestthis.com/posts/2025/2025-05-intro-to-data-engineering-concepts-08): Introduction to the terms in data engineering
- [Introduction to Data Engineering Concepts | Storage Formats and Compression](https://ingestthis.com/posts/2025/2025-05-intro-to-data-engineering-concepts-09): Introduction to the terms in data engineering
- [Introduction to Data Engineering Concepts | Data Quality and Validation](https://ingestthis.com/posts/2025/2025-05-intro-to-data-engineering-concepts-10): Introduction to the terms in data engineering
- [Introduction to Data Engineering Concepts | Metadata, Lineage, and Governance](https://ingestthis.com/posts/2025/2025-05-intro-to-data-engineering-concepts-11): Introduction to the terms in data engineering
- [Introduction to Data Engineering Concepts | Metadata, Lineage, and Governance](https://ingestthis.com/posts/2025/2025-05-intro-to-data-engineering-concepts-12): Introduction to the terms in data engineering
- [Introduction to Data Engineering Concepts | Building Scalable Pipelines](https://ingestthis.com/posts/2025/2025-05-intro-to-data-engineering-concepts-13): Introduction to the terms in data engineering
- [Introduction to Data Engineering Concepts | DevOps for Data Engineering](https://ingestthis.com/posts/2025/2025-05-intro-to-data-engineering-concepts-14): Introduction to the terms in data engineering
- [Introduction to Data Engineering Concepts | Cloud Data Platforms and the Modern Stack](https://ingestthis.com/posts/2025/2025-05-intro-to-data-engineering-concepts-15): Introduction to the terms in data engineering
- [Introduction to Data Engineering Concepts | Data Lakehouse Architecture Explained](https://ingestthis.com/posts/2025/2025-05-intro-to-data-engineering-concepts-16): Introduction to the terms in data engineering
- [Introduction to Data Engineering Concepts | Apache Iceberg, Arrow, and Polaris](https://ingestthis.com/posts/2025/2025-05-intro-to-data-engineering-concepts-17): Introduction to the terms in data engineering
- [Introduction to Data Engineering Concepts | The Power of Dremio in the Modern Lakehouse](https://ingestthis.com/posts/2025/2025-05-intro-to-data-engineering-concepts-18): Introduction to the terms in data engineering
- [A Journey from AI to LLMs and MCP - 10 - Sampling and Prompts in MCP  – Making Agent Workflows Smarter and Safer](https://ingestthis.com/posts/2025/2025-04-AI-Agents-MCP-10): Sampling and Prompts in MCP : Making Agent Workflows Smarter and Safer
- [A Journey from AI to LLMs and MCP - 9 - Tools in MCP  – Giving LLMs the Power to Act](https://ingestthis.com/posts/2025/2025-04-AI-Agents-MCP-09): Tools in MCP : Giving LLMs the Power to Act
- [A Journey from AI to LLMs and MCP - 8 - Resources in MCP  – Serving Relevant Data Securely to LLMs](https://ingestthis.com/posts/2025/2025-04-AI-Agents-MCP-08): Resources in MCP : Serving Relevant Data Securely to LLMs
- [A Journey from AI to LLMs and MCP - 7 - Under the Hood  – The Architecture of MCP and Its Core Components](https://ingestthis.com/posts/2025/2025-04-AI-Agents-MCP-07): Under the Hood : The Architecture of MCP and Its Core Components
- [Journey from AI to LLMs and MCP - 6 - Enter the Model Context Protocol (MCP)  – The Interoperability Layer for AI Agents](https://ingestthis.com/posts/2025/2025-04-AI-Agents-MCP-06): Enter the Model Context Protocol (MCP) : The Interoperability Layer for AI Agents
- [A Journey from AI to LLMs and MCP - 5 - AI Agent Frameworks  – Benefits and Limitations](https://ingestthis.com/posts/2025/2025-04-AI-Agents-MCP-05): AI Agent Frameworks : Benefits and Limitations
- [A Journey from AI to LLMs and MCP - 4 - What Are AI Agents  – And Why They're the Future of LLM Applications](https://ingestthis.com/posts/2025/2025-04-AI-Agents-MCP-04): What Are AI Agents : And Why They're the Future of LLM Applications
- [A Journey from AI to LLMs and MCP - 3 - Boosting LLM Performance  – Fine-Tuning, Prompt Engineering, and RAG](https://ingestthis.com/posts/2025/2025-04-AI-Agents-MCP-03): Boosting LLM Performance : Fine-Tuning, Prompt Engineering, and RAG
- [A Journey from AI to LLMs and MCP - 2 - How LLMs Work  – Embeddings, Vectors, and Context Windows](https://ingestthis.com/posts/2025/2025-04-AI-Agents-MCP-02): How LLMs Work : Embeddings, Vectors, and Context Windows
- [A Journey from AI to LLMs and MCP - 1 - What Is AI and How It Evolved Into LLMs](https://ingestthis.com/posts/2025/2025-04-AI-Agents-MCP-01): What Is AI and How It Evolved Into LLMs
- [Introduction to Data Engineering Concepts | Scheduling and Workflow Orchestration](https://ingestthis.com/posts/2025/2025-04-basics-of-making-mcp-server): The Basics of Building a Basic MCP Server
- [Using Helm with Kubernetes - A Guide to Helm Charts and Their Implementation](https://ingestthis.com/posts/2025/2025-02-understanding-helm-charts): A Guide on when to use Helm Charts for Kubernetes Deployment
- [Crash Course on Developing AI Applications with LangChain](https://ingestthis.com/posts/2025/2025-02-crash-course-on-langchain): A guide on building AI applications with LangChain, a framework for developing AI applications powered by Large Language Models (LLMs).
- [The Data Lakehouse - The Benefits and Enhancing Implementation](https://ingestthis.com/posts/2025/2025-01-the-lakehouse-benefits-and-enhancing): Understanding the value of a lakehouse and how to get that value faster
- [2025 Comprehensive Guide to Apache Iceberg](https://ingestthis.com/posts/2025/2025-01-2025-comprehensive-apache-iceberg-guide): What is Apache Iceberg, How it Works, and Why it Matters!
- [When to use Apache Xtable or Delta Lake Uniform for Data Lakehouse Interoperability](https://ingestthis.com/posts/2025/2025-01-XTable-or-Uniform): A Guide on when to use Apache Xtable or Delta Lake Uniform for Data Lakehouse Interoperability
- [RAG Isn’t a Modeling Problem. It’s a Data Engineering Problem.](https://ingestthis.com/posts/2026/2026-01-rag-isnt-the-problem): Why retrieval-augmented generation systems fail in enterprises - and what to do about it.
- [Building Pangolin - My Holiday Break, an AI IDE, and a Lakehouse Catalog for the Curious](https://ingestthis.com/posts/2026/2026-01-the-story-of-pangolin-catalog): A personal story of how I built Pangolin Catalog over a holiday break using an AI-powered IDE.
- [2025 Guide to Architecting an Iceberg Lakehouse](https://ingestthis.com/posts/2024/2024-12-2025-guide-architecting-an-iceberg-lakehouse): A Comprehensive Guide to Building a Data Lakehouse with Apache Iceberg
- [10 Future Apache Iceberg Developments to Look forward to in 2025](https://ingestthis.com/posts/2024/2024-11-10-Iceberg-developments): What is cool about Apache Iceberg's Future
- [Deep Dive into Dremio's File-based Auto Ingestion into Apache Iceberg Tables](https://ingestthis.com/posts/2024/2024-11-deep-dive-auto-ingest-dremio-iceberg): Auto ingesting data from JSON, CSV, and Parquet files into Apache Iceberg Tables
- [Intro to SQL using Apache Iceberg and Dremio](https://ingestthis.com/posts/2024/2024-11-intro-to-sql-with-dremio): Intro to SQL using Apache Iceberg and Dremio
- [Dremio, Apache Iceberg and their role in AI-Ready Data](https://ingestthis.com/posts/2024/2024-11-Dremio-and-AI-Ready-Data): The Role of Dremio and Apache Iceberg in AI-Ready Data
- [Introduction to Cargo and cargo.toml](https://ingestthis.com/posts/2024/2024-11-rust-cargo-toml): Getting Started with Cargo and cargo.toml
- [Leveraging Python's Pattern Matching and Comprehensions for Data Analytics](https://ingestthis.com/posts/2024/2024-11-Python-Analytics-Pattern-Matching): Using Features like Pattern Matching and Comprehensions for Data Analytics
- [Hands-on with Apache Iceberg & Dremio on Your Laptop within 10 Minutes](https://ingestthis.com/posts/2024/2024-10-hands-on-with-iceberg-dremio-laptop): How to get hands-on with Apache Iceberg
- [Data Modeling - Entities and Events](https://ingestthis.com/posts/2024/2024-10-data-modeling-entities-and-events): How to Model Events and Entities
- [All About  Parquet Part 01 - An Introduction](https://ingestthis.com/posts/2024/2024-10-all-about-parquet-part-01): All about the Apache Parquet File Format
- [All About  Parquet Part 02 - Parquet's Columnar Storage Model](https://ingestthis.com/posts/2024/2024-10-all-about-parquet-part-02): All about the Apache Parquet File Format
- [All About  Parquet Part 03 - Parquet File Structure | Pages, Row Groups, and Columns](https://ingestthis.com/posts/2024/2024-10-all-about-parquet-part-03): All about the Apache Parquet File Format
- [All About  Parquet Part 04 - Schema Evolution in Parquet](https://ingestthis.com/posts/2024/2024-10-all-about-parquet-part-04): All about the Apache Parquet File Format
- [All About  Parquet Part 05 - Compression Techniques in Parquet](https://ingestthis.com/posts/2024/2024-10-all-about-parquet-part-05): All about the Apache Parquet File Format
- [All About  Parquet Part 06 - Encoding in Parquet | Optimizing for Storage](https://ingestthis.com/posts/2024/2024-10-all-about-parquet-part-06): All about the Apache Parquet File Format
- [All About  Parquet Part 07 - Metadata in Parquet | Improving Data Efficiency](https://ingestthis.com/posts/2024/2024-10-all-about-parquet-part-07): All about the Apache Parquet File Format
- [All About  Parquet Part 08 - Reading and Writing Parquet Files in Python](https://ingestthis.com/posts/2024/2024-10-all-about-parquet-part-08): All about the Apache Parquet File Format
- [All About  Parquet Part 09 - Parquet in Data Lake Architectures](https://ingestthis.com/posts/2024/2024-10-all-about-parquet-part-09): All about the Apache Parquet File Format
- [All About  Parquet Part 10 - Performance Tuning and Best Practices with Parquet](https://ingestthis.com/posts/2024/2024-10-all-about-parquet-part-10): All about the Apache Parquet File Format
- [Orchestrating Airflow DAGs with GitHub Actions - A Lightweight Approach to Data Curation Across Spark, Dremio, and Snowflake](https://ingestthis.com/posts/2024/2024-10-github-actions-dbt-airflow-data): Advanced GitHub Actions for Data Engineering
- [A Deep Dive Into GitHub Actions From Software Development to Data Engineering](https://ingestthis.com/posts/2024/2024-10-intro-to-github-actions): Learning about GitHub Actions
- [A Guide to dbt Macros - Purpose, Benefits, and Usage](https://ingestthis.com/posts/2024/2024-10-dbt-macros): Learning about dbt  Macros
- [Data Lakehouse Roundup 1 - News and Insights on the Lakehouse](https://ingestthis.com/posts/2024/2024-10-Data-Lakhouse-Roundup-1): What's Going on in the Data Lakehouse Space
- [Getting Started with Data Analytics Using PyArrow in Python](https://ingestthis.com/posts/2024/2024-10-pyarrow-and-python): Learning to work with PyArrow to run analytics
- [What is Three-Tier Data (Bronze, Silver, Gold) and How Dremio Simplifies It](https://ingestthis.com/posts/2024/2024-10-bronze-silver-gold-data): Process Data from Raw to Clean Aggregated Data
- [A Brief Guide to the Governance of Apache Iceberg Tables](https://ingestthis.com/posts/2024/2024-10-governing-apache-iceberg-tables): Controlling Access to your Apache Iceberg Tables
- [Exploring Data Operations with PySpark, Pandas, DuckDB, Polars, and DataFusion in a Python Notebook](https://ingestthis.com/posts/2024/2024-10-learning-data-with-python): Learning to work with Python to ingest and query data
- [Ultimate Directory of Apache Iceberg Resources](https://ingestthis.com/posts/2024/2024-10-ultimate-directory-of-Apache-Iceberg-Resources): Apache Iceberg Education, Tutorials and more!
- [Change Data Capture (CDC) when there is no CDC](https://ingestthis.com/posts/2024/2024-10-CDC-when-there-is-no-CDC): Handling Synching Changing Data Across Systems
- [Virtualization + Lakehouse + Mesh = Data At Scale](https://ingestthis.com/posts/2024/2024-09-decentralization-centralization-data-at-scale): Combining Centralization and Decentralization for Data at Scale
- [Deep Dive into Data Apps with Streamlit](https://ingestthis.com/posts/2024/2024-09-Deep-Dive-into-data-apps-with-streamlit): Building a Deploying Data Apps Easily
- [A Deep Dive into Docker Compose](https://ingestthis.com/posts/2024/2024-09-a-deep-dive-into-docker-compose): A Comprehensive Guide to Docker Compose
- [Hands-on with Apache Iceberg on Your Laptop - Deep Dive with Apache Spark, Nessie, Minio, Dremio, Polars and Seaborn](https://ingestthis.com/posts/2024/2024-09-hands-on-iceberg-dremio-minio-nessie): The Evolving Data Lakehouse World
- [Why Data Analysts, Engineers, Architects and Scientists Should Care about Dremio and Apache Iceberg](https://ingestthis.com/posts/2024/2024-09-Why-Dremio-Iceberg-Matters): The Evolving Data Lakehouse World
- [5 Trends in the Data Lakehouse Space](https://ingestthis.com/posts/2024/2024-09-five-trends-in-data-lakehouse): The Evolving Data Lakehouse World
- [Using the alexmerced/datanotebook Docker Image](https://ingestthis.com/posts/2024/2024-08-using-the-alexmerced-datanotebook-image): Setting up a quick and easy data environment for data science and analytics
- [Understanding Apache Iceberg Delete Files](https://ingestthis.com/posts/2024/2024-08-Understanding-Apache-Iceberg-Delete-Files): Continuing the Understand Apache Iceberg series, this article delves into the Manifest, a critical component of Apache Iceberg's architecture.
- [Understanding the Apache Iceberg Manifest](https://ingestthis.com/posts/2024/2024-08-Understanding-Apache-Iceberg-Manifest): Continuing the Understand Apache Iceberg series, this article delves into the Manifest, a critical component of Apache Iceberg's architecture.
- [Understanding the Apache Iceberg Manifest List (Snapshot)](https://ingestthis.com/posts/2024/2024-08-Understanding-Apache-Iceberg-Manifest-List): Continuing the Understand Apache Iceberg series, this article delves into the Manifest List, a critical component of Apache Iceberg's architecture.
- [What Apache Iceberg REST Catalog is and isn't](https://ingestthis.com/posts/2024/2024-08-what-apache-iceberg-rest-catalog-is-and-isnt): Understanding Iceberg Catalog Interoperability
- [ACID Guarantees and Apache Iceberg - Turning Any Storage into a Data Warehouse](https://ingestthis.com/posts/2024/2024-08-acid-guarantees-and-apache-iceberg): What are ACID Guarantees? WHy do they matter?
- [Data Lakehouse 101 - The Who, What and Why of Data Lakehouses](https://ingestthis.com/posts/2024/2024-08-data-lakehouses-101): The Who, What and Why of Data Lakehouses
- [Understanding the Polaris Iceberg Catalog and Its Architecture](https://ingestthis.com/posts/2024/2024-07-Understanding-Polaris-Apache-Iceberg-Catalog): Learn about the new open source Iceberg Catalog in Town
- [Apache Iceberg Reliability](https://ingestthis.com/posts/2024/07-Apache-Iceberg-Reliability): Why Apache Iceberg Works
- [Upcoming Data Talks from Alex Merced (And how to follow)](https://ingestthis.com/posts/2024/07-Upcoming-Events): Come see me talk live at these events
- [Databases Deconstructed - The Value of Data Lakehouses and Table Formats](https://ingestthis.com/posts/2024/07-databases-deconstructed-data-lakehouses-and-table-formats): Building up the Data Lakehouse
- [Video Course - Basics of Lakehouse Engineering - Apache Iceberg, Nessie, Dremio](https://ingestthis.com/posts/2024/06-apache-iceberg-intro-to-lakehouse-engineering-course): Introductory Course to Data Engineering for Apache Iceberg Lakehouses
- [Partitioning with Apache Iceberg - A Deep Dive](https://ingestthis.com/posts/2024/05-Apache-Iceberg-Partitioning-Deep-Dive): Benefits of Apache Iceberg Partition Evolution and Hidden Partitioning
- [3 Reasons Data Engineers Should Embrace Apache Iceberg](https://ingestthis.com/posts/2024/05-3-reasons-data-engineers-should-embrace-apache-iceberg): Benefits of Apache Iceberg
- [Running SQL on your Excel Files From Your Laptop with Dremio](https://ingestthis.com/posts/2024/05-SQL-on-Excel-Files): How to run SQL on your Excel files easily
- [Understanding the Future of Apache Iceberg Catalogs](https://ingestthis.com/posts/2024/04-The-Future-Of-Apache-Iceberg-Catalogs): Java, Rest and the expanding open lakehouse ecosystem
- [A Deep Intro to Apache Iceberg and Resources for Learning More](https://ingestthis.com/posts/2024/04-deep-dive-into-apache-iceberg): Learning about Apache Iceberg
- [End-to-End Basic Data Engineering Tutorial (Spark, Dremio, Superset)](https://ingestthis.com/posts/2024/04-End-to-End-Data-Engineering-Tutorial-Ingest-Dashboards): Ingesting Data and Building BI Dashboards
- [5 Open Source Data Projects You Should Be Following](https://ingestthis.com/posts/2024/03-5-open-source-data-project-to-follow): Apache Iceberg, Apache Arrow, Nessie, Ibis, Substrait
- [5 Reasons Dremio is the Ideal Apache Iceberg Lakehouse Platform](https://ingestthis.com/posts/2024/03-5-reasons-apache-iceberg-dremio): Understanding how catalogs work and which one to choose
- [The Apache Iceberg Lakehouse - The Great Data Equalizer](https://ingestthis.com/posts/2024/03-Apache-Iceberg-The-Great-Equalizer): Disrupting the Snowflake/Databricks status quo
- [10 Reasons to Make Apache Iceberg and Dremio Part of Your Data Lakehouse Strategy](https://ingestthis.com/posts/2024/03-10-reasons-apache-iceberg-dremio-lakehouse): Understanding how catalogs work and which one to choose
- [A deep dive into the concept and world of Apache Iceberg Catalogs](https://ingestthis.com/posts/2024/03-deep-dive-into-apache-iceberg-catalogs): Understanding how catalogs work and which one to choose
- [The Role of Ontologies in Data Management](https://ingestthis.com/posts/2024/02-What-are-ontologies): What are ontologies and why they matter
- [Introduction to ANSI SQL - Understanding the Syntax and Concepts](https://ingestthis.com/posts/2024/02-introduction-to-ansi-sql): Learning the Standard SQL Syntax
- [What is the Data Lakehouse and the Role of Apache Iceberg, Nessie and Dremio?](https://ingestthis.com/posts/2024/02-Data-Lakehouse-Apache-Iceberg-Nessie-Dremio): Understanding the Value of the Data Lakehouse
- [Partitioning Practices in Apache Hive and Apache Iceberg](https://ingestthis.com/posts/2024/02-Partitioning-Apache-Hive-Apache-Iceberg): Deep Dive in Data Lake Table Partitioning
- [Columnar vs. Row-based Data Structures in OLTP and OLAP Systems](https://ingestthis.com/posts/2024/02-Row-based-OLTP-Column-based-OLAP.md): The Fundamentals of Data Systems
- [Introduction to Data Vault Modeling](https://ingestthis.com/posts/2024/02-Data-Vault-Modeling): Understanding the Data Vault Style of Data Warehouse Modeling
- [Table Format FUD - Thinking Through the Table Format Conversion (Apache Iceberg, Apache Hudi, Delta Lake)](https://ingestthis.com/posts/2024/02-Table_Format_FUD_Apache_Iceberg_Delta_Lake_Apache_Hudi): Understanding how to choose a table format
- [Embracing the Future of Data Management - Why Choose Lakehouse, Iceberg, and Dremio?](https://ingestthis.com/posts/2024/01-Why-Choose-Data-Lakehouse-Apache-Iceberg-Dremio): The Future of Data Platforms
- [Open Lakehouse Engineering/Apache Iceberg Lakehouse Engineering - A Directory of Resources](https://ingestthis.com/posts/2024/01-Open_Lakehouse_Engineering): Resources for learning how to Engineer an Open Data Lakehouse
- [Nessie -  An Alternative to Hive & JDBC for Self-Managed Apache Iceberg Catalogs](https://ingestthis.com/posts/2024/01-Nessie-Alternative-to-Hive-JDBC-for-Apache-Iceberg.md): Nessie is the only open-source catalog implementation specifically for Apache Iceberg.
- [Apache Iceberg, Git-Like Catalog Versioning and Data Lakehouse Management - Pillars of a Robust Data Lakehouse Platform](https://ingestthis.com/posts/2024/01-Apache_Iceberg_Catalog_Versioning_Data_Lakehouse_Management): This is where the combined power of Dremio’s Lakehouse Management features and Project Nessie's catalog-level versioning comes into play.
- [No Code - Convert XLS/CSV files into Parquet with Dremio](https://ingestthis.com/posts/2023/12-Converting-xls-csv-into-parquet-no-code-with-dremio): Convert XLS/CSV Files without having to write python
- [An In-Depth Overview of Open Lakehouse Tech: Apache Iceberg & Nessie](https://ingestthis.com/posts/2023/10-an-in-depth-overview-of-iceberg-nessie): Article about An In-Depth Overview of Open Lakehouse Tech: Apache Iceberg & Nessie
- [Overview of the Open Lakehouse: Why Dremio?](https://ingestthis.com/posts/2023/10-why-dremio-iceberg-data-lakehouse): Article about Overview of the Open Lakehouse: Why Dremio?
- [An Approach to Architecting a Lower Cost, Fast and Self-Service Data Lakehouse](https://ingestthis.com/posts/2023/09-architecting-low-cost-fast-self-service-lakehouse): Article about An Approach to Architecting a Lower Cost, Fast and Self-Service Data Lakehouse
- [Creating a Local Data Lakehouse using Spark/Minio/Dremio/Nessie](https://ingestthis.com/posts/2023/08-creating-a-iceberg-nessie-data-lakehouse-on-your-laptop copy): Article about Creating a Local Data Lakehouse using Spark/Minio/Dremio/Nessie
- [Project Nessie: A Look in the Depths](https://ingestthis.com/posts/2023/07-project-nessie-in-depth): Article about Project Nessie: A Look in the Depths
- [Overview of File Encryption Algorithms for Everyone](https://ingestthis.com/posts/2023/06-file-encryption-overview): Article about Overview of File Encryption Algorithms for Everyone
- [Parquet File Compression for Everyone (zstd, brotli, lz4, gzip, snappy)](https://ingestthis.com/posts/2023/06-parquet-file-compression): Article about Parquet File Compression for Everyone (zstd, brotli, lz4, gzip, snappy)
- [Dremio and Modern Data Architecture: Data Lakes, Data Lakehouses and Data Mesh](https://ingestthis.com/posts/2023/06-Dremio-Modern-Data-Architecture): Article about Dremio and Modern Data Architecture: Data Lakes, Data Lakehouses and Data Mesh
- [What is Nessie and Why as a Data Engineer or Architect you should care?](https://ingestthis.com/posts/2023/05-Why-Project-Nessie-Matters-For-Data-Engineers): Article about What is Nessie and Why as a Data Engineer or Architect you should care?
- [Resources for Learning more about Catalog level versioning with Project Nessie & Dremio Arctic (Rollbacks, Branching, Tagging and Multi-Table Txns)](https://ingestthis.com/posts/2023/05-Dremio-Arctic-Project-Nessie-Resources.md): Article about Resources for Learning more about Catalog level versioning with Project Nessie & Dremio Arctic (Rollbacks, Branching, Tagging and Multi-Table Txns)
- [Overview of the Data Lakehouse, Dremio and Apache Iceberg](https://ingestthis.com/posts/2023/04-overview-of-the-data-lakehouse): Article about Overview of the Data Lakehouse, Dremio and Apache Iceberg
- [Why Dremio is a must for Apache Iceberg Data Lakehouses](https://ingestthis.com/posts/2023/11-Why-Dremio-is-a-must-for-apache-iceberg): Why is Dremio so useful for Apache Iceberg data lakehouses
- [Understanding Spark Configurations with Apache Iceberg](https://ingestthis.com/posts/2022/11-configuring-apache-spark-for-apache-iceberg): How to configure Spark for using Apache Iceberg
- [5 Reasons Your Data Lakehouse should Embrace Dremio Cloud](https://ingestthis.com/posts/2022/08-five-reasons-you-should-embrace-dremio-cloud): How your data lakehouse can expand what's possible with Dremio Cloud.
- [Brief Hands on Intro to Apache Iceberg](https://ingestthis.com/posts/2022/07-Hands-on-intro-to-apache-iceberg-in-five-minutes): Engineer a Data Lakehouse with Apache Iceberg
- [Introduction to The World of Data - (OLTP, OLAP, Data Warehouses, Data Lakes and more)](https://ingestthis.com/posts/2022/06-intro-to-the-world-of-data): An accessible high-level guide for data and non-data professionals
- [A 2022 Introduction to SQL](https://ingestthis.com/posts/2022/05-introduction-to-SQL): Learning Structured Query Language
- [2022 MongooseJS Cheatsheet](https://ingestthis.com/posts/2022/04-2022-mongoosejs-cheatsheet): Details on working with MongooseJS
- [Web Storage API Part 1 - LocalStorage and SessionStorage](https://ingestthis.com/posts/2022/03-localstorage): Article about Web Storage API Part 1 - LocalStorage and SessionStorage
- [What is Batch and Streaming Data? (Data 101)](https://ingestthis.com/posts/2022/03-batch-streaming): Article about What is Batch and Streaming Data? (Data 101)

## Events
- [Agentic Lakehouse Events](https://luma.com/agenticlakehouse): global meetups and webinars on agentic analytics
- [Data Lakehouse Hub Events](https://luma.com/DataLakehouseHub): global lakehouse meetups, linkups and webinars

## Community
- [Data Lakehouse Hub Slack](https://join.slack.com/t/thedatalakehousehub/shared_invite/zt-274yc8sza-mI2zhCW8LGkOh1uxuf8T5Q): practitioner community for lakehouse architecture
- [Data Events Slack](https://join.slack.com/t/data-events/shared_invite/zt-38vgrooy9-U9ral_gr3NAz_Siih1QwmQ): announcements for data conferences and meetups
- [Data & Tech Slack](https://join.slack.com/t/datatechcommunity/shared_invite/zt-12xrk4qmd-y~6jUFFd7kdaLhgLURKwoA): broader data and technology community
- [r/datalakehouseandai](https://www.reddit.com/r/datalakehouseandai/): subreddit for data lakehouse and AI discussion
- [Data Lakehouse Hub on LinkedIn](https://www.linkedin.com/company/data-lakehouse-hub/): company page for the Data Lakehouse Hub
- [Alex Merced Tech on YouTube](https://www.youtube.com/@AlexMercedCoder): software development and engineering channel
- [Alex Merced Data & AI on YouTube](https://www.youtube.com/@alexmerceddata): data lakehouse and AI channel