Apache Iceberg

What Is Apache Iceberg?

Iceberg employs a layered approach to data management, distinguishing between the metadata management layer and the data storage layer.

Data Layer: This layer contains the actual data files, such as Parquet and ORC files, which are the physical carriers for storing data.

Metadata Layer: The metadata layer in Iceberg is multi-tiered and is responsible for storing the structure of the table and the indexing information of the data files.

Catalog Layer: This layer stores pointers to the locations of the metadata files, serving as an index for the metadata. Iceberg provides several catalog implementations, such as:

• HadoopCatalog: Directly uses Hadoop filesystem directories to manage table information.
• HiveCatalog: Stores table metadata in the Hive Metastore, facilitating integration with the existing Hive ecosystem.

The metadata management is further divided into three key components:

• Metadata File: Stores the current version's metadata, including all snapshot information.
• Snapshot: Represents a specific operation's snapshot, with each commit generating a new snapshot containing multiple manifests detailing the addresses of the generated data files.
• Manifest: Lists the data files associated with a snapshot, providing a comprehensive view of the data's organization and facilitating efficient data retrieval and modification.

At its core, Iceberg aims to track all changes to a table over time through snapshots, which represent complete collections of table data files at any given moment. Each update operation generates a new snapshot, ensuring data consistency and facilitating historical data analysis and incremental reads.  

Key Aspects of Iceberg Table Format

Hidden Partition

• Hidden Partitions: Iceberg allows users to organize data partitions based on timestamps with different granularities (e.g., year, month, day, or hour). Partition information is transparent to users.
• Syntax Usage: Use syntax like CREATE TABLE catalog.MyTable (..., ts TIMESTAMP) PARTITIONED BY days(ts) to create partitions based on daily timestamps. New data is automatically placed in the corresponding partitions.

Schema Evolution

• Flexible Schema Evolution: Iceberg provides flexible schema evolution capabilities, supporting DDL operations like adding, deleting, renaming, and updating columns.
• Metadata Updates: Schema changes only update metadata without rewriting or moving data files. Each change is recorded in the metadata, ensuring historical schemas can be tracked and queried.
• Unique Column IDs: Each column in the table is tracked with a unique column ID, ensuring the correctness of read and write operations on files.

Partition Evolution

• Partition Strategy Modification: When modifying partition strategies, existing data files retain their original partition strategy, while new data applies the new strategy.
• Historical Partition Metadata: Metadata records all historical partition schema information.
• Execution Plans: Iceberg can generate different execution plans to handle data involving different partition strategies.

MVCC (Multi-Version Concurrency Control)

• MVCC Mechanism: Ensures that data writes do not interfere with read operations. Each read accesses the latest data snapshot.
• Delete and Overwrite Operations: Modify the status of Parquet files in the Manifest file rather than directly deleting data files.

Consistency

• Concurrent Data Writes: Supports concurrent data writes using an optimistic locking mechanism to handle write conflicts.
• Snapshot-Based Writes: Write operations are based on the same data snapshot. If there are no conflicts, both operations can succeed; if there is a conflict, only one operation succeeds.

Row-Level Update: COW & MOR

• COW (Copy on Write): Supported in V1 table format. Updates data by copying the original data and applying changes to the copy.
• MOR (Merge on Read): Supported in V2 table format. Introduces position delete files and equality delete files to perform logical deletions, reflecting the latest state during data reads.
  • Position Delete: Records the position of data rows to be deleted, suitable for scenarios where data changes infrequently.
  • Equality Delete: Saves the value of the data to be deleted and executes the deletion through comparison during reads.

Exploring the Benefits of Apache Iceberg

Where and When to Use Apache Iceberg

As one of the core components of a universal data lake solution, Iceberg is primarily suitable for the following scenarios:

Real-time Data Import and Querying

Data flows in real-time from upstream to the Iceberg data lake, where it can be immediately queried. For example, in logging scenarios, Iceberg or Spark streaming jobs are initiated to import log data into Iceberg tables in real-time. This data can then be queried in real-time using Hive, Spark, Iceberg, or Presto. Moreover, Iceberg's support for ACID transactions ensures the isolation of data inflow and querying, preventing dirty data.

Data Deletion or Updating

Most data warehouses struggle to efficiently perform row-level data deletions or updates, typically requiring offline jobs to extract the entire table's raw data, modify it, and then write it back to the original table. Iceberg, however, narrows the scope of changes from the table level to the file level, allowing for localized changes to execute business logic for data modifications or deletions. Within the Iceberg data lake, you can directly execute commands like DELETE FROM test_table WHERE id > 10 to make changes to the data in the table.

Data Quality Control

With the validation function of the Iceberg Schema, abnormal data is excluded during data import, or further processing is performed on abnormal data.

Data Schema Changes

The schema of the data is not fixed and can change; Iceberg supports making changes to the table structure using Spark SQL's DDL statements. When changing the table structure in Iceberg, it is not necessary to re-export all historical data according to the new schema, which greatly speeds up the schema change process. Additionally, Iceberg's support for ACID transactions effectively isolates schema changes from affecting existing read tasks, allowing you to access consistently accurate data.

Real-time Machine Learning

In machine learning scenarios, a significant amount of time is often spent processing data, such as cleaning, transforming, and extracting features, as well as dealing with both historical and real-time data. Iceberg simplifies this workflow, turning the entire data processing process into a complete, reliable real-time stream. Operations like data cleaning, transformation, and feature engineering are all node actions on the stream, eliminating the need to handle historical and real-time data separately. Furthermore, Iceberg also supports a native Python SDK, making it very user-friendly for developers of machine learning algorithms.

• Apache Iceberg: Stands out for its broad engine compatibility (including Spark, Flink, Presto, and Trino), making it highly versatile. Iceberg's design is optimized for cloud storage efficiency and performance at petabyte scale, addressing issues like metadata scalability and file listing overheads inherent in cloud object stores.
• Apache Hudi: Provides similar capabilities for record-level inserts, updates, and deletes. Hudi is designed to offer faster incremental processing and comes with built-in support for change data capture (CDC), stream processing, and data privacy use cases.
• Delta Lake: Like Iceberg, Delta Lake offers ACID transactions, schema evolution, and time travel. However, it is closely integrated with Spark and Databricks, potentially limiting its flexibility with other processing engines.