◐ Shell
reader mode source ↗
Skip to content

feat: Support table format: Iceberg, Delta, and Hudi#5650

Merged
HaoXuAI merged 17 commits into
masterfrom
support-table-format
Nov 6, 2025
Merged

feat: Support table format: Iceberg, Delta, and Hudi#5650
HaoXuAI merged 17 commits into
masterfrom
support-table-format

Conversation

@HaoXuAI

@HaoXuAI HaoXuAI commented Oct 5, 2025 •
edited
Loading

Copy link
Copy Markdown
Collaborator

What this PR does / why we need it:

examples:

  from feast.table_format import IcebergFormat
  from feast.infra.offline_stores.contrib.spark_offline_store.spark_source import SparkSource

  # Create Iceberg format with snapshot option
  iceberg_format = IcebergFormat()
  iceberg_format.set_property("snapshot-id", "your_snapshot_id")

  spark_source = SparkSource(
      name="iceberg_features",
      path="my_catalog.my_database.my_table",  # Catalog table name
      table_format=iceberg_format
  )

  Generated Spark code:
  spark.read.format("iceberg").option("snapshot-id", "your_snapshot_id").load("my_catalog.my_database.my_table")

  ✅ Delta Example:

  from feast.table_format import DeltaFormat

  delta_format = DeltaFormat()
  delta_format.set_property("versionAsOf", "1")

  spark_source = SparkSource(
      name="delta_features",
      path="s3://bucket/delta-table",
      table_format=delta_format
  )

  Generated Spark code:
  spark.read.format("delta").option("versionAsOf", "1").load("s3://bucket/delta-table")

  ✅ Hudi Example:

  from feast.table_format import HudiFormat

  hudi_format = HudiFormat()
  hudi_format.set_property("hoodie.datasource.query.type", "snapshot")
  hudi_format.set_property("as.of.instant", "20231010120000")

  spark_source = SparkSource(
      name="hudi_features",
      path="s3://bucket/hudi-table",
      table_format=hudi_format
  )

  Generated Spark code:
  spark.read.format("hudi")\
    .option("hoodie.datasource.query.type", "snapshot")\
    .option("as.of.instant", "20231010120000")\
    .load("s3://bucket/hudi-table")

Which issue(s) this PR fixes:

Misc

HaoXuAI added 3 commits October 5, 2025 14:42
@HaoXuAI
add support for table format such as Iceberg, Delta, Hudi etc. …
8ae5ed5 
Signed-off-by: HaoXuAI <sduxuhao@gmail.com>
@HaoXuAI
linting …
456b97f 
Signed-off-by: HaoXuAI <sduxuhao@gmail.com>
@HaoXuAI
linting …
afa1444 
Signed-off-by: HaoXuAI <sduxuhao@gmail.com>
@HaoXuAI HaoXuAI requested a review from a team as a code owner October 5, 2025 21:47
@HaoXuAI HaoXuAI changed the title feat: Support table format Oct 5, 2025
HaoXuAI added 2 commits October 5, 2025 15:08
@HaoXuAI
add tests …
7a34eaf 
Signed-off-by: HaoXuAI <sduxuhao@gmail.com>
@HaoXuAI
fix tests …
99f85b1 
Signed-off-by: HaoXuAI <sduxuhao@gmail.com>
@tokoko

tokoko commented Oct 6, 2025

Copy link
Copy Markdown
Collaborator

+1 on the inclusion of all 3 formats. Still I think we might be able to better design data-source side such that data source definitions don't tie the sources to specific offline stores. For example right now FileSource already supports specifying delta as a format... I don't like that design either because delta and especially iceberg might require more configuration that plain files, but the upside in that design was that Source class wasn't tied to a single offline store and it is actually usable from both dask and duckdb offline stores. Hiding formats behind SparkSource will mean that when a user stores a data source instance in the registry only spark offline store can be used to retrieve the data.

I think we can have best of both worlds if we instead go for adding all these formats as separate independent data sources (DeltaSource, IcebergSource, HudiSource) and then simply teach spark engine (and in the future others) how to read it.

@franciscojavierarceo

franciscojavierarceo commented Oct 6, 2025

Copy link
Copy Markdown
Member

I think we can have best of both worlds if we instead go for adding all these formats as separate independent data sources (DeltaSource, IcebergSource, HudiSource) and then simply teach spark engine (and in the future others) how to read it.

+1

@HaoXuAI

HaoXuAI commented Oct 6, 2025

Copy link
Copy Markdown
Collaborator Author

@franciscojavierarceo @tokoko consolidation with FilleFormat and new data sources could break the backward compatibility, so I want to do it pace by pace.

@franciscojavierarceo

franciscojavierarceo commented Oct 7, 2025

Copy link
Copy Markdown
Member

That makes sense

@tokoko

tokoko commented Oct 7, 2025

Copy link
Copy Markdown
Collaborator

@HaoXuAI Why would new data sources break backwards compatibility though?

@HaoXuAI

HaoXuAI commented Oct 9, 2025

Copy link
Copy Markdown
Collaborator Author

@HaoXuAI Why would new data sources break backwards compatibility though?

There will be some proto changes, no 100% sure if there will be API changes exposed to users but I think might be the case

HaoXuAI added 3 commits October 8, 2025 23:01
@HaoXuAI
fix tests …
eb15424 
Signed-off-by: HaoXuAI <sduxuhao@gmail.com>
@HaoXuAI
Merge branch 'master' into support-table-format
05e3a65
@HaoXuAI
linting …
b11c083 
Signed-off-by: HaoXuAI <sduxuhao@gmail.com>
@HaoXuAI

HaoXuAI commented Oct 10, 2025

Copy link
Copy Markdown
Collaborator Author

@franciscojavierarceo @ntkathole mind take a look

franciscojavierarceo
franciscojavierarceo requested changes Oct 10, 2025
View reviewed changes

@franciscojavierarceo franciscojavierarceo left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hide comment

@HaoXuAI i don't see use actually using or testing Spark Table, Iceberg, or Hudi format's outside of our definitions, can you add that?

Can you also add documentation that these formats are now supported?

Otherwise lgtm.

@HaoXuAI

HaoXuAI commented Oct 14, 2025

Copy link
Copy Markdown
Collaborator Author

@HaoXuAI i don't see use actually using or testing Spark Table, Iceberg, or Hudi format's outside of our definitions, can you add that?

Can you also add documentation that these formats are now supported?

Otherwise lgtm.

@HaoXuAI HaoXuAI closed this Oct 14, 2025
@HaoXuAI HaoXuAI reopened this Oct 14, 2025
@HaoXuAI

HaoXuAI commented Oct 14, 2025

Copy link
Copy Markdown
Collaborator Author

@HaoXuAI i don't see use actually using or testing Spark Table, Iceberg, or Hudi format's outside of our definitions, can you add that?
Can you also add documentation that these formats are now supported?
Otherwise lgtm.

Gonna update to add the TableFormat proto in the next PR, after that I'll add the docs. And I think the test will need to be changed as well.

HaoXuAI and others added 7 commits October 29, 2025 21:36
@HaoXuAI
Merge branch 'master' into support-table-format
429958a
@HaoXuAI
add tableformat proto …
e19467d 
Signed-off-by: hao-xu5 <hxu44@apple.com>
@HaoXuAI
update …
931fdbd 
Signed-off-by: hao-xu5 <hxu44@apple.com>
@HaoXuAI
update doc …
4919fba 
Signed-off-by: hao-xu5 <hxu44@apple.com>
@HaoXuAI
fix linting …
63bba87 
Signed-off-by: hao-xu5 <hxu44@apple.com>
@HaoXuAI
Merge branch 'master' into support-table-format
b9bd58e
@HaoXuAI
fix test …
c014d6e 
Signed-off-by: hao-xu5 <hxu44@apple.com>
@HaoXuAI

HaoXuAI commented Nov 6, 2025

Copy link
Copy Markdown
Collaborator Author

@franciscojavierarceo mind take another look?

franciscojavierarceo
franciscojavierarceo approved these changes Nov 6, 2025
View reviewed changes

@franciscojavierarceo franciscojavierarceo left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hide comment

Lgtm

Hide details View details @HaoXuAI HaoXuAI merged commit 2915ad1 into master Nov 6, 2025
19 of 20 checks passed
franciscojavierarceo pushed a commit that referenced this pull request Nov 13, 2025
@feast-ci-bot
chore(release): release 0.57.0 …
cc39768 
# [0.57.0](v0.56.0...v0.57.0) (2025-11-13)

### Bug Fixes

* Improve trino to feast type mapping with (real,varchar,timestamp,decimal) ([#5691](#5691)) ([f855ad2](f855ad2))
* Materialize API - ODFV views not looked-up (thinks views non existant)  - crashes materialize ([#5716](#5716)) ([1b050b3](1b050b3))
* Support historical feature retrieval with start_date/end_date in RemoteOfflineStore ([#5703](#5703)) ([ad32756](ad32756))
* Thread safe Clickhouse offline store ([#5710](#5710)) ([5f446ed](5f446ed))

### Features

* Add annotations to cronjob CRDs ([#5701](#5701)) ([be6e6c2](be6e6c2))
* Add batch commit mode for MySQL OnlineStore ([#5699](#5699)) ([3cfe4eb](3cfe4eb))
* Add possibility to materialize only latest values, to increase performance ([#5713](#5713)) ([8d77b72](8d77b72))
* Support table format: Iceberg, Delta, and Hudi ([#5650](#5650)) ([2915ad1](2915ad1))
@franciscojavierarceo franciscojavierarceo mentioned this pull request Jan 14, 2026
Open
Copilot AI mentioned this pull request Jan 14, 2026
Draft
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@franciscojavierarceo franciscojavierarceo franciscojavierarceo approved these changes

@ntkathole ntkathole Awaiting requested review from ntkathole

Assignees

No one assigned

Labels

ok-to-test

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@HaoXuAI @tokoko @franciscojavierarceo