◐ Shell
reader mode source ↗
Skip to content

feat(spark): Add compute-on-read support for BatchFeatureView in get_…#6357

Merged
ntkathole merged 7 commits into
feast-dev:masterfrom
SIDDHESH1564:feature/spark-bfv-compute-on-read
May 3, 2026
Merged

feat(spark): Add compute-on-read support for BatchFeatureView in get_…#6357
ntkathole merged 7 commits into
feast-dev:masterfrom
SIDDHESH1564:feature/spark-bfv-compute-on-read

Conversation

@SIDDHESH1564

@SIDDHESH1564 SIDDHESH1564 commented May 1, 2026

Copy link
Copy Markdown
Contributor

What this PR does / why we need it:

When using @batch_feature_view with TransformationMode.PYTHON in the Spark offline store, get_historical_features() fails with UNRESOLVED_COLUMN errors. This occurs because the PIT join SQL reads directly from the raw batch_source and expects transformed feature columns (e.g., aggregated outputs) to already exist in the source data. However, BFV transformations are only executed during feast materialize, not during offline retrieval.

This PR introduces compute-on-read support for BatchFeatureView in SparkOfflineStore. Before generating the PIT join SQL, BFVs with a UDF are detected and their transformations are applied:

  1. Read the raw source into a Spark DataFrame
  2. Invoke the BFV's feature_transformation.udf() (same function used during materialization)
  3. Register the transformed DataFrame as a Spark temporary view
  4. Replace the table_subquery in the query context with the temp view name

This enables reuse of BFV definitions during offline training without requiring pre-materialization or external ETL pipelines. The entire pipeline remains fully distributed in Spark.

Which issue(s) this PR fixes:

Fixes #6345

Checks

  • I've made sure the tests are passing.
  • My commits are signed off (git commit -s)
  • My PR title follows conventional commits format

Testing Strategy

  • Unit tests
  • Integration tests
  • Manual tests
  • Testing is not required for this change

Added 7 unit tests covering:

  • BFV with UDF → table_subquery replaced with temp view
  • UDF invoked with source DataFrame
  • Transformed DataFrame registered as temp view
  • Plain FeatureView passes through unchanged
  • BFV without UDF passes through unchanged
  • Mixed BFV + plain FeatureView scenarios
  • All non-transformation context fields preserved

Misc

Changes:

  • sdk/python/feast/infra/offline_stores/contrib/spark_offline_store/spark.py:
    • Added BatchFeatureView import
    • Added _apply_bfv_transformations() helper function
    • Integrated call into get_historical_features() between query context construction and PIT join SQL generation
  • sdk/python/tests/unit/infra/offline_stores/contrib/spark_offline_store/test_spark_bfv_compute_on_read.py (new):
    • 7 unit tests for compute-on-read behavior

@SIDDHESH1564 SIDDHESH1564 requested a review from a team as a code owner May 1, 2026 19:12
@franciscojavierarceo

Copy link
Copy Markdown
Member

@copilot can you apply make format-python on this PR?

@franciscojavierarceo franciscojavierarceo changed the title feat(spark): add compute-on-read support for BatchFeatureView in get_… May 1, 2026
…historical_features

Signed-off-by: Siddhesh Khairnar <khairnarsiddhesh4057@gmail.com>
@SIDDHESH1564 SIDDHESH1564 force-pushed the feature/spark-bfv-compute-on-read branch from fcdf0e6 to 11d69be Compare May 2, 2026 04:28
…n logic

Signed-off-by: Siddhesh Khairnar <khairnarsiddhesh4057@gmail.com>
@SIDDHESH1564 SIDDHESH1564 requested a review from ntkathole May 2, 2026 16:52
…V source resolution

Signed-off-by: Siddhesh Khairnar <khairnarsiddhesh4057@gmail.com>
Hide details View details @ntkathole ntkathole merged commit 630d9f8 into feast-dev:master May 3, 2026
25 of 26 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

BFV Compute-on-Read for get_historical_features() in SparkOfflineStore

3 participants