◐ Shell
reader mode source ↗
Skip to content

feat: Make online_write_batch_size configurable in MaterializationConfig#6268

Merged
ntkathole merged 2 commits into
feast-dev:masterfrom
cutoutsy:feat/configurable-online-write-batch-size
Apr 14, 2026
Merged

feat: Make online_write_batch_size configurable in MaterializationConfig#6268
ntkathole merged 2 commits into
feast-dev:masterfrom
cutoutsy:feat/configurable-online-write-batch-size

Conversation

@cutoutsy

@cutoutsy cutoutsy commented Apr 13, 2026

Copy link
Copy Markdown
Contributor

What this PR does / why we need it:

Add a configurable online_write_batch_size option to MaterializationConfig for controlling batch size during online store writes.

Changes:

  • Add online_write_batch_size: Optional[int] = None to MaterializationConfig
  • When None (default): write all rows in a single batch (backward compatible)
  • When set to a positive integer: write rows in batches of that size

Usage in feature_store.yaml:

materialization:
  online_write_batch_size: 10000

This allows users to control memory usage during materialization for large datasets.

Which issue(s) this PR fixes:

Fixes #6160


Open with Devin

@cutoutsy cutoutsy requested a review from a team as a code owner April 13, 2026 03:56
devin-ai-integration[bot]

This comment was marked as resolved.

@cutoutsy cutoutsy force-pushed the feat/configurable-online-write-batch-size branch 2 times, most recently from de8fe04 to 654b5e4 Compare April 13, 2026 06:15
@ntkathole

Copy link
Copy Markdown
Member

@cutoutsy can we please also have a test case that exercises the batched write path ?

@cutoutsy cutoutsy force-pushed the feat/configurable-online-write-batch-size branch from d653115 to bff1499 Compare April 14, 2026 02:27
@cutoutsy

Copy link
Copy Markdown
Contributor Author

@cutoutsy can we please also have a test case that exercises the batched write path ?

Added two unit tests for the batched write functionality.

@ntkathole ntkathole left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hide comment

looks good, Thanks @cutoutsy

@ntkathole ntkathole force-pushed the feat/configurable-online-write-batch-size branch from bff1499 to 8b4ea7a Compare April 14, 2026 10:06
Hide details View details @ntkathole ntkathole merged commit d41becf into feast-dev:master Apr 14, 2026
22 of 26 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Materializes all rows as Python proto objects — high memory usage at scale

2 participants