Bytewax materializer materializes entire dataset on every pod

james-crabtree-sp

Expected Behavior

Each pod should materialize a separate batch of data

Current Behavior

Every pod redundantly materializes the entire dataset, causing a massive overspend on processing time and database writes

Steps to reproduce

Observed with snowflake offline store and dynamo db online store. Materialize a feature with enough records for the snowflake s3 integration to output multiple files. View the logs of each pod and add up the total records materialized. Observe that every pod has materialized the entire dataset

Specifications

Version: 0.31
Platform: fedora linux
Subsystem: bytewax batch_engine

Possible Solution

Use JOB_COMPLETION_INDEX in each pods dataflow to select a single file to process