◐ Shell
clean mode source ↗

fix: Pre-create S3A event log dir before SparkContext init by abhijeet-dhumal · Pull Request #6317 · feast-dev/feast

abhijeet-dhumal

devin-ai-integration[bot]

@ntkathole ntkathole changed the title fix(spark): pre-create S3A event log dir before SparkContext init fix: Pre-create S3A event log dir before SparkContext init

Apr 22, 2026

R-behera

ntkathole

ntkathole

ntkathole

ntkathole

…prevent silent materialize failure

Spark's EventLogFileWriter.requireLogBaseDirAsDirectory() is called
inside SparkContext.__init__. When spark.eventLog.dir points to an S3A
path that doesn't exist yet (S3 has no real directories), SparkContext
fails to initialise — silently from Feast's perspective because
_materialize_one() catches the exception and returns an ERROR job.

Add _ensure_s3a_event_log_dir() to utils.py: before building the
SparkSession, check if the S3A prefix exists and write a zero-byte
placeholder if it doesn't. Uses boto3 (already a Feast dep via S3 offline
store). Non-fatal: logs a warning and lets Spark surface its own error
if the write fails.

Signed-off-by: abhijeet-dhumal <abhijeetdhumal652@gmail.com>
… config, add session token support

Signed-off-by: abhijeet-dhumal <abhijeetdhumal652@gmail.com>
…linting

Signed-off-by: abhijeet-dhumal <abhijeetdhumal652@gmail.com>
Signed-off-by: abhijeet-dhumal <abhijeetdhumal652@gmail.com>
Signed-off-by: abhijeet-dhumal <abhijeetdhumal652@gmail.com>