◐ Shell
clean mode source ↗

feat: Upgrading python version to 3.11, adding support for 3.11 as well. by lokeshrangineni · Pull Request #4159 · feast-dev/feast

feast-dev

12 changes: 6 additions & 6 deletions .github/fork_workflows/fork_pr_integration_tests_aws.yml

Original file line number Diff line number Diff line change
Expand Up @@ -33,21 +33,21 @@ jobs:
id: image-tag
run: echo "::set-output name=DOCKER_IMAGE_TAG::`git rev-parse HEAD`"
- name: Cache Public ECR Image
id: lambda_python_3_9
id: lambda_python_3_11
uses: actions/cache@v2
with:
path: ~/cache
key: lambda_python_3_9
key: lambda_python_3_11
- name: Handle Cache Miss (pull public ECR image & save it to tar file)
if: steps.cache-primes.outputs.cache-hit != 'true'
run: |
mkdir -p ~/cache
docker pull public.ecr.aws/lambda/python:3.9
docker save public.ecr.aws/lambda/python:3.9 -o ~/cache/lambda_python_3_9.tar
docker pull public.ecr.aws/lambda/python:3.11
docker save public.ecr.aws/lambda/python:3.11 -o ~/cache/lambda_python_3_11.tar
- name: Handle Cache Hit (load docker image from tar file)
if: steps.cache-primes.outputs.cache-hit == 'true'
run: |
docker load -i ~/cache/lambda_python_3_9.tar
docker load -i ~/cache/lambda_python_3_11.tar
- name: Build and push
env:
ECR_REGISTRY: ${{ steps.login-ecr.outputs.registry }}
Expand All @@ -67,7 +67,7 @@ jobs:
strategy:
fail-fast: false
matrix:
python-version: [ "3.9" ]
python-version: [ "3.11" ]
os: [ ubuntu-latest ]
env:
OS: ${{ matrix.os }}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ jobs:
strategy:
fail-fast: false
matrix:
python-version: [ "3.9" ]
python-version: [ "3.11" ]
os: [ ubuntu-latest ]
env:
OS: ${{ matrix.os }}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ jobs:
strategy:
fail-fast: false
matrix:
python-version: [ "3.9" ]
python-version: [ "3.11" ]
os: [ ubuntu-latest ]
env:
OS: ${{ matrix.os }}
Expand Down

6 changes: 3 additions & 3 deletions .github/workflows/build_wheels.yml

Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,7 @@ jobs:
- name: Setup Python
uses: actions/setup-python@v3
with:
python-version: "3.9"
python-version: "3.11"
architecture: x64
- name: Setup Node
uses: actions/setup-node@v3
Expand All @@ -86,7 +86,7 @@ jobs:
id: setup-python
uses: actions/setup-python@v3
with:
python-version: "3.10"
python-version: "3.11"
architecture: x64
- name: Setup Node
uses: actions/setup-node@v3
Expand Down Expand Up @@ -137,7 +137,7 @@ jobs:
strategy:
matrix:
os: [ubuntu-latest, macos-13 ]
python-version: ["3.9", "3.10"]
python-version: ["3.9", "3.10", "3.11"]
from-source: [ True, False ]
env:
# this script is for testing servers
Expand Down

6 changes: 3 additions & 3 deletions .github/workflows/java_master_only.yml

Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ jobs:
uses: actions/setup-python@v3
id: setup-python
with:
python-version: "3.9"
python-version: "3.11"
architecture: x64
- name: Authenticate to Google Cloud
uses: 'google-github-actions/auth@v1'
Expand Down Expand Up @@ -95,7 +95,7 @@ jobs:
if: github.repository == 'feast-dev/feast'
runs-on: ubuntu-latest
env:
PYTHON: 3.9
PYTHON: 3.11
steps:
- uses: actions/checkout@v4
with:
Expand All @@ -110,7 +110,7 @@ jobs:
uses: actions/setup-python@v3
id: setup-python
with:
python-version: 3.9
python-version: 3.11
architecture: x64
- name: Get pip cache dir
id: pip-cache
Expand Down

8 changes: 4 additions & 4 deletions .github/workflows/java_pr.yml

Original file line number Diff line number Diff line change
Expand Up @@ -88,7 +88,7 @@ jobs:
uses: actions/setup-python@v3
id: setup-python
with:
python-version: "3.9"
python-version: "3.11"
architecture: x64
- name: Authenticate to Google Cloud
uses: 'google-github-actions/auth@v1'
Expand All @@ -111,7 +111,7 @@ jobs:
runs-on: ubuntu-latest
needs: unit-test-java
env:
PYTHON: 3.9
PYTHON: 3.11
steps:
- uses: actions/checkout@v4
with:
Expand All @@ -128,7 +128,7 @@ jobs:
architecture: x64
- uses: actions/setup-python@v3
with:
python-version: '3.9'
python-version: '3.11'
architecture: 'x64'
- uses: actions/cache@v2
with:
Expand Down Expand Up @@ -158,7 +158,7 @@ jobs:
uses: actions/setup-python@v3
id: setup-python
with:
python-version: 3.9
python-version: 3.11
architecture: x64
- name: Get pip cache dir
id: pip-cache
Expand Down

4 changes: 2 additions & 2 deletions .github/workflows/linter.yml

Original file line number Diff line number Diff line change
Expand Up @@ -6,14 +6,14 @@ jobs:
lint-python:
runs-on: [ubuntu-latest]
env:
PYTHON: 3.9
PYTHON: 3.11
steps:
- uses: actions/checkout@v4
- name: Setup Python
id: setup-python
uses: actions/setup-python@v5
with:
python-version: "3.9"
python-version: "3.11"
architecture: x64
- name: Upgrade pip version
run: |
Expand Down

12 changes: 6 additions & 6 deletions .github/workflows/master_only.yml

Original file line number Diff line number Diff line change
Expand Up @@ -30,21 +30,21 @@ jobs:
id: image-tag
run: echo "::set-output name=DOCKER_IMAGE_TAG::`git rev-parse HEAD`"
- name: Cache Public ECR Image
id: lambda_python_3_9
id: lambda_python_3_11
uses: actions/cache@v2
with:
path: ~/cache
key: lambda_python_3_9
key: lambda_python_3_11
- name: Handle Cache Miss (pull public ECR image & save it to tar file)
if: steps.cache-primes.outputs.cache-hit != 'true'
run: |
mkdir -p ~/cache
docker pull public.ecr.aws/lambda/python:3.9
docker save public.ecr.aws/lambda/python:3.9 -o ~/cache/lambda_python_3_9.tar
docker pull public.ecr.aws/lambda/python:3.11
docker save public.ecr.aws/lambda/python:3.11 -o ~/cache/lambda_python_3_11.tar
- name: Handle Cache Hit (load docker image from tar file)
if: steps.cache-primes.outputs.cache-hit == 'true'
run: |
docker load -i ~/cache/lambda_python_3_9.tar
docker load -i ~/cache/lambda_python_3_11.tar
- name: Build and push
env:
ECR_REGISTRY: ${{ steps.login-ecr.outputs.registry }}
Expand All @@ -65,7 +65,7 @@ jobs:
strategy:
fail-fast: false
matrix:
python-version: ["3.9", "3.10" ]
python-version: ["3.9", "3.10", "3.11"]
os: [ ubuntu-latest ]
env:
OS: ${{ matrix.os }}
Expand Down

18 changes: 9 additions & 9 deletions .github/workflows/nightly-ci.yml

Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ jobs:
uses: actions/setup-python@v5
id: setup-python
with:
python-version: "3.9"
python-version: "3.11"
architecture: x64
- name: Set up AWS SDK
uses: aws-actions/configure-aws-credentials@v1
Expand Down Expand Up @@ -89,21 +89,21 @@ jobs:
id: image-tag
run: echo "::set-output name=DOCKER_IMAGE_TAG::`git rev-parse HEAD`"
- name: Cache Public ECR Image
id: lambda_python_3_9
id: lambda_python_3_11
uses: actions/cache@v4
with:
path: ~/cache
key: lambda_python_3_9
key: lambda_python_3_11
- name: Handle Cache Miss (pull public ECR image & save it to tar file)
if: steps.lambda_python_3_9.outputs.cache-hit != 'true'
if: steps.lambda_python_3_11.outputs.cache-hit != 'true'
run: |
mkdir -p ~/cache
docker pull public.ecr.aws/lambda/python:3.9
docker save public.ecr.aws/lambda/python:3.9 -o ~/cache/lambda_python_3_9.tar
docker pull public.ecr.aws/lambda/python:3.11
docker save public.ecr.aws/lambda/python:3.11 -o ~/cache/lambda_python_3_11.tar
- name: Handle Cache Hit (load docker image from tar file)
if: steps.lambda_python_3_9.outputs.cache-hit == 'true'
if: steps.lambda_python_3_11.outputs.cache-hit == 'true'
run: |
docker load -i ~/cache/lambda_python_3_9.tar
docker load -i ~/cache/lambda_python_3_11.tar
- name: Build and push
env:
ECR_REGISTRY: ${{ steps.login-ecr.outputs.registry }}
Expand All @@ -124,7 +124,7 @@ jobs:
strategy:
fail-fast: false
matrix:
python-version: [ "3.9" ]
python-version: [ "3.11" ]
os: [ ubuntu-latest ]
env:
OS: ${{ matrix.os }}
Expand Down

12 changes: 6 additions & 6 deletions .github/workflows/pr_integration_tests.yml

Original file line number Diff line number Diff line change
Expand Up @@ -47,21 +47,21 @@ jobs:
id: image-tag
run: echo "::set-output name=DOCKER_IMAGE_TAG::`git rev-parse HEAD`"
- name: Cache Public ECR Image
id: lambda_python_3_9
id: lambda_python_3_11
uses: actions/cache@v2
with:
path: ~/cache
key: lambda_python_3_9
key: lambda_python_3_11
- name: Handle Cache Miss (pull public ECR image & save it to tar file)
if: steps.cache-primes.outputs.cache-hit != 'true'
run: |
mkdir -p ~/cache
docker pull public.ecr.aws/lambda/python:3.9
docker save public.ecr.aws/lambda/python:3.9 -o ~/cache/lambda_python_3_9.tar
docker pull public.ecr.aws/lambda/python:3.11
docker save public.ecr.aws/lambda/python:3.11 -o ~/cache/lambda_python_3_11.tar
- name: Handle Cache Hit (load docker image from tar file)
if: steps.cache-primes.outputs.cache-hit == 'true'
run: |
docker load -i ~/cache/lambda_python_3_9.tar
docker load -i ~/cache/lambda_python_3_11.tar
- name: Build and push
env:
ECR_REGISTRY: ${{ steps.login-ecr.outputs.registry }}
Expand All @@ -86,7 +86,7 @@ jobs:
strategy:
fail-fast: false
matrix:
python-version: [ "3.10" ]
python-version: [ "3.9", "3.10", "3.11" ]
os: [ ubuntu-latest ]
env:
OS: ${{ matrix.os }}
Expand Down

2 changes: 1 addition & 1 deletion .github/workflows/pr_local_integration_tests.yml

Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ jobs:
strategy:
fail-fast: false
matrix:
python-version: [ "3.10" ]
python-version: [ "3.11" ]
os: [ ubuntu-latest ]
env:
OS: ${{ matrix.os }}
Expand Down

2 changes: 1 addition & 1 deletion .github/workflows/publish.yml

Original file line number Diff line number Diff line change
Expand Up @@ -160,7 +160,7 @@ jobs:
architecture: x64
- uses: actions/setup-python@v3
with:
python-version: '3.9'
python-version: '3.11'
architecture: 'x64'
- uses: actions/cache@v2
with:
Expand Down

2 changes: 1 addition & 1 deletion .github/workflows/unit_tests.yml

Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ jobs:
strategy:
fail-fast: false
matrix:
python-version: [ "3.9", "3.10" ]
python-version: [ "3.9", "3.10", "3.11"]
os: [ ubuntu-latest, macos-13 ]
exclude:
- os: macos-13
Expand Down

2 changes: 1 addition & 1 deletion Makefile

Original file line number Diff line number Diff line change
Expand Up @@ -357,7 +357,7 @@ kill-trino-locally:
cd ${ROOT_DIR}; docker stop trino

install-protoc-dependencies:
pip install --ignore-installed protobuf==4.23.4 "grpcio-tools>=1.56.2,<2" mypy-protobuf==3.1.0
pip install --ignore-installed protobuf==4.24.0 "grpcio-tools>=1.56.2,<2" mypy-protobuf==3.1.0

install-feast-ci-locally:
pip install -e ".[ci]"
Expand Down

3 changes: 1 addition & 2 deletions docs/SUMMARY.md

Original file line number Diff line number Diff line change
Expand Up @@ -106,8 +106,7 @@
* [Google Cloud Platform](reference/providers/google-cloud-platform.md)
* [Amazon Web Services](reference/providers/amazon-web-services.md)
* [Azure](reference/providers/azure.md)
* [Batch Materialization Engines](reference/batch-materialization/README.md)
* [Bytewax](reference/batch-materialization/bytewax.md)
* [Batch Materialization Engines](reference/batch-materialization/README.md)
* [Snowflake](reference/batch-materialization/snowflake.md)
* [AWS Lambda (alpha)](reference/batch-materialization/lambda.md)
* [Spark (contrib)](reference/batch-materialization/spark.md)
Expand Down

21 changes: 1 addition & 20 deletions docs/how-to-guides/running-feast-in-production.md

Original file line number Diff line number Diff line change
Expand Up @@ -57,28 +57,9 @@ To keep your online store up to date, you need to run a job that loads feature d
Out of the box, Feast's materialization process uses an in-process materialization engine. This engine loads all the data being materialized into memory from the offline store, and writes it into the online store.

This approach may not scale to large amounts of data, which users of Feast may be dealing with in production.
In this case, we recommend using one of the more [scalable materialization engines](./scaling-feast.md#scaling-materialization), such as the [Bytewax Materialization Engine](../reference/batch-materialization/bytewax.md), or the [Snowflake Materialization Engine](../reference/batch-materialization/snowflake.md).
In this case, we recommend using one of the more [scalable materialization engines](./scaling-feast.md#scaling-materialization), such as [Snowflake Materialization Engine](../reference/batch-materialization/snowflake.md).
Users may also need to [write a custom materialization engine](../how-to-guides/customizing-feast/creating-a-custom-materialization-engine.md) to work on their existing infrastructure.

The Bytewax materialization engine can run materialization on an existing Kubernetes cluster. An example configuration of this in a `feature_store.yaml` is as follows:

```yaml
batch_engine:
type: bytewax
namespace: bytewax
image: bytewax/bytewax-feast:latest
env:
- name: AWS_ACCESS_KEY_ID
valueFrom:
secretKeyRef:
name: aws-credentials
key: aws-access-key-id
- name: AWS_SECRET_ACCESS_KEY
valueFrom:
secretKeyRef:
name: aws-credentials
key: aws-secret-access-key
```

### 2.2 Scheduled materialization with Airflow

Expand Down