◐ Shell
reader mode source ↗
Skip to content
Merged
Show file tree
Changes from all commits
File filter
Conversations
Jump to
Diff view
Apply and reload
Show whitespace
Diff view
Apply and reload
12 changes: 6 additions & 6 deletions .github/fork_workflows/fork_pr_integration_tests_aws.yml
Original file line number Diff line number Diff line change
@@ -33,21 +33,21 @@ jobs:
id: image-tag
run: echo "::set-output name=DOCKER_IMAGE_TAG::`git rev-parse HEAD`"
- name: Cache Public ECR Image
id: lambda_python_3_9
uses: actions/cache@v2
with:
path: ~/cache
key: lambda_python_3_9
- name: Handle Cache Miss (pull public ECR image & save it to tar file)
if: steps.cache-primes.outputs.cache-hit != 'true'
run: |
mkdir -p ~/cache
docker pull public.ecr.aws/lambda/python:3.9
docker save public.ecr.aws/lambda/python:3.9 -o ~/cache/lambda_python_3_9.tar
- name: Handle Cache Hit (load docker image from tar file)
if: steps.cache-primes.outputs.cache-hit == 'true'
run: |
docker load -i ~/cache/lambda_python_3_9.tar
- name: Build and push
env:
ECR_REGISTRY: ${{ steps.login-ecr.outputs.registry }}
Expand All @@ -67,7 +67,7 @@ jobs:
strategy:
fail-fast: false
matrix:
python-version: [ "3.9" ]
os: [ ubuntu-latest ]
env:
OS: ${{ matrix.os }}
Expand Down
2 changes: 1 addition & 1 deletion .github/fork_workflows/fork_pr_integration_tests_gcp.yml
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ jobs:
strategy:
fail-fast: false
matrix:
python-version: [ "3.9" ]
os: [ ubuntu-latest ]
env:
OS: ${{ matrix.os }}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ jobs:
strategy:
fail-fast: false
matrix:
python-version: [ "3.9" ]
os: [ ubuntu-latest ]
env:
OS: ${{ matrix.os }}
Expand Down
6 changes: 3 additions & 3 deletions .github/workflows/build_wheels.yml
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,7 @@ jobs:
- name: Setup Python
uses: actions/setup-python@v3
with:
python-version: "3.9"
architecture: x64
- name: Setup Node
uses: actions/setup-node@v3
Expand All @@ -86,7 +86,7 @@ jobs:
id: setup-python
uses: actions/setup-python@v3
with:
python-version: "3.10"
architecture: x64
- name: Setup Node
uses: actions/setup-node@v3
Expand Up @@ -137,7 +137,7 @@ jobs:
strategy:
matrix:
os: [ubuntu-latest, macos-13 ]
python-version: ["3.9", "3.10"]
from-source: [ True, False ]
env:
# this script is for testing servers
8 changes: 4 additions & 4 deletions .github/workflows/java_pr.yml
Original file line number Diff line number Diff line change
Expand Up @@ -88,7 +88,7 @@ jobs:
uses: actions/setup-python@v3
id: setup-python
with:
python-version: "3.9"
architecture: x64
- name: Authenticate to Google Cloud
uses: 'google-github-actions/auth@v1'
Expand All @@ -111,7 +111,7 @@ jobs:
runs-on: ubuntu-latest
needs: unit-test-java
env:
PYTHON: 3.9
steps:
- uses: actions/checkout@v4
with:
@@ -128,7 +128,7 @@ jobs:
architecture: x64
- uses: actions/setup-python@v3
with:
python-version: '3.9'
architecture: 'x64'
- uses: actions/cache@v2
with:
Expand Down Expand Up @@ -158,7 +158,7 @@ jobs:
uses: actions/setup-python@v3
id: setup-python
with:
python-version: 3.9
architecture: x64
- name: Get pip cache dir
id: pip-cache
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/linter.yml
Original file line number Diff line number Diff line change
Expand Up @@ -6,14 +6,14 @@ jobs:
lint-python:
runs-on: [ubuntu-latest]
env:
PYTHON: 3.9
steps:
- uses: actions/checkout@v4
- name: Setup Python
id: setup-python
uses: actions/setup-python@v5
with:
python-version: "3.9"
architecture: x64
- name: Upgrade pip version
run: |
Expand Down
12 changes: 6 additions & 6 deletions .github/workflows/pr_integration_tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -47,21 +47,21 @@ jobs:
id: image-tag
run: echo "::set-output name=DOCKER_IMAGE_TAG::`git rev-parse HEAD`"
- name: Cache Public ECR Image
id: lambda_python_3_9
uses: actions/cache@v2
with:
path: ~/cache
key: lambda_python_3_9
- name: Handle Cache Miss (pull public ECR image & save it to tar file)
if: steps.cache-primes.outputs.cache-hit != 'true'
run: |
mkdir -p ~/cache
docker pull public.ecr.aws/lambda/python:3.9
docker save public.ecr.aws/lambda/python:3.9 -o ~/cache/lambda_python_3_9.tar
- name: Handle Cache Hit (load docker image from tar file)
if: steps.cache-primes.outputs.cache-hit == 'true'
run: |
docker load -i ~/cache/lambda_python_3_9.tar
- name: Build and push
env:
ECR_REGISTRY: ${{ steps.login-ecr.outputs.registry }}
@@ -86,7 +86,7 @@ jobs:
strategy:
fail-fast: false
matrix:
python-version: [ "3.10" ]
os: [ ubuntu-latest ]
env:
OS: ${{ matrix.os }}
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/pr_local_integration_tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ jobs:
strategy:
fail-fast: false
matrix:
python-version: [ "3.10" ]
os: [ ubuntu-latest ]
env:
OS: ${{ matrix.os }}
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/publish.yml
Original file line number Diff line number Diff line change
Expand Up @@ -160,7 +160,7 @@ jobs:
architecture: x64
- uses: actions/setup-python@v3
with:
python-version: '3.9'
architecture: 'x64'
- uses: actions/cache@v2
with:
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/unit_tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ jobs:
strategy:
fail-fast: false
matrix:
python-version: [ "3.9", "3.10" ]
os: [ ubuntu-latest, macos-13 ]
exclude:
- os: macos-13
Expand Down
3 changes: 1 addition & 2 deletions docs/SUMMARY.md
Original file line number Diff line number Diff line change
Expand Up @@ -106,8 +106,7 @@
* [Google Cloud Platform](reference/providers/google-cloud-platform.md)
* [Amazon Web Services](reference/providers/amazon-web-services.md)
* [Azure](reference/providers/azure.md)
* [Batch Materialization Engines](reference/batch-materialization/README.md)
* [Bytewax](reference/batch-materialization/bytewax.md)
* [Snowflake](reference/batch-materialization/snowflake.md)
* [AWS Lambda (alpha)](reference/batch-materialization/lambda.md)
* [Spark (contrib)](reference/batch-materialization/spark.md)
Expand Down
21 changes: 1 addition & 20 deletions docs/how-to-guides/running-feast-in-production.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,28 +57,9 @@ To keep your online store up to date, you need to run a job that loads feature d
Out of the box, Feast's materialization process uses an in-process materialization engine. This engine loads all the data being materialized into memory from the offline store, and writes it into the online store.

This approach may not scale to large amounts of data, which users of Feast may be dealing with in production.
In this case, we recommend using one of the more [scalable materialization engines](./scaling-feast.md#scaling-materialization), such as the [Bytewax Materialization Engine](../reference/batch-materialization/bytewax.md), or the [Snowflake Materialization Engine](../reference/batch-materialization/snowflake.md).
Users may also need to [write a custom materialization engine](../how-to-guides/customizing-feast/creating-a-custom-materialization-engine.md) to work on their existing infrastructure.

The Bytewax materialization engine can run materialization on an existing Kubernetes cluster. An example configuration of this in a `feature_store.yaml` is as follows:

```yaml
batch_engine:
type: bytewax
namespace: bytewax
image: bytewax/bytewax-feast:latest
env:
- name: AWS_ACCESS_KEY_ID
valueFrom:
secretKeyRef:
name: aws-credentials
key: aws-access-key-id
- name: AWS_SECRET_ACCESS_KEY
valueFrom:
secretKeyRef:
name: aws-credentials
key: aws-secret-access-key
```

### 2.2 Scheduled materialization with Airflow

Expand Down
Loading
Toggle all file notes Toggle all file annotations