Online store latency (redis) is growing over time

Expected Behavior

The online-store latency remains the same and doesn't grow over the time. We assume that amount of data in redis is not growing.

Current Behavior

The median latency growth over the past 30 days is on the chart above.

Steps to reproduce

We use a custom aiohttp python service to relay requests to the online-store, where we invoke the native feast client. The service is completely stateless, and I don't expect it to be the source of the problem.

here is our feast config:

config = RepoConfig(
    project='feast',
    registry='gs://our-bucket/feast/registry_file_3.db',
    provider='gcp',
    online_store=RedisOnlineStoreConfig(
        connection_string=get_redis_connection_string(),
        redis_type='redis',
        key_ttl_seconds=REDIS_TTL_SECONDS,
    ),
    offline_store=BigQueryOfflineStoreConfig(
        dataset='feature_store',
        project_id='project-id',
    ),
    entity_key_serialization_version=2,
)

as you see, we use the default registry cache TTL (=600).

Specifications

• Version: 0.28.0
• Platform: amd64
• Subsystem: debian 10

Possible Solution

We noticed that changing the path to file-based registry (i.e. effectively re-creating it) eliminates the latency growth and it back to normal (today's chart):

Therefore, a solution might be related to fixing the registry caching mechanism.

Let me know if further details are needed!