◐ Shell
clean mode source ↗

Support podAnnotations on Deployment pod template to enable OpenTelemetry auto-instrumentation

ntkathole

Problem

The Feast Operator currently has no way to set annotations on the Deployment's pod template. The setDeployment() function replaces the entire deploy.Spec on every reconcile, which means:

  1. Any pod template annotations added manually via kubectl patch are wiped on the next reconcile
  2. There is no CRD field to declaratively set pod template annotations

This blocks users from enabling OpenTelemetry Operator auto-instrumentation, which requires the pod annotation instrumentation.opentelemetry.io/inject-python: "true" to inject the OTEL SDK and export logs, traces, and metrics to backends like Dynatrace, Elasticsearch, etc.

Context

  • PR fix: Allow custom annotations on Operator installed objects #5339 fixed annotation preservation on Services and ConfigMaps, but the Deployment pod template was not addressed since it uses full Spec replacement (deploy.Spec = appsv1.DeploymentSpec{...})
  • The CronJob already supports PodTemplateAnnotations via spec.cronJob.jobSpec.podTemplateAnnotations, but no equivalent exists for the main Deployment
  • The Python feature server uses standard logging.getLogger() throughout (online, offline, registry servers), which OTEL auto-instrumentation can capture automatically — no Feast code changes needed
  • Feast already ships sample OTEL manifests (infra/charts/feast-feature-server/samples/instrumentation.yaml) and documentation (docs/getting-started/components/open-telemetry.md), but these cannot be used with operator-managed deployments today

Impact

Without this, users deploying Feast via the operator cannot:

  • Export Python server logs (info/debug/warn/error) to observability backends via OTLP
  • Enable distributed tracing for feature serving requests
  • Use OTEL auto-instrumentation for any purpose (not just logging — also Istio sidecar injection, Vault agent injection, and other annotation-driven integrations)

Proposed Solution

Add a podAnnotations field to FeatureStoreServices and apply it in setDeployment(), following the existing PodTemplateAnnotations pattern from the CronJob:

apiVersion: feast.dev/v1
kind: FeatureStore
metadata:
  name: my-feast
spec:
  services:
    podAnnotations:
      instrumentation.opentelemetry.io/inject-python: "true"
    onlineStore:
      server:
        metrics: true