Operator E2E test sporadically failing due to the timing issue

lokeshrangineni

Operator E2E test sporadically failing due to the timing issue based on the availability of the pods status. We have to expect reliable execution on every PR to avoid the inconsistency.

For Example:

https://github.com/feast-dev/feast/actions/runs/12896520988/job/35959688148?pr=4930

Here is the error we are seeing.

STEP: deploying the Simple Feast Custom Resource to Kubernetes @ 01/21/25 21:59:21.019
  running: kubectl apply -f test/testdata/feast_integration_test_crs/v1alpha1_default_featurestore.yaml -n default
  [FAILED] in [It] - /home/runner/work/feast/feast/infra/feast-operator/test/e2e/e2e_test.go:161 @ 01/21/25 21:59:21.129
  STEP: Uninstalling the feast CRD @ 01/21/25 21:59:21.129
  running: kubectl delete deployment feast-operator-controller-manager -n feast-operator-system
• [FAILED] [466.659 seconds]
controller Operator E2E Tests [It] Should be able to deploy and run a default feature store CR successfully
/home/runner/work/feast/feast/infra/feast-operator/test/e2e/e2e_test.go:100

  [FAILED] Error occurred while checking FeatureStore simple-feast-setup is having remote registry or not. 
  Error: kubectl get featurestore command returned empty output

  Unexpected error:
      <*errors.errorString | 0xc0002a65b0>: 
      kubectl get featurestore command returned empty output
      {
          s: "kubectl get featurestore command returned empty output",
      }
  occurred
  In [It] at: /home/runner/work/feast/feast/infra/feast-operator/test/e2e/e2e_test.go:161 @ 01/21/25 21:59:21.129
------------------------------
S [SKIPPED] [0.000 seconds]
controller Operator E2E Tests [It] Should be able to deploy and run a feature store with remote registry CR successfully
/home/runner/work/feast/feast/infra/feast-operator/test/e2e/e2e_test.go:118

  [SKIPPED] Spec skipped because an earlier spec in an ordered container failed
  In [It] at: /home/runner/work/feast/feast/infra/feast-operator/test/e2e/e2e_test.go:118 @ 01/21/25 21:59:21.18
------------------------------

Summarizing 1 Failure:
  [FAIL] controller Operator E2E Tests [It] Should be able to deploy and run a default feature store CR successfully
  /home/runner/work/feast/feast/infra/feast-operator/test/e2e/e2e_test.go:161

Ran 1 of 2 Specs in [46](https://github.com/feast-dev/feast/actions/runs/12896520988/job/35959688148?pr=4930#step:7:47)6.659 seconds
FAIL! -- 0 Passed | 1 Failed | 0 Pending | 1 Skipped
--- FAIL: TestE2E (466.66s)
FAIL
FAIL	github.com/feast-dev/feast/infra/feast-operator/test/e2e	466.665s
FAIL
make: *** [Makefile:121: test-e2e] Error 1
Error: Process completed with exit code 2.