◐ Shell
reader mode source ↗
Skip to content

feat: Add integration tests for dbt import with local dbt project#5877

Draft
Copilot wants to merge 8 commits into
masterfrom
copilot/add-integration-tests-for-dbt-import
Draft

feat: Add integration tests for dbt import with local dbt project#5877
Copilot wants to merge 8 commits into
masterfrom
copilot/add-integration-tests-for-dbt-import

Conversation

Copilot AI commented Jan 16, 2026

Copy link
Copy Markdown
Contributor

What this PR does / why we need it:

Adds comprehensive integration tests for the dbt import feature (PR #5827), which previously lacked end-to-end testing with actual dbt project setup and Feast CLI integration.

Changes

Test Infrastructure

  • Created sdk/python/tests/integration/dbt/ with 600+ lines covering:
    • Manifest parsing with dbt-artifacts-parser
    • Tag filtering (feast, ml, recommendations)
    • Model selection by name
    • All 3 data source types (BigQuery, Snowflake, File)
    • Type mapping validation (INT32/64, FLOAT32/64, STRING, TIMESTAMP)
    • FeatureView generation with schema validation
    • Column exclusion (entity, timestamp, custom)
    • Code generation workflow

Test dbt Project

  • Minimal dbt project in test_dbt_project/ with 3 models:
    • driver_features: INT types, multiple tags
    • customer_features: STRING entity
    • product_features: FLOAT32, tag filtering test
  • Seed data (CSV files) for all three models to enable actual dbt execution
  • DuckDB-only profile configuration for local testing without external service dependencies
  • Models use ref() to reference seeds, enabling full dbt workflow testing
  • Manifest generation: Removed pre-committed manifest.json; CI now generates it via dbt build to ensure tests validate actual dbt output

CI/CD - End-to-End Workflow

  • GitHub Actions workflow dbt-integration-tests.yml
  • Runs on Python 3.11 & 3.12
  • Triggers on dbt code/test changes
  • Uses Makefile for dependency installation (make install-python-dependencies-ci)
  • Executes full dbt workflow: dbt deps, dbt build, dbt test with seed data
  • Tests Feast CLI integration:
    • Creates Feast project with SQLite online store configuration
    • Runs feast dbt import command with generated manifest
    • Verifies Feast objects (entities, feature views) are created correctly
  • Validates both dbt compilation and Feast import work together

Bug Fixes

  • Removed deprecated PytestUnhandledCoroutineWarning from pytest.ini
  • Fixed test failures by ensuring manifest.json is generated by dbt during CI rather than pre-committed

Test Example

def test_get_models_with_tag_filter(self, parser):
    """Test filtering models by dbt tag."""
    # Filter by 'ml' tag
    ml_models = parser.get_models(tag_filter="ml")
    assert len(ml_models) == 2
    
    # Filter by 'recommendations' tag
    rec_models = parser.get_models(tag_filter="recommendations")
    assert len(rec_models) == 1

Which issue(s) this PR fixes:

Related to #3335

Misc

Comprehensive documentation added for test structure and dbt project. The workflow now provides complete end-to-end testing from dbt model definition through to Feast feature store integration, including actual data materialization to SQLite online store. The manifest.json is dynamically generated during CI to ensure tests validate real dbt compilation output.

Original prompt

This section details on the original issue you should resolve

<issue_title>Add integration tests for dbt import with local dbt project</issue_title>
<issue_description>## Context
PR #5827 added dbt integration but lacks proper integration tests with an actual dbt project setup.

Description

As mentioned by @franciscojavierarceo in PR #5827, we need a lightweight local dbt project that:

  • Sets up a minimal dbt project structure
  • Runs dbt compile to generate manifest.json
  • Tests the full feast dbt import workflow end-to-end
  • Validates generated Feast objects

Acceptance Criteria

  • Create test dbt project in sdk/python/tests/integration/dbt/
  • Add test that compiles dbt project and imports to Feast
  • Test all three data source types (bigquery, snowflake, file)
  • Verify FeatureViews are created correctly
  • Test tag filtering and model selection

Related

<agent_instructions>please setup dbt in a github action integration test and test the dbt parser that was implemented in PR #5827</agent_instructions>

Comments on the Issue (you are @copilot in this section)


💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.

Copilot AI and others added 3 commits January 16, 2026 20:25
Co-authored-by: franciscojavierarceo <4163062+franciscojavierarceo@users.noreply.github.com>
Co-authored-by: franciscojavierarceo <4163062+franciscojavierarceo@users.noreply.github.com>
Co-authored-by: franciscojavierarceo <4163062+franciscojavierarceo@users.noreply.github.com>
Copilot AI changed the title [WIP] Add integration tests for dbt import with local dbt project Jan 16, 2026
@franciscojavierarceo

Copy link
Copy Markdown
Member

@YassinNouh21 mind reviewing?

Co-authored-by: franciscojavierarceo <4163062+franciscojavierarceo@users.noreply.github.com>
Co-authored-by: franciscojavierarceo <4163062+franciscojavierarceo@users.noreply.github.com>

@YassinNouh21 YassinNouh21 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hide comment

See inline comments for specific fixes needed.

@YassinNouh21 YassinNouh21 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hide comment

fix those issues copilot

@YassinNouh21 YassinNouh21 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hide comment

Found several API mismatches that will cause test failures

@YassinNouh21

Copy link
Copy Markdown
Collaborator

@franciscojavierarceo Found some issues in this PR that need attention:

Critical bugs (will cause test failures):

  • Line 230: entity.join_keys → should be entity.join_key (singular)
  • Line 254: feature_view.entities[0] stores entity names as strings, not Entity objects
  • Line 527: Object count should be 9, not 12 (3 entities + 3 sources + 3 views)

Architecture concerns:

  • These are labeled "integration tests" but don't actually run dbt or use a real database - they're unit tests testing the parser against a static manifest
  • Manifest uses outdated v9 schema instead of v12
  • Missing error handling & edge case coverage

The inline comments on lines 191, 215, 488 show the exact fixes needed for the critical bugs. Happy to help if you need clarification on any of these.

@franciscojavierarceo

Copy link
Copy Markdown
Member

@YassinNouh21 you have to tag @copilot in every comment for it to update the code 😅

@YassinNouh21

Copy link
Copy Markdown
Collaborator

@franciscojavierarceo I thought he will continue working on it 😂

Co-authored-by: franciscojavierarceo <4163062+franciscojavierarceo@users.noreply.github.com>
@franciscojavierarceo

Copy link
Copy Markdown
Member

@copilot tests are failing

…generate it

Co-authored-by: franciscojavierarceo <4163062+franciscojavierarceo@users.noreply.github.com>

Copilot AI commented Jan 22, 2026

Copy link
Copy Markdown
Contributor Author

@copilot tests are failing

Fixed in commit 586f825. The issue was an incorrectly pre-committed manifest.json from unit tests (with wrong project name "sample_dbt_project" instead of "feast_integration_test"). The manifest is now generated fresh by dbt build in the CI workflow before tests run.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add integration tests for dbt import with local dbt project

3 participants