* Enhance AI/BI dashboard skill with comprehensive widget specs
Added missing documentation from production dashboard generation:
1-widget-specifications.md:
- Combo charts (bar + line on same widget) with version 1
- Counter number formatting (currency, percent, plain number)
- Widget name max length (60 characters)
- Color scale restrictions (no scheme/colorRamp/mappings)
- Quantitative color encoding for gradient effects
- Bar chart group vs stacked decision criteria with examples
2-filters.md:
- Date range picker complete example
- Multi-dataset filter binding (one query per dataset)
- Global filter performance note (auto WHERE clause)
SKILL.md:
- ORDER BY guidance for time series and rankings
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Add TOP-N + Other bucketing guidance for high-cardinality dimensions
When a dimension has too many values (50+ stores, products, etc.),
charts become unreadable. Added guidance to:
- Check cardinality via get_table_details before charting
- Use TOP-N + "Other" SQL pattern to bucket low-value items
- Aggregate to higher abstraction level as alternative
- Use table widgets for high-cardinality data
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Remove async deploy_dashboard function for consistency
The codebase doesn't use async anywhere else, so remove the unused
async version of deploy_dashboard and keep only the synchronous one.
- Remove asyncio import
- Remove async deploy_dashboard function (was using asyncio.to_thread)
- Rename deploy_dashboard_sync to deploy_dashboard
- Update exports in __init__.py
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Add genie_space_id parameter to dashboard creation
Allow linking a Genie space to a dashboard by passing genie_space_id.
This enables the "Ask Genie" button on the dashboard UI.
The Genie space config is injected into the serialized_dashboard JSON
under uiSettings.genieSpace with isEnabled=true and enablementMode=ENABLED.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Add catalog and schema parameters to dashboard creation
Allow setting default catalog and schema for dashboard datasets via
the dataset_catalog and dataset_schema API parameters. These defaults
apply to unqualified table names in SQL queries.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Add comprehensive date range filtering documentation
- Document field-based filtering (automatic IN_RANGE on date fields)
- Document parameter-based filtering (:date_range.min/max in SQL)
- Show how to combine both approaches in one filter
- Add guidance on when NOT to apply date filtering (MRR, all-time totals)
- Update SKILL.md tools table with new genie_space_id, catalog, schema params
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Restructure AI/BI dashboard skill with improved organization
- Split widget specs into basic (1-widget-specifications.md) and advanced
(2-advanced-widget-specifications.md) files
- Add area chart, scatter plot, combo chart, and choropleth map documentation
- Rename files for consistent numbering (3-filters, 4-examples, 5-troubleshooting)
- Remove duplicate information across files (versions, naming rules, etc.)
- Add widget display formatting guidance (currency, percentage, displayName)
- Simplify SKILL.md quality checklist with link to version table
- Shorten verbose examples while preserving all critical information
- Clarify query naming convention for charts vs filters
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Add back critical behavioral instructions for text widgets and filters
* Restore important behavioral instructions removed during restructure
- Counter: full Pattern 2 example with CRITICAL field name matching note
- Table: disaggregated:true guidance and bold emphasis
- Line/Bar: x,y,color encodings and disaggregated guidance
- Pie: 3-8 category limit for readability
* Restore detailed guidance that was removed during restructure
- 5-troubleshooting.md: Restore full troubleshooting content with version
guidance, filter debugging, and detailed error explanations
- SKILL.md: Restore full 10-item quality checklist
- SKILL.md: Restore standard dashboard structure example
- SKILL.md: Restore cardinality guidance table (with softer 'suggested' language)
* Optimize MCP tool docstrings for token efficiency
- Reduce docstring verbosity across all 18 tool files (~89% reduction)
- Keep all functional information while being concise
- Add skill references to complex tools (dashboards, vector search, genie, jobs, pipelines, lakebase, unity catalog, serving, apps, agent bricks)
- Maintain human readability with bullet points and structure
- Preserve critical warnings (ASK USER FIRST, CONFIRM WITH USER)
- Keep return format hints for AI parsing
Net reduction: 1,843 lines across 18 files
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Add parameter context for ambiguous docstring params
- agent_bricks.py: Add context for description, instructions, volume_path, examples
- genie.py: Add context for table_identifiers, description, sample_questions, serialized_space
- jobs.py: Add context for tasks, job_clusters, environments, schedule, git_source
- lakebase.py: Add context for source_branch, ttl_seconds, is_protected, autoscaling params,
and sync source/target table names
- pipelines.py: Add context for root_path, workspace_file_paths, extra_settings, full_refresh
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Consolidate MCP tools from 77 to 44 (43% reduction)
Tool consolidations:
- pipelines.py: 10→2 (manage_pipeline, manage_pipeline_run)
- volume_files.py: 6→1 (manage_volume_files)
- aibi_dashboards.py: 4→1 (manage_dashboard)
- vector_search.py: 8→4 (manage_vs_endpoint, manage_vs_index, query_vs_index, manage_vs_data)
- genie.py: 5→2 (manage_genie, ask_genie)
- serving.py: 3→1 (manage_serving_endpoint)
- apps.py: 3→1 (manage_app)
- file.py: 2→1 (manage_workspace_files)
- sql.py: 6→5 (manage_warehouse replaces list/get_best)
- lakebase.py: 8→4 (manage_lakebase_database, manage_lakebase_branch, manage_lakebase_sync, generate_lakebase_credential)
Key patterns:
- All consolidated tools use an action parameter
- Each action has required params documented in docstring
- Error messages specify which params are required
- Hot paths (query_vs_index, ask_genie) kept separate for clarity
- All skills updated with action tables and examples
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Add integration test infrastructure and fix tool bugs
Test infrastructure:
- Add comprehensive integration tests for all MCP tools
- Add test runner script with parallel execution support
- Add fixtures for workspace, catalog, and resource cleanup
- Add test resources (PDFs, SQL files, app configs)
Bug fixes in databricks-tools-core:
- Fix workspace file upload for directories
- Fix job notebook path handling
- Fix vector search index operations
- Fix apps API responses
- Fix dashboard widget handling
- Fix agent bricks manager listing
Bug fixes in MCP server tools:
- Add quota skip handling for apps test
- Fix genie space operations
- Fix lakebase database operations
- Fix compute cluster lifecycle handling
- Fix dashboard operations
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Change manage_dashboard to use file path instead of inline JSON
- Replace serialized_dashboard param with dashboard_file_path
- Tool reads JSON from local file for easier iterative development
- Update SKILL.md with new workflow documentation
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Update dashboard tests for file-based approach
- Change simple_dashboard_json fixture to simple_dashboard_file
- Update all manage_dashboard calls to use dashboard_file_path
- Add tempfile imports and tmp_path usage for update test
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Fix deploy_app to correctly handle SDK Wait[AppDeployment] return type
The Databricks SDK's w.apps.deploy() returns a Wait[AppDeployment] object,
not an AppDeployment directly. The previous code passed the Wait object
to _deployment_to_dict(), which caused getattr() to return None for all
attributes since the Wait object doesn't have them.
This fix uses wait_obj.response to get the actual AppDeployment object
before converting it to a dictionary.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Fix deploy_app to correctly handle SDK Wait[AppDeployment] return type
The Databricks SDK's w.apps.deploy() returns a Wait[AppDeployment] object,
not an AppDeployment directly. The previous code passed the Wait object
to _deployment_to_dict(), which caused getattr() to return None for all
attributes since the Wait object doesn't have them.
This fix uses wait_obj.response to get the actual AppDeployment object
before converting it to a dictionary.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Clarify MCP tool usage in Genie skill documentation
- Add tools summary table at top of MCP Tools section
- Change code blocks from python syntax to plain text
- Add "# MCP Tool: <name>" comments to clarify these are tool calls, not Python code
- Move Supporting Tools table to main tools table
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Fix typo in aibi_dashboards.py docstring
Remove garbage characters from widget documentation.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Fix genie tools to use SDK methods instead of manager
- Use w.genie.trash_space() in _delete_genie_resource
- Add _find_space_by_name() using SDK's list_spaces with pagination
- Use w.genie.update_space() and w.genie.create_space() for space management
- Use w.genie.get_space() with include_serialized_space in _get_genie_space
- Fix validation to allow space_id for updates without display_name
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Improve integration test reliability and timeout handling
- Add per-suite timeout in run_tests.py (10 min default, configurable)
- Improve apps test with better cleanup and assertions
- Add skip logic for quota-exceeded scenarios
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Improve Unity Catalog tool docstrings with comprehensive parameter documentation
Add detailed parameter documentation to all 9 Unity Catalog MCP tools:
- manage_uc_objects: Document parameters by object_type (catalog/schema/volume/function)
- manage_uc_grants: Add privilege lists per securable type
- manage_uc_storage: Detail credential and external_location parameters
- manage_uc_connections: Document connection_type options and create_foreign_catalog
- manage_uc_tags: Detail set_tags/unset_tags/query parameters
- manage_uc_security_policies: Document row filter and column mask parameters
- manage_uc_monitors: Detail monitor creation and refresh parameters
- manage_uc_sharing: Document share/recipient/provider resource types
- manage_metric_views: Detail dimension/measure format and query parameters
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Add CRITICAL validation steps to dashboard tool docstring
Add clear instructions requiring users to:
0. Review the databricks-aibi-dashboards skill for widget JSON structure
1. Call get_table_stats_and_schema() for table schemas
2. Call execute_sql() to test EVERY query before use
This prevents widgets from showing errors due to untested queries.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Add design best practices section and use relative file paths
- Add Design Best Practices section for default dashboard behaviors
- Change /tmp paths to ./ for less opinionated examples
- Update parent_path example to use {user_email} placeholder
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Improve code execution and synthetic data gen skills
Execution compute skill:
- Split into 3 clear execution modes: Databricks Connect (default), Serverless Job, Interactive Cluster
- Add decision matrix for choosing execution mode
- Add job_extra_params for custom dependencies in serverless jobs
- Create dedicated reference files for each mode
Synthetic data gen skill:
- Emphasize business story: problem → impact → analysis → solution
- Add guidance to propose compelling stories by default
- Consolidate reference files (6 → 2)
- Add critical rules for data coherence and Databricks value
- Clarify when to read reference files
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Add performance rules for synthetic data generation
- Add Critical Rule #12: No Python loops or .collect()
- Add Performance Rules section with anti-pattern table
- Emphasize Spark parallelism over driver-side iteration
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Clarify parquet path is folder with table name
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Improve AI/BI dashboard skill documentation with comprehensive examples
- Replace basic NYC taxi examples with complete Sales Analytics dashboard
- Add critical widget version requirements table to SKILL.md
- Add data validation guidance to verify dashboards tell intended story
- Document key patterns: page types, KPI formatting, filter binding, layout grid
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Add skill reading requirement to dashboard MCP tool docstring
Require agent to read 4-examples.md before creating dashboards, and if
unfamiliar, read full skill documentation first. Valid JSON is critical.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Emphasize Databricks Connect serverless, add setup requirements
- Rule #10: Use Databricks Connect Serverless by default (avoid execute_code)
- Setup: Use uv, require Python 3.12 and databricks-connect>=16.4
- Common issues: Add DatabricksEnv ImportError and Python version fixes
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Fix MCP server crash on request cancellation
When a client cancels a long-running MCP request, there's a race condition
between the cancellation and normal response paths:
1. Client cancels request → RequestResponder.cancel() sends error response
and sets _completed = True
2. Middleware catches CancelledError and returns a ToolResult
3. MCP SDK tries to call message.respond(response)
4. Crash: assert not self._completed fails
Fix: Re-raise CancelledError instead of returning a result, allowing the
MCP SDK's cancellation handler to properly manage the response lifecycle.
See: modelcontextprotocol/python-sdk#1153
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Add structured_content to error responses for MCP SDK validation
When tools have an outputSchema (auto-generated from return type like
Dict[str, Any]), MCP SDK requires structured_content in all responses.
The middleware was returning ToolResult without structured_content for
error cases (timeout, exceptions), causing validation errors:
"Output validation error: outputSchema defined but no structured output returned"
Fix: Include structured_content with the same error data in all error responses.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Migrate KA operations to Python SDK and fix name lookup issues
- Migrate ka_create, ka_get, ka_sync_sources to use Python SDK
- Keep ka_update using raw API 2.1 due to SDK FieldMask bug (converts
snake_case to camelCase but API expects snake_case)
- Fix find_by_name to sanitize names (spaces→underscores) before lookup
- Fix ka_create_or_update to lookup by name when no tile_id provided,
preventing ALREADY_EXISTS errors on repeated calls
- Update MCP tool layer to use new flat response format
- Map SDK state values (ACTIVE, CREATING, FAILED) to endpoint_status
- Add integration test for updating existing KA via create_or_update
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Fix knowledge source description requirement and test ordering
- Provide default description for knowledge sources when not specified
(API requires non-empty knowledge_source.description)
- Move KA update test to after endpoint is ONLINE (update requires ACTIVE state)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Fix structured_content not populated for tools with return type annotations
FastMCP auto-generates outputSchema from return type annotations (e.g.,
-> Dict[str, Any]) but doesn't populate structured_content in ToolResult.
MCP SDK validation then fails: "outputSchema defined but no structured output"
Fix: Intercept successful results and populate structured_content from
JSON text content when missing. Only modifies results when:
1. structured_content is missing
2. There's exactly one TextContent item
3. The text is valid JSON that parses to a dict
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(mcp): apply async wrapper on all platforms to prevent cancellation crashes
The asyncio.to_thread() wrapper was only applied on Windows, but it's
needed on ALL platforms to enable proper cancellation handling.
Without this fix, when a sync tool runs longer than the client timeout:
1. Client sends cancellation
2. Sync tool blocks event loop, can't receive CancelledError
3. Tool eventually returns, but MCP SDK already responded to cancel
4. AssertionError: "Request already responded to" → server crashes
This was discovered when uploading 7,375 files triggered a timeout,
crashing the MCP server on macOS.
Extends the fix from PR #411 which added CancelledError handling in
middleware - that fix only works when cancellation can propagate,
which requires async execution via to_thread().
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Fix: don't set structured_content on error responses
Setting structured_content causes MCP SDK to validate it against the
tool's outputSchema. For error responses, the error dict {"error": True, ...}
doesn't match the expected return type (e.g., Union[str, List[Dict]]),
causing "Output validation error: 'result' is a required property".
Fix: Only set structured_content for successful responses, not errors.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Improve dashboard skill structure based on error analysis
- Add JSON skeleton section to SKILL.md showing required structure
- Add Genie note clarifying it's not a widget (use genie_space_id param)
- Move Key Patterns to top of 4-examples.md for discoverability
- Clarify example is reference only - adapt to user's actual requirements
- Add structural errors table to 5-troubleshooting.md
Root cause fixes:
- queryLines must be array, not "query": "string"
- Widgets must be inline in layout[].widget, not separate array
- pageType required on every page
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: Quentin Ambard <quentin.ambard@databricks.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: calreynolds <calrey98@gmail.com>