Skip to content

Feat/mssql python Add optional mssql-python backend, make pyodbc optional, and update docs / CI#681

Open
axellpadilla wants to merge 10 commits into
dbt-msft:masterfrom
axellpadilla:feat/mssql-python
Open

Feat/mssql python Add optional mssql-python backend, make pyodbc optional, and update docs / CI#681
axellpadilla wants to merge 10 commits into
dbt-msft:masterfrom
axellpadilla:feat/mssql-python

Conversation

@axellpadilla
Copy link
Copy Markdown
Collaborator

This PR introduces the new optional mssql-python connection backend while keeping the legacy pyodbc ODBC path available as an explicit extra.

Key changes:

  • Add lazy-loaded backend support in sqlserver_connections.py
  • Make both pyodbc and mssql-python optional extras in pyproject.toml
  • Update sqlserver_credentials.py for backend selection
  • Add/adjust unit tests for lazy import and backend behavior
  • Update integration test matrix in integration-tests-sqlserver.yml to preserve coverage without increasing total run count
  • Add devcontainer support for both backend extras in setup_env.sh
  • Clarify installation and usage in README.md
  • Add developer setup notes and dependency guidance to CONTRIBUTING.md
  • Small support changes in test.env.sample, conftest.py, and uv.lock

Why this change:

  • enables the modern mssql-python backend behind currently gated use_mssql_python: true
  • Avoids requiring system dependencies unless a backend is explicitly selected
  • keeps existing pyodbc behavior intact for legacy profiles
  • improves documentation and local development setup

Testing:

  • updated unit tests for both backend import behavior
  • CI workflow preserves the same total run count by partitioning backends across Python versions
  • devcontainer setup installs both pyodbc and mssql-python extras for local validation

@axellpadilla axellpadilla added this to the v1.10.0 milestone May 21, 2026
@Benjamin-Knight
Copy link
Copy Markdown
Collaborator

Ok, its a big change so will take a while to review, I can see a few issues I think so I'll find some time at start providing feedback.

My main concern other than the actual code change is the install method for the adapter now changes as we have to specify the driver we want to use, that is a big change. This doesn't just affect us, DBT themselves will have the wrong install instructions.

https://docs.getdbt.com/docs/local/connect-data-platform/mssql-setup?version=1.12#prerequisites

Copy link
Copy Markdown
Collaborator

@Benjamin-Knight Benjamin-Knight left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably need some new tests as well but will look at those after these comments.

Comment thread dbt/adapters/sqlserver/sqlserver_connections.py Outdated
Comment thread dbt/adapters/sqlserver/sqlserver_connections.py Outdated
Comment thread dbt/adapters/sqlserver/sqlserver_connections.py Outdated
Comment thread dbt/adapters/sqlserver/sqlserver_credentials.py Outdated
Comment thread pyproject.toml Outdated
Comment thread dbt/adapters/sqlserver/sqlserver_connections.py Outdated
Comment thread .github/workflows/integration-tests-sqlserver.yml
Comment thread dbt/adapters/sqlserver/sqlserver_connections.py Outdated
Comment thread dbt/adapters/sqlserver/sqlserver_connections.py Outdated
Comment thread tests/unit/adapters/mssql/test_sqlserver_connection_manager.py
@axellpadilla
Copy link
Copy Markdown
Collaborator Author

@Benjamin-Knight I'm moving this to v1.11, I'm more interested in the other patches for v1.10 and its true we will need more time holding the rc1 pending until dbt docs are updated

@axellpadilla axellpadilla modified the milestones: v1.10.0, v1.11.0 May 21, 2026
…and docs matching and improve optional mssql-python support

Upgrade to backend enum
Update README install docs to explain the default pyodbc backend and optional mssql extra for mssql-python
Add azure dep to lazy loading
Add pyodbc as a core dependency and update mssql-python, azure-identity, and azure-core extras in pyproject.toml and uv.lock
Add authentication normalization and validation for SQL Server profile fields
Add mssql-python-specific connection string handling for MSI and Integrated auth flows
Expand unit tests for mssql-python auth normalization and connection behavior
- Update `_validate_pyodbc_requirements()` to reject `None`, empty, and whitespace-only `driver` values
- Add test covering `None`, `""`, and `"   "` driver inputs
…hentication and connection string features

- Added support for Active Directory access token authentication.
- Improved handling of connection string sanitization for logging.
- Introduced validation for connection requirements and query timeout.
- Enhanced tests for various authentication methods and connection string formats.
- Refactored connection handling to support new backend options.
@axellpadilla axellpadilla modified the milestones: v1.11.0, v1.10.1 May 27, 2026
@axellpadilla
Copy link
Copy Markdown
Collaborator Author

@Benjamin-Knight Kept pyodbc by default and already running pyodbc part on production so changed milestone to v1.10.1, this way we don't need to update upstream docs still, and can announce deprecation if we really start to mostly use the new driver.
All lazy imports behavior intact, this mostly makes a mandatory install of pyodbc by default even if we don't use it.
This should be squash merged after v1.10.0

@Benjamin-Knight
Copy link
Copy Markdown
Collaborator

@Benjamin-Knight Kept pyodbc by default and already running pyodbc part on production so changed milestone to v1.10.1, this way we don't need to update upstream docs still, and can announce deprecation if we really start to mostly use the new driver. All lazy imports behavior intact, this mostly makes a mandatory install of pyodbc by default even if we don't use it. This should be squash merged after v1.10.0

Ok, Its a big change so we will want to have a much longer period for a release candidate then we did with the 1.9.1 release to give people time to feedback but I think its good to go now that the default behaviour is still pyodbc.

Have you flipped the adapter into mssql driver mode for a large corpus of models?

@axellpadilla
Copy link
Copy Markdown
Collaborator Author

@Benjamin-Knight Kept pyodbc by default and already running pyodbc part on production so changed milestone to v1.10.1, this way we don't need to update upstream docs still, and can announce deprecation if we really start to mostly use the new driver. All lazy imports behavior intact, this mostly makes a mandatory install of pyodbc by default even if we don't use it. This should be squash merged after v1.10.0

Ok, Its a big change so we will want to have a much longer period for a release candidate then we did with the 1.9.1 release to give people time to feedback but I think its good to go now that the default behaviour is still pyodbc.

Have you flipped the adapter into mssql driver mode for a large corpus of models?

Going to on a couple days of pyodbc, current side effect is a speed gain on sqlfluff formatting because the lazy stuff

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants