CHORE: Add OneBranch release pipelines for mssql-python-odbc#664
Open
jahnvi480 wants to merge 33 commits into
Open
CHORE: Add OneBranch release pipelines for mssql-python-odbc#664jahnvi480 wants to merge 33 commits into
jahnvi480 wants to merge 33 commits into
Conversation
Adds the build and release pipelines for the standalone mssql-python-odbc package: per-platform build stages (Windows x64/arm64, macOS universal2, Linux manylinux_2_28 + musllinux_1_2 for x86_64/aarch64), a consolidate job that gathers all 7 py3-none wheels, the build orchestration pipeline, and the ESRP-based official release pipeline. The build stages run no compilation (data-only package); Linux builds run in tdslibrs ACR PyPA containers and the musllinux tag is auto-detected via libc_ver. Infra follow-ups (ESRP onboarding of the mssql-python-odbc name, registering the ADO build definition to set odbcBuildDefinitionId, and sharing the ESRP variable group) are marked with TODO(infra) comments.
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Pull request overview
Adds Azure DevOps OneBranch build + official release pipeline YAMLs to produce and publish the standalone mssql-python-odbc “data-only” wheel set (7 platform-specific py3-none-<platform> wheels), aligning the CI/CD half of the ODBC-driver split with the existing mssql-python OneBranch approach.
Changes:
- Introduces a new OneBranch build pipeline that fans out into Windows/macOS/Linux wheel build stages and then consolidates artifacts.
- Adds per-OS stage templates for building the ODBC wheels (Windows x64/arm64, macOS universal2, Linux manylinux/musllinux x86_64/aarch64).
- Adds an official ESRP-based release pipeline to publish the consolidated wheels to PyPI (gated by
releaseToPyPI).
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| OneBranchPipelines/build-release-odbc-pipeline.yml | Orchestrates multi-stage build for 7 ODBC wheels + consolidation; configures global SDL settings for this package. |
| OneBranchPipelines/stages/build-odbc-windows-stage.yml | Builds/stages Windows ODBC wheels per-arch and publishes per-stage artifacts. |
| OneBranchPipelines/stages/build-odbc-macos-stage.yml | Builds/stages macOS universal2 ODBC wheel and publishes per-stage artifacts. |
| OneBranchPipelines/stages/build-odbc-linux-stage.yml | Builds/stages Linux ODBC wheels via PyPA containers (manylinux/musllinux) and publishes per-stage artifacts. |
| OneBranchPipelines/jobs/consolidate-odbc-artifacts-job.yml | Downloads all stage artifacts, consolidates wheels into a single dist/, and publishes the consolidated artifact. |
| OneBranchPipelines/official-release-odbc-pipeline.yml | Downloads consolidated artifact and (optionally) releases to PyPI via ESRP. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
📊 Code Coverage Report
Diff CoverageDiff: main...HEAD, staged and unstaged changesNo lines with coverage information in this diff. 📋 Files Needing Attention📉 Files with overall lowest coverage (click to expand)mssql_python.pybind.logger_bridge.cpp: 59.2%
mssql_python.pybind.ddbc_bindings.h: 59.9%
mssql_python.pybind.logger_bridge.hpp: 70.8%
mssql_python.pybind.ddbc_bindings.cpp: 76.1%
mssql_python.pybind.connection.connection.cpp: 76.2%
mssql_python.__init__.py: 77.3%
mssql_python.row.py: 77.6%
mssql_python.ddbc_bindings.py: 79.6%
mssql_python.connection.py: 83.6%
mssql_python.logging.py: 85.5%🔗 Quick Links
|
…nux twine check + container cleanup, reword infra TODOs
…; drop unused targetArch var on Windows stage
Per review feedback, build and release the standalone mssql-python-odbc package from the existing Build-Release-Package-Pipeline (def 2199) and official release pipeline instead of separate ADO definitions. Build pipeline: add odbcWindows/Macos/LinuxConfigs params (2+1+4=7 data-only wheels), reference the 3 odbc stage templates alongside the mssql-python stages, and add a separate ConsolidateOdbc stage publishing drop_ConsolidateOdbc_ConsolidateArtifacts (distinct from mssql-python's drop_Consolidate_ConsolidateArtifacts). Release pipeline: add releasePackage parameter (mssql-python | mssql-python-odbc); select the consolidated artifact accordingly and skip symbol publishing + mssql-py-core version validation for odbc. Delete the redundant standalone orchestrators (build-release-odbc-pipeline.yml, official-release-odbc-pipeline.yml). This removes the need to register a new ADO build definition and the odbcBuildDefinitionId placeholder. Both packages now build from def 2199; the odbc stages consume setup_odbc.py from PR #663, so #663 must merge before these stages can build.
Def 2199 now builds mssql-python and mssql-python-odbc in one run. Both consolidate jobs used DownloadPipelineArtifact@2 with buildType: current, which downloads every artifact in the run and copies all *.whl into dist/. In the merged run this caused: (1) the main Consolidate to sweep the 7 odbc wheels into the mssql-python release (count is warning-only, so silent), and (2) ConsolidateOdbc to see 34 wheels while expecting 7 and hard-fail. Add itemPattern to scope each download to its own package's build-stage artifacts (main: drop_Win_*/drop_MacOS_*/drop_Linux_*; odbc: drop_ODBC_*), giving clean partitioning with no wheel leakage.
…-> 34) The matrix is 9 Windows + 5 macOS + 4 Linux stages x 5 Python versions (cp310-cp314) = 34 wheels across 18 stages. The diagnostic said 27 (7+4+16), stale from an older 3.10-3.13 / 7-Windows matrix. Count check remains warning-only (no behavior change).
official-release-pipeline.yml: gate 'Published symbols' dry-run/summary text to mssql-python so odbc runs no longer claim symbol publishing that never happens; add a mssql-python-only step asserting the wheel metadata pins mssql-python-odbc==18.6.0 (fails by design if released from a pre-pin build). build-odbc-{linux,macos,windows}-stage.yml: assert each built wheel actually contains the ODBC driver binary (msodbcsql*), guarding against the .gitignore '*.so' trap / silent sync_libs() skip shipping an empty data wheel that still passes twine check.
Flip the 34-wheel count check from WARNING-only to a hard exit 1, symmetric with the odbc consolidate job's count guard. A mismatch means a build stage silently produced a partial wheel set (e.g. a missing Python version) or the matrix changed without updating the expected count -- previously this shipped silently.
Mirrors the official-release-pipeline odbc changes: releasePackage parameter, consolidatedArtifactName selection for both artifact downloads, and gating the mssql-py-core version validation + symbol publishing to mssql-python only. Stays on Maven ContentType so it never publishes to PyPI.
Adds the mssql-python-odbc pin guard (gated to mssql-python, matching official-release-pipeline) and gates the 'symbols published' summary messaging on releasePackage==mssql-python so the dummy pipeline exercises the exact same mssql-python validation/messaging as the official pipeline for both packages.
Gate mssql-python and mssql-python-odbc stage groups behind a buildPackage parameter (mssql-python default) so only the selected package builds, halving run time and enabling concurrent per-package builds. Mirrors the releasePackage selector pattern in the release pipelines.
Add effectiveBuildPackage variable that forces 'both' on scheduled (Build.Reason=Schedule) runs while manual/PR runs honour the buildPackage parameter. Mirrors the effectiveOneBranchType pattern so the nightly keeps refreshing every artifact.
bewithgaurav
requested changes
Jul 16, 2026
sumitmsft
previously approved these changes
Jul 16, 2026
Adds the build and release pipelines for the standalone mssql-python-odbc package: per-platform build stages (Windows x64/arm64, macOS universal2, Linux manylinux_2_28 + musllinux_1_2 for x86_64/aarch64), a consolidate job that gathers all 7 py3-none wheels, the build orchestration pipeline, and the ESRP-based official release pipeline. The build stages run no compilation (data-only package); Linux builds run in tdslibrs ACR PyPA containers and the musllinux tag is auto-detected via libc_ver. Infra follow-ups (ESRP onboarding of the mssql-python-odbc name, registering the ADO build definition to set odbcBuildDefinitionId, and sharing the ESRP variable group) are marked with TODO(infra) comments.
…nux twine check + container cleanup, reword infra TODOs
…; drop unused targetArch var on Windows stage
Per review feedback, build and release the standalone mssql-python-odbc package from the existing Build-Release-Package-Pipeline (def 2199) and official release pipeline instead of separate ADO definitions. Build pipeline: add odbcWindows/Macos/LinuxConfigs params (2+1+4=7 data-only wheels), reference the 3 odbc stage templates alongside the mssql-python stages, and add a separate ConsolidateOdbc stage publishing drop_ConsolidateOdbc_ConsolidateArtifacts (distinct from mssql-python's drop_Consolidate_ConsolidateArtifacts). Release pipeline: add releasePackage parameter (mssql-python | mssql-python-odbc); select the consolidated artifact accordingly and skip symbol publishing + mssql-py-core version validation for odbc. Delete the redundant standalone orchestrators (build-release-odbc-pipeline.yml, official-release-odbc-pipeline.yml). This removes the need to register a new ADO build definition and the odbcBuildDefinitionId placeholder. Both packages now build from def 2199; the odbc stages consume setup_odbc.py from PR #663, so #663 must merge before these stages can build.
Def 2199 now builds mssql-python and mssql-python-odbc in one run. Both consolidate jobs used DownloadPipelineArtifact@2 with buildType: current, which downloads every artifact in the run and copies all *.whl into dist/. In the merged run this caused: (1) the main Consolidate to sweep the 7 odbc wheels into the mssql-python release (count is warning-only, so silent), and (2) ConsolidateOdbc to see 34 wheels while expecting 7 and hard-fail. Add itemPattern to scope each download to its own package's build-stage artifacts (main: drop_Win_*/drop_MacOS_*/drop_Linux_*; odbc: drop_ODBC_*), giving clean partitioning with no wheel leakage.
…-> 34) The matrix is 9 Windows + 5 macOS + 4 Linux stages x 5 Python versions (cp310-cp314) = 34 wheels across 18 stages. The diagnostic said 27 (7+4+16), stale from an older 3.10-3.13 / 7-Windows matrix. Count check remains warning-only (no behavior change).
official-release-pipeline.yml: gate 'Published symbols' dry-run/summary text to mssql-python so odbc runs no longer claim symbol publishing that never happens; add a mssql-python-only step asserting the wheel metadata pins mssql-python-odbc==18.6.0 (fails by design if released from a pre-pin build). build-odbc-{linux,macos,windows}-stage.yml: assert each built wheel actually contains the ODBC driver binary (msodbcsql*), guarding against the .gitignore '*.so' trap / silent sync_libs() skip shipping an empty data wheel that still passes twine check.
Flip the 34-wheel count check from WARNING-only to a hard exit 1, symmetric with the odbc consolidate job's count guard. A mismatch means a build stage silently produced a partial wheel set (e.g. a missing Python version) or the matrix changed without updating the expected count -- previously this shipped silently.
Mirrors the official-release-pipeline odbc changes: releasePackage parameter, consolidatedArtifactName selection for both artifact downloads, and gating the mssql-py-core version validation + symbol publishing to mssql-python only. Stays on Maven ContentType so it never publishes to PyPI.
Adds the mssql-python-odbc pin guard (gated to mssql-python, matching official-release-pipeline) and gates the 'symbols published' summary messaging on releasePackage==mssql-python so the dummy pipeline exercises the exact same mssql-python validation/messaging as the official pipeline for both packages.
Gate mssql-python and mssql-python-odbc stage groups behind a buildPackage parameter (mssql-python default) so only the selected package builds, halving run time and enabling concurrent per-package builds. Mirrors the releasePackage selector pattern in the release pipelines.
Add effectiveBuildPackage variable that forces 'both' on scheduled (Build.Reason=Schedule) runs while manual/PR runs honour the buildPackage parameter. Mirrors the effectiveOneBranchType pattern so the nightly keeps refreshing every artifact.
… ADO design Add hardened wheel-installation-test-stage.yml (positive+negative driver proof, source-tree shadowing guards, TESTED==0 fails stage) and wire two 'both'-gated TestBothWheels_* stages (manylinux_2_28 + musllinux x86_64) after ConsolidateOdbc, replacing the old single 3.12 invocation.
…crosoft/mssql-python into jahnvi/odbc-release-pipelines # Conflicts: # OneBranchPipelines/build-release-package-pipeline.yml
jahnvi480
added a commit
that referenced
this pull request
Jul 20, 2026
…ies (#663) ### Work Item / Issue Reference > [AB#46217](https://sqlclientdrivers.visualstudio.com/c6d89619-62de-46a0-8b46-70b92a84d85e/_workitems/edit/46217) ------------------------------------------------------------------- ### Summary Splits the ODBC driver binaries out of `mssql-python` into a new standalone, pure-data package `mssql-python-odbc` (import name `mssql_python_odbc`, version 18.6.0). This is the code / packaging half of the split; the ADO release pipelines are in companion PR #664. #### What is included - `mssql_python_odbc/__init__.py` - pure-Python package (no native extension) that ships the ODBC driver `libs/` tree and exposes `get_driver_path()` and `get_libs_dir()` with the same per-platform layout the C++ loader expects. - `setup_odbc.py` - builds the `mssql-python-odbc` wheels. Each wheel is platform-specific but Python-agnostic (`py3-none-<platform>`), so we ship 7 wheels total (Windows amd64/arm64, macOS universal2, manylinux_2_28 x86_64/aarch64, musllinux_1_2 x86_64/aarch64) instead of a per-Python matrix. On Windows the wheel intentionally ships the VC++ runtime (`vcredist/msvcp140.dll` + license) alongside the driver binaries - see "VC++ runtime" below. - `mssql_python/pybind/ddbc_bindings.cpp` - `LoadDriverOrThrowException()` now resolves the driver / libs base directory via a new `GetOdbcLibsBaseDir()` helper, which imports `mssql_python_odbc` when present and falls back to the bundled `mssql_python` libs when it is not. The fallback is GIL-safe and Alpine/musl-safe. - `tests/test_024_odbc_package_split.py` - verifies Python vs C++ driver-path parity. - `.gitignore` - ignores local `mssql_python_odbc/libs/` build copies and egg-info. #### VC++ runtime (`vcredist`) on Windows The Windows wheel ships `libs/windows/<arch>/vcredist/msvcp140.dll` (with its license). This is a deliberate, documented choice - **not** "parity" with the main wheel's file layout, and not a reversal of a licensing/size decision: - The main `mssql-python` wheel also ships the VC++ runtime, just relocated: `build.bat` copies `msvcp140.dll` out of `vcredist/` to the package root next to the compiled `ddbc_bindings` extension, and `setup.py` then excludes the now-duplicate `vcredist/` folder. That exclusion is **de-duplication**, not a removal of the runtime. - This package has no compiled extension, so there is no package-root `.pyd` to place the runtime beside. Keeping `msvcp140.dll` in its original `vcredist/` folder keeps the driver binaries and the runtime they were built against together, and keeps the wheel self-contained. - No Phase-2 runtime skew: `msodbcsql18.dll`'s dependency on `msvcp140.dll` is satisfied in-process by the mssql-python extension (built `/MD`, which loads `msvcp140.dll` from beside the `.pyd`), so the external-package and bundled-libs paths behave identically. This copy is completeness + attribution, which is why `GetOdbcLibsBaseDir()`'s completeness check verifies the driver + `mssql-auth.dll` but not `vcredist`. #### Phasing (no breaking change in this PR) - Phase 2 (this PR): `mssql-python` prefers the external package but keeps bundling `libs/` as a fallback, so existing installs keep working. - Phase 3 (future, v2.0.0): remove the bundled `libs/` and depend on `mssql-python-odbc` exclusively. #### Testing - `test_024` parity suite: 7 passed, 1 skipped. - Built `mssql_python_odbc-18.6.0-py3-none-win_amd64.whl` locally: correct wheel tag, `Root-Is-Purelib: false`, driver DLLs present, `vcredist` (`msvcp140.dll` + license) present, `twine check` passes. #### Notes Companion PR (release pipelines): #664. Infra registration (ESRP onboarding of `mssql-python-odbc`, ADO build-definition id) is tracked there.
… jahnvi/odbc-release-pipelines
…crosoft/mssql-python into jahnvi/odbc-release-pipelines
- Suppress DevSkim DS162092 inline on the in-container sqlcmd readiness probe (localhost is the mssql container's own loopback, not debug code). - Reorder to run the full suite with bundled libs intact (external ODBC driver is already preferred) then a decisive post-delete SELECT 1 connect. - Make pytest.ini copy mandatory and add explicit -m 'not stress'.
sumitmsft
approved these changes
Jul 20, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Work Item / Issue Reference
Summary
Adds the ADO / OneBranch release pipelines for the standalone
mssql-python-odbcpackage. This is the CI/CD half of the ODBC driver split; the package code and build script are in companion PR #663.Files
OneBranchPipelines/build-release-odbc-pipeline.yml- build orchestration. Fans out to 7 platform build stages plus a consolidate stage, with path-filtered triggers scoped tosetup_odbc.py,mssql_python_odbc/,mssql_python/libs/, and these pipeline files. SDL keeps credscan / policheck / sbom / tsa and disables binskim / apiscan / armory / codeql (data-only package, no first-party compiled binaries) with justifications.OneBranchPipelines/stages/build-odbc-windows-stage.yml- Windows amd64 / arm64.OneBranchPipelines/stages/build-odbc-macos-stage.yml- macOS universal2.OneBranchPipelines/stages/build-odbc-linux-stage.yml- manylinux_2_28 and musllinux_1_2 (x86_64 / aarch64) built in tdslibrs ACR PyPA containers. No compilation happens here; the musllinux tag is auto-detected insetup_odbc.pyviaplatform.libc_ver(), so the hardcodedMANYLINUX_TAGonly affects glibc builds.OneBranchPipelines/jobs/consolidate-odbc-artifacts-job.yml- gathers all wheels and verifies the count is exactly 7.OneBranchPipelines/official-release-odbc-pipeline.yml- ESRP-based PyPI release, gated byreleaseToPyPI. TheEsrpRelease@9step is identical to the existing mssql-python release pipeline. A preflight step fails with a clear message ifodbcBuildDefinitionIdis still the0placeholder, so the artifact download can't fail cryptically.Wheels produced (7, py3-none-)
win_amd64,win_arm64,macosx_15_0_universal2,manylinux_2_28_x86_64,manylinux_2_28_aarch64,musllinux_1_2_x86_64,musllinux_1_2_aarch64.Infra follow-ups (marked "Infra setup:" in the YAML)
mssql-python-odbcpackage name to ESRP for PyPI release.odbcBuildDefinitionIdinofficial-release-odbc-pipeline.yml(currently placeholder0; the release pipeline now preflight-checks this and errors clearly if left at0), and confirmpipelines.sourcematches the registered definition name.ESRP Federated Creds (AME)variable group.Notes
These pipelines are inert until they are registered as ADO build definitions; merging the YAML alone runs nothing. Pairs with #663.