[ci] Install libyang3 + python3-libyang alongside libyang1 by yutongzhang-microsoft · Pull Request #1194 · sonic-net/sonic-swss-common

yutongzhang-microsoft · 2026-05-22T02:24:17Z

Why I did it

sonic-net/sonic-buildimage#26584 ported sonic-yang-mgmt from the libyang1 Python bindings (import yang) to the libyang3 Python bindings (import libyang) on 2026-05-19. After that merge the sonic-buildimage common-lib artifact ships a sonic_yang_mgmt-1.0-py3-none-any.whl whose top-level import is now import libyang as ly.

Every swss-common build started failing the next day (master 20260521.1 / 20260521.2, plus every open PR — #1180, #1188, #1191, #1192, #1193, ...) with:

File "/usr/local/lib/python3.10/dist-packages/sonic_yang.py", line 3, in <module>
    import libyang as ly
ModuleNotFoundError: No module named 'libyang'
make[2]: *** [Makefile:3670: common/cfg_schema.h] Error 1

because our CI still only downloads / installs the libyang v1 artifacts (libyang_1.0*.deb, libyang-*_1.0*.deb, libyang-cpp_*.deb, python3-yang_*.deb) while the freshly built sonic_yang_mgmt wheel now requires the libyang3 C library and the libyang3 Python binding (python3-libyang) to import.

How I did it

Pull libyang3 (libyang3_3*.deb) and the libyang3 Python binding (python3-libyang_*.deb) in parallel with the existing libyang1 artifacts in every place we touch the Azure.sonic-buildimage.common_libs pipeline:

azure-pipelines.yml — ubuntu-22.04 build job (explicit dpkg install path)
.azure-pipelines/build-template.yml — both the bullseye/bookworm and trixie download tasks (the install step already uses find ./download -name *.deb, so just adding the patterns is enough)
.azure-pipelines/build-sairedis-template.yml
.azure-pipelines/build-swss-template.yml
.azure-pipelines/test-docker-sonic-vs-template.yml (explicit dpkg install path)

libyang v1 is kept side-by-side so existing C/C++ link-time deps (libyang-cpp, libyang_1.0) and any downstream consumer that still loads the v1 Python binding keep working — this mirrors the side-by-side approach sonic-buildimage#26584 took in rules/swss-common.mk (LIBYANG + LIBYANG3 both listed in _DEPENDS / _RDEPENDS).

Only the runtime libyang3 libs are pulled in; libyang3-dev is intentionally not installed because the libyang Makefile declares it as a conflict against libyang-dev (the v1 dev package), and the C++ compile of swss-common still links against libyang1 — there is no need for v3 headers in this CI.

How to verify it

Trigger Azure.sonic-swss-common on this PR. The Compile sonic swss common and Compile sonic swss common with coverage enabled jobs should no longer fail at gen_cfg_schema.py; the build should reach the linker and the unit-test stages.

sonic-net/sonic-buildimage#26584 ported sonic-yang-mgmt from the libyang1 Python bindings (import yang) to the libyang3 Python bindings (import libyang). After that merge, every swss-common build (including master) dies at the gen_cfg_schema.py step with: File ".../sonic_yang.py", line 3, in <module> import libyang as ly ModuleNotFoundError: No module named 'libyang' make[2]: *** [Makefile:3670: common/cfg_schema.h] Error 1 because the CI still only downloads / installs the libyang v1 artifacts (libyang_1.0*.deb, libyang-cpp_*.deb, python3-yang_*.deb) while the freshly built sonic_yang_mgmt wheel now requires the libyang3 C library + python3-libyang binding to import. Pull libyang3 (libyang3_3*.deb) and the libyang3 Python binding (python3-libyang_*.deb) in parallel with the existing libyang1 artifacts in every place we touch the common-lib pipeline: - azure-pipelines.yml (ubuntu-22.04 build job) - .azure-pipelines/build-template.yml (both bullseye/bookworm and trixie download tasks) - .azure-pipelines/build-sairedis-template.yml - .azure-pipelines/build-swss-template.yml - .azure-pipelines/test-docker-sonic-vs-template.yml libyang1 is kept side-by-side so existing C/C++ link-time deps (libyang-cpp, libyang_1.0) and any consumer that still loads the v1 Python binding keep working. Only the runtime libs are pulled in; libyang3-dev is intentionally not installed because it conflicts with the libyang1 -dev package and is not needed for the build. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Yutong Zhang <yutongzhang@microsoft.com>

mssonicbld · 2026-05-22T02:24:25Z

/azp run

azure-pipelines · 2026-05-22T02:24:39Z

Azure Pipelines successfully started running 1 pipeline(s).

Copilot

Pull request overview

Updates the Azure Pipelines CI setup for sonic-swss-common to install libyang v3 runtime packages (and Python bindings) alongside the existing libyang v1 packages, unblocking gen_cfg_schema.py after sonic_yang_mgmt switched to import libyang.

Changes:

Download libyang3_3*.deb and python3-libyang_*.deb from Azure.sonic-buildimage.common_libs artifacts in all relevant pipeline templates.
Install the libyang3 runtime + Python bindings in jobs that use explicit dpkg -i paths.
Keep libyang v1 packages installed side-by-side to preserve existing v1 C/C++ and Python dependencies.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
`azure-pipelines.yml`	Adds libyang3 + python3-libyang to artifact download and installs them in the Ubuntu 22.04 build job.
`.azure-pipelines/build-template.yml`	Extends common-lib download patterns (bullseye/bookworm + trixie paths) so containerized builds pick up libyang3 and python3-libyang.
`.azure-pipelines/build-sairedis-template.yml`	Extends common-lib download patterns so sairedis builds also install libyang3 runtime and Python bindings.
`.azure-pipelines/build-swss-template.yml`	Extends common-lib download patterns so swss builds also install libyang3 runtime and Python bindings.
`.azure-pipelines/test-docker-sonic-vs-template.yml`	Adds libyang3 + python3-libyang to artifact download and installs them for VS test job setup.

Follow-up to the first push on this branch. The container-based jobs in build-template.yml are bookworm (Python 3.11) and the python3-libyang_3.1.0-1 .deb installs cleanly there. The host job in azure-pipelines.yml runs on the Microsoft-hosted ubuntu-22.04 image, which ships Python 3.10. The bookworm .deb's metadata declares `Depends: python3 (>= 3.11~)`, so dpkg refuses to configure it on Python 3.10: dpkg: dependency problems prevent configuration of python3-libyang: python3-libyang depends on python3 (>= 3.11~); however: Version of python3 on system is 3.10.6-1~22.04.1. Switch the ubuntu-22.04 host install (azure-pipelines.yml and test-docker-sonic-vs-template.yml) to install only the libyang3 C library .deb and pin the Python binding via PyPI (`pip3 install libyang==3.1.0`). CESNET ships manylinux wheels on PyPI for cp310, so this works on Python 3.10 without needing libyang3-dev (which would conflict with libyang-dev v1 that the C++ compile still uses). The containerized build-template.yml and the linker-only build-sairedis-template.yml / build-swss-template.yml are unchanged -- they only need the libyang3 system library. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Yutong Zhang <yutongzhang@microsoft.com>

mssonicbld · 2026-05-22T02:34:21Z

/azp run

azure-pipelines · 2026-05-22T02:34:34Z

Azure Pipelines successfully started running 1 pipeline(s).

Previous attempt used `pip3 install libyang==3.1.0` expecting a binary wheel, but PyPI only publishes an sdist. The sdist build then failed with: Exception: Version mismatch: this is the 'cffi' package version 2.0.0 ... we get version 1.15.0, located in '/usr/lib/python3/ dist-packages/_cffi_backend.cpython-310-x86_64-linux-gnu.so'. That's because pip's build isolation pulled cffi 2.0 into an overlay while the system _cffi_backend.so is still 1.15. Switch to building libyang's sdist against a private sysroot extracted (not installed) from libyang-dev_3*.deb, with --no-build-isolation so the system python3-cffi 1.15 is used end- to-end. We can't dpkg-install libyang-dev_3*.deb system-wide because it has the same package name as the libyang1 libyang-dev that swss-common's C++ code still needs. Apply the same approach to .azure-pipelines/test-docker-sonic-vs- template.yml. Containerized jobs in build-template.yml stay on the bookworm python3-libyang.deb path since they have Python 3.11. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Yutong Zhang <yutongzhang@microsoft.com>

mssonicbld · 2026-05-22T02:56:14Z

/azp run

azure-pipelines · 2026-05-22T02:56:26Z

Azure Pipelines successfully started running 1 pipeline(s).

After the previous fix, `Install yang deb` succeeded but `Compile sonic swss common` failed at gen_cfg_schema.py: ImportError: /usr/local/lib/python3.10/dist-packages/ _libyang.cpython-310-x86_64-linux-gnu.so: undefined symbol: lyplg_ext_parse_log Root cause: the linker resolved `-lyang` to libyang1's `libyang.so` (from libyang-dev_1.0). Our sysroot extracted from libyang-dev_3*.deb only contained the `libyang.so` symlink, not the actual `libyang.so.3` it points to (that file lives in libyang3_3*.deb, installed system-wide). Because the relative symlink target didn't exist in the sysroot, the linker silently fell through to /usr/lib/x86_64-linux-gnu/libyang.so which the libyang1 libyang-dev had already populated as a v1 symlink. Result: _libyang.so linked against v1 ABI and missed the v3 `lyplg_ext_parse_log` symbol at dlopen. Fix: also dpkg-deb -x libyang3_3*.deb into the same sysroot so the relative `libyang.so` -> `libyang.so.3` symlink resolves inside the sysroot and the linker picks up the v3 SONAME. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Yutong Zhang <yutongzhang@microsoft.com>

mssonicbld · 2026-05-22T03:04:06Z

/azp run

azure-pipelines · 2026-05-22T03:04:19Z

Azure Pipelines successfully started running 1 pipeline(s).

After the libyang3 install + swss-common compile started passing, the downstream BuildSwss stage (which rebuilds sonic-swss against the just-built libswsscommon) failed with: /usr/bin/ld: cannot find -lnexthopgroup: No such file or directory sonic-swss's own build-template.yml downloads `libnexthopgroup_*.deb` and `libnexthopgroup-dev_*.deb` from common-lib because fpmsyncd now links `-lnexthopgroup`. The swss-common pipeline's build-swss-template.yml did not download those, so when it builds sonic-swss the linker can't resolve the symbol. Add the two patterns to the common-lib download step. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Yutong Zhang <yutongzhang@microsoft.com>

mssonicbld · 2026-05-22T04:05:47Z

/azp run

azure-pipelines · 2026-05-22T04:06:01Z

Azure Pipelines successfully started running 1 pipeline(s).

BuildTrixie failed with:: ModuleNotFoundError: No module named 'libyang' at gen_cfg_schema.py because the trixie phase pins to common-lib pipelineId 926659, which doesn't ship python3-libyang_*.deb (it was added by sonic-net/sonic-buildimage#26584 only on the bookworm path, and trixie's pin is older). Debian trixie itself ships python3-libyang natively, so just apt-install it as a fallback when the deb isn't in the downloaded common-lib artifact. Guarded behind `dpkg -s` so we don't run apt unnecessarily once the artifact catches up. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Yutong Zhang <yutongzhang@microsoft.com>

mssonicbld · 2026-05-22T05:39:08Z

/azp run

azure-pipelines · 2026-05-22T05:39:21Z

Azure Pipelines successfully started running 1 pipeline(s).

Previous attempt fell back to `apt-get install python3-libyang` on trixie, but trixie's apt repo doesn't have that package: E: Unable to locate package python3-libyang Use the same approach as the ubuntu-22.04 host: build the CESNET libyang Python binding from PyPI sdist against a private sysroot extracted from libyang3 + libyang-dev_3*.deb. Also pull libyang-dev_3*.deb from common-lib for both bookworm and trixie download patterns, and exclude it from the dpkg install because it has the same package name (`libyang-dev`) as the libyang1 libyang-dev that the C++ compile still needs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Yutong Zhang <yutongzhang@microsoft.com>

mssonicbld · 2026-05-22T06:24:55Z

/azp run

azure-pipelines · 2026-05-22T06:25:08Z

Azure Pipelines successfully started running 1 pipeline(s).

The previous sysroot+pip fallback hardcoded x86_64-linux-gnu in LDFLAGS, which broke armhf/arm64 trixie jobs: the -L path didn't exist on those arches, so the linker silently fell back to the system /usr/lib/<triple>/libyang.so from libyang1, producing a v1-ABI _libyang*.so that fails at runtime with:: ImportError: .../_libyang.cpython-313-arm-linux-gnueabihf.so: undefined symbol: lyplg_ext_parse_log Compute the multiarch triple dynamically via `dpkg-architecture -qDEB_HOST_MULTIARCH` (falling back to `gcc -print-multiarch`) so the link picks up the v3 SONAME from our extracted sysroot regardless of host arch. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Yutong Zhang <yutongzhang@microsoft.com>

mssonicbld · 2026-05-22T07:25:11Z

/azp run

azure-pipelines · 2026-05-22T07:25:25Z

Azure Pipelines successfully started running 1 pipeline(s).

bhouse-nexthop · 2026-05-23T17:38:07Z

I've hit quite a few build issues that I guess need to be resolved before the libyang3 port can actually merge #973 ... This looks reasonable, if not ugly, but at least its temporary. It looks like you'd need a change similar to 1431c3d#diff-5e23c1bacebc44187fd782c77bacbcb5e811fba1840a90fbab3ddcb03f486559 for your last failure there, but unfortunately I think you'll find when you rebuild you're going to hit a few more issues.

zmq: prevent use-after-free of ZmqConsumerStateTable handler #1196 -- but is intermittent
ci: download libnexthopgroup deb for the sonic-swss build stage #1197 -- broken by sonic-net/sonic-swss@e6c482ca

Then after that, I hit things that I'm not sure how to resolve, they seem like regressions in sonic-swss itself: https://dev.azure.com/mssonic/build/_build/results?buildId=1120488&view=logs&j=c5067a1a-605b-593b-ff36-aff34370f9a0&t=97ee0541-39b3-5849-9492-51357ad8f087&s=ee3800fd-6e81-525f-e564-94108585217d

bhouse-nexthop · 2026-05-24T07:03:46Z

https://dev.azure.com/mssonic/build/_build/results?buildId=1120488&view=logs&j=c5067a1a-605b-593b-ff36-aff34370f9a0&t=97ee0541-39b3-5849-9492-51357ad8f087&s=ee3800fd-6e81-525f-e564-94108585217d

I think sonic-net/sonic-swss#4599 should fix that above remaining issue.

vstest pool's `Install dependencies` failed building the libyang Python sdist:: /tmp/libyang3-sysroot/usr/include/libyang/tree_data.h:32:10: fatal error: pcre2.h: No such file or directory libyang3's public headers `#include <pcre2.h>`. The Microsoft-hosted ubuntu-22.04 image and the bookworm/trixie slave containers happen to ship `libpcre2-dev` preinstalled, but the self-hosted vstest pool doesn't. Add it explicitly to the apt list in all three sysroot+pip fallbacks so the cffi build of the Python binding can compile against libyang3 on every pool. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Yutong Zhang <yutongzhang@microsoft.com>

mssonicbld · 2026-05-25T01:45:43Z

/azp run

azure-pipelines · 2026-05-25T01:45:56Z

Azure Pipelines successfully started running 1 pipeline(s).

prsunny · 2026-05-26T16:22:17Z

@qiluo-msft , would you review/merge this PR?

radha-danda · 2026-05-27T08:13:58Z

@prsunny, @qiluo-msft, there seems a dependent PR sonic-net/sonic-swss#4599 to have
"[Azure.sonic-swss-common (Test vstest)](https://github.com/sonic-net/sonic-swss-common/pull/1194/checks?check_run_id=77679729482)Failing after 80m" to pass"

Please review the dependent PR sonic-net/sonic-swss#4599.
Kindly refer to @bhouse-nexthop comments from the PR

radha-danda · 2026-05-27T08:15:34Z

@yutongzhang-microsoft, Please resolve merge conflicts.

lguohan · 2026-05-28T05:38:47Z

obseleted by #973

Copilot AI review requested due to automatic review settings May 22, 2026 02:24

Copilot started reviewing on behalf of yutongzhang-microsoft May 22, 2026 02:24 View session

Copilot AI reviewed May 22, 2026

View reviewed changes

bhouse-nexthop self-requested a review May 23, 2026 17:38

bhouse-nexthop previously approved these changes May 23, 2026

View reviewed changes

yutongzhang-microsoft dismissed bhouse-nexthop’s stale review via 470ad49 May 25, 2026 01:45

lguohan approved these changes May 26, 2026

View reviewed changes

lguohan mentioned this pull request May 27, 2026

sonic-swss-common: port to libyang3 #973

Merged

lguohan closed this May 28, 2026

Conversation

yutongzhang-microsoft commented May 22, 2026

Why I did it

How I did it

How to verify it

Related

Uh oh!

mssonicbld commented May 22, 2026

Uh oh!

azure-pipelines Bot commented May 22, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

mssonicbld commented May 22, 2026

Uh oh!

azure-pipelines Bot commented May 22, 2026

Uh oh!

mssonicbld commented May 22, 2026

Uh oh!

azure-pipelines Bot commented May 22, 2026

Uh oh!

mssonicbld commented May 22, 2026

Uh oh!

azure-pipelines Bot commented May 22, 2026

Uh oh!

mssonicbld commented May 22, 2026

Uh oh!

azure-pipelines Bot commented May 22, 2026

Uh oh!

mssonicbld commented May 22, 2026

Uh oh!

azure-pipelines Bot commented May 22, 2026

Uh oh!

mssonicbld commented May 22, 2026

Uh oh!

azure-pipelines Bot commented May 22, 2026

Uh oh!

mssonicbld commented May 22, 2026

Uh oh!

azure-pipelines Bot commented May 22, 2026

Uh oh!

bhouse-nexthop commented May 23, 2026

Uh oh!

bhouse-nexthop commented May 24, 2026

Uh oh!

mssonicbld commented May 25, 2026

Uh oh!

azure-pipelines Bot commented May 25, 2026

Uh oh!

prsunny commented May 26, 2026

Uh oh!

radha-danda commented May 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

radha-danda commented May 27, 2026

Uh oh!

lguohan commented May 28, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

radha-danda commented May 27, 2026 •

edited

Loading