Skip to content

fix(cli): resolve subpackage register paths before deduplication#2428

Open
Anai-Guo wants to merge 1 commit intohuggingface:mainfrom
Anai-Guo:fix/cli-symlink-dedupe
Open

fix(cli): resolve subpackage register paths before deduplication#2428
Anai-Guo wants to merge 1 commit intohuggingface:mainfrom
Anai-Guo:fix/cli-symlink-dedupe

Conversation

@Anai-Guo
Copy link
Copy Markdown

Problem

Closes #2417.

On RHEL / CentOS-derived distributions lib64 is a symlink to lib, so both site-packages directories end up in commands_register_spec.submodule_search_locations:

/home/user/.venv/lib64/python3.12/site-packages
/home/user/.venv/lib/python3.12/site-packages

register_optimum_cli_subcommands deduplicates that list with a plain set(...), which compares the raw path strings. Since the strings differ, the same register module is imported twice — once via lib, once via lib64. The duplicate REGISTER_COMMANDS entries then try to register identical subparsers and any optimum-cli invocation that pulls in a subpackage (e.g. optimum-cli export onnx --help) blows up with:

argparse.ArgumentError: argument {onnx}: conflicting subparser: onnx

Ubuntu / Debian only ship lib, so the bug never surfaced there. The reporter narrowed it down on CentOS Stream 10; it also reproduces on Fedora, RHEL 8/9 and any virtualenv created on those distros.

Fix

Resolve each candidate path with Path.resolve() before building the deduplication set. Symlinked siblings collapse to one canonical absolute path, so the registration loop runs exactly once per real directory.

register_paths = {
    Path(register_path).resolve()
    for register_path in commands_register_spec.submodule_search_locations
}
for register_path in register_paths:
    ...

The rest of the loop is unchanged — register_path was already being wrapped in Path(...) on the first line of the body, so the resolved Path object is a drop-in replacement.

Why this is safe

  • No behavior change on Ubuntu / Debian / Windows / macOSsubmodule_search_locations contains a single entry, Path.resolve() returns it unchanged.
  • No new dependencypathlib.Path is already imported at the top of the module.
  • No double-iteration — the comprehension materializes the set before the loop begins, matching the previous semantics.
  • commands_register_spec.submodule_search_locations is None short-circuit is preserved (lines above the change).

Reproduction (matches the issue)

# CentOS Stream 10 / Fedora / RHEL 8+
python3 -m venv .venv && source .venv/bin/activate
pip install optimum optimum-onnx
python -c "import sys; [print(p) for p in sys.path if 'site-packages' in p]"
# -> shows both lib and lib64 paths
optimum-cli export onnx --help
# Before: argparse.ArgumentError: conflicting subparser: onnx
# After:  prints the help text

🤖 Generated with Claude Code

On RHEL/CentOS-derived distros lib64 is a symlink to lib, so both paths appear in submodule_search_locations. The previous set() compared raw strings and treated the symlink siblings as distinct entries, causing each register module to be imported twice and raising 'argparse.ArgumentError: conflicting subparser' at optimum-cli startup.

Resolving each path with Path.resolve() before building the set lets symlinked locations collapse to a single canonical entry, so the registration loop runs once per real directory.

Fixes huggingface#2417
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Cannot run sub-commands on RHL because optimum mishandles the lib64 symlink to lib

1 participant