feat: add native C extension support for numpy, pandas, Pillow, rapidfuzz, wordcloud#645
Open
ppenna wants to merge 1 commit into
Open
feat: add native C extension support for numpy, pandas, Pillow, rapidfuzz, wordcloud#645ppenna wants to merge 1 commit into
ppenna wants to merge 1 commit into
Conversation
This was referenced May 14, 2026
There was a problem hiding this comment.
Pull request overview
Adds CPython-side glue to expose numpy, pandas, Pillow, rapidfuzz, and wordcloud C/C++ extensions as statically-linked builtin modules for the Nanvix i686 target. Each extension gets a _*_builtin.c shim that re-exports its PyInit_* under a flat name matching Modules/Setup.local, plus a .nanvix/<pkg>.py helper that generates the Setup.local entries (and for some packages, stages the corresponding Python sources into the sysroot). .nanvix/build.py now aggregates these into a single Setup.local generator.
Changes:
- Add 55 new
_*_builtin.cshim files inModules/that bridge each upstreamPyInit_*to a flat builtin name. - Add
.nanvix/{numpy_mod,pandas_mod,pillow,rapidfuzz,wordcloud}.pybuild helpers and refactor.nanvix/lxml.pyto expose agenerate_setup_local_linesAPI used by a new combined generator in.nanvix/build.py. - Remove
nanvix.lockfrom.nanvix/.gitignore.
Show a summary per file
| File | Description |
|---|---|
| Modules/_np_multiarray_umath_builtin.c | numpy core builtin shim |
| Modules/pd*_builtin.c (44 files) | pandas builtin shims for _libs, tslibs, window |
| Modules/_pil_imaging{,math,morph}_builtin.c | Pillow builtin shims |
| Modules/rf*_builtin.c (7 files) | rapidfuzz builtin shims |
| .nanvix/numpy_mod.py | Generates numpy Setup.local lines; defines (uncalled) staging helper |
| .nanvix/pandas_mod.py | Generates pandas Setup.local lines for 44 modules |
| .nanvix/pillow.py | Generates Pillow Setup.local lines |
| .nanvix/rapidfuzz.py | Generates rapidfuzz Setup.local lines and (uncalled) staging + Python shim writer |
| .nanvix/wordcloud.py | Generates wordcloud Setup.local entry referencing a missing C shim; (uncalled) staging helper |
| .nanvix/lxml.py | Refactored to expose generate_setup_local_lines |
| .nanvix/build.py | Aggregates Setup.local lines from all extension helpers |
| .nanvix/.gitignore | Drops nanvix.lock from ignored entries |
Copilot's findings
Tip
Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- Files reviewed: 63/63 changed files
- Comments generated: 3
Comment on lines
+14
to
+22
|
|
||
| _SETUP_LOCAL_LINES = """\ | ||
| # wordcloud Cython extension module (statically linked via pre-built archive). | ||
| _wc_query_integral_image _wc_query_integral_image_builtin.c -L{sysroot}/lib -lquery_integral_image | ||
| """ | ||
|
|
||
|
|
||
| def generate_setup_local_lines(sysroot: Path) -> str: | ||
| """Return Setup.local lines for wordcloud modules.""" |
Comment on lines
+35
to
+62
| def stage_rapidfuzz_runtime(repo_root: Path, sysroot: Path) -> None: | ||
| """Copy rapidfuzz Python files from buildroot into the sysroot. | ||
|
|
||
| Looks for rapidfuzz in ``.nanvix/buildroot/python-packages/rapidfuzz/``. | ||
| Then writes Python shim modules that bridge the flat builtin names | ||
| to the expected ``rapidfuzz.*`` / ``rapidfuzz.distance.*`` import paths. | ||
| """ | ||
| rf_src = repo_root / ".nanvix" / "buildroot" / "python-packages" / "rapidfuzz" | ||
| if not rf_src.is_dir(): | ||
| print( | ||
| f"[rapidfuzz] Python package not found at {rf_src}; " | ||
| "skipping runtime staging." | ||
| ) | ||
| return | ||
|
|
||
| py_lib = sysroot / "lib" / config.PYTHON_LIB_DIR | ||
| if not py_lib.is_dir(): | ||
| raise RuntimeError(f"Python runtime library directory is missing: {py_lib}") | ||
|
|
||
| dst = py_lib / "rapidfuzz" | ||
| if dst.exists(): | ||
| shutil.rmtree(dst) | ||
| shutil.copytree(rf_src, dst) | ||
|
|
||
| # Write Python shims that bridge flat builtin names to package paths | ||
| _write_shims(dst) | ||
|
|
||
| print(f"[rapidfuzz] Staged {rf_src} -> {dst}") |
| black.toml | ||
| env.json | ||
| nanvix.lock | ||
| pyrightconfig.json |
ppenna
force-pushed
the
feature/native-extensions
branch
3 times, most recently
from
b232c31 to
e9aec55
Compare
…fuzz, wordcloud Add builtin shim modules and build infrastructure for cross-compiling Python C extensions as statically-linked modules for Nanvix (i686): - numpy: 1 module (_np_multiarray_umath) - pandas: 44 modules (_pd_* covering _libs, tslibs, window, parsers, etc.) - Pillow: 3 modules (_pil_imaging, _pil_imagingmath, _pil_imagingmorph) - rapidfuzz: 6 modules (_rf_* covering fuzz, distance metrics, etc.) - wordcloud: uses existing lxml build infrastructure Build helpers added: - numpy_mod.py: numpy extension build configuration - pandas_mod.py: pandas extension build configuration - pillow.py: Pillow extension build configuration - rapidfuzz.py: rapidfuzz extension build configuration - wordcloud.py: wordcloud extension build configuration Each C extension is wrapped in a _*_builtin.c shim that bridges the PyInit function to a flat builtin name for CPython's inittab. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
ppenna
force-pushed
the
feature/native-extensions
branch
from
e9aec55 to
3560d6b
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add builtin shim modules and build infrastructure for cross-compiling Python C extensions as statically-linked modules for Nanvix (i686).
Extensions added
_np_multiarray_umath— core array/math engine_pd_*— tslibs, parsers, window, ops, etc._pil_imaging,_pil_imagingmath,_pil_imagingmorph_rf_*— fuzz, distance metrics, feature detectionArchitecture
Each C extension is wrapped in a
_*_builtin.cshim that bridges thePyInit_Xfunction to a flat builtin name for CPython'sinittab. Build helpers (.nanvix/*.py) generateSetup.localentries and link flags.Result
CPython now has 135 built-in modules (up from 91), producing a 33MB
python.elf.Benchmarks (host-side wall clock)
Related PRs