fix: access built-in system Python instead of using venv for runtime by deanq · Pull Request #30 · runpod-workers/flash

deanq · 2025-09-17T15:32:31Z

#24 is a prerequisite
See docs/System_Python_Runtime_Architecture.md

Summary

This PR refactors the worker architecture to use built-in system Python instead of virtual environments for runtime execution. This change addresses PyTorch installation issues (AE-1165) and simplifies the deployment model by removing persistent volume workspace dependencies.

Key Changes

🔧 System Python Runtime: Switch from venv-based execution to direct system Python usage
🐳 Simplified Docker: Single-stage builds instead of multi-stage for both GPU and CPU images
🎯 Environment Detection: Smart handling of Docker vs local development environments
⚡ Streamlined Dependencies: Direct system installation with environment-aware configuration

Problem Solved

Fixes AE-1165 where PyTorch installations were failing due to complex venv interactions in the containerized RunPod environment. The persistent volume workspace approach was causing dependency conflicts and installation failures.

Technical Details

Architecture Changes

Removed: /runpod-volume persistent workspace management
Added: Environment-aware dependency installation (docker vs local)
Centralized: All subprocess operations through run_logged_subprocess
Simplified: Direct system Python package installation

Docker Optimization

GPU Image: Single-stage build with PyTorch base
CPU Image: Single-stage build with Python slim base
Installation: Direct system package installation via uv pip install --system

Breaking Changes

⚠️ Architecture Change: This removes persistent volume workspace support. Functions will no longer have persistent package state between executions.

Testing

All existing handler tests pass (make test-handler)
Environment detection works in both Docker and local contexts
PyTorch installation verified in containerized environment
Quality checks pass (make quality-check)

Migration Notes

For existing deployments:

No user-facing API changes - @remote decorator usage remains identical
Package installation may be slightly slower without persistent caching
Container startup should be faster due to simplified architecture

Test Plan

Verify make test-handler passes all test cases
Test PyTorch-specific workloads that previously failed
Validate both GPU and CPU Docker images
Confirm environment detection in RunPod Serverless

🤖 Generated with Claude Code

Add core download acceleration modules with aria2c integration: - download_accelerator.py: Main acceleration classes with multi-connection downloads - huggingface_accelerator.py: Specialized HF model acceleration - constants.py: Download acceleration configuration constants - __init__.py: Package structure for src module

Enhanced dependency installation with intelligent acceleration: - Auto-detects large packages for acceleration (torch, transformers, etc.) - Integrates with remote executor for acceleration control - Maintains backward compatibility with existing workflows - Provides graceful fallback when aria2c unavailable

Enhanced workspace manager with HuggingFace model pre-caching: - Pre-cache specified HF models before function execution - Integrates with volume-aware caching system - Optimizes cold start times for ML workloads

Comprehensive test suite for download acceleration: - Integration tests for aria2 detection and fallback behavior - HF model acceleration testing with authentication - Volume-aware acceleration scenarios - Error handling and performance validation

- Update test files moved to src/ directory - Enhanced test coverage for acceleration features - Updated dependencies and documentation - Submodule updates for tetra-rp

- Added nala accelerated installation for large system packages - Enhanced DependencyInstaller with automatic nala fallback to apt-get - Updated Docker images to include nala package manager - Added comprehensive system package acceleration tests - Improved acceleration logging with system package status

Simplify dependency installation by removing aria2c acceleration for Python packages. UV's built-in parallel downloading and caching is superior and eliminates the need for additional complexity. Changes: - Remove LARGE_PACKAGE_PATTERNS from constants.py - Simplify DependencyInstaller.install_dependencies() to single parameter - Remove Python package acceleration logic and related methods - Update RemoteExecutor to use simplified API - Update tests to match new simplified interface System package acceleration (nala) and HuggingFace model acceleration remain intact as they provide meaningful performance benefits over standard tools. Core functionality verified: - All handler tests pass (8/8) - All unit tests pass (98/98) - Code quality checks pass (format, lint, typecheck)

Add conditional acceleration logic - passes accelerate_downloads to installers, HF model caching only when accelerated + models specified

…bled Implement _install_with_pip() method and route between UV (accelerated) vs pip (standard) based on accelerate_downloads parameter

Add HfXetDownloader for subsequent downloads, implement smart strategy: hf_xet for cached files → hf_transfer for fresh downloads → fallback

Add tests for both acceleration enabled/disabled scenarios, verify UV vs pip routing, update existing test assertions

Update test expectations to handle accelerate_downloads parameter in integration scenarios

Update build files and dependency locks to support new acceleration functionality

Always use UV for Python package installation regardless of acceleration setting. The _install_with_pip method has been removed as UV provides more reliable virtual environment handling and package management. - Remove _install_with_pip() method (70 lines) - Simplify install_dependencies() to always use UV - Maintain differential installation when acceleration is enabled

Update dependency installer tests to reflect the removal of pip support: - Fix test_install_dependencies_with_acceleration_disabled to expect UV - Rename test_install_dependencies_pip_failure to test_install_dependencies_uv_failure - Update assertions to check for "uv pip" commands - Update test descriptions and expected error messages All tests now correctly validate UV-only package installation behavior.

Rename test_pip_no_acceleration.json to test_uv_no_acceleration.json and update content to reflect UV-only package installation: - Update function name from test_pip_installation_without_acceleration to test_uv_installation_without_acceleration - Update success message to reference UV instead of pip - Maintain same test logic for package import validation This test validates that packages installed with accelerate_downloads=False are properly available using UV package manager.

Add parallel installation of dependencies when acceleration is enabled: - Add async wrappers for dependency and model download methods - Implement _install_dependencies_parallel() using asyncio.gather() - Add _install_dependencies_sequential() for non-accelerated path - Add _process_parallel_results() for error handling - Route between parallel/sequential execution based on accelerate_downloads flag When accelerate_downloads=True, system packages, Python packages, and HF model downloads execute concurrently for improved performance.

Add accelerate_model_download_async() method to WorkspaceManager to support parallel execution of model downloads when acceleration is enabled. This async wrapper allows HF model downloads to run concurrently with dependency installations for improved performance.

Update test mocks and expectations for parallel execution implementation: - Fix AsyncMock setup for async dependency installation methods - Update test_dependency_management.py for async method calls - Update test_download_acceleration_integration.py for parallel execution - Update test_remote_executor.py with proper AsyncMock usage All tests now properly mock async methods and validate parallel execution behavior when acceleration is enabled.

- Remove 4 obsolete test files (debug logging, subprocess debug, vLLM symlink, redundant HF) - Add 6 new comprehensive test files covering advanced functionality: * test_system_dependencies.json - System package installation * test_class_persistence.json - Instance reuse with instance_id * test_function_args.json - Serialized arguments/kwargs testing * test_mixed_dependencies.json - Combined system + Python dependencies * test_class_custom_method.json - Custom method execution * test_error_scenarios.json - Error handling and edge cases - Update CLAUDE.md to fix test file location references Total test coverage: 11 files (was 5) covering all handler functionality

- Remove custom HfXetDownloader class (~160 lines) - now redundant - Update huggingface_hub requirement to >=0.32.0 for automatic hf_xet - Leverage HF Hub's native snapshot_download() with transparent acceleration - Simplify HuggingFaceAccelerator to use HF's built-in caching and Xet support - Update workspace_manager to trust HF's cache hierarchy (HF_HOME only) - Remove manual Xet detection and file-by-file download logic - Update tests to reflect native HF Hub integration approach - Add documentation for automatic HF acceleration features Benefits: - Automatic chunk-level deduplication via native hf_xet integration - Simplified codebase with 332 fewer lines of redundant code - Better performance using HF's battle-tested acceleration - Future-proof - automatically works with new Xet-enabled repos - Transparent operation - no code changes needed for acceleration

- Add strategy pattern for HF model downloads with tetra and native implementations - Implement model pattern matching for selective acceleration - Add comprehensive test coverage for download strategies - Integrate with existing workspace and cache management systems

…rchs-not-found

test-handler should always use env python

jhcipar · 2025-09-22T20:33:41Z

If I'm understanding this correctly, it seems like one downside here is that losing the persistent volume for dependency management means that net new workers will have overall worse initial starts (because w/o venv persistence each new worker has to install deps), but then better subsequent cold starts because they're not running a venv off network storage?

deanq · 2025-09-22T23:59:26Z

If I'm understanding this correctly, it seems like one downside here is that losing the persistent volume for dependency management means that net new workers will have overall worse initial starts (because w/o venv persistence each new worker has to install deps), but then better subsequent cold starts because they're not running a venv off network storage?

@jhcipar Right. This seemed like a great idea until I spent time benchmarking every example we had. They were all 4x slower due to the network lag between disk + memory operations. This PR-25 completely removes the use of network volumes as runtimes, relegating them to warm cache storage only.

…rchs-not-found

…age)

deanq added 30 commits August 15, 2025 17:05

feat: add workspace acceleration support

046eb58

Enhanced workspace manager with HuggingFace model pre-caching: - Pre-cache specified HF models before function execution - Integrates with volume-aware caching system - Optimizes cold start times for ML workloads

chore: moved test-handler files to src/

ce51390

feat: runtime uses aria2 for accelerated parallel downloads

6c04de1

chore: update project structure and dependencies

66eb286

- Update test files moved to src/ directory - Enhanced test coverage for acceleration features - Updated dependencies and documentation - Submodule updates for tetra-rp

chore: updated tetra-rp

1930b4b

build: local-execution-test use make test-handler

731fd56

chore: update CLAUDE.md

e829140

chore: move these values to constants.py for maintainability

104b2da

test: uv is no longer part of download accelerator

d7c996d

feat: implement accelerate_downloads parameter logic in RemoteExecutor

2ab93e3

Add conditional acceleration logic - passes accelerate_downloads to installers, HF model caching only when accelerated + models specified

feat: add pip fallback for Python dependencies when acceleration disa…

b50a7bf

…bled Implement _install_with_pip() method and route between UV (accelerated) vs pip (standard) based on accelerate_downloads parameter

feat: enhance HF model caching with hf_transfer/hf_xet strategy

440d00d

Add HfXetDownloader for subsequent downloads, implement smart strategy: hf_xet for cached files → hf_transfer for fresh downloads → fallback

test: add comprehensive coverage for accelerate_downloads parameter

0320e4d

Add tests for both acceleration enabled/disabled scenarios, verify UV vs pip routing, update existing test assertions

test: update integration tests for new acceleration parameter

034f770

Update test expectations to handle accelerate_downloads parameter in integration scenarios

chore: update dependencies and constants for download acceleration

9531079

Update build files and dependency locks to support new acceleration functionality

chore: memory correction

e1db417

deanq added 5 commits September 17, 2025 18:20

chore: vscode config to point to src

b980e51

chore: no need for a NALA_CHECK_CMD constant

0bda8a1

build: update submodule

177c3d2

chore: logs namespace is now just tetra

d365f2a

Merge branch 'deanq/ae-962-log-streaming' into deanq/ae-1165-bug-pyto…

8b2cf5f

…rchs-not-found

deanq marked this pull request as ready for review September 18, 2025 01:41

deanq requested review from jhcipar and pandyamarut September 18, 2025 01:41

deanq added 4 commits September 17, 2025 19:03

docs: System Python Runtime Architecture

b95451d

docs: Centralized Log Streaming System

011890a

chore: update and cleanup

f11b4dc

fix: docker uses system's conda python; local uses uv

e2fe764

test-handler should always use env python

pandyamarut reviewed Sep 22, 2025

View reviewed changes

Comment thread Dockerfile

Comment thread src/dependency_installer.py

pandyamarut approved these changes Sep 22, 2025

View reviewed changes

fix: docker uses uv --system or conda python; local uses uv

39a32de

jhcipar approved these changes Sep 23, 2025

View reviewed changes

Merge branch 'deanq/ae-962-log-streaming' into deanq/ae-1165-bug-pyto…

0447519

…rchs-not-found

Base automatically changed from deanq/ae-962-log-streaming to main September 23, 2025 18:02

deanq added 7 commits September 23, 2025 13:44

Merge branch 'main' into deanq/ae-1165-bug-pytorchs-not-found

43888d5

build: fix CI to activate venv before make test-handler

564192a

build: use Docker-based tests for CI

d26ef79

fix: conda path wasn't found on CPU image (applies only to Pytorch im…

3f1b07f

…age)

Merge branch 'main' into deanq/ae-1165-bug-pytorchs-not-found

f4cbf30

fix: completely remove local-execution-test-gpu

f3d4a03

build: use --rebase and not --merge

3902147

deanq merged commit d11a7fb into main Sep 25, 2025
13 checks passed

deanq deleted the deanq/ae-1165-bug-pytorchs-not-found branch September 25, 2025 22:04

github-actions Bot mentioned this pull request Sep 25, 2025

chore(main): release 0.6.0 #29

Merged

Navigation Menu

Uh oh!

fix: access built-in system Python instead of using venv for runtime - #30

fix: access built-in system Python instead of using venv for runtime#30
deanq merged 71 commits into
mainfrom
deanq/ae-1165-bug-pytorchs-not-found

deanq commented Sep 17, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

jhcipar commented Sep 22, 2025

Uh oh!

deanq commented Sep 22, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

deanq commented Sep 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Key Changes

Problem Solved

Technical Details

Architecture Changes

Docker Optimization

Breaking Changes

Testing

Migration Notes

Test Plan

Uh oh!

Uh oh!

Uh oh!

jhcipar commented Sep 22, 2025

Uh oh!

deanq commented Sep 22, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

deanq commented Sep 17, 2025 •

edited

Loading