Repository navigation
fix(apps): keep transitive torch out of GPU artifacts - #623
Draft
justinwlin wants to merge 2 commits into
Draft
justinwlin wants to merge 2 commits into
justinwlin wants to merge 2 commits into
Conversation
Packages that depend on torch (peft, accelerate, sentence-transformers) installed it into the artifact even on builtin GPU images, because only top-level requirements were excluded. The vendored PyPI torch shadowed the image's build and failed to load its CUDA libraries, so workers crash-looped. runpod-flash stripped these at archive time. After vendoring, remove every installed excluded distribution, plus the wheels only the torch family depends on (nvidia/cuda, sympy, networkx), using each wheel's RECORD and Requires-Dist. Fails closed to name-only removal when metadata is missing or a requirement has no name. Script paths pip --target records as ../../bin are mapped onto the target. Auto-exclusion now applies when any resource is GPU on a builtin image and none uses a custom image, matching runpod-flash for mixed CPU+GPU apps. A vendored PyPI torch cannot import on the builtin CPU images either, so CPU resources lose nothing.
Workers pip-install manifest excludedPackages their image lacks at cold start. In a mixed CPU+GPU app the GPU images already have the torch family, so only CPU workers act on the list, and a CPU stage that never uses torch tried to install CUDA torch onto its default 10 GB disk and failed to boot. In mixed apps, keep a torch-family name in the manifest only when the CPU resources' dependency closure (their decorator dependencies plus requirements.txt) reaches it, and warn that those workers install it at cold start. Unreadable metadata keeps every name, as before.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
GPU apps whose dependencies pull in torch (
peft,accelerate,sentence-transformers) crash-loop afterrp flash deploy. runpod-flash didn't have this problem, so Flash apps migrating to 2.0 break.Cause
split_exclusionsonly skips torch-family packages listed as top-level requirements. pip still installs them as dependencies, and the vendored PyPI torch shadows the GPU image's build. A torch built off Linux has no NVIDIA libraries, so it fails to load (libcudart.so.13/libcublasLt.so not found). runpod-flash removes these at archive time.Change
torchrun). This uses each wheel'sRECORDandRequires-Dist, the same approach as fix(build): keep torch's dependency tree out of tarballs聽flash#390. Without complete metadata, or when a requirement has no name (URL, path), only the excluded packages themselves are removed.excludedPackagesthat their image lacks. In mixed apps, torch is listed only when the CPU resources' dependencies need it, and the build warns that those workers will install it at cold start. Otherwise a CPU stage that never uses torch tries to install CUDA torch onto its 10 GB disk and fails to boot.dependencies=["peft"]2.9.1+cu128, CUDA availableVerification
rp flash deployof a GPU queue app withdependencies=["peft"]. The worker returned{"cuda": true, "file": "/usr/local/lib/python3.12/site-packages/torch/__init__.py", "torch": "2.9.1+cu128", "peft": "0.21.2"}.humanize) plus a GPU queue (torch,peft). Both returned results. Before the manifest change, the CPU worker failed bootstrapverifywithNo space left on devicewhile installing torch.docker run python:3.12-slimwith a macOS-vendored PyPI torch onPYTHONPATHfails withlibcublasLt.so.*[0-9] not found, so vendoring torch never worked on the CPU images.nvidia/dirs,../../binscript paths, missingRECORD/METADATA, unnamed requirements, user--excludedependencies, custom images, mixed apps, manifest contents).pytest tests: 1494 passed, coverage 91.94%.