in rules_python 1.8.1 venv, tensorflow can't find its dependent nvidia so libraries

[gfrankliu]

With the release of rules_python 1.8.1, I assume the venv support is ready for general use, so I decide to give it a try, with latest bazel 8.

common --@rules_python//python/config_settings:bootstrap_impl=script
common --@rules_python//python/config_settings:venvs_site_packages=yes
common --@rules_python//python/config_settings:venvs_use_declare_symlink=yes

I wrote a dummy py_binary utils/tf_test.py

import tensorflow as tf

print(f"TensorFlow version: {tf.__version__}")
gpus = tf.config.list_physical_devices("GPU")
print(f"TF devices (GPU): {gpus}")

requirements.in:

tensorflow[and-cuda]==2.19.1

utils/BUILD:

load("@rules_python//python:defs.bzl", "py_binary")
load("@third_party_lib//:requirements.bzl", "requirement")

py_binary(
    name = "tf_test",
    srcs = ["tf_test.py"],
    deps = [
        requirement("tensorflow"),
    ],
)

bazel run //utils:tf_test complains

2026-01-21 19:46:05.488619: I external/local_xla/xla/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2026-01-21 19:46:05.514317: I external/local_xla/xla/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.

We do have libcudart.so.12 in bazel-bin/utils/tf_test.runfiles/_main/utils/_tf_test.venv/lib/python3.10/site-packages/nvidia/cuda_runtime/lib/libcudart.so.12

While troubleshooting, I noticed in the bazel-bin/utils/tf_test.runfiles/_main/utils/_tf_test.venv/lib/python3.10/site-packages/tensorflow/ directory, all the py files have relative symlinks, but the so files have absolute symlinks to my user bazel cache dir. Is that expected? Why didn't we also use relative symlink for so files?

gfrankliu-ws ➜  pwd
/home/gfrankliu/test/bazel-bin/utils/tf_test.runfiles/_main/utils/_tf_test.venv/lib/python3.10/site-packages/tensorflow
gfrankliu-ws ➜  ls -l
total 68
lrwxrwxrwx 1 gfrankliu gfrankliu  118 Jan 21 18:34 THIRD_PARTY_NOTICES.txt -> ../../../../../../../rules_python++pip+third_party_lib_310_tensorflow/site-packages/tensorflow/THIRD_PARTY_NOTICES.txt
lrwxrwxrwx 1 gfrankliu gfrankliu  106 Jan 21 18:34 __init__.py -> ../../../../../../../rules_python++pip+third_party_lib_310_tensorflow/site-packages/tensorflow/__init__.py
drwxr-xr-x 2 gfrankliu gfrankliu 4096 Jan 21 18:34 __pycache__
lrwxrwxrwx 1 gfrankliu gfrankliu   99 Jan 21 18:34 _api -> ../../../../../../../rules_python++pip+third_party_lib_310_tensorflow/site-packages/tensorflow/_api
lrwxrwxrwx 1 gfrankliu gfrankliu  103 Jan 21 18:34 compiler -> ../../../../../../../rules_python++pip+third_party_lib_310_tensorflow/site-packages/tensorflow/compiler
drwxr-xr-x 4 gfrankliu gfrankliu 4096 Jan 21 18:34 core
lrwxrwxrwx 1 gfrankliu gfrankliu  105 Jan 21 18:34 distribute -> ../../../../../../../rules_python++pip+third_party_lib_310_tensorflow/site-packages/tensorflow/distribute
lrwxrwxrwx 1 gfrankliu gfrankliu  102 Jan 21 18:34 dtensor -> ../../../../../../../rules_python++pip+third_party_lib_310_tensorflow/site-packages/tensorflow/dtensor
lrwxrwxrwx 1 gfrankliu gfrankliu  102 Jan 21 18:34 include -> ../../../../../../../rules_python++pip+third_party_lib_310_tensorflow/site-packages/tensorflow/include
lrwxrwxrwx 1 gfrankliu gfrankliu  183 Jan 21 18:34 libtensorflow_cc.so.2 -> /home/gfrankliu/.cache/bazel/_bazel_gfrankliu/0c06fba63a99dbe47301a331635af967/external/rules_python++pip+third_party_lib_310_tensorflow/site-packages/tensorflow/libtensorflow_cc.so.2
lrwxrwxrwx 1 gfrankliu gfrankliu  190 Jan 21 18:34 libtensorflow_framework.so.2 -> /home/gfrankliu/.cache/bazel/_bazel_gfrankliu/0c06fba63a99dbe47301a331635af967/external/rules_python++pip+third_party_lib_310_tensorflow/site-packages/tensorflow/libtensorflow_framework.so.2
lrwxrwxrwx 1 gfrankliu gfrankliu   99 Jan 21 18:34 lite -> ../../../../../../../rules_python++pip+third_party_lib_310_tensorflow/site-packages/tensorflow/lite
drwxr-xr-x 4 gfrankliu gfrankliu 4096 Jan 21 18:34 python
lrwxrwxrwx 1 gfrankliu gfrankliu  103 Jan 21 18:34 security -> ../../../../../../../rules_python++pip+third_party_lib_310_tensorflow/site-packages/tensorflow/security
lrwxrwxrwx 1 gfrankliu gfrankliu  100 Jan 21 18:34 tools -> ../../../../../../../rules_python++pip+third_party_lib_310_tensorflow/site-packages/tensorflow/tools
lrwxrwxrwx 1 gfrankliu gfrankliu   98 Jan 21 18:34 tsl -> ../../../../../../../rules_python++pip+third_party_lib_310_tensorflow/site-packages/tensorflow/tsl
lrwxrwxrwx 1 gfrankliu gfrankliu  114 Jan 21 18:34 xla_aot_runtime_src -> ../../../../../../../rules_python++pip+third_party_lib_310_tensorflow/site-packages/tensorflow/xla_aot_runtime_src
gfrankliu-ws ➜  

I also tried LD_DEBUG=libs bazel run //utils:tf_test and see it search the python packages, but using the absolute path, thus failed. See attached find_libcudart.txt

[gfrankliu]

find_libcudart.txt
Attachment

[gfrankliu]

Looking through find_libcudart.txt, this line would've worked:

  7375:       trying file=/home/gfrankliu/.cache/bazel/_bazel_gfrankliu/0c06fba63a99dbe47301a331635af967/execroot/_main/bazel-out/k8-fastbuild/bin/utils/tf_test.runfiles/_main/utils/_tf_test.venv/lib/python3.10/site-packages/tensorflow/python/platform/../../../nvidia/cuda_runtime/lib/libcudart.so.12

The site-packages/tensorflow/python/platform/../../../ would've been the "site-packages", which would have "nvidia" in it. But when I manually tested, the "site-packages" doesn't have "nvidia" or any other packages that are supposed to be in venv:

gfrankliu-ws ➜  pwd
/home/gfrankliu/test/bazel-bin/utils/tf_test.runfiles/_main/utils/_tf_test.venv/lib/python3.10/site-packages/tensorflow/python/platform
gfrankliu-ws ➜  ls ../../..
__init__.py  tensorflow  tensorflow-2.19.1.dist-info
gfrankliu-ws ➜  ls ../../../..
__init__.py  site-packages
gfrankliu-ws ➜  ls ../../../   
__init__.py  tensorflow  tensorflow-2.19.1.dist-info
gfrankliu-ws ➜  

It seems to be we are actually in the "/home/gfrankliu/test/bazel-bin/utils/tf_test.runfiles/rules_python++pip+third_party_lib_310_tensorflow" or the bazel cache "/home/gfrankliu/.cache/bazel/_bazel_gfrankliu/0c06fba63a99dbe47301a331635af967/external/rules_python++pip+third_party_lib_310_tensorflow/site-packages/tensorflow", not in the venv. Below test confirms my suspicion:

gfrankliu-ws ➜  pwd
/home/gfrankliu/test/bazel-bin/utils/tf_test.runfiles/rules_python++pip+third_party_lib_310_tensorflow/site-packages/tensorflow/python/platform
gfrankliu-ws ➜  ls ../../..
__init__.py  tensorflow  tensorflow-2.19.1.dist-info
gfrankliu-ws ➜  ls ../../../..
__init__.py  site-packages
gfrankliu-ws ➜  platform 

[gfrankliu]

Here is my test MODULE.bazel, in case that matters

bazel_dep(name = "rules_python", version = "1.8.1")
bazel_dep(name = "rules_uv", version = "0.77.0")

# -- module extensions -- #
# rules_python
python = use_extension("@rules_python//python/extensions:python.bzl", "python")
python.toolchain(
    is_default = True,
    python_version = "3.10",
)
use_repo(python, "python_3_10", "python_3_10_host")

pip = use_extension("@rules_python//python/extensions:pip.bzl", "pip")
pip.parse(
    enable_implicit_namespace_pkgs = True,
    # If true, disables conversion of native namespace packages into pkg-util style namespace packages. When set all py_binary
    # and py_test targets must specify either `legacy_create_init=False` or the global Bazel option
    # `--incompatible_default_to_explicit_init_py` to prevent `__init__.py` being automatically generated in every directory.
    extra_pip_args = ["--no-cache-dir"],
    hub_name = "third_party_lib",
    python_version = "3.10",
    requirements_lock = "//:requirements.txt",
)

use_repo(pip, "third_party_lib")

[aignas]

Could you please rephrase the issue so that it is clear what the issue in your case is? What did you expect and what did not work as expected?

[gfrankliu]

Python packages (such as tensorflow) that depends on nvidia packages usually uses relative path (eg: ../../nvidia/ ) to locate the neighboring nvidia packages within the same "site-packages" directory, likely using rpath in the so file, eg:

$ readelf -d libtensorflow_framework.so.2 | grep RUNPATH
 0x000000000000001d (RUNPATH)            Library runpath: [$ORIGIN/:$ORIGIN/../../nvidia/cublas/lib:$ORIGIN/../nvidia/cublas/lib:$ORIGIN/../../nvidia/cudnn/lib:$ORIGIN/../nvidia/cudnn/lib:$ORIGIN/../../nvidia/cufft/lib:$ORIGIN/../nvidia/cufft/lib:$ORIGIN/../../nvidia/cusolver/lib:$ORIGIN/../nvidia/cusolver/lib:$ORIGIN/../../nvidia/cusparse/lib:$ORIGIN/../nvidia/cusparse/lib:$ORIGIN/../../tensorrt:$ORIGIN/../tensorrt:$ORIGIN/../../nvidia/cuda_runtime/lib:$ORIGIN/../nvidia/cuda_runtime/lib]

That's one of the main reason rules_python wants to support venv so that we can bring all packages within the same "site-packages".

This means we need to have real directories within venv and only symlink files. If you use directory symlink, those ../.. would likely bring you outside of venv, so you won't be able to find the neighboring packages in the same "site-packages" within venv.

In my test case, I can see "platform" is not a real directory, but rather a symlink to outside of venv:

gfrankliu-ws ➜ pwd
/home/gfrankliu/test/bazel-bin/utils/tf_test.runfiles/_main/utils/_tf_test.venv/lib/python3.10/site-packages/tensorflow/python
gfrankliu-ws ➜ ls -l | grep platform
lrwxrwxrwx 1 gfrankliu gfrankliu  113 Jan 21 18:34 platform -> ../../../../../../../../rules_python++pip+third_party_lib_310_tensorflow/site-packages/tensorflow/python/platform

This explains why ../../.. becomes rules_python++pip+third_party_lib_310_tensorflow/site-packages, NOT the site-packages in venv, so of course there is no "nvidia" there.

If I check all those runfiles/rules_python++pip+* directories, I can see they follow this rule: create real directories and only symlink files to user's bazel cache (absolute link). Can we do the same thing for everything in venv: create real directories and only symlink files (relative symlink to runfiles/rules_python++pip+*).

[aignas]

Thank you for the explanation.

[shayanhoshyari]

2026-01-21 19:46:05.488619: I external/local_xla/xla/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2026-01-21 19:46:05.514317: I external/local_xla/xla/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.

This does not seem error about finding .so files though. I am not sure this is venv structure related or something else.

Those usually look like something:

E0000 00:00:1769130358.743355    4900 cuda_executor.cc:1228] INTERNAL: CUDA Runtime error: Failed call to cudaGetRuntimeVersion: Error loading CUDA libraries. GPU will not be used.: Error loading CUDA libraries. GPU will not be used.
W0000 00:00:1769130358.746154    4900 gpu_device.cc:2341] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.

[gfrankliu]

If you check the attached find_libcudart.txt (LD_DEBUG=libs output) in my above second post, they resulted from "external/local_xla/xla/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine" . That find_libcudart.txt shows how dynamic linker is attempting to find the nvidia runtime library "libcudart.so.12", eventually failed, and cudart_stub.cc throws "Could not find cuda drivers on your machine".

[shayanhoshyari]

Ok, I tried a few things.

1. Just using symlinks for all files (not parent folders) works fine. E.g. if using UV

$ uv venv .venv
$ uv pip install tensorflow[and-cuda]==2.19.1 --link-mode=symlink
$ python
   
>>> import tensorflow as tf
>>> gpus = tf.config.list_physical_devices("GPU")
>>> print(f"TF devices (GPU): {gpus}")
TF devices (GPU): [PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')]

2. I tried your reproducer

With https://github.com/shayanhoshyari/issue-reports/tree/main/rules_python/import_tensorflow, indeed I get

 E0000 00:00:1769130358.743355    4900 cuda_executor.cc:1228] INTERNAL: CUDA Runtime error: Failed call to cudaGetRuntimeVersion: Error loading CUDA libraries. GPU will not be used.: Error loading CUDA libraries. GPU will not be used.
 W0000 00:00:1769130358.746154    4900 gpu_device.cc:2341] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.

Seems the issue is here

      5981:	find library=libcuda.so.1 [0]; searching
      5981:	  trying file=/mnt/localssd/.cache/bazel/_bazel_colligo/7be7a2025a435f69f4a7b9365bfd122e/execroot/_main/bazel-out/k8-fastbuild/bin/app.runfiles/_main/_app.venv/lib/python3.12/site-packages/tensorflow/python/platform/../../libcuda.so.1

But tensorflow/python/platform is a link, not actual folder. This makes sense, since the heuristic @rickeylev implemented makes sure we don't symlink folders that contain lib*.so.* stuff. But in this case tensorflow/python/platform does not include them.

ls .../execroot/_main/bazel-out/k8-fastbuild/bin/app.runfiles/_main/_app.venv/lib/python3.12/site-packages/tensorflow/python/platform  

__init__.py  _pywrap_cpu_feature_guard.pyi  _pywrap_stacktrace_handler.pyi  _pywrap_tf2.pyi  analytics.py  benchmark.py   device_context.py  gfile.py       remote_utils.py     self_check.py  test.py
__pycache__  _pywrap_cpu_feature_guard.so   _pywrap_stacktrace_handler.so   _pywrap_tf2.so   app.py        build_info.py  flags.py           googletest.py  resource_loader.py  sysconfig.py   tf_logging.py

that being said, even if this folder was not a link, the path tensorflow/python/platform/../../libcuda.so.1 would become tensorflow/libcuda.so.1 which is still wrong.

What to do?

I suggest see how tensorflow decides where to look at for CUDA.

For torch, it puts $ORIGIN/../../nvidia/blahblah/libcuda.so in the rpath of its .so. And so the heuristic of just making sure the .so is linked directly (instead of its parent folder) worked. But for tensorflow it seems it is doing something slightly different.

[gfrankliu]

Thanks @shayanhoshyari for trying out my test!

Your successful uv test with symlink files gives a good news. This means we only need to fix the directories in rules_python venv/site-packages as I suggested in earlier comment.

I tried manually fixing the directory symlink:

cd tf_test.runfiles/_main/utils/_tf_test.venv/lib/python3.10/site-packages/tensorflow/python
mkdir platform.real
cp -a platform/* platform.real/
mv platform platform.symlink
mv platform.real platform

Now bazel run //utils:tf_test works, no longer failing finding cuda runtime libraries. Did you say @rickeylev already implemented this to make all directories inside venv/site-packages real, no more symlink directories? I guess this hasn't been merged into 1.8?

[gfrankliu]

ok, I reviewed the debug logs. When python imports tensorflow, it dlopens site-packages/tensorflow/python/platform/_pywrap_cpu_feature_guard.so. The effectively resets the $ORIGIN to site-packages/tensorflow/python/platform/ even though _pywrap_cpu_feature_guard.so is a symlink. So all future attempts to find nvidia libraries are using this $ORIGIN with RUNPATH, instead of the default dynamic linker behavior of using realpath of the symlink.

This means we can safely use file symlinks, as long as the directories are real.