undefined symbol: cudaRuntimeGetVersion Error

[hchoi-moveworks]

Device setup:

NVIDIA-SMI 510.47.03    Driver Version: 510.47.03    CUDA Version: 11.6

nvidia-smi --query-gpu=gpu_name,gpu_bus_id,vbios_version --format=csv
name, pci.bus_id, vbios_version
Tesla T4, 00000000:00:1B.0, 90.04.96.00.02
Tesla T4, 00000000:00:1C.0, 90.04.96.00.02
Tesla T4, 00000000:00:1D.0, 90.04.96.00.02
Tesla T4, 00000000:00:1E.0, 90.04.96.00.02

Code:

In [1]: from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
In [2]: import torch
In [3]: model_name = "t5-3b-sharded"
In [4]: model_id=f"ybelkada/{model_name}"
In [5]: model_8bit = AutoModelForSeq2SeqLM.from_pretrained(model_id, device_map="auto", load_in_8bit=True)

StackTrace:

AttributeError                            Traceback (most recent call last)
<ipython-input-6-f8bf7f53fe60> in <module>
----> 1 model_8bit = AutoModelForSeq2SeqLM.from_pretrained(model_id, device_map="auto", load_in_8bit=True)

~/anaconda3/lib/python3.8/site-packages/transformers/models/auto/auto_factory.py in from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
    461         elif type(config) in cls._model_mapping.keys():
    462             model_class = _get_model_class(config, cls._model_mapping)
--> 463             return model_class.from_pretrained(
    464                 pretrained_model_name_or_path, *model_args, config=config, **hub_kwargs, **kwargs
    465             )

~/anaconda3/lib/python3.8/site-packages/transformers/modeling_utils.py in from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
   2231
   2232         if load_in_8bit:
-> 2233             from .utils.bitsandbytes import get_keys_to_not_convert, replace_8bit_linear
   2234
   2235             logger.info("Detected 8-bit loading: activating 8-bit loading for this model")

~/anaconda3/lib/python3.8/site-packages/transformers/utils/bitsandbytes.py in <module>
      8     import torch.nn as nn
      9
---> 10     import bitsandbytes as bnb
     11
     12 if is_accelerate_available():

~/anaconda3/lib/python3.8/site-packages/bitsandbytes/__init__.py in <module>
      4 # LICENSE file in the root directory of this source tree.
      5
----> 6 from .autograd._functions import (
      7     MatmulLtState,
      8     bmm_cublas,

~/anaconda3/lib/python3.8/site-packages/bitsandbytes/autograd/_functions.py in <module>
      3
      4 import torch
----> 5 import bitsandbytes.functional as F
      6
      7 from dataclasses import dataclass

~/anaconda3/lib/python3.8/site-packages/bitsandbytes/functional.py in <module>
     11 from torch import Tensor
     12
---> 13 from .cextension import COMPILED_WITH_CUDA, lib
     14 from functools import reduce  # Required in Python 3
     15

~/anaconda3/lib/python3.8/site-packages/bitsandbytes/cextension.py in <module>
    111
    112
--> 113 lib = CUDASetup.get_instance().lib
    114 try:
    115     if lib is None and torch.cuda.is_available():

~/anaconda3/lib/python3.8/site-packages/bitsandbytes/cextension.py in get_instance(cls)
    107         if cls._instance is None:
    108             cls._instance = cls.__new__(cls)
--> 109             cls._instance.initialize()
    110         return cls._instance
    111

~/anaconda3/lib/python3.8/site-packages/bitsandbytes/cextension.py in initialize(self)
     57
     58         from .cuda_setup.main import evaluate_cuda_setup
---> 59         binary_name, cudart_path, cuda, cc, cuda_version_string = evaluate_cuda_setup()
     60         self.cudart_path = cudart_path
     61         self.cuda = cuda

~/anaconda3/lib/python3.8/site-packages/bitsandbytes/cuda_setup/main.py in evaluate_cuda_setup()
    123     cuda = get_cuda_lib_handle()
    124     cc = get_compute_capability(cuda)
--> 125     cuda_version_string = get_cuda_version(cuda, cudart_path)
    126
    127     failure = False

~/anaconda3/lib/python3.8/site-packages/bitsandbytes/cuda_setup/main.py in get_cuda_version(cuda, cudart_path)
     43
     44     version = ctypes.c_int()
---> 45     check_cuda_result(cuda, cudart.cudaRuntimeGetVersion(ctypes.byref(version)))
     46     version = int(version.value)
     47     major = version//1000

~/anaconda3/lib/python3.8/ctypes/__init__.py in __getattr__(self, name)
    392         if name.startswith('__') and name.endswith('__'):
    393             raise AttributeError(name)
--> 394         func = self.__getitem__(name)
    395         setattr(self, name, func)
    396         return func

~/anaconda3/lib/python3.8/ctypes/__init__.py in __getitem__(self, name_or_ordinal)
    397
    398     def __getitem__(self, name_or_ordinal):
--> 399         func = self._FuncPtr((name_or_ordinal, self))
    400         if not isinstance(name_or_ordinal, int):
    401             func.__name__ = name_or_ordinal

AttributeError: /home/hchoi/anaconda3/bin/python: undefined symbol: cudaRuntimeGetVersion

[tshadley]

Was able to workaround this with setenv LD_LIBRARY_PATH /usr/lib/x86_64-linux-gnu which is where my libcudart.so is (Ubuntu 22.04). Not sure why it misses that.

[RamonBeast]

I have the exact same issue, but in my case the workaround does not seem to work, unfortunately.

[65536william]

I had this error and was stumped by it until I realised that my CUDA toolkit had been installed ~incorrectly and was in a place where bitsandbytes couldn't find it. So when I installed PyTorch and the toolkit via conda instead of pip, the library was able to locate the toolkit and this error went away.

[0xdevalias]

Following along from the following StackOverflow, this seems to be a solution to ensuring conda is using the 'right pip':

• https://stackoverflow.com/questions/41060382/using-pip-to-install-packages-to-anaconda-environment

⇒ conda init

⇒ conda create --name foo

⇒ conda activate foo

⇒ which -a pip
/opt/conda/bin/pip

⇒ conda install pip

⇒ which -a pip
/opt/conda/envs/dreambooth/bin/pip
/opt/conda/bin/pip

---

And based on this StackOverflow + a little extra digging, I found that the reason it wasn't able to find my libcudart.so was because presumably I hadn't actually installed cudatoolkit yet:

• https://stackoverflow.com/questions/37817962/what-is-the-path-for-libcudart-so

⇒ find /opt/conda -name libcudart.so
/opt/conda/lib/libcudart.so
/opt/conda/pkgs/cudatoolkit-11.3.1-ha36c431_9/lib/libcudart.so

⇒ pip install cudatoolkit
ERROR: Could not find a version that satisfies the requirement cudatoolkit (from versions: none)
ERROR: No matching distribution found for cudatoolkit

⇒ conda install cudatoolkit
Collecting package metadata (current_repodata.json): done
Solving environment: done

==> WARNING: A newer version of conda exists. <==
  current version: 4.13.0
  latest version: 22.9.0

Please update conda by running

    $ conda update -n base -c defaults conda

## Package Plan ##

  environment location: /opt/conda/envs/dreambooth

  added / updated specs:
    - cudatoolkit

The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    cudatoolkit-11.3.1         |       h2bc3f7f_2       549.3 MB
    ------------------------------------------------------------
                                           Total:       549.3 MB

The following NEW packages will be INSTALLED:

  cudatoolkit        pkgs/main/linux-64::cudatoolkit-11.3.1-h2bc3f7f_2

Proceed ([y]/n)? y

Downloading and Extracting Packages
cudatoolkit-11.3.1   | 549.3 MB  | ######################################################################################################################################################################################################## | 100%
Preparing transaction: done
Verifying transaction: done
Executing transaction: / By downloading and using the CUDA Toolkit conda packages, you accept the terms and conditions of the CUDA End User License Agreement (EULA): https://docs.nvidia.com/cuda/eula/index.html

done

⇒ find /opt/conda -name libcudart.so
/opt/conda/envs/dreambooth/lib/libcudart.so
/opt/conda/lib/libcudart.so
/opt/conda/pkgs/cudatoolkit-11.3.1-ha36c431_9/lib/libcudart.so
/opt/conda/pkgs/cudatoolkit-11.3.1-h2bc3f7f_2/lib/libcudart.so

[RamonBeast]

Do you think there is a workaround for those that are using pip with virtualenv instead of conda?
libcudart.so is already part of the cuda_runtime, so I'm wondering if the entire cudatoolkit is required or we can get away with just the runtime pkg.

[0xdevalias]

Do you think there is a workaround for those that are using pip with virtualenv instead of conda?

@RamonBeast Just stumbled across this page, seems like it should probably work for you?

• https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html#pip-wheels-installation-linux

pip install --extra-index-url https://pypi.ngc.nvidia.com --dry-run nvidia-cuda-runtime-cu11
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting nvidia-cuda-runtime-cu11
  Downloading https://developer.download.nvidia.com/compute/redist/nvidia-cuda-runtime-cu11/nvidia-cuda-runtime-cu11-2022.4.25.tar.gz (16 kB)

• https://developer.download.nvidia.com/compute/cuda/redist/
  • https://developer.download.nvidia.com/compute/cuda/redist/cuda_cudart/

⇒  ls -la ~/Desktop/cuda_cudart-linux-x86_64-11.8.89-archive/lib
total 2900
drwxr-xr-x 13 devalias staff     416 Sep 22 04:45 ./
drwxr-xr-x  7 devalias staff     224 Nov 10 09:55 ../
lrwxrwxrwx  1 devalias staff      14 Sep 22 03:41 libOpenCL.so -> libOpenCL.so.1*
lrwxrwxrwx  1 devalias staff      16 Sep 22 03:41 libOpenCL.so.1 -> libOpenCL.so.1.0*
lrwxrwxrwx  1 devalias staff      18 Sep 22 03:41 libOpenCL.so.1.0 -> libOpenCL.so.1.0.0*
-rwxr-xr-x  1 devalias staff   30856 Sep 22 03:41 libOpenCL.so.1.0.0*
-rw-r--r--  1 devalias staff 1021860 Sep 22 03:41 libcudadevrt.a
lrwxrwxrwx  1 devalias staff      17 Sep 22 03:41 libcudart.so -> libcudart.so.11.0*
lrwxrwxrwx  1 devalias staff      20 Sep 22 03:41 libcudart.so.11.0 -> libcudart.so.11.8.89*
-rwxr-xr-x  1 devalias staff  679264 Sep 22 03:41 libcudart.so.11.8.89*
-rw-r--r--  1 devalias staff 1198880 Sep 22 03:41 libcudart_static.a
-rw-r--r--  1 devalias staff   30922 Sep 22 03:41 libculibos.a
drwxr-xr-x  3 devalias staff      96 Sep 22 04:45 stubs/

---

And for reference, here is the conda version of those docs too:

• https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html#conda
  • https://anaconda.org/nvidia/repo
    • https://anaconda.org/nvidia/cuda-toolkit
    • https://anaconda.org/nvidia/cudatoolkit
  • https://anaconda.org/search?q=cudatoolkit

---

libcudart.so is already part of the cuda_runtime, so I'm wondering if the entire cudatoolkit is required or we can get away with just the runtime pkg

@RamonBeast I haven't specifically tried it myself yet, but it could well be the case

• https://anaconda.org/nvidia/cuda-runtime

[0xdevalias]

In case it's helpful to people, here are the lists of the various packages that are installed for each of cuda-toolkit, cuda-runtime, and cuda-cudart:

---

• https://anaconda.org/nvidia/cuda-toolkit

⇒ conda install -c nvidia cuda-toolkit
Collecting package metadata (current_repodata.json): done
Solving environment: done


==> WARNING: A newer version of conda exists. <==
  current version: 4.13.0
  latest version: 22.9.0

Please update conda by running

    $ conda update -n base -c defaults conda



## Package Plan ##

  environment location: /opt/conda/envs/scratch

  added / updated specs:
    - cuda-toolkit


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    cuda-cccl-11.8.89          |                0         1.2 MB  nvidia
    cuda-command-line-tools-11.8.0|                0           1 KB  nvidia
    cuda-compiler-11.8.0       |                0           1 KB  nvidia
    cuda-cudart-11.8.89        |                0         197 KB  nvidia
    cuda-cudart-dev-11.8.89    |                0         1.1 MB  nvidia
    cuda-cuobjdump-11.8.86     |                0         229 KB  nvidia
    cuda-cupti-11.8.87         |                0        25.3 MB  nvidia
    cuda-cuxxfilt-11.8.86      |                0         291 KB  nvidia
    cuda-documentation-11.8.86 |                0          89 KB  nvidia
    cuda-driver-dev-11.8.89    |                0          16 KB  nvidia
    cuda-gdb-11.8.86           |                0         4.8 MB  nvidia
    cuda-libraries-11.8.0      |                0           1 KB  nvidia
    cuda-libraries-dev-11.8.0  |                0           2 KB  nvidia
    cuda-memcheck-11.8.86      |                0         168 KB  nvidia
    cuda-nsight-11.8.86        |                0       113.6 MB  nvidia
    cuda-nsight-compute-11.8.0 |                0           1 KB  nvidia
    cuda-nvcc-11.8.89          |                0        50.8 MB  nvidia
    cuda-nvdisasm-11.8.86      |                0        48.7 MB  nvidia
    cuda-nvml-dev-11.8.86      |                0          83 KB  nvidia
    cuda-nvprof-11.8.87        |                0         4.4 MB  nvidia
    cuda-nvprune-11.8.86       |                0          65 KB  nvidia
    cuda-nvrtc-11.8.89         |                0        19.1 MB  nvidia
    cuda-nvrtc-dev-11.8.89     |                0        17.0 MB  nvidia
    cuda-nvtx-11.8.86          |                0          57 KB  nvidia
    cuda-nvvp-11.8.87          |                0       114.4 MB  nvidia
    cuda-profiler-api-11.8.86  |                0          18 KB  nvidia
    cuda-sanitizer-api-11.8.86 |                0        16.6 MB  nvidia
    cuda-toolkit-11.8.0        |                0           1 KB  nvidia
    cuda-tools-11.8.0          |                0           1 KB  nvidia
    cuda-visual-tools-11.8.0   |                0           1 KB  nvidia
    gds-tools-1.4.0.31         |                0           2 KB  nvidia
    libcublas-11.11.3.6        |                0       364.0 MB  nvidia
    libcublas-dev-11.11.3.6    |                0       394.1 MB  nvidia
    libcufft-10.9.0.58         |                0       142.8 MB  nvidia
    libcufft-dev-10.9.0.58     |                0       275.8 MB  nvidia
    libcufile-1.4.0.31         |                0         548 KB  nvidia
    libcufile-dev-1.4.0.31     |                0         1.6 MB  nvidia
    libcurand-10.3.0.86        |                0        53.2 MB  nvidia
    libcurand-dev-10.3.0.86    |                0        53.7 MB  nvidia
    libcusolver-11.4.1.48      |                0        96.5 MB  nvidia
    libcusolver-dev-11.4.1.48  |                0        66.3 MB  nvidia
    libcusparse-11.7.5.86      |                0       176.3 MB  nvidia
    libcusparse-dev-11.7.5.86  |                0       359.7 MB  nvidia
    libnpp-11.8.0.86           |                0       147.8 MB  nvidia
    libnpp-dev-11.8.0.86       |                0       144.5 MB  nvidia
    libnvjpeg-11.9.0.86        |                0         2.4 MB  nvidia
    libnvjpeg-dev-11.9.0.86    |                0         2.1 MB  nvidia
    nsight-compute-2022.3.0.22 |                0       610.0 MB  nvidia
    ------------------------------------------------------------
                                           Total:        3.23 GB

The following NEW packages will be INSTALLED:

  cuda-cccl          nvidia/linux-64::cuda-cccl-11.8.89-0
  cuda-command-line~ nvidia/linux-64::cuda-command-line-tools-11.8.0-0
  cuda-compiler      nvidia/linux-64::cuda-compiler-11.8.0-0
  cuda-cudart        nvidia/linux-64::cuda-cudart-11.8.89-0
  cuda-cudart-dev    nvidia/linux-64::cuda-cudart-dev-11.8.89-0
  cuda-cuobjdump     nvidia/linux-64::cuda-cuobjdump-11.8.86-0
  cuda-cupti         nvidia/linux-64::cuda-cupti-11.8.87-0
  cuda-cuxxfilt      nvidia/linux-64::cuda-cuxxfilt-11.8.86-0
  cuda-documentation nvidia/linux-64::cuda-documentation-11.8.86-0
  cuda-driver-dev    nvidia/linux-64::cuda-driver-dev-11.8.89-0
  cuda-gdb           nvidia/linux-64::cuda-gdb-11.8.86-0
  cuda-libraries     nvidia/linux-64::cuda-libraries-11.8.0-0
  cuda-libraries-dev nvidia/linux-64::cuda-libraries-dev-11.8.0-0
  cuda-memcheck      nvidia/linux-64::cuda-memcheck-11.8.86-0
  cuda-nsight        nvidia/linux-64::cuda-nsight-11.8.86-0
  cuda-nsight-compu~ nvidia/linux-64::cuda-nsight-compute-11.8.0-0
  cuda-nvcc          nvidia/linux-64::cuda-nvcc-11.8.89-0
  cuda-nvdisasm      nvidia/linux-64::cuda-nvdisasm-11.8.86-0
  cuda-nvml-dev      nvidia/linux-64::cuda-nvml-dev-11.8.86-0
  cuda-nvprof        nvidia/linux-64::cuda-nvprof-11.8.87-0
  cuda-nvprune       nvidia/linux-64::cuda-nvprune-11.8.86-0
  cuda-nvrtc         nvidia/linux-64::cuda-nvrtc-11.8.89-0
  cuda-nvrtc-dev     nvidia/linux-64::cuda-nvrtc-dev-11.8.89-0
  cuda-nvtx          nvidia/linux-64::cuda-nvtx-11.8.86-0
  cuda-nvvp          nvidia/linux-64::cuda-nvvp-11.8.87-0
  cuda-profiler-api  nvidia/linux-64::cuda-profiler-api-11.8.86-0
  cuda-sanitizer-api nvidia/linux-64::cuda-sanitizer-api-11.8.86-0
  cuda-toolkit       nvidia/linux-64::cuda-toolkit-11.8.0-0
  cuda-tools         nvidia/linux-64::cuda-tools-11.8.0-0
  cuda-visual-tools  nvidia/linux-64::cuda-visual-tools-11.8.0-0
  gds-tools          nvidia/linux-64::gds-tools-1.4.0.31-0
  libcublas          nvidia/linux-64::libcublas-11.11.3.6-0
  libcublas-dev      nvidia/linux-64::libcublas-dev-11.11.3.6-0
  libcufft           nvidia/linux-64::libcufft-10.9.0.58-0
  libcufft-dev       nvidia/linux-64::libcufft-dev-10.9.0.58-0
  libcufile          nvidia/linux-64::libcufile-1.4.0.31-0
  libcufile-dev      nvidia/linux-64::libcufile-dev-1.4.0.31-0
  libcurand          nvidia/linux-64::libcurand-10.3.0.86-0
  libcurand-dev      nvidia/linux-64::libcurand-dev-10.3.0.86-0
  libcusolver        nvidia/linux-64::libcusolver-11.4.1.48-0
  libcusolver-dev    nvidia/linux-64::libcusolver-dev-11.4.1.48-0
  libcusparse        nvidia/linux-64::libcusparse-11.7.5.86-0
  libcusparse-dev    nvidia/linux-64::libcusparse-dev-11.7.5.86-0
  libnpp             nvidia/linux-64::libnpp-11.8.0.86-0
  libnpp-dev         nvidia/linux-64::libnpp-dev-11.8.0.86-0
  libnvjpeg          nvidia/linux-64::libnvjpeg-11.9.0.86-0
  libnvjpeg-dev      nvidia/linux-64::libnvjpeg-dev-11.9.0.86-0
  nsight-compute     nvidia/linux-64::nsight-compute-2022.3.0.22-0

---

• https://anaconda.org/nvidia/cuda-runtime

⇒ conda install -c nvidia cuda-runtime
Collecting package metadata (current_repodata.json): done
Solving environment: done


==> WARNING: A newer version of conda exists. <==
  current version: 4.13.0
  latest version: 22.9.0

Please update conda by running

    $ conda update -n base -c defaults conda



## Package Plan ##

  environment location: /opt/conda/envs/scratch

  added / updated specs:
    - cuda-runtime


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    cuda-cudart-11.8.89        |                0         197 KB  nvidia
    cuda-libraries-11.8.0      |                0           1 KB  nvidia
    cuda-nvrtc-11.8.89         |                0        19.1 MB  nvidia
    cuda-runtime-11.8.0        |                0           1 KB  nvidia
    libcublas-11.11.3.6        |                0       364.0 MB  nvidia
    libcufft-10.9.0.58         |                0       142.8 MB  nvidia
    libcufile-1.4.0.31         |                0         548 KB  nvidia
    libcurand-10.3.0.86        |                0        53.2 MB  nvidia
    libcusolver-11.4.1.48      |                0        96.5 MB  nvidia
    libcusparse-11.7.5.86      |                0       176.3 MB  nvidia
    libnpp-11.8.0.86           |                0       147.8 MB  nvidia
    libnvjpeg-11.9.0.86        |                0         2.4 MB  nvidia
    ------------------------------------------------------------
                                           Total:      1002.8 MB

The following NEW packages will be INSTALLED:

  cuda-cudart        nvidia/linux-64::cuda-cudart-11.8.89-0
  cuda-libraries     nvidia/linux-64::cuda-libraries-11.8.0-0
  cuda-nvrtc         nvidia/linux-64::cuda-nvrtc-11.8.89-0
  cuda-runtime       nvidia/linux-64::cuda-runtime-11.8.0-0
  libcublas          nvidia/linux-64::libcublas-11.11.3.6-0
  libcufft           nvidia/linux-64::libcufft-10.9.0.58-0
  libcufile          nvidia/linux-64::libcufile-1.4.0.31-0
  libcurand          nvidia/linux-64::libcurand-10.3.0.86-0
  libcusolver        nvidia/linux-64::libcusolver-11.4.1.48-0
  libcusparse        nvidia/linux-64::libcusparse-11.7.5.86-0
  libnpp             nvidia/linux-64::libnpp-11.8.0.86-0
  libnvjpeg          nvidia/linux-64::libnvjpeg-11.9.0.86-0

---

• https://anaconda.org/nvidia/cuda-cudart

⇒ conda install -c nvidia cuda-cudart
Collecting package metadata (current_repodata.json): done
Solving environment: done


==> WARNING: A newer version of conda exists. <==
  current version: 4.13.0
  latest version: 22.9.0

Please update conda by running

    $ conda update -n base -c defaults conda



## Package Plan ##

  environment location: /opt/conda/envs/scratch

  added / updated specs:
    - cuda-cudart


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    cuda-cudart-11.8.89        |                0         197 KB  nvidia
    ------------------------------------------------------------
                                           Total:         197 KB

The following NEW packages will be INSTALLED:

  cuda-cudart        nvidia/linux-64::cuda-cudart-11.8.89-0

⇒ ls ~/.conda/environments.txt
/root/.conda/environments.txt

⇒ cat ~/.conda/environments.txt
/opt/conda/envs/scratch

⇒ ls /opt/conda/envs/scratch
conda-meta  lib

⇒ ls -la /opt/conda/envs/scratch/lib/
total 720
drwxr-xr-x 2 root root    131 Nov  9 23:17 .
drwxr-xr-x 4 root root     35 Nov  9 23:17 ..
lrwxrwxrwx 1 root root     18 Nov  9 23:17 libOpenCL.so.1 -> libOpenCL.so.1.0.0
lrwxrwxrwx 1 root root     18 Nov  9 23:17 libOpenCL.so.1.0 -> libOpenCL.so.1.0.0
-rwxrwxr-x 2 root root  37368 Sep 21 19:07 libOpenCL.so.1.0.0
lrwxrwxrwx 1 root root     20 Nov  9 23:17 libcudart.so.11.0 -> libcudart.so.11.8.89
-rwxrwxr-x 2 root root 695712 Sep 21 19:07 libcudart.so.11.8.89

[sanderland]

What fixed this for me was:

export LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu:$LD_LIBRARY_PATH

in ~/.bashrc

along with ensuring libcudart.so exists there using:

sudo ln -s /usr/lib/x86_64-linux-gnu/libcudart.so.11.0 /usr/lib/x86_64-linux-gnu/libcudart.so

[TimDettmers]

Thank you, everyone, for contributing to the discussion!

It is a strange error because it indicates that libcudart.so could be loaded from the filesystem, but it does not have cudaRuntimeGetVersion function. I am not sure how this can happen. That changing the $LD_LIBRARY_PATH variable helps indicates that maybe multiple CUDA versions are installed. But the outputs from @0xdevalias shows that only a single version is installed with the same error.

For now, I added an error message for cases like this and added the path to libcudart.so that is being loaded. With that, it should be possible to narrow the error down.

I am going to release a new version tomorrow. If you can reproduce the error then and post the additional information printed in the CUDA SETUP error trace, that would be very helpful!

[ghost]

FWIW @0xdevalias comment solved it for me, there was one oddity though in case someone else runs into it. If you run a script in your conda env with bitsandbytes or a jupyter lab it works. If you exit the environment but have a kernel for that environment it doesn't work with the kernel in your default env.

I believe it's because the cuda libraries are installed in your specific conda env which is why it's so finicky. Not sure if full environment spec would be easier to manage here.

In case it's helpful to people, here are the lists of the various packages that are installed for each of cuda-toolkit, cuda-runtime, and cuda-cudart:

• https://anaconda.org/nvidia/cuda-toolkit

...

[Sieboldianus]

pip install --extra-index-url https://pypi.ngc.nvidia.com --dry-run nvidia-cuda-runtime-cu11

This doesn't solve the error on my side. I updated requirements.txt from Dreambooth with:

--extra-index-url https://pypi.ngc.nvidia.com
nvidia-cuda-runtime-cu11

But I still get

AttributeError: python3: undefined symbol: cudaRuntimeGetVersion

Unfortunately, I cannot use Conda.

[ryenots]

Do you think there is a workaround for those that are using pip with virtualenv instead of conda? libcudart.so is already part of the cuda_runtime, so I'm wondering if the entire cudatoolkit is required or we can get away with just the runtime pkg.

I am also using virtualenv on and Arch based system, and none of the solutions related to virtualenv have worked for me. I have /opt/cuda/targets/x86_64-linux/lib on my LD_LIBRARY_PATH, which is where my libcudart.so is located. This symlink leads to libcudart.so.12.2.53 in the same directory. I do not understand why cudaRuntimeGetVersion is still undefined with these changes to the path.

Are there more suggestions for this issue for virtualenv?

[umairafzal9245]

setenv LD_LIBRARY_PATH /usr/lib/x86_64-linux-gnu

For ubuntu 22.04, this worked but changed the command to export LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.