Module 'bitsandbytes' has no attribute 'nn'

[Dygitz]

I installed bitsandbytes-windows with pip install git+https:/Keith-Hon/bitsandbytes-windows.git and am using transformers by HuggingFace and got the below error. I am not sure if the error is caused by my CUDA version or something else. Any help appreciated!

My code

import os
os.environ["CUDA_VISIBLE_DEVICES"]="0"
import torch
import torch.nn as nn
import bitsandbytes as bnb
from transformers import AutoTokenizer, AutoConfig, AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained(
    "bigscience/bloom-7b1",
    load_in_8bit=True,
    device_map='auto',
)

tokenizer = AutoTokenizer.from_pretrained("bigscience/bloom-7b1")

Error

module 'bitsandbytes' has no attribute 'nn'

python -m bitsandbytes results

binary_path: C:\Users\golde\anaconda3\lib\site-packages\bitsandbytes\cuda_setup\libbitsandbytes_cuda116.dll
CUDA SETUP: Loading binary C:\Users\golde\anaconda3\lib\site-packages\bitsandbytes\cuda_setup\libbitsandbytes_cuda116.dll...
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++ DEBUG INFORMATION +++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

++++++++++ POTENTIALLY LIBRARY-PATH-LIKE ENV VARS ++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

WARNING: Please be sure to sanitize sensible info from any such env vars!

++++++++++++++++++++++++++ OTHER +++++++++++++++++++++++++++
COMPILED_WITH_CUDA = True
Traceback (most recent call last):
File "C:\Users\golde\anaconda3\lib\runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "C:\Users\golde\anaconda3\lib\runpy.py", line 86, in run_code
exec(code, run_globals)
File "C:\Users\golde\anaconda3\lib\site-packages\bitsandbytes_main.py", line 47, in
print(f"COMPUTE_CAPABILITIES_PER_GPU = {get_compute_capabilities(cuda)}")
File "C:\Users\golde\anaconda3\lib\site-packages\bitsandbytes\cuda_setup\main.py", line 336, in get_compute_capabilities
check_cuda_result(cuda, cuda.cuDeviceGetCount(ct.byref(nGpus)))
AttributeError: 'NoneType' object has no attribute 'cuDeviceGetCount'

nvidia-smi results

Sat Jun 10 16:23:25 2023
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.98 Driver Version: 535.98 CUDA Version: 12.2 |
|-----------------------------------------+----------------------+----------------------+
| GPU Name TCC/WDDM | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+======================+======================|
| 0 NVIDIA GeForce GTX 1080 WDDM | 00000000:01:00.0 On | N/A |
| 26% 41C P5 12W / 180W | 2118MiB / 8192MiB | 1% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=======================================================================================|
| 0 N/A N/A 2072 C+G ....5.48.0_x64__htrsf667h5kn2\AWCC.exe N/A |
| 0 N/A N/A 2560 C+G ...2txyewy\StartMenuExperienceHost.exe N/A |
| 0 N/A N/A 6460 C+G ...h-0\231.9011.36\bin\dataspell64.exe N/A |
| 0 N/A N/A 6524 C+G ...GeForce Experience\NVIDIA Share.exe N/A |
| 0 N/A N/A 10816 C+G ...AppData\Roaming\Spotify\Spotify.exe N/A |
| 0 N/A N/A 12196 C+G ...on\wallpaper_engine\wallpaper32.exe N/A |
| 0 N/A N/A 13316 C+G ...5n1h2txyewy\ShellExperienceHost.exe N/A |
| 0 N/A N/A 14240 C+G ...oogle\Chrome\Application\chrome.exe N/A |
| 0 N/A N/A 16136 C+G ...m Files (x86)\Overwolf\Overwolf.exe N/A |
| 0 N/A N/A 17464 C+G ...CBS_cw5n1h2txyewy\TextInputHost.exe N/A |
| 0 N/A N/A 18088 C+G ...\Local\slack\app-4.32.122\slack.exe N/A |
| 0 N/A N/A 18240 C+G ...pIntegrations\Grammarly.Desktop.exe N/A |
| 0 N/A N/A 19300 C+G ...ne\Binaries\Win64\EpicWebHelper.exe N/A |
| 0 N/A N/A 20252 C+G ...al\Discord\app-1.0.9013\Discord.exe N/A |
| 0 N/A N/A 21688 C+G ....Search_cw5n1h2txyewy\SearchApp.exe N/A |
| 0 N/A N/A 21752 C+G C:\Windows\explorer.exe N/A |
| 0 N/A N/A 22304 C+G ...inaries\Win64\EpicGamesLauncher.exe N/A |
| 0 N/A N/A 23652 C+G ...wolf\0.224.0.25\OverwolfBrowser.exe N/A |
+---------------------------------------------------------------------------------------+

[TimDettmers]

Windows is currently not supported. You can find more information here: #30

[ihongxx]

$ sudo apt install gcc-10 g++-10
$ export CC=/usr/bin/gcc-10
$ export CXX=/usr/bin/g++-10
$ export CUDA_ROOT=/usr/local/cuda
$ ln -s /usr/bin/gcc-10 $CUDA_ROOT/bin/gcc
$ ln -s /usr/bin/g++-10 $CUDA_ROOT/bin/g++
$ git clone https:/timdettmers/bitsandbytes.git
$ cd bitsandbytes
$ CUDA_HOME=/usr/local/cuda-11.6 CUDA_VERSION=116 make cuda11x
$ python setup.py install
I succeeded！