-
Notifications
You must be signed in to change notification settings - Fork 619
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
load_in_8bit is not working for some huggingface model #14
Comments
Hi @sanyalsunny111 However, this model does not support |
@younesbelkada Thank you for your previous response. you rightly mentioned device map auto is not supported yet and without that we cannot run a 8 bit model. But my question is how you have used |
Hi @sanyalsunny111 |
Hi @sanyalsunny111 ! |
Hey @younesbelkada Thank you very much sir. It is working fine. |
Great ! Very happy that you made it work! 💪 Do not hesitate to open an issue if you face into any new issue |
Hey @younesbelkada the device_map='auto' is actually affecting the distributed data parallel (DDP). I am using 8 GPUs and trying to run a faster inference. Here is the error I am getting Could you please suggest how to use load_in_8bit with DDP? |
Hi @sanyalsunny111 |
Hey @younesbelkada Sorry to bother you with more error. Yes, with |
[FIX] passing of sparse in StableEmbedding
improve the gemv 4bit accuracy by forcing the hipcub to 32
I have updated the transformers package and I am using ViLT model: https://huggingface.co/docs/transformers/model_doc/vilt#transformers.ViltForQuestionAnswering
I am getting this error is load_in_8bit is not integrated will all hugging face models ? Could you please let me know how to use load_in_8bit for any huggingface model not just BLOOM and T5.
The text was updated successfully, but these errors were encountered: