[Feature] Add Quantization Support for VeRA Method

[ZiadHelal]

Feature request

This pull request aims to introduce support for quantization in the VeRA PEFT method. This enhancement is in response to the issue #1921, which highlighted the need for compatibility between VeRA and quantized models.

Motivation

The motivation behind this feature request is to extend the functionality of the VeRA method to support models that have been quantized using techniques such as bitsandbytes. Currently, users face issues with dimension mismatches and incorrect tensor shapes when applying VeRA to quantized models. By adding support for quantization, we can improve the usability and flexibility of VeRA, making it a more powerful tool for a broader range of use cases.

Your contribution

I am willing to submit a pull request with the implementation of this feature. The PR will include:

• Code changes to support quantized weights in the VeRA method.
• (Possible) Tests to validate the correct handling of quantized models.
• Updated documentation and examples demonstrating the new functionality.

Please let me know if there are any additional requirements or considerations for this feature. I look forward to contributing this enhancement to the project and am open to any feedback or suggestions. @BenjaminBossan

[BenjaminBossan]

Thanks a lot @ZiadHelal for offering to work on this feature. Please go ahead and submit a PR. You can also create a draft PR that does not yet cover all your points, and I can give early feedback, which is often better than trying to finish everything in one step.