Adding LoKrModel Class to paddle.peft library #9269
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Thanks for your contribution! |
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## develop #9269 +/- ##
===========================================
- Coverage 53.02% 52.85% -0.17%
===========================================
Files 657 662 +5
Lines 106311 106999 +688
===========================================
+ Hits 56367 56556 +189
- Misses 49944 50443 +499 ☔ View full report in Codecov by Sentry. |
|
按照建议我已经提交了只有Linear Layer的LoKr实现供你们整体查看。麻烦有时间审阅下并给出需要修改的意见。 |
|
好的收到，我们这边会尽快review |
lugimzzz
reviewed
Oct 17, 2024
|
|
||
| # This module is set to be in alignment with code design paradiam of ...utils.env | ||
|
|
||
| LOKR_WEIGHTS_NAME = "lokr_model_state.pdparams" |
There was a problem hiding this comment.
| @@ -0,0 +1,19 @@ | |||
| # Copyright 2023-present the HuggingFace Inc. team. | |||
| # Copyright 2023-present the HuggingFace Inc. team. | ||
| # | ||
| # Licensed under the Apache License, Version 2.0 (the "License"); | ||
| # you may not use this file except in compliance with the License. |
There was a problem hiding this comment.
在llm目录下添加使用示例和相应的单测，具体可参考vera相关pr
#8722
| def add_lora_split_mapping(self, module_name, is_column=False): | ||
| self.lora_split_mapping[module_name] = is_column | ||
|
|
||
| def _get_tensor_parallel_mappings(self, config, is_split=True): |
There was a problem hiding this comment.
没有使用tensor_parallel 和pipeline parallel先把没用到的相关的逻辑删掉
| self.quantized = True | ||
| if lokr_module is None: | ||
| raise ValueError("LoKr strategy only supports paddle.nn.Linear right now") | ||
| if getattr(lokr_module, "quant_weight", None) is not None: |
| if any(isinstance(layer, lokr_layer) for lokr_layer in AVAILABLE_LAYERS): | ||
| layer.disable_lokr = True | ||
|
|
||
| def enable_lokr(self): |
There was a problem hiding this comment.
可以参考lora，在lora_layer中加入disable_lokr这个参数
|
辛苦了！第一次参与开源。可能错误较多，我会尽早修改问题，重新提交，供你们审阅。 Have a good day! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
PR types
New features
PR changes
Others
Description
Adding LoKrModel, LoKrLinear and LoKrConfig to support a new lora-like adapter. Current implementation only supports contains Linear Modules. Motivation and discussion on such PR issue is at: #9226
Please provide suggestions on the current implementation!