add metadata format to save_file in FullModelHFCheckpointer
#1126
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
AutoModel.xx
classes from hf's transformers do not work otherwise, and we get an error because the metadata isNone
while loading the model here: https:/huggingface/transformers/blob/main/src/transformers/modeling_utils.py#L518-L522That said, I only have very basic knowledge of this so might be wrong here. I personally had to re-save the model files with metadata added for them to work with
AutoModel...
classes.Context
What is the purpose of this PR? Is it to
Please link to any issues this PR addresses.
Changelog
Adds pt metadata while saving safetensors in
FullModelHFCheckpointer
.Test plan
Let me know if I have to do any of these? Also how 😅 ?
Please make sure to do each of the following if applicable to your PR. (If you're not sure about any one of these just ask and we will happily help.)
pre-commit install
)pytest tests
pytest tests -m integration_test