-
Notifications
You must be signed in to change notification settings - Fork 958
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
examples/bert/build.py does not use model weights #2197
Comments
This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 15 days." |
This issue was closed because it has been stalled for 15 days with no activity. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
System Info
Currently the example TensorRT LLM engine builder for Bert models simply ignores model weights if those are present in the model directory, it only reads the
config.json
file, making it essentially impossible to generate a working engine from a pretrained model.A possible fix is available in #2187
Who can help?
@byshiue
Information
Tasks
examples
folder (such as GLUE/SQuAD, ...)Reproduction
Scenario 1 (simplest)
config.json
and the weights file).examples/bert/build.py --model_dir input_model
Scenario 2 (use of the weights)
config.json
and the weights file).examples/bert/build.py --model_dir input_model
Expected behavior
Scenario 1 (simplest)
build.py
shall show an error message complaining about invalid weights file.Scenario 2 (use of the weights)
The output tensors shall have numerically close components.
actual behavior
Scenario 1 (simplest)
build.py
finished successfully, generatingbert_outputs/config.json
andbert_outputs/BertModel_float16_tp1_rank0.engine
.Scenario 2 (use of the weights)
The output tensors look totally unrelated and different from each other.
additional notes
The problem is that the script code only loads the config and does not do anything to load the weights. The fix is available in #2187.
The text was updated successfully, but these errors were encountered: