-
Notifications
You must be signed in to change notification settings - Fork 129
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[BUG] model_config prefix is changed in payload for ML inference processors #2822
Comments
maybe one way to fix this bug is to move the reconstructing payload within local model only @rbhavna please share your thoughts in this fix? |
This issue is blocking using custom prompt in the ml inference response processor, I have another idea to make this genAI use case to work, when the model_config contains the the field from input_map and output_map, we should try to cast the datatype into escape string. for example, ${input_map.context} in going to be in the substring of ${model_config.prompt}, we can try cast the ${input_map.context} into escape string or it won't be a valid payload later. @rbhavna can you try this case using the latest main branch and reproduce this issue?
|
What is the bug?
since this change in 7cd5291 in ModelExecutor, the fields in model_config prefix is changed in payload.
before this commit, when using the
model_config
parameters, the parameter are usually prefix withparameters
, for exampleparameters.prompt
.after this commit, when using the
model_config
parameters, the parameter will be prefix with 'model_config', for example,model_config.prompt
.now that using ml inference processor with model_config, on both search request, response and ingest, for remote model, we need to set
model_input
to rewrite the prefix, or the model_config won't be recognized by the connector response bodyThe pros about the parameters prefix is that it can identify the same parameters name within
input_map
andmodel_config
when compilingmodel_input
. However, for remote models, the inference processor passed over config such asmodel_config.prompt
, it won't be recognized in the response body, because in the connector, all parameters are prefix withparameter
, for exampleparameters.prompt
.How can one reproduce the bug?
Steps to reproduce the behavior:
example 1, this is failing:
example 2, this is success when we rewrite the
model_config.prompt
intoparameters.prompt
What is the expected behavior?
example 1 should be succeed.
What is your host/environment?
Do you have any screenshots?
If applicable, add screenshots to help explain your problem.
Do you have any additional context?
Add any other context about the problem.
The text was updated successfully, but these errors were encountered: