-
Notifications
You must be signed in to change notification settings - Fork 354
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
TCAV with a DeBERTa model #1465
Comments
Hi @elenagaz! Thanks for your interest in LIT. TCAV is included as one of LIT's default interpreters, but it is only compatible with certain model APIs, and specifically with classification models. Compatibility checks happen at initialization time for any To summarize the linked function, at least one model needs to 1) predict a I know the above can be a lot to process, so if there's any way you can share your model classes and server script (e.g., a Gist or PR), I'm happy to take a look and provide feedback about why you might not be seeing the TCAV option and how to make it work. |
Hi @RyanMullins Yes, it does incorporate a few things. As I wrote before, I have used the glue_models.py example included in the code. I did make some changes, to be able to use a DeBERTa model. But as you pointed out, the ouput_spec has proven to be a bit difficult. The model I am using is a classification model that has an input such as ‘I would like to test this model.’ but returns 9 probabilities, in comparison to the glue_models I have seen. This is the way I input the data when using my model with the glue_model.py structure.
This would be the output - with the input: ‘I would like to test this model.’ With the unedited probabilities like this: I particularly have trouble computing the grads If you have any input on how I could edit the input or output specification, I would appreciate it. |
Hi, I want to test a classification model of type DeBERTa and definitely want to use TCAV for my evaluation.
I have gone through the glue_model.py files and the glue_demo.py, but I cannot seem to find where exactly the TCAV implementation is added to a new demo.
Is there any documentation on where and how the TCAV can be added?
And are there any prerequisites for it to work?
Thanks in advance.
The text was updated successfully, but these errors were encountered: