Attempting to test automatically the `_keys_to_ignore`. #20042

Narsil · 2022-11-03T15:04:22Z

What does this PR do?

This adds a new part of the tied_weights test that aims at detecting automatically
when _keys_to_ignore is incorrectly set.

_keys_to_ignore aims to ignore weights that are supposed to be tied in the
final model, meaning it's OK if the parameter is missing from the on-disk weights.
The weights are really empty during the load, but they end up being tied afterwards
so we should ignore them during the load if they are missing.

The test also aims to detect _keys_to_ignore that might have been set but
could be misleading because the parameters are actually NOT tied anymore.

Fixes # (issue)

Before submitting

This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
Did you read the contributor guideline,
Pull Request section?
Was this discussed/approved via a Github issue or the forum? Please add a link
to it if that's the case.
Did you make sure to update the documentation with your changes? Here are the
documentation guidelines, and
here are tips on formatting docstrings.
Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

HuggingFaceDocBuilderDev · 2022-11-03T15:28:48Z

The documentation is not available anymore as the PR was closed or merged.

ydshieh · 2022-11-03T16:05:48Z

src/transformers/models/bert/modeling_bert.py

@@ -1153,7 +1155,7 @@ def forward(
 class BertLMHeadModel(BertPreTrainedModel):

 _keys_to_ignore_on_load_unexpected = [r"pooler"]
- _keys_to_ignore_on_load_missing = [r"position_ids", r"predictions.decoder.bias"]
+ _keys_to_ignore_on_load_missing = [r"position_ids", r"predictions.decoder.bias", r"cls.predictions.decoder.weight"]


Let me review this slowly 🙏 and verify a few things. But do you think r"predictions.decoder.bias" is a mistake and should be r"cls.predictions.decoder.bias" ?

I am going to check myself anyway.

A re match is done, so while the exact name is indeed r"cls.predictions.decoder.bias", this works. But would be great to fix just in case one day a weight named predictions.decoder.bias that should not be ignore appears ;-)

ydshieh · 2022-11-03T16:06:09Z

src/transformers/models/bert/modeling_bert.py

@@ -1153,7 +1155,7 @@ def forward(
 class BertLMHeadModel(BertPreTrainedModel):

 _keys_to_ignore_on_load_unexpected = [r"pooler"]
- _keys_to_ignore_on_load_missing = [r"position_ids", r"predictions.decoder.bias"]
+ _keys_to_ignore_on_load_missing = [r"position_ids", r"predictions.decoder.bias", r"cls.predictions.decoder.weight"]


Let me review this slowly 🙏 and verify a few things. But do you think r"predictions.decoder.bias" is a mistake and should be r"cls.predictions.decoder.bias" ?

I am going to check myself anyway.

Maybe. I'm not sure but the test does yell if we're hiding a valid key (we I don't try to yell when we have an unused _keys_to_ignore. Want me to try and add it to the test ?

sgugger

Interesting way of testing those attributes!

sgugger · 2022-11-03T16:08:42Z

src/transformers/models/bert/modeling_bert.py

@@ -1153,7 +1155,7 @@ def forward(
 class BertLMHeadModel(BertPreTrainedModel):

 _keys_to_ignore_on_load_unexpected = [r"pooler"]
- _keys_to_ignore_on_load_missing = [r"position_ids", r"predictions.decoder.bias"]
+ _keys_to_ignore_on_load_missing = [r"position_ids", r"predictions.decoder.bias", r"cls.predictions.decoder.weight"]


A re match is done, so while the exact name is indeed r"cls.predictions.decoder.bias", this works. But would be great to fix just in case one day a weight named predictions.decoder.bias that should not be ignore appears ;-)

sgugger · 2022-11-03T16:10:04Z

tests/test_modeling_common.py

@@ -1538,6 +1538,36 @@ def check_same_values(layer_1, layer_2):
 # # Check that the embedding layer and decoding layer are the same in size and in value
 # self.assertTrue(model.transformer.wte.weight.shape, model.lm_head.weight.shape)
 # self.assertTrue(check_same_values(model.transformer.wte, model.lm_head))
+ with tempfile.TemporaryDirectory() as d:


Add a comment here on what's check? Also maybe it should be in its own test?

Moving to its own test + comment on how it works.

Narsil · 2022-11-04T08:10:57Z

src/transformers/modeling_utils.py

@@ -2422,7 +2422,8 @@ def _fix_key(key):

 if remove_prefix_from_model:
 expected_keys_not_prefixed = [s for s in expected_keys if not s.startswith(prefix)]
- expected_keys = [".".join(s.split(".")[1:]) if s.startswith(prefix) else s for s in expected_keys]
+ _prefix = f"{prefix}."
+ expected_keys = [".".join(s.split(".")[1:]) if s.startswith(_prefix) else s for s in expected_keys]


This would fail for esm/esmfold since esm is a prefix for esmfold it would remove emsfold. prefix instead of the intended esm.

This was silent before, but with the new test addition the following modification would fail to get key[-1] when trying to identify missing keys.

Seems like the right fix, thanks!

Ok, there was a second place to update the fix to make the splinter test pass.

https:/huggingface/transformers/pull/20042/files#diff-6b72b98c4c2dcfc6cc606843917733f5d858374fbc22a735ff483bbc0c1e63eaR2655

Narsil · 2022-11-04T08:15:04Z

tests/test_modeling_common.py

@@ -1485,11 +1485,24 @@ def test_correct_missing_keys(self):
 base_model_prefix = model.base_model_prefix

 if hasattr(model, base_model_prefix):
+


This is an important new modification.

This test tries to verify that we're correctly warning when we're saving the "core" of a model, and failing to save the head parameters, and that we yell when loading the head from this.

This test is great, and different from the newly added one.
But, since we fixed a lot of Decoder head which are merely a copy of the embeddings matrix, then they are a head, but they DO NOT have new extra parameters. hence this test would fail.

There is self.test_missing_keys flag to deactivate that particular test on the model testing. But it would deactivate ALL potential heads (as far as I understand), potentially opening gaps in our testing.

So I modified this test to try and detect those edge cases. I don't like putting logic like that in test as it makes reasoning hard but I failed to see another way.

This test verifies that the head does have extra parameters before actually doing the verification.

Make sure you account in the test for upcoming models which have weights with the same name in the model with head and the base model (Niels is adding one in #19981)

I don't see which weights are sharing names.

Seems relatively regular to me: https:/huggingface/transformers/pull/19981/files#diff-a2ff7478a3e58383ae065b28d9e5e4e1e1bace51b32bc91b6eb086fabb0bd702R448-R470

Narsil · 2022-11-04T08:18:07Z

tests/test_modeling_common.py

+ " tying its weights",
+ )
+
+ # self.assertEqual(


I deactivated this part because token_type_ids and position_ids would yell pretty much everywhere.

I have no idea of the perfect solution for this.
I think position_ids should be a non-parameter if it's not, or a parameter if it is learned or something.
Given how many models it affect we could add an escape hatch for just these peculiar names.

But I think flagging actually too inclusive _keys_to_ignore can provide good value, especially since we're using regexp so it's easy to make a mistake, or modify the model later in such a way that it doesn't reflect properly anymore in those keys.

Narsil · 2022-11-04T08:33:19Z

@ydshieh the splinter test failing is normal ?

FAILED tests/models/splinter/test_modeling_splinter.py::SplinterModelTest::test_save_load_fast_init_from_base - AssertionError: 3069.73388671875 not less than or equal to 0.001 : splinter_qass.query_start_transform.dense.weight not identical

ydshieh · 2022-11-04T13:17:29Z

@Narsil

I am not able to reproduce the splinter test failure you mentioned above with current main on a GCP GPU VM. Could you provide more information about your environment and how you launched the test?

Narsil · 2022-11-04T13:19:13Z

IT's this failure : https://app.circleci.com/jobs/github/huggingface/transformers/608396

in case they will trigger warnigns in the `transformers` side. Even if the model is perfectly fine, core maintainers fear an influx of opened issues. This is perfectly legit. On the `transformers` side fixes are on the way: huggingface/transformers#20042 We can wait for this PR to hit `main` before communicating super widely. In the meantime this script of convertion will now prevent converting models that would trigger such warnings (so the output of the script **will** depend on the `transformers` freshness.

ydshieh

Just a few comments - only for the test test_tied_model_weights_key_ignore so far.

ydshieh · 2022-11-04T16:07:10Z

tests/test_modeling_common.py

@@ -1539,6 +1552,44 @@ def check_same_values(layer_1, layer_2):
 # self.assertTrue(model.transformer.wte.weight.shape, model.lm_head.weight.shape)
 # self.assertTrue(check_same_values(model.transformer.wte, model.lm_head))

+ def test_tied_model_weights_key_ignore(self):


I would suggest to make the name more generic (i.e. not using tied_model in the name)

The check in this test is not really specific to tied weights, although they are one of the main source of missing keys.

test_model_weights_key_ignore + a comment to explain its objective would be fine.

In my mind it is only about tied weights (because it's the only mechanism I know that can create the weight sharing).

Happy to update the name

ydshieh · 2022-11-04T16:11:41Z

tests/test_modeling_common.py

+ # We are nuking ALL weights on file, so every parameter should
+ # yell on load. We're going to detect if we yell too much, or too little.
+ with open(os.path.join(d, "pytorch_model.bin"), "wb") as f:
+ torch.save({}, f)


A comment here would be nice, as this is unusual.

I gave you the wrong link (on an old commit, the comment is actually there).

Is it readable ?

Oh, yes! Sorry, I actually didn't understand that We are nuking ALL weights on file, so every parameter should ... which is quite clear already.

ydshieh · 2022-11-04T16:18:16Z

tests/test_modeling_common.py

+
+ prefix = f"{model_reloaded.base_model_prefix}."
+ params = dict(model_reloaded.named_parameters())
+ params.update(dict(model_reloaded.named_buffers()))


You mentioned on DM

They are listed as two different things during from_pretrained but only appear once when doing model.named_parameters()

I haven't 100% understand it (despite got some rough feeling about it). Could you explain

why not using model_reloaded.state_dict() -- (from your word, I guess we will get 2 things, and it will match the missing keys, which will pass the test when the situation should fail)

Give an example, say what happens to bert or gpt2 for this block against current main

In any case, a comment (inside the code) should be added here regarding the usage of named_parameters and named_parameters 🙏 .

why not using model_reloaded.state_dict() -- (from your word, I guess we will get 2 things, and it will match the missing keys, which will pass the test when the situation should fail)

Exactly ! Actually we could use state_dict() and check iteratively the tensors which are the same (for instance using tensor.data_ptr()). to detect the duplicates.

model = GPT2LMHeadModel.from_pretrained("gpt2") "lm_head.weight" in model.state_dict().keys() # True "lm_head.weight" in model.named_parameters() # False In [6]: model.lm_head.weight.data_ptr() Out[6]: 139901378371648 In [9]: model.transformer.wte.weight.data_ptr() Out[9]: 139901378371648 # Same PTR, it's the same DATA ! we would need to check for stride too to be 100% accurate.

I'll add a comment.

As long as using named_parameters/named_buffer works, I wouldn't request to this fancier check :-)

) * Adding the convert scripts which will now prevent converting models in case they will trigger warnigns in the `transformers` side. Even if the model is perfectly fine, core maintainers fear an influx of opened issues. This is perfectly legit. On the `transformers` side fixes are on the way: huggingface/transformers#20042 We can wait for this PR to hit `main` before communicating super widely. In the meantime this script of convertion will now prevent converting models that would trigger such warnings (so the output of the script **will** depend on the `transformers` freshness. * Adding a nicer diff for the error when reloading.

ydshieh · 2022-11-04T16:39:27Z

tests/test_modeling_common.py

+ self.assertEqual(
+ extra_missing,
+ set(),
+ f"This model {model_class.__name__} might be missing some `keys_to_ignore`: {extra_missing} when"


Forgot to mention, we could remove when tying its weights here.

sgugger

Amazing work! Some styling comments and parts that seem orthogonal to this PR.

sgugger · 2022-11-04T16:39:29Z

src/transformers/modeling_utils.py

@@ -2422,7 +2422,8 @@ def _fix_key(key):

 if remove_prefix_from_model:
 expected_keys_not_prefixed = [s for s in expected_keys if not s.startswith(prefix)]
- expected_keys = [".".join(s.split(".")[1:]) if s.startswith(prefix) else s for s in expected_keys]
+ _prefix = f"{prefix}."
+ expected_keys = [".".join(s.split(".")[1:]) if s.startswith(_prefix) else s for s in expected_keys]


Seems like the right fix, thanks!

sgugger · 2022-11-04T16:40:00Z

src/transformers/modeling_utils.py

- module_keys = module_keys.union(set([".".join(key.split(".")[:-2]) for key in names if key[-1].isdigit()]))
+ module_keys = module_keys.union(
+ set([".".join(key.split(".")[:-2]) for key in names if key and key[-1].isdigit()])
+ )


Why the change here? Can we revert if it's just styling?

It's not just styling I'm adding if key and key[-1] vs if key[-1] it should prevent failing even if there is a bad prefix.

I still could revert though. I left it to prevent further failure, but we're already in a bad state if this fails so...

if key -> if key is not None or if len(key) > 0 (not sure what you are testing).

GitHub made it hard to read the diff, sorry.

It was "" so len(key) > 0 it is.

src/transformers/models/bart/modeling_bart.py

src/transformers/models/bigbird_pegasus/modeling_bigbird_pegasus.py

src/transformers/models/longt5/modeling_longt5.py

src/transformers/models/mvp/modeling_mvp.py

sgugger · 2022-11-04T16:43:28Z

src/transformers/models/prophetnet/modeling_prophetnet.py

- assert list(hidden_states.size()) == [
- batch_size,
- ngram_sequence_length,
- hidden_size,
- ], (
+ assert list(hidden_states.size()) == [batch_size, ngram_sequence_length, hidden_size], (
 f"`hidden_states` should be of shape {batch_size, ngram_sequence_length, hidden_size}, but is of shape"
 f" {hidden_states.shape}"
 )


Seems unrelated, and if we are going to change it, let's make it a real test and an exception raised :-p

I'm pretty sure it's my black that does this. However it should be 22.3 Like `transformers expect. Not sure where the diff comes from.

And fixup doesn't revert those :(

sgugger · 2022-11-04T16:43:47Z

src/transformers/models/xlm_prophetnet/modeling_xlm_prophetnet.py

- ngram_sequence_length,
- hidden_size,
- ], (
+ assert list(hidden_states.size()) == [batch_size, ngram_sequence_length, hidden_size], (


Same as before.

Black is not happy about this the old code it forces me to use this new layout.

I tried to revert this.

sgugger · 2022-11-04T16:45:53Z

tests/test_modeling_common.py

@@ -1485,11 +1485,24 @@ def test_correct_missing_keys(self):
 base_model_prefix = model.base_model_prefix

 if hasattr(model, base_model_prefix):
+


Make sure you account in the test for upcoming models which have weights with the same name in the model with head and the base model (Niels is adding one in #19981)

LysandreJik

Impressive work! The changes in common look good to me.
The tests look quite robust as well.

Narsil · 2022-11-08T17:08:11Z

@ydshieh This tests now fails in the CI tests/models/wav2vec2_conformer/test_modeling_wav2vec2_conformer.py::Wav2Vec2ConformerModelTest::test_save_load_fast_init_from_base

However I can't seem to be able to reproduce locally ? Do you mind trying if it's my setup failing or the CI ?

the missing keys check.

Co-authored-by: Sylvain Gugger <[email protected]>

Narsil · 2022-11-09T15:03:28Z

Merging.

fingers crossed :)

…20042) * Attempting to test automatically the `_keys_to_ignore`. * Style. * First fix pass. * Moving test on its own. * Another batch. * Second round removing BatchNorm * Fixing layoutlmv{2,3} + support older Python. * Disable miss missing warning. * Removing dodgy additions. * Big pass. * mbart. * More corrections. * Fixup. * Updating test_correct_missing_keys * Add escape hatch for when the head has no extra params so doesn't need the missing keys check. * Fixing test. * Greener. * Green ! (except for weird splinter bug). * Adding a test about `named_parameters` usage. * Shorten message. * Apply suggestions from code review Co-authored-by: Sylvain Gugger <[email protected]> * After rebase modifications. * More explicit condition checking. * Fixing slow tests issues. * Remove extra pdb. * Remove print. * Attempt to make failure consistent + fixing roc_bert. * Removing the seed (all tests passing with it). Co-authored-by: Sylvain Gugger <[email protected]>

younesbelkada · 2022-11-21T16:21:03Z

Hey!
I am not sure if it is because of this PR but loading NLLB (that is affected by this PR) now gives:

│ /home/younes_huggingface_co/debug_issues/code/transformers/src/transformers/modeling_utils.py:24 │
│ 59 in _load_pretrained_model                                                                     │
│                                                                                                  │
│   2456 │   │   │   for key in missing_keys:                                                      │
│   2457 │   │   │   │   if key.startswith(prefix):                                                │
│   2458 │   │   │   │   │   key = ".".join(key.split(".")[1:])                                    │
│ ❱ 2459 │   │   │   │   param = model_state_dict[key]                                             │
│   2460 │   │   │   │   if param.device == torch.device("meta"):                                  │
│   2461 │   │   │   │   │   if not load_in_8bit:                                                  │
│   2462 │   │   │   │   │   │   set_module_tensor_to_device(model, key, "cpu", torch.empty(*para  │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
KeyError: 'encoder.embed_positions.weights'

Here is the snippet to reproduce the error:

from transformers import AutoModelForSeq2SeqLM, AutoTokenizer

src_lang = "eng_Latn"
tgt_lang = "spa_Latn"

tokenizer = AutoTokenizer.from_pretrained("facebook/nllb-200-distilled-600M", src_lang=src_lang)
model = AutoModelForSeq2SeqLM.from_pretrained("facebook/nllb-200-distilled-600M",
                                              device_map= "auto")

I did not followed entirely this PR but I will dig into that now and see what exactly caused the issue 💪

cc @Narsil @sgugger

younesbelkada · 2022-11-21T16:23:33Z

src/transformers/models/m2m_100/modeling_m2m_100.py

@@ -1244,12 +1246,8 @@ class M2M100ForConditionalGeneration(M2M100PreTrainedModel):
 r"encoder.version",
 r"decoder.version",
 r"lm_head.weight",
- r"model.encoder.embed_positions.weights",


I realized these are removed, is this intended (I see that it's one of the only model where some keys are totally removed )? Maybe it has been done by mistake but I am not sure

@Narsil would know best

Seems I forgot to re-add this.

This was removed because of the test which is now commented which tries to remove _key_to_ignore which are masking real weights.

This SinusoidalPositionEmbeddings, seems to be a buffer and not a parameter, but something is wrong somewhere.
Since it's a nn.Module, maybe it still sees weights as being a Parameter somehow, idk.

We can safely revert this part anyway.

Thanks a lot for the explanation ! 🚀 The PR to revert is in #20381

…20042) * Attempting to test automatically the `_keys_to_ignore`. * Style. * First fix pass. * Moving test on its own. * Another batch. * Second round removing BatchNorm * Fixing layoutlmv{2,3} + support older Python. * Disable miss missing warning. * Removing dodgy additions. * Big pass. * mbart. * More corrections. * Fixup. * Updating test_correct_missing_keys * Add escape hatch for when the head has no extra params so doesn't need the missing keys check. * Fixing test. * Greener. * Green ! (except for weird splinter bug). * Adding a test about `named_parameters` usage. * Shorten message. * Apply suggestions from code review Co-authored-by: Sylvain Gugger <[email protected]> * After rebase modifications. * More explicit condition checking. * Fixing slow tests issues. * Remove extra pdb. * Remove print. * Attempt to make failure consistent + fixing roc_bert. * Removing the seed (all tests passing with it). Co-authored-by: Sylvain Gugger <[email protected]>

ydshieh reviewed Nov 3, 2022

View reviewed changes

sgugger reviewed Nov 3, 2022

View reviewed changes

Narsil commented Nov 4, 2022

View reviewed changes

Narsil mentioned this pull request Nov 4, 2022

Adding the convert scripts which will now prevent converting models huggingface/safetensors#59

Merged

ydshieh reviewed Nov 4, 2022

View reviewed changes

sgugger reviewed Nov 4, 2022

View reviewed changes

sgugger requested a review from LysandreJik November 4, 2022 16:47

Narsil force-pushed the tied_weights_warning_check branch from 5c25bf8 to 72493aa Compare November 4, 2022 17:04

LysandreJik approved these changes Nov 4, 2022

View reviewed changes

Narsil added 12 commits November 9, 2022 11:01

Attempting to test automatically the _keys_to_ignore.

134f8a8

Style.

6706337

First fix pass.

9e33f8a

Moving test on its own.

2225e65

Another batch.

0d1642c

Second round removing BatchNorm

602edaf

Fixing layoutlmv{2,3} + support older Python.

7851df5

Disable miss missing warning.

8f6a6ae

Removing dodgy additions.

f53fb25

Big pass.

7a7b378

mbart.

e7cb3df

More corrections.

1d8811c

Narsil and others added 14 commits November 9, 2022 11:01

Fixup.

e66d960

Updating test_correct_missing_keys

7039418

Add escape hatch for when the head has no extra params so doesn't need

8e7c939

the missing keys check.

Fixing test.

3f38678

Greener.

79c7e6e

Green ! (except for weird splinter bug).

6c399d4

Adding a test about named_parameters usage.

c3bfad1

Shorten message.

81b3692

Apply suggestions from code review

fb6007b

Co-authored-by: Sylvain Gugger <[email protected]>

After rebase modifications.

56411ac

More explicit condition checking.

d845b3a

Fixing slow tests issues.

85c4bb2

Remove extra pdb.

e2f6a84

Remove print.

3354c8e

Narsil force-pushed the tied_weights_warning_check branch from ea9a45c to 3354c8e Compare November 9, 2022 10:01

Narsil added 2 commits November 9, 2022 11:28

Attempt to make failure consistent + fixing roc_bert.

f552a91

Removing the seed (all tests passing with it).

bb3673c

sgugger approved these changes Nov 9, 2022

View reviewed changes

Narsil merged commit bac2d29 into huggingface:main Nov 9, 2022

Narsil deleted the tied_weights_warning_check branch November 9, 2022 15:03

ydshieh mentioned this pull request Nov 14, 2022

mark test_save_load_fast_init_from_base as is_flaky #20200

Merged

younesbelkada reviewed Nov 21, 2022

View reviewed changes

		@@ -1485,11 +1485,24 @@ def test_correct_missing_keys(self):
		base_model_prefix = model.base_model_prefix

		if hasattr(model, base_model_prefix):

Attempting to test automatically the _keys_to_ignore. #20042

Attempting to test automatically the _keys_to_ignore. #20042

Conversation

Narsil commented Nov 3, 2022

What does this PR do?

Before submitting

Who can review?

HuggingFaceDocBuilderDev commented Nov 3, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sgugger left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Narsil commented Nov 4, 2022

ydshieh commented Nov 4, 2022 • edited Loading

Narsil commented Nov 4, 2022

ydshieh left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ydshieh Nov 4, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sgugger left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

LysandreJik left a comment

Choose a reason for hiding this comment

Narsil commented Nov 8, 2022

Narsil commented Nov 9, 2022

younesbelkada commented Nov 21, 2022 • edited Loading

younesbelkada Nov 21, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

younesbelkada Nov 22, 2022 • edited Loading

Choose a reason for hiding this comment

Attempting to test automatically the `_keys_to_ignore`. #20042

Attempting to test automatically the `_keys_to_ignore`. #20042

HuggingFaceDocBuilderDev commented Nov 3, 2022 •

edited

Loading

ydshieh commented Nov 4, 2022 •

edited

Loading

ydshieh Nov 4, 2022 •

edited

Loading

younesbelkada commented Nov 21, 2022 •

edited

Loading

younesbelkada Nov 21, 2022 •

edited

Loading

younesbelkada Nov 22, 2022 •

edited

Loading