Fix HubertRobustTest PT/TF equivalence test on GPU

[ydshieh]

What does this PR do?

Fix HubertRobustTest PT/TF equivalence test on GPU.

Note that HubertRobustModelTest has

    def setUp(self):
        self.model_tester = HubertModelTester(
            self, conv_stride=(3, 3, 3), feat_extract_norm="layer", do_stable_layer_norm=True
        )

but get_config() had no do_stable_layer_norm=self.do_stable_layer_norm

To investigate further

• Why no issue on CPU even without this PR
• Why using conv_stride=(4, 4, 4) (the default value) has no issue on GPU, even without this PR

(Does this suggest we have PT/TF Hubert behave differently with do_stable_layer_norm=False on GPU when conv_stride=(3, 3, 3) etc?)

@patrickvonplaten You might have some idea about these points ..?

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[ydshieh]

Think it might be a great idea to add a check to avoid such situation occur in the future. Will do it in another PR.

[anton-l]

Not sure why different combinations of conv_stride and do_stable_layer_norm introduce instabilities.

• conv_stride affects the length of the transformer in the sequence axis (larger strides = stronger subsampling of the input sequence = shorter model)
• do_stable_layer_norm affects the position of LayerNorm in the transformer layer (either before or after self-attention)

So this definitely needs further investigation