use scale=1.0 in floats_tensor called in speech model testers

[ydshieh]

What does this PR do?

Fix the failure of Speech2TextModelTest.test_pt_tf_model_equivalence. This is caused by

transformers/tests/speech_to_text/test_modeling_speech_to_text.py

Lines 134 to 136 in e6f00a1

	input_features = floats_tensor(
	[self.batch_size, self.seq_length, self.input_feat_per_channel], self.vocab_size
	)

where the input_features get a large magnitude of 1e2 (from self.vocab_size=99).

(probably this happens because we just copied the input_ids = ids_tensor([self.batch_size, self.seq_length], self.vocab_size) from NLP models?)

I changed it to scale=1.0, but need @patrickvonplaten's expertise to make sure there was no particular reason to use self.vocab_size.

Details

Current speech model testers have

def prepare_config_and_inputs(self):
    input_values = floats_tensor([self.batch_size, self.seq_length], self.vocab_size)

The self.vocab_size argument is the scale, so the generated dummy input_values has the magnitude of self.vocab_size.
For Speech2TextModelTester, we have vocab_size=99.

Furthermore, Speech2TextEncoder has

transformers/src/transformers/models/speech_to_text/modeling_speech_to_text.py

Line 705 in e6f00a1

self.embed_scale = math.sqrt(embed_dim) if config.scale_embedding else 1.0

and from the tester's hidden_size=16, we get embed_scale=4.

The input_features goes through the conv layer(s) and being scaled:

transformers/src/transformers/models/speech_to_text/modeling_speech_to_text.py

Lines 767 to 768 in e6f00a1

	inputs_embeds = self.conv(input_features)
	inputs_embeds = self.embed_scale * inputs_embeds

On CPU however, the conv layers of PT/TF gives diff. with a magnitude of 1e-7 for input values with 1s. So with the above 2 scalings, this error becomes 4e-5, and the PT/TF equiv. test fails.

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[patrickvonplaten]

wow good catch!

[patrickvonplaten]

You're 100% right - this was indeed a bad copy paste!

[patrickvonplaten]

Thanks for fixing all the tests!

[sgugger]

Nice fix! Thanks a lot!