【Hackathon No57】add_bf16_fp16 unittest for conv3d & conv3d_transpose #52195

Difers · 2023-03-27T10:33:26Z

PR types

Others

PR changes

APIs

Describe

为conv3d添加BF16单测， FP16单测添加反向
为从conv3d_transpose添加FP16,BF16单测

paddle-bot · 2023-03-27T10:33:31Z

你的PR提交成功，感谢你对开源项目的贡献!
请关注后续CI自动化测试结果，详情请参考Paddle-CI手册。
Your PR has been submitted. Thanks for your contribution!
Please wait for the result of CI firstly. See Paddle CI Manual for details.

Vvsmile · 2023-03-27T11:38:10Z

python/paddle/fluid/tests/unittests/test_conv3d_op.py

+ # TODO(wangzhongpu): support mkldnn op in dygraph mode
+ place = core.CUDAPlace(0) if self.has_cudnn() else core.CPUPlace()
+ self.check_output_with_place(
+ place, atol=1e-2, check_dygraph=(not self.use_mkldnn)


atol在BF16下，默认值就是1e-2，可以移除掉

Vvsmile · 2023-03-27T11:40:35Z

python/paddle/fluid/tests/unittests/test_conv3d_op.py

+ check_dygraph=(not self.use_mkldnn),
+ )
+
+ cls_name = "{0}_{1}".format(parent.__name__, "CUDNNBF16")


“CUDNNBF16” 改为“CUDNNBF16OP”

Vvsmile · 2023-03-27T11:40:37Z

python/paddle/fluid/tests/unittests/test_conv3d_transpose_op.py

+ place, ['Filter'], 'Output', no_grad_set=set(['Input'])
+ )
+
+ cls_name = "{0}_{1}".format(parent.__name__, "CUDNNFp16")


“CUDNNFp16”改为“CUDNNFP16OP”

Vvsmile · 2023-03-27T11:42:39Z

python/paddle/fluid/tests/unittests/test_conv3d_transpose_op.py

+ if core.is_compiled_with_cuda():
+ place = core.CUDAPlace(0)
+ if core.is_float16_supported(place):
+ self.check_output_with_place(place, atol=1e-5)


atol可以1e-5有些太低了，请使用默认值1e-3（FP16阈值内部默认1e-3，所以可以不设置此atol参数）

Vvsmile · 2023-03-27T11:43:55Z

python/paddle/fluid/tests/unittests/test_conv3d_transpose_op.py

+
+ def test_check_output(self):
+ place = core.CUDAPlace(0)
+ self.check_output_with_place(place, atol=1e-5)


bf16中atol的1e-5的设置太低了，请使用1e-2（BF16阈值内部默认1e-2，所以可以不设置此atol参数）

Vvsmile · 2023-03-27T11:44:12Z

python/paddle/fluid/tests/unittests/test_conv3d_transpose_op.py

+ user_defined_grads=[numeric_grads],
+ )
+
+ cls_name = "{0}_{1}".format(parent.__name__, "CUDNNBF16")


"CUDNNBF16"改为“CUDNNBF16OP”

Vvsmile · 2023-04-03T07:54:26Z

python/paddle/fluid/tests/unittests/test_conv3d_op.py

- 'Filter': OpTest.np_dtype_to_fluid_dtype(filter),
- }
+ if self.is_bfloat16_op():
+ output = output.astype(np.float32)


output应该也要用convert_float_to_uint16来转换一下

Vvsmile · 2023-04-03T07:56:56Z

python/paddle/fluid/tests/unittests/test_conv3d_transpose_op.py

+ ['Input'],
+ 'Output',
+ no_grad_set={'Filter'},
+ user_defined_grads=[numeric_grads],


这个地方请先用不带user_defined_grads的check_grad_with_place参数尝试一下

Vvsmile · 2023-04-03T07:57:05Z

python/paddle/fluid/tests/unittests/test_conv3d_transpose_op.py

+ ['Filter'],
+ 'Output',
+ no_grad_set={'Input'},
+ user_defined_grads=[numeric_grads],


这个地方请先用不带user_defined_grads的check_grad_with_place参数尝试一下

Vvsmile · 2023-04-03T07:58:32Z

python/paddle/fluid/tests/unittests/test_conv3d_transpose_op.py

 ).astype("float32")

+ if self.is_bfloat16_op():
+ output = output.astype(np.float32)


bf16类型下的output需要调用convert_float_to_uint16转换一下

这里使用convert_float_to_uint16有一些问题，对于NHWC格式的数据它会进行一下transpose，导致check_output时expect_np与actual_np维度对应不上出错

Difers · 2023-04-05T01:12:39Z

done,辛苦再review下~

Difers · 2023-04-11T13:01:01Z

@Vvsmile 我发现convert_float_to_uint16维度转换只有四维，而test_conv3d_transpose_op.py中应为5维，就自行添加了一个，辛苦有空review下~

ZzSean · 2023-04-13T11:56:02Z

python/paddle/fluid/tests/unittests/test_conv3d_op.py

+ def test_check_grad(self):
+ if hasattr(self, "no_need_check_grad") and self.no_need_check_grad:
+ return
+ place = core.CUDAPlace(0) if self.has_cudnn() else core.CPUPlace()


这里只测CUDAPlace，且has_cudnn为False的时候好像不支持吧

ZzSean · 2023-04-13T11:58:46Z

python/paddle/fluid/tests/unittests/test_conv3d_op.py

- 'Filter': OpTest.np_dtype_to_fluid_dtype(filter),
- }
+ if self.is_bfloat16_op():
+ output = convert_float_to_uint16(output)


这里跟之前的错误类似，当uint16的时候顺序不太对，应该先初始化为float32，计算完结果后将输入输出convert

ZzSean · 2023-04-13T12:06:49Z

python/paddle/fluid/tests/unittests/test_conv3d_transpose_op.py

+ @unittest.skipIf(
+ not core.is_compiled_with_cuda(), "core is not compiled with CUDA"
+ )
+ class TestConv3DCUDNNFP16(parent):


名字需要修改下，加上Transpose

ZzSean · 2023-04-13T12:07:46Z

python/paddle/fluid/tests/unittests/test_conv3d_transpose_op.py


 from paddle.fluid import core


+def convert_float_to_uint16(float_list, data_format="NCHW"):


这里为啥要重写一个？

用原先的，对NCHW dataf_format维度变换是np.transpose(float_list, [0, 3, 1, 2]),但这里data_format是NCDHW，维度变换应该是5维的[0, 4, 1, 2, 3]

ZzSean

LGTM

…addlePaddle#52195) * add test+conv3d_transpose_part2 * fix some merge error * fix codestyle * fix typo * fix codestyle * fix some error * add redef float2uint * fix conv3d and conv3d_transpose

paddle-bot bot added contributor External developers status: proposed labels Mar 27, 2023

Vvsmile reviewed Mar 27, 2023

View reviewed changes

luotao1 assigned luotao1, ZzSean and Ligoml Mar 28, 2023

Difers force-pushed the add_bf_fp_conv branch 4 times, most recently from daa919c to 06bc17d Compare March 30, 2023 06:51

luotao1 assigned Vvsmile Mar 30, 2023

Difers force-pushed the add_bf_fp_conv branch 2 times, most recently from 71bfe22 to d9b4be3 Compare March 31, 2023 02:15

Difers changed the title ~~【Hackathon No57】add_conv3d_conv3dtp_bf_fp16~~ 【Hackathon No57】add_bf16_fp16 unittest for conv3d & conv3d_transpose Mar 31, 2023

Difers force-pushed the add_bf_fp_conv branch from d9b4be3 to e8f6b03 Compare March 31, 2023 14:12

Difers mentioned this pull request Mar 31, 2023

【PaddlePaddle Hackathon 第四期】任务总览 #51281

Closed

Difers force-pushed the add_bf_fp_conv branch from 449d8be to 2e132df Compare April 1, 2023 01:24

Vvsmile reviewed Apr 3, 2023

View reviewed changes

Difers force-pushed the add_bf_fp_conv branch from 2e132df to c3407fa Compare April 4, 2023 11:57

Difers force-pushed the add_bf_fp_conv branch from c3407fa to 3ea44aa Compare April 11, 2023 06:37

ZzSean reviewed Apr 13, 2023

View reviewed changes

Difers added 8 commits April 18, 2023 13:53

add test+conv3d_transpose_part2

b8d4b35

fix some merge error

5172c6c

fix codestyle

2bd3be2

fix typo

6f661ce

fix codestyle

6d68270

fix some error

574ffba

add redef float2uint

43451f8

fix conv3d and conv3d_transpose

4d11f4d

Difers force-pushed the add_bf_fp_conv branch from 3ea44aa to 4d11f4d Compare April 18, 2023 13:53

ZzSean approved these changes Apr 25, 2023

View reviewed changes

luotao1 merged commit eb67710 into PaddlePaddle:develop Apr 25, 2023

ZzSean mentioned this pull request May 5, 2023

[Cherry-Pick] AMP OP&Test support from Hackathon #53522

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

【Hackathon No57】add_bf16_fp16 unittest for conv3d & conv3d_transpose #52195

【Hackathon No57】add_bf16_fp16 unittest for conv3d & conv3d_transpose #52195

Difers commented Mar 27, 2023 •

edited

Loading

paddle-bot bot commented Mar 27, 2023

Vvsmile Mar 27, 2023

Vvsmile Mar 27, 2023

Vvsmile Mar 27, 2023

Vvsmile Mar 27, 2023 •

edited

Loading

Vvsmile Mar 27, 2023

Vvsmile Mar 27, 2023

Difers Mar 27, 2023

Vvsmile Apr 3, 2023

Vvsmile Apr 3, 2023

Vvsmile Apr 3, 2023

Vvsmile Apr 3, 2023

Difers Apr 4, 2023

Difers commented Apr 5, 2023

Difers commented Apr 11, 2023

ZzSean Apr 13, 2023 •

edited

Loading

ZzSean Apr 13, 2023

ZzSean Apr 13, 2023

ZzSean Apr 13, 2023

Difers Apr 13, 2023

ZzSean left a comment


		from paddle.fluid import core


		def convert_float_to_uint16(float_list, data_format="NCHW"):

【Hackathon No57】add_bf16_fp16 unittest for conv3d & conv3d_transpose #52195

【Hackathon No57】add_bf16_fp16 unittest for conv3d & conv3d_transpose #52195

Conversation

Difers commented Mar 27, 2023 • edited Loading

PR types

PR changes

Describe

paddle-bot bot commented Mar 27, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Vvsmile Mar 27, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Difers commented Apr 5, 2023

Difers commented Apr 11, 2023

ZzSean Apr 13, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ZzSean left a comment

Choose a reason for hiding this comment

Difers commented Mar 27, 2023 •

edited

Loading

Vvsmile Mar 27, 2023 •

edited

Loading

ZzSean Apr 13, 2023 •

edited

Loading