fastpath inference is not supported

[bigtree2020]

I found the fastpath inference seems not supported, which is an optimization in
https:/mlfoundations/open_clip/blob/main/src/open_clip/transformer.py#L224.

May I know what is the reason to avoid using this in the current impl?

[rwightman]

we kept it in default batch_first=False only because the original was like that, could change it around so that fastpath could work in inference cases...

[rwightman]

I merged low risk changes, but haven't touched the masking which will prevent fastpath from being used for the text transformer still