Tolerate misbehaving 3rd party kafka clients with wrong member metadata

[krvladislav]

I test migration from segmentio/kafka-go to franz-go and I met errors like next one when deploy franz-go version application over kafka-go version:

join and sync loop errored%!(EXTRA string=group, string=results, string=err, *errors.errorString=unable to create group member balancer: unable to read member metadata: response did not contain enough data to be valid, string=consecutive_errors, int=1, string=backoff, time.Duration=253.795577ms)

If I am not mistaken, there is a misbehavior in kafka-go segmentio/kafka-go#1156, which sends member metadata v1 version without OwnedPartitions that doesn't comply Kafka Protocol. When decoding metadata, franz-go expects OwnedPartitions must be provided and return the error https:/twmb/franz-go/blob/master/pkg/kmsg/generated.go#L1653

My suggestion is to provide soft handling like sarama does here https:/Shopify/sarama/blob/main/consumer_group_members.go#L43 . It will help to migrate from another kafka clients to franz-go smoothly.

Kafka Version
2.6.0.0
franz-go
v1.13.4

[twmb]

I'll add a workaround that, if the first two bytes are {0, 1} and the decoding fails, retries as v0.

I'll leave this workaround only for v1 at the moment -- hopefully clients don't claim higher and higher support without actually supporting the higher protocol fields.

[twmb]

Specifically this commit, if you want to give it a shot: 2572b9a
I aim to merge soonish.

[krvladislav]

Great, thanks for the fast response

[krvladislav]

I tried 1.14.1 on a migration from segmentio/kafka-go and now it works smoothly. Thanks for the resolution.