Support kafka 2.1, and misc improvements. #6
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
[KAFKA-6] Revamp to remove zookeeper usage
…variable overriding.
Switch back to kingpin for arg parsing
…group position, report latest partition offsets, only collect stats for topics that a consumer group is actually tracking
Report absolute consumer and topic offsets
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Greetings. This is a bit of a beefy pull request, so I'd like to outline the changes that we made and why, and hopefully we can discuss any of the issues in detail.
Upgraded the supported Kafka version to 2.1. This is the biggest change and the driver for most of the rest of the work. Kafka 2 no longer stores consumer group information in zookeeper. Instead it stores it in Kafka proper. With this, users no longer need to provide zookeeper info, as everything is queryable from Kafka itself. Right now its hardcoded to expect kafka 2.1, but that's really because that's the only version I tested it on (and coded against).
Swapped out the statsd library. The statsd format for tagged stats differs between influxdb and datadog. The tagging format used previously ended up sending garbage to datadog. The new statsd library properly serializes tagged stats for those 2 providers and the tagless fallback.
Removed the
onbuilddocker dependency. Onbuild images have been deprecated (see [Proposal] Deprecateonbuildtags docker-library/official-images#2076). Instead we're using a normal golang image and a multi-stage build to keep the image size and layer count down.Exposed cli options as environment variables. This was specifically an issue because we launched this statsd daemon on kubernetes. In our kubernetes cluster, each worker has a datadog agent running locally. We wanted this kafka-statsd to talk to the local datadog agent, and you do this in kubernetes via an environment variable definition. This isn't possible when specifying parameters as command line options. After that fix it only made sense to expose the rest of the options as environment variables.
Recording absolute partition and consumer group offsets. In addition to the consumer lag, we're now also reporting partition and group offsets. It was useful for debugging purposes. In theory we don't actually need to record consumer group lag anymore since that can be calculated from the partition and group offsets, however that would be breaking backwards compatibility.
Kafka calls made concurrent. We need to make a lot of calls to kafka to pull stats information. We just made them concurrent to reduce the chance of falling behind the sampling frequency.