Skip to content

Commit

Permalink
bench: use no compression by default, remove -no-compression flag
Browse files Browse the repository at this point in the history
Sometimes the default of snappy trips people up, and it's better to
default to high throughput for benchmarking and let people change
settings manually.
  • Loading branch information
twmb committed Oct 17, 2021
1 parent d368d11 commit af4fce4
Show file tree
Hide file tree
Showing 2 changed files with 16 additions and 23 deletions.
8 changes: 3 additions & 5 deletions examples/bench/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ in TLS or SASL.
go run . -topic foo
go run . -topic foo -consume
go run . -topic foo -no-compression
go run . -topic foo -compression snappy
go run . -tls -sasl-method scram-sha-256 -sasl-user user -sasl-pass pass -consume -group group -topic foo
```
Expand All @@ -33,7 +33,7 @@ This client has a few different default flags in comparison to librdkafka's
`rdkafka_performance` utility. For starters, this prints less stats, but stats
can be added if requested.

To operate similarly to librdkafka's benchmarker, use `-linger 1s -compression none -static-record`.
To operate similarly to librdkafka's benchmarker, use `-linger 1s -static-record`.

This will match `rdkafka_performance` flags of `./rdkafka_performance -P -t <topic> -b <brokers> -s 100 -i 1000`.

Expand All @@ -60,9 +60,7 @@ any comma delimited set of brokers.

`-batch-max-bytes` specifies the maximum amount of bytes per partition when producing. This must be less than Kafka's max.message.bytes value.

`-no-compression` disables compression.

`-compression` sets the compression to use, overriding the default of snappy. Supports "", "none", "snappy", "lz4", and "zstd".
`-compression` sets the compression to use. Supports "", "none", "snappy", "lz4", and "zstd".

`-pool` enables using a `sync.Pool` to reuse records and value slices, reducing
garbage as a factor of the benchmark.
Expand Down
31 changes: 13 additions & 18 deletions examples/bench/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -32,8 +32,7 @@ var (
useStaticValue = flag.Bool("static-record", false, "if true, use the same record value for every record (eliminates creating and formatting values for records; implies -pool)")

recordBytes = flag.Int("record-bytes", 100, "bytes per record (producing)")
noCompression = flag.Bool("no-compression", false, "set to disable compression (alias for -compression none, producing)")
compression = flag.String("compression", "snappy", "compression algorithm to use (none,gzip,snappy,lz4,zstd, for producing)")
compression = flag.String("compression", "none", "compression algorithm to use (none,gzip,snappy,lz4,zstd, for producing)")
poolProduce = flag.Bool("pool", false, "if true, use a sync.Pool to reuse record structs/slices (producing)")
noIdempotency = flag.Bool("disable-idempotency", false, "if true, disable idempotency (force 1 produce rps)")
linger = flag.Duration("linger", 0, "if non-zero, linger to use when producing")
Expand Down Expand Up @@ -137,23 +136,19 @@ func main() {
if *linger != 0 {
opts = append(opts, kgo.ProducerLinger(*linger))
}
if *noCompression {
switch strings.ToLower(*compression) {
case "", "none":
opts = append(opts, kgo.ProducerBatchCompression(kgo.NoCompression()))
} else {
switch strings.ToLower(*compression) {
case "", "none":
opts = append(opts, kgo.ProducerBatchCompression(kgo.NoCompression()))
case "gzip":
opts = append(opts, kgo.ProducerBatchCompression(kgo.GzipCompression()))
case "snappy":
opts = append(opts, kgo.ProducerBatchCompression(kgo.SnappyCompression()))
case "lz4":
opts = append(opts, kgo.ProducerBatchCompression(kgo.Lz4Compression()))
case "zstd":
opts = append(opts, kgo.ProducerBatchCompression(kgo.ZstdCompression()))
default:
die("unrecognized compression %s", *compression)
}
case "gzip":
opts = append(opts, kgo.ProducerBatchCompression(kgo.GzipCompression()))
case "snappy":
opts = append(opts, kgo.ProducerBatchCompression(kgo.SnappyCompression()))
case "lz4":
opts = append(opts, kgo.ProducerBatchCompression(kgo.Lz4Compression()))
case "zstd":
opts = append(opts, kgo.ProducerBatchCompression(kgo.ZstdCompression()))
default:
die("unrecognized compression %s", *compression)
}

if *dialTLS {
Expand Down

0 comments on commit af4fce4

Please sign in to comment.