Enable default gzip compression for otlp exporters #4587
Comments
100% agree with you, but I would like to understand the CPU cost as well. |
|
There are several dimensions that impact the performance benefit / cost of compression. Geo distribution of client and server: clients with long round trips to servers pay a steep price for unnecessary bytes they send. Clients can be put under memory pressure from data queuing up while awaiting the trips, which can then become CPU pressure. So somewhat paradoxically, spending extra CPU cycles on compression can result in lower overall CPU pressure. Additionally, the randomness of the data can change the equation. Highly random data won't compress as well, but will still incur the CPU cost. Still, despite the fact that compression isn't always the right decision, it is the right decision in the vast majority of cases. And the default behavior really should aim to provide good results most of the time. It seems quite challenging to incorporate network latency and varying payload shapes into a performance test suite. Perhaps a simplified analysis would suffice? What if we just tried to characterize the CPU cost of compression, but not the overall impact of smaller payloads? I.e. try to answer the question: "how long does it take to gzip compress 1kb"? At least that gives you an idea of the price you pay in the worst case scenario in which smaller payloads provide little benefit. |
|
I've done some benchmarks of various grpc compression algorithms on a branch here. Notes:
Raw Results: Results with some additional analysis:
The last column Snappy doesn't compress as well or zstd or gzip, but is much faster is the best bang for your buck if the process is CPU bound. If the process is network bound, gzip and zstd are likely better options as they offer better compression. However, Edited 1/4/2022: Changed compression ratio from |
|
That's great info, I think this should be part of the documentation for this feature. The only thing I would request is to get the formula for the "compress ratio" changed so that it shows |
|
I defined the compression ratio as |
|
"4.46 to 1" is great, way better than a percentage. |
Change OTLP gRPC exporter to use `gzip` compression by default. If this is merged, I can followup with the OTLP HTTP exporter to use `gzip` by default as well. **Link to tracking Issue:** #4587 **Testing:** - Added benchmarks comparing different compression algorithms. - Updated `configgrpc_test.go` to validate default behavior **Documentation:** Added details to `/config/configgrpc/README.md` with benchmark results and notes on compression.
|
Resolved with #4632. |
Describe the solution you'd like
Recently there was a spec PR about enabling gzip compression by default for OTLP in the SDK. The TC concluded that the default OTLP settings are optimized for exporting to a local collector, where compression is not useful.
The collector does not have a default OTLP export location, and it is much more common to export to a remote network address than to another collector running locally.
I propose that we enable gzip compression by default for OTLP exporters. Compression is great for clients and servers alike. Its better if users have to opt out of compression than opting in, as it will improve the likelihood of casual users having it enabled.
Additional context
Reminder that according to the spec, all server components MUST support gzip compression. So changing this behavior will not impact spec compliant OTLP receivers.
Compression ratio varies based on shape of the payload, but in some simple local tests of OTLP payloads I've seen anywhere from 15% to 60+% reduction in payload size. This can have a very significant positive impact on performance!
The text was updated successfully, but these errors were encountered: