s2: Add concurrent stream decompression #397
Labels
Comments
klauspost
added a commit
that referenced
this issue
May 25, 2022
Add concurrent stream decompression of snappy/s2 streams Fixes #397 ``` e:\gopath\src\github.com\klauspost\compress\s2\cmd\s2d (master -> klauspost) λ s2d -bench=3 -cpu=1 enwik10.snappy Reading enwik10.snappy... Decompressing... 4340323220 -> 10000000000 [230.40%]; 8.681s, 1098.6MB/s Decompressing... 4340323220 -> 10000000000 [230.40%]; 8.744s, 1090.7MB/s Decompressing... 4340323220 -> 10000000000 [230.40%]; 8.711s, 1094.8MB/s e:\gopath\src\github.com\klauspost\compress\s2\cmd\s2d (master -> klauspost) λ s2d -bench=3 -cpu=2 enwik10.snappy Reading enwik10.snappy... Decompressing... 4340323220 -> 10000000000 [230.40%]; 5.259s, 1813.4MB/s Decompressing... 4340323220 -> 10000000000 [230.40%]; 5.241s, 1819.8MB/s Decompressing... 4340323220 -> 10000000000 [230.40%]; 5.255s, 1814.7MB/s e:\gopath\src\github.com\klauspost\compress\s2\cmd\s2d (master -> klauspost) λ s2d -bench=3 -cpu=4 enwik10.snappy Reading enwik10.snappy... Decompressing... 4340323220 -> 10000000000 [230.40%]; 2.63s, 3625.6MB/s Decompressing... 4340323220 -> 10000000000 [230.40%]; 2.646s, 3604.8MB/s Decompressing... 4340323220 -> 10000000000 [230.40%]; 2.644s, 3606.5MB/s e:\gopath\src\github.com\klauspost\compress\s2\cmd\s2d (master -> klauspost) λ s2d -bench=3 -cpu=8 enwik10.snappy Reading enwik10.snappy... Decompressing... 4340323220 -> 10000000000 [230.40%]; 1.38s, 6910.6MB/s Decompressing... 4340323220 -> 10000000000 [230.40%]; 1.387s, 6873.4MB/s Decompressing... 4340323220 -> 10000000000 [230.40%]; 1.387s, 6875.8MB/s e:\gopath\src\github.com\klauspost\compress\s2\cmd\s2d (master -> klauspost) λ s2d -bench=3 -cpu=16 enwik10.snappy Reading enwik10.snappy... Decompressing... 4340323220 -> 10000000000 [230.40%]; 894ms, 10668.5MB/s Decompressing... 4340323220 -> 10000000000 [230.40%]; 882ms, 10818.2MB/s Decompressing... 4340323220 -> 10000000000 [230.40%]; 903ms, 10558.8MB/s ```
klauspost
added a commit
that referenced
this issue
May 26, 2022
Add concurrent stream decompression of snappy/s2 streams Fixes #397 For full stream decompression S2 offers a `DecodeConcurrent` function that will decode a full stream using multiple goroutines. Example scaling, AMD Ryzen 3950X, 16 cores, decompression using `s2d -bench=3 <input>`, best of 3: | Input | `-cpu=1` | `-cpu=2` | `-cpu=4` | `-cpu=8` | `-cpu=16` | |-------------------------------------------|------------|------------|------------|------------|-------------| | enwik10.snappy | 1098.6MB/s | 1819.8MB/s | 3625.6MB/s | 6910.6MB/s | 10818.2MB/s | | enwik10.s2 | 1303.5MB/s | 2606.1MB/s | 4847.9MB/s | 8878.4MB/s | 9592.1MB/s | | sofia-air-quality-dataset.tar.snappy | 1302.0MB/s | 2165.0MB/s | 4244.5MB/s | 8241.0MB/s | 12920.5MB/s | | sofia-air-quality-dataset.tar.s2 | 1399.2MB/s | 2463.2MB/s | 5196.5MB/s | 9639.8MB/s | 11439.5MB/s | | sofia-air-quality-dataset.tar.s2 (no asm) | 837.5MB/s | 1652.6MB/s | 3183.6MB/s | 5945.0MB/s | 9620.7MB/s | Scaling can be expected to be pretty linear until memory bandwidth is saturated. For now the `DecodeConcurrent` can only be used for full streams without seeking or combining with regular reads.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
No description provided.
The text was updated successfully, but these errors were encountered: