System.IO.Compression work planned for .NET 7

[carlossanlop]

This issue captures the planned work for .NET 7. This list is expected to change throughout the release cycle according to ongoing planning and discussions, with possible additions and subtractions to the scope.

Planned for .NET 7

• Add the ArchiveComment property for ZipArchive class #1547
• Detect password protected file with ZipArchive #57197
• [API Proposal]: APIs to support tar archives #65951

Backlog (roughly in priority order)

• Explore if we should consume the zlib-ng native cross-platform implementation to benefit ARM64.
  • PR testing this: Add zlib-ng library, optimized for ARM64  #61883
• Improve the way our Zip APIs load entries in memory.
  • System.IO.Compression: ZipArchive loads entire file in memory on .Dispose #1543
  • System.IO.Compression: ZipArchiveEntry always stores uncompressed data in memory #1544
  • Add support for adding new entries to large zip archives #49149
  • ZipArchive.Open will read non seakable stream to memory & fail with bad error on large streams #59027
• Add Flush to ZipArchive #24149
• Explore if we should implement LZMA/7z or recommend an existing 3rd party package and contribute to it.
  • Add support for 7z/LZMA1/LZMA2 to System.IO.Compression #1542
• Allow opening (raw) compressed archive entries in ZipArchiveEntry
  • [API Proposal]: Allow opening (raw) compressed archive entries in ZipArchiveEntry #63155
• Add async compression APIs.
  • Add async ZipFile APIs #1541
• Add stream-less compression APIs
  • Add static compression helper methods #44793
• Ensure we provide full support for zip64 (reading, creating, updating).
  • Zip64ExtraField should read StartDiskNumber as uint32 #31825
• Add password to ZipArchive #1545
• Explore if we should implement ZStandard or recommend an existing 3rd party package and contribute to it.
  • Add support for Zstandard to System.IO.Compression #59591
• Clarify documentation ambiguity between some of our compression types: DeflateStream, GZipStream, ZLibStream.
• Explore if we should implement other algorithms or file formats, or recommend an existing 3rd party package and contribute to it.
• Continuous performance improvements.
• Improve documentation and add more examples.

[ghost]

Tagging subscribers to this area: @dotnet/area-system-io-compression
See info in area-owners.md if you want to be subscribed.

Issue Details

---

These are the higher priority Compression issues and requests affecting the community.

P0

• Improve the way our Zip APIs load entries in memory.
  • System.IO.Compression: ZipArchive loads entire file in memory on .Dispose #1543
  • System.IO.Compression: ZipArchiveEntry always stores uncompressed data in memory #1544
  • Add support for adding new entries to large zip archives #49149
  • ZipArchive.Open will read non seakable stream to memory & fail with bad error on large streams #59027
• Explore adding a Flush method to ZipArchive.
  • Add Flush to ZipArchive #24149
• Detect and apply ZipArchive encryption.
  • Add password to ZipArchive #1545
  • Detect password protected file with ZipArchive #57197

P1

• Implement the Tar archive format.
  • Support for tar archives #1540
• Explore if we should consume the zlib-ng native cross-platform implementation to benefit ARM64.
  • PR testing this: Add zlib-ng library, optimized for ARM64  #61883
• Explore if we should implement LZMA/7z or recommend an existing 3rd party package and contribute to it.
  • Add support for 7z/LZMA1/LZMA2 to System.IO.Compression #1542
• Add async compression APIs.
  • Add async ZipFile APIs #1541
• Add stream-less compression APIs
  • Add static compression helper methods #44793

P2

• Add comments to ZipArchive and ZipArchiveEntry.
  • Issue: Add the ArchiveComment property for ZipArchive class #1547
  • PR: Expose Comment in ZipArchive and ZipArchiveEntry #59442
• Ensure we provide full support for zip64 (reading, creating, updating).
  • Zip64ExtraField should read StartDiskNumber as uint32 #31825
• Explore if we should implement ZStandard or recommend an existing 3rd party package and contribute to it.
  • Add support for Zstandard to System.IO.Compression #59591
• Clarify documentation ambiguity between some of our compression types: DeflateStream, GZipStream, ZLibStream.

Longer term (.NET 8+)

• Explore if we should implement other algorithms or file formats, or recommend an existing 3rd party package and contribute to it.
• Continuous performance improvements.
• Improve documentation and add more examples.

Author:	carlossanlop
Assignees:	-
Labels:	area-System.IO.Compression, Bottom Up Work
Milestone:	Future

[danmoseley]

Thanks @carlossanlop . Are there opportunities for community contributions here? Perhaps there's something that's suitable for someone interested to get started on

[carlossanlop]

Yes, I think I can mark a couple of issues as up-for-grabs, but I first need to add a comment with some context first.

[AraHaan]

More things I would like done for .NET 7 (Zlib):

• [API Proposal]: Add ZlibEncoder and ZlibDecoder to System.IO.Compression. #62113 (Span based ZlibEncoder/ZlibDecoder impl)
  • prerec: Standardize pattern for exposing advanced configuration for compression streams #42820 (comment)

For this it might be until P3~P5 or 6 at the latest (to give time for review, approval, PR, tests, and merge).

[Genbox]

Finally, System.IO.Compression gets some much-needed attention!

Ask 1:
Can we revert this rather crazy input validation? It completely removes the ability to provide an alternative value, even though the related issue describes it as "a range check between 0 and 11" which out make much more sense.

Edit: It seems all other encoder/decoders have the same behavior. I respect that all are now doing the same thing, but I much rather have a range validation check than a strict map from enum to hardcoded level.

Every sine encoder/decoder that currently exists and all future ones should have sane input validation on the compression level to ensure there is no unexpected behavior.

[Genbox]

Ask 2:
Can we change CompressionLevel to something a lot more user-friendly?

Current values are:

• Optimal
• Fastest
• NoCompression
• SmallestSize

I propose we change it to:

• Store
• Fast
• Normal (or Default)
• High
• Maximum

If that is not possible for whatever reason, can we at least change "Optimal" to "Default"?

The main reason is that .NET devs and users alike seem to not understand what "optimal" means. I've had this discussion in 2018 and about 15 times since.

[Genbox]

Ask 3:

Please expose more advanced configurations. Encoders/Decoders have a million options like providing a set of pre-defined wordlists, Huffman encoding specific options, window sizes, and much more. There is no way to squeeze all that into constructors as single values. Properties are not a solution as some of these values needs to be available before the encoder/decoder is used, so they need to come in the constructor.

There is an excellent proposal here, which would solve that problem altogether: #42820

[Genbox]

Ask 4:

Zlib is great, but the implementation is dated.

There are several highly optimized versions out there:

• Intel (whitepaper and source)
• Cloudflare (source and AWS's patches)
• Chromium (source)
• Zlib-ng (source and bechmark)

Zlib-ng seems like a good next step:

• It has the optimizations from intel and Cloudflare
• It has a Zlib compatible API
• It has the same Zlib license
• It is 2x faster than Zlib out of the box

[AraHaan]

I been working on a span based zlib that calls into the same native version under the hood that provides more config. However it will need to be marked as ready for review so it can have a PR be created.

As for the PR I would have to place 2 related proposals into the same PR as well they depend on eachother on changes (new types would be added to expose all compression levels for all of the currently implemented compressors in the runtime for better configuration).

I would have made the options classes structs in that proposal but sadly C# does not allow structs to inherit from other structs.

As for other zlib native libraries, while they may be beneficial for performance, what prevents them from accidentally breaking something in the library they are working on. I noticed something similar when I was trying to implement the same faster stuff in my fully C# zlib library but it only broke on MacOS where all the compressed data was simply:

• zlib header
• original data
• zlib footer

but on other OSs:

• zlib header
• compressed data
• zlib footer

As such, something as trivial as an native dependency change could break someone even if it provides a performance boost unless tested extensively in anyone's possible situation that can happen.

[jeffhandley]

This list in this issue has been edited to reflect changes from our ongoing planning and discussions. The headings have been updated to reflect what is still planned for .NET 7 and what is in the backlog. The list is still expected to change throughout the release cycle with periodic reviews of our plans.

[AraHaan]

@jeffhandley any updates for what I proposed here for zlib (all linked here): #62658 (comment) ?

I really need them in .NET 7 at least if possible because span based compression apis should be a priority as well (I think) for those of us who hate stream based due to perf from loading a files as bytes, then create a stream, use that stream with compression apis to compress/decompress, do other stuff that expects spans, and then save the compressed/decompressed output. The stream part becomes a pain as with span based compression apis they can be avoided entirely.

[jeffhandley]

@jeffhandley any updates for what I proposed here for zlib (all linked here): #62658 (comment) ?

@AraHaan I'm sorry I overlooked that question months back and didn't notice until now. The proposals you made for zlib are valuable, but did not make it into .NET 7.0. We will consider them while we do our .NET 8 planning. Thank you.