xz decompressor

[daurnimator]

xz is a lossless compression program and file format which incorporates the LZMA/LZMA2 compression algorithms.

Useful for:

• reading compressed debug info (external and compressed debug info #2300)
• potentially the package manager (package manager #943)
• potentially the glibc ABI info (support glibc version as part of the target #2847 (comment))

Related to:

• DEFLATE implementation (deflate implementation in standard library #213)

[shawnl]

xz is a nice algorithm, but zstd got close while having many other use-cases for fast compression too. This is because zstd uses finite state entropy, which is quite new.

[daurnimator]

Although zstd is preferred these days; we still need XZ to support e.g. compressed debug info https://sourceware.org/gdb/onlinedocs/gdb/MiniDebugInfo.html, unless gdb and other common utilities are changing any time soon?

[shawnl]

@daurnimator note that LZMA is similar, but not the same as XZ (they are both supported by liblzma2). (LZMA was the beta version, that because become so widely used, the final version got a different name.)

I really don't see why we can't just build the C version, and use that.

[data-man]

lzma in ofelas/zigtest

[daurnimator]

@ofelas would you like to make that into a PR to the standard library?

[shawnl]

And his response should be to add a license to his repository. GitHub has a good page on choosing a license (I this in this case only BSD-2/3 or MIT would be acceptable) which I would link to if I was connected to internet. El jue., 26 sept. 2019 8:19, Vesa Kaihlavirta <[email protected]> escribió:

…

Since that code/rope doesn't have any license information, @ofelas <https:/ofelas> should probably be contacted whether it's ok to just add that to Zig. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#2851?email_source=notifications&email_token=AAD4W4SYU7NCJ2HCDOI7T73QLSSEXA5CNFSM4H65FTA2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD7VLSOI#issuecomment-535476537>, or mute the thread <https:/notifications/unsubscribe-auth/AAD4W4UVLWFFQQU5CZCK3SDQLSSEXANCNFSM4H65FTAQ> .

[ofelas]

I added a LICENSE file (MIT) as the original Rust code has in case anyone wants to use said code.
Unfortunately, I have little or no time to spend on this at the moment.

[tetsuo-cpp]

@daurnimator
Is this still something you want in the Zig standard library? I'm thinking of giving this a try.

I could either prepare and clean up the code linked above or write the implementation by hand using liblzma2 as a reference.

[alexpana]

@tetsuo-cpp I'm also willing to give this a try. Did you make any progress?

[tetsuo-cpp]

@alexpana
Sounds good! I haven't made any progress on this.
Feel free to ping me if you'd like another pair of eyes.

[b-ncMN]

Hi, I would like to give this a try, has some progress been made on that?

[matu3ba]

User squirl on discord:
"I'd like to mention that xz has some issues[1] - personally I'd recommend zstd instead, since it's more flexible, and tends to be faster for the same compression ratios [1]: https://lzip.nongnu.org/xz_inadequate.html

I'm not familiar with the details of zstd though so it may be too complex to include in std - in that case I'd recommend lzip, which is simpler and more efficient than xz, while also solving its reliability issues"

Main issue I see is that the reference implementation is GPLv2 and the public domain one "hopefully compliant" https://www.nongnu.org/lzip/pdlzip.html, which makes me wonder if the spec is ambiguous or if there are too few test cases etc.

[silversquirl]

• Lzip draft standard: https://datatracker.ietf.org/doc/html/draft-diaz-lzip-06
• Zstd spec: https:/facebook/zstd/blob/dev/doc/zstd_compression_format.md

[matu3ba]

I dont think its useful to add all the compatibility boilerplate on top of lzip, if distros are switching to zstd https://en.wikipedia.org/wiki/XZ_Utils.

[silversquirl]

Note that lzip uses LZMA, which @daurnimator mentions is needed anyway for parsing compressed debug info, so using lzip over zstd could provide a benefit in terms of code reuse.