Add zlib-ng library, optimized for ARM64 #61883
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -10,7 +10,34 @@ if (GEN_SHARED_LIB) | |
| include (GenerateExportHeader) | ||
| endif() | ||
|
|
||
| if(${CLR_CMAKE_HOST_ARCH} STREQUAL arm OR ${CLR_CMAKE_HOST_ARCH} STREQUAL arm64) | ||
|
There was a problem hiding this comment. This commit should probably be broken into two. One that only adds the zlib-ng files. And another that connects dotnet runtime build system to use them. It would make it easier to review because then we can see what build system changes are needed for zlib-ng. But I realize at this point it would be difficult to split it. There was a problem hiding this comment. Good point. I'll do that. |
||
| set(NATIVECOMPRESSION_SOURCES | ||
| zlib-ng/adler32.c | ||
| zlib-ng/chunkset.c | ||
| zlib-ng/compare258.c | ||
| zlib-ng/compress.c | ||
| zlib-ng/crc32_comb.c | ||
| zlib-ng/crc32.c | ||
| zlib-ng/deflate_fast.c | ||
| zlib-ng/deflate_medium.c | ||
| zlib-ng/deflate_quick.c | ||
| zlib-ng/deflate_slow.c | ||
| zlib-ng/deflate.c | ||
| zlib-ng/functable.c | ||
| # Exclude files to fix error LNK2019: unresolved external symbol 'gzFile' open referenced in functions gz_open, gz_close, gz_load, gz_comp | ||
|
There was a problem hiding this comment. Should find out why it is giving that linker error. If dotnet does not use any of the gz_* functions then you don’t need those sources and can also set -DWITH_GZFILEOP=0. There was a problem hiding this comment. Why not just use add_subdirectory to add our CMake project? There was a problem hiding this comment. I'm looking into this again now that we're back from the Thanksgiving holidays. I'm exploring the use of There was a problem hiding this comment.
Maybe you can put zlib-ng under src/native/external as part of this rebase as discussed in |
||
| # zlib-ng/gzlib.c | ||
| # zlib-ng/gzread.c | ||
| # zlib-ng/gzwrite.c | ||
| zlib-ng/infback.c | ||
| zlib-ng/inffast.c | ||
| zlib-ng/inflate.c | ||
| zlib-ng/inftrees.c | ||
| zlib-ng/insert_string.c | ||
| zlib-ng/trees.c | ||
| zlib-ng/uncompr.c | ||
| zlib-ng/zutil.c | ||
| ) | ||
| elseif(${CLR_CMAKE_HOST_ARCH} STREQUAL x86 OR ${CLR_CMAKE_HOST_ARCH} STREQUAL x64) | ||
| set(NATIVECOMPRESSION_SOURCES | ||
| zlib-intel/adler32.c | ||
| zlib-intel/compress.c | ||
|
|
@@ -27,7 +54,7 @@ if(${CLR_CMAKE_HOST_ARCH} STREQUAL x86 OR ${CLR_CMAKE_HOST_ARCH} STREQUAL x64) | |
| zlib-intel/trees.c | ||
| zlib-intel/x86.c | ||
| zlib-intel/zutil.c | ||
| ) | ||
| else() | ||
| set(NATIVECOMPRESSION_SOURCES | ||
| zlib/adler32.c | ||
|
|
@@ -42,7 +69,7 @@ else() | |
| ) | ||
| endif() | ||
|
|
||
| set(NATIVECOMPRESSION_SOURCES | ||
| ${NATIVECOMPRESSION_SOURCES} | ||
| ../../AnyOS/zlib/pal_zlib.c | ||
| ../../AnyOS/brotli/common/constants.c | ||
|
|
||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,19 @@ | ||
| (C) 1995-2013 Jean-loup Gailly and Mark Adler | ||
|
There was a problem hiding this comment. Presumably we also need to update https:/dotnet/runtime/blob/main/THIRD-PARTY-NOTICES.TXT ? |
||
|
|
||
| This software is provided 'as-is', without any express or implied | ||
| warranty. In no event will the authors be held liable for any damages | ||
| arising from the use of this software. | ||
|
|
||
| Permission is granted to anyone to use this software for any purpose, | ||
| including commercial applications, and to alter it and redistribute it | ||
| freely, subject to the following restrictions: | ||
|
|
||
| 1. The origin of this software must not be misrepresented; you must not | ||
| claim that you wrote the original software. If you use this software | ||
| in a product, an acknowledgment in the product documentation would be | ||
| appreciated but is not required. | ||
|
|
||
| 2. Altered source versions must be plainly marked as such, and must not be | ||
| misrepresented as being the original software. | ||
|
|
||
| 3. This notice may not be removed or altered from any source distribution. | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,220 @@ | ||
| ## zlib-ng | ||
| *zlib data compression library for the next generation systems* | ||
|
|
||
| Maintained by Hans Kristian Rosbach | ||
| aka Dead2 (zlib-ng àt circlestorm dót org) | ||
|
|
||
| |CI|Status| | ||
| |:-|-| | ||
| |GitHub Actions|[](https:/zlib-ng/zlib-ng/actions) [](https:/zlib-ng/zlib-ng/actions) [](https:/zlib-ng/zlib-ng/actions)| | ||
| |Buildkite|[](https://buildkite.com/circlestorm-productions/zlib-ng)| | ||
| |CodeFactor|[](https://www.codefactor.io/repository/github/zlib-ng/zlib-ng)| | ||
| |OSS-Fuzz|[](https://bugs.chromium.org/p/oss-fuzz/issues/list?sort=-opened&can=1&q=proj:zlib-ng) | ||
| |Codecov|[](https://codecov.io/github/zlib-ng/zlib-ng/)| | ||
|
|
||
|
|
||
| Features | ||
| -------- | ||
|
|
||
| * Zlib compatible API with support for dual-linking | ||
| * Modernized native API based on zlib API for ease of porting | ||
| * Modern C99 syntax and a clean code layout | ||
| * Deflate medium and quick algorithms based on Intels zlib fork | ||
| * Support for CPU intrinsics when available | ||
| * Adler32 implementation using SSSE3, AVX2, Neon & VSX | ||
| * CRC32-B implementation using PCLMULQDQ & ACLE | ||
| * Hash table implementation using CRC32-C intrinsics on x86 and ARM | ||
| * Slide hash implementations using SSE2, AVX2, Neon & VSX | ||
| * Compare256/258 implementations using SSE4.2 & AVX2 | ||
| * Inflate chunk copying using SSE2, AVX2 & Neon | ||
| * Support for hardware-accelerated deflate using IBM Z DFLTCC | ||
| * Unaligned memory read/writes and large bit buffer improvements | ||
| * Includes improvements from Cloudflare and Intel forks | ||
| * Configure, CMake, and NMake build system support | ||
| * Comprehensive set of CMake unit tests | ||
| * Code sanitizers, fuzzing, and coverage | ||
| * GitHub Actions continuous integration on Windows, macOS, and Linux | ||
| * Emulated CI for ARM, AARCH64, PPC, PPC64, SPARC64, S390x using qemu | ||
|
|
||
|
|
||
| History | ||
| ------- | ||
|
|
||
| The motivation for this fork came after seeing several 3rd party | ||
| contributions containing new optimizations not getting implemented | ||
| into the official zlib repository. | ||
|
|
||
| Mark Adler has been maintaining zlib for a very long time, and he has | ||
| done a great job and hopefully he will continue for a long time yet. | ||
| The idea of zlib-ng is not to replace zlib, but to co-exist as a | ||
| drop-in replacement with a lower threshold for code change. | ||
|
|
||
| zlib has a long history and is incredibly portable, even supporting | ||
| lots of systems that predate the Internet. This is great, but it does | ||
| complicate further development and maintainability. | ||
| The zlib code has numerous workarounds for old compilers that do not | ||
| understand ANSI-C or to accommodate systems with limitations such as | ||
| operating in a 16-bit environment. | ||
|
|
||
| Many of these workarounds are only maintenance burdens, some of them | ||
| are pretty huge code-wise. For example, the [v]s[n]printf workaround | ||
| code has a whopping 8 different implementations just to cater to | ||
| various old compilers. With this many workarounds cluttered throughout | ||
| the code, new programmers with an idea/interest for zlib will need | ||
| to take some time to figure out why all of these seemingly strange | ||
| things are used, and how to work within those confines. | ||
|
|
||
| So I decided to make a fork, merge all the Intel optimizations, merge | ||
| the Cloudflare optimizations that did not conflict, plus a couple | ||
| of other smaller patches. Then I started cleaning out workarounds, | ||
| various dead code, all contrib and example code as there is little | ||
| point in having those in this fork for various reasons. | ||
|
|
||
| A lot of improvements have gone into zlib-ng since its start, and | ||
| numerous people and companies have contributed both small and big | ||
| improvements, or valuable testing. | ||
|
|
||
| Please read LICENSE.md, it is very simple and very liberal. | ||
|
|
||
|
|
||
| Build | ||
| ----- | ||
|
|
||
| There are two ways to build zlib-ng: | ||
|
|
||
| ### Cmake | ||
|
|
||
| To build zlib-ng using the cross-platform makefile generator cmake. | ||
|
|
||
| ``` | ||
| cmake . | ||
| cmake --build . --config Release | ||
| ctest --verbose -C Release | ||
| ``` | ||
|
|
||
| Alternatively, you can use the cmake configuration GUI tool ccmake: | ||
|
|
||
| ``` | ||
| ccmake . | ||
| ``` | ||
|
|
||
| ### Configure | ||
|
|
||
| To build zlib-ng using the bash configure script: | ||
|
|
||
| ``` | ||
| ./configure | ||
| make | ||
| make test | ||
| ``` | ||
|
|
||
| Build Options | ||
| ------------- | ||
|
|
||
| | CMake | configure | Description | Default | | ||
| |:-------------------------|:-------------------------|:--------------------------------------------------------------------------------------|---------| | ||
| | ZLIB_COMPAT | --zlib-compat | Compile with zlib compatible API | OFF | | ||
| | ZLIB_ENABLE_TESTS | | Build test binaries | ON | | ||
| | WITH_GZFILEOP | --without-gzfileops | Compile with support for gzFile related functions | ON | | ||
| | WITH_OPTIM | --without-optimizations | Build with optimisations | ON | | ||
| | WITH_NEW_STRATEGIES | --without-new-strategies | Use new strategies | ON | | ||
| | WITH_NATIVE_INSTRUCTIONS | --native | Compiles with full instruction set supported on this host (gcc/clang -march=native) | OFF | | ||
| | WITH_SANITIZER | --with-sanitizer | Build with sanitizer (memory, address, undefined) | OFF | | ||
| | WITH_FUZZERS | --with-fuzzers | Build test/fuzz | OFF | | ||
| | WITH_MAINTAINER_WARNINGS | | Build with project maintainer warnings | OFF | | ||
| | WITH_CODE_COVERAGE | | Enable code coverage reporting | OFF | | ||
|
|
||
|
|
||
| Install | ||
| ------- | ||
|
|
||
| WARNING: We do not recommend manually installing unless you really | ||
| know what you are doing, because this can potentially override the system | ||
| default zlib library, and any incompatibility or wrong configuration of | ||
| zlib-ng can make the whole system unusable, requiring recovery or reinstall. | ||
| If you still want a manual install, we recommend using the /opt/ path prefix. | ||
|
|
||
| For Linux distros, an alternative way to use zlib-ng (if compiled in | ||
| zlib-compat mode) instead of zlib, is through the use of the | ||
| _LD_PRELOAD_ environment variable. If the program is dynamically linked | ||
| with zlib, then zlib-ng will temporarily be used instead by the program, | ||
| without risking system-wide instability. | ||
|
|
||
| ``` | ||
| LD_PRELOAD=/opt/zlib-ng/libz.so.1.2.11.zlib-ng /usr/bin/program | ||
| ``` | ||
|
|
||
| ### Cmake | ||
|
|
||
| To install zlib-ng system-wide using cmake: | ||
|
|
||
| ``` | ||
| cmake --build . --target install | ||
| ``` | ||
|
|
||
| ### Configure | ||
|
|
||
| To install zlib-ng system-wide using the configure script: | ||
|
|
||
| ``` | ||
| make install | ||
| ``` | ||
|
|
||
| Contributing | ||
| ------------ | ||
|
|
||
| Zlib-ng is a aiming to be open to contributions, and we would be delighted to | ||
| receive pull requests on github. | ||
| Just remember that any code you submit must be your own and it must be zlib licensed. | ||
| Help with testing and reviewing of pull requests etc is also very much appreciated. | ||
|
|
||
| If you are interested in contributing, please consider joining our | ||
| IRC channel #zlib-ng on the Freenode IRC network. | ||
|
|
||
|
|
||
| Acknowledgments | ||
| ---------------- | ||
|
|
||
| Thanks to Servebolt.com for sponsoring my maintainership of zlib-ng. | ||
|
|
||
| Thanks go out to all the people and companies who have taken the time to contribute | ||
| code reviews, testing and/or patches. Zlib-ng would not have been nearly as good without you. | ||
|
|
||
| The deflate format used by zlib was defined by Phil Katz. | ||
| The deflate and zlib specifications were written by L. Peter Deutsch. | ||
|
|
||
| zlib was originally created by Jean-loup Gailly (compression) | ||
| and Mark Adler (decompression). | ||
|
|
||
|
|
||
| Advanced Build Options | ||
| ---------------------- | ||
|
|
||
| | CMake | configure | Description | Default | | ||
| |:--------------------------------|:----------------------|:--------------------------------------------------------------------|------------------------| | ||
| | ZLIB_DUAL_LINK | | Dual link tests with system zlib | OFF | | ||
| | UNALIGNED_OK | | Allow unaligned reads | ON (x86, arm) | | ||
| | | --force-sse2 | Skip runtime check for SSE2 instructions (Always on for x86_64) | OFF (x86) | | ||
| | WITH_AVX2 | | Build with AVX2 intrinsics | ON | | ||
| | WITH_SSE2 | | Build with SSE2 intrinsics | ON | | ||
| | WITH_SSE4 | | Build with SSE4 intrinsics | ON | | ||
| | WITH_PCLMULQDQ | | Build with PCLMULQDQ intrinsics | ON | | ||
| | WITH_ACLE | --without-acle | Build with ACLE intrinsics | ON | | ||
| | WITH_NEON | --without-neon | Build with NEON intrinsics | ON | | ||
| | WITH_POWER8 | | Build with POWER8 optimisations | ON | | ||
| | WITH_DFLTCC_DEFLATE | --with-dfltcc-deflate | Build with DFLTCC intrinsics for compression on IBM Z | OFF | | ||
| | WITH_DFLTCC_INFLATE | --with-dfltcc-inflate | Build with DFLTCC intrinsics for decompression on IBM Z | OFF | | ||
| | WITH_UNALIGNED | | Allow optimizations that use unaligned reads if safe on current arch| ON | | ||
| | WITH_INFLATE_STRICT | | Build with strict inflate distance checking | OFF | | ||
| | WITH_INFLATE_ALLOW_INVALID_DIST | | Build with zero fill for inflate invalid distances | OFF | | ||
| | INSTALL_UTILS | | Copy minigzip and minideflate during install | OFF | | ||
|
|
||
|
|
||
| Related Projects | ||
| ---------------- | ||
|
|
||
| * Fork of the popular minigzip https:/zlib-ng/minizip-ng | ||
| * Python tool to benchmark minigzip/minideflate https:/zlib-ng/deflatebench | ||
| * Python tool to benchmark pigz https:/zlib-ng/pigzbench | ||
| * 3rd party patches for zlib-ng compatibility https:/zlib-ng/patches | ||
|
|
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We don't want a zlib compatible API? What does this actually affect?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
https:/zlib-ng/zlib-ng/blob/develop/PORTING.md