Reproducibility and test suite issues from upgrading to commons-compress 1.22+

[emmileaf]

See #3856 (comment), #3964 (currently blocked PR) for attempts to upgrade from 1.21 to 1.22 and 1.23.

Unit tests failing with:

com.google.cloud.tools.jib.tar.TarExtractorTest > testExtract_modificationTimePreserved FAILED
    value of: getLastModifiedTime(...)
    expected: 2019-08-01T16:13:09Z
    but was : 2019-08-01T16:13:09.72Z
        at com.google.cloud.tools.jib.tar.TarExtractorTest.testExtract_modificationTimePreserved(TarExtractorTest.java:96)

com.google.cloud.tools.jib.tar.TarExtractorTest > testExtract_reproducibleTimestampsEnabled FAILED
    value of: getLastModifiedTime(...)
    expected: 2021-01-29T21:10:02Z
    but was : 2021-01-29T21:10:02.78Z
        at com.google.cloud.tools.jib.tar.TarExtractorTest.testExtract_reproducibleTimestampsEnabled(TarExtractorTest.java:123)

com.google.cloud.tools.jib.image.ReproducibleLayerBuilderTest > testToBlob_reproducibility FAILED
    value of: writeToByteArray(...)
...

The first two errors can be addressed in the test suite by increasing precision of the expected values for FileTime in the tests. The third error testToBlob_reproducibility, is the main blocker for this upgrade.

As noted in #3962, commons-compress 1.22.0 adds atime and ctime support (in addition to mtime), which breaks image reproducibility in Jib (ref of upstream changes: TarArchiveEntry, PR#254)

I think that upgrading to 1.22+ would also need additional handling of atime and ctime in TarArchiveEntry and FileEntry (set defaults for reproducibility, and potentially also supporting custom configuration), in a similar way as mtime (modification time) in the current implementation.

[PapaNappa]

As discovered by #3962 (comment), this issue already happens for the current Jib version when you have other dependencies pulling in common-compress 1.22+.

For better searchability:
I stumbled across this issue when I was too wondering why for some of my builds, images were not reproducible.
As a last step, I resorted to comparing the layer tars themselves, after having found no differences in any of the files.
As it turns out, the non-reproducible layers have included pax headers.

The reproducible tars have UStar format. In my example, the tar has a layout like this:

app/<NULs><flags><NULs>ustar 00<more header stuff>
app/classpath/<more header stuff>ustar 00<more header stuff>
<files and contents>

The non-reproducible tars have included pax headers. In my example, the tar has a layout like this:

./PaxHeaders.X/app_<NULs><flags and pax flags>ustar 00<more header stuff>28 atime=1688711020.1460680
28 ctime=1686738930.9122330
46 LIBARCHIVE.creationtime=1686738930.9122330
<NULs>app/<more header stuff>
./PaxHeaders.X/app_classpath_<more header stuff>ustar 00<more header stuff>28 atime=1688711020.1460680
28 ctime=1686738930.9122330
46 LIBARCHIVE.creationtime=1686738930.9122330
<NULs>app/classpath/<more header stuff>ustar 00<more header stuff><files and contents>

[agascon]

Same issue here. When using jib gradle plugin with for example any using Apache Avro +1.11.0, common-compress 1.22+ is pulled and reproducible builds are broken.

It's a tricky issue as could remain unnoticed in many cases.

[mamuf]

I have a related issue, another plugin update in our build bumped commons-compress to 1.26 and jib then fails with

java.lang.ClassNotFoundException: org.apache.commons.codec.Charsets

I've downgraded the other plugin for now (it's nothing critical).

[chanseokoh]

I believe this is fixed by #4204, although the fix is not yet in the lastest Jib release.

[mpeddada1]

Update on this: Closing this issue as jib-gradle-plugin:3.4.2 and jib-maven-plugin:3.4.2 have been released with a fix! Please open an new issue if this is still occurring. Thank you.