Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update apache-commons to 1.26.0 and set PAX headers to address build reproducibility issue #4204

Merged
merged 16 commits into from
Mar 11, 2024
Merged

Update apache-commons to 1.26.0 and set PAX headers to address build reproducibility issue #4204

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
2f19840
Set access, creation, and last-modified time to zero
bjornbugge Nov 17, 2023
8d5bedd
Add small test that ensures that modtime doesn't change layer
bjornbugge Nov 17, 2023
3180eda
Also clear time-related PAX headers on parent-dir entries
bjornbugge Nov 17, 2023
db224ae
Add blank line for checkstyle
bjornbugge Jan 24, 2024
a352e30
Fix the modification-time tests
bjornbugge Jan 25, 2024
0894421
Rename `clearTimeHeaders` to `clearPaxTimeHeaders`
bjornbugge Jan 25, 2024
85c9dd1
Set PAX headers to EPOCH plus 1
bjornbugge Jan 29, 2024
028db50
Rename to`clearTimeHeaders` and call `setModTime` in helper
bjornbugge Jan 29, 2024
909bfcd
File modification time set by JIB (and possibly overridden by user) i…
izogfif Mar 2, 2024
b885461
Bump org.apache.commons:commons-compress from 1.21 to 1.26.0. Wrong c…
izogfif Mar 3, 2024
6b22b68
TarExtractorTest fixes: modification time from tar archive is now ret…
izogfif Mar 3, 2024
2663c80
CHANGELOG files were updated.
izogfif Mar 4, 2024
b979655
Tests should now accommodate file systems incapable of storing file m…
izogfif Mar 7, 2024
8256b8a
Tar hashes and file names were updated.
izogfif Mar 7, 2024
5af8470
Typo fix.
izogfif Mar 7, 2024
042bcb1
ReproducibleImageTest error fix.
izogfif Mar 7, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions build.gradle
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ project.ext.dependencyStrings = [
GRADLE_EXTENSION: 'com.google.cloud.tools:jib-gradle-plugin-extension-api:0.4.0',
MAVEN_EXTENSION: 'com.google.cloud.tools:jib-maven-plugin-extension-api:0.4.0',

COMMONS_COMPRESS: 'org.apache.commons:commons-compress:1.21',
COMMONS_COMPRESS: 'org.apache.commons:commons-compress:1.26.0',
ZSTD_JNI: 'com.github.luben:zstd-jni:1.5.5-5',
COMMONS_TEXT: 'org.apache.commons:commons-text:1.10.0',
JACKSON_BOM: 'com.fasterxml.jackson:jackson-bom:2.15.2',
Expand Down Expand Up @@ -371,7 +371,7 @@ subprojects {
// sourceProject(Project) accepts a project and adds it as a dependency in a special manner:
// 1. force evaluation of the project first
// 2. add the project classes as "compileOnly" and make it available to tests in "testImplementation"
// 3. add the project's depedencies as "implementation"
// 3. add the project's dependencies as "implementation"
// 4. remove any transitive reference of any sourceProject depenency that may have appeared
// 5. add the project's classes to the final jar
// Other nice effects (vs shadowJar)
Expand Down
3 changes: 2 additions & 1 deletion jib-core/CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,9 +7,10 @@ All notable changes to this project will be documented in this file.
-

### Changed
-
- deps: bump org.apache.commons:commons-compress from 1.21 to 1.26.0 ([#4204](https:/GoogleContainerTools/jib/pull/4204))

### Fixed
- fix: image builds should become reproducible once again ([#4204](https:/GoogleContainerTools/jib/pull/4204))

## 0.26.0

Expand Down
10 changes: 5 additions & 5 deletions jib-core/src/integration-test/java/com/google/cloud/tools/jib/api/ReproducibleImageTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -102,9 +102,9 @@ public void testTarballStructure() throws IOException {

assertThat(actual)
.containsExactly(
"c46572ef74f58d95e44dd36c1fbdfebd3752e8b56a794a13c11cfed35a1a6e1c.tar.gz",
"6d2763b0f3940d324ea6b55386429e5b173899608abf7d1bff62e25dd2e4dcea.tar.gz",
"530c1954a2b087d0b989895ea56435c9dc739a973f2d2b6cb9bb98e55bbea7ac.tar.gz",
"98682a867906d9d07cf3c51a4fb9e08e9d5baddd1ca5dc7834f58f434c9cb15c.tar.gz",
"527db49d4e0c4159346119b4971d59016bfedceed874abab2b510ce433f6b15c.tar.gz",
"16d03883198935b4119896dcea0ea14e1bf105b6ac0a35a88820d08bc0263306.tar.gz",
"config.json",
"manifest.json")
.inOrder();
Expand All @@ -114,7 +114,7 @@ public void testTarballStructure() throws IOException {
public void testManifest() throws IOException {
String expectedManifest =
"[{\"Config\":\"config.json\",\"RepoTags\":[\"jib-core/reproducible:latest\"],"
+ "\"Layers\":[\"c46572ef74f58d95e44dd36c1fbdfebd3752e8b56a794a13c11cfed35a1a6e1c.tar.gz\",\"6d2763b0f3940d324ea6b55386429e5b173899608abf7d1bff62e25dd2e4dcea.tar.gz\",\"530c1954a2b087d0b989895ea56435c9dc739a973f2d2b6cb9bb98e55bbea7ac.tar.gz\"]}]";
+ "\"Layers\":[\"98682a867906d9d07cf3c51a4fb9e08e9d5baddd1ca5dc7834f58f434c9cb15c.tar.gz\",\"527db49d4e0c4159346119b4971d59016bfedceed874abab2b510ce433f6b15c.tar.gz\",\"16d03883198935b4119896dcea0ea14e1bf105b6ac0a35a88820d08bc0263306.tar.gz\"]}]";
String generatedManifest = extractFromTarFileAsString(imageTar, "manifest.json");
assertThat(generatedManifest).isEqualTo(expectedManifest);
}
Expand All @@ -125,7 +125,7 @@ public void testConfiguration() throws IOException {
"{\"created\":\"1970-01-01T00:00:00Z\",\"architecture\":\"amd64\",\"os\":\"linux\","
+ "\"config\":{\"Env\":[],\"Entrypoint\":[\"echo\",\"Hello World\"],\"ExposedPorts\":{},\"Labels\":{},\"Volumes\":{}},"
+ "\"history\":[{\"created\":\"1970-01-01T00:00:00Z\",\"author\":\"Jib\",\"created_by\":\"jib-core:null\",\"comment\":\"\"},{\"created\":\"1970-01-01T00:00:00Z\",\"author\":\"Jib\",\"created_by\":\"jib-core:null\",\"comment\":\"\"},{\"created\":\"1970-01-01T00:00:00Z\",\"author\":\"Jib\",\"created_by\":\"jib-core:null\",\"comment\":\"\"}],"
+ "\"rootfs\":{\"type\":\"layers\",\"diff_ids\":[\"sha256:18e4f44e6d1835bd968339b166057bd17ab7d4cbb56dc7262a5cafea7cf8d405\",\"sha256:13369c34f073f2b9c1fa6431e23d925f1a8eac65b1726c8cc8fcc2596c69b414\",\"sha256:4f92c507112d7880ca0f504ef8272b7fdee107263270125036a260a741565923\"]}}";
+ "\"rootfs\":{\"type\":\"layers\",\"diff_ids\":[\"sha256:2fcc2157bf42c89195676ef6e973a96d7b018c9d30ba89db95e9e0722e1c8ef3\",\"sha256:21f521f3217067d277af37512a08c72281d90fdd02d7174db632c8c3a34403bd\",\"sha256:6beba018395265af5061864b7f4678e831eb2daebb1045487c641fc8b142e319\"]}}";
String generatedConfig = extractFromTarFileAsString(imageTar, "config.json");
assertThat(generatedConfig).isEqualTo(expectedConfig);
}
Expand Down
19 changes: 17 additions & 2 deletions jib-core/src/main/java/com/google/cloud/tools/jib/image/ReproducibleLayerBuilder.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@
import java.io.IOException;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.time.Instant;
import java.util.ArrayList;
import java.util.Comparator;
import java.util.HashSet;
Expand Down Expand Up @@ -76,11 +77,11 @@ private void add(TarArchiveEntry tarArchiveEntry) throws IOException {
if (namePath.getParent() != namePath.getRoot()) {
Path tarArchiveParentDir = Verify.verifyNotNull(namePath.getParent());
TarArchiveEntry dir = new TarArchiveEntry(DIRECTORY_FILE, tarArchiveParentDir.toString());
dir.setModTime(FileEntriesLayer.DEFAULT_MODIFICATION_TIME.toEpochMilli());
dir.setUserId(0);
dir.setGroupId(0);
dir.setUserName("");
dir.setGroupName("");
clearTimeHeaders(dir, FileEntriesLayer.DEFAULT_MODIFICATION_TIME);
add(dir);
}

Expand All @@ -95,6 +96,20 @@ private List<TarArchiveEntry> getSortedEntries() {
}
}

private static void clearTimeHeaders(TarArchiveEntry entry, Instant modTime) {
entry.setModTime(modTime.toEpochMilli());

String headerTime = Long.toString(modTime.getEpochSecond());
final long nanos = modTime.getNano();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not familiar with the PAX headers, so just asking: does it have to be the nanosecond precision? Like, what if someone decides to go with milli or microseconds? Is it possible? Not that I care as long as nano works.

Copy link
Contributor Author

@izogfif izogfif Mar 4, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here is a post I was able to find:

This is important for example with the tarfile module with the pax tar format. The POSIX tar standard mandates storing the mtime in the extended header (if it is not an integer) with as much precision as is available in the underlying file system, and likewise to restore this time properly upon extraction.

So I'd say that it's better to store time with nanosecond precision. File systems that don't support this precision won't fail upon extraction: they'll simply truncate values.

if (nanos > 0) {
headerTime += "." + nanos;
}
entry.addPaxHeader("mtime", headerTime);
entry.addPaxHeader("atime", headerTime);
entry.addPaxHeader("ctime", headerTime);
entry.addPaxHeader("LIBARCHIVE.creationtime", headerTime);
}

private static void setUserAndGroup(TarArchiveEntry entry, FileEntry layerEntry) {
entry.setUserId(0);
entry.setGroupId(0);
Expand Down Expand Up @@ -156,8 +171,8 @@ public Blob build() throws IOException {
// Sets the entry's permissions by masking out the permission bits from the entry's mode (the
// lowest 9 bits) then using a bitwise OR to set them to the layerEntry's permissions.
entry.setMode((entry.getMode() & ~0777) | layerEntry.getPermissions().getPermissionBits());
entry.setModTime(layerEntry.getModificationTime().toEpochMilli());
setUserAndGroup(entry, layerEntry);
clearTimeHeaders(entry, layerEntry.getModificationTime());

uniqueTarArchiveEntries.add(entry);
}
Expand Down
64 changes: 46 additions & 18 deletions jib-core/src/test/java/com/google/cloud/tools/jib/tar/TarExtractorTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@
import java.nio.file.Paths;
import java.nio.file.attribute.FileTime;
import java.time.Instant;
import java.time.temporal.ChronoUnit;
import java.util.stream.Collectors;
import java.util.stream.Stream;
import org.junit.Assert;
Expand Down Expand Up @@ -92,16 +93,31 @@ public void testExtract_modificationTimePreserved() throws URISyntaxException, I

TarExtractor.extract(source, destination);

assertThat(Files.getLastModifiedTime(destination.resolve("file A")))
.isEqualTo(FileTime.from(Instant.parse("2019-08-01T16:13:09Z")));
assertThat(Files.getLastModifiedTime(destination.resolve("file B")))
.isEqualTo(FileTime.from(Instant.parse("2019-08-01T16:12:00Z")));
assertThat(Files.getLastModifiedTime(destination.resolve("folder")))
.isEqualTo(FileTime.from(Instant.parse("2019-08-01T16:12:33Z")));
assertThat(Files.getLastModifiedTime(destination.resolve("folder/nested folder")))
.isEqualTo(FileTime.from(Instant.parse("2019-08-01T16:13:30Z")));
assertThat(Files.getLastModifiedTime(destination.resolve("folder/nested folder/file C")))
.isEqualTo(FileTime.from(Instant.parse("2019-08-01T16:12:21Z")));
assertThat(
Files.getLastModifiedTime(destination.resolve("file A"))
.toInstant()
.truncatedTo(ChronoUnit.SECONDS))
.isEqualTo(Instant.parse("2019-08-01T16:13:09Z"));
assertThat(
Files.getLastModifiedTime(destination.resolve("file B"))
.toInstant()
.truncatedTo(ChronoUnit.SECONDS))
.isEqualTo(Instant.parse("2019-08-01T16:12:00Z"));
assertThat(
Files.getLastModifiedTime(destination.resolve("folder"))
.toInstant()
.truncatedTo(ChronoUnit.SECONDS))
.isEqualTo(Instant.parse("2019-08-01T16:12:33Z"));
assertThat(
Files.getLastModifiedTime(destination.resolve("folder/nested folder"))
.toInstant()
.truncatedTo(ChronoUnit.SECONDS))
.isEqualTo(Instant.parse("2019-08-01T16:13:30Z"));
assertThat(
Files.getLastModifiedTime(destination.resolve("folder/nested folder/file C"))
.toInstant()
.truncatedTo(ChronoUnit.SECONDS))
.isEqualTo(Instant.parse("2019-08-01T16:12:21Z"));
}

@Test
Expand All @@ -113,14 +129,26 @@ public void testExtract_reproducibleTimestampsEnabled() throws URISyntaxExceptio

TarExtractor.extract(source, destination, true);

assertThat(Files.getLastModifiedTime(destination.resolve("level-1")))
.isEqualTo(FileTime.fromMillis(1000L));
assertThat(Files.getLastModifiedTime(destination.resolve("level-1/level-2")))
.isEqualTo(FileTime.fromMillis(1000L));
assertThat(Files.getLastModifiedTime(destination.resolve("level-1/level-2/level-3")))
.isEqualTo(FileTime.fromMillis(1000L));
assertThat(Files.getLastModifiedTime(destination.resolve("level-1/level-2/level-3/file.txt")))
.isEqualTo(FileTime.from(Instant.parse("2021-01-29T21:10:02Z")));
assertThat(
Files.getLastModifiedTime(destination.resolve("level-1"))
.toInstant()
.truncatedTo(ChronoUnit.SECONDS))
.isEqualTo(FileTime.fromMillis(1000L).toInstant());
assertThat(
Files.getLastModifiedTime(destination.resolve("level-1/level-2"))
.toInstant()
.truncatedTo(ChronoUnit.SECONDS))
.isEqualTo(FileTime.fromMillis(1000L).toInstant());
assertThat(
Files.getLastModifiedTime(destination.resolve("level-1/level-2/level-3"))
.toInstant()
.truncatedTo(ChronoUnit.SECONDS))
.isEqualTo(FileTime.fromMillis(1000L).toInstant());
assertThat(
Files.getLastModifiedTime(destination.resolve("level-1/level-2/level-3/file.txt"))
.toInstant()
.truncatedTo(ChronoUnit.SECONDS))
.isEqualTo(Instant.parse("2021-01-29T21:10:02Z"));
}

@Test
Expand Down
3 changes: 2 additions & 1 deletion jib-gradle-plugin/CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,9 +7,10 @@ All notable changes to this project will be documented in this file.
-

### Changed
-
- deps: bump org.apache.commons:commons-compress from 1.21 to 1.26.0 ([#4204](https:/GoogleContainerTools/jib/pull/4204))

### Fixed
- fix: image builds should become reproducible once again ([#4204](https:/GoogleContainerTools/jib/pull/4204))


## 3.4.1
Expand Down
3 changes: 2 additions & 1 deletion jib-maven-plugin/CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,9 +7,10 @@ All notable changes to this project will be documented in this file.
-

### Changed
-
- deps: bump org.apache.commons:commons-compress from 1.21 to 1.26.0 ([#4204](https:/GoogleContainerTools/jib/pull/4204))

### Fixed
- fix: image builds should become reproducible once again ([#4204](https:/GoogleContainerTools/jib/pull/4204))

## 3.4.1

Expand Down
Loading