From 30212e447a507467e56f28900dca4067c7bd9e13 Mon Sep 17 00:00:00 2001 From: panbingkun Date: Fri, 22 Dec 2023 23:17:58 +0800 Subject: [PATCH] GH-39265: [Java] Make it run well with the netty newest version 4.1.104 (#39266) When I used `netty arrow memory 14.0.1` and `netty 4.1.104.Final` in Spark, the following error occurred, After pr: https://github.com/netty/netty/pull/13613, `PoolArena` no longer extends `SizeClasses`, but instead uses it as one of its fields, as follows: image in order to ensure that `netty arrow memory 14.0.1` works well with `netty 4.1.104.Final` version, I suggest making similar modifications here. 1.Compilation errors are as follows: https://github.com/panbingkun/spark/actions/runs/7237466030/job/19717162391 image 2.Some bugs have been fixed in `netty 4.1.104.Final` as follows: image image 4.1.104.Final release note: https://netty.io/news/2023/12/15/4-1-104-Final.html 4.1.103.Final release note: https://netty.io/news/2023/12/13/4-1-103-Final.html 4.1.101.Final release note: https://netty.io/news/2023/11/09/4-1-101-Final.html Java * Closes: #39265 Authored-by: panbingkun Signed-off-by: David Li GH-38537: [Java] upgrade to netty 4.1.100.Final (#38538) Screenshot 2023-11-01 at 14 58 19 Screenshot 2023-11-01 at 14 58 36 new version of netty. no * Closes: #38537 Authored-by: Ivan Chesnov Signed-off-by: David Li GH-38996: [Java] Update dependencies and plugins for JPMS modules (#38994) Update dependencies and plugins to versions that work correctly with JPMS modules. * Update several plugins to use module-enabled versions: * maven-compiler-plugin is updated to 3.11.0 * maven-shade-plugin is updated to 3.2.4 * maven-dependency-plugin is updated to 3.1.2 * CycloneDX is updated to 2.7.10 * Update grpc-java to 1.59 for module support * Update jackson to 2.16.0 as 2.15.1 had corrupt module-info.class files that broke module support. Existing tests only Users may need to update their own dependencies if they have the same ones. * Closes: #38996 Authored-by: James Duong Signed-off-by: Sutou Kouhei MINOR: [Java] Bump dep.junit.jupiter.version from 5.9.0 to 5.10.1 in /java (#39102) Bumps `dep.junit.jupiter.version` from 5.9.0 to 5.10.1. Updates `org.junit.jupiter:junit-jupiter-engine` from 5.9.0 to 5.10.1
Release notes

Sourced from org.junit.jupiter:junit-jupiter-engine's releases.

JUnit 5.10.1 = Platform 1.10.1 + Jupiter 5.10.1 + Vintage 5.10.1

See Release Notes.

Full Changelog: https://github.com/junit-team/junit5/compare/r5.10.0...r5.10.1

JUnit 5.10.0 = Platform 1.10.0 + Jupiter 5.10.0 + Vintage 5.10.0

See Release Notes.

Full Changelog: https://github.com/junit-team/junit5/compare/r5.10.0-RC2...r5.10.0

JUnit 5.10.0-RC2 = Platform 1.10.0-RC2+ Jupiter 5.10.0-RC2 + Vintage 5.10.0-RC2

See Release Notes.

JUnit 5.10.0-RC1 = Platform 1.10.0-RC1 + Jupiter 5.10.0-RC1 + Vintage 5.10.0-RC1

See Release Notes.

JUnit 5.10.0-M1 = Platform 1.10.0-M1 + Jupiter 5.10.0-M1 + Vintage 5.10.0-M1

See Release Notes.

JUnit 5.9.3 = Platform 1.9.3 + Jupiter 5.9.3 + Vintage 5.9.3

See Release Notes.

JUnit 5.9.2 = Platform 1.9.2 + Jupiter 5.9.2 + Vintage 5.9.2

See Release Notes.

JUnit 5.9.1 = Platform 1.9.1 + Jupiter 5.9.1 + Vintage 5.9.1

See Release Notes.

Commits
  • e5f50d8 Release 5.10.1
  • ac86d18 Fix typo in AfterAll documentation
  • 388c5be Harmonize application of method and field filters in search algorithms
  • f82dd1e Apply field predicate before searching type hierarchy
  • 1d1eb85 Polishing
  • 5ce280e Update picocli to 4.7.5 and enable help width computation
  • fea05c3 Fix ConsoleLauncherTests and StandaloneTests
  • c556735 Use same expected files for all JDK versions
  • 808493a Run StandaloneTests for Java 8 under Java 8
  • 9ec5766 Unify messages about exit codes in StandaloneTests
  • Additional commits viewable in compare view

Updates `org.junit.jupiter:junit-jupiter-api` from 5.9.0 to 5.10.1
Release notes

Sourced from org.junit.jupiter:junit-jupiter-api's releases.

JUnit 5.10.1 = Platform 1.10.1 + Jupiter 5.10.1 + Vintage 5.10.1

See Release Notes.

Full Changelog: https://github.com/junit-team/junit5/compare/r5.10.0...r5.10.1

JUnit 5.10.0 = Platform 1.10.0 + Jupiter 5.10.0 + Vintage 5.10.0

See Release Notes.

Full Changelog: https://github.com/junit-team/junit5/compare/r5.10.0-RC2...r5.10.0

JUnit 5.10.0-RC2 = Platform 1.10.0-RC2+ Jupiter 5.10.0-RC2 + Vintage 5.10.0-RC2

See Release Notes.

JUnit 5.10.0-RC1 = Platform 1.10.0-RC1 + Jupiter 5.10.0-RC1 + Vintage 5.10.0-RC1

See Release Notes.

JUnit 5.10.0-M1 = Platform 1.10.0-M1 + Jupiter 5.10.0-M1 + Vintage 5.10.0-M1

See Release Notes.

JUnit 5.9.3 = Platform 1.9.3 + Jupiter 5.9.3 + Vintage 5.9.3

See Release Notes.

JUnit 5.9.2 = Platform 1.9.2 + Jupiter 5.9.2 + Vintage 5.9.2

See Release Notes.

JUnit 5.9.1 = Platform 1.9.1 + Jupiter 5.9.1 + Vintage 5.9.1

See Release Notes.

Commits
  • e5f50d8 Release 5.10.1
  • ac86d18 Fix typo in AfterAll documentation
  • 388c5be Harmonize application of method and field filters in search algorithms
  • f82dd1e Apply field predicate before searching type hierarchy
  • 1d1eb85 Polishing
  • 5ce280e Update picocli to 4.7.5 and enable help width computation
  • fea05c3 Fix ConsoleLauncherTests and StandaloneTests
  • c556735 Use same expected files for all JDK versions
  • 808493a Run StandaloneTests for Java 8 under Java 8
  • 9ec5766 Unify messages about exit codes in StandaloneTests
  • Additional commits viewable in compare view

Updates `org.junit.vintage:junit-vintage-engine` from 5.9.0 to 5.10.1
Release notes

Sourced from org.junit.vintage:junit-vintage-engine's releases.

JUnit 5.10.1 = Platform 1.10.1 + Jupiter 5.10.1 + Vintage 5.10.1

See Release Notes.

Full Changelog: https://github.com/junit-team/junit5/compare/r5.10.0...r5.10.1

JUnit 5.10.0 = Platform 1.10.0 + Jupiter 5.10.0 + Vintage 5.10.0

See Release Notes.

Full Changelog: https://github.com/junit-team/junit5/compare/r5.10.0-RC2...r5.10.0

JUnit 5.10.0-RC2 = Platform 1.10.0-RC2+ Jupiter 5.10.0-RC2 + Vintage 5.10.0-RC2

See Release Notes.

JUnit 5.10.0-RC1 = Platform 1.10.0-RC1 + Jupiter 5.10.0-RC1 + Vintage 5.10.0-RC1

See Release Notes.

JUnit 5.10.0-M1 = Platform 1.10.0-M1 + Jupiter 5.10.0-M1 + Vintage 5.10.0-M1

See Release Notes.

JUnit 5.9.3 = Platform 1.9.3 + Jupiter 5.9.3 + Vintage 5.9.3

See Release Notes.

JUnit 5.9.2 = Platform 1.9.2 + Jupiter 5.9.2 + Vintage 5.9.2

See Release Notes.

JUnit 5.9.1 = Platform 1.9.1 + Jupiter 5.9.1 + Vintage 5.9.1

See Release Notes.

Commits
  • e5f50d8 Release 5.10.1
  • ac86d18 Fix typo in AfterAll documentation
  • 388c5be Harmonize application of method and field filters in search algorithms
  • f82dd1e Apply field predicate before searching type hierarchy
  • 1d1eb85 Polishing
  • 5ce280e Update picocli to 4.7.5 and enable help width computation
  • fea05c3 Fix ConsoleLauncherTests and StandaloneTests
  • c556735 Use same expected files for all JDK versions
  • 808493a Run StandaloneTests for Java 8 under Java 8
  • 9ec5766 Unify messages about exit codes in StandaloneTests
  • Additional commits viewable in compare view

Updates `org.junit.jupiter:junit-jupiter-params` from 5.9.0 to 5.10.1
Release notes

Sourced from org.junit.jupiter:junit-jupiter-params's releases.

JUnit 5.10.1 = Platform 1.10.1 + Jupiter 5.10.1 + Vintage 5.10.1

See Release Notes.

Full Changelog: https://github.com/junit-team/junit5/compare/r5.10.0...r5.10.1

JUnit 5.10.0 = Platform 1.10.0 + Jupiter 5.10.0 + Vintage 5.10.0

See Release Notes.

Full Changelog: https://github.com/junit-team/junit5/compare/r5.10.0-RC2...r5.10.0

JUnit 5.10.0-RC2 = Platform 1.10.0-RC2+ Jupiter 5.10.0-RC2 + Vintage 5.10.0-RC2

See Release Notes.

JUnit 5.10.0-RC1 = Platform 1.10.0-RC1 + Jupiter 5.10.0-RC1 + Vintage 5.10.0-RC1

See Release Notes.

JUnit 5.10.0-M1 = Platform 1.10.0-M1 + Jupiter 5.10.0-M1 + Vintage 5.10.0-M1

See Release Notes.

JUnit 5.9.3 = Platform 1.9.3 + Jupiter 5.9.3 + Vintage 5.9.3

See Release Notes.

JUnit 5.9.2 = Platform 1.9.2 + Jupiter 5.9.2 + Vintage 5.9.2

See Release Notes.

JUnit 5.9.1 = Platform 1.9.1 + Jupiter 5.9.1 + Vintage 5.9.1

See Release Notes.

Commits
  • e5f50d8 Release 5.10.1
  • ac86d18 Fix typo in AfterAll documentation
  • 388c5be Harmonize application of method and field filters in search algorithms
  • f82dd1e Apply field predicate before searching type hierarchy
  • 1d1eb85 Polishing
  • 5ce280e Update picocli to 4.7.5 and enable help width computation
  • fea05c3 Fix ConsoleLauncherTests and StandaloneTests
  • c556735 Use same expected files for all JDK versions
  • 808493a Run StandaloneTests for Java 8 under Java 8
  • 9ec5766 Unify messages about exit codes in StandaloneTests
  • Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@ dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- java/algorithm/pom.xml | 1 + java/flight/flight-core/pom.xml | 12 ++++--- java/flight/flight-grpc/pom.xml | 11 ++++--- java/flight/flight-sql-jdbc-driver/pom.xml | 1 - java/flight/flight-sql/pom.xml | 2 ++ java/memory/memory-netty/pom.xml | 2 +- .../netty/buffer/PooledByteBufAllocatorL.java | 7 ++-- java/performance/pom.xml | 7 +++- java/pom.xml | 33 ++++++++++++------- java/tools/pom.xml | 3 +- java/vector/pom.xml | 8 ++++- 11 files changed, 56 insertions(+), 31 deletions(-) diff --git a/java/algorithm/pom.xml b/java/algorithm/pom.xml index 60f9b6c8e25fb..abf76d352506e 100644 --- a/java/algorithm/pom.xml +++ b/java/algorithm/pom.xml @@ -32,6 +32,7 @@ arrow-vector ${project.version} test-jar + test org.apache.arrow diff --git a/java/flight/flight-core/pom.xml b/java/flight/flight-core/pom.xml index be91203d73d05..6fce81c0d3b69 100644 --- a/java/flight/flight-core/pom.xml +++ b/java/flight/flight-core/pom.xml @@ -58,10 +58,6 @@ io.grpc grpc-core - - io.grpc - grpc-context - io.grpc grpc-protobuf @@ -153,7 +149,13 @@ org.apache.maven.plugins maven-shade-plugin - 3.1.1 + + 3.2.4 shade-main diff --git a/java/flight/flight-grpc/pom.xml b/java/flight/flight-grpc/pom.xml index 864c629f6b044..318495bf73b31 100644 --- a/java/flight/flight-grpc/pom.xml +++ b/java/flight/flight-grpc/pom.xml @@ -49,12 +49,13 @@ io.grpc - grpc-core + grpc-stub + + + io.grpc + grpc-inprocess + test - - io.grpc - grpc-stub - org.apache.arrow arrow-memory-core diff --git a/java/flight/flight-sql-jdbc-driver/pom.xml b/java/flight/flight-sql-jdbc-driver/pom.xml index a0edf112f73e7..b7060f77932e4 100644 --- a/java/flight/flight-sql-jdbc-driver/pom.xml +++ b/java/flight/flight-sql-jdbc-driver/pom.xml @@ -164,7 +164,6 @@ org.apache.maven.plugins maven-shade-plugin - 3.4.1 package diff --git a/java/flight/flight-sql/pom.xml b/java/flight/flight-sql/pom.xml index bcefc4aeb830c..e760c9a345bf1 100644 --- a/java/flight/flight-sql/pom.xml +++ b/java/flight/flight-sql/pom.xml @@ -56,6 +56,7 @@ org.apache.arrow arrow-jdbc ${project.version} + test com.google.guava @@ -74,6 +75,7 @@ org.slf4j slf4j-api + test org.apache.derby diff --git a/java/memory/memory-netty/pom.xml b/java/memory/memory-netty/pom.xml index df39c0058ba33..784320c7d6acb 100644 --- a/java/memory/memory-netty/pom.xml +++ b/java/memory/memory-netty/pom.xml @@ -42,7 +42,7 @@ ch.qos.logback logback-core - 1.2.12 + 1.3.14 test diff --git a/java/memory/memory-netty/src/main/java/io/netty/buffer/PooledByteBufAllocatorL.java b/java/memory/memory-netty/src/main/java/io/netty/buffer/PooledByteBufAllocatorL.java index 06c6669cfd162..ba9aba353c351 100644 --- a/java/memory/memory-netty/src/main/java/io/netty/buffer/PooledByteBufAllocatorL.java +++ b/java/memory/memory-netty/src/main/java/io/netty/buffer/PooledByteBufAllocatorL.java @@ -71,7 +71,7 @@ public UnsafeDirectLittleEndian allocate(long size) { } public int getChunkSize() { - return allocator.chunkSize; + return allocator.chunkSize(); } public long getHugeBufferSize() { @@ -137,7 +137,6 @@ private class InnerAllocator extends PooledByteBufAllocator { private final PoolArena[] directArenas; private final MemoryStatusThread statusThread; - private final int chunkSize; public InnerAllocator() { super(true); @@ -150,8 +149,6 @@ public InnerAllocator() { throw new RuntimeException("Failure while initializing allocator. Unable to retrieve direct arenas field.", e); } - this.chunkSize = directArenas[0].chunkSize; - if (memoryLogger.isTraceEnabled()) { statusThread = new MemoryStatusThread(this); statusThread.start(); @@ -166,7 +163,7 @@ private UnsafeDirectLittleEndian newDirectBufferL(int initialCapacity, int maxCa if (directArena != null) { - if (initialCapacity > directArena.chunkSize) { + if (initialCapacity > chunkSize()) { // This is beyond chunk size so we'll allocate separately. ByteBuf buf = UnpooledByteBufAllocator.DEFAULT.directBuffer(initialCapacity, maxCapacity); diff --git a/java/performance/pom.xml b/java/performance/pom.xml index 685b502d27250..f520ddae1515f 100644 --- a/java/performance/pom.xml +++ b/java/performance/pom.xml @@ -26,6 +26,7 @@ org.openjdk.jmh jmh-core ${jmh.version} + test org.openjdk.jmh @@ -38,11 +39,13 @@ arrow-vector ${project.version} ${arrow.vector.classifier} + test org.apache.arrow arrow-memory-core ${project.version} + test org.apache.arrow @@ -54,11 +57,13 @@ org.apache.avro avro ${dep.avro.version} + test org.apache.arrow arrow-avro ${project.version} + test com.h2database @@ -70,6 +75,7 @@ org.apache.arrow arrow-jdbc ${project.version} + test org.apache.arrow @@ -109,7 +115,6 @@ org.apache.maven.plugins maven-shade-plugin - 2.2 package diff --git a/java/pom.xml b/java/pom.xml index dea9baac371a8..b3514f63b903e 100644 --- a/java/pom.xml +++ b/java/pom.xml @@ -30,13 +30,13 @@ ${project.build.directory}/generated-sources 1.9.0 - 5.9.0 - 1.7.25 - 31.1-jre - 4.1.96.Final - 1.56.0 + 5.10.1 + 2.0.9 + 33.0.0-jre + 4.1.104.Final + 1.60.0 3.23.1 - 2.15.1 + 2.16.0 2.7.1 1.12.0 1.10.0 @@ -45,7 +45,7 @@ true 9+181-r4173-1 2.22.0 - 3.10.1 + 3.11.0 5.5.0 5.2.0 @@ -308,7 +308,7 @@ org.slf4j jcl-over-slf4j - 1.7.5 + ${dep.slf4j.version} @@ -361,7 +361,7 @@ org.cyclonedx cyclonedx-maven-plugin - 2.7.6 + 2.7.10 package @@ -378,7 +378,12 @@ org.apache.maven.plugins maven-dependency-plugin - 3.0.1 + + 3.1.2 org.apache.rat @@ -395,6 +400,7 @@ maven-compiler-plugin ${maven-compiler-plugin.version} + false org.immutables @@ -408,6 +414,11 @@ maven-enforcer-plugin 3.0.0-M2 + + org.apache.maven.plugins + maven-shade-plugin + 3.5.1 + maven-surefire-plugin 3.0.0-M7 @@ -679,7 +690,7 @@ ch.qos.logback logback-classic - 1.2.3 + 1.3.14 test diff --git a/java/tools/pom.xml b/java/tools/pom.xml index aa6f08c8dd70a..62b821f87cc26 100644 --- a/java/tools/pom.xml +++ b/java/tools/pom.xml @@ -40,6 +40,7 @@ com.google.guava guava + test commons-cli @@ -49,7 +50,7 @@ ch.qos.logback logback-classic - 1.2.3 + 1.3.14 runtime diff --git a/java/vector/pom.xml b/java/vector/pom.xml index 31861862fa9ab..423b119e1e1ed 100644 --- a/java/vector/pom.xml +++ b/java/vector/pom.xml @@ -177,7 +177,13 @@ org.apache.maven.plugins maven-shade-plugin - 3.1.1 + + 3.2.4 package