java.lang.NoSuchFieldError: NAME_RESOLUTION_DELAYED

[vorburger]

What version of gRPC-Java are you using?

1.58.0

What is your environment?

Linux

java -version
openjdk version "17.0.8" 2023-07-18
OpenJDK Runtime Environment (Red_Hat-17.0.8.0.7-1.fc38) (build 17.0.8+7)
OpenJDK 64-Bit Server VM (Red_Hat-17.0.8.0.7-1.fc38) (build 17.0.8+7, mixed mode, sharing)

What did you expect to see?

Test that worked in 1.54.1 fails when upgrading to 1.58.0.

What did you see instead?

java.lang.NoSuchFieldError: NAME_RESOLUTION_DELAYED
        at io.grpc.internal.ManagedChannelImpl$RealChannel$PendingCall.reprocess(ManagedChannelImpl.java:1083)
        at io.grpc.internal.ManagedChannelImpl$RealChannel.updateConfigSelector(ManagedChannelImpl.java:1013)
        at io.grpc.internal.ManagedChannelImpl$NameResolverListener$1NamesResolved.run(ManagedChannelImpl.java:1810)
        at io.grpc.SynchronizationContext.drain(SynchronizationContext.java:95)
        at io.grpc.SynchronizationContext.execute(SynchronizationContext.java:127)
        at io.grpc.internal.ManagedChannelImpl$NameResolverListener.onResult(ManagedChannelImpl.java:1862)
        at io.grpc.internal.RetryingNameResolver$RetryingListener.onResult(RetryingNameResolver.java:98)
        at io.grpc.internal.DnsNameResolver$Resolve.run(DnsNameResolver.java:333)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)

Steps to reproduce the bug

It currently appears inside a documentation generation integration test on enola-dev/enola#295.

I will make it easier to reproduce by creating a unit test.

[vorburger]

As I'm seeing this in the context of a gRPC upgrade from 1.54.1 to 1.58.0, where I've hit #10576, I was wondering this NoSuchFieldError might be caused by the exclusion of grpc-util. I have tried to add an explicit dependency to it, but that didn't do the trick.

It currently appears inside a documentation generation integration test on enola-dev/enola#295.

It's reproducible by grabbing that PR and running docs/use/connector/demo-grpc.bash.

I will make it easier to reproduce by creating a unit test.

There actually already was a Unit Test covering the same - and that one works.

That demo-grpc.bash script (indirectly) runs a bazel-bin/... something. I'm wondering if that runtime dependency on grpc-util (from #10576) could be the culprit. But then it should work. But if that were so, the depending on grpc-util manually, as I tried, should have fixed it - and it doesn't.

@ejona86 might you have any suggestion what could be causing this?

[vorburger]

java.lang.NoSuchFieldError: NAME_RESOLUTION_DELAYED
        at io.grpc.internal.ManagedChannelImpl$RealChannel$PendingCall.reprocess(ManagedChannelImpl.java:1083)

So that's here:

grpc-java/core/src/main/java/io/grpc/internal/ManagedChannelImpl.java

Line 1083 in f9562b1

CallOptions delayResolutionOption = callOptions.withOption(NAME_RESOLUTION_DELAYED, true);

which uses this: https:/grpc/grpc-java/blame/2cbc290755639bcbd57d7a1e5ecebe49a6562537/api/src/main/java/io/grpc/ClientStreamTracer.java#L34

which was introduced only 5 months ago in #10044 ... so it is in 1.58.0 but probably wasn't in 1.54.1 yet (haven't checked exactly).

Which makes this smell like a classpath dependency hell issue - my project may have different versions of gRPC JAR?

b query "@maven//:*" | grep grpc only shows 1.58.0 - but I might have non-Maven (not from rules_jvm_external) depencies?! Checking...

[ejona86]

This is almost certainly caused by version skew between the artifacts. Excluding grpc-util shouldn't be related.

I expect somehow grpc-api 1.54.1 is being used (the symbol was added in 1.55.0). That shouldn't happen, as grpc-core is clearly 1.55+ and depends on grpc-api of the same version. That isn't part of the coursier results, as we can see in the lock file it is using 1.58.0. Also, this would be a more surprising result, as generally I only expect Maven to be bad enough to downgrade dependencies.

Since it doesn't seem to be caused by maven_install, I suggest using unzip -l and grep to find the other jar file on the path that has grpc-api contents inside.

[vorburger]

This is almost certainly caused by version skew between the artifacts. Excluding grpc-util shouldn't be related.

@ejona86 yup, there is definitely a (huge) classpath mess on my project... ;-)

I suggest using unzip -l and grep to find the other jar file on the path that has grpc-api contents inside.

I have a much more fun idea... enola-dev/enola#297!

Let me close this issue, as this is clearly not a gRPC "upstream" problem.