Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Call methods of a class generated in runtime #166

Open
dfdx opened this issue May 9, 2022 · 12 comments
Open

Call methods of a class generated in runtime #166

dfdx opened this issue May 9, 2022 · 12 comments

Comments

@dfdx
Copy link
Collaborator

dfdx commented May 9, 2022

jcall fails with NoSuchMethodError when calling methods of a dynamically compiled class.

I use InMemoryJavaCompiler to compile a new class and create its instance. (InMemoryJavaCompiler is based on javax.tools.*, so quite standard approach).

const JInMemoryJavaCompiler = @jimport org.mdkt.compiler.InMemoryJavaCompiler

function mkclass(name::String, src::String)
    jcompiler = jcall(JInMemoryJavaCompiler, "newInstance", JInMemoryJavaCompiler, ())
    return jcall(jcompiler, "compile", JClass, (JString, JString), name, src)
end

function instantiate(name::String, src::String)
    jclass = mkclass(name, src)
    return jcall(jclass, "newInstance", JObject, ())
end


init()
name = "julia.compiled.Hello"
src = """
        package julia.compiled;

        public class Hello {

            public String hello(String name) {
                return "Hello, " + name;
            }

            public void goodbye() {
                System.out.println("Goodbye!");
            }
        }
    """

obj = instantiate(name, src)

The resulting object looks like the ordinary one and I can see the methods I defined:

julia> listmethods(obj)
11-element Vector{JMethod}:
 java.lang.String hello(java.lang.String)
 void goodbye()
 void wait(long)
 void wait(long, int)
 void wait()
 boolean equals(java.lang.Object)
 java.lang.String toString()
 int hashCode()
 java.lang.Class getClass()
 void notify()
 void notifyAll()

However, calling them with jcall fails, even though calling the inherited methods works fine:

julia> jcall(obj, "hello", JString, (JString,), "Bob")
Exception in thread "main" java.lang.NoSuchMethodError: hello
ERROR: JavaCall.JavaCallError("Error calling Java: java.lang.NoSuchMethodError: hello")
Stacktrace:
 [1] geterror(allow::Bool)
   @ JavaCall ~/.julia/packages/JavaCall/MlduK/src/core.jl:418
 [2] jcall(obj::JObject, method::String, rettype::Type, argtypes::Tuple{DataType}, args::String)
   @ JavaCall ~/.julia/packages/JavaCall/MlduK/src/core.jl:244
 [3] top-level scope
   @ REPL[13]:1

julia> jcall(obj, "goodbye", Nothing, ())
Exception in thread "main" java.lang.NoSuchMethodError: goodbye
ERROR: JavaCall.JavaCallError("Error calling Java: java.lang.NoSuchMethodError: goodbye")
Stacktrace:
 [1] geterror(allow::Bool)
   @ JavaCall ~/.julia/packages/JavaCall/MlduK/src/core.jl:418
 [2] jcall(::JObject, ::String, ::Type, ::Tuple{})
   @ JavaCall ~/.julia/packages/JavaCall/MlduK/src/core.jl:244
 [3] top-level scope
   @ REPL[14]:1

julia> jcall(obj, "equals", jboolean, (JObject,), obj)
0x01

What's even more confusing, these new methods can actually be called using the reflection:

function jcall2(jobj::JavaObject, name::String, ret_type, arg_types, args...)
    jclass = getclass(jobj)
    jargs = [a for a in convert.(arg_types, args)]  # convert to Vector
    meth = jcall(jclass, "getMethod", JMethod, (JString, Vector{JClass}), name, getclass.(jargs))
    return meth(jobj, jargs...)
end

julia> jcall2(obj, "hello", JString, (JString,), "Bob")
"Hello, Bob"

julia> jcall2(obj, "goodbye", Nothing, ())
Goodbye!

Although it fails with NoSuchMethodException for the inherited method ¯_(ツ)_/¯:

julia> jcall2(obj, "equals", jboolean, (JObject,), obj)
Exception in thread "main" java.lang.NoSuchMethodException: julia.compiled.Hello.equals(julia.compiled.Hello)
        at java.base/java.lang.Class.getMethod(Class.java:2108)
ERROR: JavaCall.JavaCallError("Error calling Java: java.lang.NoSuchMethodException: julia.compiled.Hello.equals(julia.compiled.Hello)")
Stacktrace:
 [1] geterror(allow::Bool)
   @ JavaCall ~/.julia/packages/JavaCall/MlduK/src/core.jl:418
 [2] geterror
   @ ~/.julia/packages/JavaCall/MlduK/src/core.jl:403 [inlined]
 [3] _jcall(::JClass, ::Ptr{Nothing}, ::Ptr{Nothing}, ::Type, ::Tuple{DataType, DataType}, ::String, ::Vararg{Any})
   @ JavaCall ~/.julia/packages/JavaCall/MlduK/src/core.jl:373
 [4] jcall(::JClass, ::String, ::Type, ::Tuple{DataType, DataType}, ::String, ::Vararg{Any})
   @ JavaCall ~/.julia/packages/JavaCall/MlduK/src/core.jl:245
 [5] jcall2(jobj::JObject, name::String, ret_type::Type, arg_types::Tuple{DataType}, args::JObject)
   @ Main ./REPL[17]:4
 [6] top-level scope
   @ REPL[18]:1

The code above is executable from the Spark#new-api branch, but honestly I hope that I'm just doing some stupid mistake that a keen eye can catch just from the description.

@mkitti
Copy link
Member

mkitti commented May 9, 2022

Could you clarify which version of JavaCall.jl are using?

Note that there is https:/JuliaInterop/JavaCall.jl/releases/tag/v0.8.0rc-1

@dfdx
Copy link
Collaborator Author

dfdx commented May 9, 2022

I tested it on v0.7.8, but v0.8.0rc-1 gives the same result anyway:

julia> jcall(obj, "hello", JString, (JString,), "Bob")
Exception in thread "main" java.lang.NoSuchMethodError: hello
ERROR: JavaCall.JavaCallError("Error calling Java: java.lang.NoSuchMethodError: hello")
Stacktrace:
 [1] geterror()
   @ JavaCall ~/.julia/packages/JavaCall/KFY5m/src/core.jl:542
 [2] get_method_id(jnifun::typeof(JavaCall.JNI.GetMethodID), obj::JObject, method::String, rettype::Type, argtypes::Tuple{DataType})
   @ JavaCall ~/.julia/packages/JavaCall/KFY5m/src/core.jl:255
 [3] get_method_id
   @ ~/.julia/packages/JavaCall/KFY5m/src/core.jl:393 [inlined]
 [4] jcall(ref::JObject, method::String, rettype::Type, argtypes::Tuple{DataType}, args::String)
   @ JavaCall ~/.julia/packages/JavaCall/KFY5m/src/core.jl:370
 [5] top-level scope
   @ REPL[9]:1

And just in case:

julia> versioninfo()
Julia Version 1.7.2
Commit bf53498635 (2022-02-06 15:21 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
  CPU: Intel(R) Core(TM) i7-8550U CPU @ 1.80GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-12.0.1 (ORCJIT, skylake)
Environment:
  JULIA_COPY_STACKS = 1
  JULIA_EDITOR = code
  JULIA_NUM_THREADS = 

@mkitti
Copy link
Member

mkitti commented May 9, 2022

I notice that they use a dynamic class loader. Do they mention anything regarding the Java Native Interface?

@dfdx
Copy link
Collaborator Author

dfdx commented May 9, 2022

Not really. I will try it with the standard ClassLoader.

@dfdx
Copy link
Collaborator Author

dfdx commented May 9, 2022

Nope, the same error.

Self-containing example
using JavaCall


const JFile = @jimport java.io.File
const JToolProvider = @jimport javax.tools.ToolProvider
const JJavaCompiler = @jimport javax.tools.JavaCompiler
const JInputStream = @jimport java.io.InputStream
const JOutputStream = @jimport java.io.OutputStream
const JClassLoader = @jimport java.lang.ClassLoader
const JURLClassLoader = @jimport java.net.URLClassLoader
const JURI = @jimport java.net.URI
const JURL = @jimport java.net.URL


function JavaCall.classforname(name::String, loader)
    return jcall(JClass, "forName", JClass, (JString, jboolean, JClassLoader),
                 name, true, loader)
end


function mkclass(name::String, src::String)
    # based on https://stackoverflow.com/a/2946402
    cls = mktempdir(; prefix="jj-") do root
        # write source to a file
        elems = split(name, ".")
        pkg_path = joinpath(root, elems[1:end-1]...)
        mkpath(pkg_path)
        src_path = joinpath(pkg_path, elems[end] * ".java")
        open(src_path, "w") do f
            write(f, src)
        end
        # compile
        jcompiler = jcall(JToolProvider, "getSystemJavaCompiler", JJavaCompiler)
        jcall(jcompiler, "run", jint,
            (JInputStream, JOutputStream, JOutputStream, Vector{JString}),
            nothing, nothing, nothing, [src_path])
        # load class
        jfile = JFile((JString,), root)
        juri = jcall(jfile, "toURI", JURI)
        jurl = jcall(juri, "toURL", JURL)
        jloader = jcall(JURLClassLoader, "newInstance", JURLClassLoader, (Vector{JURL},), [jurl])
        classforname(name, jloader)
    end
    return cls
end


function instantiate(name::String, src::String)
    jclass = mkclass(name, src)
    return jcall(jclass, "newInstance", JObject, ())
end


JavaCall.init()
name = "julia.compiled.Hello"
src = """
        package julia.compiled;

        public class Hello {

            public String hello(String name) {
                return "Hello, " + name;
            }

            public void goodbye() {
                System.out.println("Goodbye!");
            }
        }
    """

obj = instantiate(name, src)
jcall(obj, "hello", JString, (JString,), "Bob")   # NoSuchMethodError:
jcall(obj, "goodbye", Nothing, ())                # NoSuchMethodError:
jcall(obj, "equals", jboolean, (JObject,), obj)   # ok

@mkitti
Copy link
Member

mkitti commented May 9, 2022

Well my point was the opposite actually. We need to use InMemoryJavaCompiler's dynamic classloader to load your classes, but JavaCall doesn't know anything about that. It just uses JNI's FindClass: https://docs.oracle.com/javase/8/docs/technotes/guides/jni/spec/functions.html#FindClass

@dfdx
Copy link
Collaborator Author

dfdx commented May 9, 2022

But isn't FindClass only needed to locate and load the class? I thought after loading a class is self-containing, including all information required by JNI to work properly.

Would it help if we introduce a proxy class loaded using the system class loader (and thus JNI-friendly), but in runtime loading the new generated class?

@mkitti
Copy link
Member

mkitti commented May 10, 2022

This is failing at https://docs.oracle.com/javase/8/docs/technotes/guides/jni/spec/functions.html#GetMethodID.

The question then is what are the correct arguments to pass to JavaCall.JNI.GetMethodID to get this to work? Or perhaps GetMethodID is not the right way to go.

Since reflection seems to be working, perhaps you should work off this result:

julia> listmethods(obj)
11-element Vector{JMethod}:
 java.lang.String hello(java.lang.String)
 void goodbye()
 void wait(long)
 void wait(long, int)
 void wait()
 boolean equals(java.lang.Object)
 java.lang.String toString()
 int hashCode()
 java.lang.Class getClass()
 void notify()
 void notifyAll()

v0.8.0rc-1 offers this new way of using jcall:

JavaCall.jl/src/core.jl

Lines 374 to 380 in 63026e4

function jcall(ref, method::JMethod, args...)
assertroottask_or_goodenv() && assertloaded()
jmethodId = get_method_id(method)
rettype = jimport(getreturntype(method))
argtypes = Tuple(jimport.(getparametertypes(method)))
_jcall(_jcallable(ref), jmethodId, rettype, argtypes, args...)
end

where get_method_id is implemented via JNI.FromReflectedMethod

get_method_id(method::JMethod) = @checknull JNI.FromReflectedMethod(method)

Combined with the two argument form of listmethods you might be able to achieve what you want.

JavaCall.jl/src/reflect.jl

Lines 111 to 114 in 63026e4

function listmethods(obj::Union{JavaObject{C}, Type{JavaObject{C}}}, name::AbstractString) where C
allmethods = listmethods(obj)
filter(m -> getname(m) == name, allmethods)
end

@dfdx
Copy link
Collaborator Author

dfdx commented May 15, 2022

Just wanted to post an updated on this. I tried the suggested method, but it turns out due to @checknull any call to get_method_id on a non-existing method throws an exception and prints unwanted error message to console. This makes it impossible to create a robust side-effect-free version of jcall without rewriting half of its machinery. If I have a minute, I'll make a PR here, but it's not the top priority for me at the moment.

Also, I test it on OpenJDK 11, which I already had problems with in the past. Oracle download site for their JDK 11 is broken right now, once it's fixed, I'll check if this problem exists there too.

@mkitti
Copy link
Member

mkitti commented May 16, 2022

Can you try to use JavaCall.JNI.* directly as if you were a C programmer? If you can demonstrate a string of JNI calls that work, then we can design a version of jcall that may work.

@mkitti
Copy link
Member

mkitti commented May 16, 2022

Concretely, is there any other JNI method other than FromReflectedMethod that works? Does FromReflectedMethod work in all cases?

@dfdx
Copy link
Collaborator Author

dfdx commented May 16, 2022

I tried JNI.GetMethodID, JNI.GetStaticMethodID and even JNI.GetFieldID and JNI.GetStaticFieldID, but none of them worked. JNI.FromReflectedMethod works fine so far. The code I used:

Generate object (copy from above)
# copy from one of the previous comments
using JavaCall


const JFile = @jimport java.io.File
const JToolProvider = @jimport javax.tools.ToolProvider
const JJavaCompiler = @jimport javax.tools.JavaCompiler
const JInputStream = @jimport java.io.InputStream
const JOutputStream = @jimport java.io.OutputStream
const JClassLoader = @jimport java.lang.ClassLoader
const JURLClassLoader = @jimport java.net.URLClassLoader
const JURI = @jimport java.net.URI
const JURL = @jimport java.net.URL


function JavaCall.classforname(name::String, loader)
    return jcall(JClass, "forName", JClass, (JString, jboolean, JClassLoader),
                 name, true, loader)
end


function mkclass(name::String, src::String)
    # based on https://stackoverflow.com/a/2946402
    cls = mktempdir(; prefix="jj-") do root
        # write source to a file
        elems = split(name, ".")
        pkg_path = joinpath(root, elems[1:end-1]...)
        mkpath(pkg_path)
        src_path = joinpath(pkg_path, elems[end] * ".java")
        open(src_path, "w") do f
            write(f, src)
        end
        # compile
        jcompiler = jcall(JToolProvider, "getSystemJavaCompiler", JJavaCompiler)
        jcall(jcompiler, "run", jint,
            (JInputStream, JOutputStream, JOutputStream, Vector{JString}),
            nothing, nothing, nothing, [src_path])
        # load class
        jfile = JFile((JString,), root)
        juri = jcall(jfile, "toURI", JURI)
        jurl = jcall(juri, "toURL", JURL)
        jloader = jcall(JURLClassLoader, "newInstance", JURLClassLoader, (Vector{JURL},), [jurl])
        classforname(name, jloader)
    end
    return cls
end


function instantiate(name::String, src::String)
    jclass = mkclass(name, src)
    return jcall(jclass, "newInstance", JObject, ())
end


JavaCall.init()
name = "julia.compiled.Hello"
src = """
        package julia.compiled;

        public class Hello {

            public String hello(String name) {
                return "Hello, " + name;
            }

            public void goodbye() {
                System.out.println("Goodbye!");
            }
        }
    """

obj = instantiate(name, src)
import JavaCall.JNI

sig = JavaCall.method_signature(JString, JString)
ptr = Ptr(JavaCall.metaclass(obj))

# methods
JNI.GetMethodID(ptr, "call", sig)           # ==> Ptr{Nothing} @0x0000000000000000
JNI.GetStaticMethodID(ptr, "call", sig)   # ==> Ptr{Nothing} @0x0000000000000000

# fields, just in case, with method signature
JNI.GetFieldID(ptr, "call", sig)               # ==> Ptr{Nothing} @0x0000000000000000
JNI.GetStaticFieldID(ptr, "call", sig)      # ==> Ptr{Nothing} @0x0000000000000000

# reflected method
meth = listmethods(obj, "hello")[1]
JNI.FromReflectedMethod(meth)         # ==> Ptr{Nothing} @0x00000000029d26a0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants