Suggestions for simplifying module creation

[alerque]

I'm having some trouble conceptualizing how to setup a project and am looking for any suggestions for how to keep it simple so I don't end up with 4 implementations of every function to maintain. I've had great success both using mlua as a runtime and for using it to create Lua modules. This projects needs to both at the same time and it's a bit of a mind bender.

The real project is SILE and my WIP is here, but I'll try to describe the relevant bits in MWE fashion here.

The first piece –the main project crate– is the CLI with a mlua based VM that loads a bunch of Lua code.

One complicating factor is that this needs to compile in several modes: with or without being built against system Lua sources (so the vendored feature), and also dynamic or static linking, and with the static option also a variant that has all the Lua resources (script and C modules) embedded in the binary. All that works already, but we have two more layers before things start getting weird.

I would also like to write code in Rust in the main Rust crate that is then exposed to in the end user in the Lua API. Again this is relatively easy to do directly ... some Rust functions and inject them into the Lua VM after instantiating it. In fact I started down just that read in this WIP. But it turns out this particular function was viable only because it could also be achieved on the Lua side with other dependencies. Because of the last requirement in this mess...

The last requirement is that the functions written in Rust and provided into Lua also need to be available if the Lua code is run from a system native Lua VM rather than the mlua based CLI wrapper. The whole thing needs to work as a Lua library module as well even without the CLI wrapper and mlua VM it provides.

In a naive attempt, I tried to simply add the module feature and export a Lua module from the existing crate lib.rs. That was a no-go for ugly linking reasons. Then I realized that was going to be a no-go anyway because of this limitation:

compile_error!("`vendored` and `module` features are mutually exclusive");

So my next move was to split it into 2 crates: one for the main app with the VM, and one that just repackages Rust functions and exports them into a Lua module. That isolates the CLI runtime with the Lua VM in one crate and a loadable Lua module in another. If running the application's Lua code base as a library loaded into Lua the Lua module will be able to provide all the expected API surface area.

So now we get to the actual question!

This layout has me creating two functions with two function signatures and hand converting the types for each side. For example in the POC of worked up, in the main crate's library we might have a function that returns a string:

pub fn demo() -> Result<String> {
    Ok("Hello from crate 1".to_string())
}

Simple enough. But we also need a similar function for the Lua module in the other crate:

use crate1;

fn demo(lua: &Lua, (): ()) -> LuaResult<LuaString> {
    let res = crate1::demo().unwrap();
    lua.create_string(res)
}

In this particular case it isn't too complex with no input argument types to convert and only a Result<String> to turn into a LuaResult<LuaString>, but I can see this getting more complex (or at least verbose) in a hurry. As soon as each function has a few different input args and possible multiple return arguments this is going to be some verbose casting boilerplate code.

Then we need to actually stuff that in a Lua module, so each function will also have to have an extra line in something like this:

#[mlua::lua_module]
fn mymodule(lua: &Lua) -> LuaResult<LuaTable> {
    let exports = lua.create_table().unwrap();
    let demo: LuaFunction = lua.create_function(demo).unwrap();
    exports.set("demo", demo)?;
    Ok(exports)
}

Is there something more ergonomic that I'm missing in any of this layout to perhaps do this in one crate, to somehow derive the right types for re-exporting to the Lua side, or automatically collate them in the module?

[khvzak]

Regarding the first point, functions like pub fn demo() -> Result<String> can be directly exported to Lua using Function::wrap(demo) OR Function::wrap_raw(demo) without writing much boilerplate since this commit.

[alerque]

Cool! One more reason to look forward to v0.10!

[alerque]

I just gave v0.10.0-beta.2 a spin, but it didn't work like you suggested. It did cleanup one line of code:

-let demo: LuaFunction = lua.create_function(demo).unwrap();
+let demo = Function::wrap(demo);

But what demo is here didn't get any less verbose. I tried passing it the fn demo() -> Result<String>, but it complained "expected function that takes 2 arguments". What it seems to want (and successfully builds with is still the messy fn demo(lua: &Lua, (): ()) -> LuaResult<LuaString> definition that gets the result from the actual Rust function, uwraps it, and converts it to a LuaString.

[khvzak]

The commit I referred to, was made after beta.2 version.
You could try master or wait for rc.1 which will be approx on next week

[alerque]

Oh my bad, somehow I assumed it was older and just didn't even think to look at the beta tag point! I'm working on an experimental branch mostly so I might as well give master a shake.

[alerque]

I gave it a shake with Git HEAD (main branch), and got a little further. However I struggled a but to figure out how it was going to handle errors and the result does not seem satisfactory:

// Can NOT be wrapped (what I actually want)
type Result<T> = anyhow::Result<T>;
fn demo() -> Result<String>

// Can NOT be wrapped (what I tried next when anyhow failed)
fn demo() -> Result<String, std::error:Error>

// Can NOT be wrapped
fn demo() -> Result<String>

// Can be wrapped (but not useful as a Rust function outside of the Lua module wrapper)
fn demo() -> Result<String, LuaError>

Am I missing something obvious here?

[khvzak]

Can you try Function::wrap_raw ?
The difference is - output type is not required to be result. if it's result, it will be converted to 2-values return (ok, err) rather than throwing an exception

[alerque]

That works to wrap fn demo() -> String, but that is not particularly useful since I need the Rust side of things to do proper error handling, not just unwrap everything. I had no luck using Function::wrap_wrap with any of the other forms that above that might be workable.

Would it be helpful to work up a MWE for this?

[khvzak]

Can you try the latest main branch with a new anyhow feature flag enabled.
It enable IntoLua implementation for anyhow::Error. Then Function::wrap_raw should work.