Support for --jvm option in Enso runner #10374

JaroslavTulach · 2024-06-26T10:34:08Z

Pull Request Description

Addresses one of two concerns of #5298 - adds support for --jvm argument to allow us to switch from native image built Enso binary (as developed by #10126) to regular JVM based Enso execution. This change doesn't affect production builds. The native executable continues to be only built by engine-runner/buildNativeImage which is tested on CI, but not in the production jobs.

Typical Usecase

With e67cc87 one can build:

sbt:enso> buildEngineDistribution
sbt:enso> engine-runner/buildNativeImage

and then the bin/enso launcher is going to be a native executable (of a huge size):

enso$ ls built-distribution/enso-engine-*linux-*/enso-*/bin/enso -hl
278M

The bin/enso file is native executable with built in support for executing Standard.Base library (because of #10296). However, should there be an Enso file short.enso like

from Standard.Base import IO

polyglot java import java.lang.Short

main =
    IO.println <| Short.valueOf "445"

then executing it with prebuilt native executable fails because it imports some class not used by Standard.Base and thus not compiled into the executable:

enso$ ./built-distribution/enso-engine-*linux-*/enso-*/bin/enso --run short.enso 
Execution finished with an error: Method `valueOf` of type java.lang.Short could not be found.
        at <enso> hi.main<arg-1>(hi.enso:6:19-37)
        at <enso> hi.main(hi.enso:6:5-37)

at that moment one can use the --jvm attribute and then the execution succeeds:

enso$ ./built-distribution/enso-engine-*linux-*/enso-*/bin/enso --jvm --run /home/devel/hi.enso 
445

Native executable gives us instant startup. The --jvm switch gives us compatibility with the JVM mode.

Checklist

Please ensure that the following checklist has been satisfied before submitting the PR:

The documentation has been updated, if necessary.
Screenshots/screencasts have been attached, if there are any visual changes. For interactive or animated visual changes, a screencast is preferred.
All code follows the
Scala,
Java,
TypeScript,
and
Rust
style guides. In case you are using a language not listed above, follow the Rust style guide.
Unit tests have been written where possible.
Verify the launcher works on other OSes

JaroslavTulach · 2024-06-26T10:43:27Z

engine/runner/src/main/java/org/enso/runner/Main.java

+ var loc = Main.class.getProtectionDomain().getCodeSource().getLocation();
+ println("Location: " + loc);
+ var commandAndArgs = new ArrayList<String>();
+ if (jvm == null) {


The JVM_OPTION takes zero or one argument. If the argument is present, it should be a path to the GraalVM to use to execute Enso process. The code below uses ProcessBuilder to construct such process - more or less the code is a copy of what is in bin/enso and bin/enso.bat files.

When the --jvm option is used without additional argument, then we should somehow locate the GraalVM in the installation. CCing @4e6 and @radeusgd (as author of enso launcher which deals with similar issues) to guide this part of the implementation.

engine/runner/src/main/java/org/enso/runner/Main.java

…linux-amd64/enso-0.0.0-dev/bin/enso on Linux

enso-bot · 2024-06-27T03:05:03Z

Jaroslav Tulach reports a new STANDUP for yesterday (2024-06-26):

Progress: - create PR for the --jvm argument: #10374

iterate runtimes() to find a JVM: https:/enso-org/enso/pull/10374/files#r1654685959
fighting with sbt: https://discord.com/channels/@me/948519891961544714/1255529502411591783
got help from Dmitry: Support for --jvm option in Enso runner #10374
Linux is working
verifying Mac OS X
review IdMap: Persist a subset of IdMap #10347 (review)
standups, meetings, retros It should be finished by 2024-07-03.

GitHub
Support for --jvm option in Enso launcher by JaroslavTulach · Pull Request #10374 · enso-org/enso
Pull Request Description Addresses one of two concerns of #5298 - adds support for --jvm argument to allow us to switch from native image built Enso binary (as developed by #10126) to regular JVM b...

Discord
Discord - Group Chat That’s All Fun & Games
Discord is great for playing games and chilling with friends, or even building a worldwide community. Customize your own space to talk, play, and hang out.

JaroslavTulach · 2024-06-27T07:53:32Z

The current status. enso or enso.exe is being generated into ./built-distribution/enso-engine-*/enso-*/bin/ directory on all operating systems Linux, Mac and Windows.

Why I run a simple program via ./bin/enso or .\bin\enso.exe I get:

java.lang.UnsupportedOperationException: Preview Features not enabled, need to run with --enable-preview
        at [email protected]/jdk.internal.misc.PreviewFeatures.ensureEnabled(PreviewFeatures.java:49)
        at [email protected]/java.lang.Thread.ofVirtualWithoutLoom(Thread.java:680)
        at [email protected]/java.util.concurrent.Executors.newVirtualThreadPerTaskExecutor(Executors.java:269)
        at org.enso.downloader.http.HTTPDownload$.asyncDownload(HTTPDownload.scala:82)
        at org.enso.downloader.http.HTTPDownload$.fetchString(HTTPDownload.scala:53)
        at org.enso.editions.updater.EditionUpdater.$anonfun$downloadEditionRepositoryManifest$1(EditionUpdater.scala:70)
        at scala.util.Try$.apply(Try.scala:210)
        at org.enso.editions.updater.EditionUpdater.downloadEditionRepositoryManifest(EditionUpdater.scala:67)
        at org.enso.editions.updater.EditionUpdater.$anonfun$fetchManifests$4(EditionUpdater.scala:106)

what can be wrong?

When I use the --jvm option, bin\enso.exe.exe --jvm --run f.enso then it cannot find libraries:

[TRACE] [2024-06-27T09:51:40.293] [enso.org.enso.interpreter.EnsoLanguage] Detected text mode, using a standalone lock manager.
[DEBUG] [2024-06-27T09:51:40.293] [org.enso.distribution.DistributionManager] Detected paths: DistributionPaths(
  dataRoot = C:\Users\Jaroslav Tulach\AppData\Local\enso,
  runtimes = C:\Users\Jaroslav Tulach\AppData\Local\enso\runtime,
  engines  = C:\Users\Jaroslav Tulach\AppData\Local\enso\dist,
  bundle   = None,
  config   = C:\Users\Jaroslav Tulach\AppData\Local\enso\config,
  locks    = C:\Users\Jaroslav Tulach\AppData\Local\enso\lock,
  logs     = C:\Users\Jaroslav Tulach\AppData\Local\enso\log,
  tmp      = C:\Users\Jaroslav Tulach\AppData\Local\enso\tmp,
  profiling = C:\Users\Jaroslav Tulach\AppData\Local\enso\profiling,
  ensoHome  = C:\Users\Jaroslav Tulach\enso,
  customEditions = List(C:\Users\Jaroslav Tulach\enso\editions),
  localLibrariesSearchpaths = List(C:\Users\Jaroslav Tulach\enso\libraries)
)
[DEBUG] [2024-06-27T09:51:40.464] [org.enso.distribution.config.GlobalConfigurationManager] Global config [C:\Users\Jaroslav Tulach\AppData\Local\enso\config\global-config.yaml] 
not found, falling back to defaults.
[TRACE] [2024-06-27T09:51:40.664] [org.enso.librarymanager.LibraryLocations] Local library search paths = List(C:\Users\Jaroslav Tulach\enso\libraries)
[TRACE] [2024-06-27T09:51:40.664] [org.enso.librarymanager.LibraryLocations] Primary library cache = C:\Users\Jaroslav Tulach\AppData\Local\enso\lib
[TRACE] [2024-06-27T09:51:40.664] [org.enso.librarymanager.LibraryLocations] Auxiliary (bundled) library caches = List()
[DEBUG] [2024-06-27T09:51:40.737] [enso.org.enso.compiler.Compiler] Initialising IR for [Standard.Builtins.Main].
[TRACE] [2024-06-27T09:51:40.842] [enso.org.enso.interpreter.caches.Cache] Loaded cache for Standard.Builtins.Main with 222 bytes in 93 ms
[TRACE] [2024-06-27T09:51:40.843] [enso.org.enso.interpreter.caches.Cache] Using cache for [Standard.Builtins.Main at location [C:\Users\Jaroslav Tulach\AppData\Local\enso\cache\ir\Standard\Builtins\0.0.0-dev\0.0.0-dev\Standard\Builtins].
[DEBUG] [2024-06-27T09:51:40.844] [enso.org.enso.interpreter.runtime.SerializationPool] Restored IR from cache for module [Standard.Builtins.Main] at stage [AFTER_STATIC_PASSES].[DEBUG] [2024-06-27T09:51:40.846] [enso.org.enso.interpreter.runtime.SerializationPool] Deserializing module Standard.Builtins.Main from IR file: true
[DEBUG] [2024-06-27T09:51:40.848] [enso.org.enso.compiler.Compiler] Parsing module [f].
[DEBUG] [2024-06-27T09:51:40.848] [enso.org.enso.interpreter.runtime.SerializationPool] Unable to load a cache for module [f].
[DEBUG] [2024-06-27T09:51:40.853] [enso.org.enso.interpreter.runtime.SerializationPool] Deserializing module f from IR file: false
[DEBUG] [2024-06-27T09:51:40.853] [enso.org.enso.compiler.Compiler] Loading module [f] from source.
[TRACE] [2024-06-27T09:51:40.897] [org.enso.interpreter.runtime.DefaultPackageRepository] Resolving library Standard.Base.
[TRACE] [2024-06-27T09:51:40.897] [org.enso.librarymanager.DefaultLibraryProvider] Resolved Standard.Base to [Right(0.0.0-dev)].
[TRACE] [2024-06-27T09:51:40.913] [org.enso.librarymanager.published.DefaultPublishedLibraryProvider] Standard.Base was not found in any caches, it will need to be downloaded.   
[TRACE] [2024-06-27T09:51:40.913] [org.enso.librarymanager.published.cache.DownloadingLibraryCache] Trying to install [Standard.Base:0.0.0-dev].
[DEBUG] [2024-06-27T09:51:40.928] [org.enso.downloader.http.HTTPDownload] Fetching [https://libraries.release.enso.org/libraries/Standard/Base/0.0.0-dev/manifest.yaml].
[INFO] [2024-06-27T09:51:41.061] [org.enso.interpreter.instrument.NotificationHandler$TextMode] Downloading library manifest of [Standard.Base].
[WARN] [2024-06-27T09:51:41.642] [org.enso.interpreter.runtime.DefaultPackageRepository] Resolution failed with [The package download has failed: The server has responded with 404 status.].
C:\enso\f.enso:1:1: error: Package containing the module Standard.Base.Main could not be loaded: The package download has failed: The server has responded with 404 status.
    1 | from Standard.Base import all
      | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
C:\enso\f.enso:4:5: error: The name `File` could not be found.
    4 |     File.new "C:\dev:a"
      |     ^~~~
Aborting due to 2 errors and 0 warnings.
org.graalvm.polyglot.PolyglotException: C:\enso\f.enso:1:1: error: Package containing the module Standard.Base.Main could not be loaded: The package download has failed: The server has responded with 404 status.
    1 | from Standard.Base import all
      | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~

These two results are from Windows and Mac - they both work on Linux - I probably have proper symlink setup there since I used it daily. But we want to get this working out of the box, without any setup, right? Anyway right now we are back in a mode when the Enso runtime tries to download something...

JaroslavTulach · 2024-06-27T14:13:07Z

Downloading Enso Bundle Linux to understand the layout of such a distribution. Áááá:

$ tar fxvz ~/Stažené/enso-bundle-2024.2.1-nightly.2024.6.24-linux-amd64.tar.gz
$ ./enso/bin/enso

isn't the engine-runner, but the launcher. Ach. ./enso/dist/2024.2.1-nightly.2024.6.24/bin/enso is the engine-runner! The layout of the Enso Bundle comes in Portable Enso Distribution Layout .

lib/scala/edition-updater/src/main/scala/org/enso/editions/updater/EditionManager.scala

enso-bot · 2024-06-28T18:49:31Z

Jaroslav Tulach reports a new STANDUP for yesterday (2024-06-27):

Progress: - integrated: #10380

discussing eta conversion: https://discord.com/channels/401396655599124480/1255794325447770164/1256181798686294017
discussing Enso launcher: https://discord.com/channels/401396655599124480/1256230229505085491/1256230821157802055
recognize JAVA_OPTS: 54bb7d8
no UpdatingEditionManager: f1ad21b
standups It should be finished by 2024-07-03.

Discord
Discord - Group Chat That’s All Fun & Games
Discord is great for playing games and chilling with friends, or even building a worldwide community. Customize your own space to talk, play, and hang out.

…5298

JaroslavTulach · 2024-07-05T05:24:55Z

We still have one failure:

DEBUG main_internal:Running process.{command=Command:
	ENSO_BENCHMARK_TEST_DRY_RUN="True" "/usr/bin/bash" "/runner/_work/enso/enso/built-distribution/enso-engine-2024.3.1-dev-linux-amd64/enso-2024.3.1-dev/bin/enso" "--jvm" "--run" "/runner/_work/enso/enso/test/Benchmarks"
	with environment overrides:
		ENSO_BENCHMARK_TEST_DRY_RUN=True}: ide_ci::program::command: Spawning.
TRACE ide_ci::program::command: bashℹ️ <START>
TRACE ide_ci::program::command: bash⚠️ <START>
 INFO ide_ci::program::command: bash⚠️ /runner/_work/enso/enso/built-distribution/enso-engine-2024.3.1-dev-linux-amd64/enso-2024.3.1-dev/bin/enso: /runner/_work/enso/enso/built-distribution/enso-engine-2024.3.1-dev-linux-amd64/enso-2024.3.1-dev/bin/enso: cannot execute binary file
TRACE ide_ci::program::command: bashℹ️ <ENDUT>
TRACE ide_ci::program::command: bash⚠️ <ENDUT>
 INFO main_internal:Running process.{command=Command:
	ENSO_BENCHMARK_TEST_DRY_RUN="True" "/usr/bin/bash" "/runner/_work/enso/enso/built-distribution/enso-engine-2024.3.1-dev-linux-amd64/enso-2024.3.1-dev/bin/enso" "--jvm" "--run" "/runner/_work/enso/enso/test/Benchmarks"
	with environment overrides:
		ENSO_BENCHMARK_TEST_DRY_RUN=True pid=3879376 status=126}: ide_ci::program::command: close
ERROR main_internal: enso_build_cli: error=Command failed: Command:
	ENSO_BENCHMARK_TEST_DRY_RUN="True" "/usr/bin/bash" "/runner/_work/enso/enso/built-distribution/enso-engine-2024.3.1-dev-linux-amd64/enso-2024.3.1-dev/bin/enso" "--jvm" "--run" "/runner/_work/enso/enso/test/Benchmarks"
	with environment overrides:
		ENSO_BENCHMARK_TEST_DRY_RUN=True
 INFO main_internal: enso_build_cli: close
Error: Command failed: Command:
	ENSO_BENCHMARK_TEST_DRY_RUN="True" "/usr/bin/bash" "/runner/_work/enso/enso/built-distribution/enso-engine-2024.3.1-dev-linux-amd64/enso-2024.3.1-dev/bin/enso" "--jvm" "--run" "/runner/_work/enso/enso/test/Benchmarks"
	with environment overrides:
		ENSO_BENCHMARK_TEST_DRY_RUN=True

Caused by:
    process exited unsuccessfully: exit status: 126

probably fixed by 59f3087 as we cannot wrap binary file bin/enso into bash script wrapper.

build/build/src/enso.rs

hubertp

Approving tentatively but documentation for native-image has become significantly outdated. Please make sure it is in line with your recent changes.

build.sbt

project/NativeImage.scala

build/build/src/engine/context.rs

build/build/src/enso.rs

Co-authored-by: Kaz Wesley <[email protected]>

enso-bot · 2024-07-06T06:43:13Z

Jaroslav Tulach reports a new 🔴 DELAY for yesterday (2024-07-05):

Summary: There is 5 days delay in implementation of the Support for --jvm option in Enso runner (#10374) task.
It will cause 5 days delay for the delivery of this weekly plan.

Already being integrated

Delay Cause: I was on vacation mostly, working intermittently.

Possible solutions: Just integrate

enso-bot · 2024-07-06T06:43:22Z

Jaroslav Tulach reports a new STANDUP for yesterday (2024-07-05):

Progress: - discussing ensoup: https://discord.com/channels/401396655599124480/1256230229505085491/1257597154906607688

processing emails
closing align Truffle launcher PR: Align Enso launcher with Truffle's AbstractLanguageLauncher #10335 (comment)
merged Disable UpdatingEditionProvider when running Enso CLI #10408
updated Support for --jvm option in Enso runner #10374 (comment)
discussions
fixing the native executable location on Windows: 13cb3d4
removal of runner: 3fb6a1b
build script fighting: Support for --jvm option in Enso runner #10374 (comment)
adressing review comments: 1ec943a
PR ready to merge It should be finished by 2024-07-08.

Discord
Discord - Group Chat That’s All Fun & Games
Discord is great for playing games and chilling with friends, or even building a worldwide community. Customize your own space to talk, play, and hang out.

project/NativeImage.scala

Support for --jvm option in Enso launcher

43929db

JaroslavTulach requested a review from radeusgd June 26, 2024 10:34

JaroslavTulach self-assigned this Jun 26, 2024

JaroslavTulach requested review from 4e6, hubertp and Akirathan as code owners June 26, 2024 10:34

JaroslavTulach marked this pull request as draft June 26, 2024 10:34

JaroslavTulach commented Jun 26, 2024

View reviewed changes

Iterate installed GraalVM runtimes

7d31eab

JaroslavTulach commented Jun 26, 2024

View reviewed changes

engine/runner/src/main/java/org/enso/runner/Main.java Show resolved Hide resolved

4e6 and others added 2 commits June 26, 2024 15:53

feat: engineDistributionRoot parameter

67f1632

Generate NI executable as ./built-distribution/enso-engine-0.0.0-dev-…

e67cc87

…linux-amd64/enso-0.0.0-dev/bin/enso on Linux

JaroslavTulach marked this pull request as ready for review June 26, 2024 17:36

JaroslavTulach added 2 commits June 28, 2024 17:24

Ignore --jvm when already in the right JVM

8f18b2f

Recognize JAVA_OPTS environment variable and pass it to the JVM

54bb7d8

JaroslavTulach force-pushed the wip/jtulach/JvmOpt5298 branch from 7a7c0b5 to 54bb7d8 Compare June 28, 2024 17:16

JaroslavTulach added 2 commits June 28, 2024 19:36

Let EditionManager work with any EditionProvider, not just updating one

ec51515

Don't use UpdatingEditionManager in regular execution

f1ad21b

JaroslavTulach commented Jun 28, 2024

View reviewed changes

lib/scala/edition-updater/src/main/scala/org/enso/editions/updater/EditionManager.scala Outdated Show resolved Hide resolved

Derive languageHome from code location

cfedcb6

JaroslavTulach requested review from jdunkerley, GregoryTravis, AdRiley and marthasharkey as code owners June 29, 2024 05:38

Search for the parser in component subdirectory

2f3e832

JaroslavTulach requested a review from farmaazon as a code owner June 29, 2024 06:05

GregoryTravis approved these changes Jul 1, 2024

View reviewed changes

JaroslavTulach mentioned this pull request Jul 2, 2024

Align Enso launcher with Truffle's AbstractLanguageLauncher #10335

Closed

3 tasks

JaroslavTulach added 4 commits July 3, 2024 07:33

Merge remote-tracking branch 'origin/develop' into wip/jtulach/JvmOpt…

fa7a62d

…5298

native-image needs location without .exe on Windows

13cb3d4

There is no runner anymore

3fb6a1b

Use .exe in native runner tests

8e4834e

JaroslavTulach added 2 commits July 5, 2024 07:45

run_benchmarks without bash script wrapper

59f3087

Supressing question mark operator is useless here warning

83f31be

JaroslavTulach commented Jul 5, 2024

View reviewed changes

build/build/src/enso.rs Outdated Show resolved Hide resolved

jdunkerley approved these changes Jul 5, 2024

View reviewed changes

hubertp approved these changes Jul 5, 2024

View reviewed changes

build.sbt Show resolved Hide resolved

project/NativeImage.scala Outdated Show resolved Hide resolved

hubertp added the CI: Clean build required CI runners will be cleaned before and after this PR is built. label Jul 5, 2024

kazcw reviewed Jul 5, 2024

View reviewed changes

build/build/src/engine/context.rs Outdated Show resolved Hide resolved

build/build/src/engine/context.rs Outdated Show resolved Hide resolved

build/build/src/enso.rs Outdated Show resolved Hide resolved

build/build/src/enso.rs Outdated Show resolved Hide resolved

JaroslavTulach and others added 7 commits July 6, 2024 07:56

Updating documentation to new location of runner

1ec943a

Just return benchmarks

13c9518

Co-authored-by: Kaz Wesley <[email protected]>

No need to disable linter

ffb2fb8

Co-authored-by: Kaz Wesley <[email protected]>

Use with_executable_extension

60595ba

Co-authored-by: Kaz Wesley <[email protected]>

Use with_executable_extension II

49700cd

Co-authored-by: Kaz Wesley <[email protected]>

cargo fmt

2bd08b1

Using longer local variable name

bbbbf25

JaroslavTulach added the CI: Ready to merge This PR is eligible for automatic merge label Jul 6, 2024

mergify bot merged commit 515d823 into develop Jul 6, 2024
42 checks passed

mergify bot deleted the wip/jtulach/JvmOpt5298 branch July 6, 2024 07:02

JaroslavTulach mentioned this pull request Jul 8, 2024

Avoid name duplication of the enso universal launcher #10476

Closed

somebody1234 reviewed Jul 12, 2024

View reviewed changes

project/NativeImage.scala Show resolved Hide resolved

somebody1234 reviewed Jul 12, 2024

View reviewed changes

project/NativeImage.scala Show resolved Hide resolved

JaroslavTulach mentioned this pull request Jul 19, 2024

Prevent re-entrant execution of finalizers #10602

Merged

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support for --jvm option in Enso runner #10374

Support for --jvm option in Enso runner #10374

JaroslavTulach commented Jun 26, 2024 •

edited

Loading

JaroslavTulach Jun 26, 2024

enso-bot bot commented Jun 27, 2024 •

edited by unfurl-links bot

Loading

JaroslavTulach commented Jun 27, 2024 •

edited

Loading

JaroslavTulach commented Jun 27, 2024 •

edited

Loading

enso-bot bot commented Jun 28, 2024 •

edited by unfurl-links bot

Loading

JaroslavTulach commented Jul 5, 2024 •

edited

Loading

hubertp left a comment

enso-bot bot commented Jul 6, 2024

enso-bot bot commented Jul 6, 2024 •

edited by unfurl-links bot

Loading

Support for --jvm option in Enso runner #10374

Support for --jvm option in Enso runner #10374

Conversation

JaroslavTulach commented Jun 26, 2024 • edited Loading

Pull Request Description

Typical Usecase

Checklist

JaroslavTulach Jun 26, 2024

Choose a reason for hiding this comment

enso-bot bot commented Jun 27, 2024 • edited by unfurl-links bot Loading

JaroslavTulach commented Jun 27, 2024 • edited Loading

JaroslavTulach commented Jun 27, 2024 • edited Loading

enso-bot bot commented Jun 28, 2024 • edited by unfurl-links bot Loading

JaroslavTulach commented Jul 5, 2024 • edited Loading

hubertp left a comment

Choose a reason for hiding this comment

enso-bot bot commented Jul 6, 2024

enso-bot bot commented Jul 6, 2024 • edited by unfurl-links bot Loading

JaroslavTulach commented Jun 26, 2024 •

edited

Loading

enso-bot bot commented Jun 27, 2024 •

edited by unfurl-links bot

Loading

JaroslavTulach commented Jun 27, 2024 •

edited

Loading

JaroslavTulach commented Jun 27, 2024 •

edited

Loading

enso-bot bot commented Jun 28, 2024 •

edited by unfurl-links bot

Loading

JaroslavTulach commented Jul 5, 2024 •

edited

Loading

enso-bot bot commented Jul 6, 2024 •

edited by unfurl-links bot

Loading