progress view shows many AsyncLiveExpression refresh messages and doesn't respond anymore

[martinlippert]

After starting up my STS4 4.11.0 installation with an existing workspace (from an internal project), the workspace comes up, but the progress view fills up with AsyncLiveExpression refresh items and the IDE is not very responsive anymore. CPU consumption is high.

The thread dump reveals many threads in this state:

"Worker-11: AsyncLiveExpression refresh" #64 prio=5 os_prio=31 cpu=74908.87ms elapsed=77.19s tid=0x00007ff133808200 nid=0x19f3b runnable  [0x000070000fbf0000]
   java.lang.Thread.State: RUNNABLE
	at org.eclipse.jdt.internal.compiler.util.SimpleLookupTable.get(SimpleLookupTable.java:74)
	at org.eclipse.jdt.internal.core.builder.ClasspathJar.findPackageSet(ClasspathJar.java:86)
	at org.eclipse.jdt.internal.core.builder.ClasspathJar.scanContent(ClasspathJar.java:361)
	at org.eclipse.jdt.internal.core.builder.ClasspathJar.listPackages(ClasspathJar.java:418)
	at org.eclipse.jdt.internal.core.search.matching.JavaSearchNameEnvironment.indexPackageNames(JavaSearchNameEnvironment.java:223)
	at org.eclipse.jdt.internal.core.search.matching.JavaSearchNameEnvironment.computeClasspathLocations(JavaSearchNameEnvironment.java:202)
	at org.eclipse.jdt.internal.core.search.matching.JavaSearchNameEnvironment.<init>(JavaSearchNameEnvironment.java:93)
	at org.eclipse.jdt.internal.core.search.matching.IndexBasedJavaSearchEnvironment.create(IndexBasedJavaSearchEnvironment.java:361)
	at org.eclipse.jdt.internal.core.search.matching.MatchLocator.initialize(MatchLocator.java:1222)
	at org.eclipse.jdt.internal.core.search.matching.MatchLocator.locateMatches(MatchLocator.java:1260)
	at org.eclipse.jdt.internal.core.search.matching.MatchLocator.locateMatches(MatchLocator.java:1373)
	at org.eclipse.jdt.internal.core.search.matching.MatchLocator.locateMatches(MatchLocator.java:1515)
	at org.eclipse.jdt.internal.core.search.JavaSearchParticipant.locateMatches(JavaSearchParticipant.java:135)
	at org.eclipse.jdt.internal.core.search.BasicSearchEngine.findMatches(BasicSearchEngine.java:250)
	at org.eclipse.jdt.internal.core.search.BasicSearchEngine.search(BasicSearchEngine.java:601)
	at org.eclipse.jdt.core.search.SearchEngine.search(SearchEngine.java:670)
	at org.eclipse.jdt.internal.debug.ui.launcher.MainMethodSearchEngine.searchMainMethods(MainMethodSearchEngine.java:97)
	at org.springframework.ide.eclipse.boot.dash.model.BootProjectDashElement$1.compute(BootProjectDashElement.java:83)
	at org.springframework.ide.eclipse.boot.dash.model.BootProjectDashElement$1.compute(BootProjectDashElement.java:1)
	at org.springsource.ide.eclipse.commons.livexp.core.LiveExpression.refresh(LiveExpression.java:74)
	- locked <0x0000000087cfc790> (a org.springframework.ide.eclipse.boot.dash.model.BootProjectDashElement$1)
	at org.springsource.ide.eclipse.commons.livexp.core.AsyncLiveExpression.syncRefresh(AsyncLiveExpression.java:125)
	at org.springsource.ide.eclipse.commons.livexp.core.AsyncLiveExpression$1.run(AsyncLiveExpression.java:95)
	at org.eclipse.core.internal.jobs.Worker.run(Worker.java:63)

So it looks like the search for the main method is going crazy somehow. Maybe the JDT search engine is not safe to be used in this async way?

[martinlippert]

sts4-11-0-crazy-progress-view.threds.txt

[sisqo]

Same problem: lots of AsyncLiveExpression refresh and CPU is going crazy, the problem appears in Eclipse 2021-06, in Eclipse 2021-03 all seems ok...

[foal]

Is it possible to switch it off somehow?

[BoykoAlex]

Please check if the below helps at all:

Click on the filters icon in the Boot Dashboard view toolbar:

Then uncheck Hide local non-runnable apps

[foal]

I have it checked already. It doesn't help :) Anyway, it just hides some processes from the view, but I am asking about switch it off because CPU load is very high and Eclipse literally freezes for several seconds.

[BoykoAlex]

You mean unchecked? It should be unchecked not to load JDT with searches for main methods...

One more thing to check, just a hypothesis... How many editor tabs do you have open? How about closing them all except a few you currently working with... Wonder if this helps...

[BoykoAlex]

When you run jps command in the terminal... What do you get?

[foal]

You are right. I understood it wrong and had it checked. Now I uncheck it. I will see if it helps.

[sisqo]

Please check if the below helps at all:

Click on the filters icon in the Boot Dashboard view toolbar:

Then uncheck Hide local non-runnable apps

It doesn't work, even with that configuration I have still lot of AsyncLiveExpression refresh processes

[BoykoAlex]

The fact that you're seeing a lot of AsyncLiveExpression refresh processes in the progress view is OK but they should all be done quickly and not stay around for long. These are just tiny async jobs running off the UI thread to refresh something in async fashion. They have been around for many version of the STS4 tooling.
How does your thread dump looks like after unchecking Hide local non-runnable apps? Since it didn't help i'm curious to see the blocked AsyncLiveExpression threads to get a feeling what is blocking everything...
Thanks in advance for your help!

[foal]

My workspace has tens (~ 40) modules and I see the AsyncLiveExpression mostly after some refactoring. Typical view:

Maybe a lot of AsyncLiveExpression refresh processes are just a visual effect, and yes, they disappear relatively fast but I see it each time during the build several times (~ 10). By feeling the building takes more time and CPU.

[sisqo]

My workspace has tens (~ 40) modules and I see the AsyncLiveExpression mostly after some refactoring. Typical view:

Maybe a lot of AsyncLiveExpression refresh processes are just a visual effect, and yes, they disappear relatively fast but I see it each time during the build several times (~ 10). By feeling the building takes more time and CPU.

Yes, same case for me. They disappear fast but they appear continuously.

[BoykoAlex]

Looks like unchecking that filter doesn't prevent main method search jobs from being created, scheduled, executed.
Therefore, I've pushed the fix that makes:

• Hide local non-runnable apps is unchecked (false) by default
• Main method searches are not performed while Hide local non-runnable apps is off (unchecked)

I'll post the update site here once the build is done thus you can try it out and it's likely it will let you work with STS without the extra lag main method search jobs add.

Hope this will unblock you using the latest STS. Meanwhile, I'll see what can be done about main method search jobs (async live expression refresh). Hope it is just a matter of finding the proper scheduling rule for them that allows build and java indexing to go first.

Hope this sounds good to you!-)

[BoykoAlex]

Use this update site: https://dist.springsource.com/snapshot/TOOLS/sts4/nightly/e4.20 (adjust the number at the end for your Eclipse version)

After the update async live expression refreshes number will decrease almost to none if Hide local non-runnable apps is unchecked.

The problem with main method async searches (async live expression refreshes) is that there is a big number of them (for each project) sitting in the job queue scheduled in sequence not letting build jobs to succeed first. Also, we need to bound how many of these jobs executed at the same time such that there is only a bounded number of threads created for them.

[BoykoAlex]

Pushed this 7155f72 to address the issue.

1. 5 threads can be around at most to search for main methods (instead of unbound number)
2. Main methods searches are triggered far less frequently

Lets see if this helps. @sisqo @foal any chance you can try these changes out please? It is an addition to the previous change for the case of check-marked Hide local non-runnable apps

[hbrands]

@BoykoAlex May I ask: how is the MainMethod-Search supposed to work? Should it search only in the direct sources of a project at hand or will it search in transitive maven project dependencies as well (with workspace resolution enabled).
(I ask because a project A is listed in the boot dashboard in my workspace although it does not contain any main method. But it has a maven dependency to a spring boot project B that is opened in the workspace as well. If I comment out this dependency B in the POM of A then A disappears from the boot dashboard. After re-adding the dependency, A shows up again in the dashboard)
Perhaps the search can be optimized further in this regard or is the transitive search intentional?

[BoykoAlex]

@hbrands here is how main method is searched for: https:/spring-projects/sts4/blob/main/eclipse-extensions/org.springframework.ide.eclipse.boot.dash/src/org/springframework/ide/eclipse/boot/dash/model/BootProjectDashElement.java#L89
Based on the javadocs for the call SearchEngine.createJavaSearchScope(new IJavaElement[]{jp}, IJavaSearchScope.SOURCES) I assume that it should only search sources of the project passed and not the referenced projects because they are neither specified in the first parameter nor in the second parameter via the mask REFERENCED_PROJECTS. However, if you see something wrong with this code please do no keep it to yourself :-)
If you're to create a launch config for project A and search for main methods in the launch config dialog do you see the project's B main methods?
(I'll try to reproduce this myself but if you have sample project already that you can share please attach them here)

I suspect however that even if referenced projects are searched as well it won't have a big impact on the performance of this feature... Once java index is ready searches are lightning quick...

[BoykoAlex]

@hbrands One more thing! If the project has an existing launch config then it'll just be shown as runnable without checking for main method. Thus if project A has no main method but has a launch config it shows up for this reason.

[hbrands]

@hbrands One more thing! If the project has an existing launch config then it'll just be shown as runnable without checking for main method. Thus if project A has no main method but has a launch config it shows up for this reason.

@BoykoAlex Bingo! There was an (incomplete) LaunchConfig for this project. After deleting it, the search works as expected. Thanks for the tip. 👍

[foal]

I have installed the update and don't see the AsyncLiveExpression refresh anymore.

[martinlippert]

I have updated to the latest CI builds and haven't seen any hiccups from this anymore. Therefore, closing this as fixed now.