[mono] Failed to load AOT module "Methodical_*"

[matouskozak]

Build Information

Build: https://dev.azure.com/dnceng-public/public/_build/results?buildId=751577
Build error leg or test failing: Methodical_[r,d][o,1]
Affected CI: linux_x64_release_allsubsets_mono_minifullaot_runtimetests (runtime-extra-platforms)
Range of commits: b252fa5...9477e94

Steps to reproduce locally:

1. ./build.sh -s mono+libs+clr.hosts -c Debug
2. ./src/tests/build.sh -mono debug -tree:JIT/Methodical/ /p:LibrariesConfiguration=Debug
3. ./src/tests/build.sh -log:MonoAot -mono_fullaot debug -tree:JIT/Methodical/ /p:LibrariesConfiguration=Debug /p:RuntimeVariant=minifullaot
4. MONO_ENV_OPTIONS=--full-aot bash ./artifacts/tests/coreclr/linux.x64.Debug/JIT/Methodical/Methodical_do/Methodical_do.sh -coreroot=$PWD/artifacts/tests/coreclr/linux.x64.Debug/Tests/Core_Root

Error Message

Fill the error message using step by step known issues guidance.

{
  "ErrorPattern": "Failed to load AOT module '.*\\/JIT\\/Methodical\\/Methodical_[r,d][o,1]\\/Methodical_[r,d][o,1]\\.dll\\.so' \\('.*\\/JIT\\/Methodical\\/Methodical_[r,d][o,1]\\/Methodical_[r,d][o,1]\\.dll'\\) in aot-only mode\\.",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=751577
Error message validated: [Failed to load AOT module '.*\/JIT\/Methodical\/Methodical_[r,d][o,1]\/Methodical_[r,d][o,1]\.dll\.so' \('.*\/JIT\/Methodical\/Methodical_[r,d][o,1]\/Methodical_[r,d][o,1]\.dll'\) in aot-only mode\.]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 7/24/2024 12:41:43 PM UTC

Report

Build	Definition	Test	Pull Request
772314	dotnet/runtime	Methodical_ro.WorkItemExecution
761344	dotnet/runtime	Methodical_ro.WorkItemExecution
758343	dotnet/runtime	Methodical_ro.WorkItemExecution
757260	dotnet/runtime	Methodical_ro.WorkItemExecution
756458	dotnet/runtime	Methodical_ro.WorkItemExecution
755747	dotnet/runtime	Methodical_ro.WorkItemExecution
751577	dotnet/runtime	Methodical_ro.WorkItemExecution

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
1	1	7

[matouskozak]

Could the regression be introduced by #105156 @steveisok ? We've seen the same failure on the Bump LLVM 19 PR #103585 (comment) but since it's failing on fullAOT-mini job I suspect it's likely not related...

[steveisok]

Could the regression be introduced by #105156 @steveisok ? We've seen the same failure on the Bump LLVM 19 PR #103585 (comment) but since it's failing on fullAOT-mini job I suspect it's likely not related...

I don't think it's likely. The change in #105156 was on the AOT compiler side and specific to marshaling COM interop types. It went from the AOT compiler exiting with a not supported message to emitting an exception instead.

[steveisok]

That means the test would have to have failed before, but just in a different way. That does not seem to be the case.

[matouskozak]

That means the test would have to have failed before, but just in a different way. That does not seem to be the case.

Is there something else from this range b252fa5...9477e94 that strikes you as a possible cause for this?

[steveisok]

Is there something else from this range b252fa5...9477e94 that strikes you as a possible cause for this?

No, I suspect it's not related to that commit range. Can you check to see if it happens every time now?

[matouskozak]

Is there something else from this range b252fa5...9477e94 that strikes you as a possible cause for this?

No, I suspect it's not related to that commit range. Can you check to see if it happens every time now?

yes, last 3 rolling builds of runtime-extra-platforms

[matouskozak]

I bisected the range and seems to be caused by the LLVM bump #103585. Radek had an idea that perhaps we default to use the new clang 19 shipped with the LLVM for AOT compilation...

[steveisok]

@matouskozak can you reproduce locally and get more info from the backtrace? The logs on CI aren't very helpful.

[matouskozak]

@matouskozak can you reproduce locally and get more info from the backtrace? The logs on CI aren't very helpful.

I believe that @BrzVlad has a fixed for these already based on #103585 (comment). It looks like the way the tests were disabled is not the correct way for Mono fullAOT.

[steveisok]

Ok, I'm going to proactively close this issue then.