Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Skip AOT of Microsoft.CodeAnalysis.CSharp.dll for WASM, due to out-of-memory issues in CI #63958

Merged
merged 19 commits into from
Feb 15, 2022

Conversation

directhex
Copy link
Contributor

@directhex directhex commented Jan 18, 2022

We compile our wasm AOT tests on Helix rather than on AzDO due to build time concerns (it takes upwards of 3 hours to build the tests in serial all on one machine). However, the machines in Helix only have 8GiB of RAM, and some tests - notably those which consume Microsoft.CodeAnalysis.CSharp.dll will OOM the hardware before the test finishes building.

This PR adds that assembly to an exclude list, which should enable us to actually build/test adequately.

Closes: #61339
Closes: #51961

@dotnet-issue-labeler
Copy link

I couldn't figure out the best area label to add to this PR. If you have write-permissions please help me learn by adding exactly one area label.

@ghost ghost assigned directhex Jan 18, 2022
@dotnet dotnet deleted a comment from azure-pipelines bot Jan 24, 2022
@radical
Copy link
Member

radical commented Feb 3, 2022

Is #64294 needed for this?

@vargaz
Copy link
Contributor

vargaz commented Feb 3, 2022

Yes, without it, it will fail at runtime.

@jeffhandley jeffhandley added arch-wasm WebAssembly architecture area-Infrastructure labels Feb 4, 2022
@ghost
Copy link

ghost commented Feb 4, 2022

Tagging subscribers to 'arch-wasm': @lewing
See info in area-owners.md if you want to be subscribed.

Issue Details

We compile our wasm AOT tests on Helix rather than on AzDO due to build time concerns (it takes upwards of 3 hours to build the tests in serial all on one machine). However, the machines in Helix only have 8GiB of RAM, and some tests - notably those which consume Microsoft.CodeAnalysis.CSharp.dll will OOM the hardware before the test finishes building.

This PR adds that assembly to an exclude list, which should enable us to actually build/test adequately.

Closes: #61339
Closes: #51961

Author: directhex
Assignees: directhex
Labels:

arch-wasm, area-Infrastructure

Milestone: -

@directhex directhex marked this pull request as ready for review February 8, 2022 15:46
@dotnet dotnet deleted a comment from azure-pipelines bot Feb 8, 2022
@radical
Copy link
Member

radical commented Feb 9, 2022

/azp run runtime-wasm

@azure-pipelines
Copy link

Azure Pipelines failed to run 1 pipeline(s).

@radical
Copy link
Member

radical commented Feb 9, 2022

@radekdoulik any idea about:

warning: unknown warning option '-Wl%2C-O0'; did you mean '-WCL4'? [-Wunknown-warning-option] [/datadisks/disk1/work/AA3B0981/w/B15B09A6/e/publish/ProxyProjectForAOTOnHelix.proj]
warning: unknown warning option '-Wl%2C-lto-O0'; did you mean '-Wlong-long'? [-Wunknown-warning-option] [/datadisks/disk1/work/AA3B0981/w/B15B09A6/e/publish/ProxyProjectForAOTOnHelix.proj]

@radical
Copy link
Member

radical commented Feb 9, 2022

/azp run runtime-wasm

@azure-pipelines
Copy link

Azure Pipelines successfully started running 1 pipeline(s).

@vargaz
Copy link
Contributor

vargaz commented Feb 9, 2022

Its from another attempt to solve this same problem:
5a078b5

../eng/testing/tests.wasm.targets
121:      <EmccLinkOptimizationFlag Condition="'$(EmccLinkOptimizationFlag)' == ''">-Oz -Wl%252C-O0 -Wl%252C-lto-O0</EmccLinkOptimizationFlag>

@directhex
Copy link
Contributor Author

Its from another attempt to solve this same problem: 5a078b5

../eng/testing/tests.wasm.targets
121:      <EmccLinkOptimizationFlag Condition="'$(EmccLinkOptimizationFlag)' == ''">-Oz -Wl%252C-O0 -Wl%252C-lto-O0</EmccLinkOptimizationFlag>

Looks like it originally came in from @radekdoulik in #53603. %252c is a double-encoded comma (i.e. %25 is % and %2c is ,) - it seems weird to do this encoding here? Is that simply a copy-paste error?

@radekdoulik
Copy link
Member

Its from another attempt to solve this same problem: 5a078b5

../eng/testing/tests.wasm.targets
121:      <EmccLinkOptimizationFlag Condition="'$(EmccLinkOptimizationFlag)' == ''">-Oz -Wl%252C-O0 -Wl%252C-lto-O0</EmccLinkOptimizationFlag>

Looks like it originally came in from @radekdoulik in #53603. %252c is a double-encoded comma (i.e. %25 is % and %2c is ,) - it seems weird to do this encoding here? Is that simply a copy-paste error?

I think back then we had these options on command line and they needed to be escaped. Now we have them in response file(s) and so we should not need to escape them anymore.

@directhex
Copy link
Contributor Author

/azp run runtime-wasm

@azure-pipelines
Copy link

Azure Pipelines successfully started running 1 pipeline(s).

@directhex
Copy link
Contributor Author

The System.Collections.Immutable failure isn't me

[19:43:35] fail: {}
                 
[19:43:35] fail: RuntimeError: memory access out of bounds
                 
[19:43:35] fail:     at <anonymous>:wasm-function[218]:0x1a7d5
                 
[19:43:35] fail:     at <anonymous>:wasm-function[1699]:0x7c4c4
                 
[19:43:35] fail:     at <anonymous>:wasm-function[1698]:0x7c469
                 
[19:43:35] fail:     at <anonymous>:wasm-function[1734]:0x7e1e1
                 
[19:43:35] fail:     at <anonymous>:wasm-function[529]:0x371e6
                 
[19:43:35] fail:     at <anonymous>:wasm-function[8374]:0x1b64a1
                 
[19:43:35] fail:     at <anonymous>:wasm-function[8373]:0x1b635e
                 
[19:43:35] fail:     at <anonymous>:wasm-function[526]:0x37141
                 
[19:43:35] fail:     at <anonymous>:wasm-function[551]:0x37781
                 
[19:43:35] fail:     at <anonymous>:wasm-function[1541]:0x75c32
                 
[19:43:35] fail:     at <anonymous>:wasm-function[1540]:0x757a0
                 
[19:43:35] fail:     at <anonymous>:wasm-function[1525]:0x745c2
                 
[19:43:35] fail:     at <anonymous>:wasm-function[1524]:0x7411c
                 
[19:43:35] fail:     at <anonymous>:wasm-function[359]:0x2710e
                 
[19:43:35] fail:     at <anonymous>:wasm-function[327]:0x240cb
                 
[19:43:35] fail:     at <anonymous>:wasm-function[379]:0x28933
                 
[19:43:35] fail:     at <anonymous>:wasm-function[223]:0x1a857
                 
[19:43:35] fail:     at <anonymous>:wasm-function[169]:0x18b9f
                 
[19:43:35] fail:     at <anonymous>:wasm-function[168]:0xbccf
                 
[19:43:35] fail:     at <anonymous>:wasm-function[7732]:0x196d0d
                 
[19:43:35] fail:     at <anonymous>:wasm-function[1900]:0x83f79
                 
[19:43:35] fail:     at <anonymous>:wasm-function[1904]:0x845e0
                 
[19:43:35] fail:     at mono_wasm_invoke_method (<anonymous>:wasm-function[118]:0x991c)
                 
[19:43:35] fail:     at Module._mono_wasm_invoke_method (./dotnet.js:12:128850)
                 
[19:43:35] fail:     at Object.t.<computed> (./dotnet.js:3:4100)
                 
[19:43:35] fail:     at _call_method_with_converted_args (./dotnet.js:3:48191)
                 
[19:43:35] fail:     at call_method (./dotnet.js:3:47309)
                 
[19:43:35] fail:     at ./dotnet.js:3:49132
                 
[19:43:35] fail:     at mono_call_assembly_entry_point (./dotnet.js:3:49260)
                 
[19:43:35] fail:     at Object.mono_run_main (./dotnet.js:3:57765)
                 
[19:43:35] fail:     at Object.init (test-main.js:214:43)
                 
[19:43:35] fail:     at Object.onDotnetReady (test-main.js:153:17)
                 
[19:43:35] fail:     at finalize_startup (./dotnet.js:3:62585)
                 
[19:43:35] fail:     at mono_wasm_after_runtime_initialized (./dotnet.js:3:59479)
                 
[19:43:35] fail:     at callRuntimeCallbacks (./dotnet.js:12:17800)
                 
[19:43:35] fail:     at postRun (./dotnet.js:12:13708)
                 
[19:43:35] fail:     at doRun (./dotnet.js:12:145047)
                 
[19:43:35] fail:     at run (./dotnet.js:12:145204)
                 
[19:43:35] fail:     at runCaller (./dotnet.js:12:144684)
                 
[19:43:35] fail:     at removeRunDependency (./dotnet.js:12:14500)
                 
[19:43:35] fail:     at receiveInstance (./dotnet.js:12:16355)
                 
[19:43:35] fail:     at receiveInstantiationResult (./dotnet.js:12:16476)
                 
[19:43:35] info: Process v8 exited with 1

@lewing
Copy link
Member

lewing commented Feb 14, 2022

@directhex #65242

@directhex
Copy link
Contributor Author

This seems to have largely passed the expected tests? runtime-wasm AOT tests all passed

@radical
Copy link
Member

radical commented Feb 14, 2022

The System.Collections.Immutable crash is #65335 . #65242 is for aot.

@radical
Copy link
Member

radical commented Feb 15, 2022

All the tests passed. Thank you, @directhex for fixing this!

@ghost ghost locked as resolved and limited conversation to collaborators Mar 17, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
arch-wasm WebAssembly architecture area-Infrastructure
Projects
None yet
6 participants