rust build on ARMv8 fails in test macro-stepping.rs, other debuginfo-gdb tests.

[vielmetti]

I built rust on a 96-core ARMv8 machine, and "make -j check" failed in one test:

test [debuginfo-gdb] debuginfo-gdb/macro-stepping.rs ... FAILED

I expected to see everything pass.

A second time through, I got this, with four failures:

thread 'main' panicked at 'Some tests failed', /home/emv/src/rust-lang/rust/src/tools/compiletest/src/main.rs:296
make: *** [tmp/check-stage2-T-aarch64-unknown-linux-gnu-H-aarch64-unknown-linux-gnu-debuginfo-gdb.ok] Error 101

test [debuginfo-gdb] debuginfo-gdb/issue13213.rs ... FAILED
test [debuginfo-gdb] debuginfo-gdb/cross-crate-type-uniquing.rs ... FAILED
test [debuginfo-gdb] debuginfo-gdb/cross-crate-spans.rs ... FAILED
test [debuginfo-gdb] debuginfo-gdb/macro-stepping.rs ... FAILED
test result: FAILED. 95 passed; 4 failed; 6 ignored; 0 measured

Meta

$ ./aarch64-unknown-linux-gnu/stage2/bin/rustc --version --verbose
rustc 1.14.0-dev (6dc035ed9 2016-10-15)
binary: rustc
commit-hash: 6dc035ed911672c6a1f7afc9eed15fb08e574e5b
commit-date: 2016-10-15
host: aarch64-unknown-linux-gnu
release: 1.14.0-dev

% uname -a
Linux armv8hello.local.lan 4.4.0-38-generic #57-Ubuntu SMP Wed Sep 7 10:19:14 UTC 2016 aarch64 aarch64 aarch64 GNU/Linux

[    0.000000] Linux version 4.4.0-38-generic (buildd@bos01-arm64-006) (gcc version 5.4.0 20160609 (Ubuntu/Linaro 5.4.0-6ubuntu1~16.04.2) ) #57-Ubuntu SMP Wed Sep 7 10:19:14 UTC 2016 (Ubuntu 4.4.0-38.57-generic 4.4.19)

[vielmetti]

This might be related to #36323

[vielmetti]

I'm not an expert here (on gdb or on rust), but if someone wants to get access to this hardware I'm happy to make an introduction, it's currently in beta test.

[alexcrichton]

Thanks for the report! Could you also gist the full build logs of the failed tests as well? I don't think anyone's ever run the full test suite on armv8 yet so this may just be a standard "need to do some porting" failure.

[vielmetti]

Sure! Four build logs here.

https://gist.github.com/94ae1bb0318aba060c7b9a39c46c8f7b
https://gist.github.com/893e2f1cb7b7b361beaa087a709f83f2
https://gist.github.com/ec3451e365c77dfb9a32ad12e0a031c8
https://gist.github.com/57d08cf8d298d0859c08e9ac94fa3c0b

[jacobsmith928]

@alexcrichton happy to give you access to a machine for testing, just let me know! Thanks @vielmetti

[alexcrichton]

Looks like the primarily failing test is macro-stepping.rs (all of those logs have this failing test):

---- [debuginfo-gdb] debuginfo-gdb/macro-stepping.rs stdout ----
    NOTE: compiletest thinks it is using GDB version 7.11

error: line not found in debugger output: [...]#loc6[...]
status: exit code: 0
command: gdb -quiet -batch -nx -command=aarch64-unknown-linux-gnu/test/debuginfo-gdb/macro-stepping.debugger.script
stdout:
------------------------------------------
GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.04) 7.11.1
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "aarch64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word".
Breakpoint 1 at 0x19dc: file /home/emv/src/rust-lang/rust/src/test/debuginfo/macro-stepping.rs, line 87.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/aarch64-linux-gnu/libthread_db.so.1".

Breakpoint 1, macro_stepping::main () at /home/emv/src/rust-lang/rust/src/test/debuginfo/macro-stepping.rs:87
87      zzz(); // #break
89      foo!(); // #loc1
#0  macro_stepping::main () at /home/emv/src/rust-lang/rust/src/test/debuginfo/macro-stepping.rs:89
89      foo!(); // #loc1
91      foo2!(); // #loc2
#0  macro_stepping::main () at /home/emv/src/rust-lang/rust/src/test/debuginfo/macro-stepping.rs:91
91      foo2!(); // #loc2
93      let x = vec![42]; // #loc3
#0  macro_stepping::main () at /home/emv/src/rust-lang/rust/src/test/debuginfo/macro-stepping.rs:93
93      let x = vec![42]; // #loc3
93      let x = vec![42]; // #loc3
#0  macro_stepping::main () at /home/emv/src/rust-lang/rust/src/test/debuginfo/macro-stepping.rs:93
93      let x = vec![42]; // #loc3
95      new_scope!(); // #loc4
#0  macro_stepping::main () at /home/emv/src/rust-lang/rust/src/test/debuginfo/macro-stepping.rs:95
95      new_scope!(); // #loc4
97      println!("Hello {}", // #loc5
#0  macro_stepping::main () at /home/emv/src/rust-lang/rust/src/test/debuginfo/macro-stepping.rs:97
97      println!("Hello {}", // #loc5
A debugging session is active.

    Inferior 1 [process 7811] will be killed.

Quit anyway? (y or n) [answered Y; input not from terminal]

------------------------------------------
stderr:
------------------------------------------

------------------------------------------

thread '[debuginfo-gdb] debuginfo-gdb/macro-stepping.rs' panicked at 'explicit panic', /home/emv/src/rust-lang/rust/src/tools/compiletest/src/runtest.rs:2372
note: Run with `RUST_BACKTRACE=1` for a backtrace.


failures:
[debuginfo-gdb] debuginfo-gdb/macro-stepping.rs

Other failures look like:

error: could not write output to [..]: No such file or directory

The former may just be a legit bug with AArch64 on our end (or LLVM's) and the latter may just be a bug in the build system or something like that maybe? Does the machine here have anything odd with its filesystem that may cause that to arise?

[vielmetti]

The file system it is using is an iscsi mount. I will run tests again on a local file system.

[vielmetti]

I'm able to replicate a failing test on a different file system. I captured a gist with RUST_BACKTRACE=1 to show more of what is going on.

https://gist.github.com/a0304ca94a51fd55f4853f609db65d67

[alexcrichton]

Ah ok looks like those filesystem errors aren't related to any weird mount issues or anything like that.

@michaelwoerister do you thoughts about the errors here? Either the what appears to be legit error or the filesystem issues?

[vielmetti]

The next thing I'm going to try is "make check" instead of "make -j check" just in case there are parallelism issues in the makefile.

[michaelwoerister]

Off the top of my head I can't say what the problem could be. The file system errors occur during builds of auxiliary libraries which is something that many other kinds of tests do too.

As for the stepping, it might be a problem with the aarch64 code generator? I'll try if I can reproduce the problem.

[vielmetti]

The only thing that comes to mind (which we ran into in the Go language port) was PAGE_SIZE variances among aarch64 systems. A little bit of detail is here

https://marcin.juszkiewicz.com.pl/2015/09/14/from-a-diary-of-aarch64-porter-page_size/

I'm stabbing in the dark here, but that said, this seems to be a common portability issue with aarch64.

[jacobsmith928]

@michaelwoerister nice to see a fellow NYC infrastructure-r on this thread. If you need any access to ARMv8 test boxes, we can give you access to our on demand beta, just email me jacob at packet dot net.

[michaelwoerister]

@jacobsmith928 Thanks! I recently got an ODROID C2 which should have the right architecture, right?

[vielmetti]

@michaelwoerister One of the portability issues with aarch64 code mentioned above is how to handle PAGE_SIZE, which different operating systems set differently (notably Fedora and Ubuntu do not set it the same, Fedora uses 64k pages and Ubuntu uses 4k pages). So to do a proper test you want to make sure that you cross-test between the two systems. (I don't know if that's the particular problem here.)

[jacobsmith928]

@michaelwoerister the ODROID C2 architecture is probably just fine, but if you want something more powerful or just some testing diversity, our machines have 2 x 48 core cavium chips and 128 GB of RAM.

[infinity0]

Hey, if it helps also feel free to look through the Debian build logs - click on "Maybe-Successful" or "Maybe-Failed" for the full logs. For each build in that table, the machine that performed the build is listed in the "Builder" column and you can look up its details here. For example, arm-arm-04.debian.org built our first successful arm64 rustc builds back in July.

[infinity0]

Looks like we're getting this in Debian for 1.13.0 stable too: https://buildd.debian.org/status/fetch.php?pkg=rustc&arch=arm64&ver=1.13.0%2Bdfsg1-1&stamp=1479008240

[lucab]

Bump. Confirming that this bug is breaking the testsuite of 1.13.0 on debian arm64 autobuilders (full log in the link above).

It looks like the frame info in last step is missing the final call:

#0  macro_stepping::main () at /«BUILDDIR»/rustc-1.13.0+dfsg1/src/test/debuginfo/macro-stepping.rs:97
97	    println!("Hello {}", // #loc5
A debugging session is active.

	Inferior 1 [process 7030] will be killed.
error: line not found in debugger output: [...]#loc6[...]

I noticed that the lldb test doesn't actually check for #loc6, but I don't have enough context to understand the reason.

/cc @vadimcn who introduced it in #35238

[vadimcn]

I skipped #loc6 for LLDB because Xcode LLDB (and we only test with LLDB on OSX) seems to have a bug in this case. (I think it's an LLDB bug because it also reproes with an equivalent C program).
I've reported this to Apple, and they dupped it against some other bug, but, in a typical Apple fashion, they gave no indication of what is causing the problem or when it might be fixed.

Linux LLDB 4.0 and GDB seemed to work fine so far...

[michaelwoerister]

@rust-lang/tools Since aarch64 is a tier 2 platform, does anybody object to just ignore this test case there? It looks like an unfortunate interaction between LLVM codegen and debuggers, I don't know if there's much we can do on our side.

[alexcrichton]

@michaelwoerister sounds good to me, yeah.