spurious crash when compiling cargo on dist-mips64{el}-linux

[Noratrieb]

#110721 (comment)
#112695 (comment)
#112827 (comment)
#112616 (comment)
#111850 (comment)
#112591 (comment)
#112591 (comment)

Most recent ones were on mips64 and those don't seem to date back that far, but mips64el seems to have been a problem for several months.

see https://rust-lang.zulipchat.com/#narrow/stream/131828-t-compiler/topic/MIPS.20failure.20on.20CI

[Noratrieb]

Somehow #112591 is blocked by this and cannot get past it.

[lqd]

• Rollup of 5 pull requests #113014 (comment)
• Refactor metadata emission to avoid visiting HIR #98867 (comment)
• Refactor metadata emission to avoid visiting HIR #98867 (comment)
• Normalize opaques with late-bound vars again #113108 (comment)
• Normalize opaques with late-bound vars again #113108 (comment)
• Normalize opaques with late-bound vars again #113108 (comment)

[Noratrieb]

I managed to get a backtrace by running the builder locally and then adding a bash call after the build invocation, then using gdb inside the container to rerun the command output by adding -vvv to the dist command.

gdb backtrace

#0  0x00007fffee326574 in llvm::MVT::getSizeInBits() const () from /checkout/obj/build/x86_64-unknown-linux-gnu/stage1/lib/librustc_driver-803b3a685fc1945d.so
#1  0x00007fffef074652 in llvm::SDValue::getScalarValueSizeInBits() const [clone .isra.0] () from /checkout/obj/build/x86_64-unknown-linux-gnu/stage1/lib/librustc_driver-803b3a685fc1945d.so
#2  0x00007fffef0ab57e in llvm::TargetLowering::SimplifyDemandedVectorElts(llvm::SDValue, llvm::APInt const&, llvm::APInt&, llvm::APInt&, llvm::TargetLowering::TargetLoweringOpt&, unsigned int, bool) const [clone .localalias] () from /checkout/obj/build/x86_64-unknown-linux-gnu/stage1/lib/librustc_driver-803b3a685fc1945d.so
#3  0x00007fffef0a95b7 in llvm::TargetLowering::SimplifyDemandedVectorElts(llvm::SDValue, llvm::APInt const&, llvm::APInt&, llvm::APInt&, llvm::TargetLowering::TargetLoweringOpt&, unsigned int, bool) const [clone .localalias] () from /checkout/obj/build/x86_64-unknown-linux-gnu/stage1/lib/librustc_driver-803b3a685fc1945d.so
#4  0x00007fffeeed92eb in (anonymous namespace)::DAGCombiner::SimplifyDemandedVectorElts(llvm::SDValue, llvm::APInt const&, bool) ()
   from /checkout/obj/build/x86_64-unknown-linux-gnu/stage1/lib/librustc_driver-803b3a685fc1945d.so
#5  0x00007fffeef0250a in (anonymous namespace)::DAGCombiner::visitEXTRACT_VECTOR_ELT(llvm::SDNode*) () from /checkout/obj/build/x86_64-unknown-linux-gnu/stage1/lib/librustc_driver-803b3a685fc1945d.so
#6  0x00007fffeef650b5 in (anonymous namespace)::DAGCombiner::combine(llvm::SDNode*) () from /checkout/obj/build/x86_64-unknown-linux-gnu/stage1/lib/librustc_driver-803b3a685fc1945d.so
#7  0x00007fffeef6654f in llvm::SelectionDAG::Combine(llvm::CombineLevel, llvm::AAResults*, llvm::CodeGenOpt::Level) ()
   from /checkout/obj/build/x86_64-unknown-linux-gnu/stage1/lib/librustc_driver-803b3a685fc1945d.so
#8  0x00007fffef05da86 in llvm::SelectionDAGISel::CodeGenAndEmitDAG() [clone .localalias] () from /checkout/obj/build/x86_64-unknown-linux-gnu/stage1/lib/librustc_driver-803b3a685fc1945d.so
#9  0x00007fffef0605ca in llvm::SelectionDAGISel::SelectAllBasicBlocks(llvm::Function const&) [clone .localalias] ()
   from /checkout/obj/build/x86_64-unknown-linux-gnu/stage1/lib/librustc_driver-803b3a685fc1945d.so
#10 0x00007fffef062a06 in llvm::SelectionDAGISel::runOnMachineFunction(llvm::MachineFunction&) [clone .part.0] () from /checkout/obj/build/x86_64-unknown-linux-gnu/stage1/lib/librustc_driver-803b3a685fc1945d.so
#11 0x00007ffff0b84bcf in llvm::MipsDAGToDAGISel::runOnMachineFunction(llvm::MachineFunction&) () from /checkout/obj/build/x86_64-unknown-linux-gnu/stage1/lib/librustc_driver-803b3a685fc1945d.so
#12 0x00007fffef2a52a4 in llvm::MachineFunctionPass::runOnFunction(llvm::Function&) [clone .part.0] () from /checkout/obj/build/x86_64-unknown-linux-gnu/stage1/lib/librustc_driver-803b3a685fc1945d.so
#13 0x00007ffff0288a70 in llvm::FPPassManager::runOnFunction(llvm::Function&) [clone .localalias] () from /checkout/obj/build/x86_64-unknown-linux-gnu/stage1/lib/librustc_driver-803b3a685fc1945d.so
#14 0x00007ffff0288be4 in llvm::FPPassManager::runOnModule(llvm::Module&) [clone .localalias] () from /checkout/obj/build/x86_64-unknown-linux-gnu/stage1/lib/librustc_driver-803b3a685fc1945d.so
#15 0x00007ffff02895e4 in llvm::legacy::PassManagerImpl::run(llvm::Module&) [clone .localalias] () from /checkout/obj/build/x86_64-unknown-linux-gnu/stage1/lib/librustc_driver-803b3a685fc1945d.so
#16 0x00007fffee2dd598 in LLVMRustWriteOutputFile () from /checkout/obj/build/x86_64-unknown-linux-gnu/stage1/lib/librustc_driver-803b3a685fc1945d.so
#17 0x00007fffee2a79ef in rustc_codegen_llvm::back::write::write_output_file () at compiler/rustc_codegen_llvm/src/back/write.rs:71
#18 0x00007fffee2abf88 in rustc_codegen_llvm::back::write::codegen::{closure#2} () at compiler/rustc_codegen_llvm/src/back/write.rs:749
#19 rustc_codegen_llvm::back::write::codegen::with_codegen<rustc_codegen_llvm::back::write::codegen::{closure_env#2}, core::result::Result<(), rustc_span::fatal_error::FatalError>> ()
    at compiler/rustc_codegen_llvm/src/back/write.rs:611
#20 rustc_codegen_llvm::back::write::codegen () at compiler/rustc_codegen_llvm/src/back/write.rs:748
#21 0x00007fffee1bf4d6 in rustc_codegen_llvm::{impl#3}::codegen () at compiler/rustc_codegen_llvm/src/lib.rs:233
#22 rustc_codegen_ssa::back::write::finish_intra_module_work<rustc_codegen_llvm::LlvmCodegenBackend> () at compiler/rustc_codegen_ssa/src/back/write.rs:913
#23 0x00007fffee27039f in rustc_codegen_ssa::back::write::execute_lto_work_item<rustc_codegen_llvm::LlvmCodegenBackend> () at compiler/rustc_codegen_ssa/src/back/write.rs:899
#24 rustc_codegen_ssa::back::write::spawn_work::{closure#0}<rustc_codegen_llvm::LlvmCodegenBackend> () at compiler/rustc_codegen_ssa/src/back/write.rs:1689
#25 rustc_codegen_llvm::{impl#2}::spawn_named_thread::{closure#0}<rustc_codegen_ssa::back::write::spawn_work::{closure_env#0}<rustc_codegen_llvm::LlvmCodegenBackend>, ()> ()
    at compiler/rustc_codegen_llvm/src/lib.rs:168
#26 std::sys_common::backtrace::__rust_begin_short_backtrace<rustc_codegen_llvm::{impl#2}::spawn_named_thread::{closure_env#0}<rustc_codegen_ssa::back::write::spawn_work::{closure_env#0}<rustc_codegen_llvm::LlvmCodegenBackend>, ()>, ()> () at library/std/src/sys_common/backtrace.rs:135
#27 0x00007fffee240a46 in std::thread::{impl#0}::spawn_unchecked_::{closure#1}::{closure#0}<rustc_codegen_llvm::{impl#2}::spawn_named_thread::{closure_env#0}<rustc_codegen_ssa::back::write::spawn_work::{closure_env#0}<rustc_codegen_llvm::LlvmCodegenBackend>, ()>, ()> () at library/std/src/thread/mod.rs:529
#28 core::panic::unwind_safe::{impl#23}::call_once<(), std::thread::{impl#0}::spawn_unchecked_::{closure#1}::{closure_env#0}<rustc_codegen_llvm::{impl#2}::spawn_named_thread::{closure_env#0}<rustc_codegen_ssa::back::write::spawn_work::{closure_env#0}<rustc_codegen_llvm::LlvmCodegenBackend>, ()>, ()>> () at library/core/src/panic/unwind_safe.rs:271
#29 std::panicking::try::do_call<core::panic::unwind_safe::AssertUnwindSafe<std::thread::{impl#0}::spawn_unchecked_::{closure#1}::{closure_env#0}<rustc_codegen_llvm::{impl#2}::spawn_named_thread::{closure_env#0}<rustc_codegen_ssa::back::write::spawn_work::{closure_env#0}<rustc_codegen_llvm::LlvmCodegenBackend>, ()>, ()>>, ()> () at library/std/src/panicking.rs:500
#30 std::panicking::try<(), core::panic::unwind_safe::AssertUnwindSafe<std::thread::{impl#0}::spawn_unchecked_::{closure#1}::{closure_env#0}<rustc_codegen_llvm::{impl#2}::spawn_named_thread::{closure_env#0}<rustc_codegen_ssa::back::write::spawn_work::{closure_env#0}<rustc_codegen_llvm::LlvmCodegenBackend>, ()>, ()>>> () at library/std/src/panicking.rs:464
#31 std::panic::catch_unwind<core::panic::unwind_safe::AssertUnwindSafe<std::thread::{impl#0}::spawn_unchecked_::{closure#1}::{closure_env#0}<rustc_codegen_llvm::{impl#2}::spawn_named_thread::{closure_env#0}<rustc_codegen_ssa::back::write::spawn_work::{closure_env#0}<rustc_codegen_llvm::LlvmCodegenBackend>, ()>, ()>>, ()> () at library/std/src/panic.rs:142
#32 std::thread::{impl#0}::spawn_unchecked_::{closure#1}<rustc_codegen_llvm::{impl#2}::spawn_named_thread::{closure_env#0}<rustc_codegen_ssa::back::write::spawn_work::{closure_env#0}<rustc_codegen_llvm::LlvmCodegenBackend>, ()>, ()> () at library/std/src/thread/mod.rs:528
#33 core::ops::function::FnOnce::call_once<std::thread::{impl#0}::spawn_unchecked_::{closure_env#1}<rustc_codegen_llvm::{impl#2}::spawn_named_thread::{closure_env#0}<rustc_codegen_ssa::back::write::spawn_work::{closure_env#0}<rustc_codegen_llvm::LlvmCodegenBackend>, ()>, ()>, ()> () at library/core/src/ops/function.rs:250
#34 0x00007fffecf13bbe in alloc::boxed::{impl#47}::call_once<(), dyn core::ops::function::FnOnce<(), Output=()>, alloc::alloc::Global> () at library/alloc/src/boxed.rs:1993
#35 alloc::boxed::{impl#47}::call_once<(), alloc::boxed::Box<dyn core::ops::function::FnOnce<(), Output=()>, alloc::alloc::Global>, alloc::alloc::Global> () at library/alloc/src/boxed.rs:1993
#36 std::sys::unix::thread::{impl#2}::new::thread_start () at library/std/src/sys/unix/thread.rs:108
#37 0x00007fffeccb9b43 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#38 0x00007fffecd4ba00 in ?? () from /lib/x86_64-linux-gnu/libc.so.6

I got this on bb7de68e2d432b4a695f23f1f76eeb2ad3f5c647, which reliably reproduces the crash for me.

[lqd]

cc @rust-lang/wg-llvm is the backtrace enough to understand the cause of the issue ?

[Noratrieb]

I have a single .ll file that reproduces the segfault when compiled with just llc, even outside the container with an llc i got off nixpkgs. I can't upload the file here right now since it's 416MB but I'll work on reducing it.
edit: https://cdn.discordapp.com/attachments/957720175619215380/1124044197008253080/bad-cgu.bc

[Noratrieb]

; ModuleID = 'reduced.bc'
target datalayout = "E-m:e-i8:8:32-i16:16:32-i64:64-n32:64-S128"
target triple = "mips64-unknown-linux-gnuabi64"

define internal fastcc void @uwu(<5 x i32> %self.8.val) {
start:
  store <5 x i32> %self.8.val, ptr null, align 4
  ret void
}

I'll open an upstream issue.

[Noratrieb]

llvm/llvm-project#63608

[workingjubilee]

cc @ayrtonm @Grommish @Cyanoxygen @chenx97 @Fearyncess @709924470

Your targets may be affected by the underlying codegen bug. You are not required to address this: you are, formally speaking, maintainers of different targets which do not have tier 2 support requirements. This is purely informative. Unless you were planning to apply for raising your target to tier 2... then it is informing you this will likely block that until it is resolved.

[chenx97]

The issue stayed unnoticed until recently, so it may or may not affect us. But hopefully, we can address this problem soon. And as I said earlier somewhere else, I've always wished to get in touch with Tier-2 MIPS GNU maintainers.

[chenx97]

Oh wait, it's cargo. Lemme patch it for MIPSr6 and see if it breaks as well.

[nikic]

The cause for this issue is probably #111999, which emits operations on non-power-of-two vector types like <5 x i32>. This means that we are exercising a lot of new code paths in LLVM now, and should expect to hit more issues in the future (the miscompile at llvm/llvm-project#63475 is I believe also only exposed due to this).

[Noratrieb]

MIPS is tier 3 now, which means we don't build cargo anymore.
Additionally, the LLVM bug has been fixed upstream.