lldb tests only run on macos

[ehuss]

The lldb tests only run on macos on CI. I think it might be worthwhile to run them on one CI builder for Linux, just to give a little more coverage. However, it's not really clear if anyone is actually using rust-lldb.

I discovered this because the debuginfo tests always fail on my local development system because I have lldb installed. This was a little frustrating, so I investigated and discovered that the pretty-std-collections.rs was not being tested on CI because it is ignored on macos, and it was broken a while ago.

It might be as easy as just adding apt install lldb to a builder. I don't think it is important to try to build it from source. Note that due to some oddity with how lldb apt package is installed, there is a missing symlink. Something like ln -s /usr/lib/llvm-10/bin/lldb-server /usr/bin/lldb-server-10.0.0 may be needed to get it to work.

[artemmukhin]

A related and possibly helpful comment: #83891 (comment)

[pnkfelix]

I want to discuss this at the T-compiler meeting.

The on-going breakage of the debuginfo tests (and subsequent failure to catch such breakage in CI) is pretty worrisome, and frustrating as a contributor when all you want to do is run x.py test and let it go, without having to think too hard about "well which test suites are going to run that may be broken and that I have to skip myself by hand."

[apiraino]

Issue has been discussed in T-compiler meeting on Zulip. This will probably need a follow up with T-infra.

Thus removing nomination:
@rustbot label -I-nominated

[VladimirMakaev]

I think something needs to be done here to make CI more robust. This isn't just about who uses "rust-lldb" (btw we do use the synth provider shipped with rust-lang in our tooling) this is also about changes in compiler potentially breaking LLDB. For instance while working on #124781 (comment) i've added and fixed a bunch of tests and made sure they work on LLDB 18.1.

I propose compiling LLDB from src/llvm-project and use this on CI (#124501 should help to set the binary up). We do compile Clang , do we? So compiling lldb should be straightforward. (We can do this on Linux only for instance at start). I can try dig into this direction or if there are pointers I can use please let me know.

[saethlin]

All our apple CI jobs use an lldb version before 1800. All the changes in that PR are untested on apple, and indeed based on local testing, all those tests do not pass on aarch64-apple, using either lldb from homebrew's llvm@18 or the lldb that Apple ships with XCode. I do not have access to any x86_64 apple hardware to test with.

[ehuss]

FWIW, xcode 16 beta looks like it will use LLVM 17. Unfortunately we are stuck on xcode 14 (#121058).

[saethlin]

Surely it's possible to install lldb separately from xcode, right? I am pretty sure I just did that on a laptop, though I don't know if we have the same ability to install software in CI.

[ehuss]

Yea, we can use homebrew to install llvm. It looks like it uses the system debug server to avoid requiring code signing. I'm not sure how wise it would be to rely on homebrew so heavily, or what kind of complications it would have.

I don't know how long it takes to build lldb from source. If it doesn't take too long, that might be the best option.

EDIT: Though I don't know how much of a burden that would be for @nikic whenever they update llvm. If it ends up breaking a bunch of debuginfo tests every time, I could imagine that being annoying.

[saethlin]

If we were building lldb from source, we could cache our builds somewhere, right?

If it ends up breaking a bunch of debuginfo tests every time, I could imagine that being annoying.

This is an unavoidable side effect of testing our debuginfo by using a debugger. It's a bummer to dump this all onto one person, but the work does need to get done. Trying to support a range of debugger versions is a battle that has already been lost.

[ehuss]

If we were building lldb from source, we could cache our builds somewhere, right?

LLVM builds are cached via sccache onto an S3 bucket. However, it's still a bit slow. Probably not something to worry about too much.