GDB Docs

[kotval]

When attempting to use gdb to debug why a process crashed with renode, I attempted to follow the docs: https://betrusted.io/xous-book/ch03-04-debugging-programs.html, but ran into some issues.

1. The docs reference port :3456, but doesn't seem to exist. GDB seems to be started in renode on :3333 on the SoC here.

2. The book recommends to use riscv-none-elf-gdb. It is unclear which binary that is. Is it from here? I tried just using riscv64-unknown-elf-gdb from here and using set architecture riscv:rv32. It seems to work, but could be causing the issues I see below?

3. I changed opt-level to 1 and debug to true in the profile in the main Cargo.toml, ran cargo xtask renode-image --gdb-stub, ran renode emulation/xous-release.resc, and connected using riscv64-unknown-elf-gdb -ex 'tar ext :3333'. Loading symbols for the kernel with file target/riscv32imac-unknown-none-elf/release/xous-kernel seems to work, but loading symbols for any process, e.g. file target/riscv32imac-unknown-xous-elf/release/shellchat, does not. I can enable breakpoints in the kernel, but breakpoints seem to be ignored in processes. Furthermore, mon pr does not work saying No such command or device: pr.

I did all of this on main without my changes to any process to see if I had messed something up. When running on my branch, root-keys crashes, and I would like to connect over gdb to see why, but could not. Does it appear that I am doing anything incorrect?

[bunnie]

@xobs I can confirm that the GDB default port on my machine is 3333. I'm wondering, is the 3456 port that you're getting maybe something that's windows-specific? I want to check before I edit that over, so we're not papering over a configuration that might work in a different environment.

For the binary to use, any RV GDB should work (I haven't rigorously checked but I also have been pretty will-nilly about which host/arch combo I use and it's never been a problem). That being said, we're trying to harmonize around the xpack target triple so riscv-none-elf would be the target triple for that distribution.

I think for (3): I see two possible problems that I'll clarify in the docs now.

1. It might be that you have to attach to the process you are loading symbols for before you load them. I don't know 100% for sure if this is the case, but if it was the case that the symbols didn't line up that would be a thing I'd try reflexively but not document.
2. For breakpoints to work, your target process cannot be XIP. So it does matter what command line you use to build your image -- if you use app-image-xip that will not be debuggable, because processes resident in FLASH can't have breakpoints set since in our implementation breakpoints rely on being able to modify the program to insert a breakpoint instruction.

[bunnie]

i pushed up some changes to the GDB chapter that should address the major issues brought up here. sorry you had to step on the xip landmine, i see now that the sample command literally specifies to use xip, which it should not have. I think that's my bad, I copy/pasted that out of another script when assembling the documentation, but didn't test it.

[xobs]

There are basically two ways to debug something under Renode:

1. Use the built-in GDB, which is great for debugging the kernel and bootloader, and
2. Use the Xous gdbserver, which is better for debugging processes.

The port for the Renode GDB server is 3333, which is the port you should use if you're debugging the kernel.

I believe there are some commands that are missing from the Renode script in order to support debugging using the Xous gdbserver:

# Connect the serial port to a TCPport
emulation CreateServerSocketTerminal 3456 "gdb" False
connector Connect sysbus.app_uart gdb

With that, you can use the same application-aware gdbserver that is present on hardware by connecting to port 3456.

[bunnie]

huh. which script is that? can you add a PR to put those missing lines into the main repo?

[bunnie]

Also, I'm confused, since I've only ever used GDB on hardware. Are you saying that what I get on port 3333 on hardware is different from what's on port 3333 on Renode?

Or is it the case that on hardware, you can't get application aware gdbserver?

Or that on hardware, you never get the port 3333 low-level version of GDB?

[bunnie]

(This is the first time I've even been aware there was a difference and multiple ports possibly available on GDB, so clearly I'm missing something really important about how this is all configured...)

[xobs]

I don't think hardware exposes a network port, unless you're talking about the gdb server presented by wishbone-util which I don't think has been used in ages, in which case that would be the closest analogue to the Renode gdbserver on port 3333.

The GDB server mentioned above on port 3456 is what you get when you attach app_uart to a network socket. In that sense, port 3456 is identical to what you get on hardware when you mux the 3rd uart to the serial port.

The snippet above comes from a line that I've made a PR for: #513

[bunnie]

Ah I see, so the difference is that in hardware, we have ... 3 actual UARTs, which are muxed.

But in Renode you have...well, I guess you do have as many UARTs as you want. But you still have to tell GDB which one to use. Why not use the 3333 port on Renode so that the behavior matches hardware? Is the idea that 3333 is kept around because it has some utilities that this new 3456 port wouldn't have, and so that's why there's a difference?

It's a rhetorical question, I am just trying to understand the differences because I have to grok it if I'm to correctly guide other people on how to use this feature.

[bunnie]

Ah ok, I see now, the docs in the Xous Book had no chance of working because #513 wasn't in the tree. It's not that 3456 is the wrong port. It's the "right" port, it just didn't exist in the main tree. OK, glad that's cleared up now. Sorry about that run-around @kotval .

[xobs]

3333 is useful for when Xous itself has crashed, or when you're debugging the bootloader and Xous isn't even running. It's a stop-the-world debugger. Like "God Mode" on the Vex, where you really can do anything. Debugging there has no effect on the emulated world, so it's like stopping time and looking at things.

3456 is what you get on hardware. It's invasive, since processes will keep running when you attach but their timing will be skewed. It does, however, let you attach to a given process and get actual translated memory pages. With 3333 you kind of just hope that you don't have to deal with any MMU pages in a process, which is a nonissue as long as you're just debugging the kernel or bootloader.

[bunnie]

That's a great explanation. I've added that to the Xous Book chapter: https://betrusted.io/xous-book/ch03-04-debugging-programs.html

[xobs]

It should be noted that 3333 has no concept of processes or threads. You can single-step, and you can insert an unlimited number of breakpoints, but it's somewhat arbitrary what process you're in any time you pause the debugger. Renode is not very good at debugging operating systems that have a MMU active, so it just kind of tries to do its best.