RZG2UL - KGDB - Kernel debuging

Hello,

I have some issues with a driver (esp_hosted_ng):

  • When we start the driver for the first time "it works", then if we stop it and restart it, we have a failure in drivers/base/dd.c: really_probe()
  • We would like to use GDB to understand what happens, (have the callstack etc)

I have read these links:

Currently for the debugging this is what I am doing:

0. Configure U-Boot:

setenv setbootargs 'setenv bootargs rw rootwait earlycon root=${mmcroot} rdinit=/sbin/init loglevel=7 ignore_loglevel nohlt pti=0 maxcpus=1'
saveenv
saveenv
boot


1. Disable the watchdog from systemd: Comment `RuntimeWatchdogSec` in `/etc/systemd/system.conf`.
2. Disable the watchdog from watchdog service:

systemctl disable watchdog


3. reboot
4. Transfer the module to the device:
# PC Side
scp /path_to/esp-hosted-ng.ko [email protected]:/lib/modules/5.10.201-cip41-yocto-standard/extra

# Device side
depmod -a
modprobe esp_hosted_ng

5. Get the addresses of .text,.data,.bss... of the module when loaded:

BASE_ESP_DIR=/sys/module/esp_hosted_ng/sections && echo "add-symbol-file /path_to/esp-hosted-ng.ko $(cat $BASE_ESP_DIR/.text) -s .data $(cat $BASE_ESP_DIR/.data) -s .bss $(cat $BASE_ESP_DIR/.bss) -s .rodata.str1.8 $(cat $BASE_ESP_DIR/.rodata.str1.8)"


6. Start GDB on the computer:

/path_to_from_yocto_toolchain/aarch64-poky-linux-gdb /path_to/vmlinux

add-symbol-file /path_to/esp-hosted-ng.ko 0xffff800008b25000 -s .data 0xffff800008b35000 -s .bss 0xffff800008b363c0 -s .rodata.str1.8 0xffff800008b31350



7. Start KGDB on the device:

echo ttyS0,115200 > /sys/module/kgdboc/parameters/kgdboc
echo g > /proc/sysrq-trigger


8. On the computer start debuging using GDB:

set substitute-path /usr/src/kernel /path_to/kernel-source
show substitute-path
break sdio/esp_sdio.c:esp_probe
break drivers/base/dd.c:really_probe
set serial baud 115200
target remote /dev/ttyUSB0
next
continue


Doing like this makes the debug working on console but there are some issues:

  • All the paths returned by GDB are wrong. The substitute path don't have effect
  • I am not able to debug on eclipse (i got like timeouts) even If I use the "proxy tool"

GDB Logs:

$ /opt/my_toolchain/5.10-hardknott/sysroots/x86_64-pokysdk-linux/usr/bin/aarch64-poky-linux/aarch64-poky-linux-gdb /path_to/vmlinux
GNU gdb (GDB) 10.1
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "--host=x86_64-pokysdk-linux --target=aarch64-poky-linux".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /path_to/vmlinux...
(gdb) set substitute-path /usr/src/kernel /path_to/kernel-source
(gdb) show substitute-path
List of all source path substitution rules:
`/usr/src/kernel' -> `/path_to/kernel-source'.
(gdb) break drivers/base/dd.c:really_probe
Breakpoint 1 at 0xffff80001062a3a0: file /usr/src/kernel/include/asm-generic/atomic-instrumented.h, line 28.
(gdb) set serial baud 115200
(gdb) target remote /dev/ttyUSB0
Remote debugging using /dev/ttyUSB0
[Switching to Thread 4294967294]
arch_kgdb_breakpoint () at /usr/src/kernel/arch/arm64/include/asm/kgdb.h:21
21 asm ("brk %0" : : "I" (KGDB_COMPILED_DBG_BRK_IMM));
(gdb)

I also try to use the proxy tool:

git clone http://git.kernel.org/pub/scm/utils/kernel/kgdb/agent-proxy.git
cd agent-proxy ; make
./agent-proxy 5550^5551 0 /dev/ttyUSB0,115200



# telnet 127.0.0.1 5550 is OK (i have the prompt)

# gdbserver on 127.0.0.1 5551 is always broken:


(gdb) target remote 127.0.0.1:5551
Remote debugging using 127.0.0.1:5551
Ignoring packet error, continuing...
warning: unrecognized item "timeout" in "qSupported" response
Ignoring packet error, continuing...
Remote replied unexpectedly to 'vMustReplyEmpty': timeout
(gdb)

Did someone know how to debug a kernel module on eclipse ?

Did someone know how to fix substitute path feature ?

Has someone already managed to debug on RZG2UL boards the kernel ?

Is it possible to use the USB port for debuging instead of ttyS0 ?

My kernel configuration looks like this:

CONFIG_KALLSYMS=y
CONFIG_KPROBES=y
CONFIG_KRETPROBES=y
# CONFIG_SERIAL_KGDB_NMI is not set
CONFIG_CONSOLE_POLL=y
CONFIG_MAGIC_SYSRQ=y
CONFIG_MAGIC_SYSRQ_DEFAULT_ENABLE=0x1
CONFIG_MAGIC_SYSRQ_SERIAL=y
CONFIG_MAGIC_SYSRQ_SERIAL_SEQUENCE=""
CONFIG_KGDB=y
CONFIG_KGDB_HONOUR_BLOCKLIST=y
CONFIG_KGDB_SERIAL_CONSOLE=y
# CONFIG_KGDB_TESTS is not set
# CONFIG_KGDB_KDB is not set
# CONFIG_FUNCTION_ERROR_INJECTION is not set
# CONFIG_KPROBES_SANITY_TEST is not set
CONFIG_DEBUG_INFO=y
# CONFIG_DEBUG_INFO_REDUCED is not set
# CONFIG_DEBUG_INFO_COMPRESSED is not set
# CONFIG_DEBUG_INFO_SPLIT is not set
# CONFIG_DEBUG_INFO_DWARF4 is not set
# CONFIG_DEBUG_INFO_BTF is not set
CONFIG_GDB_SCRIPTS=y