Kernel debugging with kmemd

Kmemd is a tool for reading kernel memory of a running kernel using GDB. This can be very useful for debugging and development. Read the state of the kernel without adding a debug prints and rebooting. And because it is read-only, it doesn’t interrupt the running kernel and cause problems with hardware watchdogs. Though, the drawback is you can’t do breakpoints or step through the code.

This is a simple introduction to using it. Aimed at debugging a remote system.

These are the requirements of the target kernel defconfig. Note that the last line is =n to explicitly disable reduced debug info, in case that is set.

CONFIG_BPF=y
CONFIG_BPF_SYSCALL=y
CONFIG_GDB_SCRIPTS=y
CONFIG_PERF_EVENTS=y
CONFIG_KPROBES=y
CONFIG_KPROBE_EVENTS=y
CONFIG_FTRACE=y
CONFIG_DEBUG_INFO_DWARF5=y
CONFIG_DEBUG_INFO_REDUCED=n

I hope I got everything included.

Getting started

If you are using cross-compilation, install gdb-multiarch.

sudo apt install gdb-multiarch

Some dependencies are required for kmemd. This guide assumes you know how to install them on your target system.

kmemd -> libbpf -> libelf -> bzip2, lzma

Once installed, start kmemd with a TCP socket on the target system and the IP of that system.

kmemd -s 10.1.2.3:1234

Start GDB on your machine targeting the vmlinux file corresponding to the target system. If you have external modules, add them from inside the GDB shell. Verify you have necessary debug information by printing a struct, e.g. struct net_device. Finally, attach to kmemd.

gdb-multiarch ./vmlinux
(gdb) add-symbol-file path/to/module/<module_name>.ko
(gdb) ptype struct net_device
type = struct net_device {
__u8 __cacheline_group_begin__net_device_read_tx[0];
...
}
(gdb) target remote 10.1.2.3:1234

Inspecting a net device

Create a file with the following contents, e.g. read_netdev.py.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54

import gdb

class ReadNetDevice(gdb.Command):
    """Inspect a net_device struct"""

    def __init__(self):
        super(ReadNetDevice, self).__init__("read-netdev", gdb.COMMAND_DATA)

    def invoke(self, arg, from_tty):
        args = gdb.string_to_argv(arg)
        if not args:
            print("Usage: read-netdev <address_or_ifname>")
            return

        target = args[0]
        dev = None

        if target.startswith("0x"):
            addr = int(target, 16)
            netdev_type = gdb.lookup_type("struct net_device").pointer()
            dev = gdb.Value(addr).cast(netdev_type)
        else:
            # init_net is a global variable containing everything network-related.
            # Traverse the list of net_devices.
            init_net = gdb.parse_and_eval("init_net")
            head = init_net['dev_base_head']
            curr = head['next']

            # Retrieve offset of dev_list inside struct net_device
            netdev_type = gdb.lookup_type("struct net_device")
            offset = netdev_type['dev_list'].bitpos // 8

            while curr != head.address:
                # Calculate container_of(curr, struct net_device, dev_list)
                dev_addr = int(curr) - offset
                candidate = gdb.Value(dev_addr).cast(netdev_type.pointer())
                if candidate['name'].string() == target:
                    dev = candidate
                    break

                curr = curr['next']

        if not dev:
            print(f"Device '{target}' not found.")
            return

        print(f"===== {dev['name'].string()} =====")
        print(f"Address:  {hex(int(dev))}")
        print(f"ifindex:  {int(dev['ifindex'])}")
        print(f"MTU:      {int(dev['mtu'])}")
        mac = [f"{int(dev['dev_addr'][i]):02x}" for i in range(6)]
        print(f"MAC Addr: {':'.join(mac)}")

ReadNetDevice()

Load the file in GDB and call the function.

(gdb) source path/to/read_netdev.py
(gdb) read-netdev eth0
===== eth0 =====
Address:  0xb5aa0000
ifindex:  2
MTU:      1500
MAC Addr: a8:2b:dd:01:02:03

Inspecting isn’t super fast. If you have a lot of network interfaces it takes a couple seconds to search through them all. Once you have the address as shown above, you can input that to access it directly.

(gdb) read-netdev 0xb5aa0000
===== eth0 =====
Address:  0xb5aa0000
ifindex:  2
MTU:      1500
MAC Addr: a8:2b:dd:01:02:03

From here on you can inspect further down the drivers using the Python script to cast dev['priv'] into whatever struct the driver uses.

priv_type = gdb.lookup_type("struct e1000_adapter")
priv = dev['priv'].cast(priv_type.pointer())

The full Python GDB API is documented here.

Using over SSH

I regularly work on a remote PC that has the connections to my hardware, while the build artefacts are on my laptop. This is easily solved with SSH forwarding.

ssh -L 1234:10.1.2.3:1234 my-server

I can then attach to kmemd with GDB.

(gdb) target remote :1234