From nobody Wed Feb 11 11:30:25 2026 Received: from mail-pl1-f201.google.com (mail-pl1-f201.google.com [209.85.214.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B6BAE15DBB3 for ; Wed, 29 Jan 2025 00:27:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738110427; cv=none; b=PEmkqEi9cRifFRio3sInTgdRGpyamFz+XpjZoI0oS+tPwW+OjhQMaKv43NXZrOQaQWoRw5A7ES5x1RS0fDUrWVG0Hpv/58O8m3gI71D9fdBX5XxR+i+Cvx+NV7JjXkm+0sH4xwGNDC9exwLiPp0wNr/KEZEjr1NMMzklykvUDGI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738110427; c=relaxed/simple; bh=8Sgp5yu8OBVqSwtXE0quymzsxKTHzx8BbCOes46TEKM=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=hCTh2sc7O96qvWi7g3oXsl1v721mx3t3P1vKiYGMfl0gkoWZGZnae8SCGwpqgODVdm776rPqpQprCAkWJoYHrJdUDsB/rAXC8NLeNXb7BZbX7IoYseahWUumKHR0nQVczz5XrJG0Eyvsbes3AdPKOX3/+JisC7/5I4HN82nKzJc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ctshao.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=lUKCHnN/; arc=none smtp.client-ip=209.85.214.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ctshao.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="lUKCHnN/" Received: by mail-pl1-f201.google.com with SMTP id d9443c01a7336-216717543b7so164524925ad.0 for ; Tue, 28 Jan 2025 16:27:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1738110425; x=1738715225; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=sBEvePub/SR9/IDN5Bs+wN21Z641MzpSYW1O9rZmtN0=; b=lUKCHnN/EKzsGdHBnOen1OcljRUlEcBpZBTRlzMHmGg8qeMQSZldnVyyqYcZ/j0/K1 nWDB1mtJU6abQLgmB36NgX0XeMAewS+xsbtc6BEcoPjZlBIvgtp+jMTPRLijA2ndkcLz R9cJMfeNRjPTVNOnWPYY+c3LKKNYDjZb0V2Iy1M8cOqDzrOnyE+FQPO540NY33OCH06w Bw4KuHiSRsMuSCJkpfMy/Laj0tVOQ+d69uiqQ7XbluPUsM4UGgKNOP7Vzub+RAsErZsh B4V+1ectM2mfy4aCZhA9GB+yOjLhKCc83PjkERmNbDj55JRimUVwAujJinluEawGtPRS TrJQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738110425; x=1738715225; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=sBEvePub/SR9/IDN5Bs+wN21Z641MzpSYW1O9rZmtN0=; b=HhsSzobfE9Fgk9zt7UwuRkyKQadc2PL3WLQjgRevGdoAqmeEvJQTSnEmQvFHKU92xu Hkx34z8T7Fp7MmcbEwTZdfZ4zKAy3tbFOj9v02ker62rncmA8tKl4jTY6iXcZtHrFTL8 tus/mQPeFYPYhuU3mk/mGAJP4jwLxaJSnfco/TmBq9EtYTtL/lONjsPOvpmif+qxNxgC 1w19KAU1iBK9ShX47VqtFmezKLYprAbOGbBO8nzaBEH2eics+ImLqapUKiowWR7gx2LG gVGMVr14dpUyqRwxf/9VwX8p3G3MvXXR73Z3TbL8Y/jn3/j9Qq4mkIvROIqOqwHetpAH T+pw== X-Gm-Message-State: AOJu0YwJpD4uZmxgFRJGY/EX8es3IbAjYkrhc+TgKJq+qCXqiRJ+wvh0 mrE38Xhu738BrMmbLp9sb1kl0yzGZGJWrxtVMv/Vnauoqln61b2eFuQ3jCzRRvaSFd87KSLg2kO GlLFt+K62fD2P1kdH+v+XDreXDo+qZ9fztXRH/ZKVX1X7SxrbOWkAKfjNbvY8PXi1qyx7ybhQxw Ycc8QwU4B5o1avCuuqN3YoU/3Dp4sOE/ze+bdIph9e X-Google-Smtp-Source: AGHT+IGLXoWuDc+BL3s0utjGdQvHw6YCJ9vbchFUR1aiImf8YniBCyB4qvpzZenwbn+L4cac0UqIBCcC0M0= X-Received: from pjc3.prod.google.com ([2002:a17:90b:2f43:b0:2ef:7352:9e97]) (user=ctshao job=prod-delivery.src-stubby-dispatcher) by 2002:a17:903:2446:b0:215:603e:2141 with SMTP id d9443c01a7336-21dd7c625cemr15041855ad.19.1738110424789; Tue, 28 Jan 2025 16:27:04 -0800 (PST) Date: Tue, 28 Jan 2025 16:15:00 -0800 In-Reply-To: <20250129001905.619859-1-ctshao@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250129001905.619859-1-ctshao@google.com> X-Mailer: git-send-email 2.48.1.262.g85cc9f2d1e-goog Message-ID: <20250129001905.619859-5-ctshao@google.com> Subject: [PATCH v3 4/5] perf lock: Report owner stack in usermode From: Chun-Tse Shao To: linux-kernel@vger.kernel.org Cc: Chun-Tse Shao , peterz@infradead.org, mingo@redhat.com, acme@kernel.org, namhyung@kernel.org, mark.rutland@arm.com, alexander.shishkin@linux.intel.com, jolsa@kernel.org, irogers@google.com, adrian.hunter@intel.com, kan.liang@linux.intel.com, nathan@kernel.org, ndesaulniers@google.com, morbo@google.com, justinstitt@google.com, linux-perf-users@vger.kernel.org, bpf@vger.kernel.org, llvm@lists.linux.dev Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Parse `owner_lock_stat` into a rb tree, and report owner lock stats with stack trace in order. Example output: $ sudo ~/linux/tools/perf/perf lock con -abvo -Y mutex-spin -E3 perf benc= h sched pipe ... contended total wait max wait avg wait type caller 171 1.55 ms 20.26 us 9.06 us mutex pipe_rea= d+0x57 0xffffffffac6318e7 pipe_read+0x57 0xffffffffac623862 vfs_read+0x332 0xffffffffac62434b ksys_read+0xbb 0xfffffffface604b2 do_syscall_64+0x82 0xffffffffad00012f entry_SYSCALL_64_after_hwfram= e+0x76 36 193.71 us 15.27 us 5.38 us mutex pipe_wri= te+0x50 0xffffffffac631ee0 pipe_write+0x50 0xffffffffac6241db vfs_write+0x3bb 0xffffffffac6244ab ksys_write+0xbb 0xfffffffface604b2 do_syscall_64+0x82 0xffffffffad00012f entry_SYSCALL_64_after_hwfram= e+0x76 4 51.22 us 16.47 us 12.80 us mutex do_epoll= _wait+0x24d 0xffffffffac691f0d do_epoll_wait+0x24d 0xffffffffac69249b do_epoll_pwait.part.0+0xb 0xffffffffac693ba5 __x64_sys_epoll_pwait+0x95 0xfffffffface604b2 do_syscall_64+0x82 0xffffffffad00012f entry_SYSCALL_64_after_hwfram= e+0x76 =3D=3D=3D owner stack trace =3D=3D=3D 3 31.24 us 15.27 us 10.41 us mutex pipe_rea= d+0x348 0xffffffffac631bd8 pipe_read+0x348 0xffffffffac623862 vfs_read+0x332 0xffffffffac62434b ksys_read+0xbb 0xfffffffface604b2 do_syscall_64+0x82 0xffffffffad00012f entry_SYSCALL_64_after_hwfram= e+0x76 ... Signed-off-by: Chun-Tse Shao --- tools/perf/builtin-lock.c | 20 ++++++++-- tools/perf/util/bpf_lock_contention.c | 54 +++++++++++++++++++++++++++ tools/perf/util/lock-contention.h | 7 ++++ 3 files changed, 78 insertions(+), 3 deletions(-) diff --git a/tools/perf/builtin-lock.c b/tools/perf/builtin-lock.c index 9bebc186286f..d9b0d7472aea 100644 --- a/tools/perf/builtin-lock.c +++ b/tools/perf/builtin-lock.c @@ -42,6 +42,7 @@ #include #include #include +#include =20 static struct perf_session *session; static struct target target; @@ -1817,6 +1818,22 @@ static void print_contention_result(struct lock_cont= ention *con) break; } =20 + if (con->owner && con->save_callstack) { + struct rb_root root =3D RB_ROOT; + + if (symbol_conf.field_sep) + fprintf(lock_output, "# owner stack trace:\n"); + else + fprintf(lock_output, "\n=3D=3D=3D owner stack trace =3D=3D=3D\n\n"); + while ((st =3D pop_owner_stack_trace(con))) + insert_to(&root, st, compare); + + while ((st =3D pop_from(&root))) { + print_lock_stat(con, st); + zfree(st); + } + } + if (print_nr_entries) { /* update the total/bad stats */ while ((st =3D pop_from_result())) { @@ -1962,9 +1979,6 @@ static int check_lock_contention_options(const struct= option *options, } } =20 - if (show_lock_owner) - show_thread_stats =3D true; - return 0; } =20 diff --git a/tools/perf/util/bpf_lock_contention.c b/tools/perf/util/bpf_lo= ck_contention.c index 795e2374facc..cf3267e46589 100644 --- a/tools/perf/util/bpf_lock_contention.c +++ b/tools/perf/util/bpf_lock_contention.c @@ -549,6 +549,60 @@ static const char *lock_contention_get_name(struct loc= k_contention *con, return name_buf; } =20 +struct lock_stat *pop_owner_stack_trace(struct lock_contention *con) +{ + int stacks_fd, stat_fd; + u64 *stack_trace; + s32 stack_id; + struct contention_key ckey =3D {}; + struct contention_data cdata =3D {}; + size_t stack_size =3D con->max_stack * sizeof(*stack_trace); + struct lock_stat *st; + char name[KSYM_NAME_LEN]; + + stacks_fd =3D bpf_map__fd(skel->maps.owner_stacks); + stat_fd =3D bpf_map__fd(skel->maps.owner_stat); + if (!stacks_fd || !stat_fd) + return NULL; + + stack_trace =3D zalloc(stack_size); + if (stack_trace =3D=3D NULL) + return NULL; + + if (bpf_map_get_next_key(stacks_fd, NULL, stack_trace)) + return NULL; + + bpf_map_lookup_elem(stacks_fd, stack_trace, &stack_id); + ckey.stack_id =3D stack_id; + bpf_map_lookup_elem(stat_fd, &ckey, &cdata); + + st =3D zalloc(sizeof(struct lock_stat)); + if (!st) + return NULL; + + strcpy(name, + stack_trace[0] ? lock_contention_get_name(con, NULL, stack_trace, = 0) : "unknown"); + + st->name =3D strdup(name); + if (!st->name) + return NULL; + + st->flags =3D cdata.flags; + st->nr_contended =3D cdata.count; + st->wait_time_total =3D cdata.total_time; + st->wait_time_max =3D cdata.max_time; + st->wait_time_min =3D cdata.min_time; + st->callstack =3D stack_trace; + + if (cdata.count) + st->avg_wait_time =3D cdata.total_time / cdata.count; + + bpf_map_delete_elem(stacks_fd, stack_trace); + bpf_map_delete_elem(stat_fd, &ckey); + + return st; +} + int lock_contention_read(struct lock_contention *con) { int fd, stack, err =3D 0; diff --git a/tools/perf/util/lock-contention.h b/tools/perf/util/lock-conte= ntion.h index a09f7fe877df..97fd33c57f17 100644 --- a/tools/perf/util/lock-contention.h +++ b/tools/perf/util/lock-contention.h @@ -168,6 +168,8 @@ int lock_contention_stop(void); int lock_contention_read(struct lock_contention *con); int lock_contention_finish(struct lock_contention *con); =20 +struct lock_stat *pop_owner_stack_trace(struct lock_contention *con); + #else /* !HAVE_BPF_SKEL */ =20 static inline int lock_contention_prepare(struct lock_contention *con __ma= ybe_unused) @@ -187,6 +189,11 @@ static inline int lock_contention_read(struct lock_con= tention *con __maybe_unuse return 0; } =20 +struct lock_stat *pop_owner_stack_trace(struct lock_contention *con) +{ + return NULL; +} + #endif /* HAVE_BPF_SKEL */ =20 #endif /* PERF_LOCK_CONTENTION_H */ --=20 2.48.1.262.g85cc9f2d1e-goog