From nobody Thu Dec 18 19:04:24 2025 Received: from mail-pj1-f74.google.com (mail-pj1-f74.google.com [209.85.216.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 001D44A24 for ; Thu, 27 Feb 2025 00:37:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740616643; cv=none; b=NUruzVIF28/Cm4nx7BVXzFzUdNZJaHeFG+FkCY2zfrrkE7Lawg4ClpPifaCOADnzJC+p843pAm9nyTO0TwT23nV7dw+z+ABDrM0F2OtH3uSYtmhvTRCRyBHzn3sf81ydrAT4QV7ztz2X1RE4qcdf3V+hfwOuFF12ngbh0l3VgoE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740616643; c=relaxed/simple; bh=ra0KxU6OZ2Kntw539Gq8DHkuYwWOUFGpZweLpDYUOnk=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=Vp1LVKcXFjsoE0oSrswWCcH6EX/N186i/rAe8AKLIhQl2J+RsDu5wYP6MvFG7UlIq7FzzlTUdd9ENati9XeFiIjE3wfaLQ0Ew0zU+c3rMDlhJJYJqYLiA6ncJslmW611ZEeT1YQjuyrCoRcscj+4z7yJ5mFxBJBlVPTaYpCAFIw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ctshao.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=gj9ZwhK6; arc=none smtp.client-ip=209.85.216.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ctshao.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="gj9ZwhK6" Received: by mail-pj1-f74.google.com with SMTP id 98e67ed59e1d1-2fc518f0564so900361a91.2 for ; Wed, 26 Feb 2025 16:37:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1740616641; x=1741221441; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=ySFre/DS9Ie6BpbqXRUL93PnxWJmHNbJVUPdTcYqivQ=; b=gj9ZwhK6cy7d5uq2CD1PKYQ1l5JIzsVXymr77b4AFmZiOvtMM5hydf+UOeqH99Kaed FlZ5o8ybEvs0Tan6r3iz9q0Ddaj3GK6n2KtJl/5dRG5PFBw2boKuij5BWIKkQNNHHdES jgbxFKydik14S3khEGKEsO3+o5/zTfgv72miT9oz+3nRelxLJN/JkTu5J70CHySei46B XJPYwgnscFy8B/jdQNcC8QsU328xEatQ1fOBeBIiA+fUZwGrzUn163p3tVCubPwYSTnI VG0KgoZJ/jpTLf0C90UGN98BBrl63axNy7B/KyV81UZTpnfOkdHQvn5IErSyYh6RXAk8 XSiw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740616641; x=1741221441; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=ySFre/DS9Ie6BpbqXRUL93PnxWJmHNbJVUPdTcYqivQ=; b=q+6GTP7Qs0njK7aM09eVgZ3hPYn2nUe/RrbJD4y2OwfYX/pZFsPp7KvMtEexCZn5hr 748fyqfRgBX8giv/FzrqqitEY8je9AuMbwONbRrJqT79GCWgXkW+SOwERiyc122lvFKP AIXoeIzpTclXQvv9A1SfXD9zQmF4gLKDTVsgAWtn/xgzgyVYO3t/9mzqSQhl5DjhbWiI kEKblAAULTvfoKEDVTJbVgr2l/WRqeQLeWvgdRn8IkgLZyXqO9feS96JJMPs5TQWB4Ep y44YYLRLZo6sRCjNU+lKW9cMBhV2oATpSIXPFZNIY25q5JlJPIHqSnhwINz46LNUZM9b SJnA== X-Gm-Message-State: AOJu0YwakxnoxoePBI5/dvXiuW+l2OaNu4GY5WKUn/TmOb5ctqMluEJ/ dSZB4Q2bKmvd0WRmAyUgo175JFOT/ejxDqN07sD+PfDYRtOyMK0Gkrewrwe+3pmpgViYgQxR6+p qU8kh6BirtQ8dW3n387r4/JTl3wBlSQOeHRv/hqRkVauI9riF2ZBS/GpaVHRv2/F+rG44L+M6nc resul23t1ljcM3O2C+PYWCsWDucHRml6P5JB5DrblW X-Google-Smtp-Source: AGHT+IF/GB2SXwZz3UpivVRMccErkd62rEc6mCFXFCH/iGAfZUTC6MXPC2bQf8nMbVhWRK4P5ZBkHEd5nmg= X-Received: from pfbft2.prod.google.com ([2002:a05:6a00:81c2:b0:730:8970:1f9c]) (user=ctshao job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a00:3254:b0:734:9157:7456 with SMTP id d2e1a72fcca58-734915776d1mr4869228b3a.19.1740616641195; Wed, 26 Feb 2025 16:37:21 -0800 (PST) Date: Wed, 26 Feb 2025 16:28:56 -0800 In-Reply-To: <20250227003359.732948-1-ctshao@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250227003359.732948-1-ctshao@google.com> X-Mailer: git-send-email 2.48.1.658.g4767266eb4-goog Message-ID: <20250227003359.732948-5-ctshao@google.com> Subject: [PATCH v8 4/4] perf lock: Report owner stack in usermode From: Chun-Tse Shao To: linux-kernel@vger.kernel.org Cc: Chun-Tse Shao , peterz@infradead.org, mingo@redhat.com, acme@kernel.org, namhyung@kernel.org, mark.rutland@arm.com, alexander.shishkin@linux.intel.com, jolsa@kernel.org, irogers@google.com, adrian.hunter@intel.com, kan.liang@linux.intel.com, nick.forrington@arm.com, linux-perf-users@vger.kernel.org, bpf@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This patch parses `owner_lock_stat` into a RB tree, enabling ordered reporting of owner lock statistics with stack traces. It also updates the documentation for the `-o` option in contention mode, decouples `-o` from `-t`, and issues a warning to inform users about the new behavior of `-ov`. Example output: $ sudo ~/linux/tools/perf/perf lock con -abvo -Y mutex-spin -E3 perf benc= h sched pipe ... contended total wait max wait avg wait type caller 171 1.55 ms 20.26 us 9.06 us mutex pipe_rea= d+0x57 0xffffffffac6318e7 pipe_read+0x57 0xffffffffac623862 vfs_read+0x332 0xffffffffac62434b ksys_read+0xbb 0xfffffffface604b2 do_syscall_64+0x82 0xffffffffad00012f entry_SYSCALL_64_after_hwfram= e+0x76 36 193.71 us 15.27 us 5.38 us mutex pipe_wri= te+0x50 0xffffffffac631ee0 pipe_write+0x50 0xffffffffac6241db vfs_write+0x3bb 0xffffffffac6244ab ksys_write+0xbb 0xfffffffface604b2 do_syscall_64+0x82 0xffffffffad00012f entry_SYSCALL_64_after_hwfram= e+0x76 4 51.22 us 16.47 us 12.80 us mutex do_epoll= _wait+0x24d 0xffffffffac691f0d do_epoll_wait+0x24d 0xffffffffac69249b do_epoll_pwait.part.0+0xb 0xffffffffac693ba5 __x64_sys_epoll_pwait+0x95 0xfffffffface604b2 do_syscall_64+0x82 0xffffffffad00012f entry_SYSCALL_64_after_hwfram= e+0x76 =3D=3D=3D owner stack trace =3D=3D=3D 3 31.24 us 15.27 us 10.41 us mutex pipe_rea= d+0x348 0xffffffffac631bd8 pipe_read+0x348 0xffffffffac623862 vfs_read+0x332 0xffffffffac62434b ksys_read+0xbb 0xfffffffface604b2 do_syscall_64+0x82 0xffffffffad00012f entry_SYSCALL_64_after_hwfram= e+0x76 ... Signed-off-by: Chun-Tse Shao --- tools/perf/Documentation/perf-lock.txt | 5 +- tools/perf/builtin-lock.c | 22 +++++++- tools/perf/util/bpf_lock_contention.c | 71 +++++++++++++++++++++++--- tools/perf/util/lock-contention.h | 7 +++ 4 files changed, 94 insertions(+), 11 deletions(-) diff --git a/tools/perf/Documentation/perf-lock.txt b/tools/perf/Documentat= ion/perf-lock.txt index d3793054f7d3..859dc11a7372 100644 --- a/tools/perf/Documentation/perf-lock.txt +++ b/tools/perf/Documentation/perf-lock.txt @@ -179,8 +179,9 @@ CONTENTION OPTIONS =20 -o:: --lock-owner:: - Show lock contention stat by owners. Implies --threads and - requires --use-bpf. + Show lock contention stat by owners. This option can be combined with -t, + which shows owner's per thread lock stats, or -v, which shows owner's + stacktrace. Requires --use-bpf. =20 -Y:: --type-filter=3D:: diff --git a/tools/perf/builtin-lock.c b/tools/perf/builtin-lock.c index 9bebc186286f..05e7bc30488a 100644 --- a/tools/perf/builtin-lock.c +++ b/tools/perf/builtin-lock.c @@ -1817,6 +1817,22 @@ static void print_contention_result(struct lock_cont= ention *con) break; } =20 + if (con->owner && con->save_callstack && verbose > 0) { + struct rb_root root =3D RB_ROOT; + + if (symbol_conf.field_sep) + fprintf(lock_output, "# owner stack trace:\n"); + else + fprintf(lock_output, "\n=3D=3D=3D owner stack trace =3D=3D=3D\n\n"); + while ((st =3D pop_owner_stack_trace(con))) + insert_to(&root, st, compare); + + while ((st =3D pop_from(&root))) { + print_lock_stat(con, st); + free(st); + } + } + if (print_nr_entries) { /* update the total/bad stats */ while ((st =3D pop_from_result())) { @@ -1962,8 +1978,10 @@ static int check_lock_contention_options(const struc= t option *options, } } =20 - if (show_lock_owner) - show_thread_stats =3D true; + if (show_lock_owner && !show_thread_stats) { + pr_warning("Now -o try to show owner's callstack instead of pid and comm= .\n"); + pr_warning("Please use -t option too to keep the old behavior.\n"); + } =20 return 0; } diff --git a/tools/perf/util/bpf_lock_contention.c b/tools/perf/util/bpf_lo= ck_contention.c index 76542b86e83f..5af8f6d1bc95 100644 --- a/tools/perf/util/bpf_lock_contention.c +++ b/tools/perf/util/bpf_lock_contention.c @@ -460,7 +460,6 @@ static const char *lock_contention_get_name(struct lock= _contention *con, { int idx =3D 0; u64 addr; - const char *name =3D ""; static char name_buf[KSYM_NAME_LEN]; struct symbol *sym; struct map *kmap; @@ -475,13 +474,14 @@ static const char *lock_contention_get_name(struct lo= ck_contention *con, if (pid) { struct thread *t =3D machine__findnew_thread(machine, /*pid=3D*/-1, pid= ); =20 - if (t =3D=3D NULL) - return name; - if (!bpf_map_lookup_elem(task_fd, &pid, &task) && - thread__set_comm(t, task.comm, /*timestamp=3D*/0)) - name =3D task.comm; + if (t !=3D NULL && + !bpf_map_lookup_elem(task_fd, &pid, &task) && + thread__set_comm(t, task.comm, /*timestamp=3D*/0)) { + snprintf(name_buf, sizeof(name_buf), "%s", task.comm); + return name_buf; + } } - return name; + return ""; } =20 if (con->aggr_mode =3D=3D LOCK_AGGR_ADDR) { @@ -549,6 +549,63 @@ static const char *lock_contention_get_name(struct loc= k_contention *con, return name_buf; } =20 +struct lock_stat *pop_owner_stack_trace(struct lock_contention *con) +{ + int stacks_fd, stat_fd; + u64 *stack_trace =3D NULL; + s32 stack_id; + struct contention_key ckey =3D {}; + struct contention_data cdata =3D {}; + size_t stack_size =3D con->max_stack * sizeof(*stack_trace); + struct lock_stat *st =3D NULL; + + stacks_fd =3D bpf_map__fd(skel->maps.owner_stacks); + stat_fd =3D bpf_map__fd(skel->maps.owner_stat); + if (!stacks_fd || !stat_fd) + goto out_err; + + stack_trace =3D zalloc(stack_size); + if (stack_trace =3D=3D NULL) + goto out_err; + + if (bpf_map_get_next_key(stacks_fd, NULL, stack_trace)) + goto out_err; + + bpf_map_lookup_elem(stacks_fd, stack_trace, &stack_id); + ckey.stack_id =3D stack_id; + bpf_map_lookup_elem(stat_fd, &ckey, &cdata); + + st =3D zalloc(sizeof(struct lock_stat)); + if (!st) + goto out_err; + + st->name =3D strdup(stack_trace[0] ? lock_contention_get_name(con, NULL, = stack_trace, 0) : + "unknown"); + if (!st->name) + goto out_err; + + st->flags =3D cdata.flags; + st->nr_contended =3D cdata.count; + st->wait_time_total =3D cdata.total_time; + st->wait_time_max =3D cdata.max_time; + st->wait_time_min =3D cdata.min_time; + st->callstack =3D stack_trace; + + if (cdata.count) + st->avg_wait_time =3D cdata.total_time / cdata.count; + + bpf_map_delete_elem(stacks_fd, stack_trace); + bpf_map_delete_elem(stat_fd, &ckey); + + return st; + +out_err: + free(stack_trace); + free(st); + + return NULL; +} + int lock_contention_read(struct lock_contention *con) { int fd, stack, err =3D 0; diff --git a/tools/perf/util/lock-contention.h b/tools/perf/util/lock-conte= ntion.h index a09f7fe877df..1da779d75b5f 100644 --- a/tools/perf/util/lock-contention.h +++ b/tools/perf/util/lock-contention.h @@ -168,6 +168,8 @@ int lock_contention_stop(void); int lock_contention_read(struct lock_contention *con); int lock_contention_finish(struct lock_contention *con); =20 +struct lock_stat *pop_owner_stack_trace(struct lock_contention *con); + #else /* !HAVE_BPF_SKEL */ =20 static inline int lock_contention_prepare(struct lock_contention *con __ma= ybe_unused) @@ -187,6 +189,11 @@ static inline int lock_contention_read(struct lock_con= tention *con __maybe_unuse return 0; } =20 +struct lock_stat *pop_owner_stack_trace(struct lock_contention *con __mayb= e_unused) +{ + return NULL; +} + #endif /* HAVE_BPF_SKEL */ =20 #endif /* PERF_LOCK_CONTENTION_H */ --=20 2.48.1.658.g4767266eb4-goog