From nobody Thu Apr 30 09:29:53 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7B5C4C433EF for ; Wed, 1 Jun 2022 06:58:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237128AbiFAG64 (ORCPT ); Wed, 1 Jun 2022 02:58:56 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42748 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1350046AbiFAG6v (ORCPT ); Wed, 1 Jun 2022 02:58:51 -0400 Received: from mail-pg1-x52d.google.com (mail-pg1-x52d.google.com [IPv6:2607:f8b0:4864:20::52d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AC56C28E3B; Tue, 31 May 2022 23:58:50 -0700 (PDT) Received: by mail-pg1-x52d.google.com with SMTP id x12so1054600pgj.7; Tue, 31 May 2022 23:58:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=sender:from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=JZlhNhz/TCvzULnkKwCUIM+LX24oR54eteKzGDWd7nw=; b=I/T+oNhe0uGdEuggQYwCVXaS4L3h9iHEHdCWNdlpnt55zQGOY5S5nwDADJtWsjytpx WbrRClSZGhs/zL9NecVAJHYda6sEtsvTtP4CPuUpH8Jxp0FpA/WFhlskaLyovDEYVKAV RaBdHpQK4iCTJ63V07/1ZW0/SYp2F14yKhsgllg88JMdR8L1MyqoUObzTU72npL0oUe3 KnBE4aI5w153bhlXbngTh1uXp/o1K/qsoP6Q4mX15bCBmDuocreR42T1OLosajSB+P1H H+ggLOyXzxqs1SJ0+fEi1oceWEwU8GHJBa5ZtAYib4epBliDc1mUPhk/84qlytfDzxbI L7Tw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:sender:from:to:cc:subject:date:message-id :in-reply-to:references:mime-version:content-transfer-encoding; bh=JZlhNhz/TCvzULnkKwCUIM+LX24oR54eteKzGDWd7nw=; b=sEAaGKcJiWUmSpRjMUvgAni9o3C5DJMDUIBJiUnV9TMzEY7cfS6HiH77AQ78H2N6/9 gsgSzvPL4xVFEQKyEkOmSx3IwLwx9lgEiGWS6F9hXDi6McGMgP+nplxpSGJXsn1ayNGJ tdTxrBSyFbpTkkVYcJxGl5pPL635SPasa4M/JUasfs4H4cV1APQZi7PHmGuI/JVX69xb UtrAZALQQJssWXPUCzqzPVVOmhy2yT3WjUoXiSU9fbyYAwDoq/UMpOXU2p7t2Xnje1HI 2inWbWzpKSZCUOjWKwQpvMRwuP6ZOFDxU0QJrbPDW1xLiyhVNMTADwV8kwfmxtcd6Ra1 56Hw== X-Gm-Message-State: AOAM530qfw2cEGnM7kZbwRoCjL0r4FRLzpxnFRcAsZebtYyu5db/jC02 sd5BKIHKN3RVNZ6At/gfsP0= X-Google-Smtp-Source: ABdhPJzer0OJBnQUKHRhqiyNT4MlgFmlhN/Gf4Ck2k9RpwWh//6Bfm3CQy8eLxKl4DLsGJBXLzMpdw== X-Received: by 2002:a62:d15c:0:b0:51b:b64d:fc69 with SMTP id t28-20020a62d15c000000b0051bb64dfc69mr914931pfl.7.1654066730165; Tue, 31 May 2022 23:58:50 -0700 (PDT) Received: from balhae.hsd1.ca.comcast.net ([2601:647:6780:cb0:82d7:adf7:b62d:6020]) by smtp.gmail.com with ESMTPSA id b10-20020a17090a5a0a00b001cd4989febcsm3207271pjd.8.2022.05.31.23.58.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 31 May 2022 23:58:49 -0700 (PDT) Sender: Namhyung Kim From: Namhyung Kim To: Arnaldo Carvalho de Melo , Jiri Olsa Cc: Ingo Molnar , Peter Zijlstra , LKML , Ian Rogers , linux-perf-users@vger.kernel.org, Will Deacon , Waiman Long , Boqun Feng , Davidlohr Bueso Subject: [PATCH 1/5] perf lock: Change to synthesize task events Date: Tue, 31 May 2022 23:58:42 -0700 Message-Id: <20220601065846.456965-2-namhyung@kernel.org> X-Mailer: git-send-email 2.36.1.255.ge46751e96f-goog In-Reply-To: <20220601065846.456965-1-namhyung@kernel.org> References: <20220601065846.456965-1-namhyung@kernel.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" With -t/--threads option, it needs to display task names so synthesize task related events at the beginning. Fixes: 7c3bcbdf449f ("perf lock: Add -t/--thread option for report") Acked-by: Ian Rogers Signed-off-by: Namhyung Kim --- tools/perf/builtin-lock.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tools/perf/builtin-lock.c b/tools/perf/builtin-lock.c index b1200b7340a6..23a33ac15e68 100644 --- a/tools/perf/builtin-lock.c +++ b/tools/perf/builtin-lock.c @@ -1083,7 +1083,7 @@ static int __cmd_report(bool display_info) static int __cmd_record(int argc, const char **argv) { const char *record_args[] =3D { - "record", "-R", "-m", "1024", "-c", "1", "--synth", "no", + "record", "-R", "-m", "1024", "-c", "1", "--synth", "task", }; unsigned int rec_argc, i, j, ret; const char **rec_argv; base-commit: 9dde6cadb92b5670b23b97ec53091df0530ec38b --=20 2.36.1.255.ge46751e96f-goog From nobody Thu Apr 30 09:29:53 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id DC962C433EF for ; Wed, 1 Jun 2022 06:59:01 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1350079AbiFAG7A (ORCPT ); Wed, 1 Jun 2022 02:59:00 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42866 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1350050AbiFAG6x (ORCPT ); Wed, 1 Jun 2022 02:58:53 -0400 Received: from mail-pj1-x1030.google.com (mail-pj1-x1030.google.com [IPv6:2607:f8b0:4864:20::1030]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D4D632AC77; Tue, 31 May 2022 23:58:51 -0700 (PDT) Received: by mail-pj1-x1030.google.com with SMTP id j7so1137810pjn.4; Tue, 31 May 2022 23:58:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=sender:from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=C4OURF1zHZFFhVzsDI1Pk89wZ1Ejc7c9NHHrBvA5gsw=; b=VbvvzD9vYe1DHYsYmAj1Kklq1M14msRaqs339+solqCDvZghv6D+MrwAPf7MZry4bm ynnsmMHE9NFWLVen5sbW1MjYj0yEuNz7gk1TQgdDCyEtw03IPxGla2m97wU/eekLHNxH lvOB6jeE2EeeBaiQu0SJYj1Ah7Aqb9PoFrXtNNNA6rtbT91QfLnIcha8fiqjGLSp0fSa 6nsjS7PLlTn2xxO1u9Ha2MsdMWi3ToLFxi386QBMMDEFS2hOs6K2SdnxyRO3d/GlaqPW I8qDhBzHP16ZvjGAbWO6C1zgc7TlbHN/l1Q2qIHUu4u0b25yEJyMWKaCphAWU5imUDKf qkKw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:sender:from:to:cc:subject:date:message-id :in-reply-to:references:mime-version:content-transfer-encoding; bh=C4OURF1zHZFFhVzsDI1Pk89wZ1Ejc7c9NHHrBvA5gsw=; b=DWtUv/F6qCZW5m7wbsd2I9hu1MhuOWtuwaQdC0sc/xY/Dw+7+An2Ds8D0xDqUIHomO XbmOPQueraaj3NOYJmiecbbdea2Iyczz4l6SYW5RVqeNK3lCv7Kpf/L7Zyx2nrnvJAIq ArC3Rsh1q2JEoS05+m6MspzPErsDi62/Z5UbJJYUxbRg1aS9pKPKOqUfTlwzHOIhyF0o XNFst+UEgkVZsw0IqmaRSy6KWISZ5OzBx44fh7PQcmDzY5J/Nr+V3q/NWEH2FGnFYWuR aif/ChaE5rqUwKn4ePCsSlCeL9TvXoNstMbrrwod0Ia7Ee7Ry/m1vyxPno4WhQPv+OGu iA4w== X-Gm-Message-State: AOAM532bzH5tZoHIhc37NwK9a+gK+cn9bVPEj7MpvyZYWb0suAW12KEy NelKeS0fIKObR++EVHd2CN8= X-Google-Smtp-Source: ABdhPJyEr9majNavpLa8V6xpaG85gA6Dp6kEHB+hTRnTLHdltWzynmtYMJ6+mZ8IdnJx9EpfPzJ1LA== X-Received: by 2002:a17:902:f70a:b0:153:88c7:774 with SMTP id h10-20020a170902f70a00b0015388c70774mr64365924plo.166.1654066731360; Tue, 31 May 2022 23:58:51 -0700 (PDT) Received: from balhae.hsd1.ca.comcast.net ([2601:647:6780:cb0:82d7:adf7:b62d:6020]) by smtp.gmail.com with ESMTPSA id b10-20020a17090a5a0a00b001cd4989febcsm3207271pjd.8.2022.05.31.23.58.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 31 May 2022 23:58:51 -0700 (PDT) Sender: Namhyung Kim From: Namhyung Kim To: Arnaldo Carvalho de Melo , Jiri Olsa Cc: Ingo Molnar , Peter Zijlstra , LKML , Ian Rogers , linux-perf-users@vger.kernel.org, Will Deacon , Waiman Long , Boqun Feng , Davidlohr Bueso Subject: [PATCH 2/5] perf lock: Add lock contention tracepoints record support Date: Tue, 31 May 2022 23:58:43 -0700 Message-Id: <20220601065846.456965-3-namhyung@kernel.org> X-Mailer: git-send-email 2.36.1.255.ge46751e96f-goog In-Reply-To: <20220601065846.456965-1-namhyung@kernel.org> References: <20220601065846.456965-1-namhyung@kernel.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" When LOCKDEP and LOCK_STAT events are not available, it falls back to record the new lock contention tracepoints. Signed-off-by: Namhyung Kim --- tools/perf/builtin-lock.c | 70 +++++++++++++++++++++++++++++++++++---- 1 file changed, 63 insertions(+), 7 deletions(-) diff --git a/tools/perf/builtin-lock.c b/tools/perf/builtin-lock.c index 23a33ac15e68..3e3320b8cede 100644 --- a/tools/perf/builtin-lock.c +++ b/tools/perf/builtin-lock.c @@ -495,6 +495,12 @@ struct trace_lock_handler { =20 int (*release_event)(struct evsel *evsel, struct perf_sample *sample); + + int (*contention_begin_event)(struct evsel *evsel, + struct perf_sample *sample); + + int (*contention_end_event)(struct evsel *evsel, + struct perf_sample *sample); }; =20 static struct lock_seq_stat *get_seq(struct thread_stat *ts, u64 addr) @@ -822,6 +828,20 @@ static int evsel__process_lock_release(struct evsel *e= vsel, struct perf_sample * return 0; } =20 +static int evsel__process_contention_begin(struct evsel *evsel, struct per= f_sample *sample) +{ + if (trace_handler->contention_begin_event) + return trace_handler->contention_begin_event(evsel, sample); + return 0; +} + +static int evsel__process_contention_end(struct evsel *evsel, struct perf_= sample *sample) +{ + if (trace_handler->contention_end_event) + return trace_handler->contention_end_event(evsel, sample); + return 0; +} + static void print_bad_events(int bad, int total) { /* Output for debug, this have to be removed */ @@ -1023,6 +1043,11 @@ static const struct evsel_str_handler lock_tracepoin= ts[] =3D { { "lock:lock_release", evsel__process_lock_release, }, /* CONFIG_LOCKD= EP */ }; =20 +static const struct evsel_str_handler contention_tracepoints[] =3D { + { "lock:contention_begin", evsel__process_contention_begin, }, + { "lock:contention_end", evsel__process_contention_end, }, +}; + static bool force; =20 static int __cmd_report(bool display_info) @@ -1086,20 +1111,41 @@ static int __cmd_record(int argc, const char **argv) "record", "-R", "-m", "1024", "-c", "1", "--synth", "task", }; unsigned int rec_argc, i, j, ret; + unsigned int nr_tracepoints; const char **rec_argv; + bool has_lock_stat =3D true; =20 for (i =3D 0; i < ARRAY_SIZE(lock_tracepoints); i++) { if (!is_valid_tracepoint(lock_tracepoints[i].name)) { - pr_err("tracepoint %s is not enabled. " - "Are CONFIG_LOCKDEP and CONFIG_LOCK_STAT enabled?\n", - lock_tracepoints[i].name); - return 1; + pr_debug("tracepoint %s is not enabled. " + "Are CONFIG_LOCKDEP and CONFIG_LOCK_STAT enabled?\n", + lock_tracepoints[i].name); + has_lock_stat =3D false; + break; + } + } + + if (has_lock_stat) + goto setup_args; + + for (i =3D 0; i < ARRAY_SIZE(contention_tracepoints); i++) { + if (!is_valid_tracepoint(contention_tracepoints[i].name)) { + pr_err("tracepoint %s is not enabled.\n", + contention_tracepoints[i].name); + return 1; } } =20 +setup_args: rec_argc =3D ARRAY_SIZE(record_args) + argc - 1; + + if (has_lock_stat) + nr_tracepoints =3D ARRAY_SIZE(lock_tracepoints); + else + nr_tracepoints =3D ARRAY_SIZE(contention_tracepoints); + /* factor of 2 is for -e in front of each tracepoint */ - rec_argc +=3D 2 * ARRAY_SIZE(lock_tracepoints); + rec_argc +=3D 2 * nr_tracepoints; =20 rec_argv =3D calloc(rec_argc + 1, sizeof(char *)); if (!rec_argv) @@ -1108,9 +1154,19 @@ static int __cmd_record(int argc, const char **argv) for (i =3D 0; i < ARRAY_SIZE(record_args); i++) rec_argv[i] =3D strdup(record_args[i]); =20 - for (j =3D 0; j < ARRAY_SIZE(lock_tracepoints); j++) { + for (j =3D 0; j < nr_tracepoints; j++) { + const char *ev_name; + + if (has_lock_stat) + ev_name =3D strdup(lock_tracepoints[j].name); + else + ev_name =3D strdup(contention_tracepoints[j].name); + + if (!ev_name) + return -ENOMEM; + rec_argv[i++] =3D "-e"; - rec_argv[i++] =3D strdup(lock_tracepoints[j].name); + rec_argv[i++] =3D ev_name; } =20 for (j =3D 1; j < (unsigned int)argc; j++, i++) --=20 2.36.1.255.ge46751e96f-goog From nobody Thu Apr 30 09:29:53 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id F37FEC433F5 for ; Wed, 1 Jun 2022 06:59:09 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1350090AbiFAG7I (ORCPT ); Wed, 1 Jun 2022 02:59:08 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42968 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1350062AbiFAG6y (ORCPT ); Wed, 1 Jun 2022 02:58:54 -0400 Received: from mail-pj1-x102e.google.com (mail-pj1-x102e.google.com [IPv6:2607:f8b0:4864:20::102e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 19DA028E3B; Tue, 31 May 2022 23:58:53 -0700 (PDT) Received: by mail-pj1-x102e.google.com with SMTP id 3-20020a17090a174300b001e426a02ac5so2117652pjm.2; Tue, 31 May 2022 23:58:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=sender:from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=SvywXQR5u5eLQGrEXYtQtLKz9fPRbHwv8WFahECd0Ho=; b=hkDgB0dJd0jTool2E/W8gSNApqM2U1u6+6qzuJZYb9UrwhA6Zi8xafBaQRpUBaooU9 Ez7RG9iNn8lQ82KuwwYhZ1d6gsofDIbQCbJkeZhXiNHmv5Zh02aSrjHQwwPDXUpEeEjM z1olgHARgw/jDezZCboiN8GY/hItvXKJ+iw/gac1kWET37FWoFFwgTjKuziov4Osf/89 Ch8aOzfzisSLCGx2Xr/Zv2eaQ8mctpaTylw5q372TmyKCHShwBW/zX2S0Gwydnu01jmw MQGYPlPkGTmLMdvCKaCb/ufr8s3m7WrpTx6jaY40yaWJwda6riOb+nrapsHPrJtO/c6i 3SOg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:sender:from:to:cc:subject:date:message-id :in-reply-to:references:mime-version:content-transfer-encoding; bh=SvywXQR5u5eLQGrEXYtQtLKz9fPRbHwv8WFahECd0Ho=; b=LfljWEnry0324OSrPBGKa2pbQBrffadw/Cw+iF2nZxAlJYY23dPKhpvieysXjX2Dxr KG1WUGNyF2FD0tL5UgJV5Yj9wa1RBWTU1gQREe5f3PhVZK1aJxhUD8+3YsKNKj0fyI7H saDOF+FufxgxHiDbYMDnTHSCoK7K6RBeHRd8tDuY1VQRYu2KIViGJHrN3br2m4Zb5/2H Me1tInl2PwxmZBPjiXmhaa00JSTmQssnNKi7ejl6PAomnVmCZfUzuK8al7OX3Wvz7JJr 1z3QiFSwQhdVVMLEHUoTH5d88d22U0aZvlptNjuqC7xK3YTTO4m7ldebgu08RLbteV9W xjoA== X-Gm-Message-State: AOAM533obb4/sUmnYBU+plaPA2Lg42eYLDZMvAWgM+bVjI24e/y6XMuh Ss/3KbRp/K3t0Qg4tpK/6KA= X-Google-Smtp-Source: ABdhPJyZB9sm5lrcSl295Dxrb8f0ukTuAPEaDSCKWRoknHi9p8ouSHE+ejWaWRXQ0EiwNuGrS6lYlw== X-Received: by 2002:a17:903:1c8:b0:165:1395:dcb3 with SMTP id e8-20020a17090301c800b001651395dcb3mr1836834plh.76.1654066732590; Tue, 31 May 2022 23:58:52 -0700 (PDT) Received: from balhae.hsd1.ca.comcast.net ([2601:647:6780:cb0:82d7:adf7:b62d:6020]) by smtp.gmail.com with ESMTPSA id b10-20020a17090a5a0a00b001cd4989febcsm3207271pjd.8.2022.05.31.23.58.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 31 May 2022 23:58:52 -0700 (PDT) Sender: Namhyung Kim From: Namhyung Kim To: Arnaldo Carvalho de Melo , Jiri Olsa Cc: Ingo Molnar , Peter Zijlstra , LKML , Ian Rogers , linux-perf-users@vger.kernel.org, Will Deacon , Waiman Long , Boqun Feng , Davidlohr Bueso Subject: [PATCH 3/5] perf lock: Handle lock contention tracepoints Date: Tue, 31 May 2022 23:58:44 -0700 Message-Id: <20220601065846.456965-4-namhyung@kernel.org> X-Mailer: git-send-email 2.36.1.255.ge46751e96f-goog In-Reply-To: <20220601065846.456965-1-namhyung@kernel.org> References: <20220601065846.456965-1-namhyung@kernel.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" When the lock contention events are used, there's no tracking of acquire and release. So the state machine is simplified to use UNINITIALIZED -> CONTENDED -> ACQUIRED only. Note that CONTENDED state is re-entrant since mutex locks can hit two or more consecutive contention_begin events for optimistic spinning and sleep. Signed-off-by: Namhyung Kim Acked-by: Ian Rogers --- tools/perf/builtin-lock.c | 125 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 125 insertions(+) diff --git a/tools/perf/builtin-lock.c b/tools/perf/builtin-lock.c index 3e3320b8cede..82cbf0f2e4cd 100644 --- a/tools/perf/builtin-lock.c +++ b/tools/perf/builtin-lock.c @@ -789,6 +789,124 @@ static int report_lock_release_event(struct evsel *ev= sel, return 0; } =20 +static int report_lock_contention_begin_event(struct evsel *evsel, + struct perf_sample *sample) +{ + struct lock_stat *ls; + struct thread_stat *ts; + struct lock_seq_stat *seq; + u64 addr =3D evsel__intval(evsel, sample, "lock_addr"); + + if (show_thread_stats) + addr =3D sample->tid; + + ls =3D lock_stat_findnew(addr, "No name"); + if (!ls) + return -ENOMEM; + + ts =3D thread_stat_findnew(sample->tid); + if (!ts) + return -ENOMEM; + + seq =3D get_seq(ts, addr); + if (!seq) + return -ENOMEM; + + switch (seq->state) { + case SEQ_STATE_UNINITIALIZED: + case SEQ_STATE_ACQUIRED: + break; + case SEQ_STATE_CONTENDED: + /* + * It can have nested contention begin with mutex spinning, + * then we would use the original contention begin event and + * ignore the second one. + */ + goto end; + case SEQ_STATE_ACQUIRING: + case SEQ_STATE_READ_ACQUIRED: + case SEQ_STATE_RELEASED: + /* broken lock sequence */ + if (!ls->broken) { + ls->broken =3D 1; + bad_hist[BROKEN_CONTENDED]++; + } + list_del_init(&seq->list); + free(seq); + goto end; + default: + BUG_ON("Unknown state of lock sequence found!\n"); + break; + } + + if (seq->state !=3D SEQ_STATE_CONTENDED) { + seq->state =3D SEQ_STATE_CONTENDED; + seq->prev_event_time =3D sample->time; + ls->nr_contended++; + } +end: + return 0; +} + +static int report_lock_contention_end_event(struct evsel *evsel, + struct perf_sample *sample) +{ + struct lock_stat *ls; + struct thread_stat *ts; + struct lock_seq_stat *seq; + u64 contended_term; + u64 addr =3D evsel__intval(evsel, sample, "lock_addr"); + + if (show_thread_stats) + addr =3D sample->tid; + + ls =3D lock_stat_findnew(addr, "No name"); + if (!ls) + return -ENOMEM; + + ts =3D thread_stat_findnew(sample->tid); + if (!ts) + return -ENOMEM; + + seq =3D get_seq(ts, addr); + if (!seq) + return -ENOMEM; + + switch (seq->state) { + case SEQ_STATE_UNINITIALIZED: + goto end; + case SEQ_STATE_CONTENDED: + contended_term =3D sample->time - seq->prev_event_time; + ls->wait_time_total +=3D contended_term; + if (contended_term < ls->wait_time_min) + ls->wait_time_min =3D contended_term; + if (ls->wait_time_max < contended_term) + ls->wait_time_max =3D contended_term; + break; + case SEQ_STATE_ACQUIRING: + case SEQ_STATE_ACQUIRED: + case SEQ_STATE_READ_ACQUIRED: + case SEQ_STATE_RELEASED: + /* broken lock sequence */ + if (!ls->broken) { + ls->broken =3D 1; + bad_hist[BROKEN_CONTENDED]++; + } + list_del_init(&seq->list); + free(seq); + goto end; + default: + BUG_ON("Unknown state of lock sequence found!\n"); + break; + } + + seq->state =3D SEQ_STATE_ACQUIRED; + ls->nr_acquired++; + ls->avg_wait_time =3D ls->wait_time_total/ls->nr_acquired; +end: + return 0; +} + /* lock oriented handlers */ /* TODO: handlers for CPU oriented, thread oriented */ static struct trace_lock_handler report_lock_ops =3D { @@ -796,6 +914,8 @@ static struct trace_lock_handler report_lock_ops =3D { .acquired_event =3D report_lock_acquired_event, .contended_event =3D report_lock_contended_event, .release_event =3D report_lock_release_event, + .contention_begin_event =3D report_lock_contention_begin_event, + .contention_end_event =3D report_lock_contention_end_event, }; =20 static struct trace_lock_handler *trace_handler; @@ -1081,6 +1201,11 @@ static int __cmd_report(bool display_info) goto out_delete; } =20 + if (perf_session__set_tracepoints_handlers(session, contention_tracepoint= s)) { + pr_err("Initializing perf session tracepoint handlers failed\n"); + goto out_delete; + } + if (setup_output_field(output_fields)) goto out_delete; =20 --=20 2.36.1.255.ge46751e96f-goog From nobody Thu Apr 30 09:29:53 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1A83AC433EF for ; Wed, 1 Jun 2022 06:59:13 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1350094AbiFAG7K (ORCPT ); Wed, 1 Jun 2022 02:59:10 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43036 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1350064AbiFAG6y (ORCPT ); Wed, 1 Jun 2022 02:58:54 -0400 Received: from mail-pj1-x102c.google.com (mail-pj1-x102c.google.com [IPv6:2607:f8b0:4864:20::102c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 421632B182; Tue, 31 May 2022 23:58:54 -0700 (PDT) Received: by mail-pj1-x102c.google.com with SMTP id v5-20020a17090a7c0500b001df84fa82f8so1174535pjf.5; Tue, 31 May 2022 23:58:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=sender:from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=NjKuGihzL/MP0z9+OCRyWKWmjzrMu46K8cjCG0rnT4o=; b=qO0Nup+l63oUcIC4Fx3l5w+xOQo/ReDtZIkCcVF+cml3GF/+s6zBmZ49+REKewd9PQ Cgp2Xumia7koGuVg+ww9V2x0fZAP8zyj55u3PrmPaezDvEiijFYcxhub6dBg+HPpkDkW c6Jbipz+vOrBVxAM5L1zR23AYoXmE1uDMDFwPr9+XpxETZRQ54JCzPzOjEuH9Jlu8Cv/ To9bR+KTVtY0KyZ84Q1SYGlbfO+AiRBRoYEr8IS70hkKec9GgvwyZyoqGEonFvyPF8+1 Tcg5KEnqsNL7rv2WfLSekgHVJ0q9MNkqbTT8Oz726UsSh0NGj75duQmsK8DHb7tyvF5c utHA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:sender:from:to:cc:subject:date:message-id :in-reply-to:references:mime-version:content-transfer-encoding; bh=NjKuGihzL/MP0z9+OCRyWKWmjzrMu46K8cjCG0rnT4o=; b=bCtTZkrFDdT2GzVwiRjdBLYE/onLUowR6up60bINcDmf6sA4bIGJI101kgDdH+1w2M uO5MPldN6F8gaQIfjBc6BddvfJcyNSD2VOV17gJSrHxrAf7txDswSfyhFa1Fl9WEbFc6 VUD/Z864KXGwGZW0BKjeJRjk33dU7gkUm/I0Cf/foP+KadGavv7k5uqbOCC8SKSYpphr xXZ1gDnesnx7gDjrROBePzAGj0FbZm6mHczAEE9e4817yglXNG5GPqLWHZWX7VvW9/XW Vawrxj+2hcv22uMkIEB9OaKzXlIv7BIsRpfHWI82/yAvLs6ahGKWyjNKoI6uDyoa058g kNVg== X-Gm-Message-State: AOAM531NJc5zSUqnUjHq2PbEQoq2KoZ7mcn9bCykxzS3OK3olRKnrKCZ NlUXsm3rjDOyx8ecYGPxQfI= X-Google-Smtp-Source: ABdhPJxmwnu1AQmw42c3A7SvAwXsGJqnihla9cDw0ZjcapNtFWFhBz096TPq3vcqUlctmfRt9vXCgA== X-Received: by 2002:a17:902:e849:b0:163:df09:9082 with SMTP id t9-20020a170902e84900b00163df099082mr13509559plg.163.1654066733779; Tue, 31 May 2022 23:58:53 -0700 (PDT) Received: from balhae.hsd1.ca.comcast.net ([2601:647:6780:cb0:82d7:adf7:b62d:6020]) by smtp.gmail.com with ESMTPSA id b10-20020a17090a5a0a00b001cd4989febcsm3207271pjd.8.2022.05.31.23.58.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 31 May 2022 23:58:53 -0700 (PDT) Sender: Namhyung Kim From: Namhyung Kim To: Arnaldo Carvalho de Melo , Jiri Olsa Cc: Ingo Molnar , Peter Zijlstra , LKML , Ian Rogers , linux-perf-users@vger.kernel.org, Will Deacon , Waiman Long , Boqun Feng , Davidlohr Bueso Subject: [PATCH 4/5] perf record: Allow to specify max stack depth of fp callchain Date: Tue, 31 May 2022 23:58:45 -0700 Message-Id: <20220601065846.456965-5-namhyung@kernel.org> X-Mailer: git-send-email 2.36.1.255.ge46751e96f-goog In-Reply-To: <20220601065846.456965-1-namhyung@kernel.org> References: <20220601065846.456965-1-namhyung@kernel.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Currently it has no interface to specify the max stack depth for perf record. Extend the command line parameter to accept a number after 'fp' to specify the depth like '--call-graph fp,32'. Signed-off-by: Namhyung Kim --- tools/perf/Documentation/perf-record.txt | 5 +++++ tools/perf/util/callchain.c | 18 ++++++++++++------ 2 files changed, 17 insertions(+), 6 deletions(-) diff --git a/tools/perf/Documentation/perf-record.txt b/tools/perf/Document= ation/perf-record.txt index b4e9ef7edfef..d2759e2ccf9e 100644 --- a/tools/perf/Documentation/perf-record.txt +++ b/tools/perf/Documentation/perf-record.txt @@ -275,6 +275,11 @@ OPTIONS User can change the size by passing the size after comma like "--call-graph dwarf,4096". =20 + When "fp" recording is used, perf tries to save stack enties + up to the number specified in sysctl.kernel.perf_event_max_stack + by default. User can change the number by passing it after comma + like "--call-graph fp,32". + -q:: --quiet:: Don't print any message, useful for scripting. diff --git a/tools/perf/util/callchain.c b/tools/perf/util/callchain.c index 5c27a4b2e7a7..7e663673f79f 100644 --- a/tools/perf/util/callchain.c +++ b/tools/perf/util/callchain.c @@ -31,6 +31,7 @@ #include "callchain.h" #include "branch.h" #include "symbol.h" +#include "util.h" #include "../perf.h" =20 #define CALLCHAIN_PARAM_DEFAULT \ @@ -266,12 +267,17 @@ int parse_callchain_record(const char *arg, struct ca= llchain_param *param) do { /* Framepointer style */ if (!strncmp(name, "fp", sizeof("fp"))) { - if (!strtok_r(NULL, ",", &saveptr)) { - param->record_mode =3D CALLCHAIN_FP; - ret =3D 0; - } else - pr_err("callchain: No more arguments " - "needed for --call-graph fp\n"); + ret =3D 0; + param->record_mode =3D CALLCHAIN_FP; + + tok =3D strtok_r(NULL, ",", &saveptr); + if (tok) { + unsigned long size; + + size =3D strtoul(tok, &name, 0); + if (size < (unsigned) sysctl__max_stack()) + param->max_stack =3D size; + } break; =20 /* Dwarf style */ --=20 2.36.1.255.ge46751e96f-goog From nobody Thu Apr 30 09:29:53 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 62703C433F5 for ; Wed, 1 Jun 2022 06:59:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1350101AbiFAG7N (ORCPT ); Wed, 1 Jun 2022 02:59:13 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43166 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1350068AbiFAG64 (ORCPT ); Wed, 1 Jun 2022 02:58:56 -0400 Received: from mail-pj1-x1036.google.com (mail-pj1-x1036.google.com [IPv6:2607:f8b0:4864:20::1036]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8E08F2BB1F; Tue, 31 May 2022 23:58:55 -0700 (PDT) Received: by mail-pj1-x1036.google.com with SMTP id nn3-20020a17090b38c300b001e0e091cf03so3212767pjb.1; Tue, 31 May 2022 23:58:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=sender:from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=1KjHGRCb/gwEYrdUgbK9V5JpdX1zhq09kZ/uUwYFWDo=; b=e4rZEv+qVKlgVHLRHoVI0W2nN2xwzY9sq++plrmNvKGX2bzSmIrY2RFs5hJmIyummw xDb6XjGhYkphZdO7VfbR1hlfRVQgcjrUSLJbiRrmFgwv2JdVZFfGhkSAMMFVcl6K7NVy UngelsP1WPInrx/Jt1iQo7oGupnWOawqo48UFw7A0wkHbQ+IqVP7Y3KPPw0bX/p1Fbu1 IJmXRuUWrukog8E6+EU5HnoNsUlRJELl7NPVqRQ9qGwegkWSB3Pk9Yk4HsmLNfTf0nXb /11iTDoypwMGZOIGsOBI2x+hP1SF2817mZv8Sdf0f7o3nlbqwRw3j+LpdzPdHh+cjXzz 7ALg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:sender:from:to:cc:subject:date:message-id :in-reply-to:references:mime-version:content-transfer-encoding; bh=1KjHGRCb/gwEYrdUgbK9V5JpdX1zhq09kZ/uUwYFWDo=; b=t+oTbu0KSGFDKIRHER6CdJ0Tt8z0F0BRR5MgApgNOHScNbiBCtMhwrHZ6T3w0YHR/x rqNW+0aEzZUP59BIbeUZpZDE45UoFF+7i/wcnfPZYQX9/IEQ82kRPq+jtLrvOYCH4z+N 8zqwd4gbkAtApSx2NaDmOaj8u4colZ+6Oodw8HU7o8omhPrKaw5JtGBxpXwS5Iu/mYYH wlXTIl0mL0Is0bYW5ujx/UVwtQoQfJLOwLOodJKpQ/lnBlDo9BYRaZmJ//SH53pDgx/o zNh+f4PQCU03rdjW3nXue4LtQsvVqMo/geD2cV37bdWrv2vQVNrekRuI5nxaguJsqYY6 WE6g== X-Gm-Message-State: AOAM5307PM5Yt1Kdk8+PExqj8Bu2xelXLrhejQ2AXmK4/ST971U0hDxg MLTc0NDhRX+ZeE9FLTenrEA= X-Google-Smtp-Source: ABdhPJwUHW0GMofOL8QRaZlHc/siRsCjjL62ZNz/Lq0p2eVYBEJUuoU9AFwGGDOWnIddMD55XMGSgQ== X-Received: by 2002:a17:902:7c0c:b0:161:ef0f:9da8 with SMTP id x12-20020a1709027c0c00b00161ef0f9da8mr28156048pll.147.1654066735036; Tue, 31 May 2022 23:58:55 -0700 (PDT) Received: from balhae.hsd1.ca.comcast.net ([2601:647:6780:cb0:82d7:adf7:b62d:6020]) by smtp.gmail.com with ESMTPSA id b10-20020a17090a5a0a00b001cd4989febcsm3207271pjd.8.2022.05.31.23.58.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 31 May 2022 23:58:54 -0700 (PDT) Sender: Namhyung Kim From: Namhyung Kim To: Arnaldo Carvalho de Melo , Jiri Olsa Cc: Ingo Molnar , Peter Zijlstra , LKML , Ian Rogers , linux-perf-users@vger.kernel.org, Will Deacon , Waiman Long , Boqun Feng , Davidlohr Bueso Subject: [PATCH 5/5] perf lock: Look up callchain for the contended locks Date: Tue, 31 May 2022 23:58:46 -0700 Message-Id: <20220601065846.456965-6-namhyung@kernel.org> X-Mailer: git-send-email 2.36.1.255.ge46751e96f-goog In-Reply-To: <20220601065846.456965-1-namhyung@kernel.org> References: <20220601065846.456965-1-namhyung@kernel.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" The lock contention tracepoints don't provide lock names. All we can do is to get stack traces and show the caller instead. To minimize the overhead it's limited to up to 8 stack traces and display the first non-lock function symbol name as a caller. $ perf lock report -F acquired,contended,avg_wait Name acquired contended avg wait (ns) update_blocked_a... 40 40 3611 kernfs_fop_open+... 5 5 3636 _nohz_idle_balance 3 3 2650 tick_do_update_j... 1 1 6041 ep_scan_ready_list 1 1 3930 ... Signed-off-by: Namhyung Kim Acked-by: Ian Rogers --- tools/perf/builtin-lock.c | 165 +++++++++++++++++++++++++++++++++++++- 1 file changed, 161 insertions(+), 4 deletions(-) diff --git a/tools/perf/builtin-lock.c b/tools/perf/builtin-lock.c index 82cbf0f2e4cd..46a5b3b50f0b 100644 --- a/tools/perf/builtin-lock.c +++ b/tools/perf/builtin-lock.c @@ -9,6 +9,7 @@ #include "util/symbol.h" #include "util/thread.h" #include "util/header.h" +#include "util/callchain.h" =20 #include #include @@ -19,6 +20,7 @@ #include "util/tool.h" #include "util/data.h" #include "util/string2.h" +#include "util/map.h" =20 #include #include @@ -32,6 +34,7 @@ #include #include #include +#include =20 static struct perf_session *session; =20 @@ -120,6 +123,17 @@ static struct rb_root thread_stats; static bool combine_locks; static bool show_thread_stats; =20 +/* + * CONTENTION_STACK_DEPTH + * Number of stack trace entries to find callers + */ +#define CONTENTION_STACK_DEPTH 8 + +static u64 sched_text_start; +static u64 sched_text_end; +static u64 lock_text_start; +static u64 lock_text_end; + static struct thread_stat *thread_stat_find(u32 tid) { struct rb_node *node; @@ -451,6 +465,18 @@ static struct lock_stat *pop_from_result(void) return container_of(node, struct lock_stat, rb); } =20 +static struct lock_stat *lock_stat_find(u64 addr) +{ + struct hlist_head *entry =3D lockhashentry(addr); + struct lock_stat *ret; + + hlist_for_each_entry(ret, entry, hash_entry) { + if (ret->addr =3D=3D addr) + return ret; + } + return NULL; +} + static struct lock_stat *lock_stat_findnew(u64 addr, const char *name) { struct hlist_head *entry =3D lockhashentry(addr); @@ -789,6 +815,116 @@ static int report_lock_release_event(struct evsel *ev= sel, return 0; } =20 +static bool is_lock_function(u64 addr) +{ + if (!sched_text_start) { + struct machine *machine =3D &session->machines.host; + struct map *kmap; + struct symbol *sym; + + sym =3D machine__find_kernel_symbol_by_name(machine, + "__sched_text_start", + &kmap); + if (!sym) { + /* to avoid retry */ + sched_text_start =3D 1; + return false; + } + + sched_text_start =3D kmap->unmap_ip(kmap, sym->start); + + /* should not fail from here */ + sym =3D machine__find_kernel_symbol_by_name(machine, + "__sched_text_end", + &kmap); + sched_text_end =3D kmap->unmap_ip(kmap, sym->start); + + sym =3D machine__find_kernel_symbol_by_name(machine, + "__lock_text_start", + &kmap); + lock_text_start =3D kmap->unmap_ip(kmap, sym->start); + + sym =3D machine__find_kernel_symbol_by_name(machine, + "__lock_text_end", + &kmap); + lock_text_start =3D kmap->unmap_ip(kmap, sym->start); + } + + /* failed to get kernel symbols */ + if (sched_text_start =3D=3D 1) + return false; + + /* mutex and rwsem functions are in sched text section */ + if (sched_text_start <=3D addr && addr < sched_text_end) + return true; + + /* spinlock functions are in lock text section */ + if (lock_text_start <=3D addr && addr < lock_text_end) + return true; + + return false; +} + +static int lock_contention_caller(struct evsel *evsel, struct perf_sample = *sample, + char *buf, int size) +{ + struct thread *thread; + struct callchain_cursor *cursor =3D &callchain_cursor; + struct symbol *sym; + int skip =3D 0; + int ret; + + /* lock names will be replaced to task name later */ + if (show_thread_stats) + return -1; + + thread =3D machine__findnew_thread(&session->machines.host, + -1, sample->pid); + if (thread =3D=3D NULL) + return -1; + + /* use caller function name from the callchain */ + ret =3D thread__resolve_callchain(thread, cursor, evsel, sample, + NULL, NULL, CONTENTION_STACK_DEPTH); + if (ret !=3D 0) { + thread__put(thread); + return -1; + } + + callchain_cursor_commit(cursor); + thread__put(thread); + + while (true) { + struct callchain_cursor_node *node; + + node =3D callchain_cursor_current(cursor); + if (node =3D=3D NULL) + break; + + /* skip first 3 entries - for lock functions */ + if (++skip <=3D 3) + goto next; + + sym =3D node->ms.sym; + if (sym && !is_lock_function(node->ip)) { + struct map *map =3D node->ms.map; + u64 offset; + + offset =3D map->map_ip(map, node->ip) - sym->start; + + if (offset) + scnprintf(buf, size, "%s+%#lx", sym->name, offset); + else + strlcpy(buf, sym->name, size); + return 0; + } + +next: + callchain_cursor_advance(cursor); + } + return -1; +} + static int report_lock_contention_begin_event(struct evsel *evsel, struct perf_sample *sample) { @@ -800,9 +936,18 @@ static int report_lock_contention_begin_event(struct e= vsel *evsel, if (show_thread_stats) addr =3D sample->tid; =20 - ls =3D lock_stat_findnew(addr, "No name"); - if (!ls) - return -ENOMEM; + ls =3D lock_stat_find(addr); + if (!ls) { + char buf[128]; + const char *caller =3D buf; + + if (lock_contention_caller(evsel, sample, buf, sizeof(buf)) < 0) + caller =3D "Unknown"; + + ls =3D lock_stat_findnew(addr, caller); + if (!ls) + return -ENOMEM; + } =20 ts =3D thread_stat_findnew(sample->tid); if (!ts) @@ -1176,6 +1321,7 @@ static int __cmd_report(bool display_info) struct perf_tool eops =3D { .sample =3D process_sample_event, .comm =3D perf_event__process_comm, + .mmap =3D perf_event__process_mmap, .namespaces =3D perf_event__process_namespaces, .ordered_events =3D true, }; @@ -1191,6 +1337,8 @@ static int __cmd_report(bool display_info) return PTR_ERR(session); } =20 + /* for lock function check */ + symbol_conf.sort_by_name =3D true; symbol__init(&session->header.env); =20 if (!perf_session__has_traces(session, "lock record")) @@ -1235,8 +1383,12 @@ static int __cmd_record(int argc, const char **argv) const char *record_args[] =3D { "record", "-R", "-m", "1024", "-c", "1", "--synth", "task", }; + const char *callgraph_args[] =3D { + "--call-graph", "fp," __stringify(CONTENTION_STACK_DEPTH), + }; unsigned int rec_argc, i, j, ret; unsigned int nr_tracepoints; + unsigned int nr_callgraph_args =3D 0; const char **rec_argv; bool has_lock_stat =3D true; =20 @@ -1261,8 +1413,10 @@ static int __cmd_record(int argc, const char **argv) } } =20 + nr_callgraph_args =3D ARRAY_SIZE(callgraph_args); + setup_args: - rec_argc =3D ARRAY_SIZE(record_args) + argc - 1; + rec_argc =3D ARRAY_SIZE(record_args) + nr_callgraph_args + argc - 1; =20 if (has_lock_stat) nr_tracepoints =3D ARRAY_SIZE(lock_tracepoints); @@ -1294,6 +1448,9 @@ static int __cmd_record(int argc, const char **argv) rec_argv[i++] =3D ev_name; } =20 + for (j =3D 0; j < nr_callgraph_args; j++, i++) + rec_argv[i] =3D callgraph_args[j]; + for (j =3D 1; j < (unsigned int)argc; j++, i++) rec_argv[i] =3D argv[j]; =20 --=20 2.36.1.255.ge46751e96f-goog