From nobody Sun Jun 14 21:08:39 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 19A6DECAAD2 for ; Mon, 29 Aug 2022 10:15:23 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230062AbiH2KPV (ORCPT ); Mon, 29 Aug 2022 06:15:21 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43180 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229656AbiH2KO6 (ORCPT ); Mon, 29 Aug 2022 06:14:58 -0400 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D3671327 for ; Mon, 29 Aug 2022 03:14:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=69XAIoGDwel9FBx285nF3+WxoH4y8KYpY4nBuO7IJh8=; b=GhXLVXKnL7l+3MH9wBD/NYWhEn zu9los2vnQFOrquulJccyn0yUSmujtV8CQI3FJINzMpNcKfYd2v3KUaqBP2FMUnuz7IFU4rQ4KF4e vmGnDBsI77Nv6lUw/ZIyE3ZyLd04qD5PX0NDkZWChxvAfedovn21p4fv5QJM/OfClVgL4gQWv5fQ0 +xMYi9/CMvZQ/TBeIEC65iV/0SmDutQJkx8q68XIpEizwH76MfPkVVw9zkSiT53m1qk1pU8cFCmHc Rd2f0CDeAvUz2Y8y96RbZ03jazj6R8OrspQD8SmQdElkGY4uX1caIdOYK6gDtJHOazhHrNSde6wJT wm9jZUdw==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1oSbn1-00304t-5k; Mon, 29 Aug 2022 10:14:45 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 830DF300137; Mon, 29 Aug 2022 12:14:41 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 6E3FA20099A4D; Mon, 29 Aug 2022 12:14:41 +0200 (CEST) Message-ID: <20220829101321.440196408@infradead.org> User-Agent: quilt/0.66 Date: Mon, 29 Aug 2022 12:10:00 +0200 From: Peter Zijlstra To: x86@kernel.org, kan.liang@linux.intel.com, eranian@google.com, ravi.bangoria@amd.com Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, acme@kernel.org, mark.rutland@arm.com, alexander.shishkin@linux.intel.com, jolsa@kernel.org, namhyung@kernel.org Subject: [PATCH v2 1/9] perf/x86: Add two more x86_pmu methods References: <20220829100959.917169441@infradead.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" In order to clean up x86_perf_event_{set_period,update)() start by adding them as x86_pmu methods. Signed-off-by: Peter Zijlstra (Intel) --- arch/x86/events/core.c | 22 +++++++++++++++++----- arch/x86/events/perf_event.h | 5 +++++ 2 files changed, 22 insertions(+), 5 deletions(-) --- a/arch/x86/events/core.c +++ b/arch/x86/events/core.c @@ -72,6 +72,9 @@ DEFINE_STATIC_CALL_NULL(x86_pmu_add, *x DEFINE_STATIC_CALL_NULL(x86_pmu_del, *x86_pmu.del); DEFINE_STATIC_CALL_NULL(x86_pmu_read, *x86_pmu.read); =20 +DEFINE_STATIC_CALL_NULL(x86_pmu_set_period, *x86_pmu.set_period); +DEFINE_STATIC_CALL_NULL(x86_pmu_update, *x86_pmu.update); + DEFINE_STATIC_CALL_NULL(x86_pmu_schedule_events, *x86_pmu.schedule_e= vents); DEFINE_STATIC_CALL_NULL(x86_pmu_get_event_constraints, *x86_pmu.get_event_= constraints); DEFINE_STATIC_CALL_NULL(x86_pmu_put_event_constraints, *x86_pmu.put_event_= constraints); @@ -1518,7 +1521,7 @@ static void x86_pmu_start(struct perf_ev =20 if (flags & PERF_EF_RELOAD) { WARN_ON_ONCE(!(event->hw.state & PERF_HES_UPTODATE)); - x86_perf_event_set_period(event); + static_call(x86_pmu_set_period)(event); } =20 event->hw.state =3D 0; @@ -1610,7 +1613,7 @@ void x86_pmu_stop(struct perf_event *eve * Drain the remaining delta count out of a event * that we are disabling: */ - x86_perf_event_update(event); + static_call(x86_pmu_update)(event); hwc->state |=3D PERF_HES_UPTODATE; } } @@ -1700,7 +1703,7 @@ int x86_pmu_handle_irq(struct pt_regs *r =20 event =3D cpuc->events[idx]; =20 - val =3D x86_perf_event_update(event); + val =3D static_call(x86_pmu_update)(event); if (val & (1ULL << (x86_pmu.cntval_bits - 1))) continue; =20 @@ -1709,7 +1712,7 @@ int x86_pmu_handle_irq(struct pt_regs *r */ handled++; =20 - if (!x86_perf_event_set_period(event)) + if (!static_call(x86_pmu_set_period)(event)) continue; =20 perf_sample_data_init(&data, 0, event->hw.last_period); @@ -2023,6 +2026,9 @@ static void x86_pmu_static_call_update(v static_call_update(x86_pmu_del, x86_pmu.del); static_call_update(x86_pmu_read, x86_pmu.read); =20 + static_call_update(x86_pmu_set_period, x86_pmu.set_period); + static_call_update(x86_pmu_update, x86_pmu.update); + static_call_update(x86_pmu_schedule_events, x86_pmu.schedule_events); static_call_update(x86_pmu_get_event_constraints, x86_pmu.get_event_const= raints); static_call_update(x86_pmu_put_event_constraints, x86_pmu.put_event_const= raints); @@ -2042,7 +2048,7 @@ static void x86_pmu_static_call_update(v =20 static void _x86_pmu_read(struct perf_event *event) { - x86_perf_event_update(event); + static_call(x86_pmu_update)(event); } =20 void x86_pmu_show_pmu_cap(int num_counters, int num_counters_fixed, @@ -2148,6 +2154,12 @@ static int __init init_hw_perf_events(vo if (!x86_pmu.guest_get_msrs) x86_pmu.guest_get_msrs =3D (void *)&__static_call_return0; =20 + if (!x86_pmu.set_period) + x86_pmu.set_period =3D x86_perf_event_set_period; + + if (!x86_pmu.update) + x86_pmu.update =3D x86_perf_event_update; + x86_pmu_static_call_update(); =20 /* --- a/arch/x86/events/perf_event.h +++ b/arch/x86/events/perf_event.h @@ -735,6 +735,8 @@ struct x86_pmu { void (*add)(struct perf_event *); void (*del)(struct perf_event *); void (*read)(struct perf_event *event); + int (*set_period)(struct perf_event *event); + u64 (*update)(struct perf_event *event); int (*hw_config)(struct perf_event *event); int (*schedule_events)(struct cpu_hw_events *cpuc, int n, int *assign); unsigned eventsel; @@ -1031,6 +1033,9 @@ static struct perf_pmu_format_hybrid_att struct pmu *x86_get_pmu(unsigned int cpu); extern struct x86_pmu x86_pmu __read_mostly; =20 +DECLARE_STATIC_CALL(x86_pmu_set_period, *x86_pmu.set_period); +DECLARE_STATIC_CALL(x86_pmu_update, *x86_pmu.update); + static __always_inline struct x86_perf_task_context_opt *task_context_opt(= void *ctx) { if (static_cpu_has(X86_FEATURE_ARCH_LBR)) From nobody Sun Jun 14 21:08:39 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4EB40ECAAD2 for ; Mon, 29 Aug 2022 10:15:14 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230033AbiH2KPL (ORCPT ); Mon, 29 Aug 2022 06:15:11 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43180 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229738AbiH2KOy (ORCPT ); Mon, 29 Aug 2022 06:14:54 -0400 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D5C5426EA for ; Mon, 29 Aug 2022 03:14:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=6ox9SsphMCojor9sb9ByAEKufXYN5OOq6Tkw9aA9+6M=; b=Xes8KpCX74mlxgfducWVp2+rUQ 6YW8fFvyNktPbnwcG3c+91k/RMRUPa75MCjWRNC1g8R7rG76/m8sTGwomxL/ns28tJtah7NE8Q+jW CTDa1G9YFBXXogi9fpEntBpPBda5z9Ung7pUjPpx43DOCqNDMUxlUclWyTd40Y8nPN8w3vewEtENN C4DruyRoyyo2YkdJpwzAASqTylgQMEr+6x6Zim53/D9TKl+kh/E1tIGyzTCLJW8rjNlSTpKABMn0D x1NwtUCZDmuEmqT2G0Yy70Hs3fVcuflkRFya2AT9fdvT6zNzLVP20jqVQr38Rk2sVWOLYLtrc5I2E UAbMlFhg==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1oSbn1-00304r-3m; Mon, 29 Aug 2022 10:14:43 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 8A80A30034E; Mon, 29 Aug 2022 12:14:41 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 7066F20180FA7; Mon, 29 Aug 2022 12:14:41 +0200 (CEST) Message-ID: <20220829101321.505933457@infradead.org> User-Agent: quilt/0.66 Date: Mon, 29 Aug 2022 12:10:01 +0200 From: Peter Zijlstra To: x86@kernel.org, kan.liang@linux.intel.com, eranian@google.com, ravi.bangoria@amd.com Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, acme@kernel.org, mark.rutland@arm.com, alexander.shishkin@linux.intel.com, jolsa@kernel.org, namhyung@kernel.org Subject: [PATCH v2 2/9] perf/x86/intel: Move the topdown stuff into the intel driver References: <20220829100959.917169441@infradead.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Use the new x86_pmu::{set_period,update}() methods to push the topdown stuff into the Intel driver, where it belongs. Signed-off-by: Peter Zijlstra (Intel) --- arch/x86/events/core.c | 7 ------- arch/x86/events/intel/core.c | 28 +++++++++++++++++++++++++--- 2 files changed, 25 insertions(+), 10 deletions(-) --- a/arch/x86/events/core.c +++ b/arch/x86/events/core.c @@ -119,9 +119,6 @@ u64 x86_perf_event_update(struct perf_ev if (unlikely(!hwc->event_base)) return 0; =20 - if (unlikely(is_topdown_count(event)) && x86_pmu.update_topdown_event) - return x86_pmu.update_topdown_event(event); - /* * Careful: an NMI might modify the previous event value. * @@ -1373,10 +1370,6 @@ int x86_perf_event_set_period(struct per if (unlikely(!hwc->event_base)) return 0; =20 - if (unlikely(is_topdown_count(event)) && - x86_pmu.set_topdown_event_period) - return x86_pmu.set_topdown_event_period(event); - /* * If we are way outside a reasonable range then just skip forward: */ --- a/arch/x86/events/intel/core.c +++ b/arch/x86/events/intel/core.c @@ -2301,7 +2301,7 @@ static void intel_pmu_nhm_workaround(voi for (i =3D 0; i < 4; i++) { event =3D cpuc->events[i]; if (event) - x86_perf_event_update(event); + static_call(x86_pmu_update)(event); } =20 for (i =3D 0; i < 4; i++) { @@ -2316,7 +2316,7 @@ static void intel_pmu_nhm_workaround(voi event =3D cpuc->events[i]; =20 if (event) { - x86_perf_event_set_period(event); + static_call(x86_pmu_set_period)(event); __x86_pmu_enable_event(&event->hw, ARCH_PERFMON_EVENTSEL_ENABLE); } else @@ -2793,7 +2793,7 @@ static void intel_pmu_add_event(struct p */ int intel_pmu_save_and_restart(struct perf_event *event) { - x86_perf_event_update(event); + static_call(x86_pmu_update)(event); /* * For a checkpointed counter always reset back to 0. This * avoids a situation where the counter overflows, aborts the @@ -2805,9 +2805,27 @@ int intel_pmu_save_and_restart(struct pe wrmsrl(event->hw.event_base, 0); local64_set(&event->hw.prev_count, 0); } + return static_call(x86_pmu_set_period)(event); +} + +static int intel_pmu_set_period(struct perf_event *event) +{ + if (unlikely(is_topdown_count(event)) && + x86_pmu.set_topdown_event_period) + return x86_pmu.set_topdown_event_period(event); + return x86_perf_event_set_period(event); } =20 +static u64 intel_pmu_update(struct perf_event *event) +{ + if (unlikely(is_topdown_count(event)) && + x86_pmu.update_topdown_event) + return x86_pmu.update_topdown_event(event); + + return x86_perf_event_update(event); +} + static void intel_pmu_reset(void) { struct debug_store *ds =3D __this_cpu_read(cpu_hw_events.ds); @@ -4635,6 +4653,10 @@ static __initconst const struct x86_pmu .enable_all =3D core_pmu_enable_all, .enable =3D core_pmu_enable_event, .disable =3D x86_pmu_disable_event, + + .set_period =3D intel_pmu_set_period, + .update =3D intel_pmu_update, + .hw_config =3D core_pmu_hw_config, .schedule_events =3D x86_schedule_events, .eventsel =3D MSR_ARCH_PERFMON_EVENTSEL0, From nobody Sun Jun 14 21:08:39 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id EFC06ECAAD2 for ; Mon, 29 Aug 2022 10:15:05 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229902AbiH2KPD (ORCPT ); Mon, 29 Aug 2022 06:15:03 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43168 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229725AbiH2KOy (ORCPT ); Mon, 29 Aug 2022 06:14:54 -0400 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D5A5326DE for ; Mon, 29 Aug 2022 03:14:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=k2xQ/LVf2SOdthxC920Ou1J9VXdYvbj3bbQ75rCmiM4=; b=FruITZ5wMFXzZuMH+u8PpjUs/w A5i5H8qgc1QaGwodBC+QbbupMbVT0/9nuP8lOrRRmCLEXBKIqVhZR1+/OJ7orkaByE0A2R5MZdk9w 5hJoKPY44pg2ly1Eo6ap9Sw2qqa+tupQ78Rl/mmti1aIsRUbSzsz7xUm3QQ9Nbq44K9WqWyXSy67Z vlb8x8sXQp2UnHVGgP7uNQo2gN/PIcauqTJyjpLnWsscvcuPk/zLaCerik+TtI36LbOuP6XHqrjeD tRkNhKVUOMiPegCGwAjoy6spUIzUSmCFOtcWc1hD4NoEKnGR1xFNIUT7nDjT2LSY/mHBQpc26UHLp AV6h9Bnw==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1oSbn1-00304u-5j; Mon, 29 Aug 2022 10:14:43 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 8D219300410; Mon, 29 Aug 2022 12:14:41 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 76E2A20088DAC; Mon, 29 Aug 2022 12:14:41 +0200 (CEST) Message-ID: <20220829101321.573713839@infradead.org> User-Agent: quilt/0.66 Date: Mon, 29 Aug 2022 12:10:02 +0200 From: Peter Zijlstra To: x86@kernel.org, kan.liang@linux.intel.com, eranian@google.com, ravi.bangoria@amd.com Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, acme@kernel.org, mark.rutland@arm.com, alexander.shishkin@linux.intel.com, jolsa@kernel.org, namhyung@kernel.org Subject: [PATCH v2 3/9] perf/x86: Change x86_pmu::limit_period signature References: <20220829100959.917169441@infradead.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" In preparation for making it a static_call, change the signature. Signed-off-by: Peter Zijlstra (Intel) --- arch/x86/events/amd/core.c | 8 +++----- arch/x86/events/core.c | 13 ++++++++----- arch/x86/events/intel/core.c | 19 ++++++++----------- arch/x86/events/perf_event.h | 2 +- 4 files changed, 20 insertions(+), 22 deletions(-) --- a/arch/x86/events/amd/core.c +++ b/arch/x86/events/amd/core.c @@ -1222,16 +1222,14 @@ static ssize_t amd_event_sysfs_show(char return x86_event_sysfs_show(page, config, event); } =20 -static u64 amd_pmu_limit_period(struct perf_event *event, u64 left) +static void amd_pmu_limit_period(struct perf_event *event, s64 *left) { /* * Decrease period by the depth of the BRS feature to get the last N * taken branches and approximate the desired period */ - if (has_branch_stack(event) && left > x86_pmu.lbr_nr) - left -=3D x86_pmu.lbr_nr; - - return left; + if (has_branch_stack(event) && *left > x86_pmu.lbr_nr) + *left -=3D x86_pmu.lbr_nr; } =20 static __initconst const struct x86_pmu amd_pmu =3D { --- a/arch/x86/events/core.c +++ b/arch/x86/events/core.c @@ -621,8 +621,9 @@ int x86_pmu_hw_config(struct perf_event event->hw.config |=3D event->attr.config & X86_RAW_EVENT_MASK; =20 if (event->attr.sample_period && x86_pmu.limit_period) { - if (x86_pmu.limit_period(event, event->attr.sample_period) > - event->attr.sample_period) + s64 left =3D event->attr.sample_period; + x86_pmu.limit_period(event, &left); + if (left > event->attr.sample_period) return -EINVAL; } =20 @@ -1396,9 +1397,9 @@ int x86_perf_event_set_period(struct per left =3D x86_pmu.max_period; =20 if (x86_pmu.limit_period) - left =3D x86_pmu.limit_period(event, left); + x86_pmu.limit_period(event, &left); =20 - per_cpu(pmc_prev_left[idx], smp_processor_id()) =3D left; + this_cpu_write(pmc_prev_left[idx], left); =20 /* * The hw event starts counting from this event offset, @@ -2675,7 +2676,9 @@ static int x86_pmu_check_period(struct p return -EINVAL; =20 if (value && x86_pmu.limit_period) { - if (x86_pmu.limit_period(event, value) > value) + s64 left =3D value; + x86_pmu.limit_period(event, &left); + if (left > value) return -EINVAL; } =20 --- a/arch/x86/events/intel/core.c +++ b/arch/x86/events/intel/core.c @@ -4342,28 +4342,25 @@ static u8 adl_get_hybrid_cpu_type(void) * Therefore the effective (average) period matches the requested period, * despite coarser hardware granularity. */ -static u64 bdw_limit_period(struct perf_event *event, u64 left) +static void bdw_limit_period(struct perf_event *event, s64 *left) { if ((event->hw.config & INTEL_ARCH_EVENT_MASK) =3D=3D X86_CONFIG(.event=3D0xc0, .umask=3D0x01)) { - if (left < 128) - left =3D 128; - left &=3D ~0x3fULL; + if (*left < 128) + *left =3D 128; + *left &=3D ~0x3fULL; } - return left; } =20 -static u64 nhm_limit_period(struct perf_event *event, u64 left) +static void nhm_limit_period(struct perf_event *event, s64 *left) { - return max(left, 32ULL); + *left =3D max(*left, 32LL); } =20 -static u64 spr_limit_period(struct perf_event *event, u64 left) +static void spr_limit_period(struct perf_event *event, s64 *left) { if (event->attr.precise_ip =3D=3D 3) - return max(left, 128ULL); - - return left; + *left =3D max(*left, 128LL); } =20 PMU_FORMAT_ATTR(event, "config:0-7" ); --- a/arch/x86/events/perf_event.h +++ b/arch/x86/events/perf_event.h @@ -783,7 +783,7 @@ struct x86_pmu { struct event_constraint *event_constraints; struct x86_pmu_quirk *quirks; int perfctr_second_write; - u64 (*limit_period)(struct perf_event *event, u64 l); + void (*limit_period)(struct perf_event *event, s64 *l); =20 /* PMI handler bits */ unsigned int late_ack :1, From nobody Sun Jun 14 21:08:39 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8E188ECAAD4 for ; Mon, 29 Aug 2022 10:15:28 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230029AbiH2KPY (ORCPT ); Mon, 29 Aug 2022 06:15:24 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43524 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229822AbiH2KO7 (ORCPT ); Mon, 29 Aug 2022 06:14:59 -0400 Received: from desiato.infradead.org (desiato.infradead.org [IPv6:2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 464037653 for ; Mon, 29 Aug 2022 03:14:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=RrsRwnufM8aJb8X9Do0ueNVFWhI9Rg8k/jxvL6pTy3A=; b=E8SAi3JaFuSLK93DPkIu0UKrrI OFKSUTh1qSx0V9zZokCjaJTstYrM1wcSGcFFsd46hYScOpU0vb7OLEDpEFMCLmBdWnte+NhLwtR6P OXgglvsJ3ikzLmAUNJpVWbkt0qC0wr2pXsAvBSArlJoAgP4+gYGG46FMNgv804GUjK6QbIkxqtS9b gsDx28h+UUePMh8pXHnJhhVvN5cORytqGnV3Kv4tf14SItzM9iODQuXb0DIVNxC87jSzEGJhytjnt R7xwKwq+9vWz3tH14c41AFskXrt3jdFxcqYT3rXylH+oXIIPDHb/o0XegZCLGep9iAvuBRm9P8dYB ZT+d+SFQ==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1oSbn1-007SzJ-Ig; Mon, 29 Aug 2022 10:14:44 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 957103004C7; Mon, 29 Aug 2022 12:14:41 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 7B9F7200A7EB8; Mon, 29 Aug 2022 12:14:41 +0200 (CEST) Message-ID: <20220829101321.640658334@infradead.org> User-Agent: quilt/0.66 Date: Mon, 29 Aug 2022 12:10:03 +0200 From: Peter Zijlstra To: x86@kernel.org, kan.liang@linux.intel.com, eranian@google.com, ravi.bangoria@amd.com Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, acme@kernel.org, mark.rutland@arm.com, alexander.shishkin@linux.intel.com, jolsa@kernel.org, namhyung@kernel.org Subject: [PATCH v2 4/9] perf/x86: Add a x86_pmu::limit_period static_call References: <20220829100959.917169441@infradead.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Avoid a branch and indirect call. Signed-off-by: Peter Zijlstra (Intel) --- arch/x86/events/core.c | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) --- a/arch/x86/events/core.c +++ b/arch/x86/events/core.c @@ -72,8 +72,9 @@ DEFINE_STATIC_CALL_NULL(x86_pmu_add, *x DEFINE_STATIC_CALL_NULL(x86_pmu_del, *x86_pmu.del); DEFINE_STATIC_CALL_NULL(x86_pmu_read, *x86_pmu.read); =20 -DEFINE_STATIC_CALL_NULL(x86_pmu_set_period, *x86_pmu.set_period); -DEFINE_STATIC_CALL_NULL(x86_pmu_update, *x86_pmu.update); +DEFINE_STATIC_CALL_NULL(x86_pmu_set_period, *x86_pmu.set_period); +DEFINE_STATIC_CALL_NULL(x86_pmu_update, *x86_pmu.update); +DEFINE_STATIC_CALL_NULL(x86_pmu_limit_period, *x86_pmu.limit_period); =20 DEFINE_STATIC_CALL_NULL(x86_pmu_schedule_events, *x86_pmu.schedule_e= vents); DEFINE_STATIC_CALL_NULL(x86_pmu_get_event_constraints, *x86_pmu.get_event_= constraints); @@ -1391,8 +1392,7 @@ int x86_perf_event_set_period(struct per if (left > x86_pmu.max_period) left =3D x86_pmu.max_period; =20 - if (x86_pmu.limit_period) - x86_pmu.limit_period(event, &left); + static_call_cond(x86_pmu_limit_period)(event, &left); =20 this_cpu_write(pmc_prev_left[idx], left); =20 @@ -2017,6 +2017,7 @@ static void x86_pmu_static_call_update(v =20 static_call_update(x86_pmu_set_period, x86_pmu.set_period); static_call_update(x86_pmu_update, x86_pmu.update); + static_call_update(x86_pmu_limit_period, x86_pmu.limit_period); =20 static_call_update(x86_pmu_schedule_events, x86_pmu.schedule_events); static_call_update(x86_pmu_get_event_constraints, x86_pmu.get_event_const= raints); From nobody Sun Jun 14 21:08:39 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 91D67ECAAD2 for ; Mon, 29 Aug 2022 10:15:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229994AbiH2KPh (ORCPT ); Mon, 29 Aug 2022 06:15:37 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43696 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229879AbiH2KPC (ORCPT ); Mon, 29 Aug 2022 06:15:02 -0400 Received: from desiato.infradead.org (desiato.infradead.org [IPv6:2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D38D37661 for ; Mon, 29 Aug 2022 03:14:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=Ey8THXCja1PgogTYIuBjl5F4YUuf3vH3bpgVEwluBX0=; b=BTsvGSUHQqUgRy5roqdQfDuzvl xbJVFkW6W7CZ5tcLhphOzYzPDg1YOzQkz8wuEflVRhpOP3bXfCAk8Joiqe1ThbPeGfWQmwdsPV8qT TeoJ7A33joiFEbPdwQzVZ0xTn6z/3qf8hzMCm0fR1np9Msczz7kzyEEiwJXG2wwIvQgwchOau+7RP sf07hbVDxV6CKIY5vxNEMHvIngRgTUjRUbw7hdxFYNS8si5FZOXHTvIAs8Digv/kN1mcao8jLovrH kvwMtVr92zg5c04kTkHD8j1sFofrlqbvX9cdCe3Uwss1I88+p01nWqI5BoqyH1b71pyMDreMr4zzY JkyqqVUQ==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1oSbn1-007SzL-Ic; Mon, 29 Aug 2022 10:14:44 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id D5187300859; Mon, 29 Aug 2022 12:14:41 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 8103F2008A3F5; Mon, 29 Aug 2022 12:14:41 +0200 (CEST) Message-ID: <20220829101321.706354189@infradead.org> User-Agent: quilt/0.66 Date: Mon, 29 Aug 2022 12:10:04 +0200 From: Peter Zijlstra To: x86@kernel.org, kan.liang@linux.intel.com, eranian@google.com, ravi.bangoria@amd.com Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, acme@kernel.org, mark.rutland@arm.com, alexander.shishkin@linux.intel.com, jolsa@kernel.org, namhyung@kernel.org Subject: [PATCH v2 5/9] perf/x86/intel: Remove x86_pmu::set_topdown_event_period References: <20220829100959.917169441@infradead.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Now that it is all internal to the intel driver, remove x86_pmu::set_topdown_event_period. Signed-off-by: Peter Zijlstra (Intel) --- arch/x86/events/intel/core.c | 16 ++++++++++------ arch/x86/events/perf_event.h | 1 - 2 files changed, 10 insertions(+), 7 deletions(-) --- a/arch/x86/events/intel/core.c +++ b/arch/x86/events/intel/core.c @@ -2520,6 +2520,8 @@ static int adl_set_topdown_event_period( return icl_set_topdown_event_period(event); } =20 +DEFINE_STATIC_CALL(intel_pmu_set_topdown_event_period, x86_perf_event_set_= period); + static inline u64 icl_get_metrics_event_value(u64 metric, u64 slots, int i= dx) { u32 val; @@ -2810,9 +2812,8 @@ int intel_pmu_save_and_restart(struct pe =20 static int intel_pmu_set_period(struct perf_event *event) { - if (unlikely(is_topdown_count(event)) && - x86_pmu.set_topdown_event_period) - return x86_pmu.set_topdown_event_period(event); + if (unlikely(is_topdown_count(event))) + return static_call(intel_pmu_set_topdown_event_period)(event); =20 return x86_perf_event_set_period(event); } @@ -6191,7 +6192,8 @@ __init int intel_pmu_init(void) intel_pmu_pebs_data_source_skl(pmem); x86_pmu.num_topdown_events =3D 4; x86_pmu.update_topdown_event =3D icl_update_topdown_event; - x86_pmu.set_topdown_event_period =3D icl_set_topdown_event_period; + static_call_update(intel_pmu_set_topdown_event_period, + &icl_set_topdown_event_period); pr_cont("Icelake events, "); name =3D "icelake"; break; @@ -6228,7 +6230,8 @@ __init int intel_pmu_init(void) intel_pmu_pebs_data_source_skl(pmem); x86_pmu.num_topdown_events =3D 8; x86_pmu.update_topdown_event =3D icl_update_topdown_event; - x86_pmu.set_topdown_event_period =3D icl_set_topdown_event_period; + static_call_update(intel_pmu_set_topdown_event_period, + &icl_set_topdown_event_period); pr_cont("Sapphire Rapids events, "); name =3D "sapphire_rapids"; break; @@ -6264,7 +6267,8 @@ __init int intel_pmu_init(void) x86_pmu.pebs_latency_data =3D adl_latency_data_small; x86_pmu.num_topdown_events =3D 8; x86_pmu.update_topdown_event =3D adl_update_topdown_event; - x86_pmu.set_topdown_event_period =3D adl_set_topdown_event_period; + static_call_update(intel_pmu_set_topdown_event_period, + &adl_set_topdown_event_period); =20 x86_pmu.filter_match =3D intel_pmu_filter_match; x86_pmu.get_event_constraints =3D adl_get_event_constraints; --- a/arch/x86/events/perf_event.h +++ b/arch/x86/events/perf_event.h @@ -890,7 +890,6 @@ struct x86_pmu { */ int num_topdown_events; u64 (*update_topdown_event)(struct perf_event *event); - int (*set_topdown_event_period)(struct perf_event *event); =20 /* * perf task context (i.e. struct perf_event_context::task_ctx_data) From nobody Sun Jun 14 21:08:39 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4B989ECAAD2 for ; Mon, 29 Aug 2022 10:15:32 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230088AbiH2KPa (ORCPT ); Mon, 29 Aug 2022 06:15:30 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43698 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229867AbiH2KPC (ORCPT ); Mon, 29 Aug 2022 06:15:02 -0400 Received: from desiato.infradead.org (desiato.infradead.org [IPv6:2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D3A197663 for ; Mon, 29 Aug 2022 03:14:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=W4onxSZKhwfLLpxsFTRSf73IGco3HKn+5oDZh6sBf9A=; b=OfVrbhROQnbSrlhQeauPRWsn27 l3k8QQuMPM3spyryd4rkdwJx7wc79vjcmq1NxYfBta/LqqGylGwsE6mOBhe5bjCkIzADBqd0P65gt S9p0/xiS2MRMBJEsQnQJG8E0as2zm3gjSCgS0Jd2KqiAORGKFghMm4I8vpa9HwGhoE31A2IOd+wOP EIHyh+jit0o7oyQvSv1aH/IQdma4AYCQtwlaxknYLiDh1V1rOw0avvtYRZ6ZdfLaldbMbI3JIHV66 c0Hf9TzJ9gqEPddx7aQk6kBJ75jFS7bz9jvPSFp9aIN8ifT55x5u32VGqu4NsFZt0tQMJyHQLcb8X Gc1Ln08Q==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1oSbn1-007SzM-J2; Mon, 29 Aug 2022 10:14:44 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id D4F4F3005DB; Mon, 29 Aug 2022 12:14:41 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 83D28207617BD; Mon, 29 Aug 2022 12:14:41 +0200 (CEST) Message-ID: <20220829101321.771635301@infradead.org> User-Agent: quilt/0.66 Date: Mon, 29 Aug 2022 12:10:05 +0200 From: Peter Zijlstra To: x86@kernel.org, kan.liang@linux.intel.com, eranian@google.com, ravi.bangoria@amd.com Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, acme@kernel.org, mark.rutland@arm.com, alexander.shishkin@linux.intel.com, jolsa@kernel.org, namhyung@kernel.org Subject: [PATCH v2 6/9] perf/x86/intel: Remove x86_pmu::update_topdown_event References: <20220829100959.917169441@infradead.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Now that it is all internal to the intel driver, remove x86_pmu::update_topdown_event. Assumes that is_topdown_count(event) can only be true when the hardware has topdown stuff and the function is set. Signed-off-by: Peter Zijlstra (Intel) --- arch/x86/events/intel/core.c | 22 ++++++++++++---------- arch/x86/events/perf_event.h | 1 - 2 files changed, 12 insertions(+), 11 deletions(-) --- a/arch/x86/events/intel/core.c +++ b/arch/x86/events/intel/core.c @@ -2672,6 +2672,7 @@ static u64 adl_update_topdown_event(stru return icl_update_topdown_event(event); } =20 +DEFINE_STATIC_CALL(intel_pmu_update_topdown_event, x86_perf_event_update); =20 static void intel_pmu_read_topdown_event(struct perf_event *event) { @@ -2683,7 +2684,7 @@ static void intel_pmu_read_topdown_event return; =20 perf_pmu_disable(event->pmu); - x86_pmu.update_topdown_event(event); + static_call(intel_pmu_update_topdown_event)(event); perf_pmu_enable(event->pmu); } =20 @@ -2691,7 +2692,7 @@ static void intel_pmu_read_event(struct { if (event->hw.flags & PERF_X86_EVENT_AUTO_RELOAD) intel_pmu_auto_reload_read(event); - else if (is_topdown_count(event) && x86_pmu.update_topdown_event) + else if (is_topdown_count(event)) intel_pmu_read_topdown_event(event); else x86_perf_event_update(event); @@ -2820,9 +2821,8 @@ static int intel_pmu_set_period(struct p =20 static u64 intel_pmu_update(struct perf_event *event) { - if (unlikely(is_topdown_count(event)) && - x86_pmu.update_topdown_event) - return x86_pmu.update_topdown_event(event); + if (unlikely(is_topdown_count(event))) + return static_call(intel_pmu_update_topdown_event)(event); =20 return x86_perf_event_update(event); } @@ -2950,8 +2950,7 @@ static int handle_pmi_common(struct pt_r */ if (__test_and_clear_bit(GLOBAL_STATUS_PERF_METRICS_OVF_BIT, (unsigned lo= ng *)&status)) { handled++; - if (x86_pmu.update_topdown_event) - x86_pmu.update_topdown_event(NULL); + static_call(intel_pmu_update_topdown_event)(NULL); } =20 /* @@ -6191,7 +6190,8 @@ __init int intel_pmu_init(void) x86_pmu.lbr_pt_coexist =3D true; intel_pmu_pebs_data_source_skl(pmem); x86_pmu.num_topdown_events =3D 4; - x86_pmu.update_topdown_event =3D icl_update_topdown_event; + static_call_update(intel_pmu_update_topdown_event, + &icl_update_topdown_event); static_call_update(intel_pmu_set_topdown_event_period, &icl_set_topdown_event_period); pr_cont("Icelake events, "); @@ -6229,7 +6229,8 @@ __init int intel_pmu_init(void) x86_pmu.lbr_pt_coexist =3D true; intel_pmu_pebs_data_source_skl(pmem); x86_pmu.num_topdown_events =3D 8; - x86_pmu.update_topdown_event =3D icl_update_topdown_event; + static_call_update(intel_pmu_update_topdown_event, + &icl_update_topdown_event); static_call_update(intel_pmu_set_topdown_event_period, &icl_set_topdown_event_period); pr_cont("Sapphire Rapids events, "); @@ -6266,7 +6267,8 @@ __init int intel_pmu_init(void) intel_pmu_pebs_data_source_adl(); x86_pmu.pebs_latency_data =3D adl_latency_data_small; x86_pmu.num_topdown_events =3D 8; - x86_pmu.update_topdown_event =3D adl_update_topdown_event; + static_call_update(intel_pmu_update_topdown_event, + &adl_update_topdown_event); static_call_update(intel_pmu_set_topdown_event_period, &adl_set_topdown_event_period); =20 --- a/arch/x86/events/perf_event.h +++ b/arch/x86/events/perf_event.h @@ -889,7 +889,6 @@ struct x86_pmu { * Intel perf metrics */ int num_topdown_events; - u64 (*update_topdown_event)(struct perf_event *event); =20 /* * perf task context (i.e. struct perf_event_context::task_ctx_data) From nobody Sun Jun 14 21:08:39 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id BCD49ECAAD2 for ; Mon, 29 Aug 2022 10:15:18 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229898AbiH2KPQ (ORCPT ); Mon, 29 Aug 2022 06:15:16 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43182 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229743AbiH2KOy (ORCPT ); Mon, 29 Aug 2022 06:14:54 -0400 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D5F9126F7 for ; Mon, 29 Aug 2022 03:14:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=cjwMj5/XK3Arj3+DPDxgPLyq06v8ipOyphQTAJ1oEvk=; b=J9FKQjfLSa3gkKEh5/6FMIOwYh olQqOGNnjxr0fBB1tSswORAa+uMxiqB7kQnBWK1iO119KCY9W2l29Ipz5bNv65Nrj6mT0COVg9mNW 2UvMW282KntYrhZ7W5MGkzXEvwIvM1Wo0DIT79Y03O3QSJDKZUC4Wc/WKzdGnJoCGrMf2BvNFf3TJ B5zqtfTZUIqcD8aetotki/FzeU4klgBZ/oluJi2OaV0eECddys/toZt9m4PLYbycmFy2g69YhtIa6 ALzomFgRK2QmF6vM/FJCiuq8nx87KqUK+EqhoXUlBebku72rJivkOObQKwU5cYjCXftlesV4Oa48A 6hpJv0sQ==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1oSbn1-003053-Nd; Mon, 29 Aug 2022 10:14:43 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id D505B300750; Mon, 29 Aug 2022 12:14:41 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 8FB9F207296FB; Mon, 29 Aug 2022 12:14:41 +0200 (CEST) Message-ID: <20220829101321.839502514@infradead.org> User-Agent: quilt/0.66 Date: Mon, 29 Aug 2022 12:10:06 +0200 From: Peter Zijlstra To: x86@kernel.org, kan.liang@linux.intel.com, eranian@google.com, ravi.bangoria@amd.com Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, acme@kernel.org, mark.rutland@arm.com, alexander.shishkin@linux.intel.com, jolsa@kernel.org, namhyung@kernel.org Subject: [PATCH v2 7/9] perf/x86/p4: Remove perfctr_second_write quirk References: <20220829100959.917169441@infradead.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Now that we have a x86_pmu::set_period() method, use it to remove the perfctr_second_write quirk from the generic code. Signed-off-by: Peter Zijlstra (Intel) --- arch/x86/events/core.c | 12 +----------- arch/x86/events/intel/p4.c | 37 +++++++++++++++++++++++++++---------- arch/x86/events/perf_event.h | 2 +- 3 files changed, 29 insertions(+), 22 deletions(-) --- a/arch/x86/events/core.c +++ b/arch/x86/events/core.c @@ -1356,7 +1356,7 @@ static void x86_pmu_enable(struct pmu *p static_call(x86_pmu_enable_all)(added); } =20 -static DEFINE_PER_CPU(u64 [X86_PMC_IDX_MAX], pmc_prev_left); +DEFINE_PER_CPU(u64 [X86_PMC_IDX_MAX], pmc_prev_left); =20 /* * Set the next IRQ period, based on the hwc->period_left value. @@ -1416,16 +1416,6 @@ int x86_perf_event_set_period(struct per if (is_counter_pair(hwc)) wrmsrl(x86_pmu_event_addr(idx + 1), 0xffff); =20 - /* - * Due to erratum on certan cpu we need - * a second write to be sure the register - * is updated properly - */ - if (x86_pmu.perfctr_second_write) { - wrmsrl(hwc->event_base, - (u64)(-left) & x86_pmu.cntval_mask); - } - perf_event_update_userpage(event); =20 return ret; --- a/arch/x86/events/intel/p4.c +++ b/arch/x86/events/intel/p4.c @@ -1006,6 +1006,29 @@ static void p4_pmu_enable_all(int added) } } =20 +static int p4_pmu_set_period(struct perf_event *event) +{ + struct hw_perf_event *hwc =3D &event->hw; + s64 left =3D this_cpu_read(pmc_prev_left[hwc->idx]); + int ret; + + ret =3D x86_perf_event_set_period(event); + + if (hwc->event_base) { + /* + * This handles erratum N15 in intel doc 249199-029, + * the counter may not be updated correctly on write + * so we need a second write operation to do the trick + * (the official workaround didn't work) + * + * the former idea is taken from OProfile code + */ + wrmsrl(hwc->event_base, (u64)(-left) & x86_pmu.cntval_mask); + } + + return ret; +} + static int p4_pmu_handle_irq(struct pt_regs *regs) { struct perf_sample_data data; @@ -1044,7 +1067,7 @@ static int p4_pmu_handle_irq(struct pt_r /* event overflow for sure */ perf_sample_data_init(&data, 0, hwc->last_period); =20 - if (!x86_perf_event_set_period(event)) + if (!static_call(x86_pmu_set_period)(event)) continue; =20 =20 @@ -1316,6 +1339,9 @@ static __initconst const struct x86_pmu .enable_all =3D p4_pmu_enable_all, .enable =3D p4_pmu_enable_event, .disable =3D p4_pmu_disable_event, + + .set_period =3D p4_pmu_set_period, + .eventsel =3D MSR_P4_BPU_CCCR0, .perfctr =3D MSR_P4_BPU_PERFCTR0, .event_map =3D p4_pmu_event_map, @@ -1334,15 +1360,6 @@ static __initconst const struct x86_pmu .max_period =3D (1ULL << (ARCH_P4_CNTRVAL_BITS - 1)) - 1, .hw_config =3D p4_hw_config, .schedule_events =3D p4_pmu_schedule_events, - /* - * This handles erratum N15 in intel doc 249199-029, - * the counter may not be updated correctly on write - * so we need a second write operation to do the trick - * (the official workaround didn't work) - * - * the former idea is taken from OProfile code - */ - .perfctr_second_write =3D 1, =20 .format_attrs =3D intel_p4_formats_attr, }; --- a/arch/x86/events/perf_event.h +++ b/arch/x86/events/perf_event.h @@ -772,7 +772,6 @@ struct x86_pmu { =20 struct event_constraint *event_constraints; struct x86_pmu_quirk *quirks; - int perfctr_second_write; void (*limit_period)(struct perf_event *event, s64 *l); =20 /* PMI handler bits */ @@ -1049,6 +1048,7 @@ static inline bool x86_pmu_has_lbr_calls } =20 DECLARE_PER_CPU(struct cpu_hw_events, cpu_hw_events); +DECLARE_PER_CPU(u64 [X86_PMC_IDX_MAX], pmc_prev_left); =20 int x86_perf_event_set_period(struct perf_event *event); From nobody Sun Jun 14 21:08:39 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id EA0D5ECAAD2 for ; Mon, 29 Aug 2022 10:15:10 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229725AbiH2KPH (ORCPT ); Mon, 29 Aug 2022 06:15:07 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43008 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229735AbiH2KOy (ORCPT ); Mon, 29 Aug 2022 06:14:54 -0400 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D5F18323 for ; Mon, 29 Aug 2022 03:14:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=BfqUEpfAtKviGOSi1l2C06+GNHziXV+kCVcVUMnSczs=; b=bVG7AOxUlP0Rrd16vth0/ROqe2 SDbP3okR9dkwOKvyOBdNZU7TvrdriFmRGp0wLcQLHkSeB7qWReCDsA+p3UnvcnkQd9ljRiQSfo0N7 fWy+/w54al02jMeCVuDmEfLMs5bLf51dmGcQd9RclCHNHN58fht6G+YCMtuE32dajQsEJEvLpO5kZ DKhQzoLJ25XvoY8x1I7AjVfTHRB/+2mu9xzLtIhAt6GFI8kKetpqRBLakHslDeejpHsH1IFnEsv0a 1oUlKJljxakJuz1VomDr6SK19c9rqBYgDAjQo4stw6uh2KsMFz2fvzJHUgHrgfBC5WPX7c/pqlUyd Jse3gvKA==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1oSbn1-003056-TE; Mon, 29 Aug 2022 10:14:44 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id D515B3007DA; Mon, 29 Aug 2022 12:14:41 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 99784207F2716; Mon, 29 Aug 2022 12:14:41 +0200 (CEST) Message-ID: <20220829101321.905673933@infradead.org> User-Agent: quilt/0.66 Date: Mon, 29 Aug 2022 12:10:07 +0200 From: Peter Zijlstra To: x86@kernel.org, kan.liang@linux.intel.com, eranian@google.com, ravi.bangoria@amd.com Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, acme@kernel.org, mark.rutland@arm.com, alexander.shishkin@linux.intel.com, jolsa@kernel.org, namhyung@kernel.org Subject: [PATCH v2 8/9] perf/x86/intel: Shadow MSR_ARCH_PERFMON_FIXED_CTR_CTRL References: <20220829100959.917169441@infradead.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Less RDMSR is more better. Signed-off-by: Peter Zijlstra (Intel) --- arch/x86/events/intel/core.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) --- a/arch/x86/events/intel/core.c +++ b/arch/x86/events/intel/core.c @@ -2405,6 +2405,8 @@ static inline void intel_clear_masks(str __clear_bit(idx, (unsigned long *)&cpuc->intel_cp_status); } =20 +static DEFINE_PER_CPU(u64, intel_fixed_ctrl); + static void intel_pmu_disable_fixed(struct perf_event *event) { struct hw_perf_event *hwc =3D &event->hw; @@ -2426,8 +2428,9 @@ static void intel_pmu_disable_fixed(stru intel_clear_masks(event, idx); =20 mask =3D 0xfULL << ((idx - INTEL_PMC_IDX_FIXED) * 4); - rdmsrl(hwc->config_base, ctrl_val); + ctrl_val =3D this_cpu_read(intel_fixed_ctrl); ctrl_val &=3D ~mask; + this_cpu_write(intel_fixed_ctrl, ctrl_val); wrmsrl(hwc->config_base, ctrl_val); } =20 @@ -2746,9 +2749,10 @@ static void intel_pmu_enable_fixed(struc mask |=3D ICL_FIXED_0_ADAPTIVE << (idx * 4); } =20 - rdmsrl(hwc->config_base, ctrl_val); + ctrl_val =3D this_cpu_read(intel_fixed_ctrl); ctrl_val &=3D ~mask; ctrl_val |=3D bits; + this_cpu_write(intel_fixed_ctrl, ctrl_val); wrmsrl(hwc->config_base, ctrl_val); } From nobody Sun Jun 14 21:08:39 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7E664ECAAD2 for ; Mon, 29 Aug 2022 10:15:36 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229890AbiH2KPd (ORCPT ); Mon, 29 Aug 2022 06:15:33 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43694 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229884AbiH2KPC (ORCPT ); Mon, 29 Aug 2022 06:15:02 -0400 Received: from desiato.infradead.org (desiato.infradead.org [IPv6:2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D37C2765F for ; Mon, 29 Aug 2022 03:14:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=OQAAMSjRHuYgCLVl5R9t0nt04B2gvVrpsDsjFaKL1VI=; b=GoW94x/AMI1g98fioJPJuyHJF1 KdxLLlUDJNiygWcb/xx2+hyr/PLqQ3xsiFpfbmK9R7ThxZTqfcRXrRdieG7DuaHAiUk0j0F9uus7E vGvIhj0nksX3GdH9NQeJUiTzaOlq+xvTt0f1yi1KKoKSIjGh0WMER0bdwAmpxiQ3htRkz30vyXxX4 Bekf/iMdZh7hE7f8VuxvTJcm5UrNn32qKmkCx0LHWnxTuCCDIRZFQg3dWtYH784xuqFpQuEjXGEjB HQVQ/qLIqgyObVExFXCnOVxIkkt/vlv3POQBrMEZMawZ4haAI/fCaDXFQ+3EwhJMgc+Et+J3/S4wt YP6tW28w==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1oSbn1-007SzK-J5; Mon, 29 Aug 2022 10:14:44 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id D504B3006C4; Mon, 29 Aug 2022 12:14:41 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id A48CD207F7FB4; Mon, 29 Aug 2022 12:14:41 +0200 (CEST) Message-ID: <20220829101321.971473694@infradead.org> User-Agent: quilt/0.66 Date: Mon, 29 Aug 2022 12:10:08 +0200 From: Peter Zijlstra To: x86@kernel.org, kan.liang@linux.intel.com, eranian@google.com, ravi.bangoria@amd.com Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, acme@kernel.org, mark.rutland@arm.com, alexander.shishkin@linux.intel.com, jolsa@kernel.org, namhyung@kernel.org Subject: [PATCH v2 9/9] perf/x86/intel: Optimize short PEBS counters References: <20220829100959.917169441@infradead.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" XXX: crazy idea; really not sure this is worth the extra complexity It is possible to have the counter programmed to a value smaller than the sampling period. In that case, the code suppresses the sample, recalculates the remaining events and reprograms the counter. This should also work for PEBS counters (and it does); however triggering a full PEBS assist and parsing the event from the DS is more overhead than is required. As such, detect this case and temporarily suppress PEBS. This will then trigger a regular PMI for the counter which will reprogram the event and re-enable PEBS once the target period is in reach. Signed-off-by: Peter Zijlstra (Intel) --- arch/x86/events/intel/core.c | 80 ++++++++++++++++++++++++++++++++++++++= ----- arch/x86/events/perf_event.h | 9 ++++ 2 files changed, 81 insertions(+), 8 deletions(-) --- a/arch/x86/events/intel/core.c +++ b/arch/x86/events/intel/core.c @@ -2722,12 +2722,7 @@ static void intel_pmu_enable_fixed(struc =20 intel_set_masks(event, idx); =20 - /* - * Enable IRQ generation (0x8), if not PEBS, - * and enable ring-3 counting (0x2) and ring-0 counting (0x1) - * if requested: - */ - if (!event->attr.precise_ip) + if (hwc->config & ARCH_PERFMON_EVENTSEL_INT) bits |=3D 0x8; if (hwc->config & ARCH_PERFMON_EVENTSEL_USR) bits |=3D 0x2; @@ -2816,12 +2811,75 @@ int intel_pmu_save_and_restart(struct pe return static_call(x86_pmu_set_period)(event); } =20 +static void intel_pmu_update_config(struct perf_event *event) +{ + struct hw_perf_event *hwc =3D &event->hw; + u64 config =3D hwc->config; + + if (hwc->idx >=3D INTEL_PMC_IDX_FIXED) { /* PEBS is limited to real PMCs = */ + u64 mask =3D 0xf, bits =3D 0; + + if (config & ARCH_PERFMON_EVENTSEL_INT) + bits |=3D 0x8; + if (config & ARCH_PERFMON_EVENTSEL_USR) + bits |=3D 0x2; + if (config & ARCH_PERFMON_EVENTSEL_OS) + bits |=3D 0x1; + + bits <<=3D (hwc->idx * 4); + mask <<=3D (hwc->idx * 4); + + config =3D this_cpu_read(intel_fixed_ctrl); + config &=3D ~mask; + config |=3D bits; + this_cpu_write(intel_fixed_ctrl, config); + } + + wrmsrl(hwc->config_base, config); +} + +static void intel_pmu_handle_short_pebs(struct perf_event *event) +{ + struct cpu_hw_events *cpuc =3D this_cpu_ptr(&cpu_hw_events); + struct hw_perf_event *hwc =3D &event->hw; + + /* if the event is not enabled; intel_pmu_pebs_enable() DTRT */ + if (!test_bit(hwc->idx, cpuc->active_mask)) + return; + + WARN_ON_ONCE(cpuc->enabled); + + if (intel_pmu_is_short_pebs(event)) { + + /* stripped down intel_pmu_pebs_disable() */ + cpuc->pebs_enabled &=3D ~(1ULL << hwc->idx); + hwc->config |=3D ARCH_PERFMON_EVENTSEL_INT; + + intel_pmu_update_config(event); + + } else if (!(cpuc->pebs_enabled & (1ULL << hwc->idx))) { + + /* stripped down intel_pmu_pebs_enable() */ + hwc->config &=3D ~ARCH_PERFMON_EVENTSEL_INT; + cpuc->pebs_enabled |=3D (1ULL << hwc->idx); + + intel_pmu_update_config(event); + } +} + static int intel_pmu_set_period(struct perf_event *event) { + int ret; + if (unlikely(is_topdown_count(event))) return static_call(intel_pmu_set_topdown_event_period)(event); =20 - return x86_perf_event_set_period(event); + ret =3D x86_perf_event_set_period(event); + + if (event->attr.precise_ip) + intel_pmu_handle_short_pebs(event); + + return ret; } =20 static u64 intel_pmu_update(struct perf_event *event) @@ -2975,6 +3033,9 @@ static int handle_pmi_common(struct pt_r * MSR_IA32_PEBS_ENABLE is not updated. Because the * cpuc->enabled has been forced to 0 in PMI. * Update the MSR if pebs_enabled is changed. + * + * Also; short counters temporarily disable PEBS, see + * intel_pmu_set_period(). */ if (pebs_enabled !=3D cpuc->pebs_enabled) wrmsrl(MSR_IA32_PEBS_ENABLE, cpuc->pebs_enabled); @@ -3856,7 +3917,10 @@ static int intel_pmu_hw_config(struct pe if ((event->attr.config & INTEL_ARCH_EVENT_MASK) =3D=3D INTEL_FIXED_VLBR= _EVENT) return -EINVAL; =20 - if (!(event->attr.freq || (event->attr.wakeup_events && !event->attr.wat= ermark))) { + if (!(event->attr.freq || + (event->attr.wakeup_events && !event->attr.watermark) || + event->attr.sample_period > x86_pmu.max_period)) { + event->hw.flags |=3D PERF_X86_EVENT_AUTO_RELOAD; if (!(event->attr.sample_type & ~intel_pmu_large_pebs_flags(event))) { --- a/arch/x86/events/perf_event.h +++ b/arch/x86/events/perf_event.h @@ -1063,6 +1063,15 @@ static inline bool x86_pmu_has_lbr_calls DECLARE_PER_CPU(struct cpu_hw_events, cpu_hw_events); DECLARE_PER_CPU(u64 [X86_PMC_IDX_MAX], pmc_prev_left); =20 +static inline bool intel_pmu_is_short_pebs(struct perf_event *event) +{ + struct hw_perf_event *hwc =3D &event->hw; + s64 counter =3D this_cpu_read(pmc_prev_left[hwc->idx]); + s64 left =3D local64_read(&hwc->period_left); + + return counter < left; +} + int x86_perf_event_set_period(struct perf_event *event); =20 /*