From nobody Fri Jan 2 15:45:01 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2C685CD6904 for ; Tue, 10 Oct 2023 08:20:00 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1442781AbjJJIT5 (ORCPT ); Tue, 10 Oct 2023 04:19:57 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59726 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1442730AbjJJITg (ORCPT ); Tue, 10 Oct 2023 04:19:36 -0400 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A344BA7; Tue, 10 Oct 2023 01:19:32 -0700 (PDT) Date: Tue, 10 Oct 2023 08:19:30 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1696925971; h=from:from:sender:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=uFcrSlmz0vzKoZCoGjeUz32KUDi6r6mYIggXErj/xdk=; b=e3O7NMUV6Gs97aV3YdsL8FeKrCPCPpDLyGUc3RGpohzbpXfTavHKrw+K81SfeiurqAjqu+ 9caoWpRJ5KGu3SOUMbbUdvASf86MVhEQEI7crK8ClXvRTdY1rrLkVHp43RFphyFWdqmmoF m44qXC3wRSuv8fvB9e0rpmhiP2xeFaw1kREDBEdYnKRcV2oiDBcLTvcEEadfN8m3b8Dand n9kza1AP8DJbfA4mh79QH8nfy2Agn+AFpgFUB9thk7dft0N0FXG659gRJI792nd6GLGux7 1IrkAA6Q+umDhY/y9N2oldt/Y0AD68flKFwlbv3pJN3DNldrdIZYrPeCdBdCXA== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1696925971; h=from:from:sender:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=uFcrSlmz0vzKoZCoGjeUz32KUDi6r6mYIggXErj/xdk=; b=vYRuMuDoR8rUV9GfaI9+086E++lqHDjRBC6hiK/6T0PwYbJgp0HGgvyLTAMlGVO7uOdT2i fjjwPDLLvgD6LpDg== From: "tip-bot2 for Sandipan Das" Sender: tip-bot2@linutronix.de Reply-to: linux-kernel@vger.kernel.org To: linux-tip-commits@vger.kernel.org Subject: [tip: perf/core] perf/x86/amd/uncore: Refactor uncore management Cc: Sandipan Das , "Peter Zijlstra (Intel)" , x86@kernel.org, linux-kernel@vger.kernel.org In-Reply-To: =?utf-8?q?=3C24b38c49a5dae65d8c96e5d75a2b96ae97aaa651=2E16964?= =?utf-8?q?25185=2Egit=2Esandipan=2Edas=40amd=2Ecom=3E?= References: =?utf-8?q?=3C24b38c49a5dae65d8c96e5d75a2b96ae97aaa651=2E169642?= =?utf-8?q?5185=2Egit=2Esandipan=2Edas=40amd=2Ecom=3E?= MIME-Version: 1.0 Message-ID: <169692597062.3135.2529737485792230446.tip-bot2@tip-bot2> Robot-ID: Robot-Unsubscribe: Contact to get blacklisted from these emails Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org The following commit has been merged into the perf/core branch of tip: Commit-ID: d6389d3ccc136a4229a8d497899c64f80fd3c5b3 Gitweb: https://git.kernel.org/tip/d6389d3ccc136a4229a8d497899c64f80= fd3c5b3 Author: Sandipan Das AuthorDate: Thu, 05 Oct 2023 10:53:11 +05:30 Committer: Peter Zijlstra CommitterDate: Mon, 09 Oct 2023 16:12:23 +02:00 perf/x86/amd/uncore: Refactor uncore management Since struct amd_uncore is used to manage per-cpu contexts, rename it to amd_uncore_ctx in order to better reflect its purpose. Add a new struct amd_uncore_pmu to encapsulate all attributes which are shared by per-cpu contexts for a corresponding PMU. These include the number of counters, active mask, MSR and RDPMC base addresses, etc. Since the struct pmu is now embedded, the corresponding amd_uncore_pmu for a given event can be found by simply using container_of(). Finally, move all PMU-specific code to separate functions. While the original event management functions continue to provide the base functionality, all PMU-specific quirks and customizations are applied in separate functions. The motivation is to simplify the management of uncore PMUs. Signed-off-by: Sandipan Das Signed-off-by: Peter Zijlstra (Intel) Link: https://lore.kernel.org/r/24b38c49a5dae65d8c96e5d75a2b96ae97aaa651.16= 96425185.git.sandipan.das@amd.com --- arch/x86/events/amd/uncore.c | 737 +++++++++++++++++----------------- 1 file changed, 379 insertions(+), 358 deletions(-) diff --git a/arch/x86/events/amd/uncore.c b/arch/x86/events/amd/uncore.c index 83f15fe..ffcecda 100644 --- a/arch/x86/events/amd/uncore.c +++ b/arch/x86/events/amd/uncore.c @@ -27,56 +27,41 @@ =20 #define COUNTER_SHIFT 16 =20 +#define NUM_UNCORES_MAX 2 /* DF (or NB) and L3 (or L2) */ +#define UNCORE_NAME_LEN 16 + #undef pr_fmt #define pr_fmt(fmt) "amd_uncore: " fmt =20 static int pmu_version; -static int num_counters_llc; -static int num_counters_nb; -static bool l3_mask; =20 static HLIST_HEAD(uncore_unused_list); =20 -struct amd_uncore { +struct amd_uncore_ctx { int id; int refcnt; int cpu; - int num_counters; - int rdpmc_base; - u32 msr_base; - cpumask_t *active_mask; - struct pmu *pmu; struct perf_event **events; struct hlist_node node; }; =20 -static struct amd_uncore * __percpu *amd_uncore_nb; -static struct amd_uncore * __percpu *amd_uncore_llc; - -static struct pmu amd_nb_pmu; -static struct pmu amd_llc_pmu; - -static cpumask_t amd_nb_active_mask; -static cpumask_t amd_llc_active_mask; - -static bool is_nb_event(struct perf_event *event) -{ - return event->pmu->type =3D=3D amd_nb_pmu.type; -} +struct amd_uncore_pmu { + char name[UNCORE_NAME_LEN]; + int num_counters; + int rdpmc_base; + u32 msr_base; + cpumask_t active_mask; + struct pmu pmu; + struct amd_uncore_ctx * __percpu *ctx; + int (*id)(unsigned int cpu); +}; =20 -static bool is_llc_event(struct perf_event *event) -{ - return event->pmu->type =3D=3D amd_llc_pmu.type; -} +static struct amd_uncore_pmu pmus[NUM_UNCORES_MAX]; +static int num_pmus __read_mostly; =20 -static struct amd_uncore *event_to_amd_uncore(struct perf_event *event) +static struct amd_uncore_pmu *event_to_amd_uncore_pmu(struct perf_event *e= vent) { - if (is_nb_event(event) && amd_uncore_nb) - return *per_cpu_ptr(amd_uncore_nb, event->cpu); - else if (is_llc_event(event) && amd_uncore_llc) - return *per_cpu_ptr(amd_uncore_llc, event->cpu); - - return NULL; + return container_of(event->pmu, struct amd_uncore_pmu, pmu); } =20 static void amd_uncore_read(struct perf_event *event) @@ -118,7 +103,7 @@ static void amd_uncore_stop(struct perf_event *event, i= nt flags) hwc->state |=3D PERF_HES_STOPPED; =20 if ((flags & PERF_EF_UPDATE) && !(hwc->state & PERF_HES_UPTODATE)) { - amd_uncore_read(event); + event->pmu->read(event); hwc->state |=3D PERF_HES_UPTODATE; } } @@ -126,15 +111,16 @@ static void amd_uncore_stop(struct perf_event *event,= int flags) static int amd_uncore_add(struct perf_event *event, int flags) { int i; - struct amd_uncore *uncore =3D event_to_amd_uncore(event); + struct amd_uncore_pmu *pmu =3D event_to_amd_uncore_pmu(event); + struct amd_uncore_ctx *ctx =3D *per_cpu_ptr(pmu->ctx, event->cpu); struct hw_perf_event *hwc =3D &event->hw; =20 /* are we already assigned? */ - if (hwc->idx !=3D -1 && uncore->events[hwc->idx] =3D=3D event) + if (hwc->idx !=3D -1 && ctx->events[hwc->idx] =3D=3D event) goto out; =20 - for (i =3D 0; i < uncore->num_counters; i++) { - if (uncore->events[i] =3D=3D event) { + for (i =3D 0; i < pmu->num_counters; i++) { + if (ctx->events[i] =3D=3D event) { hwc->idx =3D i; goto out; } @@ -142,8 +128,8 @@ static int amd_uncore_add(struct perf_event *event, int= flags) =20 /* if not, take the first available counter */ hwc->idx =3D -1; - for (i =3D 0; i < uncore->num_counters; i++) { - if (cmpxchg(&uncore->events[i], NULL, event) =3D=3D NULL) { + for (i =3D 0; i < pmu->num_counters; i++) { + if (cmpxchg(&ctx->events[i], NULL, event) =3D=3D NULL) { hwc->idx =3D i; break; } @@ -153,23 +139,13 @@ out: if (hwc->idx =3D=3D -1) return -EBUSY; =20 - hwc->config_base =3D uncore->msr_base + (2 * hwc->idx); - hwc->event_base =3D uncore->msr_base + 1 + (2 * hwc->idx); - hwc->event_base_rdpmc =3D uncore->rdpmc_base + hwc->idx; + hwc->config_base =3D pmu->msr_base + (2 * hwc->idx); + hwc->event_base =3D pmu->msr_base + 1 + (2 * hwc->idx); + hwc->event_base_rdpmc =3D pmu->rdpmc_base + hwc->idx; hwc->state =3D PERF_HES_UPTODATE | PERF_HES_STOPPED; =20 - /* - * The first four DF counters are accessible via RDPMC index 6 to 9 - * followed by the L3 counters from index 10 to 15. For processors - * with more than four DF counters, the DF RDPMC assignments become - * discontiguous as the additional counters are accessible starting - * from index 16. - */ - if (is_nb_event(event) && hwc->idx >=3D NUM_COUNTERS_NB) - hwc->event_base_rdpmc +=3D NUM_COUNTERS_L3; - if (flags & PERF_EF_START) - amd_uncore_start(event, PERF_EF_RELOAD); + event->pmu->start(event, PERF_EF_RELOAD); =20 return 0; } @@ -177,55 +153,36 @@ out: static void amd_uncore_del(struct perf_event *event, int flags) { int i; - struct amd_uncore *uncore =3D event_to_amd_uncore(event); + struct amd_uncore_pmu *pmu =3D event_to_amd_uncore_pmu(event); + struct amd_uncore_ctx *ctx =3D *per_cpu_ptr(pmu->ctx, event->cpu); struct hw_perf_event *hwc =3D &event->hw; =20 - amd_uncore_stop(event, PERF_EF_UPDATE); + event->pmu->stop(event, PERF_EF_UPDATE); =20 - for (i =3D 0; i < uncore->num_counters; i++) { - if (cmpxchg(&uncore->events[i], event, NULL) =3D=3D event) + for (i =3D 0; i < pmu->num_counters; i++) { + if (cmpxchg(&ctx->events[i], event, NULL) =3D=3D event) break; } =20 hwc->idx =3D -1; } =20 -/* - * Return a full thread and slice mask unless user - * has provided them - */ -static u64 l3_thread_slice_mask(u64 config) -{ - if (boot_cpu_data.x86 <=3D 0x18) - return ((config & AMD64_L3_SLICE_MASK) ? : AMD64_L3_SLICE_MASK) | - ((config & AMD64_L3_THREAD_MASK) ? : AMD64_L3_THREAD_MASK); - - /* - * If the user doesn't specify a threadmask, they're not trying to - * count core 0, so we enable all cores & threads. - * We'll also assume that they want to count slice 0 if they specify - * a threadmask and leave sliceid and enallslices unpopulated. - */ - if (!(config & AMD64_L3_F19H_THREAD_MASK)) - return AMD64_L3_F19H_THREAD_MASK | AMD64_L3_EN_ALL_SLICES | - AMD64_L3_EN_ALL_CORES; - - return config & (AMD64_L3_F19H_THREAD_MASK | AMD64_L3_SLICEID_MASK | - AMD64_L3_EN_ALL_CORES | AMD64_L3_EN_ALL_SLICES | - AMD64_L3_COREID_MASK); -} - static int amd_uncore_event_init(struct perf_event *event) { - struct amd_uncore *uncore; + struct amd_uncore_pmu *pmu; + struct amd_uncore_ctx *ctx; struct hw_perf_event *hwc =3D &event->hw; - u64 event_mask =3D AMD64_RAW_EVENT_MASK_NB; =20 if (event->attr.type !=3D event->pmu->type) return -ENOENT; =20 - if (pmu_version >=3D 2 && is_nb_event(event)) - event_mask =3D AMD64_PERFMON_V2_RAW_EVENT_MASK_NB; + if (event->cpu < 0) + return -EINVAL; + + pmu =3D event_to_amd_uncore_pmu(event); + ctx =3D *per_cpu_ptr(pmu->ctx, event->cpu); + if (!ctx) + return -ENODEV; =20 /* * NB and Last level cache counters (MSRs) are shared across all cores @@ -235,28 +192,14 @@ static int amd_uncore_event_init(struct perf_event *e= vent) * out. So we do not support sampling and per-thread events via * CAP_NO_INTERRUPT, and we do not enable counter overflow interrupts: */ - hwc->config =3D event->attr.config & event_mask; + hwc->config =3D event->attr.config; hwc->idx =3D -1; =20 - if (event->cpu < 0) - return -EINVAL; - - /* - * SliceMask and ThreadMask need to be set for certain L3 events. - * For other events, the two fields do not affect the count. - */ - if (l3_mask && is_llc_event(event)) - hwc->config |=3D l3_thread_slice_mask(event->attr.config); - - uncore =3D event_to_amd_uncore(event); - if (!uncore) - return -ENODEV; - /* * since request can come in to any of the shared cores, we will remap * to a single common cpu. */ - event->cpu =3D uncore->cpu; + event->cpu =3D ctx->cpu; =20 return 0; } @@ -278,17 +221,10 @@ static ssize_t amd_uncore_attr_show_cpumask(struct de= vice *dev, struct device_attribute *attr, char *buf) { - cpumask_t *active_mask; - struct pmu *pmu =3D dev_get_drvdata(dev); - - if (pmu->type =3D=3D amd_nb_pmu.type) - active_mask =3D &amd_nb_active_mask; - else if (pmu->type =3D=3D amd_llc_pmu.type) - active_mask =3D &amd_llc_active_mask; - else - return 0; + struct pmu *ptr =3D dev_get_drvdata(dev); + struct amd_uncore_pmu *pmu =3D container_of(ptr, struct amd_uncore_pmu, p= mu); =20 - return cpumap_print_to_pagebuf(true, buf, active_mask); + return cpumap_print_to_pagebuf(true, buf, &pmu->active_mask); } static DEVICE_ATTR(cpumask, S_IRUGO, amd_uncore_attr_show_cpumask, NULL); =20 @@ -396,113 +332,57 @@ static const struct attribute_group *amd_uncore_l3_a= ttr_update[] =3D { NULL, }; =20 -static struct pmu amd_nb_pmu =3D { - .task_ctx_nr =3D perf_invalid_context, - .attr_groups =3D amd_uncore_df_attr_groups, - .name =3D "amd_nb", - .event_init =3D amd_uncore_event_init, - .add =3D amd_uncore_add, - .del =3D amd_uncore_del, - .start =3D amd_uncore_start, - .stop =3D amd_uncore_stop, - .read =3D amd_uncore_read, - .capabilities =3D PERF_PMU_CAP_NO_EXCLUDE | PERF_PMU_CAP_NO_INTERRUPT, - .module =3D THIS_MODULE, -}; - -static struct pmu amd_llc_pmu =3D { - .task_ctx_nr =3D perf_invalid_context, - .attr_groups =3D amd_uncore_l3_attr_groups, - .attr_update =3D amd_uncore_l3_attr_update, - .name =3D "amd_l2", - .event_init =3D amd_uncore_event_init, - .add =3D amd_uncore_add, - .del =3D amd_uncore_del, - .start =3D amd_uncore_start, - .stop =3D amd_uncore_stop, - .read =3D amd_uncore_read, - .capabilities =3D PERF_PMU_CAP_NO_EXCLUDE | PERF_PMU_CAP_NO_INTERRUPT, - .module =3D THIS_MODULE, -}; - -static struct amd_uncore *amd_uncore_alloc(unsigned int cpu) -{ - return kzalloc_node(sizeof(struct amd_uncore), GFP_KERNEL, - cpu_to_node(cpu)); -} - -static inline struct perf_event ** -amd_uncore_events_alloc(unsigned int num, unsigned int cpu) -{ - return kzalloc_node(sizeof(struct perf_event *) * num, GFP_KERNEL, - cpu_to_node(cpu)); -} - static int amd_uncore_cpu_up_prepare(unsigned int cpu) { - struct amd_uncore *uncore_nb =3D NULL, *uncore_llc =3D NULL; - - if (amd_uncore_nb) { - *per_cpu_ptr(amd_uncore_nb, cpu) =3D NULL; - uncore_nb =3D amd_uncore_alloc(cpu); - if (!uncore_nb) - goto fail; - uncore_nb->cpu =3D cpu; - uncore_nb->num_counters =3D num_counters_nb; - uncore_nb->rdpmc_base =3D RDPMC_BASE_NB; - uncore_nb->msr_base =3D MSR_F15H_NB_PERF_CTL; - uncore_nb->active_mask =3D &amd_nb_active_mask; - uncore_nb->pmu =3D &amd_nb_pmu; - uncore_nb->events =3D amd_uncore_events_alloc(num_counters_nb, cpu); - if (!uncore_nb->events) + struct amd_uncore_pmu *pmu; + struct amd_uncore_ctx *ctx; + int node =3D cpu_to_node(cpu), i; + + for (i =3D 0; i < num_pmus; i++) { + pmu =3D &pmus[i]; + *per_cpu_ptr(pmu->ctx, cpu) =3D NULL; + ctx =3D kzalloc_node(sizeof(struct amd_uncore_ctx), GFP_KERNEL, + node); + if (!ctx) goto fail; - uncore_nb->id =3D -1; - *per_cpu_ptr(amd_uncore_nb, cpu) =3D uncore_nb; - } =20 - if (amd_uncore_llc) { - *per_cpu_ptr(amd_uncore_llc, cpu) =3D NULL; - uncore_llc =3D amd_uncore_alloc(cpu); - if (!uncore_llc) - goto fail; - uncore_llc->cpu =3D cpu; - uncore_llc->num_counters =3D num_counters_llc; - uncore_llc->rdpmc_base =3D RDPMC_BASE_LLC; - uncore_llc->msr_base =3D MSR_F16H_L2I_PERF_CTL; - uncore_llc->active_mask =3D &amd_llc_active_mask; - uncore_llc->pmu =3D &amd_llc_pmu; - uncore_llc->events =3D amd_uncore_events_alloc(num_counters_llc, cpu); - if (!uncore_llc->events) + ctx->cpu =3D cpu; + ctx->events =3D kzalloc_node(sizeof(struct perf_event *) * + pmu->num_counters, GFP_KERNEL, + node); + if (!ctx->events) goto fail; - uncore_llc->id =3D -1; - *per_cpu_ptr(amd_uncore_llc, cpu) =3D uncore_llc; + + ctx->id =3D -1; + *per_cpu_ptr(pmu->ctx, cpu) =3D ctx; } =20 return 0; =20 fail: - if (uncore_nb) { - kfree(uncore_nb->events); - kfree(uncore_nb); - } + /* Rollback */ + for (; i >=3D 0; i--) { + pmu =3D &pmus[i]; + ctx =3D *per_cpu_ptr(pmu->ctx, cpu); + if (!ctx) + continue; =20 - if (uncore_llc) { - kfree(uncore_llc->events); - kfree(uncore_llc); + kfree(ctx->events); + kfree(ctx); } =20 return -ENOMEM; } =20 -static struct amd_uncore * -amd_uncore_find_online_sibling(struct amd_uncore *this, - struct amd_uncore * __percpu *uncores) +static struct amd_uncore_ctx * +amd_uncore_find_online_sibling(struct amd_uncore_ctx *this, + struct amd_uncore_pmu *pmu) { unsigned int cpu; - struct amd_uncore *that; + struct amd_uncore_ctx *that; =20 for_each_online_cpu(cpu) { - that =3D *per_cpu_ptr(uncores, cpu); + that =3D *per_cpu_ptr(pmu->ctx, cpu); =20 if (!that) continue; @@ -523,24 +403,16 @@ amd_uncore_find_online_sibling(struct amd_uncore *thi= s, =20 static int amd_uncore_cpu_starting(unsigned int cpu) { - unsigned int eax, ebx, ecx, edx; - struct amd_uncore *uncore; - - if (amd_uncore_nb) { - uncore =3D *per_cpu_ptr(amd_uncore_nb, cpu); - cpuid(0x8000001e, &eax, &ebx, &ecx, &edx); - uncore->id =3D ecx & 0xff; - - uncore =3D amd_uncore_find_online_sibling(uncore, amd_uncore_nb); - *per_cpu_ptr(amd_uncore_nb, cpu) =3D uncore; - } - - if (amd_uncore_llc) { - uncore =3D *per_cpu_ptr(amd_uncore_llc, cpu); - uncore->id =3D get_llc_id(cpu); + struct amd_uncore_pmu *pmu; + struct amd_uncore_ctx *ctx; + int i; =20 - uncore =3D amd_uncore_find_online_sibling(uncore, amd_uncore_llc); - *per_cpu_ptr(amd_uncore_llc, cpu) =3D uncore; + for (i =3D 0; i < num_pmus; i++) { + pmu =3D &pmus[i]; + ctx =3D *per_cpu_ptr(pmu->ctx, cpu); + ctx->id =3D pmu->id(cpu); + ctx =3D amd_uncore_find_online_sibling(ctx, pmu); + *per_cpu_ptr(pmu->ctx, cpu) =3D ctx; } =20 return 0; @@ -548,195 +420,359 @@ static int amd_uncore_cpu_starting(unsigned int cpu) =20 static void uncore_clean_online(void) { - struct amd_uncore *uncore; + struct amd_uncore_ctx *ctx; struct hlist_node *n; =20 - hlist_for_each_entry_safe(uncore, n, &uncore_unused_list, node) { - hlist_del(&uncore->node); - kfree(uncore->events); - kfree(uncore); + hlist_for_each_entry_safe(ctx, n, &uncore_unused_list, node) { + hlist_del(&ctx->node); + kfree(ctx->events); + kfree(ctx); } } =20 -static void uncore_online(unsigned int cpu, - struct amd_uncore * __percpu *uncores) +static int amd_uncore_cpu_online(unsigned int cpu) { - struct amd_uncore *uncore =3D *per_cpu_ptr(uncores, cpu); + struct amd_uncore_pmu *pmu; + struct amd_uncore_ctx *ctx; + int i; =20 uncore_clean_online(); =20 - if (cpu =3D=3D uncore->cpu) - cpumask_set_cpu(cpu, uncore->active_mask); + for (i =3D 0; i < num_pmus; i++) { + pmu =3D &pmus[i]; + ctx =3D *per_cpu_ptr(pmu->ctx, cpu); + if (cpu =3D=3D ctx->cpu) + cpumask_set_cpu(cpu, &pmu->active_mask); + } + + return 0; } =20 -static int amd_uncore_cpu_online(unsigned int cpu) +static int amd_uncore_cpu_down_prepare(unsigned int cpu) { - if (amd_uncore_nb) - uncore_online(cpu, amd_uncore_nb); + struct amd_uncore_ctx *this, *that; + struct amd_uncore_pmu *pmu; + int i, j; + + for (i =3D 0; i < num_pmus; i++) { + pmu =3D &pmus[i]; + this =3D *per_cpu_ptr(pmu->ctx, cpu); + + /* this cpu is going down, migrate to a shared sibling if possible */ + for_each_online_cpu(j) { + that =3D *per_cpu_ptr(pmu->ctx, j); + + if (cpu =3D=3D j) + continue; + + if (this =3D=3D that) { + perf_pmu_migrate_context(&pmu->pmu, cpu, j); + cpumask_clear_cpu(cpu, &pmu->active_mask); + cpumask_set_cpu(j, &pmu->active_mask); + that->cpu =3D j; + break; + } + } + } =20 - if (amd_uncore_llc) - uncore_online(cpu, amd_uncore_llc); + return 0; +} + +static int amd_uncore_cpu_dead(unsigned int cpu) +{ + struct amd_uncore_ctx *ctx; + struct amd_uncore_pmu *pmu; + int i; + + for (i =3D 0; i < num_pmus; i++) { + pmu =3D &pmus[i]; + ctx =3D *per_cpu_ptr(pmu->ctx, cpu); + if (cpu =3D=3D ctx->cpu) + cpumask_clear_cpu(cpu, &pmu->active_mask); + + if (!--ctx->refcnt) { + kfree(ctx->events); + kfree(ctx); + } + + *per_cpu_ptr(pmu->ctx, cpu) =3D NULL; + } =20 return 0; } =20 -static void uncore_down_prepare(unsigned int cpu, - struct amd_uncore * __percpu *uncores) +static int amd_uncore_df_id(unsigned int cpu) { - unsigned int i; - struct amd_uncore *this =3D *per_cpu_ptr(uncores, cpu); + unsigned int eax, ebx, ecx, edx; =20 - if (this->cpu !=3D cpu) - return; + cpuid(0x8000001e, &eax, &ebx, &ecx, &edx); =20 - /* this cpu is going down, migrate to a shared sibling if possible */ - for_each_online_cpu(i) { - struct amd_uncore *that =3D *per_cpu_ptr(uncores, i); + return ecx & 0xff; +} =20 - if (cpu =3D=3D i) - continue; +static int amd_uncore_df_event_init(struct perf_event *event) +{ + struct hw_perf_event *hwc =3D &event->hw; + int ret =3D amd_uncore_event_init(event); =20 - if (this =3D=3D that) { - perf_pmu_migrate_context(this->pmu, cpu, i); - cpumask_clear_cpu(cpu, that->active_mask); - cpumask_set_cpu(i, that->active_mask); - that->cpu =3D i; - break; - } - } + if (ret || pmu_version < 2) + return ret; + + hwc->config =3D event->attr.config & + (pmu_version >=3D 2 ? AMD64_PERFMON_V2_RAW_EVENT_MASK_NB : + AMD64_RAW_EVENT_MASK_NB); + + return 0; } =20 -static int amd_uncore_cpu_down_prepare(unsigned int cpu) +static int amd_uncore_df_add(struct perf_event *event, int flags) { - if (amd_uncore_nb) - uncore_down_prepare(cpu, amd_uncore_nb); + int ret =3D amd_uncore_add(event, flags & ~PERF_EF_START); + struct hw_perf_event *hwc =3D &event->hw; + + if (ret) + return ret; + + /* + * The first four DF counters are accessible via RDPMC index 6 to 9 + * followed by the L3 counters from index 10 to 15. For processors + * with more than four DF counters, the DF RDPMC assignments become + * discontiguous as the additional counters are accessible starting + * from index 16. + */ + if (hwc->idx >=3D NUM_COUNTERS_NB) + hwc->event_base_rdpmc +=3D NUM_COUNTERS_L3; =20 - if (amd_uncore_llc) - uncore_down_prepare(cpu, amd_uncore_llc); + /* Delayed start after rdpmc base update */ + if (flags & PERF_EF_START) + amd_uncore_start(event, PERF_EF_RELOAD); =20 return 0; } =20 -static void uncore_dead(unsigned int cpu, struct amd_uncore * __percpu *un= cores) +static int amd_uncore_df_init(void) { - struct amd_uncore *uncore =3D *per_cpu_ptr(uncores, cpu); + struct attribute **df_attr =3D amd_uncore_df_format_attr; + struct amd_uncore_pmu *pmu =3D &pmus[num_pmus]; + union cpuid_0x80000022_ebx ebx; + int ret; =20 - if (cpu =3D=3D uncore->cpu) - cpumask_clear_cpu(cpu, uncore->active_mask); + if (!boot_cpu_has(X86_FEATURE_PERFCTR_NB)) + return 0; =20 - if (!--uncore->refcnt) { - kfree(uncore->events); - kfree(uncore); + /* + * For Family 17h and above, the Northbridge counters are repurposed + * as Data Fabric counters. The PMUs are exported based on family as + * either NB or DF. + */ + strscpy(pmu->name, boot_cpu_data.x86 >=3D 0x17 ? "amd_df" : "amd_nb", + sizeof(pmu->name)); + + pmu->num_counters =3D NUM_COUNTERS_NB; + pmu->msr_base =3D MSR_F15H_NB_PERF_CTL; + pmu->rdpmc_base =3D RDPMC_BASE_NB; + pmu->id =3D amd_uncore_df_id; + + if (pmu_version >=3D 2) { + *df_attr++ =3D &format_attr_event14v2.attr; + *df_attr++ =3D &format_attr_umask12.attr; + ebx.full =3D cpuid_ebx(EXT_PERFMON_DEBUG_FEATURES); + pmu->num_counters =3D ebx.split.num_df_pmc; + } else if (boot_cpu_data.x86 >=3D 0x17) { + *df_attr =3D &format_attr_event14.attr; + } + + pmu->ctx =3D alloc_percpu(struct amd_uncore_ctx *); + if (!pmu->ctx) + return -ENOMEM; + + pmu->pmu =3D (struct pmu) { + .task_ctx_nr =3D perf_invalid_context, + .attr_groups =3D amd_uncore_df_attr_groups, + .name =3D pmu->name, + .event_init =3D amd_uncore_df_event_init, + .add =3D amd_uncore_df_add, + .del =3D amd_uncore_del, + .start =3D amd_uncore_start, + .stop =3D amd_uncore_stop, + .read =3D amd_uncore_read, + .capabilities =3D PERF_PMU_CAP_NO_EXCLUDE | PERF_PMU_CAP_NO_INTERRUPT, + .module =3D THIS_MODULE, + }; + + ret =3D perf_pmu_register(&pmu->pmu, pmu->pmu.name, -1); + if (ret) { + free_percpu(pmu->ctx); + pmu->ctx =3D NULL; + return ret; } =20 - *per_cpu_ptr(uncores, cpu) =3D NULL; + pr_info("%d %s %s counters detected\n", pmu->num_counters, + boot_cpu_data.x86_vendor =3D=3D X86_VENDOR_HYGON ? "HYGON" : "", + pmu->pmu.name); + + num_pmus++; + + return 0; } =20 -static int amd_uncore_cpu_dead(unsigned int cpu) +static int amd_uncore_l3_id(unsigned int cpu) { - if (amd_uncore_nb) - uncore_dead(cpu, amd_uncore_nb); + return get_llc_id(cpu); +} + +static int amd_uncore_l3_event_init(struct perf_event *event) +{ + int ret =3D amd_uncore_event_init(event); + struct hw_perf_event *hwc =3D &event->hw; + u64 config =3D event->attr.config; + u64 mask; + + hwc->config =3D config & AMD64_RAW_EVENT_MASK_NB; + + /* + * SliceMask and ThreadMask need to be set for certain L3 events. + * For other events, the two fields do not affect the count. + */ + if (ret || boot_cpu_data.x86 < 0x17) + return ret; + + mask =3D config & (AMD64_L3_F19H_THREAD_MASK | AMD64_L3_SLICEID_MASK | + AMD64_L3_EN_ALL_CORES | AMD64_L3_EN_ALL_SLICES | + AMD64_L3_COREID_MASK); + + if (boot_cpu_data.x86 <=3D 0x18) + mask =3D ((config & AMD64_L3_SLICE_MASK) ? : AMD64_L3_SLICE_MASK) | + ((config & AMD64_L3_THREAD_MASK) ? : AMD64_L3_THREAD_MASK); + + /* + * If the user doesn't specify a ThreadMask, they're not trying to + * count core 0, so we enable all cores & threads. + * We'll also assume that they want to count slice 0 if they specify + * a ThreadMask and leave SliceId and EnAllSlices unpopulated. + */ + else if (!(config & AMD64_L3_F19H_THREAD_MASK)) + mask =3D AMD64_L3_F19H_THREAD_MASK | AMD64_L3_EN_ALL_SLICES | + AMD64_L3_EN_ALL_CORES; =20 - if (amd_uncore_llc) - uncore_dead(cpu, amd_uncore_llc); + hwc->config |=3D mask; =20 return 0; } =20 -static int __init amd_uncore_init(void) +static int amd_uncore_l3_init(void) { - struct attribute **df_attr =3D amd_uncore_df_format_attr; struct attribute **l3_attr =3D amd_uncore_l3_format_attr; - union cpuid_0x80000022_ebx ebx; - int ret =3D -ENODEV; + struct amd_uncore_pmu *pmu =3D &pmus[num_pmus]; + int ret; =20 - if (boot_cpu_data.x86_vendor !=3D X86_VENDOR_AMD && - boot_cpu_data.x86_vendor !=3D X86_VENDOR_HYGON) - return -ENODEV; + if (!boot_cpu_has(X86_FEATURE_PERFCTR_LLC)) + return 0; =20 - if (!boot_cpu_has(X86_FEATURE_TOPOEXT)) - return -ENODEV; + /* + * For Family 17h and above, L3 cache counters are available instead + * of L2 cache counters. The PMUs are exported based on family as + * either L2 or L3. + */ + strscpy(pmu->name, boot_cpu_data.x86 >=3D 0x17 ? "amd_l3" : "amd_l2", + sizeof(pmu->name)); =20 - if (boot_cpu_has(X86_FEATURE_PERFMON_V2)) - pmu_version =3D 2; + pmu->num_counters =3D NUM_COUNTERS_L2; + pmu->msr_base =3D MSR_F16H_L2I_PERF_CTL; + pmu->rdpmc_base =3D RDPMC_BASE_LLC; + pmu->id =3D amd_uncore_l3_id; =20 - num_counters_nb =3D NUM_COUNTERS_NB; - num_counters_llc =3D NUM_COUNTERS_L2; if (boot_cpu_data.x86 >=3D 0x17) { - /* - * For F17h and above, the Northbridge counters are - * repurposed as Data Fabric counters. Also, L3 - * counters are supported too. The PMUs are exported - * based on family as either L2 or L3 and NB or DF. - */ - num_counters_llc =3D NUM_COUNTERS_L3; - amd_nb_pmu.name =3D "amd_df"; - amd_llc_pmu.name =3D "amd_l3"; - l3_mask =3D true; + *l3_attr++ =3D &format_attr_event8.attr; + *l3_attr++ =3D &format_attr_umask8.attr; + *l3_attr++ =3D boot_cpu_data.x86 >=3D 0x19 ? + &format_attr_threadmask2.attr : + &format_attr_threadmask8.attr; + pmu->num_counters =3D NUM_COUNTERS_L3; } =20 - if (boot_cpu_has(X86_FEATURE_PERFCTR_NB)) { - if (pmu_version >=3D 2) { - *df_attr++ =3D &format_attr_event14v2.attr; - *df_attr++ =3D &format_attr_umask12.attr; - } else if (boot_cpu_data.x86 >=3D 0x17) { - *df_attr =3D &format_attr_event14.attr; - } + pmu->ctx =3D alloc_percpu(struct amd_uncore_ctx *); + if (!pmu->ctx) + return -ENOMEM; + + pmu->pmu =3D (struct pmu) { + .task_ctx_nr =3D perf_invalid_context, + .attr_groups =3D amd_uncore_l3_attr_groups, + .attr_update =3D amd_uncore_l3_attr_update, + .name =3D pmu->name, + .event_init =3D amd_uncore_l3_event_init, + .add =3D amd_uncore_add, + .del =3D amd_uncore_del, + .start =3D amd_uncore_start, + .stop =3D amd_uncore_stop, + .read =3D amd_uncore_read, + .capabilities =3D PERF_PMU_CAP_NO_EXCLUDE | PERF_PMU_CAP_NO_INTERRUPT, + .module =3D THIS_MODULE, + }; + + ret =3D perf_pmu_register(&pmu->pmu, pmu->pmu.name, -1); + if (ret) { + free_percpu(pmu->ctx); + pmu->ctx =3D NULL; + return ret; + } =20 - amd_uncore_nb =3D alloc_percpu(struct amd_uncore *); - if (!amd_uncore_nb) { - ret =3D -ENOMEM; - goto fail_nb; - } - ret =3D perf_pmu_register(&amd_nb_pmu, amd_nb_pmu.name, -1); - if (ret) - goto fail_nb; + pr_info("%d %s %s counters detected\n", pmu->num_counters, + boot_cpu_data.x86_vendor =3D=3D X86_VENDOR_HYGON ? "HYGON" : "", + pmu->pmu.name); =20 - if (pmu_version >=3D 2) { - ebx.full =3D cpuid_ebx(EXT_PERFMON_DEBUG_FEATURES); - num_counters_nb =3D ebx.split.num_df_pmc; - } + num_pmus++; =20 - pr_info("%d %s %s counters detected\n", num_counters_nb, - boot_cpu_data.x86_vendor =3D=3D X86_VENDOR_HYGON ? "HYGON" : "", - amd_nb_pmu.name); + return 0; +} =20 - ret =3D 0; - } +static void uncore_free(void) +{ + struct amd_uncore_pmu *pmu; + int i; =20 - if (boot_cpu_has(X86_FEATURE_PERFCTR_LLC)) { - if (boot_cpu_data.x86 >=3D 0x19) { - *l3_attr++ =3D &format_attr_event8.attr; - *l3_attr++ =3D &format_attr_umask8.attr; - *l3_attr++ =3D &format_attr_threadmask2.attr; - } else if (boot_cpu_data.x86 >=3D 0x17) { - *l3_attr++ =3D &format_attr_event8.attr; - *l3_attr++ =3D &format_attr_umask8.attr; - *l3_attr++ =3D &format_attr_threadmask8.attr; - } + for (i =3D 0; i < num_pmus; i++) { + pmu =3D &pmus[i]; + if (!pmu->ctx) + continue; =20 - amd_uncore_llc =3D alloc_percpu(struct amd_uncore *); - if (!amd_uncore_llc) { - ret =3D -ENOMEM; - goto fail_llc; - } - ret =3D perf_pmu_register(&amd_llc_pmu, amd_llc_pmu.name, -1); - if (ret) - goto fail_llc; - - pr_info("%d %s %s counters detected\n", num_counters_llc, - boot_cpu_data.x86_vendor =3D=3D X86_VENDOR_HYGON ? "HYGON" : "", - amd_llc_pmu.name); - ret =3D 0; + perf_pmu_unregister(&pmu->pmu); + free_percpu(pmu->ctx); + pmu->ctx =3D NULL; } =20 + num_pmus =3D 0; +} + +static int __init amd_uncore_init(void) +{ + int ret; + + if (boot_cpu_data.x86_vendor !=3D X86_VENDOR_AMD && + boot_cpu_data.x86_vendor !=3D X86_VENDOR_HYGON) + return -ENODEV; + + if (!boot_cpu_has(X86_FEATURE_TOPOEXT)) + return -ENODEV; + + if (boot_cpu_has(X86_FEATURE_PERFMON_V2)) + pmu_version =3D 2; + + ret =3D amd_uncore_df_init(); + if (ret) + goto fail; + + ret =3D amd_uncore_l3_init(); + if (ret) + goto fail; + /* * Install callbacks. Core will call them for each online cpu. */ if (cpuhp_setup_state(CPUHP_PERF_X86_AMD_UNCORE_PREP, "perf/x86/amd/uncore:prepare", amd_uncore_cpu_up_prepare, amd_uncore_cpu_dead)) - goto fail_llc; + goto fail; =20 if (cpuhp_setup_state(CPUHP_AP_PERF_X86_AMD_UNCORE_STARTING, "perf/x86/amd/uncore:starting", @@ -753,12 +789,8 @@ fail_start: cpuhp_remove_state(CPUHP_AP_PERF_X86_AMD_UNCORE_STARTING); fail_prep: cpuhp_remove_state(CPUHP_PERF_X86_AMD_UNCORE_PREP); -fail_llc: - if (boot_cpu_has(X86_FEATURE_PERFCTR_NB)) - perf_pmu_unregister(&amd_nb_pmu); - free_percpu(amd_uncore_llc); -fail_nb: - free_percpu(amd_uncore_nb); +fail: + uncore_free(); =20 return ret; } @@ -768,18 +800,7 @@ static void __exit amd_uncore_exit(void) cpuhp_remove_state(CPUHP_AP_PERF_X86_AMD_UNCORE_ONLINE); cpuhp_remove_state(CPUHP_AP_PERF_X86_AMD_UNCORE_STARTING); cpuhp_remove_state(CPUHP_PERF_X86_AMD_UNCORE_PREP); - - if (boot_cpu_has(X86_FEATURE_PERFCTR_LLC)) { - perf_pmu_unregister(&amd_llc_pmu); - free_percpu(amd_uncore_llc); - amd_uncore_llc =3D NULL; - } - - if (boot_cpu_has(X86_FEATURE_PERFCTR_NB)) { - perf_pmu_unregister(&amd_nb_pmu); - free_percpu(amd_uncore_nb); - amd_uncore_nb =3D NULL; - } + uncore_free(); } =20 module_init(amd_uncore_init);