From nobody Sat Jan 3 06:34:52 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E6A3EE9271C for ; Thu, 5 Oct 2023 16:41:08 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233801AbjJEQks (ORCPT ); Thu, 5 Oct 2023 12:40:48 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47112 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234813AbjJEQjV (ORCPT ); Thu, 5 Oct 2023 12:39:21 -0400 Received: from NAM11-CO1-obe.outbound.protection.outlook.com (mail-co1nam11on2061f.outbound.protection.outlook.com [IPv6:2a01:111:f400:7eab::61f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4EF564ED7; Wed, 4 Oct 2023 22:24:34 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=nRKWCSDBFzPZB1v4TvggdPSlp0osBhYz6APVeG+57IdL8ohOTmy2pIB5xq7Vd/eEhXrr7+BWpI78A2cLPQ1lWjEfFQCHH1el/IdsisOH5pO3/trEW0H5XZxtXhUF0TdqcGzAspVdKNxsAD7dEc+fcZFWLPI1ozCbqBZSxqBMTFKNl+YjGFiWn9a7Q7ucxYUpNaKEHx2NFmQrTEI0NeSJtaNAtGUr0w07YG82JwEljaIEAWVNVolEUXklamcHSetiroOLGqkR49odndOId+a6sTn+d9r60zcwr58GiwDCsV6zDnRSTy4Qhe3f+nFTKlaSEqfSMtqj9CJhY6VOx6DZAA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=0BXSBwuJ4YsRWA1Xrxys8uIlw2CPbb6D1MSI86imjQw=; b=Ei+PuaqHxy5CitsWzMOktmrzWOq0+/Czxf/4sSNw92c4rWKYgKOphQ0wVNf5ZxEGIgBCC5HvwCnvqpTHfAhI54PC7dUHVk+wJU//cEotGGOAjJayAdjY4WoG72p2ABvqEQZA6SJ+Zn4taa1wzrMWnzda6OslP2s2CtoyB2Yr9W2vjM14GFGbGbXZ3PNI9O18Xl5Jk/U8BYbSXVUgd9XVhVeIUk59k8fm9C6uMHxC3Efqmk0/QKz8sc1TPG7Or9LZMRNDR8VibP8x8JPT63nEE17+vae+S+M6wrsd+MuY2ZQ6Qkc+6/qulr4HmfS8IEk6+Idl0qIgI8lQwqEZ7prBZQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=vger.kernel.org smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=0BXSBwuJ4YsRWA1Xrxys8uIlw2CPbb6D1MSI86imjQw=; b=o/+DaQYZv2/NHkRSV/OK5DU5Vc6BIU2Rofd7ydjgQimzgX913acrHuP09PVSyBSAm/JCLEiS/9X6mkWSwP1Ek5fg+9IWGDS/NXP8AzSdK2ZeUOZAoiBwrFLF5gh4xUvwlqoGkuvWBPXBAdO+7u8WFFTwHtHHwvVGMxN0wpKdfLg= Received: from BLAPR03CA0177.namprd03.prod.outlook.com (2603:10b6:208:32f::30) by MN2PR12MB4334.namprd12.prod.outlook.com (2603:10b6:208:1d1::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6838.34; Thu, 5 Oct 2023 05:24:17 +0000 Received: from BL6PEPF0001AB4D.namprd04.prod.outlook.com (2603:10b6:208:32f:cafe::23) by BLAPR03CA0177.outlook.office365.com (2603:10b6:208:32f::30) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6863.25 via Frontend Transport; Thu, 5 Oct 2023 05:24:17 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by BL6PEPF0001AB4D.mail.protection.outlook.com (10.167.242.71) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.6838.14 via Frontend Transport; Thu, 5 Oct 2023 05:24:17 +0000 Received: from sindhu.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.27; Thu, 5 Oct 2023 00:24:10 -0500 From: Sandipan Das To: , , CC: , , , , , , , , , , , , , , , , , Subject: [PATCH v2 1/6] perf/x86/amd/uncore: Refactor uncore management Date: Thu, 5 Oct 2023 10:53:11 +0530 Message-ID: <24b38c49a5dae65d8c96e5d75a2b96ae97aaa651.1696425185.git.sandipan.das@amd.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Originating-IP: [10.180.168.240] X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BL6PEPF0001AB4D:EE_|MN2PR12MB4334:EE_ X-MS-Office365-Filtering-Correlation-Id: ffe175f6-113d-4cb2-2d20-08dbc5635263 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 4i0KpmIjsy5sRL87qalO3Z/2DF0/2lpVAYXs3ixlSHnD5gMovD6x15/5FAhVfvzZ6jS6lpO+wq2tqF/KEj2LlNqJaAmYyjilL4QRdrn9I2mQ56xwUktJxtta3wJVgk9rQDl3JAgHQpyu0DKIniYwCch+wYcR1I/GVIIKKRUjvbRPgNnnxSru2AB1qvQhGADs4LNVKNXteSd5E3RlevP3uIQsbftr2Pmsh0/+PkA22vD4QlzfcucYBZVbAAE5oY0unLRxhQQU9G3OKymr4FTeZtDphCBVVMJsXIf7nfUl3gZVnlfnvifpdhy8s8cbtn/vDv6iqgNb2otiPo1/NoYN1/OhLIkr0ZF1fXslQcwt79fWxP+TVEnU3RsnePXEc/UPIwxBL9jUNlf0H1j9LToeDsxlmw4HiNF2yXzrFPs1dSnN3+1w8oMbiD+RvVSsyzd3Oeg4R1lHiiAe4Hro/iX8x8s+MbUUcxO3/QhzRCb9tLpjAu45zudclwRc6v7jdcqY88PhgUZngJ6EuBzxfxaHj6NUwXhtQInqCX/5WMwquhVP5wX/w3OqWExiXv3AB5hk6yE2LpcbsFe0AIu3fVbVtulvcmETp1stpM7RPuE/R1VH0bUqrs3EXzrjbUFb268ZwJXMGsUfNevGRMHcSYI/AmzSiYDXR4XDkDVs+8akOeFEkX23KcT3nkM/ybCs7nkqgKJ7ij1CGzKNbl9G3dSmD/AVLiUe5MF6MiiPK9Aku6bR3cEC5QoaKYF4FvqWN+6q+yO+qrzLkMpo70adPXjFYA== X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230031)(4636009)(346002)(396003)(39860400002)(136003)(376002)(230922051799003)(82310400011)(1800799009)(186009)(64100799003)(451199024)(36840700001)(46966006)(40470700004)(36756003)(82740400003)(7696005)(81166007)(110136005)(86362001)(478600001)(356005)(54906003)(316002)(70586007)(41300700001)(2616005)(26005)(16526019)(70206006)(8936002)(4326008)(8676002)(336012)(5660300002)(426003)(83380400001)(44832011)(30864003)(40460700003)(47076005)(36860700001)(2906002)(40480700001)(7416002)(36900700001);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 Oct 2023 05:24:17.7253 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: ffe175f6-113d-4cb2-2d20-08dbc5635263 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: BL6PEPF0001AB4D.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR12MB4334 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Since struct amd_uncore is used to manage per-cpu contexts, rename it to amd_uncore_ctx in order to better reflect its purpose. Add a new struct amd_uncore_pmu to encapsulate all attributes which are shared by per-cpu contexts for a corresponding PMU. These include the number of counters, active mask, MSR and RDPMC base addresses, etc. Since the struct pmu is now embedded, the corresponding amd_uncore_pmu for a given event can be found by simply using container_of(). Finally, move all PMU-specific code to separate functions. While the original event management functions continue to provide the base functionality, all PMU-specific quirks and customizations are applied in separate functions. The motivation is to simplify the management of uncore PMUs. Signed-off-by: Sandipan Das --- arch/x86/events/amd/uncore.c | 737 ++++++++++++++++++----------------- 1 file changed, 379 insertions(+), 358 deletions(-) diff --git a/arch/x86/events/amd/uncore.c b/arch/x86/events/amd/uncore.c index 83f15fe411b3..ffcecda13d65 100644 --- a/arch/x86/events/amd/uncore.c +++ b/arch/x86/events/amd/uncore.c @@ -27,56 +27,41 @@ =20 #define COUNTER_SHIFT 16 =20 +#define NUM_UNCORES_MAX 2 /* DF (or NB) and L3 (or L2) */ +#define UNCORE_NAME_LEN 16 + #undef pr_fmt #define pr_fmt(fmt) "amd_uncore: " fmt =20 static int pmu_version; -static int num_counters_llc; -static int num_counters_nb; -static bool l3_mask; =20 static HLIST_HEAD(uncore_unused_list); =20 -struct amd_uncore { +struct amd_uncore_ctx { int id; int refcnt; int cpu; - int num_counters; - int rdpmc_base; - u32 msr_base; - cpumask_t *active_mask; - struct pmu *pmu; struct perf_event **events; struct hlist_node node; }; =20 -static struct amd_uncore * __percpu *amd_uncore_nb; -static struct amd_uncore * __percpu *amd_uncore_llc; - -static struct pmu amd_nb_pmu; -static struct pmu amd_llc_pmu; - -static cpumask_t amd_nb_active_mask; -static cpumask_t amd_llc_active_mask; - -static bool is_nb_event(struct perf_event *event) -{ - return event->pmu->type =3D=3D amd_nb_pmu.type; -} +struct amd_uncore_pmu { + char name[UNCORE_NAME_LEN]; + int num_counters; + int rdpmc_base; + u32 msr_base; + cpumask_t active_mask; + struct pmu pmu; + struct amd_uncore_ctx * __percpu *ctx; + int (*id)(unsigned int cpu); +}; =20 -static bool is_llc_event(struct perf_event *event) -{ - return event->pmu->type =3D=3D amd_llc_pmu.type; -} +static struct amd_uncore_pmu pmus[NUM_UNCORES_MAX]; +static int num_pmus __read_mostly; =20 -static struct amd_uncore *event_to_amd_uncore(struct perf_event *event) +static struct amd_uncore_pmu *event_to_amd_uncore_pmu(struct perf_event *e= vent) { - if (is_nb_event(event) && amd_uncore_nb) - return *per_cpu_ptr(amd_uncore_nb, event->cpu); - else if (is_llc_event(event) && amd_uncore_llc) - return *per_cpu_ptr(amd_uncore_llc, event->cpu); - - return NULL; + return container_of(event->pmu, struct amd_uncore_pmu, pmu); } =20 static void amd_uncore_read(struct perf_event *event) @@ -118,7 +103,7 @@ static void amd_uncore_stop(struct perf_event *event, i= nt flags) hwc->state |=3D PERF_HES_STOPPED; =20 if ((flags & PERF_EF_UPDATE) && !(hwc->state & PERF_HES_UPTODATE)) { - amd_uncore_read(event); + event->pmu->read(event); hwc->state |=3D PERF_HES_UPTODATE; } } @@ -126,15 +111,16 @@ static void amd_uncore_stop(struct perf_event *event,= int flags) static int amd_uncore_add(struct perf_event *event, int flags) { int i; - struct amd_uncore *uncore =3D event_to_amd_uncore(event); + struct amd_uncore_pmu *pmu =3D event_to_amd_uncore_pmu(event); + struct amd_uncore_ctx *ctx =3D *per_cpu_ptr(pmu->ctx, event->cpu); struct hw_perf_event *hwc =3D &event->hw; =20 /* are we already assigned? */ - if (hwc->idx !=3D -1 && uncore->events[hwc->idx] =3D=3D event) + if (hwc->idx !=3D -1 && ctx->events[hwc->idx] =3D=3D event) goto out; =20 - for (i =3D 0; i < uncore->num_counters; i++) { - if (uncore->events[i] =3D=3D event) { + for (i =3D 0; i < pmu->num_counters; i++) { + if (ctx->events[i] =3D=3D event) { hwc->idx =3D i; goto out; } @@ -142,8 +128,8 @@ static int amd_uncore_add(struct perf_event *event, int= flags) =20 /* if not, take the first available counter */ hwc->idx =3D -1; - for (i =3D 0; i < uncore->num_counters; i++) { - if (cmpxchg(&uncore->events[i], NULL, event) =3D=3D NULL) { + for (i =3D 0; i < pmu->num_counters; i++) { + if (cmpxchg(&ctx->events[i], NULL, event) =3D=3D NULL) { hwc->idx =3D i; break; } @@ -153,23 +139,13 @@ static int amd_uncore_add(struct perf_event *event, i= nt flags) if (hwc->idx =3D=3D -1) return -EBUSY; =20 - hwc->config_base =3D uncore->msr_base + (2 * hwc->idx); - hwc->event_base =3D uncore->msr_base + 1 + (2 * hwc->idx); - hwc->event_base_rdpmc =3D uncore->rdpmc_base + hwc->idx; + hwc->config_base =3D pmu->msr_base + (2 * hwc->idx); + hwc->event_base =3D pmu->msr_base + 1 + (2 * hwc->idx); + hwc->event_base_rdpmc =3D pmu->rdpmc_base + hwc->idx; hwc->state =3D PERF_HES_UPTODATE | PERF_HES_STOPPED; =20 - /* - * The first four DF counters are accessible via RDPMC index 6 to 9 - * followed by the L3 counters from index 10 to 15. For processors - * with more than four DF counters, the DF RDPMC assignments become - * discontiguous as the additional counters are accessible starting - * from index 16. - */ - if (is_nb_event(event) && hwc->idx >=3D NUM_COUNTERS_NB) - hwc->event_base_rdpmc +=3D NUM_COUNTERS_L3; - if (flags & PERF_EF_START) - amd_uncore_start(event, PERF_EF_RELOAD); + event->pmu->start(event, PERF_EF_RELOAD); =20 return 0; } @@ -177,55 +153,36 @@ static int amd_uncore_add(struct perf_event *event, i= nt flags) static void amd_uncore_del(struct perf_event *event, int flags) { int i; - struct amd_uncore *uncore =3D event_to_amd_uncore(event); + struct amd_uncore_pmu *pmu =3D event_to_amd_uncore_pmu(event); + struct amd_uncore_ctx *ctx =3D *per_cpu_ptr(pmu->ctx, event->cpu); struct hw_perf_event *hwc =3D &event->hw; =20 - amd_uncore_stop(event, PERF_EF_UPDATE); + event->pmu->stop(event, PERF_EF_UPDATE); =20 - for (i =3D 0; i < uncore->num_counters; i++) { - if (cmpxchg(&uncore->events[i], event, NULL) =3D=3D event) + for (i =3D 0; i < pmu->num_counters; i++) { + if (cmpxchg(&ctx->events[i], event, NULL) =3D=3D event) break; } =20 hwc->idx =3D -1; } =20 -/* - * Return a full thread and slice mask unless user - * has provided them - */ -static u64 l3_thread_slice_mask(u64 config) -{ - if (boot_cpu_data.x86 <=3D 0x18) - return ((config & AMD64_L3_SLICE_MASK) ? : AMD64_L3_SLICE_MASK) | - ((config & AMD64_L3_THREAD_MASK) ? : AMD64_L3_THREAD_MASK); - - /* - * If the user doesn't specify a threadmask, they're not trying to - * count core 0, so we enable all cores & threads. - * We'll also assume that they want to count slice 0 if they specify - * a threadmask and leave sliceid and enallslices unpopulated. - */ - if (!(config & AMD64_L3_F19H_THREAD_MASK)) - return AMD64_L3_F19H_THREAD_MASK | AMD64_L3_EN_ALL_SLICES | - AMD64_L3_EN_ALL_CORES; - - return config & (AMD64_L3_F19H_THREAD_MASK | AMD64_L3_SLICEID_MASK | - AMD64_L3_EN_ALL_CORES | AMD64_L3_EN_ALL_SLICES | - AMD64_L3_COREID_MASK); -} - static int amd_uncore_event_init(struct perf_event *event) { - struct amd_uncore *uncore; + struct amd_uncore_pmu *pmu; + struct amd_uncore_ctx *ctx; struct hw_perf_event *hwc =3D &event->hw; - u64 event_mask =3D AMD64_RAW_EVENT_MASK_NB; =20 if (event->attr.type !=3D event->pmu->type) return -ENOENT; =20 - if (pmu_version >=3D 2 && is_nb_event(event)) - event_mask =3D AMD64_PERFMON_V2_RAW_EVENT_MASK_NB; + if (event->cpu < 0) + return -EINVAL; + + pmu =3D event_to_amd_uncore_pmu(event); + ctx =3D *per_cpu_ptr(pmu->ctx, event->cpu); + if (!ctx) + return -ENODEV; =20 /* * NB and Last level cache counters (MSRs) are shared across all cores @@ -235,28 +192,14 @@ static int amd_uncore_event_init(struct perf_event *e= vent) * out. So we do not support sampling and per-thread events via * CAP_NO_INTERRUPT, and we do not enable counter overflow interrupts: */ - hwc->config =3D event->attr.config & event_mask; + hwc->config =3D event->attr.config; hwc->idx =3D -1; =20 - if (event->cpu < 0) - return -EINVAL; - - /* - * SliceMask and ThreadMask need to be set for certain L3 events. - * For other events, the two fields do not affect the count. - */ - if (l3_mask && is_llc_event(event)) - hwc->config |=3D l3_thread_slice_mask(event->attr.config); - - uncore =3D event_to_amd_uncore(event); - if (!uncore) - return -ENODEV; - /* * since request can come in to any of the shared cores, we will remap * to a single common cpu. */ - event->cpu =3D uncore->cpu; + event->cpu =3D ctx->cpu; =20 return 0; } @@ -278,17 +221,10 @@ static ssize_t amd_uncore_attr_show_cpumask(struct de= vice *dev, struct device_attribute *attr, char *buf) { - cpumask_t *active_mask; - struct pmu *pmu =3D dev_get_drvdata(dev); - - if (pmu->type =3D=3D amd_nb_pmu.type) - active_mask =3D &amd_nb_active_mask; - else if (pmu->type =3D=3D amd_llc_pmu.type) - active_mask =3D &amd_llc_active_mask; - else - return 0; + struct pmu *ptr =3D dev_get_drvdata(dev); + struct amd_uncore_pmu *pmu =3D container_of(ptr, struct amd_uncore_pmu, p= mu); =20 - return cpumap_print_to_pagebuf(true, buf, active_mask); + return cpumap_print_to_pagebuf(true, buf, &pmu->active_mask); } static DEVICE_ATTR(cpumask, S_IRUGO, amd_uncore_attr_show_cpumask, NULL); =20 @@ -396,113 +332,57 @@ static const struct attribute_group *amd_uncore_l3_a= ttr_update[] =3D { NULL, }; =20 -static struct pmu amd_nb_pmu =3D { - .task_ctx_nr =3D perf_invalid_context, - .attr_groups =3D amd_uncore_df_attr_groups, - .name =3D "amd_nb", - .event_init =3D amd_uncore_event_init, - .add =3D amd_uncore_add, - .del =3D amd_uncore_del, - .start =3D amd_uncore_start, - .stop =3D amd_uncore_stop, - .read =3D amd_uncore_read, - .capabilities =3D PERF_PMU_CAP_NO_EXCLUDE | PERF_PMU_CAP_NO_INTERRUPT, - .module =3D THIS_MODULE, -}; - -static struct pmu amd_llc_pmu =3D { - .task_ctx_nr =3D perf_invalid_context, - .attr_groups =3D amd_uncore_l3_attr_groups, - .attr_update =3D amd_uncore_l3_attr_update, - .name =3D "amd_l2", - .event_init =3D amd_uncore_event_init, - .add =3D amd_uncore_add, - .del =3D amd_uncore_del, - .start =3D amd_uncore_start, - .stop =3D amd_uncore_stop, - .read =3D amd_uncore_read, - .capabilities =3D PERF_PMU_CAP_NO_EXCLUDE | PERF_PMU_CAP_NO_INTERRUPT, - .module =3D THIS_MODULE, -}; - -static struct amd_uncore *amd_uncore_alloc(unsigned int cpu) -{ - return kzalloc_node(sizeof(struct amd_uncore), GFP_KERNEL, - cpu_to_node(cpu)); -} - -static inline struct perf_event ** -amd_uncore_events_alloc(unsigned int num, unsigned int cpu) -{ - return kzalloc_node(sizeof(struct perf_event *) * num, GFP_KERNEL, - cpu_to_node(cpu)); -} - static int amd_uncore_cpu_up_prepare(unsigned int cpu) { - struct amd_uncore *uncore_nb =3D NULL, *uncore_llc =3D NULL; - - if (amd_uncore_nb) { - *per_cpu_ptr(amd_uncore_nb, cpu) =3D NULL; - uncore_nb =3D amd_uncore_alloc(cpu); - if (!uncore_nb) - goto fail; - uncore_nb->cpu =3D cpu; - uncore_nb->num_counters =3D num_counters_nb; - uncore_nb->rdpmc_base =3D RDPMC_BASE_NB; - uncore_nb->msr_base =3D MSR_F15H_NB_PERF_CTL; - uncore_nb->active_mask =3D &amd_nb_active_mask; - uncore_nb->pmu =3D &amd_nb_pmu; - uncore_nb->events =3D amd_uncore_events_alloc(num_counters_nb, cpu); - if (!uncore_nb->events) + struct amd_uncore_pmu *pmu; + struct amd_uncore_ctx *ctx; + int node =3D cpu_to_node(cpu), i; + + for (i =3D 0; i < num_pmus; i++) { + pmu =3D &pmus[i]; + *per_cpu_ptr(pmu->ctx, cpu) =3D NULL; + ctx =3D kzalloc_node(sizeof(struct amd_uncore_ctx), GFP_KERNEL, + node); + if (!ctx) goto fail; - uncore_nb->id =3D -1; - *per_cpu_ptr(amd_uncore_nb, cpu) =3D uncore_nb; - } =20 - if (amd_uncore_llc) { - *per_cpu_ptr(amd_uncore_llc, cpu) =3D NULL; - uncore_llc =3D amd_uncore_alloc(cpu); - if (!uncore_llc) - goto fail; - uncore_llc->cpu =3D cpu; - uncore_llc->num_counters =3D num_counters_llc; - uncore_llc->rdpmc_base =3D RDPMC_BASE_LLC; - uncore_llc->msr_base =3D MSR_F16H_L2I_PERF_CTL; - uncore_llc->active_mask =3D &amd_llc_active_mask; - uncore_llc->pmu =3D &amd_llc_pmu; - uncore_llc->events =3D amd_uncore_events_alloc(num_counters_llc, cpu); - if (!uncore_llc->events) + ctx->cpu =3D cpu; + ctx->events =3D kzalloc_node(sizeof(struct perf_event *) * + pmu->num_counters, GFP_KERNEL, + node); + if (!ctx->events) goto fail; - uncore_llc->id =3D -1; - *per_cpu_ptr(amd_uncore_llc, cpu) =3D uncore_llc; + + ctx->id =3D -1; + *per_cpu_ptr(pmu->ctx, cpu) =3D ctx; } =20 return 0; =20 fail: - if (uncore_nb) { - kfree(uncore_nb->events); - kfree(uncore_nb); - } + /* Rollback */ + for (; i >=3D 0; i--) { + pmu =3D &pmus[i]; + ctx =3D *per_cpu_ptr(pmu->ctx, cpu); + if (!ctx) + continue; =20 - if (uncore_llc) { - kfree(uncore_llc->events); - kfree(uncore_llc); + kfree(ctx->events); + kfree(ctx); } =20 return -ENOMEM; } =20 -static struct amd_uncore * -amd_uncore_find_online_sibling(struct amd_uncore *this, - struct amd_uncore * __percpu *uncores) +static struct amd_uncore_ctx * +amd_uncore_find_online_sibling(struct amd_uncore_ctx *this, + struct amd_uncore_pmu *pmu) { unsigned int cpu; - struct amd_uncore *that; + struct amd_uncore_ctx *that; =20 for_each_online_cpu(cpu) { - that =3D *per_cpu_ptr(uncores, cpu); + that =3D *per_cpu_ptr(pmu->ctx, cpu); =20 if (!that) continue; @@ -523,24 +403,16 @@ amd_uncore_find_online_sibling(struct amd_uncore *thi= s, =20 static int amd_uncore_cpu_starting(unsigned int cpu) { - unsigned int eax, ebx, ecx, edx; - struct amd_uncore *uncore; - - if (amd_uncore_nb) { - uncore =3D *per_cpu_ptr(amd_uncore_nb, cpu); - cpuid(0x8000001e, &eax, &ebx, &ecx, &edx); - uncore->id =3D ecx & 0xff; - - uncore =3D amd_uncore_find_online_sibling(uncore, amd_uncore_nb); - *per_cpu_ptr(amd_uncore_nb, cpu) =3D uncore; - } - - if (amd_uncore_llc) { - uncore =3D *per_cpu_ptr(amd_uncore_llc, cpu); - uncore->id =3D get_llc_id(cpu); + struct amd_uncore_pmu *pmu; + struct amd_uncore_ctx *ctx; + int i; =20 - uncore =3D amd_uncore_find_online_sibling(uncore, amd_uncore_llc); - *per_cpu_ptr(amd_uncore_llc, cpu) =3D uncore; + for (i =3D 0; i < num_pmus; i++) { + pmu =3D &pmus[i]; + ctx =3D *per_cpu_ptr(pmu->ctx, cpu); + ctx->id =3D pmu->id(cpu); + ctx =3D amd_uncore_find_online_sibling(ctx, pmu); + *per_cpu_ptr(pmu->ctx, cpu) =3D ctx; } =20 return 0; @@ -548,195 +420,359 @@ static int amd_uncore_cpu_starting(unsigned int cpu) =20 static void uncore_clean_online(void) { - struct amd_uncore *uncore; + struct amd_uncore_ctx *ctx; struct hlist_node *n; =20 - hlist_for_each_entry_safe(uncore, n, &uncore_unused_list, node) { - hlist_del(&uncore->node); - kfree(uncore->events); - kfree(uncore); + hlist_for_each_entry_safe(ctx, n, &uncore_unused_list, node) { + hlist_del(&ctx->node); + kfree(ctx->events); + kfree(ctx); } } =20 -static void uncore_online(unsigned int cpu, - struct amd_uncore * __percpu *uncores) +static int amd_uncore_cpu_online(unsigned int cpu) { - struct amd_uncore *uncore =3D *per_cpu_ptr(uncores, cpu); + struct amd_uncore_pmu *pmu; + struct amd_uncore_ctx *ctx; + int i; =20 uncore_clean_online(); =20 - if (cpu =3D=3D uncore->cpu) - cpumask_set_cpu(cpu, uncore->active_mask); + for (i =3D 0; i < num_pmus; i++) { + pmu =3D &pmus[i]; + ctx =3D *per_cpu_ptr(pmu->ctx, cpu); + if (cpu =3D=3D ctx->cpu) + cpumask_set_cpu(cpu, &pmu->active_mask); + } + + return 0; } =20 -static int amd_uncore_cpu_online(unsigned int cpu) +static int amd_uncore_cpu_down_prepare(unsigned int cpu) { - if (amd_uncore_nb) - uncore_online(cpu, amd_uncore_nb); + struct amd_uncore_ctx *this, *that; + struct amd_uncore_pmu *pmu; + int i, j; + + for (i =3D 0; i < num_pmus; i++) { + pmu =3D &pmus[i]; + this =3D *per_cpu_ptr(pmu->ctx, cpu); + + /* this cpu is going down, migrate to a shared sibling if possible */ + for_each_online_cpu(j) { + that =3D *per_cpu_ptr(pmu->ctx, j); + + if (cpu =3D=3D j) + continue; + + if (this =3D=3D that) { + perf_pmu_migrate_context(&pmu->pmu, cpu, j); + cpumask_clear_cpu(cpu, &pmu->active_mask); + cpumask_set_cpu(j, &pmu->active_mask); + that->cpu =3D j; + break; + } + } + } =20 - if (amd_uncore_llc) - uncore_online(cpu, amd_uncore_llc); + return 0; +} + +static int amd_uncore_cpu_dead(unsigned int cpu) +{ + struct amd_uncore_ctx *ctx; + struct amd_uncore_pmu *pmu; + int i; + + for (i =3D 0; i < num_pmus; i++) { + pmu =3D &pmus[i]; + ctx =3D *per_cpu_ptr(pmu->ctx, cpu); + if (cpu =3D=3D ctx->cpu) + cpumask_clear_cpu(cpu, &pmu->active_mask); + + if (!--ctx->refcnt) { + kfree(ctx->events); + kfree(ctx); + } + + *per_cpu_ptr(pmu->ctx, cpu) =3D NULL; + } =20 return 0; } =20 -static void uncore_down_prepare(unsigned int cpu, - struct amd_uncore * __percpu *uncores) +static int amd_uncore_df_id(unsigned int cpu) { - unsigned int i; - struct amd_uncore *this =3D *per_cpu_ptr(uncores, cpu); + unsigned int eax, ebx, ecx, edx; =20 - if (this->cpu !=3D cpu) - return; + cpuid(0x8000001e, &eax, &ebx, &ecx, &edx); =20 - /* this cpu is going down, migrate to a shared sibling if possible */ - for_each_online_cpu(i) { - struct amd_uncore *that =3D *per_cpu_ptr(uncores, i); + return ecx & 0xff; +} =20 - if (cpu =3D=3D i) - continue; +static int amd_uncore_df_event_init(struct perf_event *event) +{ + struct hw_perf_event *hwc =3D &event->hw; + int ret =3D amd_uncore_event_init(event); =20 - if (this =3D=3D that) { - perf_pmu_migrate_context(this->pmu, cpu, i); - cpumask_clear_cpu(cpu, that->active_mask); - cpumask_set_cpu(i, that->active_mask); - that->cpu =3D i; - break; - } - } + if (ret || pmu_version < 2) + return ret; + + hwc->config =3D event->attr.config & + (pmu_version >=3D 2 ? AMD64_PERFMON_V2_RAW_EVENT_MASK_NB : + AMD64_RAW_EVENT_MASK_NB); + + return 0; } =20 -static int amd_uncore_cpu_down_prepare(unsigned int cpu) +static int amd_uncore_df_add(struct perf_event *event, int flags) { - if (amd_uncore_nb) - uncore_down_prepare(cpu, amd_uncore_nb); + int ret =3D amd_uncore_add(event, flags & ~PERF_EF_START); + struct hw_perf_event *hwc =3D &event->hw; + + if (ret) + return ret; + + /* + * The first four DF counters are accessible via RDPMC index 6 to 9 + * followed by the L3 counters from index 10 to 15. For processors + * with more than four DF counters, the DF RDPMC assignments become + * discontiguous as the additional counters are accessible starting + * from index 16. + */ + if (hwc->idx >=3D NUM_COUNTERS_NB) + hwc->event_base_rdpmc +=3D NUM_COUNTERS_L3; =20 - if (amd_uncore_llc) - uncore_down_prepare(cpu, amd_uncore_llc); + /* Delayed start after rdpmc base update */ + if (flags & PERF_EF_START) + amd_uncore_start(event, PERF_EF_RELOAD); =20 return 0; } =20 -static void uncore_dead(unsigned int cpu, struct amd_uncore * __percpu *un= cores) +static int amd_uncore_df_init(void) { - struct amd_uncore *uncore =3D *per_cpu_ptr(uncores, cpu); + struct attribute **df_attr =3D amd_uncore_df_format_attr; + struct amd_uncore_pmu *pmu =3D &pmus[num_pmus]; + union cpuid_0x80000022_ebx ebx; + int ret; =20 - if (cpu =3D=3D uncore->cpu) - cpumask_clear_cpu(cpu, uncore->active_mask); + if (!boot_cpu_has(X86_FEATURE_PERFCTR_NB)) + return 0; =20 - if (!--uncore->refcnt) { - kfree(uncore->events); - kfree(uncore); + /* + * For Family 17h and above, the Northbridge counters are repurposed + * as Data Fabric counters. The PMUs are exported based on family as + * either NB or DF. + */ + strscpy(pmu->name, boot_cpu_data.x86 >=3D 0x17 ? "amd_df" : "amd_nb", + sizeof(pmu->name)); + + pmu->num_counters =3D NUM_COUNTERS_NB; + pmu->msr_base =3D MSR_F15H_NB_PERF_CTL; + pmu->rdpmc_base =3D RDPMC_BASE_NB; + pmu->id =3D amd_uncore_df_id; + + if (pmu_version >=3D 2) { + *df_attr++ =3D &format_attr_event14v2.attr; + *df_attr++ =3D &format_attr_umask12.attr; + ebx.full =3D cpuid_ebx(EXT_PERFMON_DEBUG_FEATURES); + pmu->num_counters =3D ebx.split.num_df_pmc; + } else if (boot_cpu_data.x86 >=3D 0x17) { + *df_attr =3D &format_attr_event14.attr; + } + + pmu->ctx =3D alloc_percpu(struct amd_uncore_ctx *); + if (!pmu->ctx) + return -ENOMEM; + + pmu->pmu =3D (struct pmu) { + .task_ctx_nr =3D perf_invalid_context, + .attr_groups =3D amd_uncore_df_attr_groups, + .name =3D pmu->name, + .event_init =3D amd_uncore_df_event_init, + .add =3D amd_uncore_df_add, + .del =3D amd_uncore_del, + .start =3D amd_uncore_start, + .stop =3D amd_uncore_stop, + .read =3D amd_uncore_read, + .capabilities =3D PERF_PMU_CAP_NO_EXCLUDE | PERF_PMU_CAP_NO_INTERRUPT, + .module =3D THIS_MODULE, + }; + + ret =3D perf_pmu_register(&pmu->pmu, pmu->pmu.name, -1); + if (ret) { + free_percpu(pmu->ctx); + pmu->ctx =3D NULL; + return ret; } =20 - *per_cpu_ptr(uncores, cpu) =3D NULL; + pr_info("%d %s %s counters detected\n", pmu->num_counters, + boot_cpu_data.x86_vendor =3D=3D X86_VENDOR_HYGON ? "HYGON" : "", + pmu->pmu.name); + + num_pmus++; + + return 0; } =20 -static int amd_uncore_cpu_dead(unsigned int cpu) +static int amd_uncore_l3_id(unsigned int cpu) { - if (amd_uncore_nb) - uncore_dead(cpu, amd_uncore_nb); + return get_llc_id(cpu); +} + +static int amd_uncore_l3_event_init(struct perf_event *event) +{ + int ret =3D amd_uncore_event_init(event); + struct hw_perf_event *hwc =3D &event->hw; + u64 config =3D event->attr.config; + u64 mask; + + hwc->config =3D config & AMD64_RAW_EVENT_MASK_NB; + + /* + * SliceMask and ThreadMask need to be set for certain L3 events. + * For other events, the two fields do not affect the count. + */ + if (ret || boot_cpu_data.x86 < 0x17) + return ret; + + mask =3D config & (AMD64_L3_F19H_THREAD_MASK | AMD64_L3_SLICEID_MASK | + AMD64_L3_EN_ALL_CORES | AMD64_L3_EN_ALL_SLICES | + AMD64_L3_COREID_MASK); + + if (boot_cpu_data.x86 <=3D 0x18) + mask =3D ((config & AMD64_L3_SLICE_MASK) ? : AMD64_L3_SLICE_MASK) | + ((config & AMD64_L3_THREAD_MASK) ? : AMD64_L3_THREAD_MASK); + + /* + * If the user doesn't specify a ThreadMask, they're not trying to + * count core 0, so we enable all cores & threads. + * We'll also assume that they want to count slice 0 if they specify + * a ThreadMask and leave SliceId and EnAllSlices unpopulated. + */ + else if (!(config & AMD64_L3_F19H_THREAD_MASK)) + mask =3D AMD64_L3_F19H_THREAD_MASK | AMD64_L3_EN_ALL_SLICES | + AMD64_L3_EN_ALL_CORES; =20 - if (amd_uncore_llc) - uncore_dead(cpu, amd_uncore_llc); + hwc->config |=3D mask; =20 return 0; } =20 -static int __init amd_uncore_init(void) +static int amd_uncore_l3_init(void) { - struct attribute **df_attr =3D amd_uncore_df_format_attr; struct attribute **l3_attr =3D amd_uncore_l3_format_attr; - union cpuid_0x80000022_ebx ebx; - int ret =3D -ENODEV; + struct amd_uncore_pmu *pmu =3D &pmus[num_pmus]; + int ret; =20 - if (boot_cpu_data.x86_vendor !=3D X86_VENDOR_AMD && - boot_cpu_data.x86_vendor !=3D X86_VENDOR_HYGON) - return -ENODEV; + if (!boot_cpu_has(X86_FEATURE_PERFCTR_LLC)) + return 0; =20 - if (!boot_cpu_has(X86_FEATURE_TOPOEXT)) - return -ENODEV; + /* + * For Family 17h and above, L3 cache counters are available instead + * of L2 cache counters. The PMUs are exported based on family as + * either L2 or L3. + */ + strscpy(pmu->name, boot_cpu_data.x86 >=3D 0x17 ? "amd_l3" : "amd_l2", + sizeof(pmu->name)); =20 - if (boot_cpu_has(X86_FEATURE_PERFMON_V2)) - pmu_version =3D 2; + pmu->num_counters =3D NUM_COUNTERS_L2; + pmu->msr_base =3D MSR_F16H_L2I_PERF_CTL; + pmu->rdpmc_base =3D RDPMC_BASE_LLC; + pmu->id =3D amd_uncore_l3_id; =20 - num_counters_nb =3D NUM_COUNTERS_NB; - num_counters_llc =3D NUM_COUNTERS_L2; if (boot_cpu_data.x86 >=3D 0x17) { - /* - * For F17h and above, the Northbridge counters are - * repurposed as Data Fabric counters. Also, L3 - * counters are supported too. The PMUs are exported - * based on family as either L2 or L3 and NB or DF. - */ - num_counters_llc =3D NUM_COUNTERS_L3; - amd_nb_pmu.name =3D "amd_df"; - amd_llc_pmu.name =3D "amd_l3"; - l3_mask =3D true; + *l3_attr++ =3D &format_attr_event8.attr; + *l3_attr++ =3D &format_attr_umask8.attr; + *l3_attr++ =3D boot_cpu_data.x86 >=3D 0x19 ? + &format_attr_threadmask2.attr : + &format_attr_threadmask8.attr; + pmu->num_counters =3D NUM_COUNTERS_L3; } =20 - if (boot_cpu_has(X86_FEATURE_PERFCTR_NB)) { - if (pmu_version >=3D 2) { - *df_attr++ =3D &format_attr_event14v2.attr; - *df_attr++ =3D &format_attr_umask12.attr; - } else if (boot_cpu_data.x86 >=3D 0x17) { - *df_attr =3D &format_attr_event14.attr; - } + pmu->ctx =3D alloc_percpu(struct amd_uncore_ctx *); + if (!pmu->ctx) + return -ENOMEM; + + pmu->pmu =3D (struct pmu) { + .task_ctx_nr =3D perf_invalid_context, + .attr_groups =3D amd_uncore_l3_attr_groups, + .attr_update =3D amd_uncore_l3_attr_update, + .name =3D pmu->name, + .event_init =3D amd_uncore_l3_event_init, + .add =3D amd_uncore_add, + .del =3D amd_uncore_del, + .start =3D amd_uncore_start, + .stop =3D amd_uncore_stop, + .read =3D amd_uncore_read, + .capabilities =3D PERF_PMU_CAP_NO_EXCLUDE | PERF_PMU_CAP_NO_INTERRUPT, + .module =3D THIS_MODULE, + }; + + ret =3D perf_pmu_register(&pmu->pmu, pmu->pmu.name, -1); + if (ret) { + free_percpu(pmu->ctx); + pmu->ctx =3D NULL; + return ret; + } =20 - amd_uncore_nb =3D alloc_percpu(struct amd_uncore *); - if (!amd_uncore_nb) { - ret =3D -ENOMEM; - goto fail_nb; - } - ret =3D perf_pmu_register(&amd_nb_pmu, amd_nb_pmu.name, -1); - if (ret) - goto fail_nb; + pr_info("%d %s %s counters detected\n", pmu->num_counters, + boot_cpu_data.x86_vendor =3D=3D X86_VENDOR_HYGON ? "HYGON" : "", + pmu->pmu.name); =20 - if (pmu_version >=3D 2) { - ebx.full =3D cpuid_ebx(EXT_PERFMON_DEBUG_FEATURES); - num_counters_nb =3D ebx.split.num_df_pmc; - } + num_pmus++; =20 - pr_info("%d %s %s counters detected\n", num_counters_nb, - boot_cpu_data.x86_vendor =3D=3D X86_VENDOR_HYGON ? "HYGON" : "", - amd_nb_pmu.name); + return 0; +} =20 - ret =3D 0; - } +static void uncore_free(void) +{ + struct amd_uncore_pmu *pmu; + int i; =20 - if (boot_cpu_has(X86_FEATURE_PERFCTR_LLC)) { - if (boot_cpu_data.x86 >=3D 0x19) { - *l3_attr++ =3D &format_attr_event8.attr; - *l3_attr++ =3D &format_attr_umask8.attr; - *l3_attr++ =3D &format_attr_threadmask2.attr; - } else if (boot_cpu_data.x86 >=3D 0x17) { - *l3_attr++ =3D &format_attr_event8.attr; - *l3_attr++ =3D &format_attr_umask8.attr; - *l3_attr++ =3D &format_attr_threadmask8.attr; - } + for (i =3D 0; i < num_pmus; i++) { + pmu =3D &pmus[i]; + if (!pmu->ctx) + continue; =20 - amd_uncore_llc =3D alloc_percpu(struct amd_uncore *); - if (!amd_uncore_llc) { - ret =3D -ENOMEM; - goto fail_llc; - } - ret =3D perf_pmu_register(&amd_llc_pmu, amd_llc_pmu.name, -1); - if (ret) - goto fail_llc; - - pr_info("%d %s %s counters detected\n", num_counters_llc, - boot_cpu_data.x86_vendor =3D=3D X86_VENDOR_HYGON ? "HYGON" : "", - amd_llc_pmu.name); - ret =3D 0; + perf_pmu_unregister(&pmu->pmu); + free_percpu(pmu->ctx); + pmu->ctx =3D NULL; } =20 + num_pmus =3D 0; +} + +static int __init amd_uncore_init(void) +{ + int ret; + + if (boot_cpu_data.x86_vendor !=3D X86_VENDOR_AMD && + boot_cpu_data.x86_vendor !=3D X86_VENDOR_HYGON) + return -ENODEV; + + if (!boot_cpu_has(X86_FEATURE_TOPOEXT)) + return -ENODEV; + + if (boot_cpu_has(X86_FEATURE_PERFMON_V2)) + pmu_version =3D 2; + + ret =3D amd_uncore_df_init(); + if (ret) + goto fail; + + ret =3D amd_uncore_l3_init(); + if (ret) + goto fail; + /* * Install callbacks. Core will call them for each online cpu. */ if (cpuhp_setup_state(CPUHP_PERF_X86_AMD_UNCORE_PREP, "perf/x86/amd/uncore:prepare", amd_uncore_cpu_up_prepare, amd_uncore_cpu_dead)) - goto fail_llc; + goto fail; =20 if (cpuhp_setup_state(CPUHP_AP_PERF_X86_AMD_UNCORE_STARTING, "perf/x86/amd/uncore:starting", @@ -753,12 +789,8 @@ static int __init amd_uncore_init(void) cpuhp_remove_state(CPUHP_AP_PERF_X86_AMD_UNCORE_STARTING); fail_prep: cpuhp_remove_state(CPUHP_PERF_X86_AMD_UNCORE_PREP); -fail_llc: - if (boot_cpu_has(X86_FEATURE_PERFCTR_NB)) - perf_pmu_unregister(&amd_nb_pmu); - free_percpu(amd_uncore_llc); -fail_nb: - free_percpu(amd_uncore_nb); +fail: + uncore_free(); =20 return ret; } @@ -768,18 +800,7 @@ static void __exit amd_uncore_exit(void) cpuhp_remove_state(CPUHP_AP_PERF_X86_AMD_UNCORE_ONLINE); cpuhp_remove_state(CPUHP_AP_PERF_X86_AMD_UNCORE_STARTING); cpuhp_remove_state(CPUHP_PERF_X86_AMD_UNCORE_PREP); - - if (boot_cpu_has(X86_FEATURE_PERFCTR_LLC)) { - perf_pmu_unregister(&amd_llc_pmu); - free_percpu(amd_uncore_llc); - amd_uncore_llc =3D NULL; - } - - if (boot_cpu_has(X86_FEATURE_PERFCTR_NB)) { - perf_pmu_unregister(&amd_nb_pmu); - free_percpu(amd_uncore_nb); - amd_uncore_nb =3D NULL; - } + uncore_free(); } =20 module_init(amd_uncore_init); --=20 2.34.1