From nobody Sat Sep 26 23:52:45 2026 Received: from mail-pj1-f45.google.com (mail-pj1-f45.google.com [209.85.216.45]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3DFA03E276A for ; Fri, 28 Aug 2026 08:58:24 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.45 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787907508; cv=none; b=Op4USpNLbQSdJ121yKvT7Rad7pY44QmS8sv2QdE0iGc+jhXHj8WEOSz8jOA8jrTwVK+2jjX+GzSoynY38XsCsf6O/TqpFAgpfQr5nIHK4fpAsv7i/Dy5fiaA4MLhUiQw12jsDJR+o2U/HKAkrhE9uk5PcoOHZAwzuH63S/7tqrQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787907508; c=relaxed/simple; bh=Zigv5U8KTWbGoETWKj6n47kYEyLGEPDs/lNItATA8ig=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=W7h8DG7bwDNzPL4HITaUEPkqEKP2Tz4bPXwzw8YO0PdkSHnPu4+agMc1rKwKA86c7uagsCP0xduiidX7Mawn/W04bYg+2ziH1pHb3ri3xDrXCXtDrgTzJ9nGnelX773glem6ekcDkMcRM7Ua1wLtWeg0bDVeLnhkFOmgO44+erc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=sifive.com; spf=pass smtp.mailfrom=sifive.com; dkim=pass (2048-bit key) header.d=sifive.com header.i=@sifive.com header.b=gK9Q6LI2; arc=none smtp.client-ip=209.85.216.45 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=sifive.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=sifive.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=sifive.com header.i=@sifive.com header.b="gK9Q6LI2" Received: by mail-pj1-f45.google.com with SMTP id 98e67ed59e1d1-38dc69c74b8so620713a91.0 for ; Fri, 28 Aug 2026 01:58:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1787907504; x=1788512304; darn=vger.kernel.org; h=content-transfer-encoding:content-type:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to:content-type; bh=PmAaTzAU98udvTIz1G8zKGX+O7LjWQdY9YLkmOflWxY=; b=gK9Q6LI2sVGHLnofGla+AlX+1iGIothS2kesUlT6jnGzUInx8hkv8B7+H0b/hr/raC INHTvOBO/uYa5xp5dE+Vf0qrL007SMhnFvFG9axHb2VEJLtLVs6baSvjIxfDoXdxb0lU X6iMAnjVlVBJwsM7cFN75KFzCXuUfy81LxZyzJZDniVGeCdU+rL9IW8fyMiMEU4zOisZ K8VegECnXa64I5HV1RPAj+Y+dDN4x6SPidL3poT1h6v1XzFarrVr3CjD6MPSU2wZFqos 9tgDB/XwsXsaRgU08WMUSBD9K+Q3vCuxOlEdPQrNyA+nThEU3nLWrgaFz2zIatKYNh9N XhdQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787907504; x=1788512304; h=content-transfer-encoding:content-type:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=PmAaTzAU98udvTIz1G8zKGX+O7LjWQdY9YLkmOflWxY=; b=omtyFWxNg+XmxTQTMajqtRF4j9/CcxJwCaBGs7BLE0HKeNMhbNYSWo9ekaI69h5k6M b1h7sNrSHrqG6/ra/a/r6a0jLeCl7y+hPmeDLVjgKUpAhrp1DYNcd3+OIAbCslq8EYmI +PglN+sdXZJI4Z14EKlif/J8Rge9gH55Eatu03AY8SqG9botfyBntMwlHENvAk7HNm/4 WJZ78vyTLJe9JJG53S0erYQ3xHGJT2KlqTxqrcWBcrM6joa06NRI4+OaknpJPbVCL6/h jV0zYKd4L6vxh7uFKkjQIE5cKSq8KLKi7pcMrk11QD490CbsqmMrI6qSFC9EIRWxVqWk Y7bw== X-Forwarded-Encrypted: i=1; AHgh+RpOuhGhzuHnsI3GNe75hdknv4kMbdv742NMiT/+K6hkI1U4AjLG2TIg2VtkPOFWB+kqV47Xy8gUgxRtF+g=@vger.kernel.org X-Gm-Message-State: AFuF++llQyloPTZErDnTsNnvNz/9OrF0zeVbrOG9WSKTm7NPjo34EX0Y djgcICvlm9NsgbhMTjFoTPBMbZ5RSoWkwq/ODPZESyhsBx8LZIymYjnHb5Wuzx5B8RE= X-Gm-Gg: AR+sD13hdncbEi6lAXyHSVjXMB7YZXzAs41VpntgnmK2ayt64w+azFybkJ1M6qIJJmT WrfKhCSX/UWQs1WfYis4VICXIuHkWokxSKmW66Yi1CFPOed1ZVy8+92s0GeMG9Xvd1SM0wLOqIc ZFv2vWwnreQuDB5VTIrvuUoxgBn3XT06ZU6L/FeHYiXSu+W2YHOZpXDp1kIXb4DscCmj1kPPygC YY2HedXpsiAfAOE2LvSDM+vXfyYQQ0+2MX0qoy4bWJ/Swht1rjtQoOLj0xuvHibOXdlilMYgEHf W8ZocJPp53kIRMr71ei2Odwc02s0CzDokxI9ydza2x6YKtQu/dTV3aNc72YOttj+VXHvKmFTGRw QazvJgoLo7Gvj3nAVvaXPb6GF/cU0Ln1Syxocor0DMgBgIdml/7wtX8d37z5r1r6eMj13LCeaMt JxFMY55vWePGQ+jWF0qZTOICP+uUJKYM07loN5pIDQyJOweVmhElspMMjzzxneLI65Wu24bHV7H qcTmPhQU2Y= X-Received: by 2002:a17:90b:50c6:b0:38e:97f0:aa4b with SMTP id 98e67ed59e1d1-396d1017316mr11667952a91.13.1787907503403; Fri, 28 Aug 2026 01:58:23 -0700 (PDT) Received: from sw04.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id 5a478bee46e88-3286f9ecaa9sm3838814eec.26.2026.08.28.01.58.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 28 Aug 2026 01:58:22 -0700 (PDT) From: Zong Li To: tomasz.jeznach@linux.dev, joro@8bytes.org, will@kernel.org, robin.murphy@arm.com, pjw@kernel.org, palmer@dabbelt.com, aou@eecs.berkeley.edu, alex@ghiti.fr, mark.rutland@arm.com, andrew.jones@oss.qualcomm.com, guoren@kernel.org, david.laight.linux@gmail.com, zhangzhanpeng.jasper@bytedance.com, yang.yicong@picoheart.com, iommu@lists.linux.dev, linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, linux-perf-users@vger.kernel.org Cc: Zong Li , Chen Pei , Fangyu Yu Subject: [PATCH RESEND v7 1/3] drivers/perf: riscv-iommu: add risc-v iommu pmu driver Date: Fri, 28 Aug 2026 01:58:15 -0700 Message-ID: <20260828085819.4076449-2-zong.li@sifive.com> X-Mailer: git-send-email @GIT_VERSION@ In-Reply-To: <20260828085819.4076449-1-zong.li@sifive.com> References: <20260828085819.4076449-1-zong.li@sifive.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Add a new driver to support the RISC-V IOMMU PMU. This is an auxiliary device driver created by the parent RISC-V IOMMU driver. The performance monitor provides counters with filtering support to collect events for specific device ID/process ID, or GSCID/PSCID. The RISC-V IOMMU PMU separates the cycle counter from the event counters. The cycle counter is not associated with iohpmevt0, so a software-defined cycle event is required for the perf subsystem. The number and width of the counters are hardware-implemented and must be detected at runtime. Leave out all the dead cleanup code (i.e. .remove() operation) if the PMU driver is tied to the IOMMU driver and can never realistically be removed. PMU-related definitions are moved into the perf driver, where they are used exclusively. According to RISC-V IOMMU specification Chapter 6: Whether an 8 byte access to an IOMMU register is single-copy atomic is UNSPECIFIED. Use two separate 4 byte accesses for hardware compatibility. Tested-by: Chen Pei Tested-by: Fangyu Yu Reviewed-by: Guo Ren (Alibaba DAMO Academy) Reviewed-by: Yicong Yang Suggested-by: David Laight Suggested-by: Guo Ren Link: https://lore.kernel.org/linux-riscv/20260618143634.7f3dd6c5@pumpkin/ Signed-off-by: Zong Li --- drivers/iommu/riscv/iommu-bits.h | 61 --- drivers/perf/Kconfig | 12 + drivers/perf/Makefile | 1 + drivers/perf/riscv_iommu_pmu.c | 894 +++++++++++++++++++++++++++++++ 4 files changed, 907 insertions(+), 61 deletions(-) create mode 100644 drivers/perf/riscv_iommu_pmu.c diff --git a/drivers/iommu/riscv/iommu-bits.h b/drivers/iommu/riscv/iommu-b= its.h index f2ef9bd3cde9..6b5de913a032 100644 --- a/drivers/iommu/riscv/iommu-bits.h +++ b/drivers/iommu/riscv/iommu-bits.h @@ -192,67 +192,6 @@ enum riscv_iommu_ddtp_modes { #define RISCV_IOMMU_IPSR_PMIP BIT(RISCV_IOMMU_INTR_PM) #define RISCV_IOMMU_IPSR_PIP BIT(RISCV_IOMMU_INTR_PQ) =20 -/* 5.19 Performance monitoring counter overflow status (32bits) */ -#define RISCV_IOMMU_REG_IOCOUNTOVF 0x0058 -#define RISCV_IOMMU_IOCOUNTOVF_CY BIT(0) -#define RISCV_IOMMU_IOCOUNTOVF_HPM GENMASK_ULL(31, 1) - -/* 5.20 Performance monitoring counter inhibits (32bits) */ -#define RISCV_IOMMU_REG_IOCOUNTINH 0x005C -#define RISCV_IOMMU_IOCOUNTINH_CY BIT(0) -#define RISCV_IOMMU_IOCOUNTINH_HPM GENMASK(31, 1) - -/* 5.21 Performance monitoring cycles counter (64bits) */ -#define RISCV_IOMMU_REG_IOHPMCYCLES 0x0060 -#define RISCV_IOMMU_IOHPMCYCLES_COUNTER GENMASK_ULL(62, 0) -#define RISCV_IOMMU_IOHPMCYCLES_OF BIT_ULL(63) - -/* 5.22 Performance monitoring event counters (31 * 64bits) */ -#define RISCV_IOMMU_REG_IOHPMCTR_BASE 0x0068 -#define RISCV_IOMMU_REG_IOHPMCTR(_n) (RISCV_IOMMU_REG_IOHPMCTR_BASE + ((_n= ) * 0x8)) - -/* 5.23 Performance monitoring event selectors (31 * 64bits) */ -#define RISCV_IOMMU_REG_IOHPMEVT_BASE 0x0160 -#define RISCV_IOMMU_REG_IOHPMEVT(_n) (RISCV_IOMMU_REG_IOHPMEVT_BASE + ((_n= ) * 0x8)) -#define RISCV_IOMMU_IOHPMEVT_EVENTID GENMASK_ULL(14, 0) -#define RISCV_IOMMU_IOHPMEVT_DMASK BIT_ULL(15) -#define RISCV_IOMMU_IOHPMEVT_PID_PSCID GENMASK_ULL(35, 16) -#define RISCV_IOMMU_IOHPMEVT_DID_GSCID GENMASK_ULL(59, 36) -#define RISCV_IOMMU_IOHPMEVT_PV_PSCV BIT_ULL(60) -#define RISCV_IOMMU_IOHPMEVT_DV_GSCV BIT_ULL(61) -#define RISCV_IOMMU_IOHPMEVT_IDT BIT_ULL(62) -#define RISCV_IOMMU_IOHPMEVT_OF BIT_ULL(63) - -/* Number of defined performance-monitoring event selectors */ -#define RISCV_IOMMU_IOHPMEVT_CNT 31 - -/** - * enum riscv_iommu_hpmevent_id - Performance-monitoring event identifier - * - * @RISCV_IOMMU_HPMEVENT_INVALID: Invalid event, do not count - * @RISCV_IOMMU_HPMEVENT_URQ: Untranslated requests - * @RISCV_IOMMU_HPMEVENT_TRQ: Translated requests - * @RISCV_IOMMU_HPMEVENT_ATS_RQ: ATS translation requests - * @RISCV_IOMMU_HPMEVENT_TLB_MISS: TLB misses - * @RISCV_IOMMU_HPMEVENT_DD_WALK: Device directory walks - * @RISCV_IOMMU_HPMEVENT_PD_WALK: Process directory walks - * @RISCV_IOMMU_HPMEVENT_S_VS_WALKS: First-stage page table walks - * @RISCV_IOMMU_HPMEVENT_G_WALKS: Second-stage page table walks - * @RISCV_IOMMU_HPMEVENT_MAX: Value to denote maximum Event IDs - */ -enum riscv_iommu_hpmevent_id { - RISCV_IOMMU_HPMEVENT_INVALID =3D 0, - RISCV_IOMMU_HPMEVENT_URQ =3D 1, - RISCV_IOMMU_HPMEVENT_TRQ =3D 2, - RISCV_IOMMU_HPMEVENT_ATS_RQ =3D 3, - RISCV_IOMMU_HPMEVENT_TLB_MISS =3D 4, - RISCV_IOMMU_HPMEVENT_DD_WALK =3D 5, - RISCV_IOMMU_HPMEVENT_PD_WALK =3D 6, - RISCV_IOMMU_HPMEVENT_S_VS_WALKS =3D 7, - RISCV_IOMMU_HPMEVENT_G_WALKS =3D 8, - RISCV_IOMMU_HPMEVENT_MAX =3D 9 -}; - /* 5.24 Translation request IOVA (64bits) */ #define RISCV_IOMMU_REG_TR_REQ_IOVA 0x0258 #define RISCV_IOMMU_TR_REQ_IOVA_VPN GENMASK_ULL(63, 12) diff --git a/drivers/perf/Kconfig b/drivers/perf/Kconfig index 245e7bb763b9..8cce6c2ea626 100644 --- a/drivers/perf/Kconfig +++ b/drivers/perf/Kconfig @@ -105,6 +105,18 @@ config RISCV_PMU_SBI full perf feature support i.e. counter overflow, privilege mode filtering, counter configuration. =20 +config RISCV_IOMMU_PMU + depends on RISCV || COMPILE_TEST + depends on RISCV_IOMMU + bool "RISC-V IOMMU Hardware Performance Monitor" + default y + help + Say Y if you want to use the RISC-V IOMMU performance monitor + implementation. The performance monitor is an optional hardware + feature, and whether it is actually enabled depends on IOMMU + hardware support. If the underlying hardware does not implement + the PMU, this option will have no effect. + config STARFIVE_STARLINK_PMU depends on ARCH_STARFIVE || COMPILE_TEST depends on 64BIT diff --git a/drivers/perf/Makefile b/drivers/perf/Makefile index eb8a022dad9a..90c75f3c0ac1 100644 --- a/drivers/perf/Makefile +++ b/drivers/perf/Makefile @@ -20,6 +20,7 @@ obj-$(CONFIG_QCOM_L3_PMU) +=3D qcom_l3_pmu.o obj-$(CONFIG_RISCV_PMU) +=3D riscv_pmu.o obj-$(CONFIG_RISCV_PMU_LEGACY) +=3D riscv_pmu_legacy.o obj-$(CONFIG_RISCV_PMU_SBI) +=3D riscv_pmu_sbi.o +obj-$(CONFIG_RISCV_IOMMU_PMU) +=3D riscv_iommu_pmu.o obj-$(CONFIG_STARFIVE_STARLINK_PMU) +=3D starfive_starlink_pmu.o obj-$(CONFIG_THUNDERX2_PMU) +=3D thunderx2_pmu.o obj-$(CONFIG_XGENE_PMU) +=3D xgene_pmu.o diff --git a/drivers/perf/riscv_iommu_pmu.c b/drivers/perf/riscv_iommu_pmu.c new file mode 100644 index 000000000000..f6acd56f2f61 --- /dev/null +++ b/drivers/perf/riscv_iommu_pmu.c @@ -0,0 +1,894 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (C) 2026 SiFive + * + * Authors + * Zong Li + */ + +#include +#include +#include +#include +#include + +#include "../iommu/riscv/iommu.h" + +/* 5.19 Performance monitoring counter overflow status (32bits) */ +#define RISCV_IOMMU_REG_IOCOUNTOVF 0x0058 +#define RISCV_IOMMU_IOCOUNTOVF_CY BIT(0) +#define RISCV_IOMMU_IOCOUNTOVF_HPM GENMASK_ULL(31, 1) + +/* 5.20 Performance monitoring counter inhibits (32bits) */ +#define RISCV_IOMMU_REG_IOCOUNTINH 0x005C +#define RISCV_IOMMU_IOCOUNTINH_CY BIT(0) +#define RISCV_IOMMU_IOCOUNTINH_HPM GENMASK(31, 0) + +/* 5.21 Performance monitoring cycles counter (64bits) */ +#define RISCV_IOMMU_REG_IOHPMCYCLES 0x0060 +#define RISCV_IOMMU_IOHPMCYCLES_COUNTER GENMASK_ULL(62, 0) +#define RISCV_IOMMU_IOHPMCYCLES_OF BIT_ULL(63) +#define RISCV_IOMMU_REG_IOHPMCTR(_n) (RISCV_IOMMU_REG_IOHPMCYCLES + ((_n) = * 0x8)) + +/* 5.22 Performance monitoring event counters (31 * 64bits) */ +#define RISCV_IOMMU_REG_IOHPMCTR_BASE 0x0068 +#define RISCV_IOMMU_IOHPMCTR_COUNTER GENMASK_ULL(63, 0) + +/* 5.23 Performance monitoring event selectors (31 * 64bits) */ +#define RISCV_IOMMU_REG_IOHPMEVT_BASE 0x0160 +#define RISCV_IOMMU_REG_IOHPMEVT(_n) (RISCV_IOMMU_REG_IOHPMEVT_BASE + ((_n= ) * 0x8)) +#define RISCV_IOMMU_IOHPMEVT_EVENTID GENMASK_ULL(14, 0) +#define RISCV_IOMMU_IOHPMEVT_DMASK BIT_ULL(15) +#define RISCV_IOMMU_IOHPMEVT_PID_PSCID GENMASK_ULL(35, 16) +#define RISCV_IOMMU_IOHPMEVT_DID_GSCID GENMASK_ULL(59, 36) +#define RISCV_IOMMU_IOHPMEVT_PV_PSCV BIT_ULL(60) +#define RISCV_IOMMU_IOHPMEVT_DV_GSCV BIT_ULL(61) +#define RISCV_IOMMU_IOHPMEVT_IDT BIT_ULL(62) +#define RISCV_IOMMU_IOHPMEVT_OF BIT_ULL(63) +#define RISCV_IOMMU_IOHPMEVT_EVENT GENMASK_ULL(62, 0) + +/* The total number of counters is 31 event counters plus 1 cycle counter = */ +#define RISCV_IOMMU_HPM_COUNTER_NUM 32 + +/* Counter index 0 is the cycle counter, the event counters start at index= 1 */ +#define RISCV_IOMMU_HPM_CYCLE_IDX 0 + +static int cpuhp_state; + +/** + * enum riscv_iommu_hpmevent_id - Performance-monitoring event identifier + * + * @RISCV_IOMMU_HPMEVENT_CYCLE: Clock cycle counter + * @RISCV_IOMMU_HPMEVENT_URQ: Untranslated requests + * @RISCV_IOMMU_HPMEVENT_TRQ: Translated requests + * @RISCV_IOMMU_HPMEVENT_ATS_RQ: ATS translation requests + * @RISCV_IOMMU_HPMEVENT_TLB_MISS: TLB misses + * @RISCV_IOMMU_HPMEVENT_DD_WALK: Device directory walks + * @RISCV_IOMMU_HPMEVENT_PD_WALK: Process directory walks + * @RISCV_IOMMU_HPMEVENT_S_VS_WALKS: First-stage page table walks + * @RISCV_IOMMU_HPMEVENT_G_WALKS: Second-stage page table walks + * @RISCV_IOMMU_HPMEVENT_MAX: Value to denote maximum Event IDs + * + * The specification does not define an event ID for counting the + * number of clock cycles, meaning there is no associated 'iohpmevt0'. + * Event ID 0 is an invalid event and does not overlap with any valid + * event ID. Let's repurpose ID 0 as the cycle for perf, the cycle + * event is not actually written into any register, it serves solely + * as an identifier. + */ +enum riscv_iommu_hpmevent_id { + RISCV_IOMMU_HPMEVENT_CYCLE =3D 0, + RISCV_IOMMU_HPMEVENT_URQ =3D 1, + RISCV_IOMMU_HPMEVENT_TRQ =3D 2, + RISCV_IOMMU_HPMEVENT_ATS_RQ =3D 3, + RISCV_IOMMU_HPMEVENT_TLB_MISS =3D 4, + RISCV_IOMMU_HPMEVENT_DD_WALK =3D 5, + RISCV_IOMMU_HPMEVENT_PD_WALK =3D 6, + RISCV_IOMMU_HPMEVENT_S_VS_WALKS =3D 7, + RISCV_IOMMU_HPMEVENT_G_WALKS =3D 8, + RISCV_IOMMU_HPMEVENT_MAX =3D 9 +}; + +struct riscv_iommu_pmu { + struct pmu pmu; + struct hlist_node node; + void __iomem *reg; + int on_cpu; + unsigned int irq; + int numa_node; + unsigned int num_counters; + u64 cycle_cntr_mask; + u64 event_cntr_mask; + struct perf_event *events[RISCV_IOMMU_HPM_COUNTER_NUM]; + DECLARE_BITMAP(used_counters, RISCV_IOMMU_HPM_COUNTER_NUM); +}; + +#define to_riscv_iommu_pmu(p) (container_of(p, struct riscv_iommu_pmu, pmu= )) + +#define RISCV_IOMMU_PMU_ATTR_EXTRACTOR(_name, _mask) \ + static inline u32 get_##_name(struct perf_event *event) \ + { \ + return FIELD_GET(_mask, event->attr.config); \ + } \ + +RISCV_IOMMU_PMU_ATTR_EXTRACTOR(event, RISCV_IOMMU_IOHPMEVT_EVENTID); +RISCV_IOMMU_PMU_ATTR_EXTRACTOR(partial_matching, RISCV_IOMMU_IOHPMEVT_DMAS= K); +RISCV_IOMMU_PMU_ATTR_EXTRACTOR(pid_pscid, RISCV_IOMMU_IOHPMEVT_PID_PSCID); +RISCV_IOMMU_PMU_ATTR_EXTRACTOR(did_gscid, RISCV_IOMMU_IOHPMEVT_DID_GSCID); +RISCV_IOMMU_PMU_ATTR_EXTRACTOR(filter_pid_pscid, RISCV_IOMMU_IOHPMEVT_PV_P= SCV); +RISCV_IOMMU_PMU_ATTR_EXTRACTOR(filter_did_gscid, RISCV_IOMMU_IOHPMEVT_DV_G= SCV); +RISCV_IOMMU_PMU_ATTR_EXTRACTOR(filter_id_type, RISCV_IOMMU_IOHPMEVT_IDT); + +/* Formats */ +PMU_FORMAT_ATTR(event, "config:0-14"); +PMU_FORMAT_ATTR(partial_matching, "config:15"); +PMU_FORMAT_ATTR(pid_pscid, "config:16-35"); +PMU_FORMAT_ATTR(did_gscid, "config:36-59"); +PMU_FORMAT_ATTR(filter_pid_pscid, "config:60"); +PMU_FORMAT_ATTR(filter_did_gscid, "config:61"); +PMU_FORMAT_ATTR(filter_id_type, "config:62"); + +static struct attribute *riscv_iommu_pmu_formats[] =3D { + &format_attr_event.attr, + &format_attr_partial_matching.attr, + &format_attr_pid_pscid.attr, + &format_attr_did_gscid.attr, + &format_attr_filter_pid_pscid.attr, + &format_attr_filter_did_gscid.attr, + &format_attr_filter_id_type.attr, + NULL, +}; + +static const struct attribute_group riscv_iommu_pmu_format_group =3D { + .name =3D "format", + .attrs =3D riscv_iommu_pmu_formats, +}; + +/* Events */ +static ssize_t riscv_iommu_pmu_event_show(struct device *dev, + struct device_attribute *attr, + char *page) +{ + struct perf_pmu_events_attr *pmu_attr; + + pmu_attr =3D container_of(attr, struct perf_pmu_events_attr, attr); + + return sysfs_emit(page, "event=3D0x%02llx\n", pmu_attr->id); +} + +#define RISCV_IOMMU_PMU_EVENT_ATTR(name, id) \ + PMU_EVENT_ATTR_ID(name, riscv_iommu_pmu_event_show, id) + +static struct attribute *riscv_iommu_pmu_events[] =3D { + RISCV_IOMMU_PMU_EVENT_ATTR(cycle, RISCV_IOMMU_HPMEVENT_CYCLE), + RISCV_IOMMU_PMU_EVENT_ATTR(untranslated_req, RISCV_IOMMU_HPMEVENT_URQ), + RISCV_IOMMU_PMU_EVENT_ATTR(translated_req, RISCV_IOMMU_HPMEVENT_TRQ), + RISCV_IOMMU_PMU_EVENT_ATTR(ats_trans_req, RISCV_IOMMU_HPMEVENT_ATS_RQ), + RISCV_IOMMU_PMU_EVENT_ATTR(tlb_miss, RISCV_IOMMU_HPMEVENT_TLB_MISS), + RISCV_IOMMU_PMU_EVENT_ATTR(ddt_walks, RISCV_IOMMU_HPMEVENT_DD_WALK), + RISCV_IOMMU_PMU_EVENT_ATTR(pdt_walks, RISCV_IOMMU_HPMEVENT_PD_WALK), + RISCV_IOMMU_PMU_EVENT_ATTR(s_vs_pt_walks, RISCV_IOMMU_HPMEVENT_S_VS_WALKS= ), + RISCV_IOMMU_PMU_EVENT_ATTR(g_pt_walks, RISCV_IOMMU_HPMEVENT_G_WALKS), + NULL, +}; + +static const struct attribute_group riscv_iommu_pmu_events_group =3D { + .name =3D "events", + .attrs =3D riscv_iommu_pmu_events, +}; + +/* cpumask */ +static ssize_t riscv_iommu_cpumask_show(struct device *dev, + struct device_attribute *attr, + char *buf) +{ + struct riscv_iommu_pmu *pmu =3D to_riscv_iommu_pmu(dev_get_drvdata(dev)); + int on_cpu =3D pmu->on_cpu; + + /* + * riscv_iommu_pmu_offline_cpu() leaves on_cpu at -1 when it cannot + * find another online CPU to migrate to. Report an empty mask rather + * than feeding -1 to cpumask_of(), which indexes out of bounds. + */ + if (on_cpu < 0) + return sysfs_emit(buf, "%*pbl\n", cpumask_pr_args(cpu_none_mask)); + + return sysfs_emit(buf, "%*pbl\n", cpumask_pr_args(cpumask_of(on_cpu))); +} + +static struct device_attribute riscv_iommu_cpumask_attr =3D + __ATTR(cpumask, 0444, riscv_iommu_cpumask_show, NULL); + +static struct attribute *riscv_iommu_cpumask_attrs[] =3D { + &riscv_iommu_cpumask_attr.attr, + NULL +}; + +static const struct attribute_group riscv_iommu_pmu_cpumask_group =3D { + .attrs =3D riscv_iommu_cpumask_attrs, +}; + +static const struct attribute_group *riscv_iommu_pmu_attr_grps[] =3D { + &riscv_iommu_pmu_cpumask_group, + &riscv_iommu_pmu_format_group, + &riscv_iommu_pmu_events_group, + NULL, +}; + +/* + * Register access wrapper + * + * According to RISC-V IOMMU specification Chapter 6: + * A 4 byte access to an IOMMU register must be single-copy atomic. + * Whether an 8 byte access to an IOMMU register is single-copy atomic is = UNSPECIFIED + * + * Use two separate 4 byte accesses for hardware compatibility + */ +static u64 riscv_iommu_pmu_readq(void __iomem *addr) +{ + return hi_lo_readq(addr); +} + +static void riscv_iommu_pmu_writeq(u64 value, void __iomem *addr) +{ + hi_lo_writeq(value, addr); +} + +/* PMU Operations */ +static void riscv_iommu_pmu_set_counter(struct riscv_iommu_pmu *pmu, u32 i= dx, + u64 value) +{ + u64 counter_mask =3D idx ? pmu->event_cntr_mask : pmu->cycle_cntr_mask; + + riscv_iommu_pmu_writeq(value & counter_mask, pmu->reg + RISCV_IOMMU_REG_I= OHPMCTR(idx)); +} + +/* + * As stated in the RISC-V IOMMU Specification, Chapter 6: + * Whether an 8 byte access to an IOMMU register is single-copy atomic + * is UNSPECIFIED, and such an access may appear, internally to the + * IOMMU, as if two separate 4 byte accesses -=E2=80=89first to the high h= alf + * and second to the low half=E2=80=89-=E2=80=89were performed + * + * To make sure the driver works correctly on different hardware, + * the software will always use two 4-byte access for the counter. + * + * This function implements the hi-lo-hi pattern to detect and handle + * wraparound during the read operation: + * 1. Read high half (hi) + * 2. Read low half (lo) + * 3. Read high half again (hi_again) + * + * If both reads of the high half agree, then the low half did not carry + * into the high half in between, so the two halves belong together. If + * they differ, the low half wrapped during the read and is re-read to + * pair it with the high half observed after the carry. A second carry + * cannot follow within these few register accesses, as that would + * require the counter to advance by another 2^32 in the meantime. + * + * Note that the comparison must be made between the two reads of the + * high half within this call. Comparing against a value cached from an + * earlier call cannot work: a carry is invisible to such a check + * whenever the cached low half happens to be smaller than the low half + * observed after the wrap. + */ +static u64 riscv_iommu_pmu_get_counter(struct riscv_iommu_pmu *pmu, u32 id= x) +{ + void __iomem *addr =3D pmu->reg + RISCV_IOMMU_REG_IOHPMCTR(idx); + u64 value, counter_mask =3D idx ? pmu->event_cntr_mask : pmu->cycle_cntr_= mask; + u32 hi, lo, hi_again; + + hi =3D readl(addr + 4); + lo =3D readl(addr); + hi_again =3D readl(addr + 4); + + if (hi_again !=3D hi) { + hi =3D hi_again; + lo =3D readl(addr); + } + + value =3D (((u64)hi << 32) | lo) & counter_mask; + + /* The bit 63 of cycle counter (i.e., idx =3D=3D 0) is OF bit */ + return idx ? value : (value & ~RISCV_IOMMU_IOHPMCYCLES_OF); +} + +static bool is_cycle_event(u64 event) +{ + return FIELD_GET(RISCV_IOMMU_IOHPMEVT_EVENTID, event) =3D=3D + RISCV_IOMMU_HPMEVENT_CYCLE; +} + +static void riscv_iommu_pmu_set_event(struct riscv_iommu_pmu *pmu, u32 idx, + u64 value) +{ + /* There is no associated IOHPMEVT0 for IOHPMCYCLES */ + if (is_cycle_event(value)) + return; + + /* Event counter start from idx 1 */ + riscv_iommu_pmu_writeq(FIELD_GET(RISCV_IOMMU_IOHPMEVT_EVENT, value), + pmu->reg + RISCV_IOMMU_REG_IOHPMEVT(idx - 1)); +} + +static void riscv_iommu_pmu_enable_counter(struct riscv_iommu_pmu *pmu, u3= 2 idx) +{ + void __iomem *addr =3D pmu->reg + RISCV_IOMMU_REG_IOCOUNTINH; + u32 value =3D readl(addr); + + writel(value & ~BIT(idx), addr); +} + +static void riscv_iommu_pmu_disable_counter(struct riscv_iommu_pmu *pmu, u= 32 idx) +{ + void __iomem *addr =3D pmu->reg + RISCV_IOMMU_REG_IOCOUNTINH; + u32 value =3D readl(addr); + + writel(value | BIT(idx), addr); +} + +static void riscv_iommu_pmu_clear_ovf(struct riscv_iommu_pmu *pmu, u32 idx) +{ + u64 value; + + /* Counter is disabled here, making it safe to read and write registers */ + if (idx =3D=3D RISCV_IOMMU_HPM_CYCLE_IDX) { + value =3D riscv_iommu_pmu_readq(pmu->reg + RISCV_IOMMU_REG_IOHPMCYCLES) & + ~RISCV_IOMMU_IOHPMCYCLES_OF; + riscv_iommu_pmu_writeq(value, pmu->reg + RISCV_IOMMU_REG_IOHPMCYCLES); + } else { + /* Event counter start from idx 1 */ + value =3D riscv_iommu_pmu_readq(pmu->reg + RISCV_IOMMU_REG_IOHPMEVT(idx = - 1)) & + ~RISCV_IOMMU_IOHPMEVT_OF; + riscv_iommu_pmu_writeq(value, pmu->reg + RISCV_IOMMU_REG_IOHPMEVT(idx - = 1)); + } +} + +static void riscv_iommu_pmu_start_all(struct riscv_iommu_pmu *pmu, u32 inh= ibit) +{ + writel(inhibit, pmu->reg + RISCV_IOMMU_REG_IOCOUNTINH); +} + +/* Returns the inhibit state prior to stopping, so callers can restore it = later */ +static u32 riscv_iommu_pmu_stop_all(struct riscv_iommu_pmu *pmu) +{ + void __iomem *addr =3D pmu->reg + RISCV_IOMMU_REG_IOCOUNTINH; + u32 inhibit =3D readl(addr); + + writel(GENMASK_U32(pmu->num_counters - 1, 0), addr); + + return inhibit; +} + +/* PMU APIs */ +static void riscv_iommu_pmu_set_period(struct perf_event *event) +{ + struct riscv_iommu_pmu *pmu =3D to_riscv_iommu_pmu(event->pmu); + struct hw_perf_event *hwc =3D &event->hw; + u64 counter_mask =3D hwc->idx ? pmu->event_cntr_mask : pmu->cycle_cntr_ma= sk; + u64 period; + + /* + * Limit the maximum period to prevent the counter value + * from overtaking the one we are about to program. + * In effect we are reducing max_period to account for + * interrupt latency (and we are being very conservative). + */ + period =3D counter_mask >> 1; + riscv_iommu_pmu_set_counter(pmu, hwc->idx, period); + local64_set(&hwc->prev_count, period); +} + +static int riscv_iommu_pmu_event_init(struct perf_event *event) +{ + struct riscv_iommu_pmu *pmu =3D to_riscv_iommu_pmu(event->pmu); + struct hw_perf_event *hwc =3D &event->hw; + struct perf_event *sibling; + int total_event_counters =3D pmu->num_counters - 1; + int counters =3D 0; + int on_cpu; + + if (event->attr.type !=3D event->pmu->type) + return -ENOENT; + + if (is_sampling_event(event)) + return -EOPNOTSUPP; + + /* + * Reject event IDs this driver does not know about. Programming one + * into IOHPMEVT would be accepted by the hardware but would never + * count anything, which is indistinguishable from an idle counter. + */ + if (get_event(event) >=3D RISCV_IOMMU_HPMEVENT_MAX) + return -EINVAL; + + /* + * There is no IOHPMEVT register associated with IOHPMCYCLES, so none + * of the filtering fields can be programmed for the cycle event. + * Reject them here instead of counting unfiltered cycles behind the + * user's back. + */ + if (is_cycle_event(event->attr.config) && + (event->attr.config & ~RISCV_IOMMU_IOHPMEVT_EVENTID)) + return -EINVAL; + + if (event->cpu < 0) + return -EOPNOTSUPP; + + /* + * All events are bound to the CPU the interrupt is affine to. That + * CPU is unset while no online CPU could be found for this PMU, and + * assigning -1 here would turn this into a task bound event, which is + * not something this PMU can serve. + */ + on_cpu =3D pmu->on_cpu; + if (on_cpu < 0) + return -ENODEV; + + event->cpu =3D on_cpu; + + hwc->idx =3D -1; + hwc->config =3D event->attr.config; + + /* + * Account for this event itself first. It has to be done before the + * check below, otherwise an event which is on its own would never be + * matched against the number of counters the hardware implements. + */ + if (!is_cycle_event(event->attr.config)) + if (++counters > total_event_counters) + return -EINVAL; + + if (event->group_leader =3D=3D event) + return 0; + + if (!is_software_event(event->group_leader)) { + if (event->group_leader->pmu !=3D event->pmu) + return -EINVAL; + + if (!is_cycle_event(event->group_leader->attr.config)) + if (++counters > total_event_counters) + return -EINVAL; + } + + for_each_sibling_event(sibling, event->group_leader) { + if (is_software_event(sibling)) + continue; + + if (sibling->pmu !=3D event->pmu) + return -EINVAL; + + if (is_cycle_event(sibling->attr.config)) + continue; + + if (++counters > total_event_counters) + return -EINVAL; + } + + return 0; +} + +static void riscv_iommu_pmu_update(struct perf_event *event) +{ + struct hw_perf_event *hwc =3D &event->hw; + struct riscv_iommu_pmu *pmu =3D to_riscv_iommu_pmu(event->pmu); + u64 delta, prev, now; + u32 idx =3D hwc->idx; + u64 counter_mask =3D idx ? pmu->event_cntr_mask : pmu->cycle_cntr_mask; + + do { + prev =3D local64_read(&hwc->prev_count); + now =3D riscv_iommu_pmu_get_counter(pmu, idx); + } while (local64_cmpxchg(&hwc->prev_count, prev, now) !=3D prev); + + delta =3D (now - prev) & counter_mask; + local64_add(delta, &event->count); +} + +static void riscv_iommu_pmu_start(struct perf_event *event, int flags) +{ + struct riscv_iommu_pmu *pmu =3D to_riscv_iommu_pmu(event->pmu); + struct hw_perf_event *hwc =3D &event->hw; + + if (WARN_ON_ONCE(!(event->hw.state & PERF_HES_STOPPED))) + return; + + if (flags & PERF_EF_RELOAD) + WARN_ON_ONCE(!(event->hw.state & PERF_HES_UPTODATE)); + + hwc->state =3D 0; + riscv_iommu_pmu_set_period(event); + riscv_iommu_pmu_set_event(pmu, hwc->idx, hwc->config); + riscv_iommu_pmu_enable_counter(pmu, hwc->idx); + + perf_event_update_userpage(event); +} + +static void riscv_iommu_pmu_stop(struct perf_event *event, int flags) +{ + struct riscv_iommu_pmu *pmu =3D to_riscv_iommu_pmu(event->pmu); + struct hw_perf_event *hwc =3D &event->hw; + int idx =3D hwc->idx; + + if (hwc->state & PERF_HES_STOPPED) + return; + + riscv_iommu_pmu_disable_counter(pmu, idx); + + if ((flags & PERF_EF_UPDATE) && !(hwc->state & PERF_HES_UPTODATE)) + riscv_iommu_pmu_update(event); + + hwc->state |=3D PERF_HES_STOPPED | PERF_HES_UPTODATE; +} + +static int riscv_iommu_pmu_add(struct perf_event *event, int flags) +{ + struct riscv_iommu_pmu *pmu =3D to_riscv_iommu_pmu(event->pmu); + struct hw_perf_event *hwc =3D &event->hw; + unsigned int num_counters =3D pmu->num_counters; + unsigned int idx; + + /* Reserve index zero for iohpmcycles */ + if (is_cycle_event(event->attr.config)) + idx =3D RISCV_IOMMU_HPM_CYCLE_IDX; + else + idx =3D find_next_zero_bit(pmu->used_counters, num_counters, 1); + + /* All event counters or cycle counter are in use */ + if (idx =3D=3D num_counters || pmu->events[idx]) + return -EAGAIN; + + set_bit(idx, pmu->used_counters); + + pmu->events[idx] =3D event; + hwc->idx =3D idx; + hwc->state =3D PERF_HES_STOPPED | PERF_HES_UPTODATE; + local64_set(&hwc->prev_count, 0); + + if (flags & PERF_EF_START) + riscv_iommu_pmu_start(event, flags); + + /* Propagate changes to the userspace mapping. */ + perf_event_update_userpage(event); + + return 0; +} + +static void riscv_iommu_pmu_read(struct perf_event *event) +{ + riscv_iommu_pmu_update(event); +} + +static void riscv_iommu_pmu_del(struct perf_event *event, int flags) +{ + struct riscv_iommu_pmu *pmu =3D to_riscv_iommu_pmu(event->pmu); + struct hw_perf_event *hwc =3D &event->hw; + int idx =3D hwc->idx; + + riscv_iommu_pmu_stop(event, PERF_EF_UPDATE); + pmu->events[idx] =3D NULL; + clear_bit(idx, pmu->used_counters); + + perf_event_update_userpage(event); +} + +static int riscv_iommu_pmu_online_cpu(unsigned int cpu, struct hlist_node = *node) +{ + struct riscv_iommu_pmu *iommu_pmu; + int ret; + + iommu_pmu =3D hlist_entry_safe(node, struct riscv_iommu_pmu, node); + + if (iommu_pmu->on_cpu !=3D -1) + return 0; + + /* + * Pick a local CPU from the numa node of this iommu for the + * locality, rather than blindly binding to whichever CPU happens + * to trigger this callback (e.g. during initial registration, this + * runs once per already-online CPU in ascending CPU order). + * + * The interrupt has to follow the CPU the events are bound to, the + * same way riscv_iommu_pmu_offline_cpu() moves it when it picks a new + * one. + */ + iommu_pmu->on_cpu =3D cpumask_local_spread(0, iommu_pmu->numa_node); + ret =3D irq_set_affinity(iommu_pmu->irq, cpumask_of(iommu_pmu->on_cpu)); + if (ret) + pr_debug("failed to set irq %u affinity to cpu %d: %d\n", + iommu_pmu->irq, iommu_pmu->on_cpu, ret); + + return 0; +} + +static int riscv_iommu_pmu_offline_cpu(unsigned int cpu, struct hlist_node= *node) +{ + struct riscv_iommu_pmu *iommu_pmu; + unsigned int target_cpu; + int ret; + + iommu_pmu =3D hlist_entry_safe(node, struct riscv_iommu_pmu, node); + + if (iommu_pmu->on_cpu !=3D (int)cpu) + return 0; + + iommu_pmu->on_cpu =3D -1; + + target_cpu =3D cpumask_any_and_but(cpumask_of_node(iommu_pmu->numa_node), + cpu_online_mask, cpu); + if (target_cpu >=3D nr_cpu_ids) + target_cpu =3D cpumask_any_but(cpu_online_mask, cpu); + if (target_cpu >=3D nr_cpu_ids) + return 0; + + perf_pmu_migrate_context(&iommu_pmu->pmu, cpu, target_cpu); + iommu_pmu->on_cpu =3D target_cpu; + ret =3D irq_set_affinity(iommu_pmu->irq, cpumask_of(target_cpu)); + if (ret) + pr_debug("failed to set irq %u affinity to cpu %u: %d\n", + iommu_pmu->irq, target_cpu, ret); + + return 0; +} + +static irqreturn_t riscv_iommu_pmu_irq_handler(int irq, void *dev_id) +{ + struct riscv_iommu_pmu *pmu =3D (struct riscv_iommu_pmu *)dev_id; + DECLARE_BITMAP(ovf_bitmap, BITS_PER_TYPE(u64)); + u32 ovf, idx, inhibit; + + /* Check whether this interrupt is for PMU */ + if (!(readl_relaxed(pmu->reg + RISCV_IOMMU_REG_IPSR) & RISCV_IOMMU_IPSR_P= MIP)) + return IRQ_NONE; + + /* Process PMU IRQ */ + inhibit =3D riscv_iommu_pmu_stop_all(pmu); + + ovf =3D readl(pmu->reg + RISCV_IOMMU_REG_IOCOUNTOVF); + if (ovf) { + bitmap_from_u64(ovf_bitmap, ovf); + for_each_set_bit(idx, ovf_bitmap, pmu->num_counters) { + struct perf_event *event =3D pmu->events[idx]; + + /* + * A counter may overflow just before its event is + * removed by riscv_iommu_pmu_del(), which leaves the + * overflow bit set with no event bound to the counter. + * There is nothing to update in that case, but the + * overflow bit still has to be cleared below, otherwise + * the interrupt would stay pending forever. + */ + if (event) { + riscv_iommu_pmu_update(event); + riscv_iommu_pmu_set_period(event); + } + + riscv_iommu_pmu_clear_ovf(pmu, idx); + } + } + + /* Clear performance monitoring interrupt pending bit */ + writel_relaxed(RISCV_IOMMU_IPSR_PMIP, pmu->reg + RISCV_IOMMU_REG_IPSR); + + riscv_iommu_pmu_start_all(pmu, inhibit); + + return IRQ_HANDLED; +} + +static unsigned int riscv_iommu_pmu_get_irq_num(struct riscv_iommu_device = *iommu) +{ + /* Reuse ICVEC.CIV mask for all interrupt vectors mapping */ + int vec =3D (iommu->icvec >> (RISCV_IOMMU_INTR_PM * 4)) & RISCV_IOMMU_ICV= EC_CIV; + + return iommu->irqs[vec]; +} + +static int riscv_iommu_pmu_request_irq(struct auxiliary_device *auxdev, + struct riscv_iommu_device *iommu, + struct riscv_iommu_pmu *pmu) +{ + /* + * Bind the handler to the auxiliary device, which is the same devres + * scope that frees @pmu. Requesting it on the parent iommu device + * would keep the handler registered with a dangling dev_id once @pmu + * is freed, either on a later probe failure or on device removal. + * + * IRQF_SHARED is required because ICVEC maps the performance + * monitoring source onto one of the vectors the iommu driver already + * requested for its queues whenever fewer than RISCV_IOMMU_INTR_COUNT + * vectors are available. Both requesters have to agree on sharing, or + * this one is rejected with -EBUSY. IRQF_ONESHOT does not have to be + * matched by hand: devm_request_irq() adds IRQF_COND_ONESHOT, so this + * handler adopts whatever the first requester picked. + */ + return devm_request_irq(&auxdev->dev, pmu->irq, riscv_iommu_pmu_irq_handl= er, + IRQF_SHARED | IRQF_NOBALANCING, + dev_name(iommu->dev), pmu); +} + +static void riscv_iommu_pmu_remove_cpuhp_instance(void *data) +{ + struct riscv_iommu_pmu *pmu =3D data; + + cpuhp_state_remove_instance_nocalls(cpuhp_state, &pmu->node); +} + +static void riscv_iommu_pmu_do_unregister(void *data) +{ + struct riscv_iommu_pmu *pmu =3D data; + + perf_pmu_unregister(&pmu->pmu); +} + +static int riscv_iommu_pmu_probe(struct auxiliary_device *auxdev, + const struct auxiliary_device_id *id) +{ + struct riscv_iommu_device *iommu_dev =3D dev_get_platdata(&auxdev->dev); + struct riscv_iommu_pmu *iommu_pmu; + void __iomem *addr; + char *name; + int ret; + + iommu_pmu =3D devm_kzalloc(&auxdev->dev, sizeof(*iommu_pmu), GFP_KERNEL); + if (!iommu_pmu) + return -ENOMEM; + + iommu_pmu->reg =3D iommu_dev->reg; + + /* + * Counter number and width are hardware-implemented, detect them by + * writing 1s and reading back which bits stuck. + * + * The specification requires a minimum of one programmable event + * counter besides the cycles counter when capabilities.HPM is 1, + * which is the condition under which this device is created. So a + * compliant implementation always reports at least two counters, and + * both IOHPMCYCLES and the first IOHPMCTR are always present. A + * readback which says otherwise is non-compliant hardware and is + * rejected rather than worked around. + * + * The implemented counters are assumed to be consecutive, so that + * hweight32() of the IOCOUNTINH readback can be used as the bound on + * valid counter indices. + * + * The counter masks are also assumed to be a contiguous run of bits + * starting at bit 0, which is what riscv_iommu_pmu_update() relies on + * when it masks the difference of two samples to handle wraparound. + */ + addr =3D iommu_pmu->reg + RISCV_IOMMU_REG_IOCOUNTINH; + writel(RISCV_IOMMU_IOCOUNTINH_HPM, addr); + iommu_pmu->num_counters =3D hweight32(readl(addr)); + if (iommu_pmu->num_counters < 2) { + dev_err(&auxdev->dev, "hardware reports %u counter(s)\n", + iommu_pmu->num_counters); + return -ENODEV; + } + + /* Bit 63 of IOHPMCYCLES is the OF bit, not part of the counter */ + addr =3D iommu_pmu->reg + RISCV_IOMMU_REG_IOHPMCYCLES; + riscv_iommu_pmu_writeq(RISCV_IOMMU_IOHPMCYCLES_COUNTER, addr); + iommu_pmu->cycle_cntr_mask =3D riscv_iommu_pmu_readq(addr) & + RISCV_IOMMU_IOHPMCYCLES_COUNTER; + if (!iommu_pmu->cycle_cntr_mask) { + dev_err(&auxdev->dev, "cycles counter is not implemented\n"); + return -ENODEV; + } + + /* Assume the width of all event counters are the same */ + addr =3D iommu_pmu->reg + RISCV_IOMMU_REG_IOHPMCTR_BASE; + riscv_iommu_pmu_writeq(RISCV_IOMMU_IOHPMCTR_COUNTER, addr); + iommu_pmu->event_cntr_mask =3D riscv_iommu_pmu_readq(addr); + if (!iommu_pmu->event_cntr_mask) { + dev_err(&auxdev->dev, "event counter is not implemented\n"); + return -ENODEV; + } + + iommu_pmu->pmu =3D (struct pmu) { + .module =3D THIS_MODULE, + .parent =3D &auxdev->dev, + .task_ctx_nr =3D perf_invalid_context, + .event_init =3D riscv_iommu_pmu_event_init, + .add =3D riscv_iommu_pmu_add, + .del =3D riscv_iommu_pmu_del, + .start =3D riscv_iommu_pmu_start, + .stop =3D riscv_iommu_pmu_stop, + .read =3D riscv_iommu_pmu_read, + .attr_groups =3D riscv_iommu_pmu_attr_grps, + .capabilities =3D PERF_PMU_CAP_NO_EXCLUDE, + }; + + auxiliary_set_drvdata(auxdev, iommu_pmu); + + name =3D devm_kasprintf(&auxdev->dev, GFP_KERNEL, + "riscv_iommu_pmu_%u", auxdev->id); + if (!name) { + dev_err(&auxdev->dev, "Failed to create name riscv_iommu_pmu_%u\n", + auxdev->id); + return -ENOMEM; + } + + iommu_pmu->numa_node =3D dev_to_node(iommu_dev->dev); + iommu_pmu->irq =3D riscv_iommu_pmu_get_irq_num(iommu_dev); + + ret =3D riscv_iommu_pmu_request_irq(auxdev, iommu_dev, iommu_pmu); + if (ret) { + dev_err(&auxdev->dev, "Failed to request irq %s: %d\n", name, ret); + return ret; + } + + /* + * Bind all events to the same cpu context to avoid race enabling. + * riscv_iommu_pmu_online_cpu() picks the CPU and sets the irq + * affinity for us once the instance is registered below. + */ + iommu_pmu->on_cpu =3D -1; + + ret =3D cpuhp_state_add_instance(cpuhp_state, &iommu_pmu->node); + if (ret) { + dev_err(&auxdev->dev, "Failed to register hotplug %s: %d\n", name, ret); + return ret; + } + + ret =3D devm_add_action_or_reset(&auxdev->dev, + riscv_iommu_pmu_remove_cpuhp_instance, + iommu_pmu); + if (ret) + return ret; + + ret =3D perf_pmu_register(&iommu_pmu->pmu, name, -1); + if (ret) { + dev_err(&auxdev->dev, "Failed to register %s: %d\n", name, ret); + return ret; + } + + ret =3D devm_add_action_or_reset(&auxdev->dev, + riscv_iommu_pmu_do_unregister, + iommu_pmu); + if (ret) + return ret; + + /* + * The PMU name only carries the aux dev id, not the iommu dev name, so + * find the iommu dev name here to map this PMU back to its iommu dev. + */ + dev_info(&auxdev->dev, "%s: Registered with %u counters (iommu %s)\n", + name, iommu_pmu->num_counters, dev_name(iommu_dev->dev)); + + return 0; +} + +static const struct auxiliary_device_id riscv_iommu_pmu_id_table[] =3D { + { .name =3D "riscv-iommu.pmu" }, + {} +}; +MODULE_DEVICE_TABLE(auxiliary, riscv_iommu_pmu_id_table); + +static struct auxiliary_driver iommu_pmu_driver =3D { + .driver =3D { + .suppress_bind_attrs =3D true, + }, + .probe =3D riscv_iommu_pmu_probe, + .id_table =3D riscv_iommu_pmu_id_table, +}; + +static int __init riscv_iommu_pmu_init(void) +{ + int ret; + + cpuhp_state =3D cpuhp_setup_state_multi(CPUHP_AP_ONLINE_DYN, + "perf/riscv/iommu:online", + riscv_iommu_pmu_online_cpu, + riscv_iommu_pmu_offline_cpu); + if (cpuhp_state < 0) + return cpuhp_state; + + ret =3D auxiliary_driver_register(&iommu_pmu_driver); + if (ret) + cpuhp_remove_multi_state(cpuhp_state); + + return ret; +} +module_init(riscv_iommu_pmu_init); + +MODULE_DESCRIPTION("RISC-V IOMMU PMU"); +MODULE_LICENSE("GPL"); --=20 2.43.7 From nobody Sat Sep 26 23:52:45 2026 Received: from mail-pg1-f181.google.com (mail-pg1-f181.google.com [209.85.215.181]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A58F53DDDB3 for ; Fri, 28 Aug 2026 08:58:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.181 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787907507; cv=none; b=FbUx4NSrRbBzKYXJt+x1iNVNqjW8l7ulms/pNiBrntPV3IGaEpDoqiTI494Qo54O/CGKNtCFh2Y68+QQf3xoxQSwzOtNE/sTFMydf1tnm6iIx28HHnPtySSSd5jv9Bv2l+0vN1gJ6KkFgC/rJncFhIx8WyuEXl+aT3AzDBh2NdY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787907507; c=relaxed/simple; bh=0OuilxAE+ThUNxpU1Gxpqy4nbLhkHMBngSRff/ONKc8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=ZqmZZ//sgCYUxvjvBxYRNPKp3xr4CO1sysquK4ZN4hMGOhWng1F8wMXwCNhOBFjgskFBO6fc+UPtCfmWvU6fQLX9ZWw8Yfge0RndcZGU66MT+H1Pa2FeWqI3O7JqCLWGnw2PlW9IN4J/ya5QwJZlocTTrWz0RzzOwh+s80JnvC0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=sifive.com; spf=pass smtp.mailfrom=sifive.com; dkim=pass (2048-bit key) header.d=sifive.com header.i=@sifive.com header.b=Rh2n3wLC; arc=none smtp.client-ip=209.85.215.181 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=sifive.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=sifive.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=sifive.com header.i=@sifive.com header.b="Rh2n3wLC" Received: by mail-pg1-f181.google.com with SMTP id 41be03b00d2f7-cbedf433a99so794176a12.2 for ; Fri, 28 Aug 2026 01:58:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1787907504; x=1788512304; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=0jwxUNr7N+2l26FeCwq48z6QyCIJefuKV/bUVltiLUA=; b=Rh2n3wLC1ayEEzDkd5DUTAVtd3gI3JotUY987ulVOT7WP07stpb8lZqbkv5lzpjyum 7yT3SZmUiYS58rRle0FzyxhUJFtAMtFCYtNa6IseXHeJIs7ifKn/4K1t3PvxwUpZSKFs MEYwJ9rXQ6SJuReMDgXNahBRcQx14BxmzqnP23c91ldb+Ks/rA3exJjVwE6GjMw8dlEe 9x0tCxUxIGjowHocrseQUsPQKOwQJ684bY1rupoJmyDfL8nMQIGtcKWBua3rV/qYJoW6 3KujAe+MXMaW59pnEEFWTUZLuZOKMHBer49zfRpFtnPe+TpWCMj/ziznKmuWCR0YrrHF mItg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787907504; x=1788512304; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=0jwxUNr7N+2l26FeCwq48z6QyCIJefuKV/bUVltiLUA=; b=U5JFtvqxkz2m3uPh8wZ8tyA+nmhC5JlFy05ElIrsMAd3+tiloM4mChb4t329RhgPC4 c2gaLEf+5halclSRNIuMFXRVrHHfHM3t5vbiDMMxW0+VcoIYHpFuBpzXqm6UNlV7Y+T4 Uee+UxHybf8b+tkxStbV8elSj92zkiCY3eE8HC2OdRAh/a10eebFDFS1YfrGEG4jpZNq +YWOJ+w04PGFm5+2D04xgvJU1BFMVmwieDQdKJ3Ql4DVbWjVFydDcOO0DV4uqAWeJaqH DlBlFi3aPpuf8gqWidxV1yVlNab7DY0qBiC/bnlQK4TNXzn/mDPFwELvLkTnCBxkfENK 7lXA== X-Forwarded-Encrypted: i=1; AHgh+RornOxoTm7JRDBnyfNYYveAJsk1kkqkvLQaZaunFjbRwJwsoq/m21FGFeENhH2Uq+CP3GqiZ9Xv1opztSk=@vger.kernel.org X-Gm-Message-State: AFuF++mgpLbelTBfQZ7LrOz3YIToV5Q7iOOuFH8lZOKat7LUSKSvIMDb AWfakaFZagzcCfL8HjBGtZx4/qY92t8fVnMiKSFcD121lOa5zdwmkRBBBEJpmq9uAwQ= X-Gm-Gg: AR+sD11CShD2tdDfrgX9OsAX+hb83zPujePmyBw3GSgWsHdox47nMEmdbZhv4o0TS0k FI5tUOxufCQ8gi0sF8EfJ4K9wtkwU7X/rPI9jw4OO9L3peT1vBFp9vIrnSg4yTfyj0s0flWsLnJ 9Yq2JAiIsI4ANsh9r9t4GAzpaDlCHIcHDT1jqQYPHEgWHyHnxfxSTTUm1IgqgxH10i6Q1jfEqOB aay1b3TgMj/DHeiT6U9rLvmg0j/v8fQtAMVVZum3D13l4cxYNnO5Bc7Hj0TLVzdcYc6BqA0OlnA uTSj/0siH4sRshAXg8iL7VNgjAYC837Och+KdfXelz4DUvknfLqVrVJ5TOs7s/0vHb0cPJqbhk3 EK+cLO/XgYVG0rPFJUP171K8YoM6H6tOYdyQyq6mHV0HUVOxEMbpN12saY23yBKEoN4rcgqvTdR IyHtqpZUwgLq7m6QHNApI+N6xKYnmz3238JK2LiK3Ya+2gXU0F3HIoiqnfYV9iuQuLcFKJ3OJM X-Received: by 2002:a05:6a20:7345:b0:3d1:7238:33d9 with SMTP id adf61e73a8af0-3d26532756amr10426288637.3.1787907504506; Fri, 28 Aug 2026 01:58:24 -0700 (PDT) Received: from sw04.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id 5a478bee46e88-3286f9ecaa9sm3838814eec.26.2026.08.28.01.58.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 28 Aug 2026 01:58:24 -0700 (PDT) From: Zong Li To: tomasz.jeznach@linux.dev, joro@8bytes.org, will@kernel.org, robin.murphy@arm.com, pjw@kernel.org, palmer@dabbelt.com, aou@eecs.berkeley.edu, alex@ghiti.fr, mark.rutland@arm.com, andrew.jones@oss.qualcomm.com, guoren@kernel.org, david.laight.linux@gmail.com, zhangzhanpeng.jasper@bytedance.com, yang.yicong@picoheart.com, iommu@lists.linux.dev, linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, linux-perf-users@vger.kernel.org Cc: Zong Li , Chen Pei , Fangyu Yu , Samuel Holland Subject: [PATCH RESEND v7 2/3] iommu/riscv: create a auxiliary device for HPM Date: Fri, 28 Aug 2026 01:58:16 -0700 Message-ID: <20260828085819.4076449-3-zong.li@sifive.com> X-Mailer: git-send-email @GIT_VERSION@ In-Reply-To: <20260828085819.4076449-1-zong.li@sifive.com> References: <20260828085819.4076449-1-zong.li@sifive.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Create an auxiliary device for HPM when the IOMMU supports a hardware performance monitor. Tested-by: Chen Pei Tested-by: Fangyu Yu Reviewed-by: Guo Ren Reviewed-by: Yicong Yang Suggested-by: Samuel Holland Signed-off-by: Zong Li --- drivers/iommu/riscv/Kconfig | 1 + drivers/iommu/riscv/iommu.c | 37 +++++++++++++++++++++++++++++++++++++ 2 files changed, 38 insertions(+) diff --git a/drivers/iommu/riscv/Kconfig b/drivers/iommu/riscv/Kconfig index b86e5ab94183..8025bf0fb67f 100644 --- a/drivers/iommu/riscv/Kconfig +++ b/drivers/iommu/riscv/Kconfig @@ -10,6 +10,7 @@ config RISCV_IOMMU select GENERIC_PT select IOMMU_PT select IOMMU_PT_RISCV64 + select AUXILIARY_BUS help Support for implementations of the RISC-V IOMMU architecture that complements the RISC-V MMU capabilities, providing similar address diff --git a/drivers/iommu/riscv/iommu.c b/drivers/iommu/riscv/iommu.c index cec3ddd7ab10..7f619971bb70 100644 --- a/drivers/iommu/riscv/iommu.c +++ b/drivers/iommu/riscv/iommu.c @@ -14,6 +14,7 @@ =20 #include #include +#include #include #include #include @@ -48,6 +49,9 @@ static DEFINE_IDA(riscv_iommu_pscids); #define RISCV_IOMMU_MAX_PSCID (BIT(20) - 1) =20 +/* IOMMU PMU auxiliary device id allocation namespace. */ +static DEFINE_IDA(riscv_iommu_pmu_ida); + /* Device resource-managed allocations */ struct riscv_iommu_devres { void *addr; @@ -565,6 +569,36 @@ static irqreturn_t riscv_iommu_fltq_process(int irq, v= oid *data) return IRQ_HANDLED; } =20 +/* + * IOMMU Hardware performance monitor + */ +static void riscv_iommu_pmu_id_free(void *data) +{ + ida_free(&riscv_iommu_pmu_ida, (unsigned long)data); +} + +static int riscv_iommu_hpm_enable(struct riscv_iommu_device *iommu) +{ + struct auxiliary_device *auxdev; + int id, ret; + + id =3D ida_alloc(&riscv_iommu_pmu_ida, GFP_KERNEL); + if (id < 0) + return id; + + ret =3D devm_add_action_or_reset(iommu->dev, riscv_iommu_pmu_id_free, + (void *)(unsigned long)id); + if (ret) + return ret; + + auxdev =3D __devm_auxiliary_device_create(iommu->dev, "riscv-iommu", + "pmu", iommu, id); + if (!auxdev) + return -ENODEV; + + return 0; +} + /* Lookup and initialize device context info structure. */ static struct riscv_iommu_dc *riscv_iommu_get_dc(struct riscv_iommu_device= *iommu, unsigned int devid) @@ -1613,6 +1647,9 @@ int riscv_iommu_init(struct riscv_iommu_device *iommu) goto err_remove_sysfs; } =20 + if (iommu->caps & RISCV_IOMMU_CAPABILITIES_HPM) + riscv_iommu_hpm_enable(iommu); + return 0; =20 err_remove_sysfs: --=20 2.43.7 From nobody Sat Sep 26 23:52:45 2026 Received: from mail-pj1-f42.google.com (mail-pj1-f42.google.com [209.85.216.42]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5CFCC3E2746 for ; Fri, 28 Aug 2026 08:58:26 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.42 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787907509; cv=none; b=OkgmRJXRan6EEEvspylGwfSNBJMN+haVNB7F/7/9nyU3Qj3nZjwmEV9xOYsp3I5Wr20ovxaytvQSQQ7oAH3E56ojRjToUty5OfY036L5z1owZDm6uFpd/X+7LhoNOnA61b5S0apH52oLtqp9zZMNJi4sjo2h3mlL+81mts/GSLA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787907509; c=relaxed/simple; bh=lsBZYsAG+nr8THk4a+Io/3qdHObvPBkAo1nshLqp2i4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=TZZ6NKLACjMulhM9OFHRwl2vQ7NC7XvfUqtETMN1lGpO1nYMh4jGdg8D4vh+tWhAxArIq3wIVKHWxUZhvD9lG7WZM1xhRin9cklhgigYT+wqAxnpmiuRBfV8bewKXjz1q/d1uMXRZf1JRFOJnIgA56ge9Jtx1wD8VLi2fbGI0Jc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=sifive.com; spf=pass smtp.mailfrom=sifive.com; dkim=pass (2048-bit key) header.d=sifive.com header.i=@sifive.com header.b=WocUhwhW; arc=none smtp.client-ip=209.85.216.42 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=sifive.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=sifive.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=sifive.com header.i=@sifive.com header.b="WocUhwhW" Received: by mail-pj1-f42.google.com with SMTP id 98e67ed59e1d1-39686abd426so715966a91.1 for ; Fri, 28 Aug 2026 01:58:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1787907506; x=1788512306; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=TRFIKfxO5tXI3eQnrJwwXHHecr77RgRWRC9iwHx3Doo=; b=WocUhwhWpzCjfE65FB/qs0E4H+DCr3xElj0Vh6b8oElbzVQBo7BHEJPgx02BDO9F41 ims9HyllchcRKycebQHTl7tMIogBeRNtkgUpn6fI2G88ykzvv+WJUcYILmvbGKMSxaNU FEeZdsuuKspuyQvkH/c1DQRNOV+Bv3Ww6LQ9KZRHh/7nIQlNWdtKheaOuQrMpNiKvagu pqdwq5+bZKTuGtU3DvupM5gk1qBgOfa2YhpElrbJIulC/WuICojAlMJxdpMV7orcFb8T nl16KUVUADMzoXmesUKSVFuSvatH6jVsvcGpqMooGmYaQUZO6ClVtC3iBHM5R4vfgJjY mxFg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787907506; x=1788512306; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=TRFIKfxO5tXI3eQnrJwwXHHecr77RgRWRC9iwHx3Doo=; b=FfH4Op3fc4JjMWaIOXg5ojC4csDtZkYLss9dipW7HVhlngX85I1pGiObzfw/3elLcn gQ+Ga/cmRiLui7LO2TEt5oqgB9YxO3zz2VB9ac9vTW3r4evij1r0bImhCrC9oOC1Uomf RE+A6lNg2hrvLIJaw+vBHmu2kCHxH0uuwvSjW9fnRd0b2FaJi3YN8DiqxRiJ8h8zJ2fO 5YiL3eSg5GSswPP7D/85KMoHa3vYaRBeIOt7TI/oHazMUlDGUaYCSZgWbFT99CG6ZVZA ja/05KBgHSc9eha8JS7tcJxrYg906AnTdCM3M32TBBLRKXc/dhzD1VEWWt00lwreaVep Rzvg== X-Forwarded-Encrypted: i=1; AHgh+Ro+r+Gnxyk2WWX4zqJ3eCXz9oH7rC4bH/dZrewnUuwrc+uEyQUaHcYEjzgAd5DlwwLH+8J4zTocS6gOcmA=@vger.kernel.org X-Gm-Message-State: AFuF++mFF3xuGxxlWU9aBU77JelCmBsvtNB15wM7Qmib/6FK9dJ4a9RD Vzu9uL2rjiZ1n8xcecuiPa1bAqscePjhEwt8dd6B0aM3ZyAOaqcj50e4S427ZimW2rk= X-Gm-Gg: AR+sD13ZaA/pLzHcjGDeJEPvX/jqU7VAU3bPHVCPOhZjPSH6f5xy3iQumDfvdlcYUw2 JOdESlOR7jUZzyy+e/kWJcdK3i/dgzTNUmQuuVyJFvx8GNP4B/ZGO8/j16a5/abJkqpOfc2n9n2 7Qz+MXJQFeu3uS4vrSUCvVKbFO6hFT/ammnbAEFhFHBy14rK3OSKxMw7vwvzzh9WyevL49wtTpW +cy3csIfaOZQjl+lEZQ5drY6hqV8bowvnZk2VzZCJ9ZHHPMcqQPupjJ345rruEmtLsEXpc4JAF9 Tr9IbfrLHF5TrEgFapKmawAXl6Nj5Gc68+MW4M/k/du3BzsxRGUcmRd81EUhrtT8hFVKyU9MyUh JVkav0DLJv/YSzQwxOnf2ECa5U2FUhPNw8PsgYoMPqt6i4bbEbbucwVKAYVOQJDwwHrlQ0zFPvF ZXr1KZ8kZui4LOo9gAfn3VWKGXhTcLTxvDPBvOrRkyHglXz1UprBxwUvEsFKw5l72zaaJsVNx6 X-Received: by 2002:a17:90b:5284:b0:396:4dfb:5890 with SMTP id 98e67ed59e1d1-396d0e5bbc0mr11833357a91.2.1787907505524; Fri, 28 Aug 2026 01:58:25 -0700 (PDT) Received: from sw04.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id 5a478bee46e88-3286f9ecaa9sm3838814eec.26.2026.08.28.01.58.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 28 Aug 2026 01:58:25 -0700 (PDT) From: Zong Li To: tomasz.jeznach@linux.dev, joro@8bytes.org, will@kernel.org, robin.murphy@arm.com, pjw@kernel.org, palmer@dabbelt.com, aou@eecs.berkeley.edu, alex@ghiti.fr, mark.rutland@arm.com, andrew.jones@oss.qualcomm.com, guoren@kernel.org, david.laight.linux@gmail.com, zhangzhanpeng.jasper@bytedance.com, yang.yicong@picoheart.com, iommu@lists.linux.dev, linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, linux-perf-users@vger.kernel.org Cc: Zong Li Subject: [PATCH RESEND v7 3/3] drivers/perf: riscv-iommu: protect shared state with a raw spinlock Date: Fri, 28 Aug 2026 01:58:17 -0700 Message-ID: <20260828085819.4076449-4-zong.li@sifive.com> X-Mailer: git-send-email @GIT_VERSION@ In-Reply-To: <20260828085819.4076449-1-zong.li@sifive.com> References: <20260828085819.4076449-1-zong.li@sifive.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Events are bound to one CPU and the interrupt is affine to it, so the perf callbacks running with interrupts disabled would be enough to exclude the handler. PCI MSI/MSI-X on IMSIC breaks that: the irqchip sets IRQCHIP_MOVE_DEFERRED, so irq_set_affinity() reports success while only recording the request, and the move is applied in interrupt context upon the next device interrupt. Until then the interrupt is still routed to the CPU IMSIC picked initially, so the first overflow interrupt can run concurrently with the perf callbacks on the CPU the events are bound to. Take a raw spinlock, with interrupts disabled so that the handler can never interrupt a holder on the same CPU, rather than depending on that irqchip behaviour. It covers the state which is reachable from both sides: - IOCOUNTINH is read-modify-written by ->start()/->stop() and is saved and restored around the whole handler. - pmu->events[] is written by ->del() and read by the handler. - hw_perf_event::prev_count is updated by both. ->add() and ->del() call the unlocked __riscv_iommu_pmu_start() and __riscv_iommu_pmu_stop() so the lock is taken once per callback. Signed-off-by: Zong Li --- drivers/perf/riscv_iommu_pmu.c | 66 ++++++++++++++++++++++++++++++---- 1 file changed, 60 insertions(+), 6 deletions(-) diff --git a/drivers/perf/riscv_iommu_pmu.c b/drivers/perf/riscv_iommu_pmu.c index f6acd56f2f61..ee2f6d1fbece 100644 --- a/drivers/perf/riscv_iommu_pmu.c +++ b/drivers/perf/riscv_iommu_pmu.c @@ -101,6 +101,7 @@ struct riscv_iommu_pmu { u64 event_cntr_mask; struct perf_event *events[RISCV_IOMMU_HPM_COUNTER_NUM]; DECLARE_BITMAP(used_counters, RISCV_IOMMU_HPM_COUNTER_NUM); + raw_spinlock_t lock; }; =20 #define to_riscv_iommu_pmu(p) (container_of(p, struct riscv_iommu_pmu, pmu= )) @@ -485,7 +486,8 @@ static void riscv_iommu_pmu_update(struct perf_event *e= vent) local64_add(delta, &event->count); } =20 -static void riscv_iommu_pmu_start(struct perf_event *event, int flags) +/* Called with pmu->lock held */ +static void __riscv_iommu_pmu_start(struct perf_event *event, int flags) { struct riscv_iommu_pmu *pmu =3D to_riscv_iommu_pmu(event->pmu); struct hw_perf_event *hwc =3D &event->hw; @@ -500,11 +502,22 @@ static void riscv_iommu_pmu_start(struct perf_event *= event, int flags) riscv_iommu_pmu_set_period(event); riscv_iommu_pmu_set_event(pmu, hwc->idx, hwc->config); riscv_iommu_pmu_enable_counter(pmu, hwc->idx); +} + +static void riscv_iommu_pmu_start(struct perf_event *event, int flags) +{ + struct riscv_iommu_pmu *pmu =3D to_riscv_iommu_pmu(event->pmu); + unsigned long irqflags; + + raw_spin_lock_irqsave(&pmu->lock, irqflags); + __riscv_iommu_pmu_start(event, flags); + raw_spin_unlock_irqrestore(&pmu->lock, irqflags); =20 perf_event_update_userpage(event); } =20 -static void riscv_iommu_pmu_stop(struct perf_event *event, int flags) +/* Called with pmu->lock held */ +static void __riscv_iommu_pmu_stop(struct perf_event *event, int flags) { struct riscv_iommu_pmu *pmu =3D to_riscv_iommu_pmu(event->pmu); struct hw_perf_event *hwc =3D &event->hw; @@ -521,13 +534,26 @@ static void riscv_iommu_pmu_stop(struct perf_event *e= vent, int flags) hwc->state |=3D PERF_HES_STOPPED | PERF_HES_UPTODATE; } =20 +static void riscv_iommu_pmu_stop(struct perf_event *event, int flags) +{ + struct riscv_iommu_pmu *pmu =3D to_riscv_iommu_pmu(event->pmu); + unsigned long irqflags; + + raw_spin_lock_irqsave(&pmu->lock, irqflags); + __riscv_iommu_pmu_stop(event, flags); + raw_spin_unlock_irqrestore(&pmu->lock, irqflags); +} + static int riscv_iommu_pmu_add(struct perf_event *event, int flags) { struct riscv_iommu_pmu *pmu =3D to_riscv_iommu_pmu(event->pmu); struct hw_perf_event *hwc =3D &event->hw; unsigned int num_counters =3D pmu->num_counters; + unsigned long irqflags; unsigned int idx; =20 + raw_spin_lock_irqsave(&pmu->lock, irqflags); + /* Reserve index zero for iohpmcycles */ if (is_cycle_event(event->attr.config)) idx =3D RISCV_IOMMU_HPM_CYCLE_IDX; @@ -535,8 +561,10 @@ static int riscv_iommu_pmu_add(struct perf_event *even= t, int flags) idx =3D find_next_zero_bit(pmu->used_counters, num_counters, 1); =20 /* All event counters or cycle counter are in use */ - if (idx =3D=3D num_counters || pmu->events[idx]) + if (idx =3D=3D num_counters || pmu->events[idx]) { + raw_spin_unlock_irqrestore(&pmu->lock, irqflags); return -EAGAIN; + } =20 set_bit(idx, pmu->used_counters); =20 @@ -546,7 +574,9 @@ static int riscv_iommu_pmu_add(struct perf_event *event= , int flags) local64_set(&hwc->prev_count, 0); =20 if (flags & PERF_EF_START) - riscv_iommu_pmu_start(event, flags); + __riscv_iommu_pmu_start(event, flags); + + raw_spin_unlock_irqrestore(&pmu->lock, irqflags); =20 /* Propagate changes to the userspace mapping. */ perf_event_update_userpage(event); @@ -556,18 +586,26 @@ static int riscv_iommu_pmu_add(struct perf_event *eve= nt, int flags) =20 static void riscv_iommu_pmu_read(struct perf_event *event) { + struct riscv_iommu_pmu *pmu =3D to_riscv_iommu_pmu(event->pmu); + unsigned long irqflags; + + raw_spin_lock_irqsave(&pmu->lock, irqflags); riscv_iommu_pmu_update(event); + raw_spin_unlock_irqrestore(&pmu->lock, irqflags); } =20 static void riscv_iommu_pmu_del(struct perf_event *event, int flags) { struct riscv_iommu_pmu *pmu =3D to_riscv_iommu_pmu(event->pmu); struct hw_perf_event *hwc =3D &event->hw; + unsigned long irqflags; int idx =3D hwc->idx; =20 - riscv_iommu_pmu_stop(event, PERF_EF_UPDATE); + raw_spin_lock_irqsave(&pmu->lock, irqflags); + __riscv_iommu_pmu_stop(event, PERF_EF_UPDATE); pmu->events[idx] =3D NULL; clear_bit(idx, pmu->used_counters); + raw_spin_unlock_irqrestore(&pmu->lock, irqflags); =20 perf_event_update_userpage(event); } @@ -635,12 +673,24 @@ static irqreturn_t riscv_iommu_pmu_irq_handler(int ir= q, void *dev_id) { struct riscv_iommu_pmu *pmu =3D (struct riscv_iommu_pmu *)dev_id; DECLARE_BITMAP(ovf_bitmap, BITS_PER_TYPE(u64)); + unsigned long irqflags; u32 ovf, idx, inhibit; =20 - /* Check whether this interrupt is for PMU */ + /* + * Check whether this interrupt is for PMU. Done outside the lock so + * that a shared interrupt line is left alone as cheaply as possible. + */ if (!(readl_relaxed(pmu->reg + RISCV_IOMMU_REG_IPSR) & RISCV_IOMMU_IPSR_P= MIP)) return IRQ_NONE; =20 + /* + * Hold the lock across the whole sequence below. Stopping the + * counters, processing them and restoring the previous inhibit state + * has to be atomic against ->start()/->stop(), otherwise a counter + * enabled in between would be inhibited again by the restore. + */ + raw_spin_lock_irqsave(&pmu->lock, irqflags); + /* Process PMU IRQ */ inhibit =3D riscv_iommu_pmu_stop_all(pmu); =20 @@ -672,6 +722,8 @@ static irqreturn_t riscv_iommu_pmu_irq_handler(int irq,= void *dev_id) =20 riscv_iommu_pmu_start_all(pmu, inhibit); =20 + raw_spin_unlock_irqrestore(&pmu->lock, irqflags); + return IRQ_HANDLED; } =20 @@ -735,6 +787,8 @@ static int riscv_iommu_pmu_probe(struct auxiliary_devic= e *auxdev, =20 iommu_pmu->reg =3D iommu_dev->reg; =20 + raw_spin_lock_init(&iommu_pmu->lock); + /* * Counter number and width are hardware-implemented, detect them by * writing 1s and reading back which bits stuck. --=20 2.43.7