From nobody Sun Dec 14 23:06:10 2025 Received: from mail-pl1-f169.google.com (mail-pl1-f169.google.com [209.85.214.169]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7BF8022087 for ; Wed, 15 Jan 2025 03:03:27 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.169 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1736910210; cv=none; b=ocSKSa3dSQLls+g8jhK1m4vNdhxcLX6OWWvefciAb7qBVFb8Vz2lypzdUiyPGb7ztlKQatZ4V/Zs29VTFyDyVhMwrMML7eLwm9HLz+Uh1BtmP/C8KHGOIs1szZpctvwieJR6uwDm2VJzLSo/RXSCAmIFlPe0WoEeZl3V9H+6cK0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1736910210; c=relaxed/simple; bh=Y70Uq4YaFnQ16wnRUgsbmFS5coyRvYPYv7NDvtKp80I=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References; b=JooT5xjO6WE/BQY3/AKd2rHEFPaDMn+JXyts5CdzMK0cw2Syvq2GKm6BUHnbhnaLIpwjS9Ajpi18yntIhVphOOAlVG6uZiMcsTWlDTcD1fgvFVTOoY+HJ48RU2fCeuB3W+4QFvpzD7LSSQjLLkWCbUk9rGVfEoaBAsMkcN1UgvE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=sifive.com; spf=pass smtp.mailfrom=sifive.com; dkim=pass (2048-bit key) header.d=sifive.com header.i=@sifive.com header.b=nfPo04wc; arc=none smtp.client-ip=209.85.214.169 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=sifive.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=sifive.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=sifive.com header.i=@sifive.com header.b="nfPo04wc" Received: by mail-pl1-f169.google.com with SMTP id d9443c01a7336-21a7ed0155cso107324085ad.3 for ; Tue, 14 Jan 2025 19:03:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1736910206; x=1737515006; darn=vger.kernel.org; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=W0GVuACSZ4nNU1jOcBib3HparAzNSm90YaA55aCx84U=; b=nfPo04wcBIJh/IhEySLppukL3SYsvcSx0LSE3TTOnSrGa5QyzMBi7L2WkZnL5CgD9T CnJnrJGhGdFudlzTS9a7DvtFxC4puqYbTg6gw/wg+MCeyhKiIzKq6m+VXWYackwFGrFC Vvenzl5ORwgcTQ0iWrdafb3GB54q40KjMa6p5jsFAc9Ot1TXge5qrR1e/3TFWl+WK65c DxcfX/EOOR2BxtK22weHUxRHXf2xcp04zJs5htZwgGE2Ys9zyurEiLie8YJCj4DEu5/W Qzsmnb5PCQ/rV7JenaULWrkkKUZX1rNHPUvfItqsLhBjxhDYBN4JmZcNgP2TpaxqCie0 VvgQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736910206; x=1737515006; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=W0GVuACSZ4nNU1jOcBib3HparAzNSm90YaA55aCx84U=; b=sWLoMVgSiylSsugHKD9489J3+2zggvGMXFvpCb7P7gfRA3JWCX924Tu+DRkgO0bRrH Hn0QcdBs6aEIlvM9ZWPMtghIOBZjXLrytZBtay50XyG6TeaMVAkn9nwCLsl93SuuayjF Br1GBvUjoNYHujWnRunl7ntFmHBagMYnpersjgPj56kSg7fDpZNap7GG/F97QrfQnwEN b+deCC9/CQWc/Zz798ggcFvu5xw9x6wOPjQsLgRI0Hc/v4nF64sRR2dzTI/uEgbeT2jO Y4NL987C6qFzN1CjZ0cgf0iFJufvYkl4QNCMvyRMSz1S2KFJoy1anhpW33gTvylxYa59 vO5w== X-Forwarded-Encrypted: i=1; AJvYcCXxR+F/JqFMRlPCOLlyXgeaDairRVZ+AfeEFqj1GwtINibpZKXPSzmZWRCJhO7SaL/ZTxchkXIaejf0SXY=@vger.kernel.org X-Gm-Message-State: AOJu0YxxO0uW3Ve7Q1fJHEdZjMbhBYWjVDGfUNoDookEn/V0wXeDihlh RTA8ZpbgOKhcdY8EJ5oSFd9ovzccRhv5Gb1K1srRinV4cXkCiNyZfLDz0AzrHXc= X-Gm-Gg: ASbGncuC0y8tBrPG+QN19qdFzvUyjYAb3vLY5Y3oFQtAhJOt4TLOKsAf0iruCcL6wzY k3NW46uSFMUrumRupjcE/GUoaz7NQCBg1ZPR2wIhC1VNCbg3C+j9bVJg4JEVJFVdV9faWOy6Nsn GaEqiSL/SHzfWE3IRp8sjXwtEoPS8vXGJFbIktRGPHBvnVNo/WP1j9EVTLob7Ef0I+8Num5ZkPL 9f3Yb8IRL/0h93JcRa2bYbQgL2S1IvzktmSGkU2QYHHPENpVgD+52odDx7RhlwBz6cXL7vUyi3T DSQ= X-Google-Smtp-Source: AGHT+IFP+WuGymi19PJE7TnqwQafCTvoBmiNwn1Tfo9q5pdCPh3ZYRhf84iwGoAqVzg5FJAoLA9lMA== X-Received: by 2002:a05:6a00:2792:b0:725:b4f7:378e with SMTP id d2e1a72fcca58-72d21e0d4cemr40456055b3a.0.1736910206556; Tue, 14 Jan 2025 19:03:26 -0800 (PST) Received: from hsinchu26.internal.sifive.com ([210.176.154.34]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-72d40568dccsm8123668b3a.47.2025.01.14.19.03.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 14 Jan 2025 19:03:26 -0800 (PST) From: Zong Li To: joro@8bytes.org, will@kernel.org, robin.murphy@arm.com, tjeznach@rivosinc.com, paul.walmsley@sifive.com, palmer@dabbelt.com, aou@eecs.berkeley.edu, luxu.kernel@bytedance.com, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-riscv@lists.infradead.org Cc: Zong Li Subject: [PATCH 1/2] iommu/riscv: add RISC-V IOMMU PMU support Date: Wed, 15 Jan 2025 11:03:05 +0800 Message-Id: <20250115030306.29735-2-zong.li@sifive.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20250115030306.29735-1-zong.li@sifive.com> References: <20250115030306.29735-1-zong.li@sifive.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Support for the RISC-V IOMMU hardware performance monitor includes both counting and sampling modes. The specification does not define an event ID for counting the number of clock cycles, meaning there is no associated `iohpmevt0`. However, we need an event for counting cycle, so we reserve the maximum event ID for this purpose. Signed-off-by: Zong Li Tested-by: Xu Lu --- drivers/iommu/riscv/Makefile | 2 +- drivers/iommu/riscv/iommu-bits.h | 16 + drivers/iommu/riscv/iommu-pmu.c | 486 +++++++++++++++++++++++++++++++ drivers/iommu/riscv/iommu.h | 8 + 4 files changed, 511 insertions(+), 1 deletion(-) create mode 100644 drivers/iommu/riscv/iommu-pmu.c diff --git a/drivers/iommu/riscv/Makefile b/drivers/iommu/riscv/Makefile index f54c9ed17d41..d36625a1fd08 100644 --- a/drivers/iommu/riscv/Makefile +++ b/drivers/iommu/riscv/Makefile @@ -1,3 +1,3 @@ # SPDX-License-Identifier: GPL-2.0-only -obj-$(CONFIG_RISCV_IOMMU) +=3D iommu.o iommu-platform.o +obj-$(CONFIG_RISCV_IOMMU) +=3D iommu.o iommu-platform.o iommu-pmu.o obj-$(CONFIG_RISCV_IOMMU_PCI) +=3D iommu-pci.o diff --git a/drivers/iommu/riscv/iommu-bits.h b/drivers/iommu/riscv/iommu-b= its.h index 98daf0e1a306..60523449f016 100644 --- a/drivers/iommu/riscv/iommu-bits.h +++ b/drivers/iommu/riscv/iommu-bits.h @@ -17,6 +17,7 @@ #include #include #include +#include =20 /* * Chapter 5: Memory Mapped register interface @@ -207,6 +208,7 @@ enum riscv_iommu_ddtp_modes { /* 5.22 Performance monitoring event counters (31 * 64bits) */ #define RISCV_IOMMU_REG_IOHPMCTR_BASE 0x0068 #define RISCV_IOMMU_REG_IOHPMCTR(_n) (RISCV_IOMMU_REG_IOHPMCTR_BASE + ((_n= ) * 0x8)) +#define RISCV_IOMMU_IOHPMCTR_COUNTER GENMASK_ULL(63, 0) =20 /* 5.23 Performance monitoring event selectors (31 * 64bits) */ #define RISCV_IOMMU_REG_IOHPMEVT_BASE 0x0160 @@ -250,6 +252,20 @@ enum riscv_iommu_hpmevent_id { RISCV_IOMMU_HPMEVENT_MAX =3D 9 }; =20 +/* Use maximum event ID for cycle event */ +#define RISCV_IOMMU_HPMEVENT_CYCLE GENMASK_ULL(14, 0) + +#define RISCV_IOMMU_HPM_COUNTER_NUM 32 + +struct riscv_iommu_pmu { + struct pmu pmu; + void __iomem *reg; + int num_counters; + u64 mask_counter; + struct perf_event *events[RISCV_IOMMU_IOHPMEVT_CNT + 1]; + DECLARE_BITMAP(used_counters, RISCV_IOMMU_IOHPMEVT_CNT + 1); +}; + /* 5.24 Translation request IOVA (64bits) */ #define RISCV_IOMMU_REG_TR_REQ_IOVA 0x0258 #define RISCV_IOMMU_TR_REQ_IOVA_VPN GENMASK_ULL(63, 12) diff --git a/drivers/iommu/riscv/iommu-pmu.c b/drivers/iommu/riscv/iommu-pm= u.c new file mode 100644 index 000000000000..74eb1525cd32 --- /dev/null +++ b/drivers/iommu/riscv/iommu-pmu.c @@ -0,0 +1,486 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (C) 2024 SiFive + * + * Authors + * Zong Li + */ + +#include + +#include "iommu.h" +#include "iommu-bits.h" + +#define to_riscv_iommu_pmu(p) (container_of(p, struct riscv_iommu_pmu, pmu= )) + +#define RISCV_IOMMU_PMU_ATTR_EXTRACTOR(_name, _mask) \ + static inline u32 get_##_name(struct perf_event *event) \ + { \ + return FIELD_GET(_mask, event->attr.config); \ + } \ + +RISCV_IOMMU_PMU_ATTR_EXTRACTOR(event, RISCV_IOMMU_IOHPMEVT_EVENTID); +RISCV_IOMMU_PMU_ATTR_EXTRACTOR(partial_matching, RISCV_IOMMU_IOHPMEVT_DMAS= K); +RISCV_IOMMU_PMU_ATTR_EXTRACTOR(pid_pscid, RISCV_IOMMU_IOHPMEVT_PID_PSCID); +RISCV_IOMMU_PMU_ATTR_EXTRACTOR(did_gscid, RISCV_IOMMU_IOHPMEVT_DID_GSCID); +RISCV_IOMMU_PMU_ATTR_EXTRACTOR(filter_pid_pscid, RISCV_IOMMU_IOHPMEVT_PV_P= SCV); +RISCV_IOMMU_PMU_ATTR_EXTRACTOR(filter_did_gscid, RISCV_IOMMU_IOHPMEVT_DV_G= SCV); +RISCV_IOMMU_PMU_ATTR_EXTRACTOR(filter_id_type, RISCV_IOMMU_IOHPMEVT_IDT); + +/* Formats */ +PMU_FORMAT_ATTR(event, "config:0-14"); +PMU_FORMAT_ATTR(partial_matching, "config:15"); +PMU_FORMAT_ATTR(pid_pscid, "config:16-35"); +PMU_FORMAT_ATTR(did_gscid, "config:36-59"); +PMU_FORMAT_ATTR(filter_pid_pscid, "config:60"); +PMU_FORMAT_ATTR(filter_did_gscid, "config:61"); +PMU_FORMAT_ATTR(filter_id_type, "config:62"); + +static struct attribute *riscv_iommu_pmu_formats[] =3D { + &format_attr_event.attr, + &format_attr_partial_matching.attr, + &format_attr_pid_pscid.attr, + &format_attr_did_gscid.attr, + &format_attr_filter_pid_pscid.attr, + &format_attr_filter_did_gscid.attr, + &format_attr_filter_id_type.attr, + NULL, +}; + +static const struct attribute_group riscv_iommu_pmu_format_group =3D { + .name =3D "format", + .attrs =3D riscv_iommu_pmu_formats, +}; + +/* Events */ +static ssize_t riscv_iommu_pmu_event_show(struct device *dev, + struct device_attribute *attr, + char *page) +{ + struct perf_pmu_events_attr *pmu_attr; + + pmu_attr =3D container_of(attr, struct perf_pmu_events_attr, attr); + + return sprintf(page, "event=3D0x%02llx\n", pmu_attr->id); +} + +PMU_EVENT_ATTR(cycle, event_attr_cycle, + RISCV_IOMMU_HPMEVENT_CYCLE, riscv_iommu_pmu_event_show); +PMU_EVENT_ATTR(dont_count, event_attr_dont_count, + RISCV_IOMMU_HPMEVENT_INVALID, riscv_iommu_pmu_event_show); +PMU_EVENT_ATTR(untranslated_req, event_attr_untranslated_req, + RISCV_IOMMU_HPMEVENT_URQ, riscv_iommu_pmu_event_show); +PMU_EVENT_ATTR(translated_req, event_attr_translated_req, + RISCV_IOMMU_HPMEVENT_TRQ, riscv_iommu_pmu_event_show); +PMU_EVENT_ATTR(ats_trans_req, event_attr_ats_trans_req, + RISCV_IOMMU_HPMEVENT_ATS_RQ, riscv_iommu_pmu_event_show); +PMU_EVENT_ATTR(tlb_miss, event_attr_tlb_miss, + RISCV_IOMMU_HPMEVENT_TLB_MISS, riscv_iommu_pmu_event_show); +PMU_EVENT_ATTR(ddt_walks, event_attr_ddt_walks, + RISCV_IOMMU_HPMEVENT_DD_WALK, riscv_iommu_pmu_event_show); +PMU_EVENT_ATTR(pdt_walks, event_attr_pdt_walks, + RISCV_IOMMU_HPMEVENT_PD_WALK, riscv_iommu_pmu_event_show); +PMU_EVENT_ATTR(s_vs_pt_walks, event_attr_s_vs_pt_walks, + RISCV_IOMMU_HPMEVENT_S_VS_WALKS, riscv_iommu_pmu_event_show); +PMU_EVENT_ATTR(g_pt_walks, event_attr_g_pt_walks, + RISCV_IOMMU_HPMEVENT_G_WALKS, riscv_iommu_pmu_event_show); + +static struct attribute *riscv_iommu_pmu_events[] =3D { + &event_attr_cycle.attr.attr, + &event_attr_dont_count.attr.attr, + &event_attr_untranslated_req.attr.attr, + &event_attr_translated_req.attr.attr, + &event_attr_ats_trans_req.attr.attr, + &event_attr_tlb_miss.attr.attr, + &event_attr_ddt_walks.attr.attr, + &event_attr_pdt_walks.attr.attr, + &event_attr_s_vs_pt_walks.attr.attr, + &event_attr_g_pt_walks.attr.attr, + NULL, +}; + +static const struct attribute_group riscv_iommu_pmu_events_group =3D { + .name =3D "events", + .attrs =3D riscv_iommu_pmu_events, +}; + +static const struct attribute_group *riscv_iommu_pmu_attr_grps[] =3D { + &riscv_iommu_pmu_format_group, + &riscv_iommu_pmu_events_group, + NULL, +}; + +/* PMU Operations */ +static void riscv_iommu_pmu_set_counter(struct riscv_iommu_pmu *pmu, u32 i= dx, + u64 value) +{ + void __iomem *addr =3D pmu->reg + RISCV_IOMMU_REG_IOHPMCYCLES; + + if (WARN_ON_ONCE(idx < 0 || idx > pmu->num_counters)) + return; + + if (idx =3D=3D 0) + value =3D (value & ~RISCV_IOMMU_IOHPMCYCLES_OF) | + (readq(addr) & RISCV_IOMMU_IOHPMCYCLES_OF); + + writeq(FIELD_PREP(RISCV_IOMMU_IOHPMCTR_COUNTER, value), addr + idx * 8); +} + +static u64 riscv_iommu_pmu_get_counter(struct riscv_iommu_pmu *pmu, u32 id= x) +{ + void __iomem *addr =3D pmu->reg + RISCV_IOMMU_REG_IOHPMCYCLES; + u64 value; + + if (WARN_ON_ONCE(idx < 0 || idx > pmu->num_counters)) + return -EINVAL; + + value =3D readq(addr + idx * 8); + + if (idx =3D=3D 0) + return FIELD_GET(RISCV_IOMMU_IOHPMCYCLES_COUNTER, value); + + return FIELD_GET(RISCV_IOMMU_IOHPMCTR_COUNTER, value); +} + +static u64 riscv_iommu_pmu_get_event(struct riscv_iommu_pmu *pmu, u32 idx) +{ + void __iomem *addr =3D pmu->reg + RISCV_IOMMU_REG_IOHPMEVT_BASE; + + if (WARN_ON_ONCE(idx < 0 || idx > pmu->num_counters)) + return 0; + + /* There is no associtated IOHPMEVT0 for IOHPMCYCLES */ + if (idx =3D=3D 0) + return 0; + + return readq(addr + (idx - 1) * 8); +} + +static void riscv_iommu_pmu_set_event(struct riscv_iommu_pmu *pmu, u32 idx, + u64 value) +{ + void __iomem *addr =3D pmu->reg + RISCV_IOMMU_REG_IOHPMEVT_BASE; + + if (WARN_ON_ONCE(idx < 0 || idx > pmu->num_counters)) + return; + + /* There is no associtated IOHPMEVT0 for IOHPMCYCLES */ + if (idx =3D=3D 0) + return; + + writeq(value, addr + (idx - 1) * 8); +} + +static void riscv_iommu_pmu_enable_counter(struct riscv_iommu_pmu *pmu, u3= 2 idx) +{ + void __iomem *addr =3D pmu->reg + RISCV_IOMMU_REG_IOCOUNTINH; + u32 value =3D readl(addr); + + writel(value & ~BIT(idx), addr); +} + +static void riscv_iommu_pmu_disable_counter(struct riscv_iommu_pmu *pmu, u= 32 idx) +{ + void __iomem *addr =3D pmu->reg + RISCV_IOMMU_REG_IOCOUNTINH; + u32 value =3D readl(addr); + + writel(value | BIT(idx), addr); +} + +static void riscv_iommu_pmu_enable_ovf_intr(struct riscv_iommu_pmu *pmu, u= 32 idx) +{ + u64 value; + + if (get_event(pmu->events[idx]) =3D=3D RISCV_IOMMU_HPMEVENT_CYCLE) { + value =3D riscv_iommu_pmu_get_counter(pmu, idx) & ~RISCV_IOMMU_IOHPMCYCL= ES_OF; + writeq(value, pmu->reg + RISCV_IOMMU_REG_IOHPMCYCLES); + } else { + value =3D riscv_iommu_pmu_get_event(pmu, idx) & ~RISCV_IOMMU_IOHPMEVT_OF; + writeq(value, pmu->reg + RISCV_IOMMU_REG_IOHPMEVT_BASE + (idx - 1) * 8); + } +} + +static void riscv_iommu_pmu_disable_ovf_intr(struct riscv_iommu_pmu *pmu, = u32 idx) +{ + u64 value; + + if (get_event(pmu->events[idx]) =3D=3D RISCV_IOMMU_HPMEVENT_CYCLE) { + value =3D riscv_iommu_pmu_get_counter(pmu, idx) | RISCV_IOMMU_IOHPMCYCLE= S_OF; + writeq(value, pmu->reg + RISCV_IOMMU_REG_IOHPMCYCLES); + } else { + value =3D riscv_iommu_pmu_get_event(pmu, idx) | RISCV_IOMMU_IOHPMEVT_OF; + writeq(value, pmu->reg + RISCV_IOMMU_REG_IOHPMEVT_BASE + (idx - 1) * 8); + } +} + +static void riscv_iommu_pmu_start_all(struct riscv_iommu_pmu *pmu) +{ + int idx; + + for_each_set_bit(idx, pmu->used_counters, pmu->num_counters) { + riscv_iommu_pmu_enable_ovf_intr(pmu, idx); + riscv_iommu_pmu_enable_counter(pmu, idx); + } +} + +static void riscv_iommu_pmu_stop_all(struct riscv_iommu_pmu *pmu) +{ + writel(GENMASK_ULL(pmu->num_counters - 1, 0), + pmu->reg + RISCV_IOMMU_REG_IOCOUNTINH); +} + +/* PMU APIs */ +static int riscv_iommu_pmu_set_period(struct perf_event *event) +{ + struct riscv_iommu_pmu *pmu =3D to_riscv_iommu_pmu(event->pmu); + struct hw_perf_event *hwc =3D &event->hw; + s64 left =3D local64_read(&hwc->period_left); + s64 period =3D hwc->sample_period; + u64 max_period =3D pmu->mask_counter; + int ret =3D 0; + + if (unlikely(left <=3D -period)) { + left =3D period; + local64_set(&hwc->period_left, left); + hwc->last_period =3D period; + ret =3D 1; + } + + if (unlikely(left <=3D 0)) { + left +=3D period; + local64_set(&hwc->period_left, left); + hwc->last_period =3D period; + ret =3D 1; + } + + /* + * Limit the maximum period to prevent the counter value + * from overtaking the one we are about to program. In + * effect we are reducing max_period to account for + * interrupt latency (and we are being very conservative). + */ + if (left > (max_period >> 1)) + left =3D (max_period >> 1); + + local64_set(&hwc->prev_count, (u64)-left); + riscv_iommu_pmu_set_counter(pmu, hwc->idx, (u64)(-left) & max_period); + perf_event_update_userpage(event); + + return ret; +} + +static int riscv_iommu_pmu_event_init(struct perf_event *event) +{ + struct riscv_iommu_pmu *pmu =3D to_riscv_iommu_pmu(event->pmu); + struct hw_perf_event *hwc =3D &event->hw; + + hwc->idx =3D -1; + hwc->config =3D event->attr.config; + + if (!is_sampling_event(event)) { + /* + * For non-sampling runs, limit the sample_period to half + * of the counter width. That way, the new counter value + * is far less likely to overtake the previous one unless + * you have some serious IRQ latency issues. + */ + hwc->sample_period =3D pmu->mask_counter >> 1; + hwc->last_period =3D hwc->sample_period; + local64_set(&hwc->period_left, hwc->sample_period); + } + + return 0; +} + +static void riscv_iommu_pmu_update(struct perf_event *event) +{ + struct hw_perf_event *hwc =3D &event->hw; + struct riscv_iommu_pmu *pmu =3D to_riscv_iommu_pmu(event->pmu); + u64 delta, prev, now; + u32 idx =3D hwc->idx; + + do { + prev =3D local64_read(&hwc->prev_count); + now =3D riscv_iommu_pmu_get_counter(pmu, idx); + } while (local64_cmpxchg(&hwc->prev_count, prev, now) !=3D prev); + + delta =3D FIELD_GET(RISCV_IOMMU_IOHPMCTR_COUNTER, now - prev) & pmu->mask= _counter; + local64_add(delta, &event->count); + local64_sub(delta, &hwc->period_left); +} + +static void riscv_iommu_pmu_start(struct perf_event *event, int flags) +{ + struct riscv_iommu_pmu *pmu =3D to_riscv_iommu_pmu(event->pmu); + struct hw_perf_event *hwc =3D &event->hw; + + if (WARN_ON_ONCE(!(event->hw.state & PERF_HES_STOPPED))) + return; + + if (flags & PERF_EF_RELOAD) + WARN_ON_ONCE(!(event->hw.state & PERF_HES_UPTODATE)); + + hwc->state =3D 0; + riscv_iommu_pmu_set_period(event); + riscv_iommu_pmu_set_event(pmu, hwc->idx, hwc->config); + riscv_iommu_pmu_enable_ovf_intr(pmu, hwc->idx); + riscv_iommu_pmu_enable_counter(pmu, hwc->idx); + + perf_event_update_userpage(event); +} + +static void riscv_iommu_pmu_stop(struct perf_event *event, int flags) +{ + struct riscv_iommu_pmu *pmu =3D to_riscv_iommu_pmu(event->pmu); + struct hw_perf_event *hwc =3D &event->hw; + + if (hwc->state & PERF_HES_STOPPED) + return; + + riscv_iommu_pmu_set_event(pmu, hwc->idx, RISCV_IOMMU_HPMEVENT_INVALID); + riscv_iommu_pmu_disable_counter(pmu, hwc->idx); + + if ((flags & PERF_EF_UPDATE) && !(hwc->state & PERF_HES_UPTODATE)) + riscv_iommu_pmu_update(event); + + hwc->state |=3D PERF_HES_STOPPED | PERF_HES_UPTODATE; +} + +static int riscv_iommu_pmu_add(struct perf_event *event, int flags) +{ + struct hw_perf_event *hwc =3D &event->hw; + struct riscv_iommu_pmu *pmu =3D to_riscv_iommu_pmu(event->pmu); + unsigned int num_counters =3D pmu->num_counters; + int idx; + + /* Reserve index zero for iohpmcycles */ + if (get_event(event) =3D=3D RISCV_IOMMU_HPMEVENT_CYCLE) + idx =3D 0; + else + idx =3D find_next_zero_bit(pmu->used_counters, num_counters, 1); + + if (idx =3D=3D num_counters) + return -EAGAIN; + + set_bit(idx, pmu->used_counters); + + pmu->events[idx] =3D event; + hwc->idx =3D idx; + hwc->state =3D PERF_HES_STOPPED | PERF_HES_UPTODATE; + + if (flags & PERF_EF_START) + riscv_iommu_pmu_start(event, flags); + + /* Propagate changes to the userspace mapping. */ + perf_event_update_userpage(event); + + return 0; +} + +static void riscv_iommu_pmu_read(struct perf_event *event) +{ + riscv_iommu_pmu_update(event); +} + +static void riscv_iommu_pmu_del(struct perf_event *event, int flags) +{ + struct hw_perf_event *hwc =3D &event->hw; + struct riscv_iommu_pmu *pmu =3D to_riscv_iommu_pmu(event->pmu); + int idx =3D hwc->idx; + + riscv_iommu_pmu_stop(event, PERF_EF_UPDATE); + pmu->events[idx] =3D NULL; + clear_bit(idx, pmu->used_counters); + perf_event_update_userpage(event); +} + +irqreturn_t riscv_iommu_pmu_handle_irq(struct riscv_iommu_pmu *pmu) +{ + struct perf_sample_data data; + struct pt_regs *regs; + u32 ovf =3D readl(pmu->reg + RISCV_IOMMU_REG_IOCOUNTOVF); + int idx; + + if (!ovf) + return IRQ_NONE; + + riscv_iommu_pmu_stop_all(pmu); + + regs =3D get_irq_regs(); + + for_each_set_bit(idx, (unsigned long *)&ovf, pmu->num_counters) { + struct perf_event *event =3D pmu->events[idx]; + struct hw_perf_event *hwc; + + if (WARN_ON_ONCE(!event) || !is_sampling_event(event)) + continue; + + hwc =3D &event->hw; + + riscv_iommu_pmu_update(event); + perf_sample_data_init(&data, 0, hwc->last_period); + if (!riscv_iommu_pmu_set_period(event)) + continue; + + if (perf_event_overflow(event, &data, regs)) + riscv_iommu_pmu_stop(event, 0); + } + + riscv_iommu_pmu_start_all(pmu); + + return IRQ_HANDLED; +} + +int riscv_iommu_pmu_init(struct riscv_iommu_pmu *pmu, void __iomem *reg, + const char *dev_name) +{ + char *name; + int ret; + + pmu->reg =3D reg; + pmu->num_counters =3D RISCV_IOMMU_HPM_COUNTER_NUM; + pmu->mask_counter =3D RISCV_IOMMU_IOHPMCTR_COUNTER; + + pmu->pmu =3D (struct pmu) { + .task_ctx_nr =3D perf_invalid_context, + .event_init =3D riscv_iommu_pmu_event_init, + .add =3D riscv_iommu_pmu_add, + .del =3D riscv_iommu_pmu_del, + .start =3D riscv_iommu_pmu_start, + .stop =3D riscv_iommu_pmu_stop, + .read =3D riscv_iommu_pmu_read, + .attr_groups =3D riscv_iommu_pmu_attr_grps, + .capabilities =3D PERF_PMU_CAP_NO_EXCLUDE, + .module =3D THIS_MODULE, + }; + + name =3D kasprintf(GFP_KERNEL, "riscv_iommu_pmu_%s", dev_name); + + ret =3D perf_pmu_register(&pmu->pmu, name, -1); + if (ret) { + pr_err("Failed to register riscv_iommu_pmu_%s: %d\n", + dev_name, ret); + return ret; + } + + /* Stop all counters and later start the counter with perf */ + riscv_iommu_pmu_stop_all(pmu); + + pr_info("riscv_iommu_pmu_%s: Registered with %d counters\n", + dev_name, pmu->num_counters); + + return 0; +} + +void riscv_iommu_pmu_uninit(struct riscv_iommu_pmu *pmu) +{ + int idx; + + /* Disable interrupt and functions */ + for_each_set_bit(idx, pmu->used_counters, pmu->num_counters) { + riscv_iommu_pmu_disable_counter(pmu, idx); + riscv_iommu_pmu_disable_ovf_intr(pmu, idx); + } + + perf_pmu_unregister(&pmu->pmu); +} diff --git a/drivers/iommu/riscv/iommu.h b/drivers/iommu/riscv/iommu.h index b1c4664542b4..92659a8a75ae 100644 --- a/drivers/iommu/riscv/iommu.h +++ b/drivers/iommu/riscv/iommu.h @@ -60,11 +60,19 @@ struct riscv_iommu_device { unsigned int ddt_mode; dma_addr_t ddt_phys; u64 *ddt_root; + + /* hardware performance monitor */ + struct riscv_iommu_pmu pmu; }; =20 int riscv_iommu_init(struct riscv_iommu_device *iommu); void riscv_iommu_remove(struct riscv_iommu_device *iommu); =20 +int riscv_iommu_pmu_init(struct riscv_iommu_pmu *pmu, void __iomem *reg, + const char *name); +void riscv_iommu_pmu_uninit(struct riscv_iommu_pmu *pmu); +irqreturn_t riscv_iommu_pmu_handle_irq(struct riscv_iommu_pmu *pmu); + #define riscv_iommu_readl(iommu, addr) \ readl_relaxed((iommu)->reg + (addr)) =20 --=20 2.17.1 From nobody Sun Dec 14 23:06:10 2025 Received: from mail-pl1-f174.google.com (mail-pl1-f174.google.com [209.85.214.174]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 91F2522DFB1 for ; Wed, 15 Jan 2025 03:03:33 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.174 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1736910215; cv=none; b=LTnjkgXOxuNseD/BCnEHije8m8uJbl2dHZiE5dcds8ob1xlljqSMt7hjcrmy0HciXC9F31LDQzxcYEupGFZapIeJLJxJH7LnACJgVpE6L3D8dinMQNsK5opdg0fz6wEVtWli3PekeVndueWF68GsD5+fDR4BPEkdCs1VbjmGGyU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1736910215; c=relaxed/simple; bh=pKHvVYHDRn0ecGOYQL39aWwE9MzaJjTut+z7liM0ME0=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References; b=HnXOS6W6iVJMLnZKsCyYjJ/oAzykqC9M83cjt/lRdQOilaiBgBTF4zohq8SwgwTZtCuM25+CYHcQYZyZUTiOqslMyVl26TKO95hJFN1u9aEvLrEk7z54XzczpzSmCX34d1gLrIBkbx4lgCfPd+nf/1wtLPGqtWphK3ADYM4ClZo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=sifive.com; spf=pass smtp.mailfrom=sifive.com; dkim=pass (2048-bit key) header.d=sifive.com header.i=@sifive.com header.b=hvNixIHb; arc=none smtp.client-ip=209.85.214.174 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=sifive.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=sifive.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=sifive.com header.i=@sifive.com header.b="hvNixIHb" Received: by mail-pl1-f174.google.com with SMTP id d9443c01a7336-2166f1e589cso133388785ad.3 for ; Tue, 14 Jan 2025 19:03:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1736910213; x=1737515013; darn=vger.kernel.org; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=C2l3cW3A35MwF8MmccDu9nYzUOGU3WhdqnIlz5R1nkM=; b=hvNixIHbNWlzXmrgf6re86MKnV4Jd5jg+K9U63hEfPWvpY4grc/hYcZlc6qELs4Vvp azWg1Jdr8NB0BvGDydmsT3ilRfF8yYhA9sFI+mXVRUmiCSaT4r6tBeGRcPIWc68yssMH E4qgfQ9deAjyX5kxXAcbvrBxrRRgyjlkJBtK8MNILvGY3gOyKBFLfCzdsxjkqw6IeFOy oM1BIxkoebIO9qh4KihQq6tcSfTIWxY/HVrQZoj0iSXUzN1OxnRLFjTqTUz5JXco3RKE oZwPeGLM2F5E/8cvftoHzCkwxQxEEpibhrn1iJxKXIDG6KQBWFOHPc/zW7Qqss+q3hDC 54/Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736910213; x=1737515013; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=C2l3cW3A35MwF8MmccDu9nYzUOGU3WhdqnIlz5R1nkM=; b=huMqrsLty6Rh35ehq0ycQ4uOfVoEkcIBw+RcklD25Efdyo1Ha6eNUEihjeMxEUHYEg TJvDeO5qmrKC+9Pm8fS4rENkfBo14rorZ4ZhO+r698rJa8nPo9vW7jlmIORBqklCSsJz EgttzTmCSznOqJygM6TarcfqBjv0K32ZCjs9f1ljBY0+zWXJU4ZrZCrH2SRzknxXAyJe 7+C0qMeJeIljxo6LT0LwZt2r8f6iHNGVRO7q7zOPmLbt6pu6WnLnqdFv7z6/ipOuOS+U REKtd90EFG/ItNYLzomJLaIbn0CSrE1BWt0ao2sOsRI2ximndP/ZnycUm40yAl6Yht+k 5epQ== X-Forwarded-Encrypted: i=1; AJvYcCXYP9jfgaaWRobIS0+0Pg3Z5fqZihlVxPnF7XRURgHpwPh3JpWfEPgOjsUk1Frkp+dI6qq64dy8/OUABJM=@vger.kernel.org X-Gm-Message-State: AOJu0YzzV1QX0D1Zg7iP+58Peki3IgWrkBhOmREFHozf5tu7qLWCMPVG y93KXtho7nAylXfNDTV7cPmGUdKMqcvC4IkaVzvHrRdNayUmmNntxDW4nY0KqVg= X-Gm-Gg: ASbGncvEjvY2LzsGc4fyL4hQI2XKE5FANR5DAyySBwCv/r1zq68AOTl2DlU8s83dh3W 1C9oTwg3hfh6d8aqGLemA2oabRZ5oHoo3KqhbGdvYAkugqSv2tt8//fj1smUYERWKvS3tRo3xtA Zoqr7YdrzmNuqahgioBGz5IzmC2bFnMhWAS2qAvv8epFhT/hSUn/yiN0s8FLriUmhofNeMv1Cn8 ywGGpi4JXh5IHU89OGu+1NRBtPSayqGGh9xO4iygILZFjotvLoHgoaDbuwSiXoWqFOCZN3Upx8c X+g= X-Google-Smtp-Source: AGHT+IGutP9CT/bKHGG/K5YQHtyEvJxs9HfNEL1yUqUDRQQkupgNCYnCJpWiQyiLvCI4Vu/2v7GT3g== X-Received: by 2002:a05:6a20:7350:b0:1e0:cc4a:caab with SMTP id adf61e73a8af0-1e88d12c1e2mr42368431637.19.1736910212772; Tue, 14 Jan 2025 19:03:32 -0800 (PST) Received: from hsinchu26.internal.sifive.com ([210.176.154.34]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-72d40568dccsm8123668b3a.47.2025.01.14.19.03.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 14 Jan 2025 19:03:32 -0800 (PST) From: Zong Li To: joro@8bytes.org, will@kernel.org, robin.murphy@arm.com, tjeznach@rivosinc.com, paul.walmsley@sifive.com, palmer@dabbelt.com, aou@eecs.berkeley.edu, luxu.kernel@bytedance.com, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-riscv@lists.infradead.org Cc: Zong Li Subject: [PATCH 2/2] iommu/riscv: support HPM and interrupt handling Date: Wed, 15 Jan 2025 11:03:06 +0800 Message-Id: <20250115030306.29735-3-zong.li@sifive.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20250115030306.29735-1-zong.li@sifive.com> References: <20250115030306.29735-1-zong.li@sifive.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Initialize the PMU and uninitialize it when driver is removed. Interrupt handling is also implemented, and the handler needs to be a primary handler instead of a threaded function because pt_regs is empty when threading the IRQ. However, pt_regs is required by perf_event_overflow. Signed-off-by: Zong Li Tested-by: Xu Lu --- drivers/iommu/riscv/iommu.c | 65 +++++++++++++++++++++++++++++++++++++ 1 file changed, 65 insertions(+) diff --git a/drivers/iommu/riscv/iommu.c b/drivers/iommu/riscv/iommu.c index 8a05def774bd..20ae90471484 100644 --- a/drivers/iommu/riscv/iommu.c +++ b/drivers/iommu/riscv/iommu.c @@ -552,6 +552,62 @@ static irqreturn_t riscv_iommu_fltq_process(int irq, v= oid *data) return IRQ_HANDLED; } =20 +/* + * IOMMU Hardware performance monitor + */ + +/* HPM interrupt primary handler */ +static irqreturn_t riscv_iommu_hpm_irq_handler(int irq, void *dev_id) +{ + struct riscv_iommu_device *iommu =3D (struct riscv_iommu_device *)dev_id; + + /* Clear performance monitoring interrupt pending */ + riscv_iommu_writel(iommu, RISCV_IOMMU_REG_IPSR, RISCV_IOMMU_IPSR_PMIP); + + /* Process pmu irq */ + riscv_iommu_pmu_handle_irq(&iommu->pmu); + + return IRQ_HANDLED; +} + +/* HPM initialization */ +static int riscv_iommu_hpm_enable(struct riscv_iommu_device *iommu) +{ + int rc; + + if (!(iommu->caps & RISCV_IOMMU_CAPABILITIES_HPM)) + return 0; + + /* + * pt_regs is empty when threading the IRQ, but pt_regs is necessary + * by perf_event_overflow. Use primary handler instead of thread + * function for PM IRQ. + * + * Set the IRQF_ONESHOT flag because this IRQ might be shared with + * other threaded IRQs by other queues. + */ + rc =3D devm_request_irq(iommu->dev, + iommu->irqs[riscv_iommu_queue_vec(iommu, RISCV_IOMMU_IPSR_PMIP)], + riscv_iommu_hpm_irq_handler, IRQF_ONESHOT | IRQF_SHARED, NULL, io= mmu); + if (rc) + return rc; + + return riscv_iommu_pmu_init(&iommu->pmu, iommu->reg, dev_name(iommu->dev)= ); +} + +/* HPM uninitialization */ +static void riscv_iommu_hpm_disable(struct riscv_iommu_device *iommu) +{ + if (!(iommu->caps & RISCV_IOMMU_CAPABILITIES_HPM)) + return; + + devm_free_irq(iommu->dev, + iommu->irqs[riscv_iommu_queue_vec(iommu, RISCV_IOMMU_IPSR_PMIP)], + iommu); + + riscv_iommu_pmu_uninit(&iommu->pmu); +} + /* Lookup and initialize device context info structure. */ static struct riscv_iommu_dc *riscv_iommu_get_dc(struct riscv_iommu_device= *iommu, unsigned int devid) @@ -1596,6 +1652,9 @@ void riscv_iommu_remove(struct riscv_iommu_device *io= mmu) riscv_iommu_iodir_set_mode(iommu, RISCV_IOMMU_DDTP_IOMMU_MODE_OFF); riscv_iommu_queue_disable(&iommu->cmdq); riscv_iommu_queue_disable(&iommu->fltq); + + if (iommu->caps & RISCV_IOMMU_CAPABILITIES_HPM) + riscv_iommu_pmu_uninit(&iommu->pmu); } =20 int riscv_iommu_init(struct riscv_iommu_device *iommu) @@ -1635,6 +1694,10 @@ int riscv_iommu_init(struct riscv_iommu_device *iomm= u) if (rc) goto err_queue_disable; =20 + rc =3D riscv_iommu_hpm_enable(iommu); + if (rc) + goto err_hpm_disable; + rc =3D iommu_device_sysfs_add(&iommu->iommu, NULL, NULL, "riscv-iommu@%s", dev_name(iommu->dev)); if (rc) { @@ -1653,6 +1716,8 @@ int riscv_iommu_init(struct riscv_iommu_device *iommu) err_remove_sysfs: iommu_device_sysfs_remove(&iommu->iommu); err_iodir_off: + riscv_iommu_hpm_disable(iommu); +err_hpm_disable: riscv_iommu_iodir_set_mode(iommu, RISCV_IOMMU_DDTP_IOMMU_MODE_OFF); err_queue_disable: riscv_iommu_queue_disable(&iommu->fltq); --=20 2.17.1