From nobody Tue Feb 10 15:29:43 2026 Received: from mail-pf1-f173.google.com (mail-pf1-f173.google.com [209.85.210.173]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BE1A515F408 for ; Tue, 7 May 2024 14:26:08 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.173 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715091970; cv=none; b=AaT5z2o5j83aKniRFz9giFx7Twgn63DDbp7u2A6KU/h/J2nmesU272tdKiTjumGLSU4RxUK+TnyguJ8t0b8ObEYM9cTcGY/IMZGfH4sJftt7vIpuFSEGNuDa2kZRRMEbr8ujdvHUu4rZueiVN+l+OVDgjf62jmgKbC5vH1tTfy4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715091970; c=relaxed/simple; bh=jvmCunqWfLZMTm0TzSSifX9CqijKbrgNiDjN6poD0Ek=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References; b=HGkGkr+ZcyKf2q261AsQuYFX5XEYEyFWAoBsg1toqTv2dnEMaoeRBnMW25TcC+5XQweHmjOMEo/XjmbEBWjNhZ5LtggyMgIIr7oe5441PAGyeJ3K/5M71S31X3mdQfuEncuWRuTKfGlI6rjjwQ0Vg7n2pWH6e8K1oP7k7N7+SFs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=sifive.com; spf=pass smtp.mailfrom=sifive.com; dkim=pass (2048-bit key) header.d=sifive.com header.i=@sifive.com header.b=RQn+x2FJ; arc=none smtp.client-ip=209.85.210.173 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=sifive.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=sifive.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=sifive.com header.i=@sifive.com header.b="RQn+x2FJ" Received: by mail-pf1-f173.google.com with SMTP id d2e1a72fcca58-6f447976de7so2449019b3a.1 for ; Tue, 07 May 2024 07:26:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1715091968; x=1715696768; darn=vger.kernel.org; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=17FazZA/6yHcTXFZKWGiwqZBDbFiQzycEthhRXcysw0=; b=RQn+x2FJVyORu5HC7ji7SIT6Q9QCu2kxyJrXYhwL0x5pPt1SUxC2jADgAIrHtrvwkg bUUAohb9r364PyxkmzrSP4zdbEhJNAIh6Qln8O4B/OcdEwh3xUtzjz10b4S3Bve8Gm62 t55S9IcUqYDhqlWAmf+G4f2AcC9iPf2v+r4f8q5/HLhZeGw+MuKW1/EDIFYVDwXUjzF0 aK4uNhmEgX3SAvWaIOqd5V4JlKQMcEmSTVMVHf9J5jL3vYEOdobnh6ryzZ/tHgJdDII0 wuJpAxRzdAk4SD9HTdAkOIEmKvCZ9w6NbhzphBoBwLPwn3RVd7Zesbk86WTT7kmsHAqD YhdQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1715091968; x=1715696768; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=17FazZA/6yHcTXFZKWGiwqZBDbFiQzycEthhRXcysw0=; b=OMLHK9oU1Vfsd5GH2IJ6k9kdFT3TIOIexshc35K8B1Q2tZwTxyVMCYtveML35E4QFA 36yWOF9pTdz+b0UhqDFJRxleIoLKZc6btiOtbyInhqKvOvkSMnJu7/dQMD5iIJfJSd14 pS88SVJrCkcZsdDkABNacG+wOFNnhx7NPodPBf8PvFJjUU8zGzE+gmEXfQ6AEOHzMzaS 0klDce7z9jOOGRonvM0bL7VHoqC68Xtd1DFcqKZO1UmsI4YCkWeTpsAG3lEX2aFJKrFc fILYSRoPn7vIdQh5DRXbvftLLfrBCF0RohN0lC+E/KKeICnTmxngnphUUUEK85VN4Mnn wNlw== X-Forwarded-Encrypted: i=1; AJvYcCWJZf84KPh/zpwmpswqj0jp324UtonuBxSxCKtzEDU0EfDWHcYSPBsl4eGk7aH8XyPg1HazHuflV+eNp3CO5jNUCib/Z8ZgNB0ZLqX/ X-Gm-Message-State: AOJu0YxI0ixNKygsI3+UBjZO4t+G8o9pxtIvamwJwczOW2yZgwEcA37M hCYrKUH7kGXE2RKJEAiFqpJc4zsSMbN+jSB6mEdzIMelLd+vqRZL6mAkuXhgWCI= X-Google-Smtp-Source: AGHT+IEVF58YFSyNvpGOqdtSAF9LBHk4AkUJDCtEDamXUUk6yuu3XQnJI085PYRdN1EgiwEXBqwp0w== X-Received: by 2002:a05:6a21:920d:b0:1af:86da:3f7 with SMTP id tl13-20020a056a21920d00b001af86da03f7mr11243607pzb.4.1715091968001; Tue, 07 May 2024 07:26:08 -0700 (PDT) Received: from hsinchu26.internal.sifive.com (59-124-168-89.hinet-ip.hinet.net. [59.124.168.89]) by smtp.gmail.com with ESMTPSA id i22-20020aa79096000000b006f44bcbe7e3sm7687554pfa.201.2024.05.07.07.26.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 07 May 2024 07:26:07 -0700 (PDT) From: Zong Li To: joro@8bytes.org, will@kernel.org, robin.murphy@arm.com, tjeznach@rivosinc.com, paul.walmsley@sifive.com, palmer@dabbelt.com, aou@eecs.berkeley.edu, jgg@ziepe.ca, kevin.tian@intel.com, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-riscv@lists.infradead.org Cc: Zong Li Subject: [PATCH RFC RESEND 1/6] iommu/riscv: Add RISC-V IOMMU PMU support Date: Tue, 7 May 2024 22:25:55 +0800 Message-Id: <20240507142600.23844-2-zong.li@sifive.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20240507142600.23844-1-zong.li@sifive.com> References: <20240507142600.23844-1-zong.li@sifive.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" This patch implements the RISC-V IOMMU hardware performance monitor, it includes the counting ans sampling mode. Specification doesn't define the event ID for counting the number of clock cycles, there is no associated iohpmevt0. But we need an event for counting cycle in perf, reserve the maximum number of event ID for it now. Signed-off-by: Zong Li --- drivers/iommu/riscv/Makefile | 4 +- drivers/iommu/riscv/iommu-bits.h | 15 + drivers/iommu/riscv/iommu-pmu.c | 477 +++++++++++++++++++++++++++++++ drivers/iommu/riscv/iommu.h | 8 + 4 files changed, 502 insertions(+), 2 deletions(-) create mode 100644 drivers/iommu/riscv/iommu-pmu.c diff --git a/drivers/iommu/riscv/Makefile b/drivers/iommu/riscv/Makefile index f54c9ed17d41..1b02e07d83c9 100644 --- a/drivers/iommu/riscv/Makefile +++ b/drivers/iommu/riscv/Makefile @@ -1,3 +1,3 @@ # SPDX-License-Identifier: GPL-2.0-only -obj-$(CONFIG_RISCV_IOMMU) +=3D iommu.o iommu-platform.o -obj-$(CONFIG_RISCV_IOMMU_PCI) +=3D iommu-pci.o +obj-$(CONFIG_RISCV_IOMMU) +=3D iommu.o iommu-platform.o iommu-pmu.o +obj-$(CONFIG_RISCV_IOMMU_PCI) +=3D iommu-pci.o iommu-pmu.o diff --git a/drivers/iommu/riscv/iommu-bits.h b/drivers/iommu/riscv/iommu-b= its.h index 40c379222821..11351cf6c710 100644 --- a/drivers/iommu/riscv/iommu-bits.h +++ b/drivers/iommu/riscv/iommu-bits.h @@ -17,6 +17,7 @@ #include #include #include +#include =20 /* * Chapter 5: Memory Mapped register interface @@ -210,6 +211,7 @@ enum riscv_iommu_ddtp_modes { /* 5.22 Performance monitoring event counters (31 * 64bits) */ #define RISCV_IOMMU_REG_IOHPMCTR_BASE 0x0068 #define RISCV_IOMMU_REG_IOHPMCTR(_n) (RISCV_IOMMU_REG_IOHPMCTR_BASE + ((_n= ) * 0x8)) +#define RISCV_IOMMU_IOHPMCTR_COUNTER GENMASK_ULL(63, 0) =20 /* 5.23 Performance monitoring event selectors (31 * 64bits) */ #define RISCV_IOMMU_REG_IOHPMEVT_BASE 0x0160 @@ -251,6 +253,19 @@ enum riscv_iommu_hpmevent_id { RISCV_IOMMU_HPMEVENT_MAX =3D 9 }; =20 +/* Use maximum event ID for cycle event */ +#define RISCV_IOMMU_HPMEVENT_CYCLE GENMASK_ULL(14, 0) + +#define RISCV_IOMMU_HPM_COUNTER_NUM 32 + +struct riscv_iommu_pmu { + struct pmu pmu; + void __iomem *reg; + int num_counters; + struct perf_event *events[RISCV_IOMMU_HPM_COUNTER_NUM]; + DECLARE_BITMAP(used_counters, RISCV_IOMMU_HPM_COUNTER_NUM); +}; + /* 5.24 Translation request IOVA (64bits) */ #define RISCV_IOMMU_REG_TR_REQ_IOVA 0x0258 #define RISCV_IOMMU_TR_REQ_IOVA_VPN GENMASK_ULL(63, 12) diff --git a/drivers/iommu/riscv/iommu-pmu.c b/drivers/iommu/riscv/iommu-pm= u.c new file mode 100644 index 000000000000..6ab50763860f --- /dev/null +++ b/drivers/iommu/riscv/iommu-pmu.c @@ -0,0 +1,477 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (C) 2024 SiFive + * + * Authors + * Zong Li + */ + +#include + +#include "iommu.h" +#include "iommu-bits.h" + +#define to_riscv_iommu_pmu(p) (container_of(p, struct riscv_iommu_pmu, pmu= )) + +#define RISCV_IOMMU_PMU_ATTR_EXTRACTOR(_name, _mask) \ + static inline u32 get_##_name(struct perf_event *event) \ + { \ + return FIELD_GET(_mask, event->attr.config); \ + } \ + +RISCV_IOMMU_PMU_ATTR_EXTRACTOR(event, RISCV_IOMMU_IOHPMEVT_EVENT_ID); +RISCV_IOMMU_PMU_ATTR_EXTRACTOR(partial_matching, RISCV_IOMMU_IOHPMEVT_DMAS= K); +RISCV_IOMMU_PMU_ATTR_EXTRACTOR(pid_pscid, RISCV_IOMMU_IOHPMEVT_PID_PSCID); +RISCV_IOMMU_PMU_ATTR_EXTRACTOR(did_gscid, RISCV_IOMMU_IOHPMEVT_DID_GSCID); +RISCV_IOMMU_PMU_ATTR_EXTRACTOR(filter_pid_pscid, RISCV_IOMMU_IOHPMEVT_PV_P= SCV); +RISCV_IOMMU_PMU_ATTR_EXTRACTOR(filter_did_gscid, RISCV_IOMMU_IOHPMEVT_DV_G= SCV); +RISCV_IOMMU_PMU_ATTR_EXTRACTOR(filter_id_type, RISCV_IOMMU_IOHPMEVT_IDT); + +/* Formats */ +PMU_FORMAT_ATTR(event, "config:0-14"); +PMU_FORMAT_ATTR(partial_matching, "config:15"); +PMU_FORMAT_ATTR(pid_pscid, "config:16-35"); +PMU_FORMAT_ATTR(did_gscid, "config:36-59"); +PMU_FORMAT_ATTR(filter_pid_pscid, "config:60"); +PMU_FORMAT_ATTR(filter_did_gscid, "config:61"); +PMU_FORMAT_ATTR(filter_id_type, "config:62"); + +static struct attribute *riscv_iommu_pmu_formats[] =3D { + &format_attr_event.attr, + &format_attr_partial_matching.attr, + &format_attr_pid_pscid.attr, + &format_attr_did_gscid.attr, + &format_attr_filter_pid_pscid.attr, + &format_attr_filter_did_gscid.attr, + &format_attr_filter_id_type.attr, + NULL, +}; + +static const struct attribute_group riscv_iommu_pmu_format_group =3D { + .name =3D "format", + .attrs =3D riscv_iommu_pmu_formats, +}; + +/* Events */ +static ssize_t riscv_iommu_pmu_event_show(struct device *dev, + struct device_attribute *attr, + char *page) +{ + struct perf_pmu_events_attr *pmu_attr; + + pmu_attr =3D container_of(attr, struct perf_pmu_events_attr, attr); + + return sprintf(page, "event=3D0x%02llx\n", pmu_attr->id); +} + +PMU_EVENT_ATTR(cycle, event_attr_cycle, + RISCV_IOMMU_HPMEVENT_CYCLE, riscv_iommu_pmu_event_show); +PMU_EVENT_ATTR(dont_count, event_attr_dont_count, + RISCV_IOMMU_HPMEVENT_INVALID, riscv_iommu_pmu_event_show); +PMU_EVENT_ATTR(untranslated_req, event_attr_untranslated_req, + RISCV_IOMMU_HPMEVENT_URQ, riscv_iommu_pmu_event_show); +PMU_EVENT_ATTR(translated_req, event_attr_translated_req, + RISCV_IOMMU_HPMEVENT_TRQ, riscv_iommu_pmu_event_show); +PMU_EVENT_ATTR(ats_trans_req, event_attr_ats_trans_req, + RISCV_IOMMU_HPMEVENT_ATS_RQ, riscv_iommu_pmu_event_show); +PMU_EVENT_ATTR(tlb_miss, event_attr_tlb_miss, + RISCV_IOMMU_HPMEVENT_TLB_MISS, riscv_iommu_pmu_event_show); +PMU_EVENT_ATTR(ddt_walks, event_attr_ddt_walks, + RISCV_IOMMU_HPMEVENT_DD_WALK, riscv_iommu_pmu_event_show); +PMU_EVENT_ATTR(pdt_walks, event_attr_pdt_walks, + RISCV_IOMMU_HPMEVENT_PD_WALK, riscv_iommu_pmu_event_show); +PMU_EVENT_ATTR(s_vs_pt_walks, event_attr_s_vs_pt_walks, + RISCV_IOMMU_HPMEVENT_S_VS_WALKS, riscv_iommu_pmu_event_show); +PMU_EVENT_ATTR(g_pt_walks, event_attr_g_pt_walks, + RISCV_IOMMU_HPMEVENT_G_WALKS, riscv_iommu_pmu_event_show); + +static struct attribute *riscv_iommu_pmu_events[] =3D { + &event_attr_cycle.attr.attr, + &event_attr_dont_count.attr.attr, + &event_attr_untranslated_req.attr.attr, + &event_attr_translated_req.attr.attr, + &event_attr_ats_trans_req.attr.attr, + &event_attr_tlb_miss.attr.attr, + &event_attr_ddt_walks.attr.attr, + &event_attr_pdt_walks.attr.attr, + &event_attr_s_vs_pt_walks.attr.attr, + &event_attr_g_pt_walks.attr.attr, + NULL, +}; + +static const struct attribute_group riscv_iommu_pmu_events_group =3D { + .name =3D "events", + .attrs =3D riscv_iommu_pmu_events, +}; + +static const struct attribute_group *riscv_iommu_pmu_attr_grps[] =3D { + &riscv_iommu_pmu_format_group, + &riscv_iommu_pmu_events_group, + NULL, +}; + +/* PMU Operations */ +static void riscv_iommu_pmu_set_counter(struct riscv_iommu_pmu *pmu, u32 i= dx, + u64 value) +{ + void __iomem *addr =3D pmu->reg + RISCV_IOMMU_REG_IOHPMCYCLES; + + if (WARN_ON_ONCE(idx < 0 || idx > pmu->num_counters)) + return; + + writeq(FIELD_PREP(RISCV_IOMMU_IOHPMCTR_COUNTER, value), addr + idx * 8); +} + +static u64 riscv_iommu_pmu_get_counter(struct riscv_iommu_pmu *pmu, u32 id= x) +{ + void __iomem *addr =3D pmu->reg + RISCV_IOMMU_REG_IOHPMCYCLES; + u64 value; + + if (WARN_ON_ONCE(idx < 0 || idx > pmu->num_counters)) + return -EINVAL; + + value =3D readq(addr + idx * 8); + + return FIELD_GET(RISCV_IOMMU_IOHPMCTR_COUNTER, value); +} + +static u64 riscv_iommu_pmu_get_event(struct riscv_iommu_pmu *pmu, u32 idx) +{ + void __iomem *addr =3D pmu->reg + RISCV_IOMMU_REG_IOHPMEVT_BASE; + + if (WARN_ON_ONCE(idx < 0 || idx > pmu->num_counters)) + return 0; + + /* There is no associtated IOHPMEVT0 for IOHPMCYCLES */ + if (idx =3D=3D 0) + return 0; + + return readq(addr + (idx - 1) * 8); +} + +static void riscv_iommu_pmu_set_event(struct riscv_iommu_pmu *pmu, u32 idx, + u64 value) +{ + void __iomem *addr =3D pmu->reg + RISCV_IOMMU_REG_IOHPMEVT_BASE; + + if (WARN_ON_ONCE(idx < 0 || idx > pmu->num_counters)) + return; + + /* There is no associtated IOHPMEVT0 for IOHPMCYCLES */ + if (idx =3D=3D 0) + return; + + writeq(value, addr + (idx - 1) * 8); +} + +static void riscv_iommu_pmu_enable_counter(struct riscv_iommu_pmu *pmu, u3= 2 idx) +{ + void __iomem *addr =3D pmu->reg + RISCV_IOMMU_REG_IOCOUNTINH; + u32 value =3D readl(addr); + + writel(value & ~BIT(idx), addr); +} + +static void riscv_iommu_pmu_disable_counter(struct riscv_iommu_pmu *pmu, u= 32 idx) +{ + void __iomem *addr =3D pmu->reg + RISCV_IOMMU_REG_IOCOUNTINH; + u32 value =3D readl(addr); + + writel(value | BIT(idx), addr); +} + +static void riscv_iommu_pmu_enable_ovf_intr(struct riscv_iommu_pmu *pmu, u= 32 idx) +{ + u64 value; + + if (get_event(pmu->events[idx]) =3D=3D RISCV_IOMMU_HPMEVENT_CYCLE) { + value =3D riscv_iommu_pmu_get_counter(pmu, idx) & ~RISCV_IOMMU_IOHPMCYCL= ES_OVF; + writeq(value, pmu->reg + RISCV_IOMMU_REG_IOHPMCYCLES); + } else { + value =3D riscv_iommu_pmu_get_event(pmu, idx) & ~RISCV_IOMMU_IOHPMEVT_OF; + writeq(value, pmu->reg + RISCV_IOMMU_REG_IOHPMEVT_BASE + (idx - 1) * 8); + } +} + +static void riscv_iommu_pmu_disable_ovf_intr(struct riscv_iommu_pmu *pmu, = u32 idx) +{ + u64 value; + + if (get_event(pmu->events[idx]) =3D=3D RISCV_IOMMU_HPMEVENT_CYCLE) { + value =3D riscv_iommu_pmu_get_counter(pmu, idx) | RISCV_IOMMU_IOHPMCYCLE= S_OVF; + writeq(value, pmu->reg + RISCV_IOMMU_REG_IOHPMCYCLES); + } else { + value =3D riscv_iommu_pmu_get_event(pmu, idx) | RISCV_IOMMU_IOHPMEVT_OF; + writeq(value, pmu->reg + RISCV_IOMMU_REG_IOHPMEVT_BASE + (idx - 1) * 8); + } +} + +static void riscv_iommu_pmu_start_all(struct riscv_iommu_pmu *pmu) +{ + int idx; + + for_each_set_bit(idx, pmu->used_counters, pmu->num_counters) { + riscv_iommu_pmu_enable_ovf_intr(pmu, idx); + riscv_iommu_pmu_enable_counter(pmu, idx); + } +} + +static void riscv_iommu_pmu_stop_all(struct riscv_iommu_pmu *pmu) +{ + writel(GENMASK_ULL(pmu->num_counters - 1, 0), + pmu->reg + RISCV_IOMMU_REG_IOCOUNTINH); +} + +/* PMU APIs */ +static int riscv_iommu_pmu_set_period(struct perf_event *event) +{ + struct riscv_iommu_pmu *pmu =3D to_riscv_iommu_pmu(event->pmu); + struct hw_perf_event *hwc =3D &event->hw; + s64 left =3D local64_read(&hwc->period_left); + s64 period =3D hwc->sample_period; + u64 max_period =3D RISCV_IOMMU_IOHPMCTR_COUNTER; + int ret =3D 0; + + if (unlikely(left <=3D -period)) { + left =3D period; + local64_set(&hwc->period_left, left); + hwc->last_period =3D period; + ret =3D 1; + } + + if (unlikely(left <=3D 0)) { + left +=3D period; + local64_set(&hwc->period_left, left); + hwc->last_period =3D period; + ret =3D 1; + } + + /* + * Limit the maximum period to prevent the counter value + * from overtaking the one we are about to program. In + * effect we are reducing max_period to account for + * interrupt latency (and we are being very conservative). + */ + if (left > (max_period >> 1)) + left =3D (max_period >> 1); + + local64_set(&hwc->prev_count, (u64)-left); + riscv_iommu_pmu_set_counter(pmu, hwc->idx, (u64)(-left) & max_period); + perf_event_update_userpage(event); + + return ret; +} + +static int riscv_iommu_pmu_event_init(struct perf_event *event) +{ + struct hw_perf_event *hwc =3D &event->hw; + + hwc->idx =3D -1; + hwc->config =3D event->attr.config; + + if (!is_sampling_event(event)) { + /* + * For non-sampling runs, limit the sample_period to half + * of the counter width. That way, the new counter value + * is far less likely to overtake the previous one unless + * you have some serious IRQ latency issues. + */ + hwc->sample_period =3D RISCV_IOMMU_IOHPMCTR_COUNTER >> 1; + hwc->last_period =3D hwc->sample_period; + local64_set(&hwc->period_left, hwc->sample_period); + } + + return 0; +} + +static void riscv_iommu_pmu_update(struct perf_event *event) +{ + struct hw_perf_event *hwc =3D &event->hw; + struct riscv_iommu_pmu *pmu =3D to_riscv_iommu_pmu(event->pmu); + u64 delta, prev, now; + u32 idx =3D hwc->idx; + + do { + prev =3D local64_read(&hwc->prev_count); + now =3D riscv_iommu_pmu_get_counter(pmu, idx); + } while (local64_cmpxchg(&hwc->prev_count, prev, now) !=3D prev); + + delta =3D FIELD_GET(RISCV_IOMMU_IOHPMCTR_COUNTER, now - prev); + local64_add(delta, &event->count); + local64_sub(delta, &hwc->period_left); +} + +static void riscv_iommu_pmu_start(struct perf_event *event, int flags) +{ + struct riscv_iommu_pmu *pmu =3D to_riscv_iommu_pmu(event->pmu); + struct hw_perf_event *hwc =3D &event->hw; + + if (WARN_ON_ONCE(!(event->hw.state & PERF_HES_STOPPED))) + return; + + if (flags & PERF_EF_RELOAD) + WARN_ON_ONCE(!(event->hw.state & PERF_HES_UPTODATE)); + + hwc->state =3D 0; + riscv_iommu_pmu_set_period(event); + riscv_iommu_pmu_set_event(pmu, hwc->idx, hwc->config); + riscv_iommu_pmu_enable_ovf_intr(pmu, hwc->idx); + riscv_iommu_pmu_enable_counter(pmu, hwc->idx); + + perf_event_update_userpage(event); +} + +static void riscv_iommu_pmu_stop(struct perf_event *event, int flags) +{ + struct riscv_iommu_pmu *pmu =3D to_riscv_iommu_pmu(event->pmu); + struct hw_perf_event *hwc =3D &event->hw; + + if (hwc->state & PERF_HES_STOPPED) + return; + + riscv_iommu_pmu_set_event(pmu, hwc->idx, RISCV_IOMMU_HPMEVENT_INVALID); + riscv_iommu_pmu_disable_counter(pmu, hwc->idx); + + if ((flags & PERF_EF_UPDATE) && !(hwc->state & PERF_HES_UPTODATE)) + riscv_iommu_pmu_update(event); + + hwc->state |=3D PERF_HES_STOPPED | PERF_HES_UPTODATE; +} + +static int riscv_iommu_pmu_add(struct perf_event *event, int flags) +{ + struct hw_perf_event *hwc =3D &event->hw; + struct riscv_iommu_pmu *pmu =3D to_riscv_iommu_pmu(event->pmu); + unsigned int num_counters =3D pmu->num_counters; + int idx; + + /* Reserve index zero for iohpmcycles */ + if (get_event(event) =3D=3D RISCV_IOMMU_HPMEVENT_CYCLE) + idx =3D 0; + else + idx =3D find_next_zero_bit(pmu->used_counters, num_counters, 1); + + if (idx =3D=3D num_counters) + return -EAGAIN; + + set_bit(idx, pmu->used_counters); + + pmu->events[idx] =3D event; + hwc->idx =3D idx; + hwc->state =3D PERF_HES_STOPPED | PERF_HES_UPTODATE; + + if (flags & PERF_EF_START) + riscv_iommu_pmu_start(event, flags); + + /* Propagate changes to the userspace mapping. */ + perf_event_update_userpage(event); + + return 0; +} + +static void riscv_iommu_pmu_read(struct perf_event *event) +{ + riscv_iommu_pmu_update(event); +} + +static void riscv_iommu_pmu_del(struct perf_event *event, int flags) +{ + struct hw_perf_event *hwc =3D &event->hw; + struct riscv_iommu_pmu *pmu =3D to_riscv_iommu_pmu(event->pmu); + int idx =3D hwc->idx; + + riscv_iommu_pmu_stop(event, PERF_EF_UPDATE); + pmu->events[idx] =3D NULL; + clear_bit(idx, pmu->used_counters); + perf_event_update_userpage(event); +} + +irqreturn_t riscv_iommu_pmu_handle_irq(struct riscv_iommu_pmu *pmu) +{ + struct perf_sample_data data; + struct pt_regs *regs; + u32 ovf =3D readl(pmu->reg + RISCV_IOMMU_REG_IOCOUNTOVF); + int idx; + + if (!ovf) + return IRQ_NONE; + + riscv_iommu_pmu_stop_all(pmu); + + regs =3D get_irq_regs(); + + for_each_set_bit(idx, (unsigned long *)&ovf, pmu->num_counters) { + struct perf_event *event =3D pmu->events[idx]; + struct hw_perf_event *hwc; + + if (WARN_ON_ONCE(!event) || !is_sampling_event(event)) + continue; + + hwc =3D &event->hw; + + riscv_iommu_pmu_update(event); + perf_sample_data_init(&data, 0, hwc->last_period); + if (!riscv_iommu_pmu_set_period(event)) + continue; + + if (perf_event_overflow(event, &data, regs)) + riscv_iommu_pmu_stop(event, 0); + } + + riscv_iommu_pmu_start_all(pmu); + + return IRQ_HANDLED; +} + +int riscv_iommu_pmu_init(struct riscv_iommu_pmu *pmu, void __iomem *reg, + const char *dev_name) +{ + char *name; + int ret; + + pmu->reg =3D reg; + pmu->num_counters =3D RISCV_IOMMU_HPM_COUNTER_NUM; + + pmu->pmu =3D (struct pmu) { + .task_ctx_nr =3D perf_invalid_context, + .event_init =3D riscv_iommu_pmu_event_init, + .add =3D riscv_iommu_pmu_add, + .del =3D riscv_iommu_pmu_del, + .start =3D riscv_iommu_pmu_start, + .stop =3D riscv_iommu_pmu_stop, + .read =3D riscv_iommu_pmu_read, + .attr_groups =3D riscv_iommu_pmu_attr_grps, + .capabilities =3D PERF_PMU_CAP_NO_EXCLUDE, + .module =3D THIS_MODULE, + }; + + name =3D kasprintf(GFP_KERNEL, "riscv-iommu-pmu@%s", dev_name); + + ret =3D perf_pmu_register(&pmu->pmu, name, -1); + if (ret) { + pr_err("Failed to register riscv-iommu-pmu@%s: %d\n", + dev_name, ret); + return ret; + } + + /* Stop all counters and later start the counter with perf */ + riscv_iommu_pmu_stop_all(pmu); + + pr_info("riscv-iommu-pmu@%s: Registered with %d counters\n", + dev_name, pmu->num_counters); + + return 0; +} + +void riscv_iommu_pmu_uninit(struct riscv_iommu_pmu *pmu) +{ + int idx; + + /* Disable interrupt and functions */ + for_each_set_bit(idx, pmu->used_counters, pmu->num_counters) { + riscv_iommu_pmu_disable_counter(pmu, idx); + riscv_iommu_pmu_disable_ovf_intr(pmu, idx); + } + + perf_pmu_unregister(&pmu->pmu); +} diff --git a/drivers/iommu/riscv/iommu.h b/drivers/iommu/riscv/iommu.h index 03e0c45bc7e1..ff66822c1114 100644 --- a/drivers/iommu/riscv/iommu.h +++ b/drivers/iommu/riscv/iommu.h @@ -60,11 +60,19 @@ struct riscv_iommu_device { unsigned int ddt_mode; dma_addr_t ddt_phys; u64 *ddt_root; + + /* hardware performance monitor */ + struct riscv_iommu_pmu pmu; }; =20 int riscv_iommu_init(struct riscv_iommu_device *iommu); void riscv_iommu_remove(struct riscv_iommu_device *iommu); =20 +int riscv_iommu_pmu_init(struct riscv_iommu_pmu *pmu, void __iomem *reg, + const char *name); +void riscv_iommu_pmu_uninit(struct riscv_iommu_pmu *pmu); +irqreturn_t riscv_iommu_pmu_handle_irq(struct riscv_iommu_pmu *pmu); + #define riscv_iommu_readl(iommu, addr) \ readl_relaxed((iommu)->reg + (addr)) =20 --=20 2.17.1 From nobody Tue Feb 10 15:29:43 2026 Received: from mail-pf1-f170.google.com (mail-pf1-f170.google.com [209.85.210.170]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1F6C715FA95 for ; Tue, 7 May 2024 14:26:10 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.170 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715091973; cv=none; b=Mv9dAZV0AwQhQvEdd+SEiYsuffz2UNVdBuyNmzlv2Xsc2kMqlVa5HN0rUUpWJiVOh7TIcqbLdTBYGvK9M4hJiZDoG7B1LXCv1j7nOQhSTgOWOjJxi52BQXb/bhQw5/kbhV9WGapYlZefVS7gmyK7yBZ5mn6Y1WvsCNgv5lLFryM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715091973; c=relaxed/simple; bh=VZxNS2OxyzYq3m1acYYqTvZC3vZEhXDPbe+Lr3aCwho=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References; b=S4UoNbmg2SBn8i14jI0/rlcQPmowDTJfwp0VxGwAQt31J6ODwL5tej5mrjXqYweC8sfAJr/rTvrutOeGpwumbw7O41kka80HAXQv2Hiw/tOnBv43sxPmTgzotv8qZbGe2etM0xcT+tNAOGKLBNePPrLpTZl4oHhGZRk+SRBCu4g= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=sifive.com; spf=pass smtp.mailfrom=sifive.com; dkim=pass (2048-bit key) header.d=sifive.com header.i=@sifive.com header.b=Fl5dYrOK; arc=none smtp.client-ip=209.85.210.170 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=sifive.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=sifive.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=sifive.com header.i=@sifive.com header.b="Fl5dYrOK" Received: by mail-pf1-f170.google.com with SMTP id d2e1a72fcca58-6f490b5c23bso719980b3a.3 for ; Tue, 07 May 2024 07:26:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1715091970; x=1715696770; darn=vger.kernel.org; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=0jz4ENNoetocqIssCS0B2oXUtzcTlnoT1WDoFLx5sIU=; b=Fl5dYrOKtK/1LYFWcio9ytUp7A3TIApjuT985xUUgG/PaHbSI7PTDKI36QylG8fBve i3oLdUPMNsEw5iZCJK61cx2yS7Gn8OrYmBFOO2T9y2YnqrceOVvuh8NcRG3ps+LK2EBC LoK/7G+XPm38FYZMK6zOhSP2qudhsxcr0Ui8I3U39Q7OquWIbgfiGkCWqktTKMTOdlLD ahnV+yFL6CLXxbq0PXRe/6tEcMX4t73esGF1q6DNWVTfNtZ0e1NoKifEclkI5gcFIhxW VYKpFTmvWQNh/1XEdmb7yyuTBoJyIWo4nTcbodbPoa7msmC8SF5uBWzjxJ5aW+kl2y5G MPzA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1715091970; x=1715696770; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=0jz4ENNoetocqIssCS0B2oXUtzcTlnoT1WDoFLx5sIU=; b=beM4Brs6RKCCoISyxs3/+kO55vJkbZS7BE5dQM8iJRGJbWGC0j5+qUdA/ZfqK0le3L uuYFx7lRx1dNknc8QhwMQn42wsfFS406f8uhEYks+6dX0iUC0Ar3tQc5SPisQuve2GTz 8OFf5sv55duC0k7M0eFCe3qi8cND7WS+N1fR/FxXOHTu9c9Xj8vKiwmZKHEKCEqa8Syw JexZXzaza22fkqCaCUPG55cU+Bzbc9O6h5H+HW2yGhw6CFKw+SS6y3LkfHdYeixio65w jYtrgEkO80LK5En5uKUG2xaQotRSOgsBBYZLkRh0+00+OJGztEAzqNVld4VVVIjDXL7N nwcg== X-Forwarded-Encrypted: i=1; AJvYcCXN2GV9Qj8s+3R1/Dyr5vmArN0zNcRXAnwZtLokVyO+R07A+WjumJ6d0wG/kJCDtL3eBgeQuDHf3+WqNHwBFE9UDdDWHv6guN445AFX X-Gm-Message-State: AOJu0YwXfHJ7v25nZW64XqtXTOs7cy4DTR4fM7cr+Laz95wOHLEYmdyY awvxc5x2A+x6Z15PFAu5mAhNq7y7aDynaxwJlLYYuzNsyvqLMgxBiiLcgYZvFJE= X-Google-Smtp-Source: AGHT+IFFwJydlhv8Eo1uigoZzco+8G3QZGbkY7AV3d8a5KKB5Jnd/eMaJgfIknHgv405ntAxsJkBlA== X-Received: by 2002:a05:6a20:9c97:b0:1af:9ee6:25c4 with SMTP id mj23-20020a056a209c9700b001af9ee625c4mr8750903pzb.42.1715091970500; Tue, 07 May 2024 07:26:10 -0700 (PDT) Received: from hsinchu26.internal.sifive.com (59-124-168-89.hinet-ip.hinet.net. [59.124.168.89]) by smtp.gmail.com with ESMTPSA id i22-20020aa79096000000b006f44bcbe7e3sm7687554pfa.201.2024.05.07.07.26.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 07 May 2024 07:26:10 -0700 (PDT) From: Zong Li To: joro@8bytes.org, will@kernel.org, robin.murphy@arm.com, tjeznach@rivosinc.com, paul.walmsley@sifive.com, palmer@dabbelt.com, aou@eecs.berkeley.edu, jgg@ziepe.ca, kevin.tian@intel.com, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-riscv@lists.infradead.org Cc: Zong Li Subject: [PATCH RFC RESEND 2/6] iommu/riscv: Support HPM and interrupt handling Date: Tue, 7 May 2024 22:25:56 +0800 Message-Id: <20240507142600.23844-3-zong.li@sifive.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20240507142600.23844-1-zong.li@sifive.com> References: <20240507142600.23844-1-zong.li@sifive.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" This patch initialize the pmu stuff and uninitialize it when driver removing. The interrupt handling is also provided, this handler need to be primary handler instead of thread function, because pt_regs is empty when threading the IRQ, but pt_regs is necessary by perf_event_overflow. Signed-off-by: Zong Li --- drivers/iommu/riscv/iommu.c | 59 +++++++++++++++++++++++++++++++++++++ 1 file changed, 59 insertions(+) diff --git a/drivers/iommu/riscv/iommu.c b/drivers/iommu/riscv/iommu.c index ec701fde520f..e0bf74a9c64d 100644 --- a/drivers/iommu/riscv/iommu.c +++ b/drivers/iommu/riscv/iommu.c @@ -526,6 +526,56 @@ static irqreturn_t riscv_iommu_fltq_process(int irq, v= oid *data) return IRQ_HANDLED; } =20 +/* + * IOMMU Hardware performance monitor + */ + +/* HPM interrupt primary handler */ +static irqreturn_t riscv_iommu_hpm_irq_handler(int irq, void *dev_id) +{ + struct riscv_iommu_device *iommu =3D (struct riscv_iommu_device *)dev_id; + + /* Process pmu irq */ + riscv_iommu_pmu_handle_irq(&iommu->pmu); + + /* Clear performance monitoring interrupt pending */ + riscv_iommu_writel(iommu, RISCV_IOMMU_REG_IPSR, RISCV_IOMMU_IPSR_PMIP); + + return IRQ_HANDLED; +} + +/* HPM initialization */ +static int riscv_iommu_hpm_enable(struct riscv_iommu_device *iommu) +{ + int rc; + + if (!(iommu->caps & RISCV_IOMMU_CAP_HPM)) + return 0; + + /* + * pt_regs is empty when threading the IRQ, but pt_regs is necessary + * by perf_event_overflow. Use primary handler instead of thread + * function for PM IRQ. + */ + rc =3D devm_request_irq(iommu->dev, iommu->irqs[RISCV_IOMMU_IPSR_PMIP], + riscv_iommu_hpm_irq_handler, 0, NULL, iommu); + if (rc) + return rc; + + return riscv_iommu_pmu_init(&iommu->pmu, iommu->reg, dev_name(iommu->dev)= ); +} + +/* HPM uninitialization */ +static void riscv_iommu_hpm_disable(struct riscv_iommu_device *iommu) +{ + if (!(iommu->caps & RISCV_IOMMU_CAP_HPM)) + return; + + devm_free_irq(iommu->dev, iommu->irqs[RISCV_IOMMU_IPSR_PMIP], iommu); + + riscv_iommu_pmu_uninit(&iommu->pmu); +} + /* Lookup and initialize device context info structure. */ static struct riscv_iommu_dc *riscv_iommu_get_dc(struct riscv_iommu_device= *iommu, unsigned int devid) @@ -1551,6 +1601,9 @@ void riscv_iommu_remove(struct riscv_iommu_device *io= mmu) riscv_iommu_iodir_set_mode(iommu, RISCV_IOMMU_DDTP_MODE_OFF); riscv_iommu_queue_disable(&iommu->cmdq); riscv_iommu_queue_disable(&iommu->fltq); + + if (iommu->caps & RISCV_IOMMU_CAP_HPM) + riscv_iommu_pmu_uninit(&iommu->pmu); } =20 int riscv_iommu_init(struct riscv_iommu_device *iommu) @@ -1590,6 +1643,10 @@ int riscv_iommu_init(struct riscv_iommu_device *iomm= u) if (rc) goto err_queue_disable; =20 + rc =3D riscv_iommu_hpm_enable(iommu); + if (rc) + goto err_hpm_disable; + rc =3D iommu_device_sysfs_add(&iommu->iommu, NULL, NULL, "riscv-iommu@%s", dev_name(iommu->dev)); if (rc) { @@ -1608,6 +1665,8 @@ int riscv_iommu_init(struct riscv_iommu_device *iommu) err_remove_sysfs: iommu_device_sysfs_remove(&iommu->iommu); err_iodir_off: + riscv_iommu_hpm_disable(iommu); +err_hpm_disable: riscv_iommu_iodir_set_mode(iommu, RISCV_IOMMU_DDTP_MODE_OFF); err_queue_disable: riscv_iommu_queue_disable(&iommu->fltq); --=20 2.17.1 From nobody Tue Feb 10 15:29:43 2026 Received: from mail-pf1-f178.google.com (mail-pf1-f178.google.com [209.85.210.178]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0DC9C15FCF9 for ; Tue, 7 May 2024 14:26:14 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.178 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715091976; cv=none; b=Bwy2lBNe/G23jdTVWmYNkq7oY0GS9vkbZ09UN5oI0Njc3gl5EfbVQjcPlrrByOUcKnkRw6XwPuPPpflywMCl+mDA0t5iu3x8wgj32G9o90XQAyjOyDhSmbi+DCtis9IPDno27OtWl0aWNMRA2C6y7QTFCwRANP9MD5L61ePajvc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715091976; c=relaxed/simple; bh=dle2cRslL/eml8rQHKVoHJTE4hJnwycWESm76X81LvU=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References; b=l3FAAKMQo43W3CznA8aECQByn3GFKed+ouJbRdaCpFughBkTTk00k/aOsQJqBbOJB6/VqdTbtMPIMrV6ijmjcZd2PHLB1Rlo9/wDRqjUDaRZotxWeC4duGrFQVIxFKRKwfB14lBQcZWckVu8MQwawrrHLMp6X/9qEfeqNKBbS/Y= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=sifive.com; spf=pass smtp.mailfrom=sifive.com; dkim=pass (2048-bit key) header.d=sifive.com header.i=@sifive.com header.b=j5B0TW5n; arc=none smtp.client-ip=209.85.210.178 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=sifive.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=sifive.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=sifive.com header.i=@sifive.com header.b="j5B0TW5n" Received: by mail-pf1-f178.google.com with SMTP id d2e1a72fcca58-6f44bcbaae7so2388328b3a.2 for ; Tue, 07 May 2024 07:26:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1715091974; x=1715696774; darn=vger.kernel.org; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=Q5Yiv5A35fqkZLewYyNasBxNRcaYrtDBJ52Nob0v8A4=; b=j5B0TW5n7wposDagnyi1GHTtyStH/FVuILUgGSlu5kgE9Uop5CkGhBuXJrtNT6uvxV acxYu2wysQwfQCXzc1cYvLjDciMsvFprYQMFyWJXa/ZtJ913o4zoPzTmqy3aanGGyXlq Gsp+yGZxFbaWv/ztJ6RwFCLfRhzpbbSof0NZN8JiSIv8su2P40+RZslYobuRy83r/qlG SP5imMIXrAqHBbNyGMTG+XajteI08YtEdyLAO5/BovA45Ky0784ZDwx+V9xcloKcJUew l2o2kRD86TmXZ7hiqmNG7X4y1YLXpuexNNAulLF2PHEiMXRdIII4wn1FhUwD70Dc7zPF tIvw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1715091974; x=1715696774; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Q5Yiv5A35fqkZLewYyNasBxNRcaYrtDBJ52Nob0v8A4=; b=jkzjn8do0w8rwPYGMb24MvpjZIe4T9QWV0G5R/FFJmfUSnazcshaZKWPxNWOzFzPaC JaI3wN9/UAiBPnM4LrsRgfO8+Dcz42B88gVXxbT7xDiTV/gySsZl9DVVlY2atQkUNE6p L7eMelnqmJm5Cu8VZY1nvbyaLClu7B+fH+n89M0Nl9hohLVjDVmuxH6BDyXkz6TJTtSQ xyT2dv2duuOb8JeW/+vhYz5TySV5xZEOU72qZub73YReEntJLdvr43oiQDBFFA+RNHk4 Gv7iWyE9x48DeXfWMeyf3Xpgtwp406FoyvbRWotlQmsLc58p4fGhbvWDWzVIURPaKqYd ADJw== X-Forwarded-Encrypted: i=1; AJvYcCXHbxM5O5eL7tWxqnj9ObckvSRFzFwKYvfpF6sMBomIKTIkJCkzqnTJWwjI/DDBWNbyo2GJNGjUS71YIcaRhr6vZmPMIh/4vvbZQgKh X-Gm-Message-State: AOJu0YxT6opmd17k2BgrKAk6HQ+MNEA+b9TbaUgR26y43ov2lKjZkyeR Tb6o6asHjVWyk4Z8dSYRx2jQH4OX3uQRWWWfjDJ0PWlLKQ/AXGZ3xAljp5YBivg= X-Google-Smtp-Source: AGHT+IHdg7+7I4bSI4CruHY7MHG+tPkBhVw4YY0PCUXdy/YjCBHjgssv5B+jlzNMBIZ1jyV/U90aVQ== X-Received: by 2002:a05:6a00:4f82:b0:6ed:21b2:cb17 with SMTP id ld2-20020a056a004f8200b006ed21b2cb17mr14575610pfb.4.1715091972855; Tue, 07 May 2024 07:26:12 -0700 (PDT) Received: from hsinchu26.internal.sifive.com (59-124-168-89.hinet-ip.hinet.net. [59.124.168.89]) by smtp.gmail.com with ESMTPSA id i22-20020aa79096000000b006f44bcbe7e3sm7687554pfa.201.2024.05.07.07.26.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 07 May 2024 07:26:12 -0700 (PDT) From: Zong Li To: joro@8bytes.org, will@kernel.org, robin.murphy@arm.com, tjeznach@rivosinc.com, paul.walmsley@sifive.com, palmer@dabbelt.com, aou@eecs.berkeley.edu, jgg@ziepe.ca, kevin.tian@intel.com, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-riscv@lists.infradead.org Cc: Zong Li Subject: [PATCH RFC RESEND 3/6] iommu/riscv: support GSCID Date: Tue, 7 May 2024 22:25:57 +0800 Message-Id: <20240507142600.23844-4-zong.li@sifive.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20240507142600.23844-1-zong.li@sifive.com> References: <20240507142600.23844-1-zong.li@sifive.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" This patch adds a global ID Allocator for GSCID and a wrap for setting up GSCID in IOTLB invalidation command. Set up iohgatp to enable second stage table and flus stage-2 table if the GSCID is allocated. The GSCID of domain should be freed when release domain. GSCID will be allocated for parent domain in nested IOMMU process. Signed-off-by: Zong Li --- drivers/iommu/riscv/iommu-bits.h | 7 +++ drivers/iommu/riscv/iommu.c | 81 ++++++++++++++++++++++---------- 2 files changed, 62 insertions(+), 26 deletions(-) diff --git a/drivers/iommu/riscv/iommu-bits.h b/drivers/iommu/riscv/iommu-b= its.h index 11351cf6c710..62b1ee387357 100644 --- a/drivers/iommu/riscv/iommu-bits.h +++ b/drivers/iommu/riscv/iommu-bits.h @@ -728,6 +728,13 @@ static inline void riscv_iommu_cmd_inval_vma(struct ri= scv_iommu_command *cmd) cmd->dword1 =3D 0; } =20 +static inline void riscv_iommu_cmd_inval_gvma(struct riscv_iommu_command *= cmd) +{ + cmd->dword0 =3D FIELD_PREP(RISCV_IOMMU_CMD_OPCODE, RISCV_IOMMU_CMD_IOTINV= AL_OPCODE) | + FIELD_PREP(RISCV_IOMMU_CMD_FUNC, RISCV_IOMMU_CMD_IOTINVAL_FUNC_GVM= A); + cmd->dword1 =3D 0; +} + static inline void riscv_iommu_cmd_inval_set_addr(struct riscv_iommu_comma= nd *cmd, u64 addr) { diff --git a/drivers/iommu/riscv/iommu.c b/drivers/iommu/riscv/iommu.c index e0bf74a9c64d..d38e09b138b6 100644 --- a/drivers/iommu/riscv/iommu.c +++ b/drivers/iommu/riscv/iommu.c @@ -45,6 +45,10 @@ static DEFINE_IDA(riscv_iommu_pscids); #define RISCV_IOMMU_MAX_PSCID (BIT(20) - 1) =20 +/* IOMMU GSCID allocation namespace. */ +static DEFINE_IDA(riscv_iommu_gscids); +#define RISCV_IOMMU_MAX_GSCID (BIT(16) - 1) + /* Device resource-managed allocations */ struct riscv_iommu_devres { void *addr; @@ -826,6 +830,7 @@ struct riscv_iommu_domain { struct list_head bonds; spinlock_t lock; /* protect bonds list updates. */ int pscid; + int gscid; int numa_node; int amo_enabled:1; unsigned int pgd_mode; @@ -919,29 +924,43 @@ static void riscv_iommu_iotlb_inval(struct riscv_iomm= u_domain *domain, rcu_read_lock(); =20 prev =3D NULL; - list_for_each_entry_rcu(bond, &domain->bonds, list) { - iommu =3D dev_to_iommu(bond->dev); =20 - /* - * IOTLB invalidation request can be safely omitted if already sent - * to the IOMMU for the same PSCID, and with domain->bonds list - * arranged based on the device's IOMMU, it's sufficient to check - * last device the invalidation was sent to. - */ - if (iommu =3D=3D prev) - continue; - - riscv_iommu_cmd_inval_vma(&cmd); - riscv_iommu_cmd_inval_set_pscid(&cmd, domain->pscid); - if (len && len >=3D RISCV_IOMMU_IOTLB_INVAL_LIMIT) { - for (iova =3D start; iova < end; iova +=3D PAGE_SIZE) { - riscv_iommu_cmd_inval_set_addr(&cmd, iova); + /* + * Host domain needs to flush entries in stage-2 for MSI mapping. + * However, device is bound to s1 domain instead of s2 domain. + * We need to flush mapping without looping devices of s2 domain + */ + if (domain->gscid) { + riscv_iommu_cmd_inval_gvma(&cmd); + riscv_iommu_cmd_inval_set_gscid(&cmd, domain->gscid); + riscv_iommu_cmd_send(iommu, &cmd, 0); + riscv_iommu_cmd_iofence(&cmd); + riscv_iommu_cmd_send(iommu, &cmd, RISCV_IOMMU_QUEUE_TIMEOUT); + } else { + list_for_each_entry_rcu(bond, &domain->bonds, list) { + iommu =3D dev_to_iommu(bond->dev); + + /* + * IOTLB invalidation request can be safely omitted if already sent + * to the IOMMU for the same PSCID, and with domain->bonds list + * arranged based on the device's IOMMU, it's sufficient to check + * last device the invalidation was sent to. + */ + if (iommu =3D=3D prev) + continue; + + riscv_iommu_cmd_inval_vma(&cmd); + riscv_iommu_cmd_inval_set_pscid(&cmd, domain->pscid); + if (len && len >=3D RISCV_IOMMU_IOTLB_INVAL_LIMIT) { + for (iova =3D start; iova < end; iova +=3D PAGE_SIZE) { + riscv_iommu_cmd_inval_set_addr(&cmd, iova); + riscv_iommu_cmd_send(iommu, &cmd, 0); + } + } else { riscv_iommu_cmd_send(iommu, &cmd, 0); } - } else { - riscv_iommu_cmd_send(iommu, &cmd, 0); + prev =3D iommu; } - prev =3D iommu; } =20 prev =3D NULL; @@ -972,7 +991,7 @@ static void riscv_iommu_iotlb_inval(struct riscv_iommu_= domain *domain, * interim translation faults. */ static void riscv_iommu_iodir_update(struct riscv_iommu_device *iommu, - struct device *dev, u64 fsc, u64 ta) + struct device *dev, u64 fsc, u64 ta, u64 iohgatp) { struct iommu_fwspec *fwspec =3D dev_iommu_fwspec_get(dev); struct riscv_iommu_dc *dc; @@ -1012,6 +1031,7 @@ static void riscv_iommu_iodir_update(struct riscv_iom= mu_device *iommu, /* Update device context, write TC.V as the last step. */ WRITE_ONCE(dc->fsc, fsc); WRITE_ONCE(dc->ta, ta & RISCV_IOMMU_PC_TA_PSCID); + WRITE_ONCE(dc->iohgatp, iohgatp); WRITE_ONCE(dc->tc, tc); } } @@ -1271,6 +1291,9 @@ static void riscv_iommu_free_paging_domain(struct iom= mu_domain *iommu_domain) if ((int)domain->pscid > 0) ida_free(&riscv_iommu_pscids, domain->pscid); =20 + if ((int)domain->gscid > 0) + ida_free(&riscv_iommu_gscids, domain->gscid); + riscv_iommu_pte_free(domain, _io_pte_entry(pfn, _PAGE_TABLE), NULL); kfree(domain); } @@ -1296,7 +1319,7 @@ static int riscv_iommu_attach_paging_domain(struct io= mmu_domain *iommu_domain, struct riscv_iommu_domain *domain =3D iommu_domain_to_riscv(iommu_domain); struct riscv_iommu_device *iommu =3D dev_to_iommu(dev); struct riscv_iommu_info *info =3D dev_iommu_priv_get(dev); - u64 fsc, ta; + u64 fsc =3D 0, iohgatp =3D 0, ta; =20 if (!riscv_iommu_pt_supported(iommu, domain->pgd_mode)) return -ENODEV; @@ -1314,12 +1337,18 @@ static int riscv_iommu_attach_paging_domain(struct = iommu_domain *iommu_domain, */ riscv_iommu_iotlb_inval(domain, 0, ULONG_MAX); =20 - fsc =3D FIELD_PREP(RISCV_IOMMU_PC_FSC_MODE, domain->pgd_mode) | - FIELD_PREP(RISCV_IOMMU_PC_FSC_PPN, virt_to_pfn(domain->pgd_root)); + if (domain->gscid) + iohgatp =3D FIELD_PREP(RISCV_IOMMU_DC_IOHGATP_MODE, domain->pgd_mode) | + FIELD_PREP(RISCV_IOMMU_DC_IOHGATP_GSCID, domain->gscid) | + FIELD_PREP(RISCV_IOMMU_DC_IOHGATP_PPN, virt_to_pfn(domain->pgd_root)); + else + fsc =3D FIELD_PREP(RISCV_IOMMU_PC_FSC_MODE, domain->pgd_mode) | + FIELD_PREP(RISCV_IOMMU_PC_FSC_PPN, virt_to_pfn(domain->pgd_root)); + ta =3D FIELD_PREP(RISCV_IOMMU_PC_TA_PSCID, domain->pscid) | RISCV_IOMMU_PC_TA_V; =20 - riscv_iommu_iodir_update(iommu, dev, fsc, ta); + riscv_iommu_iodir_update(iommu, dev, fsc, ta, iohgatp); riscv_iommu_bond_unlink(info->domain, dev); info->domain =3D domain; =20 @@ -1422,7 +1451,7 @@ static int riscv_iommu_attach_blocking_domain(struct = iommu_domain *iommu_domain, struct riscv_iommu_device *iommu =3D dev_to_iommu(dev); struct riscv_iommu_info *info =3D dev_iommu_priv_get(dev); =20 - riscv_iommu_iodir_update(iommu, dev, RISCV_IOMMU_FSC_BARE, 0); + riscv_iommu_iodir_update(iommu, dev, RISCV_IOMMU_FSC_BARE, 0, 0); riscv_iommu_bond_unlink(info->domain, dev); info->domain =3D NULL; =20 @@ -1442,7 +1471,7 @@ static int riscv_iommu_attach_identity_domain(struct = iommu_domain *iommu_domain, struct riscv_iommu_device *iommu =3D dev_to_iommu(dev); struct riscv_iommu_info *info =3D dev_iommu_priv_get(dev); =20 - riscv_iommu_iodir_update(iommu, dev, RISCV_IOMMU_FSC_BARE, RISCV_IOMMU_PC= _TA_V); + riscv_iommu_iodir_update(iommu, dev, RISCV_IOMMU_FSC_BARE, RISCV_IOMMU_PC= _TA_V, 0); riscv_iommu_bond_unlink(info->domain, dev); info->domain =3D NULL; =20 --=20 2.17.1 From nobody Tue Feb 10 15:29:43 2026 Received: from mail-pf1-f169.google.com (mail-pf1-f169.google.com [209.85.210.169]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0CFD41607A7 for ; Tue, 7 May 2024 14:26:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.169 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715091979; cv=none; b=eEeT1Fxl7Cko7UB5kRs/L8t5kTgll4jW1jy1BK+PmHqDe52DS6WrEE1ehMVkk7YYofwIDivgwVkLVmM/dFf64dA/cv/yS9ft+JoVdZm648ICqmDT8O4PuKcTIP46im0RlmJx1qoWRVEw7C/VeJYQVeo2l/vk2QU5WL9z8ENLM/A= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715091979; c=relaxed/simple; bh=/u32NxJwpRrssHruSux3ZSYrSazvMyTW+Oc6RuusvT4=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References; b=lbUzRn7AE4+UBXLFNVlM/EmCzXndGTjlmtj/8vw3lTK4CUpVHIvN2/2QVcpAvZeGc6heQahp3s80mRii8XUQNmqZo30KtBSov72FXw3EYUGFrsm6KNW/t4F5TF3FnP8qMIgDU1MIIhouzVObAGmfBUxAczQAAsnsGHCfp2C70V8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=sifive.com; spf=pass smtp.mailfrom=sifive.com; dkim=pass (2048-bit key) header.d=sifive.com header.i=@sifive.com header.b=fojrQ+mI; arc=none smtp.client-ip=209.85.210.169 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=sifive.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=sifive.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=sifive.com header.i=@sifive.com header.b="fojrQ+mI" Received: by mail-pf1-f169.google.com with SMTP id d2e1a72fcca58-6f467fb2e66so2476159b3a.1 for ; Tue, 07 May 2024 07:26:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1715091977; x=1715696777; darn=vger.kernel.org; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=b+b6a14h4JCjFlQP6t364i5UnYTPW3CwsNAJ2Ic+YBk=; b=fojrQ+mIqtbTkZGc4i7aWVVypJyxH8rfo4xAJbiJdDHCsmRxPS01loIMfOV+aEpIB8 q1TKP7qGi5ApGajtePpAtpU4y+bySFitIfzglloy+R5ayyHqCzYI9CdKswAGhgcxVQ9s h8yqwN9r9wdWkf5fHCLItpxohECdBeXbYlpu0fxCIK6cqREoTBiexjO71pLdlpaclJCV ITl4ZI5KgGvJqhF2xjjvlF/yzPiQ6e1y7RIFMm325US09wRXpmjZeMCFq5N68/IeaUw6 OxsrmQBcjmAJpyrD7SGYq9jDfmOQcnTMPRtGQ9YqxOyf7Qh/PzGDRRLXk6E/7FCwcVZM zCyQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1715091977; x=1715696777; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=b+b6a14h4JCjFlQP6t364i5UnYTPW3CwsNAJ2Ic+YBk=; b=Ni5HnPXD08DyG6s0dBEyFz+ucIktuBYSOjU8ppmefoJPvakZlnt20igtseJPl2c48p Haztj1NLvkJGD72s3a8LIk+JEpLBTT4TaaUGmmwBSnsBO/FbSBsbR3+3XGU8SSfkDpvC QCEw8VpfOHcqmiaY7yQz0SLybK8a4dda6lfWbAJDUY+YRP8Pf855oSAHeqEYgHFfOX0d Z0KAPZC+uRY0la6sqIn86YqVLkoHJD+eD5mcIFQpPmcJDccidG1RX9wmeFyC12+w82r2 acEYM0+QzdUA4UapvC0RERei7oAhn7TlBzvxBD6n9QBq4pgdIIf9SQR6/6/H1PYWd9ub Ek8w== X-Forwarded-Encrypted: i=1; AJvYcCWz52gCYlv+o6hxrtXnTn4f/UU5wJgDyyunfU36EcEfN12qW4QA2d8Pk4lPJLNLHnfHZN0JqEjKx3vwyZMZaPZHN7U0F/4tGFwO79UK X-Gm-Message-State: AOJu0YyH3T87Na7RDca9TVQKgy/Oxra1CUQzTz4j8LtcT8gO8S+kSgA2 kwZgKsGMiE4lpyc897Zz1Juy1YH2pKeZZBZ767kqTwW4VF4j1yotELYLUkX/1ps= X-Google-Smtp-Source: AGHT+IGGpKiuhFlm6XAxN2dekhHNztwFPkYH1wk0fi2F0H3c/gi7fQ/EHQEjzLAwbTOOSYiKwoJeKg== X-Received: by 2002:a05:6a00:843:b0:6ea:86f2:24fb with SMTP id q3-20020a056a00084300b006ea86f224fbmr15303975pfk.25.1715091976310; Tue, 07 May 2024 07:26:16 -0700 (PDT) Received: from hsinchu26.internal.sifive.com (59-124-168-89.hinet-ip.hinet.net. [59.124.168.89]) by smtp.gmail.com with ESMTPSA id i22-20020aa79096000000b006f44bcbe7e3sm7687554pfa.201.2024.05.07.07.26.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 07 May 2024 07:26:16 -0700 (PDT) From: Zong Li To: joro@8bytes.org, will@kernel.org, robin.murphy@arm.com, tjeznach@rivosinc.com, paul.walmsley@sifive.com, palmer@dabbelt.com, aou@eecs.berkeley.edu, jgg@ziepe.ca, kevin.tian@intel.com, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-riscv@lists.infradead.org Cc: Zong Li Subject: [PATCH RFC RESEND 4/6] iommu/riscv: support nested iommu for getting iommu hardware information Date: Tue, 7 May 2024 22:25:58 +0800 Message-Id: <20240507142600.23844-5-zong.li@sifive.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20240507142600.23844-1-zong.li@sifive.com> References: <20240507142600.23844-1-zong.li@sifive.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" This patch implements .hw_info operation and the related data structures for passing the IOMMU hardware capabilities for iommufd. Signed-off-by: Zong Li --- drivers/iommu/riscv/iommu.c | 23 +++++++++++++++++++++++ include/uapi/linux/iommufd.h | 13 +++++++++++++ 2 files changed, 36 insertions(+) diff --git a/drivers/iommu/riscv/iommu.c b/drivers/iommu/riscv/iommu.c index d38e09b138b6..072251f6ad85 100644 --- a/drivers/iommu/riscv/iommu.c +++ b/drivers/iommu/riscv/iommu.c @@ -19,6 +19,7 @@ #include #include #include +#include =20 #include "../iommu-pages.h" #include "iommu-bits.h" @@ -1485,6 +1486,27 @@ static struct iommu_domain riscv_iommu_identity_doma= in =3D { } }; =20 +static void *riscv_iommu_hw_info(struct device *dev, u32 *length, u32 *typ= e) +{ + struct riscv_iommu_device *iommu =3D dev_to_iommu(dev); + struct iommu_hw_info_riscv_iommu *info; + + if (!iommu) + return ERR_PTR(-ENODEV); + + info =3D kzalloc(sizeof(*info), GFP_KERNEL); + if (!info) + return ERR_PTR(-ENOMEM); + + info->capability =3D iommu->caps; + info->fctl =3D riscv_iommu_readl(iommu, RISCV_IOMMU_REG_FCTL); + + *length =3D sizeof(*info); + *type =3D IOMMU_HW_INFO_TYPE_RISCV_IOMMU; + + return info; +} + static int riscv_iommu_device_domain_type(struct device *dev) { return 0; @@ -1560,6 +1582,7 @@ static void riscv_iommu_release_device(struct device = *dev) static const struct iommu_ops riscv_iommu_ops =3D { .pgsize_bitmap =3D SZ_4K, .of_xlate =3D riscv_iommu_of_xlate, + .hw_info =3D riscv_iommu_hw_info, .identity_domain =3D &riscv_iommu_identity_domain, .blocked_domain =3D &riscv_iommu_blocking_domain, .release_domain =3D &riscv_iommu_blocking_domain, diff --git a/include/uapi/linux/iommufd.h b/include/uapi/linux/iommufd.h index 1dfeaa2e649e..ec9aafd7d373 100644 --- a/include/uapi/linux/iommufd.h +++ b/include/uapi/linux/iommufd.h @@ -475,15 +475,28 @@ struct iommu_hw_info_vtd { __aligned_u64 ecap_reg; }; =20 +/** + * struct iommu_hw_info_riscv_iommu - RISCV IOMMU hardware information + * + * @capability: Value of RISC-V IOMMU capability register + * @fctl: Value of RISC-V IOMMU feature control register + */ +struct iommu_hw_info_riscv_iommu { + __aligned_u64 capability; + __u32 fctl; +}; + /** * enum iommu_hw_info_type - IOMMU Hardware Info Types * @IOMMU_HW_INFO_TYPE_NONE: Used by the drivers that do not report hardwa= re * info * @IOMMU_HW_INFO_TYPE_INTEL_VTD: Intel VT-d iommu info type + * @IOMMU_HW_INFO_TYPE_RISCV_IOMMU: RISC-V iommu info type */ enum iommu_hw_info_type { IOMMU_HW_INFO_TYPE_NONE, IOMMU_HW_INFO_TYPE_INTEL_VTD, + IOMMU_HW_INFO_TYPE_RISCV_IOMMU, }; =20 /** --=20 2.17.1 From nobody Tue Feb 10 15:29:43 2026 Received: from mail-pf1-f176.google.com (mail-pf1-f176.google.com [209.85.210.176]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6B6AD1607B7 for ; Tue, 7 May 2024 14:26:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.176 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715091983; cv=none; b=GhahlMWvzkDakP4RYDkhWOXsVOSI4peRVoMEA4Xx8NKfah/Qa7biJU85yHCF0VYn3vFMxfmAQnTJfLvZRZIUcyuBZPFCJVDN2HWJaBVk6VA/wzvsYxEbXSKMGZDcVaAKpCLtGMgE6zUT7DWg9/mhIERcj0dyVKsdFMeIwpYCLtI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715091983; c=relaxed/simple; bh=bqlipkF/qwmaFsPGQ6jvLZEhD8A7hXZuMOw8nPGleSE=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References; b=Z9OWWC0w6K1iKRMLEdFrYWc+mO8clgmTvcxavioA13U5H0aQ0GgC/W9lWwsWuDcXyuXWQSzT+qfdHIsjf1qSqxoOQfOw/3s22b5OMBTT4Gq9EllPuPXpeAOwxb0GcubIITisf4E1ikuHYx13Z56YH4iKZYaoGFLOjgcKhyDrhIA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=sifive.com; spf=pass smtp.mailfrom=sifive.com; dkim=pass (2048-bit key) header.d=sifive.com header.i=@sifive.com header.b=DAXrv5pR; arc=none smtp.client-ip=209.85.210.176 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=sifive.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=sifive.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=sifive.com header.i=@sifive.com header.b="DAXrv5pR" Received: by mail-pf1-f176.google.com with SMTP id d2e1a72fcca58-6f44dc475f4so2077152b3a.2 for ; Tue, 07 May 2024 07:26:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1715091981; x=1715696781; darn=vger.kernel.org; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=MTOLBzQvBvlY+EmrQy97Trzu0yuGzLHkqXUAWYArG80=; b=DAXrv5pRcA2b+bXiFHrSmBUNtmMHuUnQFaLj/g+4Cl0ajiLQXqfkRXzVsxvVY63MTW PhlJSttl74G4eQhX8PjtXG+oHQMpi1PEmBrgk5ORnK6xWfDYGnb+amWgjTFoncmD9kIa v3AcL7S0mPkG5tYyOwembQGqUICYU1h5Dn/dUVXxrKBwPW7XNoQg02vuXQy/nG86WQq1 D/u4QkWhz7NpW4TiT8628CeKc8C24R3X4ZX56B5716D8Emj0/sshN8yHopUvBeBCVSjB 2R/r5WpjAPnOlu8IB4yC22uPjChDdI6iVy2D0jUDSppdz+FtjnICS2VKRUoMuf8/8iio dneg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1715091981; x=1715696781; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=MTOLBzQvBvlY+EmrQy97Trzu0yuGzLHkqXUAWYArG80=; b=QGXhV975osaYrOKkI24y/R8PpDoFdPFK14Ss53cZ3E13mioNvQdmBw9J6e4yHTGm36 IZ6PrMtBA/NA23ac0Q7BaODSQiorAlU0vBpUD8WPhPxSCdesWxktWJeesUBAFs2g9+yt C37E8UGwWgngIh70zCy4SyL+TtdR5RxfRC5/DRQazLPaGdFpYMl1WEm2qfOOHO39QAyz 961YT/YkZe1fFQ2kolzr1j7JyaBL7W0jlKCuSBggRtaWzqlz98vaCefuuCCCHDiMu/DZ 9jrm8pe1kO3bHqSMzQV3Y1yC83UkPCQGJ30DcNEU7IT9u4VimcNiZVrdBZAqXzDl/cJU lWWA== X-Forwarded-Encrypted: i=1; AJvYcCXg7SNhr5yb7nHl7W0ymDCkqqijVnWP5JfljLPuICO9EZWHHvP9fkcPXOaxzq/adEBp9o91BZQC2hAO011C0XIVEF3h79oZQc6JvgxQ X-Gm-Message-State: AOJu0Yx5rdp5sjVo1Hi7e9gmvSf7O5wF2qi+rch7yigx7em5Q1Uoh1wY 5EsT81wCO/LWf4ycNXW8m0tnxmcOiMAtU4HolJpnExpR4xSPfb4C6EtNL2iavGI= X-Google-Smtp-Source: AGHT+IFrRpCDjD/4eOupdDn8MUYNlbPGsiI9chLNqLmFjLhbFaZkNfyNZ758Eh2Q2HfbLwdvKcrXqw== X-Received: by 2002:a05:6a00:1305:b0:6ed:88e5:53d4 with SMTP id j5-20020a056a00130500b006ed88e553d4mr14682261pfu.8.1715091979690; Tue, 07 May 2024 07:26:19 -0700 (PDT) Received: from hsinchu26.internal.sifive.com (59-124-168-89.hinet-ip.hinet.net. [59.124.168.89]) by smtp.gmail.com with ESMTPSA id i22-20020aa79096000000b006f44bcbe7e3sm7687554pfa.201.2024.05.07.07.26.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 07 May 2024 07:26:19 -0700 (PDT) From: Zong Li To: joro@8bytes.org, will@kernel.org, robin.murphy@arm.com, tjeznach@rivosinc.com, paul.walmsley@sifive.com, palmer@dabbelt.com, aou@eecs.berkeley.edu, jgg@ziepe.ca, kevin.tian@intel.com, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-riscv@lists.infradead.org Cc: Zong Li Subject: [PATCH RFC RESEND 5/6] iommu/riscv: support nested iommu for creating domains owned by userspace Date: Tue, 7 May 2024 22:25:59 +0800 Message-Id: <20240507142600.23844-6-zong.li@sifive.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20240507142600.23844-1-zong.li@sifive.com> References: <20240507142600.23844-1-zong.li@sifive.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" This patch implements .domain_alloc_user operation for creating domains owend by userspace, e.g. through IOMMUFD. Add s2 domain for parent domain for second stage, s1 domain will be the first stage. Don't remove IOMMU private data of dev in blocked domain, because it holds the user data of device, which is used when attaching device into s1 domain. Signed-off-by: Zong Li --- drivers/iommu/riscv/iommu.c | 227 ++++++++++++++++++++++++++++++++++- include/uapi/linux/iommufd.h | 17 +++ 2 files changed, 243 insertions(+), 1 deletion(-) diff --git a/drivers/iommu/riscv/iommu.c b/drivers/iommu/riscv/iommu.c index 072251f6ad85..7eda850df475 100644 --- a/drivers/iommu/riscv/iommu.c +++ b/drivers/iommu/riscv/iommu.c @@ -827,6 +827,7 @@ static int riscv_iommu_iodir_set_mode(struct riscv_iomm= u_device *iommu, =20 /* This struct contains protection domain specific IOMMU driver data. */ struct riscv_iommu_domain { + struct riscv_iommu_domain *s2; struct iommu_domain domain; struct list_head bonds; spinlock_t lock; /* protect bonds list updates. */ @@ -844,6 +845,7 @@ struct riscv_iommu_domain { /* Private IOMMU data for managed devices, dev_iommu_priv_* */ struct riscv_iommu_info { struct riscv_iommu_domain *domain; + struct riscv_iommu_dc dc_user; }; =20 /* Linkage between an iommu_domain and attached devices. */ @@ -1454,7 +1456,6 @@ static int riscv_iommu_attach_blocking_domain(struct = iommu_domain *iommu_domain, =20 riscv_iommu_iodir_update(iommu, dev, RISCV_IOMMU_FSC_BARE, 0, 0); riscv_iommu_bond_unlink(info->domain, dev); - info->domain =3D NULL; =20 return 0; } @@ -1486,6 +1487,229 @@ static struct iommu_domain riscv_iommu_identity_dom= ain =3D { } }; =20 +/** + * Nested IOMMU operations + */ + +static int riscv_iommu_attach_dev_nested(struct iommu_domain *domain, stru= ct device *dev) +{ + struct riscv_iommu_domain *riscv_domain =3D iommu_domain_to_riscv(domain); + struct riscv_iommu_device *iommu =3D dev_to_iommu(dev); + struct riscv_iommu_info *info =3D dev_iommu_priv_get(dev); + + if (riscv_domain->numa_node =3D=3D NUMA_NO_NODE) + riscv_domain->numa_node =3D dev_to_node(iommu->dev); + + riscv_iommu_bond_unlink(info->domain, dev); + + if (riscv_iommu_bond_link(riscv_domain, dev)) + return -ENOMEM; + + riscv_iommu_iotlb_inval(riscv_domain, 0, ULONG_MAX); + + riscv_iommu_iodir_update(iommu, dev, info->dc_user.fsc, info->dc_user.ta, + info->dc_user.iohgatp); + + info->domain =3D riscv_domain; + + return 0; +} + +static void riscv_iommu_domain_free_nested(struct iommu_domain *domain) +{ + struct riscv_iommu_domain *riscv_domain =3D iommu_domain_to_riscv(domain); + + kfree(riscv_domain); +} + +static const struct iommu_domain_ops riscv_iommu_nested_domain_ops =3D { + .attach_dev =3D riscv_iommu_attach_dev_nested, + .free =3D riscv_iommu_domain_free_nested, +}; + +static int +riscv_iommu_get_dc_user(struct device *dev, struct iommu_hwpt_riscv_iommu = *user_arg) +{ + struct iommu_fwspec *fwspec =3D dev_iommu_fwspec_get(dev); + struct riscv_iommu_device *iommu =3D dev_to_iommu(dev); + struct riscv_iommu_info *info =3D dev_iommu_priv_get(dev); + struct riscv_iommu_dc dc; + struct riscv_iommu_fq_record event; + u64 dc_len =3D sizeof(struct riscv_iommu_dc) >> (!(iommu->caps & RISCV_IO= MMU_CAP_MSI_FLAT)); + u64 event_len =3D sizeof(struct riscv_iommu_fq_record); + void __user *event_user =3D NULL; + + for (int i =3D 0; i < fwspec->num_ids; i++) { + event.hdr =3D + FIELD_PREP(RISCV_IOMMU_FQ_HDR_CAUSE, RISCV_IOMMU_FQ_CAUSE_DDT_INVALID) | + FIELD_PREP(RISCV_IOMMU_FQ_HDR_DID, fwspec->ids[i]); + + /* Sanity check DC of stage-1 from user data */ + if (!user_arg->out_event_uptr || user_arg->event_len !=3D event_len) + return -EINVAL; + + event_user =3D u64_to_user_ptr(user_arg->out_event_uptr); + + if (!user_arg->dc_uptr || user_arg->dc_len !=3D dc_len) + return -EINVAL; + + if (copy_from_user(&dc, u64_to_user_ptr(user_arg->dc_uptr), dc_len)) + return -EFAULT; + + if (!(dc.tc & RISCV_IOMMU_DDTE_VALID)) { + dev_dbg(dev, "Invalid DDT from user data\n"); + if (copy_to_user(event_user, &event, event_len)) + return -EFAULT; + } + + if (!dc.fsc || dc.iohgatp) { + dev_dbg(dev, "Wrong page table from user data\n"); + if (copy_to_user(event_user, &event, event_len)) + return -EFAULT; + } + + /* Save DC of stage-1 from user data */ + memcpy(&info->dc_user, + riscv_iommu_get_dc(iommu, fwspec->ids[i]), + sizeof(struct riscv_iommu_dc)); + info->dc_user.fsc =3D dc.fsc; + } + + return 0; +} + +static struct iommu_domain * +riscv_iommu_domain_alloc_nested(struct device *dev, + struct iommu_domain *parent, + const struct iommu_user_data *user_data) +{ + struct riscv_iommu_domain *s2_domain =3D iommu_domain_to_riscv(parent); + struct riscv_iommu_domain *s1_domain; + struct riscv_iommu_device *iommu =3D dev_to_iommu(dev); + struct iommu_hwpt_riscv_iommu arg; + int ret, va_bits; + + if (user_data->type !=3D IOMMU_HWPT_DATA_RISCV_IOMMU) + return ERR_PTR(-EOPNOTSUPP); + + if (parent->type !=3D IOMMU_DOMAIN_UNMANAGED) + return ERR_PTR(-EINVAL); + + ret =3D iommu_copy_struct_from_user(&arg, + user_data, + IOMMU_HWPT_DATA_RISCV_IOMMU, + out_event_uptr); + if (ret) + return ERR_PTR(ret); + + s1_domain =3D kzalloc(sizeof(*s1_domain), GFP_KERNEL); + if (!s1_domain) + return ERR_PTR(-ENOMEM); + + spin_lock_init(&s1_domain->lock); + INIT_LIST_HEAD_RCU(&s1_domain->bonds); + + s1_domain->pscid =3D ida_alloc_range(&riscv_iommu_pscids, 1, + RISCV_IOMMU_MAX_PSCID, GFP_KERNEL); + if (s1_domain->pscid < 0) { + iommu_free_page(s1_domain->pgd_root); + kfree(s1_domain); + return ERR_PTR(-ENOMEM); + } + + /* Get device context of stage-1 from user*/ + ret =3D riscv_iommu_get_dc_user(dev, &arg); + if (ret) { + kfree(s1_domain); + return ERR_PTR(-EINVAL); + } + + if (!iommu) { + va_bits =3D VA_BITS; + } else if (iommu->caps & RISCV_IOMMU_CAP_S_SV57) { + va_bits =3D 57; + } else if (iommu->caps & RISCV_IOMMU_CAP_S_SV48) { + va_bits =3D 48; + } else if (iommu->caps & RISCV_IOMMU_CAP_S_SV39) { + va_bits =3D 39; + } else { + dev_err(dev, "cannot find supported page table mode\n"); + return ERR_PTR(-ENODEV); + } + + /* + * The ops->domain_alloc_user could be directly called by the iommufd cor= e, + * instead of iommu core. So, this function need to set the default value= of + * following data member: + * - domain->pgsize_bitmap + * - domain->geometry + * - domain->type + * - domain->ops + */ + s1_domain->s2 =3D s2_domain; + s1_domain->domain.type =3D IOMMU_DOMAIN_NESTED; + s1_domain->domain.ops =3D &riscv_iommu_nested_domain_ops; + s1_domain->domain.pgsize_bitmap =3D SZ_4K; + s1_domain->domain.geometry.aperture_start =3D 0; + s1_domain->domain.geometry.aperture_end =3D DMA_BIT_MASK(va_bits - 1); + s1_domain->domain.geometry.force_aperture =3D true; + + return &s1_domain->domain; +} + +static struct iommu_domain * +riscv_iommu_domain_alloc_user(struct device *dev, u32 flags, + struct iommu_domain *parent, + const struct iommu_user_data *user_data) +{ + struct iommu_domain *domain; + struct riscv_iommu_domain *riscv_domain; + + /* Allocate stage-1 domain if it has stage-2 parent domain */ + if (parent) + return riscv_iommu_domain_alloc_nested(dev, parent, user_data); + + if (flags & ~((IOMMU_HWPT_ALLOC_NEST_PARENT | IOMMU_HWPT_ALLOC_DIRTY_TRAC= KING))) + return ERR_PTR(-EOPNOTSUPP); + + if (user_data) + return ERR_PTR(-EINVAL); + + /* domain_alloc_user op needs to be fully initialized */ + domain =3D iommu_domain_alloc(dev->bus); + if (!domain) + return ERR_PTR(-ENOMEM); + + /* + * We assume that nest-parent or g-stage only will come here + * TODO: Shadow page table doesn't be supported now. + * We currently can't distinguish g-stage and shadow + * page table here. Shadow page table shouldn't be + * put at stage-2. + */ + riscv_domain =3D iommu_domain_to_riscv(domain); + + /* pgd_root may be allocated in .domain_alloc_paging */ + if (riscv_domain->pgd_root) + iommu_free_page(riscv_domain->pgd_root); + + riscv_domain->pgd_root =3D iommu_alloc_pages_node(riscv_domain->numa_node, + GFP_KERNEL_ACCOUNT, + 2); + if (!riscv_domain->pgd_root) + return ERR_PTR(-ENOMEM); + + riscv_domain->gscid =3D ida_alloc_range(&riscv_iommu_gscids, 1, + RISCV_IOMMU_MAX_GSCID, GFP_KERNEL); + if (riscv_domain->gscid < 0) { + iommu_free_pages(riscv_domain->pgd_root, 2); + kfree(riscv_domain); + return ERR_PTR(-ENOMEM); + } + + return domain; +} + static void *riscv_iommu_hw_info(struct device *dev, u32 *length, u32 *typ= e) { struct riscv_iommu_device *iommu =3D dev_to_iommu(dev); @@ -1587,6 +1811,7 @@ static const struct iommu_ops riscv_iommu_ops =3D { .blocked_domain =3D &riscv_iommu_blocking_domain, .release_domain =3D &riscv_iommu_blocking_domain, .domain_alloc_paging =3D riscv_iommu_alloc_paging_domain, + .domain_alloc_user =3D riscv_iommu_domain_alloc_user, .def_domain_type =3D riscv_iommu_device_domain_type, .device_group =3D riscv_iommu_device_group, .probe_device =3D riscv_iommu_probe_device, diff --git a/include/uapi/linux/iommufd.h b/include/uapi/linux/iommufd.h index ec9aafd7d373..e10b6e236647 100644 --- a/include/uapi/linux/iommufd.h +++ b/include/uapi/linux/iommufd.h @@ -390,14 +390,31 @@ struct iommu_hwpt_vtd_s1 { __u32 __reserved; }; =20 +/** + * struct iommu_hwpt_riscv_iommu - RISCV IOMMU stage-1 device context table + * info (IOMMU_HWPT_TYPE_RISCV_IOMMU) + * @dc_len: Length of device context + * @dc_uptr: User pointer to the address of device context + * @event_len: Length of an event record + * @out_event_uptr: User pointer to the address of event record + */ +struct iommu_hwpt_riscv_iommu { + __aligned_u64 dc_len; + __aligned_u64 dc_uptr; + __aligned_u64 event_len; + __aligned_u64 out_event_uptr; +}; + /** * enum iommu_hwpt_data_type - IOMMU HWPT Data Type * @IOMMU_HWPT_DATA_NONE: no data * @IOMMU_HWPT_DATA_VTD_S1: Intel VT-d stage-1 page table + * @IOMMU_HWPT_DATA_RISCV_IOMMU: RISC-V IOMMU device context table */ enum iommu_hwpt_data_type { IOMMU_HWPT_DATA_NONE, IOMMU_HWPT_DATA_VTD_S1, + IOMMU_HWPT_DATA_RISCV_IOMMU, }; =20 /** --=20 2.17.1 From nobody Tue Feb 10 15:29:43 2026 Received: from mail-pf1-f180.google.com (mail-pf1-f180.google.com [209.85.210.180]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B6AAF161902 for ; Tue, 7 May 2024 14:26:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.180 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715091984; cv=none; b=G8z3Mn17Ty/kTEFmcLRK8E0h1rTRjXmZ3RhInrNu+72Ej41G0viDBMX4tg+KsfguWoBnSHg/4ghaowwnUefDHzRKBbaPU4S/eooL3Wx9nWYGYg+CnN1iHv2rBtElt3zc7HFSrk9Ziij+4ujurPRL0Bu4h0m3xVVOXppLWqElslw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715091984; c=relaxed/simple; bh=p9LzzUVq84gigk4g1jMkvkAiSTYV+AnJqJecyq5AEds=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References; b=aDbFYStaw1++dFcwrrJk1Fwpf3PmWjR0IYC9vVnsjFP3ZrqlCdqAopAy2/3SFnUpaUXsar/pLScAyR6sDZl917osaE64DZ0HS4DC8qKLwe/rZPPGTQo4ZlNGrgd3oXvhl4A4GQ0VqMPi+oyvYyxNmUqC99k0bCiUm4Qqt8LlbnQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=sifive.com; spf=pass smtp.mailfrom=sifive.com; dkim=pass (2048-bit key) header.d=sifive.com header.i=@sifive.com header.b=mjjOo1eE; arc=none smtp.client-ip=209.85.210.180 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=sifive.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=sifive.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=sifive.com header.i=@sifive.com header.b="mjjOo1eE" Received: by mail-pf1-f180.google.com with SMTP id d2e1a72fcca58-6f44b5e7f07so2268981b3a.2 for ; Tue, 07 May 2024 07:26:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1715091982; x=1715696782; darn=vger.kernel.org; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=QsG7aE0HnxZsvtjvosuXtYKV2huEonpRW6TKXMCUJnU=; b=mjjOo1eEpyGke4cQdaudCWJ/o3HM4QjmGS9ZQjwFrFoWtYIO6f/5XEXeMeT7tk/yg/ cUKrSnDcJCp7Ux3juRvMlfsVWtkXRmfs+T2qWXrhu4+E0yFBXOfuQEGHd9qxJA4Z0ZxG PIiF79Txcj03/7LOLpOLxBkeu45iuWhdcqeXJW5QJzVnrm7AHJPXRQhoTwwAIkjlIjUP 6e3jy3niKr8L2L2xIdu8KFH490mDEdTF/4NUPd4HHywquVOUGv35cD96TNuSqPizWiso 0Pt+KqjL2hGcb2e5kh4D3an0avnlrLK7CKQ8MhW1Oa71qOwVjTMIlp3n2T9cl3rqCDOQ WtBg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1715091982; x=1715696782; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=QsG7aE0HnxZsvtjvosuXtYKV2huEonpRW6TKXMCUJnU=; b=epZHJ6Fkk8Rh+pnsJDf94mnv0DG1eHz9Ce/Y6sXYapX8BgmlPDi76hjpKfH1WeB4I3 UcAR7qt407EI9wsfADbvxEEIfA6meWKxRJasziau2akEIRNp0p9kJ6pOWd2KvDBVdZp8 jzPyhLwJAbCjkTkXKWuPMfU7ItdMmWSrjpqqZwLSn7lFfyMWIhK88qGd+w9Wf+4iH8Pj Pcn+Eu1sdJHAnuBx7AwC0GyhppwaK3ipG2Wq+/0ikeWiKJW3np18MJ52h98HMyLe8Nff jq67ZP8ZxdDLHBGgIxki+gKlr8ZeDusZeDg4N9HqYvLlae1IeB01tA1w70B4680zzvAZ ikvA== X-Forwarded-Encrypted: i=1; AJvYcCUpbX/tSQtWKKFJrE7CnTxaquHOkVLvKHbVL0GiRHSYRcTN+H+p3eDBA0gUzn/dODAoMkEkLV79A28AXyHr4rX2v+wOOO8/6qwn8KqH X-Gm-Message-State: AOJu0YwRw70wmae13Nm8SoIinHaO9UsrGLJgWBum97lxrKQxa1O1hDUZ bFPrMtXYCrQewaKgIBFQg3OxHHA2AdVMcHD67nDU/nhIeLIvuMgk9t+BXUcyfJU= X-Google-Smtp-Source: AGHT+IESwO+1TpOehpzE3oOZnwENdjrLViV8PyyzyYTAIHr0YOeHnmv3rn25FWxqq/8kskMF3gIH5A== X-Received: by 2002:a05:6a00:17a7:b0:6ec:e726:b6f5 with SMTP id s39-20020a056a0017a700b006ece726b6f5mr16072960pfg.26.1715091982078; Tue, 07 May 2024 07:26:22 -0700 (PDT) Received: from hsinchu26.internal.sifive.com (59-124-168-89.hinet-ip.hinet.net. [59.124.168.89]) by smtp.gmail.com with ESMTPSA id i22-20020aa79096000000b006f44bcbe7e3sm7687554pfa.201.2024.05.07.07.26.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 07 May 2024 07:26:21 -0700 (PDT) From: Zong Li To: joro@8bytes.org, will@kernel.org, robin.murphy@arm.com, tjeznach@rivosinc.com, paul.walmsley@sifive.com, palmer@dabbelt.com, aou@eecs.berkeley.edu, jgg@ziepe.ca, kevin.tian@intel.com, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-riscv@lists.infradead.org Cc: Zong Li Subject: [PATCH RFC RESEND 6/6] iommu/riscv: support nested iommu for flushing cache Date: Tue, 7 May 2024 22:26:00 +0800 Message-Id: <20240507142600.23844-7-zong.li@sifive.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20240507142600.23844-1-zong.li@sifive.com> References: <20240507142600.23844-1-zong.li@sifive.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" This patch implements cache_invalidate_user operation for the userspace to flush the hardware caches for a nested domain through iommufd. Signed-off-by: Zong Li --- drivers/iommu/riscv/iommu.c | 91 ++++++++++++++++++++++++++++++++++++ include/uapi/linux/iommufd.h | 9 ++++ 2 files changed, 100 insertions(+) diff --git a/drivers/iommu/riscv/iommu.c b/drivers/iommu/riscv/iommu.c index 7eda850df475..4dd58fe2242d 100644 --- a/drivers/iommu/riscv/iommu.c +++ b/drivers/iommu/riscv/iommu.c @@ -1522,9 +1522,100 @@ static void riscv_iommu_domain_free_nested(struct i= ommu_domain *domain) kfree(riscv_domain); } =20 +static int riscv_iommu_fix_user_cmd(struct riscv_iommu_command *cmd, + unsigned int pscid, unsigned int gscid) +{ + u32 opcode =3D FIELD_GET(RISCV_IOMMU_CMD_OPCODE, cmd->dword0); + + switch (opcode) { + case RISCV_IOMMU_CMD_IOTINVAL_OPCODE: + u32 func =3D FIELD_GET(RISCV_IOMMU_CMD_FUNC, cmd->dword0); + + if (func !=3D RISCV_IOMMU_CMD_IOTINVAL_FUNC_GVMA && + func !=3D RISCV_IOMMU_CMD_IOTINVAL_FUNC_VMA) { + pr_warn("The IOTINVAL function: 0x%x is not supported\n", + func); + return -EOPNOTSUPP; + } + + if (func =3D=3D RISCV_IOMMU_CMD_IOTINVAL_FUNC_GVMA) { + cmd->dword0 &=3D ~RISCV_IOMMU_CMD_FUNC; + cmd->dword0 |=3D FIELD_PREP(RISCV_IOMMU_CMD_FUNC, + RISCV_IOMMU_CMD_IOTINVAL_FUNC_VMA); + } + + cmd->dword0 &=3D ~(RISCV_IOMMU_CMD_IOTINVAL_PSCID | + RISCV_IOMMU_CMD_IOTINVAL_GSCID); + riscv_iommu_cmd_inval_set_pscid(cmd, pscid); + riscv_iommu_cmd_inval_set_gscid(cmd, gscid); + break; + case RISCV_IOMMU_CMD_IODIR_OPCODE: + /* + * Ensure the device ID is right. We expect that VMM has + * transferred the device ID to host's from guest's. + */ + break; + default: + pr_warn("The user command: 0x%x is not supported\n", opcode); + return -EOPNOTSUPP; + } + + return 0; +} + +static int riscv_iommu_cache_invalidate_user(struct iommu_domain *domain, + struct iommu_user_data_array *array) +{ + struct riscv_iommu_domain *riscv_domain =3D iommu_domain_to_riscv(domain); + struct riscv_iommu_device *iommu; + struct riscv_iommu_bond *bond; + struct riscv_iommu_command cmd; + struct iommu_hwpt_riscv_iommu_invalidate inv_info; + int ret, index; + + if (!riscv_domain) + return -EINVAL; + + /* Assume attached devices in the domain go through the same IOMMU device= */ + spin_lock(&riscv_domain->lock); + list_for_each_entry_rcu(bond, &riscv_domain->bonds, list) { + if (bond->dev) { + iommu =3D dev_to_iommu(bond->dev); + break; + } + } + spin_unlock(&riscv_domain->lock); + + if (!iommu) + return -EINVAL; + + for (index =3D 0; index < array->entry_num; index++) { + ret =3D iommu_copy_struct_from_user_array(&inv_info, array, + IOMMU_HWPT_DATA_RISCV_IOMMU, + index, cmd); + if (ret) + break; + + ret =3D riscv_iommu_fix_user_cmd((struct riscv_iommu_command *)inv_info.= cmd, + riscv_domain->pscid, + riscv_domain->s2->gscid); + if (ret =3D=3D -EOPNOTSUPP) + continue; + + riscv_iommu_cmd_send(iommu, (struct riscv_iommu_command *)inv_info.cmd, = 0); + riscv_iommu_cmd_iofence(&cmd); + riscv_iommu_cmd_send(iommu, &cmd, RISCV_IOMMU_QUEUE_TIMEOUT); + } + + array->entry_num =3D index; + + return ret; +} + static const struct iommu_domain_ops riscv_iommu_nested_domain_ops =3D { .attach_dev =3D riscv_iommu_attach_dev_nested, .free =3D riscv_iommu_domain_free_nested, + .cache_invalidate_user =3D riscv_iommu_cache_invalidate_user, }; =20 static int diff --git a/include/uapi/linux/iommufd.h b/include/uapi/linux/iommufd.h index e10b6e236647..d93a8f11813d 100644 --- a/include/uapi/linux/iommufd.h +++ b/include/uapi/linux/iommufd.h @@ -689,6 +689,15 @@ struct iommu_hwpt_vtd_s1_invalidate { __u32 __reserved; }; =20 +/** + * struct iommu_hwpt_riscv_iommu_invalidate - RISCV IOMMU cache invalidati= on + * (IOMMU_HWPT_TYPE_RISCV_IOMMU) + * @cmd: An array holds a command for cache invalidation + */ +struct iommu_hwpt_riscv_iommu_invalidate { + __aligned_u64 cmd[2]; +}; + /** * struct iommu_hwpt_invalidate - ioctl(IOMMU_HWPT_INVALIDATE) * @size: sizeof(struct iommu_hwpt_invalidate) --=20 2.17.1