From nobody Sun Nov 24 04:01:41 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org ARC-Seal: i=1; a=rsa-sha256; t=1727831168; cv=none; d=zohomail.com; s=zohoarc; b=NYQD4GSMIzvBkjLXnfAvYgxMzMXoZxYMiIxXpdttoMNHH5/UnGJf7/Ssil+A0Mr8d+gSzpcCfqBdlWZRBtADwbyisII65GuJUevZwOLtAO8Z9h8mFKhLTSUErJ/Ih7cQhC2/gow6CuCDm95nSghPK9yjuvRl8cng6a2WZeWqo/Y= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1727831168; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=WylGfdlkZtWbt5bABNox0uybEHY565+NdTjNISBYimI=; b=MG29f+iCa9ZD22uBRIMtg3tQUwR/OHjvTRvbeJro8wP4SIKx+Zy/PyzLVrvJFeGDrtv//qoc1oNM+6wpj7nCI41dTluI8oKoROzPxfzHVPm79nz0qJjscDcPJuUGBrOGn1dTLlqpSCNV4ft7abfOPiGKonmiGp7b5fXRPTfcHe4= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1727831168932771.8140235106679; Tue, 1 Oct 2024 18:06:08 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1svnmv-00071U-It; Tue, 01 Oct 2024 21:04:23 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1svnmU-0006wr-Qz for qemu-devel@nongnu.org; Tue, 01 Oct 2024 21:03:57 -0400 Received: from mail-pl1-x630.google.com ([2607:f8b0:4864:20::630]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1svnmS-0002mz-Qb for qemu-devel@nongnu.org; Tue, 01 Oct 2024 21:03:54 -0400 Received: by mail-pl1-x630.google.com with SMTP id d9443c01a7336-20bcae5e482so1279705ad.0 for ; Tue, 01 Oct 2024 18:03:51 -0700 (PDT) Received: from grind.. (200-206-229-93.dsl.telesp.net.br. [200.206.229.93]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-20b37d63971sm76061295ad.4.2024.10.01.18.03.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 01 Oct 2024 18:03:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ventanamicro.com; s=google; t=1727831030; x=1728435830; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=WylGfdlkZtWbt5bABNox0uybEHY565+NdTjNISBYimI=; b=hXdrZZCleyggekrxTD46eErUHmzA3yRrOPrSImfxRMYNzGr1KOb0j+mCKsX2sp3VWV QTk5tosyp35W+dHX0KyWuIUhGeGBxEj+ZqTP5kQp1u1Kz23pQgU7cVBVTfn4lTSaJYa4 aFMvrMCW4xWDbEVm904DHX5+0B2q0lIygZCgbmN7lMMf0VoKxBoYLqIVnkr22Ngc+Ccq lsdWFRtCqL6fjZmC+778H+XCkUZMt4OwanI84OE8BxImtl9suMPYt+QotcpvoHZVDCWO IdncYy/cPFN9tORcVYpRFStbcnpbI7+z3H3qCIP/jDqzCK0h1K0baFxLFOOTDZYX4eu2 4DRw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1727831030; x=1728435830; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=WylGfdlkZtWbt5bABNox0uybEHY565+NdTjNISBYimI=; b=J0O4o37nxTf0yQ0xjD0gQpT7HXSrvNZulYSSRPVCE+rK3q9oG4wGSCSxHtUvL2O/+o z3gG6vbhIm8E0AIV+k0oN/FR8YczVf4+y0znOxer3hB/pa67iCk3k+oyKbbzcwW3fZ3s SKr4dTf7/a8a0bUnT7k1XBQtjkPWSZ73tERlazOIz/vRdX3KoYFWu1CTmgPrLQe89Pn2 3gwi/BBCwiqjTvj2Z7Bjrkbgfhwa58QhlyCMWzduf6byGBHxTVvQe2eJN4A4JSgZiLYA RgpDQAZ3eFVI1Rq66iDKG30JggW5qpWDAtKXvGO6NLanOiavHjIzev9ERQ6tg28ETL26 v1VA== X-Gm-Message-State: AOJu0YzjZX/pUkF8ZbEOnkDtNSNdlq4EpugOnthG5FyDQVhIR6JFRJg7 B6dFXEoLoHkoSXvskgCEyzRHXCOWvYYtjhdVJgZINaS354mtCuQ+2gq1+PzwoEtNgoGJizMcTYz j X-Google-Smtp-Source: AGHT+IFHSvP4VyEjNWKx5D6f9poEnt14JLqpobXEp5p/PpuDK7eVmUbloSsWlOrnwQkL/3WokR257Q== X-Received: by 2002:a17:902:d50c:b0:205:4721:19c with SMTP id d9443c01a7336-20bc5a5d0f6mr21479555ad.37.1727831030365; Tue, 01 Oct 2024 18:03:50 -0700 (PDT) From: Daniel Henrique Barboza To: qemu-devel@nongnu.org Cc: qemu-riscv@nongnu.org, alistair.francis@wdc.com, bmeng@tinylab.org, liwei1518@gmail.com, zhiwei_liu@linux.alibaba.com, palmer@rivosinc.com, peter.maydell@linaro.org, Tomasz Jeznach , Daniel Henrique Barboza , Frank Chang Subject: [PATCH v8 08/12] hw/riscv/riscv-iommu: add Address Translation Cache (IOATC) Date: Tue, 1 Oct 2024 22:03:03 -0300 Message-ID: <20241002010314.1928515-9-dbarboza@ventanamicro.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241002010314.1928515-1-dbarboza@ventanamicro.com> References: <20241002010314.1928515-1-dbarboza@ventanamicro.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=2607:f8b0:4864:20::630; envelope-from=dbarboza@ventanamicro.com; helo=mail-pl1-x630.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @ventanamicro.com) X-ZM-MESSAGEID: 1727831170812116600 Content-Type: text/plain; charset="utf-8" From: Tomasz Jeznach The RISC-V IOMMU spec predicts that the IOMMU can use translation caches to hold entries from the DDT. This includes implementation for all cache commands that are marked as 'not implemented'. There are some artifacts included in the cache that predicts s-stage and g-stage elements, although we don't support it yet. We'll introduce them next. Signed-off-by: Tomasz Jeznach Signed-off-by: Daniel Henrique Barboza Reviewed-by: Frank Chang Acked-by: Alistair Francis --- hw/riscv/riscv-iommu.c | 204 ++++++++++++++++++++++++++++++++++++++++- hw/riscv/riscv-iommu.h | 3 + 2 files changed, 203 insertions(+), 4 deletions(-) diff --git a/hw/riscv/riscv-iommu.c b/hw/riscv/riscv-iommu.c index 854b4b8a81..e3597af67c 100644 --- a/hw/riscv/riscv-iommu.c +++ b/hw/riscv/riscv-iommu.c @@ -64,6 +64,15 @@ struct RISCVIOMMUContext { uint64_t msiptp; /* MSI redirection page table pointer */ }; =20 +/* Address translation cache entry */ +struct RISCVIOMMUEntry { + uint64_t iova:44; /* IOVA Page Number */ + uint64_t pscid:20; /* Process Soft-Context identifier */ + uint64_t phys:44; /* Physical Page Number */ + uint64_t gscid:16; /* Guest Soft-Context identifier */ + uint64_t perm:2; /* IOMMU_RW flags */ +}; + /* IOMMU index for transactions without process_id specified. */ #define RISCV_IOMMU_NOPROCID 0 =20 @@ -1151,13 +1160,132 @@ static AddressSpace *riscv_iommu_space(RISCVIOMMUS= tate *s, uint32_t devid) return &as->iova_as; } =20 +/* Translation Object cache support */ +static gboolean riscv_iommu_iot_equal(gconstpointer v1, gconstpointer v2) +{ + RISCVIOMMUEntry *t1 =3D (RISCVIOMMUEntry *) v1; + RISCVIOMMUEntry *t2 =3D (RISCVIOMMUEntry *) v2; + return t1->gscid =3D=3D t2->gscid && t1->pscid =3D=3D t2->pscid && + t1->iova =3D=3D t2->iova; +} + +static guint riscv_iommu_iot_hash(gconstpointer v) +{ + RISCVIOMMUEntry *t =3D (RISCVIOMMUEntry *) v; + return (guint)t->iova; +} + +/* GV: 1 PSCV: 1 AV: 1 */ +static void riscv_iommu_iot_inval_pscid_iova(gpointer key, gpointer value, + gpointer data) +{ + RISCVIOMMUEntry *iot =3D (RISCVIOMMUEntry *) value; + RISCVIOMMUEntry *arg =3D (RISCVIOMMUEntry *) data; + if (iot->gscid =3D=3D arg->gscid && + iot->pscid =3D=3D arg->pscid && + iot->iova =3D=3D arg->iova) { + iot->perm =3D IOMMU_NONE; + } +} + +/* GV: 1 PSCV: 1 AV: 0 */ +static void riscv_iommu_iot_inval_pscid(gpointer key, gpointer value, + gpointer data) +{ + RISCVIOMMUEntry *iot =3D (RISCVIOMMUEntry *) value; + RISCVIOMMUEntry *arg =3D (RISCVIOMMUEntry *) data; + if (iot->gscid =3D=3D arg->gscid && + iot->pscid =3D=3D arg->pscid) { + iot->perm =3D IOMMU_NONE; + } +} + +/* GV: 1 GVMA: 1 */ +static void riscv_iommu_iot_inval_gscid_gpa(gpointer key, gpointer value, + gpointer data) +{ + RISCVIOMMUEntry *iot =3D (RISCVIOMMUEntry *) value; + RISCVIOMMUEntry *arg =3D (RISCVIOMMUEntry *) data; + if (iot->gscid =3D=3D arg->gscid) { + /* simplified cache, no GPA matching */ + iot->perm =3D IOMMU_NONE; + } +} + +/* GV: 1 GVMA: 0 */ +static void riscv_iommu_iot_inval_gscid(gpointer key, gpointer value, + gpointer data) +{ + RISCVIOMMUEntry *iot =3D (RISCVIOMMUEntry *) value; + RISCVIOMMUEntry *arg =3D (RISCVIOMMUEntry *) data; + if (iot->gscid =3D=3D arg->gscid) { + iot->perm =3D IOMMU_NONE; + } +} + +/* GV: 0 */ +static void riscv_iommu_iot_inval_all(gpointer key, gpointer value, + gpointer data) +{ + RISCVIOMMUEntry *iot =3D (RISCVIOMMUEntry *) value; + iot->perm =3D IOMMU_NONE; +} + +/* caller should keep ref-count for iot_cache object */ +static RISCVIOMMUEntry *riscv_iommu_iot_lookup(RISCVIOMMUContext *ctx, + GHashTable *iot_cache, hwaddr iova) +{ + RISCVIOMMUEntry key =3D { + .gscid =3D get_field(ctx->gatp, RISCV_IOMMU_DC_IOHGATP_GSCID), + .pscid =3D get_field(ctx->ta, RISCV_IOMMU_DC_TA_PSCID), + .iova =3D PPN_DOWN(iova), + }; + return g_hash_table_lookup(iot_cache, &key); +} + +/* caller should keep ref-count for iot_cache object */ +static void riscv_iommu_iot_update(RISCVIOMMUState *s, + GHashTable *iot_cache, RISCVIOMMUEntry *iot) +{ + if (!s->iot_limit) { + return; + } + + if (g_hash_table_size(s->iot_cache) >=3D s->iot_limit) { + iot_cache =3D g_hash_table_new_full(riscv_iommu_iot_hash, + riscv_iommu_iot_equal, + g_free, NULL); + g_hash_table_unref(qatomic_xchg(&s->iot_cache, iot_cache)); + } + g_hash_table_add(iot_cache, iot); +} + +static void riscv_iommu_iot_inval(RISCVIOMMUState *s, GHFunc func, + uint32_t gscid, uint32_t pscid, hwaddr iova) +{ + GHashTable *iot_cache; + RISCVIOMMUEntry key =3D { + .gscid =3D gscid, + .pscid =3D pscid, + .iova =3D PPN_DOWN(iova), + }; + + iot_cache =3D g_hash_table_ref(s->iot_cache); + g_hash_table_foreach(iot_cache, func, &key); + g_hash_table_unref(iot_cache); +} + static int riscv_iommu_translate(RISCVIOMMUState *s, RISCVIOMMUContext *ct= x, - IOMMUTLBEntry *iotlb) + IOMMUTLBEntry *iotlb, bool enable_cache) { + RISCVIOMMUEntry *iot; + IOMMUAccessFlags perm; bool enable_pid; bool enable_pri; + GHashTable *iot_cache; int fault; =20 + iot_cache =3D g_hash_table_ref(s->iot_cache); /* * TC[32] is reserved for custom extensions, used here to temporarily * enable automatic page-request generation for ATS queries. @@ -1165,9 +1293,43 @@ static int riscv_iommu_translate(RISCVIOMMUState *s,= RISCVIOMMUContext *ctx, enable_pri =3D (iotlb->perm =3D=3D IOMMU_NONE) && (ctx->tc & BIT_ULL(3= 2)); enable_pid =3D (ctx->tc & RISCV_IOMMU_DC_TC_PDTV); =20 + iot =3D riscv_iommu_iot_lookup(ctx, iot_cache, iotlb->iova); + perm =3D iot ? iot->perm : IOMMU_NONE; + if (perm !=3D IOMMU_NONE) { + iotlb->translated_addr =3D PPN_PHYS(iot->phys); + iotlb->addr_mask =3D ~TARGET_PAGE_MASK; + iotlb->perm =3D perm; + fault =3D 0; + goto done; + } + /* Translate using device directory / page table information. */ fault =3D riscv_iommu_spa_fetch(s, ctx, iotlb); =20 + if (!fault && iotlb->target_as =3D=3D &s->trap_as) { + /* Do not cache trapped MSI translations */ + goto done; + } + + /* + * We made an implementation choice to not cache identity-mapped + * translations, as allowed by the specification, to avoid + * translation cache evictions for other devices sharing the + * IOMMU hardware model. + */ + if (!fault && iotlb->translated_addr !=3D iotlb->iova && enable_cache)= { + iot =3D g_new0(RISCVIOMMUEntry, 1); + iot->iova =3D PPN_DOWN(iotlb->iova); + iot->phys =3D PPN_DOWN(iotlb->translated_addr); + iot->gscid =3D get_field(ctx->gatp, RISCV_IOMMU_DC_IOHGATP_GSCID); + iot->pscid =3D get_field(ctx->ta, RISCV_IOMMU_DC_TA_PSCID); + iot->perm =3D iotlb->perm; + riscv_iommu_iot_update(s, iot_cache, iot); + } + +done: + g_hash_table_unref(iot_cache); + if (enable_pri && fault) { struct riscv_iommu_pq_record pr =3D {0}; if (enable_pid) { @@ -1307,13 +1469,40 @@ static void riscv_iommu_process_cq_tail(RISCVIOMMUS= tate *s) if (cmd.dword0 & RISCV_IOMMU_CMD_IOTINVAL_PSCV) { /* illegal command arguments IOTINVAL.GVMA & PSCV =3D=3D 1= */ goto cmd_ill; + } else if (!(cmd.dword0 & RISCV_IOMMU_CMD_IOTINVAL_GV)) { + /* invalidate all cache mappings */ + func =3D riscv_iommu_iot_inval_all; + } else if (!(cmd.dword0 & RISCV_IOMMU_CMD_IOTINVAL_AV)) { + /* invalidate cache matching GSCID */ + func =3D riscv_iommu_iot_inval_gscid; + } else { + /* invalidate cache matching GSCID and ADDR (GPA) */ + func =3D riscv_iommu_iot_inval_gscid_gpa; } - /* translation cache not implemented yet */ + riscv_iommu_iot_inval(s, func, + get_field(cmd.dword0, RISCV_IOMMU_CMD_IOTINVAL_GSCID), 0, + cmd.dword1 & TARGET_PAGE_MASK); break; =20 case RISCV_IOMMU_CMD(RISCV_IOMMU_CMD_IOTINVAL_FUNC_VMA, RISCV_IOMMU_CMD_IOTINVAL_OPCODE): - /* translation cache not implemented yet */ + if (!(cmd.dword0 & RISCV_IOMMU_CMD_IOTINVAL_GV)) { + /* invalidate all cache mappings, simplified model */ + func =3D riscv_iommu_iot_inval_all; + } else if (!(cmd.dword0 & RISCV_IOMMU_CMD_IOTINVAL_PSCV)) { + /* invalidate cache matching GSCID, simplified model */ + func =3D riscv_iommu_iot_inval_gscid; + } else if (!(cmd.dword0 & RISCV_IOMMU_CMD_IOTINVAL_AV)) { + /* invalidate cache matching GSCID and PSCID */ + func =3D riscv_iommu_iot_inval_pscid; + } else { + /* invalidate cache matching GSCID and PSCID and ADDR (IOV= A) */ + func =3D riscv_iommu_iot_inval_pscid_iova; + } + riscv_iommu_iot_inval(s, func, + get_field(cmd.dword0, RISCV_IOMMU_CMD_IOTINVAL_GSCID), + get_field(cmd.dword0, RISCV_IOMMU_CMD_IOTINVAL_PSCID), + cmd.dword1 & TARGET_PAGE_MASK); break; =20 case RISCV_IOMMU_CMD(RISCV_IOMMU_CMD_IODIR_FUNC_INVAL_DDT, @@ -1828,6 +2017,10 @@ static void riscv_iommu_realize(DeviceState *dev, Er= ror **errp) riscv_iommu_ctx_equal, g_free, NULL); =20 + s->iot_cache =3D g_hash_table_new_full(riscv_iommu_iot_hash, + riscv_iommu_iot_equal, + g_free, NULL); + s->iommus.le_next =3D NULL; s->iommus.le_prev =3D NULL; QLIST_INIT(&s->spaces); @@ -1837,6 +2030,7 @@ static void riscv_iommu_unrealize(DeviceState *dev) { RISCVIOMMUState *s =3D RISCV_IOMMU(dev); =20 + g_hash_table_unref(s->iot_cache); g_hash_table_unref(s->ctx_cache); } =20 @@ -1844,6 +2038,8 @@ static Property riscv_iommu_properties[] =3D { DEFINE_PROP_UINT32("version", RISCVIOMMUState, version, RISCV_IOMMU_SPEC_DOT_VER), DEFINE_PROP_UINT32("bus", RISCVIOMMUState, bus, 0x0), + DEFINE_PROP_UINT32("ioatc-limit", RISCVIOMMUState, iot_limit, + LIMIT_CACHE_IOT), DEFINE_PROP_BOOL("intremap", RISCVIOMMUState, enable_msi, TRUE), DEFINE_PROP_BOOL("off", RISCVIOMMUState, enable_off, TRUE), DEFINE_PROP_BOOL("s-stage", RISCVIOMMUState, enable_s_stage, TRUE), @@ -1898,7 +2094,7 @@ static IOMMUTLBEntry riscv_iommu_memory_region_transl= ate( /* Translation disabled or invalid. */ iotlb.addr_mask =3D 0; iotlb.perm =3D IOMMU_NONE; - } else if (riscv_iommu_translate(as->iommu, ctx, &iotlb)) { + } else if (riscv_iommu_translate(as->iommu, ctx, &iotlb, true)) { /* Translation disabled or fault reported. */ iotlb.addr_mask =3D 0; iotlb.perm =3D IOMMU_NONE; diff --git a/hw/riscv/riscv-iommu.h b/hw/riscv/riscv-iommu.h index af3fcafc19..9f15f3b27f 100644 --- a/hw/riscv/riscv-iommu.h +++ b/hw/riscv/riscv-iommu.h @@ -69,6 +69,9 @@ struct RISCVIOMMUState { =20 GHashTable *ctx_cache; /* Device translation Context Cache */ =20 + GHashTable *iot_cache; /* IO Translated Address Cache */ + unsigned iot_limit; /* IO Translation Cache size limit */ + /* MMIO Hardware Interface */ MemoryRegion regs_mr; uint8_t *regs_rw; /* register state (user write) */ --=20 2.45.2