From nobody Fri Dec 19 21:17:24 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B1516C83F37 for ; Thu, 31 Aug 2023 17:46:41 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1346961AbjHaRqm (ORCPT ); Thu, 31 Aug 2023 13:46:42 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48528 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233316AbjHaRqk (ORCPT ); Thu, 31 Aug 2023 13:46:40 -0400 Received: from mail-yb1-xb4a.google.com (mail-yb1-xb4a.google.com [IPv6:2607:f8b0:4864:20::b4a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A34FF1704 for ; Thu, 31 Aug 2023 10:46:14 -0700 (PDT) Received: by mail-yb1-xb4a.google.com with SMTP id 3f1490d57ef6-d7493303b5cso940586276.2 for ; Thu, 31 Aug 2023 10:46:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1693503974; x=1694108774; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=i7VjOD8paVBTT/i2kkoQBwaJlSloOfoXnTHQ1Eza9rY=; b=caytXUIXpvgER0McdFHicRzq9EIoWKxXrt52BsiSo2OOOcthCVCCD1DH0+Hv17v+ez ZQ+omL5eCQ33ZmzgHNdnp+Qo7F5EmNUy6dTrseLgUYAi1nzm912f8QxJ9dxA5WhUbdq+ dZHn8Qz6U8vLbX/gx846ALWeWeM9AfJuEtGtnJmK/0jDy7cSydUfVzpuF4XkMGv89EQt 9CkIZZccepTXQHrcuncGYXq9guaEfTKLgJcnkaBKrJvMJv7LCvPULspD9Dlh5nuC97nQ FBwkylKiIWLumtAewiHH0x9/QeE30SWoaWL84mk3hL6pu3Iq1bmKbK57LR2O+b7zYvWB I4yA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1693503974; x=1694108774; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=i7VjOD8paVBTT/i2kkoQBwaJlSloOfoXnTHQ1Eza9rY=; b=LzUk1SQOezdofj63ENApDSWgT/jUffGn4AfU5UmcqXTKXqjCBQi9f+7yAI7T8QzlDf vR19f1MaCxiR6dGcW/fu0nM77yjh4pLwJ/eetpqg9w6czFVUa6gCdlNZ2W7z445/TMhu w+55La40h8eXi0N+Uy8lI9ugVjB+CRIN47p2qrqwXUZO26kDzBaWXaUhxBHEu7ZaK/rd vPsuV67LvX//ZxfxNkHqyJ3y+38wNGSHmfuMC10oCje9puRMD14VU4gSGDuFNJgc7EME p7BzibctVv1dqxmnHoQ+I2LXQxzXisyWnyhmjtyilkQ6H/I5+JHgZDXSLcRJ1luJ4mCr RauA== X-Gm-Message-State: AOJu0YyryA/Fj18CN2I6b8vklgqSSnFiwTJg8FTUuTwXbdP0XnDXZdxz TfYktzEfmCLY2fUDIXq6lW75/qTcsxsM X-Google-Smtp-Source: AGHT+IHRa4lOvgolrcThFwuiSmnWSM47tU5dtwzo9XjBsvC5OMF90W0tNaj+15w7MHzP+4BubJnyK4n2OPmz X-Received: from mshavit.ntc.corp.google.com ([2401:fa00:95:20c:1a0a:7338:4a5a:5f83]) (user=mshavit job=sendgmr) by 2002:a25:ba90:0:b0:d72:8661:ee29 with SMTP id s16-20020a25ba90000000b00d728661ee29mr9278ybg.2.1693503973962; Thu, 31 Aug 2023 10:46:13 -0700 (PDT) Date: Fri, 1 Sep 2023 01:44:35 +0800 In-Reply-To: <20230831174536.103472-1-mshavit@google.com> Mime-Version: 1.0 References: <20230831174536.103472-1-mshavit@google.com> X-Mailer: git-send-email 2.42.0.283.g2d96d420d3-goog Message-ID: <20230901014413.v7.6.Ice063dcf87d1b777a72e008d9e3406d2bcf6d876@changeid> Subject: [PATCH v7 6/9] iommu/arm-smmu-v3: Move CD table to arm_smmu_master From: Michael Shavit To: iommu@lists.linux.dev, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org Cc: will@kernel.org, robin.murphy@arm.com, jean-philippe@linaro.org, jgg@nvidia.com, nicolinc@nvidia.com, Michael Shavit , Jason Gunthorpe , Joerg Roedel , Kevin Tian , "Kirill A. Shutemov" , Lu Baolu , Mark Brown , Yicong Yang Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" With this change, each master will now own its own CD table instead of sharing one with other masters attached to the same domain. Attaching a stage 1 domain installs CD entries into the master's CD table. SVA writes its CD entries into each master's CD table if the domain is shared across masters. Also add the device to the devices list before writing the CD to the table so that SVA will know that the CD needs to be re-written to this device's CD table as well if it decides to update the CD's ASID concurrently with this function. Tested-by: Nicolin Chen Reviewed-by: Jason Gunthorpe Signed-off-by: Michael Shavit --- Changes in v7: - Update commit message to be more clear about locking purpose. - Removed redundant newline Changes in v6: - Grab the asid lock while writing the RID CD to prevent a race with SVA. - Add the device to the devices list before writing the CD to the table and installing the CD table. - Undo arm_smmu_finalise_s1 rename - Minor comment fix - Consistently check cdtab pointer instead of cdtab_dma Changes in v5: - Clear the 0th CD entry when the domain is detached. Not clearing it caused a bug in arm_smmu_write_ctx_desc which doesn't expect the entry to already be set. Changes in v4: - Added comment about the cd_table's dependency on the iommu core's group mutex. - Narrowed the range of code for which the domain's init_mutex is held on attach since it now only protects the arm_smmu_domain_finalise call. Changes in v2: - Allocate CD table when it's first needed instead of on probe. Changes in v1: - The master's CD table allocation was previously split to a different commit. This change now atomically allocates the new CD table, uses it, and removes the old one. drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 100 +++++++++++--------- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 7 +- 2 files changed, 58 insertions(+), 49 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/ar= m/arm-smmu-v3/arm-smmu-v3.c index 69b9bb5c7f773..bcc94a67640d6 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -1025,7 +1025,7 @@ static __le64 *arm_smmu_get_cd_ptr(struct arm_smmu_ma= ster *master, u32 ssid) unsigned int idx; struct arm_smmu_l1_ctx_desc *l1_desc; struct arm_smmu_device *smmu =3D master->smmu; - struct arm_smmu_ctx_desc_cfg *cdcfg =3D &master->domain->cd_table; + struct arm_smmu_ctx_desc_cfg *cdcfg =3D &master->cd_table; =20 if (cdcfg->s1fmt =3D=3D STRTAB_STE_0_S1FMT_LINEAR) return cdcfg->cdtab + ssid * CTXDESC_CD_DWORDS; @@ -1062,7 +1062,7 @@ int arm_smmu_write_ctx_desc(struct arm_smmu_master *m= aster, int ssid, u64 val; bool cd_live; __le64 *cdptr; - struct arm_smmu_ctx_desc_cfg *cd_table =3D &master->domain->cd_table; + struct arm_smmu_ctx_desc_cfg *cd_table =3D &master->cd_table; =20 if (WARN_ON(ssid >=3D (1 << cd_table->s1cdmax))) return -E2BIG; @@ -1125,14 +1125,13 @@ int arm_smmu_write_ctx_desc(struct arm_smmu_master = *master, int ssid, return 0; } =20 -static int arm_smmu_alloc_cd_tables(struct arm_smmu_domain *smmu_domain, - struct arm_smmu_master *master) +static int arm_smmu_alloc_cd_tables(struct arm_smmu_master *master) { int ret; size_t l1size; size_t max_contexts; struct arm_smmu_device *smmu =3D master->smmu; - struct arm_smmu_ctx_desc_cfg *cdcfg =3D &smmu_domain->cd_table; + struct arm_smmu_ctx_desc_cfg *cdcfg =3D &master->cd_table; =20 cdcfg->stall_enabled =3D master->stall_enabled; cdcfg->s1cdmax =3D master->ssid_bits; @@ -1176,12 +1175,12 @@ static int arm_smmu_alloc_cd_tables(struct arm_smmu= _domain *smmu_domain, return ret; } =20 -static void arm_smmu_free_cd_tables(struct arm_smmu_domain *smmu_domain) +static void arm_smmu_free_cd_tables(struct arm_smmu_master *master) { int i; size_t size, l1size; - struct arm_smmu_device *smmu =3D smmu_domain->smmu; - struct arm_smmu_ctx_desc_cfg *cdcfg =3D &smmu_domain->cd_table; + struct arm_smmu_device *smmu =3D master->smmu; + struct arm_smmu_ctx_desc_cfg *cdcfg =3D &master->cd_table; =20 if (cdcfg->l1_desc) { size =3D CTXDESC_L2_ENTRIES * (CTXDESC_CD_DWORDS << 3); @@ -1289,7 +1288,7 @@ static void arm_smmu_write_strtab_ent(struct arm_smmu= _master *master, u32 sid, if (smmu_domain) { switch (smmu_domain->stage) { case ARM_SMMU_DOMAIN_S1: - cd_table =3D &smmu_domain->cd_table; + cd_table =3D &master->cd_table; break; case ARM_SMMU_DOMAIN_S2: case ARM_SMMU_DOMAIN_NESTED: @@ -2075,14 +2074,10 @@ static void arm_smmu_domain_free(struct iommu_domai= n *domain) =20 free_io_pgtable_ops(smmu_domain->pgtbl_ops); =20 - /* Free the CD and ASID, if we allocated them */ + /* Free the ASID or VMID */ if (smmu_domain->stage =3D=3D ARM_SMMU_DOMAIN_S1) { - struct arm_smmu_ctx_desc_cfg *cd_table =3D &smmu_domain->cd_table; - /* Prevent SVA from touching the CD while we're freeing it */ mutex_lock(&arm_smmu_asid_lock); - if (cd_table->cdtab) - arm_smmu_free_cd_tables(smmu_domain); arm_smmu_free_asid(&smmu_domain->cd); mutex_unlock(&arm_smmu_asid_lock); } else { @@ -2113,10 +2108,6 @@ static int arm_smmu_domain_finalise_s1(struct arm_sm= mu_domain *smmu_domain, if (ret) goto out_unlock; =20 - ret =3D arm_smmu_alloc_cd_tables(smmu_domain, master); - if (ret) - goto out_free_asid; - cd->asid =3D (u16)asid; cd->ttbr =3D pgtbl_cfg->arm_lpae_s1_cfg.ttbr; cd->tcr =3D FIELD_PREP(CTXDESC_CD_0_TCR_T0SZ, tcr->tsz) | @@ -2128,17 +2119,9 @@ static int arm_smmu_domain_finalise_s1(struct arm_sm= mu_domain *smmu_domain, CTXDESC_CD_0_TCR_EPD1 | CTXDESC_CD_0_AA64; cd->mair =3D pgtbl_cfg->arm_lpae_s1_cfg.mair; =20 - ret =3D arm_smmu_write_ctx_desc(master, 0, cd); - if (ret) - goto out_free_cd_tables; - mutex_unlock(&arm_smmu_asid_lock); return 0; =20 -out_free_cd_tables: - arm_smmu_free_cd_tables(smmu_domain); -out_free_asid: - arm_smmu_free_asid(cd); out_unlock: mutex_unlock(&arm_smmu_asid_lock); return ret; @@ -2400,6 +2383,14 @@ static void arm_smmu_detach_dev(struct arm_smmu_mast= er *master) master->domain =3D NULL; master->ats_enabled =3D false; arm_smmu_install_ste_for_dev(master); + /* + * Clearing the CD entry isn't strictly required to detach the domain + * since the table is uninstalled anyway, but it helps avoid confusion + * in the call to arm_smmu_write_ctx_desc on the next attach (which + * expects the entry to be empty). + */ + if (smmu_domain->stage =3D=3D ARM_SMMU_DOMAIN_S1 && master->cd_table.cdta= b) + arm_smmu_write_ctx_desc(master, 0, NULL); } =20 static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device = *dev) @@ -2434,23 +2425,14 @@ static int arm_smmu_attach_dev(struct iommu_domain = *domain, struct device *dev) if (!smmu_domain->smmu) { smmu_domain->smmu =3D smmu; ret =3D arm_smmu_domain_finalise(domain, master); - if (ret) { + if (ret) smmu_domain->smmu =3D NULL; - goto out_unlock; - } - } else if (smmu_domain->smmu !=3D smmu) { - ret =3D -EINVAL; - goto out_unlock; - } else if (smmu_domain->stage =3D=3D ARM_SMMU_DOMAIN_S1 && - master->ssid_bits !=3D smmu_domain->cd_table.s1cdmax) { + } else if (smmu_domain->smmu !=3D smmu) ret =3D -EINVAL; - goto out_unlock; - } else if (smmu_domain->stage =3D=3D ARM_SMMU_DOMAIN_S1 && - smmu_domain->cd_table.stall_enabled !=3D - master->stall_enabled) { - ret =3D -EINVAL; - goto out_unlock; - } + + mutex_unlock(&smmu_domain->init_mutex); + if (ret) + return ret; =20 master->domain =3D smmu_domain; =20 @@ -2464,16 +2446,42 @@ static int arm_smmu_attach_dev(struct iommu_domain = *domain, struct device *dev) if (smmu_domain->stage !=3D ARM_SMMU_DOMAIN_BYPASS) master->ats_enabled =3D arm_smmu_ats_supported(master); =20 - arm_smmu_install_ste_for_dev(master); - spin_lock_irqsave(&smmu_domain->devices_lock, flags); list_add(&master->domain_head, &smmu_domain->devices); spin_unlock_irqrestore(&smmu_domain->devices_lock, flags); =20 + if (smmu_domain->stage =3D=3D ARM_SMMU_DOMAIN_S1) { + if (!master->cd_table.cdtab) { + ret =3D arm_smmu_alloc_cd_tables(master); + if (ret) { + master->domain =3D NULL; + goto out_list_del; + } + } + + /* + * Prevent SVA from concurrently modifying the CD or writing to + * the CD entry + */ + mutex_lock(&arm_smmu_asid_lock); + ret =3D arm_smmu_write_ctx_desc(master, 0, &smmu_domain->cd); + mutex_unlock(&arm_smmu_asid_lock); + if (ret) { + master->domain =3D NULL; + goto out_list_del; + } + } + + arm_smmu_install_ste_for_dev(master); + arm_smmu_enable_ats(master); + return 0; + +out_list_del: + spin_lock_irqsave(&smmu_domain->devices_lock, flags); + list_del(&master->domain_head); + spin_unlock_irqrestore(&smmu_domain->devices_lock, flags); =20 -out_unlock: - mutex_unlock(&smmu_domain->init_mutex); return ret; } =20 @@ -2718,6 +2726,8 @@ static void arm_smmu_release_device(struct device *de= v) arm_smmu_detach_dev(master); arm_smmu_disable_pasid(master); arm_smmu_remove_master(master); + if (master->cd_table.cdtab) + arm_smmu_free_cd_tables(master); kfree(master); } =20 diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/ar= m/arm-smmu-v3/arm-smmu-v3.h index 00f8e6388848e..2f4b832e0deb4 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h @@ -695,6 +695,8 @@ struct arm_smmu_master { struct arm_smmu_domain *domain; struct list_head domain_head; struct arm_smmu_stream *streams; + /* Locked by the iommu core using the group mutex */ + struct arm_smmu_ctx_desc_cfg cd_table; unsigned int num_streams; bool ats_enabled; bool stall_enabled; @@ -721,11 +723,8 @@ struct arm_smmu_domain { =20 enum arm_smmu_domain_stage stage; union { - struct { struct arm_smmu_ctx_desc cd; - struct arm_smmu_ctx_desc_cfg cd_table; - }; - struct arm_smmu_s2_cfg s2_cfg; + struct arm_smmu_s2_cfg s2_cfg; }; =20 struct iommu_domain domain; --=20 2.42.0.283.g2d96d420d3-goog