From nobody Sat Feb 7 11:52:10 2026 Received: from SN4PR0501CU005.outbound.protection.outlook.com (mail-southcentralusazon11011030.outbound.protection.outlook.com [40.93.194.30]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BBAFC265CC2 for ; Tue, 27 Jan 2026 03:09:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.93.194.30 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1769483387; cv=fail; b=aFz0Q7o/b58E5Sp8z+oibedeiZ7eb4hZf4FIaVIquFrO/EknTqyJDgia7slkkdLXp9R+4rYOuG/q2R91hs1/9fiHky7c516DXwVs+YuFmukhCO/B7ZpoSQ8xDryDsq6DaqUTqW22azOaNGduVXBm+okBT3Wl0B+RSXx5Y6aNmns= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1769483387; c=relaxed/simple; bh=e8w+kJJTWpDvNnIri4qzlJLkamQnd9zmXkfKm5e180o=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=nogsdDpCeRmhldMiPFY4teuHViJa5NzAcSkZ0mnrx7/T3sKC7u7yjUTxKG3gekh8wyPPTZSN/WiooSFKz5t50cw4wMt5zEF9nkvR459a+L+qmR3btHamzGh+HlAB82NmricV7a828FM43bteIRfyoVUply44i+s1043vauAsS9c= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=XA6vuvBo; arc=fail smtp.client-ip=40.93.194.30 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="XA6vuvBo" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=JXioHBJVFLRpnuboZLQzkONi+Is8CKHVeUClgrTnfkI/+HHBnw+qpBI4s7qJMmvGYMqhHhsiNpEqF8WqhJPQQs6d3Obg3BnyCWtNr+KNeZRb2AzirByccTW6foGrQp2ou1q7/r50aJz8SE40qgsr7P6F1cfkfUu62twCF0TMjxCS6keKq0ohid+ToxJPdjX/Do3XVAQU6i2MaeNJWC+OGmco0sAEewq3BuDnLvTKgE+ip/bJlM44RrIbBzwMN1sJUDU/uOc4yl2ZE8oCV0rRnJHL1fi4RUGKjOOR39mPuBDeI09IozHvPDYcBTpiEULsaYVYGrZgI5cZBFq/CsXvdA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=bR1WV+hnSofJm2a+W10Fau6sfiq5FT+in6HkPRcmHnA=; b=ueCifcrMV1xT533SLv/UT6OhiB4UuQCizMWCn6Ze1Gpmp/C9KSTin5AzjWGFwiBJxu4b8H5sXYky8oAZqfVyhRpEplVkiODfp8gmVA+vmclpHaFiaVbwlVo/KHnXFw/gEJUu5tZHIvwezoWxjJQtIRLtI/1jFhOiJnLW+4EapvGWcqrLFTnGVdIx9Y1aXnelszOxJ6bt7FZ3Gzi7tAYlDF2SASWE5dtMaOpIMnQo4ORmnK9ECsvW9WRNvJqbih+qf61LdSKDcTaAwNyLjxyHbYdSMWA9PSEML8tkKmCstJfxzrGqeHKJ/bBtIm8L5mmaZc8Aqu3pAhIKcyWXRFpRwQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.232) smtp.rcpttodomain=kernel.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=bR1WV+hnSofJm2a+W10Fau6sfiq5FT+in6HkPRcmHnA=; b=XA6vuvBodTJrs8u9WKnU9s5kIXTmoPhNugsQi9z43VXbYweroezo9mOH0YNXotZY8UaTT7AMr/iBGrZEhvVrENXXK7RL5kMBuljEc2BjMeEjmLHPRE42V9SslyyXPifOHxHzX6vM7uMGpS8R1DTBqUurNVrqpNDTMqLtP3Q4OmLHl3yDGgZktRakYZD4E6tKDIu0kZKZlOjxKiMld8WBlu1o/dFrVMHIHP4Q18ZZBQXxgRfk21URoliRweeWeE32K0rqzwL+lapAjHX5MFluzaqAObfYssISYAsW7UxfO4e6E3LlRclqiaXL8joLa/l7QzGmLjvFCriTqMNsq9BrIg== Received: from CH0PR13CA0032.namprd13.prod.outlook.com (2603:10b6:610:b2::7) by DM4PR12MB6086.namprd12.prod.outlook.com (2603:10b6:8:b2::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9542.16; Tue, 27 Jan 2026 03:09:41 +0000 Received: from CH1PEPF0000AD7C.namprd04.prod.outlook.com (2603:10b6:610:b2:cafe::63) by CH0PR13CA0032.outlook.office365.com (2603:10b6:610:b2::7) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.9564.7 via Frontend Transport; Tue, 27 Jan 2026 03:09:13 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.232) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.232 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.232; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.118.232) by CH1PEPF0000AD7C.mail.protection.outlook.com (10.167.244.84) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9564.3 via Frontend Transport; Tue, 27 Jan 2026 03:09:41 +0000 Received: from drhqmail203.nvidia.com (10.126.190.182) by mail.nvidia.com (10.127.129.5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.20; Mon, 26 Jan 2026 19:09:34 -0800 Received: from drhqmail201.nvidia.com (10.126.190.180) by drhqmail203.nvidia.com (10.126.190.182) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.20; Mon, 26 Jan 2026 19:09:33 -0800 Received: from Asurada-Nvidia.nvidia.com (10.127.8.14) by mail.nvidia.com (10.126.190.180) with Microsoft SMTP Server id 15.2.2562.20 via Frontend Transport; Mon, 26 Jan 2026 19:09:33 -0800 From: Nicolin Chen To: CC: , , , , , , , , , , , Subject: [PATCH v10 1/8] iommu/arm-smmu-v3: Add a missing dma_wmb() for hitless STE update Date: Mon, 26 Jan 2026 19:09:12 -0800 Message-ID: X-Mailer: git-send-email 2.43.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH1PEPF0000AD7C:EE_|DM4PR12MB6086:EE_ X-MS-Office365-Filtering-Correlation-Id: dfa9c70b-4199-438e-4f97-08de5d5183cc X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|7416014|376014|36860700013|82310400026|1800799024|7053199007; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?lkFV7ovoHYJFhTK52A96KTWxGdonXlt87P2iNeyR5Ln5D/ZzyT3g1nQNFHGd?= =?us-ascii?Q?8MCzbFgLQqqsdXtJkxYZYfxMta7+IpyM4oQodvN0/liyJ9wErSPQZTkw+23Y?= =?us-ascii?Q?rMyf3a81A7HsnU0kiNswYWYssHSXEi5gOEuVrGYXOvs1dNrTY7w5+9U0NCfq?= =?us-ascii?Q?PcCKfvtHdk+nlSCOIcgo0sJdTX27cjB+/PbKWbeEJgnnssCPOdySZsEhqsVZ?= =?us-ascii?Q?XcQDbh2hHk6JVGmSS0NBaa+cRf09feYXvKekwLFpxHcdPvjUgiYnLF1Hu+os?= =?us-ascii?Q?JeXpzD21CivvnH3y65JXZf+/CcrzRtgmz/gakrRcYzIjp2geyEWY339dTPPa?= =?us-ascii?Q?dhpFIjo8fkBEwU/II7Ms75Gy3MjFotfxBiwa14YPEWXATo/w6ofrnAczzU0N?= =?us-ascii?Q?TIb9MeH20CtRi1acQeV2faFf3VXY9jw3ubmFx2FHBkNPiPn+SvyqfyDBTTOY?= =?us-ascii?Q?VENdXityEW2yLmbaUDCH/C4XpCK7jelcH3MbpOd5SOkhs8IH6bvVAuyHU14Y?= =?us-ascii?Q?ag2oh4zvs5R0Pg3W2miELctvkuapnYvc+k32KKxH3COncA2xcq2bW5JK7F0A?= =?us-ascii?Q?JG/vyoSkHd02DqPAEKDCTUR3VvVhWXLUqGjYI+CU4fi3JS3KuGOBH7RcD+nd?= =?us-ascii?Q?eDVA9O5o/uw0xM9wzaTIwcqmtE7BGnnArfWoHCWFsh488ZQbbmDLlo4JXGq7?= =?us-ascii?Q?hlMro+MFbLM/nByxHyUBFQJO5ZcvcidtcChC0Bwnd2ttqyp7iqfP2NKfUljC?= =?us-ascii?Q?uZmAdrpFDeW6BQHIYtiv1y91b0Z+3qkNPLrDn7rWNHs8DcuiMPKyo87knnMg?= =?us-ascii?Q?lZjTpWdlSzzx9ZEZHKjUSbgHmLExkLJPOEr2BfxXSivLXokWJz0uqLxewMc5?= =?us-ascii?Q?z/nsicrb/VUoWvpAiUvc7R1QPG5B3KAytFuuq+e9ASKheYN7uUT90t132HLM?= =?us-ascii?Q?QOOgv5eCvU/W0MURJAtlkG8hP9r3F2ToxssyesGCJWs170U2bNXConE0DQnm?= =?us-ascii?Q?CHp73Czs/j1J3uVJGbSJTJtW2EicqCkCGL/jqWtPTfTrMRhNMiT3dm598B5u?= =?us-ascii?Q?F1tDbFy+4XSw7XzvsSvMnVqiVxhjI5YV0+L6rMEf6REGsorDuk9T88WrL36I?= =?us-ascii?Q?kpzI5sDsqgXPgI7Bkg5yq3hrCEeHDmY5ZeQ/tUUTtbVqSyqk+BpyghzncXNh?= =?us-ascii?Q?5dTLvCOkv27dEfYnmzUuZm5WX+ZWCYyhKQo7XKQ8CCqCTxCyd2MKJZ47sw8l?= =?us-ascii?Q?bQeHUzDis54sj3JLTlGYnLg/m4tiT8WBsKi7Y3LsE5ogpJ3Uh6TYNzKaOkPr?= =?us-ascii?Q?P6hlim1hslwwgbHcwlQuwCrsyjZHdPIOzKeUVzfKZuaeUHFVjzZxdVOt6sLr?= =?us-ascii?Q?zE/p+tDlBU66HqeHohV/UoaE3Na+mYkS0SdOKc0xSbJBk2KM6wtuehX5+diW?= =?us-ascii?Q?fEX+MipkaBLI4usJgBtp91vTVE7HnYj3AYMaH8BpcQgltN4lfzknwNveIAeI?= =?us-ascii?Q?S2ida5TAkL/GW08dKP4pT8PJbRYEDPjrZcIzSEmK0h/gG+sL8N8u6IO5BUUX?= =?us-ascii?Q?4oBBAyPlKMA4X7JqCH7mBCplfmxib07nH3lN+Nrl4x5BerZLO3RDOvHlsLaV?= =?us-ascii?Q?r5hxMbfpeNw5O6Kma2rJF4pMGtotc0rC9ULdNXfb4vqJwjFsB9x/lY5XOSVg?= =?us-ascii?Q?V57KWg=3D=3D?= X-Forefront-Antispam-Report: CIP:216.228.118.232;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc7edge1.nvidia.com;CAT:NONE;SFS:(13230040)(7416014)(376014)(36860700013)(82310400026)(1800799024)(7053199007);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Jan 2026 03:09:41.6591 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: dfa9c70b-4199-438e-4f97-08de5d5183cc X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.118.232];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CH1PEPF0000AD7C.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM4PR12MB6086 Content-Type: text/plain; charset="utf-8" When writing a new (previously invalid) valid IOPTE to a page table, then installing the page table into an STE hitlesslessly (e.g. in S2TTB field), there is a window before an STE invalidation, where the page-table may be accessed by SMMU but the new IOPTE is still siting in the CPU cache. This could occur when we allocate an iommu_domain and immediately install it hitlessly, while there would be no dma_wmb() for the page table memory prior to the earliest point of HW reading the STE. Fix it by adding a dma_wmb() prior to updating the STE. Fixes: 56e1a4cc2588 ("iommu/arm-smmu-v3: Add unit tests for arm_smmu_write_= entry") Cc: stable@vger.kernel.org Reported-by: Will Deacon Closes: https://lore.kernel.org/linux-iommu/aXdlnLLFUBwjT0V5@willie-the-tru= ck/ Suggested-by: Jason Gunthorpe Signed-off-by: Nicolin Chen --- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/ar= m/arm-smmu-v3/arm-smmu-v3.c index 852379845359..f0e3b407c293 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -1236,6 +1236,13 @@ void arm_smmu_write_entry(struct arm_smmu_entry_writ= er *writer, __le64 *entry, __le64 unused_update[NUM_ENTRY_QWORDS]; u8 used_qword_diff; =20 + /* + * Many of the entry structures have pointers to other structures that + * need to have their updates be visible before any writes of the entry + * happen. + */ + dma_wmb(); + used_qword_diff =3D arm_smmu_entry_qword_diff(writer, entry, target, unused_update); if (hweight8(used_qword_diff) =3D=3D 1) { --=20 2.43.0 From nobody Sat Feb 7 11:52:10 2026 Received: from MW6PR02CU001.outbound.protection.outlook.com (mail-westus2azon11012046.outbound.protection.outlook.com [52.101.48.46]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E05662F5492 for ; Tue, 27 Jan 2026 03:09:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=52.101.48.46 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1769483391; cv=fail; b=MG/aajM6m2H0WBM5pcOi6ZhGQxBaf9XI1djKwb+bmRuLh88PAWfUEIE553tGq2fTB7NZ7syPtmXxDXJF3bioA1Dlm20ybQRP7M1fQcOADID93zZ3rwCGLzPZkxPnZ/OjNhPLRNBcbCOzqlyIfAvsWE8aPVRpCqeaZUtAm02VjZg= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1769483391; c=relaxed/simple; bh=/Zmmdbf2RP+Mzk6BaK+3ZN9TpfLi6P6DOgpbG7bI/qE=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=JzDTviIJh7546Cm/wkPhRY1eyFGAg4mTd2JLxQnBqmLy8VRpXbBWuIuVhIrET+dU/8EIw9NQZ95ZZ3bbu4LI7DkE+o8BcaPxOxstDCs37zODhZryGsRNwtyM8EmykKqrYJcP9SCAlOQQmSQ2eBUtj6q0N1pytpGcwGmMJ7BhBxQ= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=d+I6NhrF; arc=fail smtp.client-ip=52.101.48.46 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="d+I6NhrF" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=y+f7MEYy1QauNKE0TWFHoef7aq5+joHuEp2O/8BvYwehXXBR90MxSD9EoC5Vmkh7ppXpOeXF93nmah+W7NCbrp7947NRtRi5k6UV5O/1BHKNN9IKMCg6Ct6GWGrV6oBtUXf51Zorrbw+F9aeq1wsDViKtSpFLdBBQL3Z+psWxHcwS3+6FatxMOV0EGt/j7eu1+rQ6P4dKoOQLDTzctyzugSazrRLGNYABdVoXT47N/9vipm4Pr2i2Kw42tehwBVahMgSmyGC0cOYTNfJmuQnaH6m5IRnm3aleolfzJVKQFEw3Vkd8JI/t4bIs4lNB4QCyjE/zaocmFXakf0Vj6yyQQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Gg7O4T6+DcumjFbxgHM0BeoJlMr/0VGkrnBfw7zDC9k=; b=TO1mGKQIbM9tFI9dryyL/hpfGM8ujAjyeW+siJl8S23nycD7rAIsad0QpXOie9fOewCHFUdN6Q4ulhC0l+Dn189+ONI8QeTse4dDPFGukX5t3IfC87juu8gHe46aTi/M15F3rfTCRsDwhhiUodmVx6RmqPNQ30Q+Ank4lE7ZhmlGF2J9nhV12hoquEjPiSP1MMYz81mCX6nA+ZQePzbZSR/GUmn3i6rxGRLcEbOMIE9pnvNsLAPcseiV0KC9B1XlMXbiDZxFqoltqJwljGLFcdxbXYvpX+6R1qBnBwbB7x173CPc6QK7PQds09aCWa9WyHJbVokLHR96fqmr12gj8g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.233) smtp.rcpttodomain=kernel.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Gg7O4T6+DcumjFbxgHM0BeoJlMr/0VGkrnBfw7zDC9k=; b=d+I6NhrFtCYdg+NJvEfrpx5NPB1HxEh2aGgW4SWH22XPLuZ+YDNScNOsl/DcA3RvLPTzFQzyZLk19tTfg0YcTF0iBoATQWHcUtarSmU5E3o+LU+lVdSpw1Hb0fzAnDkRW/kaFE5jwkR8BpkCQa2Td0FVrRieAbB4F3OrEUluUyWioYxEX9c6DkpuxQHoFtMKXiiOJEElu3ExFJz/WNJVYyP2hisrDU82TafEVmW8nnGetUXAC/1pRUhGCYcBZI74DBZd6Yl9LJYUdPpYzVn6+lgf3QAByhXdVTTXW9wL39LHwobtcqpWNQ2sxX3r9oNU3JQrBhUWrGirju+PF84B8Q== Received: from LV3P220CA0009.NAMP220.PROD.OUTLOOK.COM (2603:10b6:408:234::18) by SN7PR12MB8817.namprd12.prod.outlook.com (2603:10b6:806:347::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9542.9; Tue, 27 Jan 2026 03:09:46 +0000 Received: from BN2PEPF000044A8.namprd04.prod.outlook.com (2603:10b6:408:234:cafe::c1) by LV3P220CA0009.outlook.office365.com (2603:10b6:408:234::18) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.9542.16 via Frontend Transport; Tue, 27 Jan 2026 03:09:45 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.233) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.233 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.233; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.118.233) by BN2PEPF000044A8.mail.protection.outlook.com (10.167.243.102) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9564.3 via Frontend Transport; Tue, 27 Jan 2026 03:09:46 +0000 Received: from drhqmail202.nvidia.com (10.126.190.181) by mail.nvidia.com (10.127.129.6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.20; Mon, 26 Jan 2026 19:09:35 -0800 Received: from drhqmail201.nvidia.com (10.126.190.180) by drhqmail202.nvidia.com (10.126.190.181) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.20; Mon, 26 Jan 2026 19:09:34 -0800 Received: from Asurada-Nvidia.nvidia.com (10.127.8.14) by mail.nvidia.com (10.126.190.180) with Microsoft SMTP Server id 15.2.2562.20 via Frontend Transport; Mon, 26 Jan 2026 19:09:34 -0800 From: Nicolin Chen To: CC: , , , , , , , , , , , Subject: [PATCH v10 2/8] iommu/arm-smmu-v3: Explicitly set smmu_domain->stage for SVA Date: Mon, 26 Jan 2026 19:09:13 -0800 Message-ID: <444724fc12a62f3746b6ad38d04e88c2872ec9a0.1769476588.git.nicolinc@nvidia.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BN2PEPF000044A8:EE_|SN7PR12MB8817:EE_ X-MS-Office365-Filtering-Correlation-Id: 26a52b20-b3d7-400f-cab7-08de5d5186aa X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|82310400026|7416014|376014|36860700013|1800799024; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?dVblUuzNlXIjxatoPMv3Vpf8+W+3hPGKlnV5dmQpGKgTi5NS4WFFjXZrCh5V?= =?us-ascii?Q?t2E5kZ4j5w19X0tNer/7Nlw9PUCGskWFeO1t/c77m++v9Kg8lbY+f+kNBeMZ?= =?us-ascii?Q?45D8aVpIW8dJoNNNYN1W2+9fey3y3+09czPzHekhRW+Zzr4RIasyhox69nIr?= =?us-ascii?Q?zH8DN9oiGnS6XY5y0bBUg2sqqlwU1ho+rpgwupAAg0m9f+8wz4lCBuK3/sWG?= =?us-ascii?Q?poa/xsPMAmTC9fOy8zRBhqPYJJcI5wlSZOw24SNlCfvQx532JHvxkJP9y4OG?= =?us-ascii?Q?iK1/CyruOM8c6DMJzKV2y9ogXtRvBzic90m4sxe59bIjrrFcKzxsyFwVs2Z+?= =?us-ascii?Q?pqx0KAxosuOTcInZu6GJaUz7mFbvWCSFgSRb5uqi9pTV8jk6DR7EwjBOd/m9?= =?us-ascii?Q?veqP9ukB8R3oElrWE3mtkW8PYaJ1CsaE/qvQhIJB2LFLhoEmceokMonLHZmB?= =?us-ascii?Q?jWt2f5Xk5PqYFS7OjUJFRjB518YkKLYhiXS1Z6MAzqc337HqI97Yz9EeoOlY?= =?us-ascii?Q?Z2nuBd4mlRHajJhl0cYjjBiI3As5uHMizyVEqQtawAUraXwvDZTmul4NwIw5?= =?us-ascii?Q?FwPXcDHxF999c23Qw+taCgi44eimchAzAQVkmvwfQbJ1Z0VjfCN1BcS+L1Ix?= =?us-ascii?Q?Dm3/CXUuoFi3on2IzuXFgEFBjA+kvx8SkGF9m0srAziMIzcXQf8f9QBS2jFr?= =?us-ascii?Q?kgonB5bCE8EJgK0XeLlQp9I/BiTrbq4AvlU+3874RHjvW0A9l/nuUQiM2nYS?= =?us-ascii?Q?BiI6YeE3pAUyJiXOMwakKlYKCWbZZPbkSMcEN0OOBPEHbefMgB8onFdCHU8P?= =?us-ascii?Q?XdvQjQT5qDxH+obR3Wb4tcZrK0HFeUWbBzUkSWYHx7mqtU3mxp1LF510tOla?= =?us-ascii?Q?MMh0KsYmZbdxAPMpMPhSxeZ19Kaq7313oqWkBsdgJmSkpEdEKFl6yjjIkmDW?= =?us-ascii?Q?WUdLIQKSNFei9aG9rqb8VNKPl7qU5bCAN5Se7wLkekPiZHN7VLDgshRBhl9E?= =?us-ascii?Q?SYua7+1rFHEBWVq0oftBwA+KBAiv9AdwuIBaNAqkhZSV+aLzZ00JljPQdyIG?= =?us-ascii?Q?BBKCr2EA7cClHRX2snCRKsKZ3/33VA0XWz/xpmRKFW2rpbHWlF93pI+N0irz?= =?us-ascii?Q?MWykUO39oN0p1Nl9RGn1cjLGb+mEGBJ9CaVVUCvYP8x6kSvNr4t1+FI0ug8k?= =?us-ascii?Q?THnb6gFaWF5IFRbgdCCtZhasw7t9V+OEgcUuNI+omiD7/iOWfxls7OUZzUi4?= =?us-ascii?Q?gHnUMuQ44eKVYth4RqcmsSTDL9DLKZvwxmYGZ4mspDScSEyf+q/OkM7ozaQ3?= =?us-ascii?Q?WVXVIt8vZkLWyIUi61o3JRmSn7U1Za5oaUwA83U/MhsCRBPCgoKEHilwOsuW?= =?us-ascii?Q?+hqq2dhlLlEHjTvIv+XGFgjHqbbzceuOFEiWWYSLCLuAZA7yLiVHPk1JXMvV?= =?us-ascii?Q?cJyNpNGm4Z7OhIn2dR0Tc92ZGHXTq8k1U+hCisTaW5XX63cHq7A15npl90ry?= =?us-ascii?Q?D2JS0FlFv5U6dAXFeRXx8jqxPE54a0EQxGA371G44BsslsdtLapUVLAIDuqd?= =?us-ascii?Q?tObzBdyvj+uh5yfZCIpMBhUqvBcATnEitBnmiXb9p6E2y2xtCrFToKxsExna?= =?us-ascii?Q?SWKY7ian8bb8kq0RMBn9fSjeHnEEWMeC8ttt3fmyMfLZgtgr2LKvNefVzbtM?= =?us-ascii?Q?8i4bkA=3D=3D?= X-Forefront-Antispam-Report: CIP:216.228.118.233;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc7edge2.nvidia.com;CAT:NONE;SFS:(13230040)(82310400026)(7416014)(376014)(36860700013)(1800799024);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Jan 2026 03:09:46.4500 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 26a52b20-b3d7-400f-cab7-08de5d5186aa X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.118.233];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BN2PEPF000044A8.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN7PR12MB8817 Content-Type: text/plain; charset="utf-8" Both the ARM_SMMU_DOMAIN_S1 case and the SVA case use ASID, requiring ASID based invalidation commands to flush the TLB. Define an ARM_SMMU_DOMAIN_SVA to make the SVA case clear to share the same path with the ARM_SMMU_DOMAIN_S1 case, which will be a part of the routine to build a new per-domain invalidation array. There is no function change. Suggested-by: Jason Gunthorpe Acked-by: Balbir Singh Reviewed-by: Jason Gunthorpe Reviewed-by: Pranjal Shrivastava Signed-off-by: Nicolin Chen --- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 1 + drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 1 + drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 3 +++ 3 files changed, 5 insertions(+) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/ar= m/arm-smmu-v3/arm-smmu-v3.h index 3c6d65d36164..24894b163004 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h @@ -856,6 +856,7 @@ struct arm_smmu_master { enum arm_smmu_domain_stage { ARM_SMMU_DOMAIN_S1 =3D 0, ARM_SMMU_DOMAIN_S2, + ARM_SMMU_DOMAIN_SVA, }; =20 struct arm_smmu_domain { diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iomm= u/arm/arm-smmu-v3/arm-smmu-v3-sva.c index 59a480974d80..6097f1f540d8 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c @@ -346,6 +346,7 @@ struct iommu_domain *arm_smmu_sva_domain_alloc(struct d= evice *dev, * ARM_SMMU_FEAT_RANGE_INV is present */ smmu_domain->domain.pgsize_bitmap =3D PAGE_SIZE; + smmu_domain->stage =3D ARM_SMMU_DOMAIN_SVA; smmu_domain->smmu =3D smmu; =20 ret =3D xa_alloc(&arm_smmu_asid_xa, &asid, smmu_domain, diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/ar= m/arm-smmu-v3/arm-smmu-v3.c index f0e3b407c293..a55f9ae95411 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -3133,6 +3133,9 @@ static int arm_smmu_attach_dev(struct iommu_domain *d= omain, struct device *dev, arm_smmu_install_ste_for_dev(master, &target); arm_smmu_clear_cd(master, IOMMU_NO_PASID); break; + default: + WARN_ON(true); + break; } =20 arm_smmu_attach_commit(&state); --=20 2.43.0 From nobody Sat Feb 7 11:52:10 2026 Received: from BN1PR04CU002.outbound.protection.outlook.com (mail-eastus2azon11010015.outbound.protection.outlook.com [52.101.56.15]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2C9502FD1B6 for ; Tue, 27 Jan 2026 03:09:52 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=52.101.56.15 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1769483394; cv=fail; b=qAoE+raRrOLrpKgwcHhC6JgleTbzo75EUUbdFpDNTEoPmknuY3PdXgtW70rXcVWH5GzwanzZS1aYFyjlUY8bVf6RJpO8tD6GEgCRrE+LxdX5hhfxzrqRcyD+cFBGG6H/1m3vy5a8q+1bsjjZq0obXAmbD3DEc3S4y+rFl0fOkfg= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1769483394; c=relaxed/simple; bh=T2m2Y2n3KgDnVar93IMm9LDwJsbzrC2XYX1p93fe1bk=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=t3LsigoH91TzmDJGUtnH1A1hIHylnFwpilIkBe/V3ZQaQouuFCRkKyEX380YTEV6FRjajwJbDdhbjrNqKeW3GP1UsNW/IeV049FV70tjrVb/DMd8vc2Ty0HiLknnJ/9MUALIHQlzix7YDm7XuE0yxUTfn5JAWpXp2KDf8DDFQHw= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=V9zoFqJu; arc=fail smtp.client-ip=52.101.56.15 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="V9zoFqJu" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=majBG1kFxu69l78HmxldkOatyoAv/5Xc1Rx9RQJU0aIncEI3cMENwqXoH+3pzNcpvUE6VsiYYRwz1RlAIkCzoiUG8dTBEHuSs+lOuaA3DUYk8BojqnEcEeLQrR8Wqk6DntCzRU6ADIiMo+PvSDIrV2L/492TQGwMyZ7A1p7K1Hq+jOD5YzdhcWG0eYV5jzdC0TzzRZvOuWuHSMfn3uSR2q1DrfQ7tdyzfCX28mLB41wFDbAF58VTYmrcmXQtvr3n9T/MlbTcZ1SxhqTOn620AHRBEVlvxmNU7+1ZEVfHXlneCcFBjGA//XJXs/B+AnidI0Ckeiq+EjdewC0uhO4hVg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=9EsUegwYvh5X0cWkaY9RCge07+sCBL++7IVOrlemteE=; b=kYM050MDDELaSBKYsrBPjxaFpUf0MSMdLXMVyrQi3kquDyU7qxJPPLPYckiNb2jlHxxXvTNGAnbrKZt21JF+bi3mP9L0IG2e4UEV3H6RIz0317vZv3905KRtggukgo+FSstCN9fh9mXsTr8n4Jzckxv1xCVaQq3d1TDwPfihdC3bFaHE5yvCAny0ygLOUWWxK8NSMxGFftzjxh/TkN3ttcc2QQe7Vq90SSrlpoEQSa3SeYn+I8z8IS8af9jvi+eCEA2iVOj3XgInjJjbA8NE80lsNR1sMN6VnENw02OFjlexkmJrrwxJUAOcHuGOovEItIZt87Y83T7VmJ7RrI2qwA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.233) smtp.rcpttodomain=kernel.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=9EsUegwYvh5X0cWkaY9RCge07+sCBL++7IVOrlemteE=; b=V9zoFqJuuSbohFdm7aGgMKRI5wIUTTehteWUY+8TjPUYSeO33QJmXeb5dMiXu/Dgq4tEyJrfCyuvFzZnlQnSAzFhwLpbnuHtpFAH4KfxnTrPZFLFfnBEXbAJQgtLv56HRZ+BlkYi+hjqi1+ljg1aJSTTKcaOYizy2MIeYWeXVXuqTRyclm3i18YASzsTc89+siUZNh5qVhnakoaLcsZQaRE78bKeI0iZu1lHz1vaWkq1rT91KJROW1iI0sxOuEQyEi2R44nbteWa6PcWzZ3gVrpEO1n+00IpS5jTlEE2jF64zuV69uMGpeHKoDb+R13mjFhXNobO8tgwxuvrxoDNRQ== Received: from BN9PR03CA0172.namprd03.prod.outlook.com (2603:10b6:408:f4::27) by PH8PR12MB6818.namprd12.prod.outlook.com (2603:10b6:510:1c9::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9542.15; Tue, 27 Jan 2026 03:09:48 +0000 Received: from BN2PEPF000044A6.namprd04.prod.outlook.com (2603:10b6:408:f4:cafe::ef) by BN9PR03CA0172.outlook.office365.com (2603:10b6:408:f4::27) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.9542.15 via Frontend Transport; Tue, 27 Jan 2026 03:09:39 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.233) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.233 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.233; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.118.233) by BN2PEPF000044A6.mail.protection.outlook.com (10.167.243.100) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9564.3 via Frontend Transport; Tue, 27 Jan 2026 03:09:47 +0000 Received: from drhqmail203.nvidia.com (10.126.190.182) by mail.nvidia.com (10.127.129.6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.20; Mon, 26 Jan 2026 19:09:36 -0800 Received: from drhqmail201.nvidia.com (10.126.190.180) by drhqmail203.nvidia.com (10.126.190.182) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.20; Mon, 26 Jan 2026 19:09:35 -0800 Received: from Asurada-Nvidia.nvidia.com (10.127.8.14) by mail.nvidia.com (10.126.190.180) with Microsoft SMTP Server id 15.2.2562.20 via Frontend Transport; Mon, 26 Jan 2026 19:09:35 -0800 From: Nicolin Chen To: CC: , , , , , , , , , , , Subject: [PATCH v10 3/8] iommu/arm-smmu-v3: Add an inline arm_smmu_domain_free() Date: Mon, 26 Jan 2026 19:09:14 -0800 Message-ID: <7e0e6c59e2cdcbdb9eeab642e878749dc6468807.1769476588.git.nicolinc@nvidia.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BN2PEPF000044A6:EE_|PH8PR12MB6818:EE_ X-MS-Office365-Filtering-Correlation-Id: 6061cd12-78dd-47b6-c7bd-08de5d518749 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|36860700013|1800799024|7416014|376014|82310400026; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?7KEjt7kd8oBdz3vR0jg8kSvBZCtFvx8FA5dwRsI4FBIQtf0USPpeXl4Z9zOn?= =?us-ascii?Q?4hvUhGcHk8AM6Q9s+HYzUkjJzHmLJ/kM/U4fvDW6tTeJoeqKOTvOYOuMzc42?= =?us-ascii?Q?0JmVCzcoaOitYsWJresVPljQXlSpFgC3AVaaltQADbdj5qv1pJuhwwWmvIb+?= =?us-ascii?Q?zdmQVYeFbZH/XEOkf1ZgD9FG6RkGLWFH7uhs2wp/SsLTz0SeGfy3mPcFO5Zm?= =?us-ascii?Q?TYDhGYWAIvrOWMcpY14vD1Rq/sMYJvtyrYTSBivtOyP0sZ/C6AFcLq+uK/3o?= =?us-ascii?Q?ZrNQ8uDdxQrZWVHXc0MoEVs2Dv6gh1ep7OA5Lsskp4OsvUJXhBcUxpSLe2kr?= =?us-ascii?Q?sLSC3ClzXpiidtcF34QwSzYdsf75GEHsLhmuWVDiqqcnQppSj1gqwYqBYAVK?= =?us-ascii?Q?cWdyr+sS/tYNWE5YlCOqx7NVaXUUl8KTAVXv8u+noY35iPPEntl3O52iRhXd?= =?us-ascii?Q?f+982L3ydQ6829GqBSYGg4wli+GtjNWDCMya/gSRS7hqrKBRqut0EP8Qb4FV?= =?us-ascii?Q?WneEcfsKljhf7BdVJxYY/Vgl+GkVkvTy/WL0CZ44R1HDx1+iJD/fzrxB1t8c?= =?us-ascii?Q?Y7Mke64dnDZFhjL2F+EFUYVYFSoDbRp3H+JksTQ2Gg1YYhSOBwJN5bcKDIHp?= =?us-ascii?Q?qNf8oK0reZTFJNRu4E+WMHklpTeTf1FIq/xYLojPG5bBg8JcEGCl6Y8mLORJ?= =?us-ascii?Q?y3zKzFvHLRnBgCGvqkRwQL0w+DZ/LKmfQg4b1xEM1pe2DX7RdI+DpQ8K+h6y?= =?us-ascii?Q?dYfU0xeFV8K0TSQ0DZRwEk0yIJbEgahz0Xl5QHEcLSEcPWmy0AtYuty3gfw2?= =?us-ascii?Q?eVXESkjl5m4ffzrM6VcrLZpUMDP07AsGZHjQ1H+00O4b4LH7Qg9i0bgBGWPw?= =?us-ascii?Q?AsvSPMx8OgCYdNFbyVbxfvsIvvT9hyk2xlFbkP/kXv2SxzUCX7z3GvTaUM2K?= =?us-ascii?Q?w6YgJFe/sIuGfHcSgX7XqIlNwoFcRosI/0/GlDZDVoZ24w+xZ/Lr4Iophk8q?= =?us-ascii?Q?I/lJQYAS78iPHja8waiDTZBOo8f6asRyh2nPN1Cxq7Uuvf6U0dm8SOTiUPq/?= =?us-ascii?Q?lhoi5zuXwGB9Ekfu8X3tE3sgetTS5Y46yDX/RW44tmW3NoADeao0cxPdQ6VD?= =?us-ascii?Q?L73RREsL2jYeUZ9D0DbAm+GtNoJbnzFVbFW10XXeDZ/jkam4qHEzo7xXaABz?= =?us-ascii?Q?wWZ2tZSD7HvSHRBkbLGQKcFo5oSzC6laAxdyOAs3/jGTOFX2/0G8l23FD5/4?= =?us-ascii?Q?IfkCPLvodX3qMy9zNp3SPzxGVosA6vXuCEJQ2i10KFYRKQPBJ6XDVMr4yxZU?= =?us-ascii?Q?wtGJrqkiYRHDsKa0ZvzXPai1yKKEFYVJPIDWOr9merMTBtqMUO/2ITc3FUsu?= =?us-ascii?Q?i24m9ddQTWlF3GzSQQd3uQg1wUc8fe0k7HDjrlz9x7j3lymwiZZbCoWIAAc/?= =?us-ascii?Q?r766w3u7pfZdQaZ+WdvvExMAX0H0wySDQQIPhghGVRVYOm/aMMgMb+H0oMX+?= =?us-ascii?Q?yil2b40F9rG3OilZ5e9/hzKX5zdPY64cmsa4YrO/BUjy+NhIah/exAr5rm0/?= =?us-ascii?Q?LyjyyqKT0DdeN/028b1F7Zk1oAX3YjLmXyxzu5dyyXPBDYUKgVGSPXzf9GH9?= =?us-ascii?Q?j7jSd0LrQk8WXmWbEAjprKGSRgw25Pgwf4WECGjtlS1zt98toSQJKOdMWme/?= =?us-ascii?Q?xZU4iw=3D=3D?= X-Forefront-Antispam-Report: CIP:216.228.118.233;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc7edge2.nvidia.com;CAT:NONE;SFS:(13230040)(36860700013)(1800799024)(7416014)(376014)(82310400026);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Jan 2026 03:09:47.4953 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 6061cd12-78dd-47b6-c7bd-08de5d518749 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.118.233];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BN2PEPF000044A6.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH8PR12MB6818 Content-Type: text/plain; charset="utf-8" There will be a bit more things to free than smmu_domain itself. So keep a simple inline function in the header to share aross files. Suggested-by: Jason Gunthorpe Reviewed-by: Jason Gunthorpe Acked-by: Balbir Singh Reviewed-by: Pranjal Shrivastava Signed-off-by: Nicolin Chen --- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 5 +++++ drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 5 +++-- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 4 ++-- 3 files changed, 10 insertions(+), 4 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/ar= m/arm-smmu-v3/arm-smmu-v3.h index 24894b163004..cfbedb76c8ba 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h @@ -956,6 +956,11 @@ extern struct mutex arm_smmu_asid_lock; =20 struct arm_smmu_domain *arm_smmu_domain_alloc(void); =20 +static inline void arm_smmu_domain_free(struct arm_smmu_domain *smmu_domai= n) +{ + kfree(smmu_domain); +} + void arm_smmu_clear_cd(struct arm_smmu_master *master, ioasid_t ssid); struct arm_smmu_cd *arm_smmu_get_cd_ptr(struct arm_smmu_master *master, u32 ssid); diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iomm= u/arm/arm-smmu-v3/arm-smmu-v3-sva.c index 6097f1f540d8..440ad8cc07de 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c @@ -197,7 +197,8 @@ static void arm_smmu_mm_release(struct mmu_notifier *mn= , struct mm_struct *mm) =20 static void arm_smmu_mmu_notifier_free(struct mmu_notifier *mn) { - kfree(container_of(mn, struct arm_smmu_domain, mmu_notifier)); + arm_smmu_domain_free( + container_of(mn, struct arm_smmu_domain, mmu_notifier)); } =20 static const struct mmu_notifier_ops arm_smmu_mmu_notifier_ops =3D { @@ -365,6 +366,6 @@ struct iommu_domain *arm_smmu_sva_domain_alloc(struct d= evice *dev, err_asid: xa_erase(&arm_smmu_asid_xa, smmu_domain->cd.asid); err_free: - kfree(smmu_domain); + arm_smmu_domain_free(smmu_domain); return ERR_PTR(ret); } diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/ar= m/arm-smmu-v3/arm-smmu-v3.c index a55f9ae95411..6beffa8e7c74 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -2560,7 +2560,7 @@ static void arm_smmu_domain_free_paging(struct iommu_= domain *domain) ida_free(&smmu->vmid_map, cfg->vmid); } =20 - kfree(smmu_domain); + arm_smmu_domain_free(smmu_domain); } =20 static int arm_smmu_domain_finalise_s1(struct arm_smmu_device *smmu, @@ -3427,7 +3427,7 @@ arm_smmu_domain_alloc_paging_flags(struct device *dev= , u32 flags, return &smmu_domain->domain; =20 err_free: - kfree(smmu_domain); + arm_smmu_domain_free(smmu_domain); return ERR_PTR(ret); } =20 --=20 2.43.0 From nobody Sat Feb 7 11:52:10 2026 Received: from CY7PR03CU001.outbound.protection.outlook.com (mail-westcentralusazon11010008.outbound.protection.outlook.com [40.93.198.8]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5542E2FF679 for ; Tue, 27 Jan 2026 03:10:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.93.198.8 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1769483404; cv=fail; b=DfqcvmrYgeptOPMpkv+sfIS3Ks5Rgb404mSjqbMCpPDqQ1/P/GqrNa1CzOmqIIr/IKIdfQ4BkA7PkoIcH+r08gLvU4KqiPvGpFyuIqtD+JV7T25HsSaY1qzTf2zBx5RlOhv+qxBmV5dYvGXOwIXPuYfnHnhwJex+IvdsYCAbrWE= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1769483404; c=relaxed/simple; bh=vOlZdLUj3TXtopVT8LrKMgTt2Mx8SgUWC5Gxjp4IFAk=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=TcCuQ4KG6bj98SPEVQ9wdJSKqyfh3GSsqTwGNin4Z27T47Z+NvH+x313P/c7/mlXaLbEfuHFYApLkebYLnJvnVXiFtR/K2VQMMdTYEN+DGQ7rS8L217hBszMHvtzlHWAeKgJCvFgpOBHlwmVAFWB+bF7MmGCZ5ve8x71caNUu3E= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=Hi71Hchq; arc=fail smtp.client-ip=40.93.198.8 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="Hi71Hchq" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=J2t+HzPjedBQZzeChPf3rM+AE0AYSZ9EuWN37z9mSYyBu59dV1FPYJs4Yz/Cr0305XPSWY2/QqiOIaOKZJX0Uc/tsNfSWOvUcC8j+j2unZ2KYPi0JizwWStVRbPElYrc5SKKDJhaRCIblwJB+xf+qkngKREuP9WIDEshzllD/vnJdj3l1F6Ua5BOnxRvGcnPVS4hYkym+EG2GEHwaJ8o7ycSgSe5hyAXsMSGlGB8nrIB1S+MNcDZx3ivqakGjAou1GQYpIKDKZrslvrlUPAq99o5PNvKn23h9FbIkI0EC+1m3BwMvSwCeXmE9EpwjlywvrgykMBxrNRxB8zSrFoMIw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=wpdF2YjpOzQANOUp/YBZ2XUg6x/g/giQ0hrK3lELJ8U=; b=fKdyn7d4n3op2pF6X5GU82iBeIpDE+K99rugZS/+PdaeEVe6bNQY6Esba4gnXxZBDPP7H3zqEJSoh4RKskNIzxIAk3n+r1gGDWXBZKaD7fN0J1ulLX22Iovp1m9dkCJDuptdaID0WWug53qWHT8JABn8OrE6O+mzAePtTR8eH8FRIX2e9OdqmoKgnChSrhMixSJ9fyd0BdovvdCM7ZxLNIYZZCHQs4ikznBiaFGUJ/eNWO+G+ZOAlXq6KSKJ+GD+Tlky2Vd04vfCLCh8gcsUTuTWrNA2Rv6ujMoPBydxgW2vIYX2+5w1Li5YVG+KnZavcxKuMdBst8/j9ewfOJtt5Q== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.232) smtp.rcpttodomain=kernel.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=wpdF2YjpOzQANOUp/YBZ2XUg6x/g/giQ0hrK3lELJ8U=; b=Hi71HchqZLWeix8k/umr12vEHUQU+EKcjzFDcQ/yYgEv8D3aiDXJ8sNa+QWDn6wqZA9bLHZODThYvLF30eLnq/z/doyHbHR20mPJwm9T/YUJmFRVqnXdLL7SaCA9U/4EDwIfHIDpSfxDR7KsPxs7DKCnfCzO54KLv880eaG9BiLg6Vbq2maJQdrF+4VH3wdwPpExZtkdwTbuLncJ20Y05O0mhcIKhm0A9QJeUqYrEBcdITaSVK23rLiWfW+Opbp0yxgkFIZd79t9vdNQT/L2VQ+Z+z8y0s4PXP5SQpsBTs1sj8mNK+Cl1Gw1Alcq5BIUhnL0h3FxJ1id/jZWb4wanQ== Received: from CH0PR13CA0050.namprd13.prod.outlook.com (2603:10b6:610:b2::25) by IA1PR12MB6307.namprd12.prod.outlook.com (2603:10b6:208:3e5::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9542.16; Tue, 27 Jan 2026 03:09:47 +0000 Received: from CH1PEPF0000AD7C.namprd04.prod.outlook.com (2603:10b6:610:b2:cafe::6e) by CH0PR13CA0050.outlook.office365.com (2603:10b6:610:b2::25) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.9564.7 via Frontend Transport; Tue, 27 Jan 2026 03:09:34 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.232) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.232 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.232; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.118.232) by CH1PEPF0000AD7C.mail.protection.outlook.com (10.167.244.84) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9564.3 via Frontend Transport; Tue, 27 Jan 2026 03:09:46 +0000 Received: from drhqmail202.nvidia.com (10.126.190.181) by mail.nvidia.com (10.127.129.5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.20; Mon, 26 Jan 2026 19:09:37 -0800 Received: from drhqmail201.nvidia.com (10.126.190.180) by drhqmail202.nvidia.com (10.126.190.181) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.20; Mon, 26 Jan 2026 19:09:36 -0800 Received: from Asurada-Nvidia.nvidia.com (10.127.8.14) by mail.nvidia.com (10.126.190.180) with Microsoft SMTP Server id 15.2.2562.20 via Frontend Transport; Mon, 26 Jan 2026 19:09:36 -0800 From: Nicolin Chen To: CC: , , , , , , , , , , , Subject: [PATCH v10 4/8] iommu/arm-smmu-v3: Introduce a per-domain arm_smmu_invs array Date: Mon, 26 Jan 2026 19:09:15 -0800 Message-ID: <2aa5e516f9e22be2c96d1d5b69252011210da5e8.1769476588.git.nicolinc@nvidia.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH1PEPF0000AD7C:EE_|IA1PR12MB6307:EE_ X-MS-Office365-Filtering-Correlation-Id: e9e46c47-632b-43f3-942f-08de5d5186e4 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|7416014|36860700013|1800799024|82310400026; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?VOsHZ5B/9vkCTQz7ts/66OrTJqpV1YfAnemf7zjiC7dCOPkdh5wU+LLtHtwa?= =?us-ascii?Q?tUtZfaeA/Vj4CAP2tBNAPX4+FXIBNYeGsx0DE2Cq/egDWCBPAAHfQ0X6A5+/?= =?us-ascii?Q?901E21gEqkA8AzQg9kAs45j+VYDZOZ2vusL5EqnTJHsGk/REAXCy1Gf08veT?= =?us-ascii?Q?ZIy7qKXWUDzBzGIhE6R8j5yOBBLp6kb1NSN8Ei7GASuCD2kkeC52E0cLSplY?= =?us-ascii?Q?QqGo2qgds1nTOswGFoxUBEh8Klqw3sYVd7GDinyUoSR0Qa2lYiaXxjxsZR8Y?= =?us-ascii?Q?GF+auPOfIRgJ8UkwzL+pgFPX+gEhM+yb6zxJNpv70rrvLgAq//O4Hxmuw7cc?= =?us-ascii?Q?kOz3lGBh2dw6eyOeIqfHt3OAFJZkd8OvagfznQthJJhwDxw1Ix01+MAAyfrC?= =?us-ascii?Q?EWttDx0F8e/cf1e2HWFpinsgYLvMeIYHNxuJlsiUBHDhDEMm0IEtHcrYSoX6?= =?us-ascii?Q?bcDd7tP/+glkS/cXv3WnvikTeI9dLlas7iywCp/x9lKZDXrf6simIFhps9OH?= =?us-ascii?Q?jVt3Q85Z4hWeO5/sy906Ty4q8e5Y3UJeWJExw8/I1NwlEAsETD/tFmDBCpNd?= =?us-ascii?Q?Ed615ZuX7H/bpbLnQaw1xSvnVTr3aoPSw2nV65DZUsTjr9WnTMiN/Zhqx8M0?= =?us-ascii?Q?ltjBqHSU3GzqSwnBKTQnjiGYykJltTYO+S7Kzbh4VLmmaxHPBEvGjCpBsxJf?= =?us-ascii?Q?FnWo3xt+aIk/m6tgpS7wJYhNM5MSxpaA+bNx/FEMppQ0/UPKpF9v74IiLeYM?= =?us-ascii?Q?NqTrZfG0DtCk+2SmcEBaA49c01ioauQfDSkY3pAKZ+dnrukitG+ve2693UK4?= =?us-ascii?Q?mYF7VYvv4BKL8/VmF+n8wNM0o0SCb+5vSolrs5GM68BbvDH5qKIFQI2NZDdT?= =?us-ascii?Q?v4Xm++o8+79hEjjF8H/Rd9wCI33WSegGVZiRyNsLCBs4cAlD2f0fnV/aek6K?= =?us-ascii?Q?uFHcP9ba9m5vSEFD1X1b9VeUdPpNFjt+xyKKKf/bLZAR+18ChzA5t3gar3IS?= =?us-ascii?Q?0gD73CojG0C2CcztjdMYiH9H+W9xsdU+gbx5EJFFFjVT2afg9CRrJa9v7Dnx?= =?us-ascii?Q?inGs8xLL12cy0I0wXAfC6x/OnYSj11tUa//2UeoFuGVm/+yMT4P9mOx+WnrI?= =?us-ascii?Q?WZ9jL8JxxBNj5Og2rG11ykre36uKneU5842wWwMwAOkgJiNqQNM4DO7iItR0?= =?us-ascii?Q?gs4vpgKrRIgsW65UKTaPr+DliZ1dp62XFRd6SIlv4VZJVAAjW6rxSHKN0l1N?= =?us-ascii?Q?ugcKWkdtn6ibC4EnqBbcTSpBiLSKB69O4Qog+e7ZxbYaiEADYzi4EsV8leCC?= =?us-ascii?Q?jOSCfRlnsLYtcAdgfQnyC9rHC+s8PtR3/XeMTM/Aj77BUoySJcjvFYH26Oh8?= =?us-ascii?Q?HmQKZTYk8P5A+00EDC7V8Vk+q72+ZcbxI+prjr/wJ4YpRJoYdDjBVqPr0qXc?= =?us-ascii?Q?X6xfF0gQrkj920mr+YysUM+f6Bpmlai7m1bFLPnwkuJlpA9qa7NXVQFbSi15?= =?us-ascii?Q?WCSJE595ErYpH+U5T+FchIs7Cevao+EZinSEkETbJA99s/G1H5Z5cv4kGyrV?= =?us-ascii?Q?mkHRzD6fI2P0C+otb52x3DSeOj2m3KASVbq29A8qDAresuhgJV6qmMd80vzJ?= =?us-ascii?Q?iJ3z0LHxvOUdlHiNnOk34TAKmHo8pS4+0y3phPGiTCwLaEn4IrTQvLEnTiw0?= =?us-ascii?Q?SA9Nzw=3D=3D?= X-Forefront-Antispam-Report: CIP:216.228.118.232;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc7edge1.nvidia.com;CAT:NONE;SFS:(13230040)(376014)(7416014)(36860700013)(1800799024)(82310400026);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Jan 2026 03:09:46.8875 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: e9e46c47-632b-43f3-942f-08de5d5186e4 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.118.232];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CH1PEPF0000AD7C.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR12MB6307 Content-Type: text/plain; charset="utf-8" From: Jason Gunthorpe Create a new data structure to hold an array of invalidations that need to be performed for the domain based on what masters are attached, to replace the single smmu pointer and linked list of masters in the current design. Each array entry holds one of the invalidation actions - S1_ASID, S2_VMID, ATS or their variant with information to feed invalidation commands to HW. It is structured so that multiple SMMUs can participate in the same array, removing one key limitation of the current system. To maximize performance, a sorted array is used as the data structure. It allows grouping SYNCs together to parallelize invalidations. For instance, it will group all the ATS entries after the ASID/VMID entry, so they will all be pushed to the PCI devices in parallel with one SYNC. To minimize the locking cost on the invalidation fast path (reader of the invalidation array), the array is managed with RCU. Provide a set of APIs to add/delete entries to/from an array, which cover cannot-fail attach cases, e.g. attaching to arm_smmu_blocked_domain. Also add kunit coverage for those APIs. Signed-off-by: Jason Gunthorpe Reviewed-by: Jason Gunthorpe Reviewed-by: Pranjal Shrivastava Co-developed-by: Nicolin Chen Signed-off-by: Nicolin Chen --- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 97 +++++++ .../iommu/arm/arm-smmu-v3/arm-smmu-v3-test.c | 92 ++++++ drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 265 ++++++++++++++++++ 3 files changed, 454 insertions(+) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/ar= m/arm-smmu-v3/arm-smmu-v3.h index cfbedb76c8ba..ed8820f12ba3 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h @@ -648,6 +648,93 @@ struct arm_smmu_cmdq_batch { int num; }; =20 +/* + * The order here also determines the sequence in which commands are sent = to the + * command queue. E.g. TLBI must be done before ATC_INV. + */ +enum arm_smmu_inv_type { + INV_TYPE_S1_ASID, + INV_TYPE_S2_VMID, + INV_TYPE_S2_VMID_S1_CLEAR, + INV_TYPE_ATS, + INV_TYPE_ATS_FULL, +}; + +struct arm_smmu_inv { + struct arm_smmu_device *smmu; + u8 type; + u8 size_opcode; + u8 nsize_opcode; + u32 id; /* ASID or VMID or SID */ + union { + size_t pgsize; /* ARM_SMMU_FEAT_RANGE_INV */ + u32 ssid; /* INV_TYPE_ATS */ + }; + + int users; /* users=3D0 to mark as a trash to be purged */ +}; + +static inline bool arm_smmu_inv_is_ats(const struct arm_smmu_inv *inv) +{ + return inv->type =3D=3D INV_TYPE_ATS || inv->type =3D=3D INV_TYPE_ATS_FUL= L; +} + +/** + * struct arm_smmu_invs - Per-domain invalidation array + * @max_invs: maximum capacity of the flexible array + * @num_invs: number of invalidations in the flexible array. May be smalle= r than + * @max_invs after a tailing trash entry is excluded, but must = not be + * greater than @max_invs + * @num_trashes: number of trash entries in the array for arm_smmu_invs_pu= rge(). + * Must not be greater than @num_invs + * @rwlock: optional rwlock to fench ATS operations + * @has_ats: flag if the array contains an INV_TYPE_ATS or INV_TYPE_ATS_FU= LL + * @rcu: rcu head for kfree_rcu() + * @inv: flexible invalidation array + * + * The arm_smmu_invs is an RCU data structure. During a ->attach_dev callb= ack, + * arm_smmu_invs_merge(), arm_smmu_invs_unref() and arm_smmu_invs_purge() = will + * be used to allocate a new copy of an old array for addition and deletio= n in + * the old domain's and new domain's invs arrays. + * + * The arm_smmu_invs_unref() mutates a given array, by internally reducing= the + * users counts of some given entries. This exists to support a no-fail ro= utine + * like attaching to an IOMMU_DOMAIN_BLOCKED. And it could pair with a fol= lowup + * arm_smmu_invs_purge() call to generate a new clean array. + * + * Concurrent invalidation thread will push every invalidation described i= n the + * array into the command queue for each invalidation event. It is designe= d like + * this to optimize the invalidation fast path by avoiding locks. + * + * A domain can be shared across SMMU instances. When an instance gets rem= oved, + * it would delete all the entries that belong to that SMMU instance. Then= , a + * synchronize_rcu() would have to be called to sync the array, to prevent= any + * concurrent invalidation thread accessing the old array from issuing com= mands + * to the command queue of a removed SMMU instance. + */ +struct arm_smmu_invs { + size_t max_invs; + size_t num_invs; + size_t num_trashes; + rwlock_t rwlock; + bool has_ats; + struct rcu_head rcu; + struct arm_smmu_inv inv[] __counted_by(max_invs); +}; + +static inline struct arm_smmu_invs *arm_smmu_invs_alloc(size_t num_invs) +{ + struct arm_smmu_invs *new_invs; + + new_invs =3D kzalloc(struct_size(new_invs, inv, num_invs), GFP_KERNEL); + if (!new_invs) + return NULL; + new_invs->max_invs =3D num_invs; + new_invs->num_invs =3D num_invs; + rwlock_init(&new_invs->rwlock); + return new_invs; +} + struct arm_smmu_evtq { struct arm_smmu_queue q; struct iopf_queue *iopf; @@ -873,6 +960,8 @@ struct arm_smmu_domain { =20 struct iommu_domain domain; =20 + struct arm_smmu_invs __rcu *invs; + /* List of struct arm_smmu_master_domain */ struct list_head devices; spinlock_t devices_lock; @@ -925,6 +1014,12 @@ void arm_smmu_make_cdtable_ste(struct arm_smmu_ste *t= arget, void arm_smmu_make_sva_cd(struct arm_smmu_cd *target, struct arm_smmu_master *master, struct mm_struct *mm, u16 asid); + +struct arm_smmu_invs *arm_smmu_invs_merge(struct arm_smmu_invs *invs, + struct arm_smmu_invs *to_merge); +void arm_smmu_invs_unref(struct arm_smmu_invs *invs, + struct arm_smmu_invs *to_unref); +struct arm_smmu_invs *arm_smmu_invs_purge(struct arm_smmu_invs *invs); #endif =20 struct arm_smmu_master_domain { @@ -958,6 +1053,8 @@ struct arm_smmu_domain *arm_smmu_domain_alloc(void); =20 static inline void arm_smmu_domain_free(struct arm_smmu_domain *smmu_domai= n) { + /* No concurrency with invalidation is possible at this point */ + kfree(rcu_dereference_protected(smmu_domain->invs, true)); kfree(smmu_domain); } =20 diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-test.c b/drivers/iom= mu/arm/arm-smmu-v3/arm-smmu-v3-test.c index 69c9ef441fc1..7b8035b1db24 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-test.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-test.c @@ -637,6 +637,97 @@ static void arm_smmu_v3_write_cd_test_sva_release(stru= ct kunit *test) NUM_EXPECTED_SYNCS(2)); } =20 +static void arm_smmu_v3_invs_test_verify(struct kunit *test, + struct arm_smmu_invs *invs, + int num_invs, const int num_trashes, + const int *ids, const int *users) +{ + KUNIT_EXPECT_EQ(test, invs->num_invs, num_invs); + KUNIT_EXPECT_EQ(test, invs->num_trashes, num_trashes); + while (num_invs--) { + KUNIT_EXPECT_EQ(test, invs->inv[num_invs].id, ids[num_invs]); + KUNIT_EXPECT_EQ(test, READ_ONCE(invs->inv[num_invs].users), + users[num_invs]); + } +} + +static struct arm_smmu_invs invs1 =3D { + .num_invs =3D 3, + .inv =3D { { .type =3D INV_TYPE_S2_VMID, .id =3D 1, }, + { .type =3D INV_TYPE_S2_VMID_S1_CLEAR, .id =3D 1, }, + { .type =3D INV_TYPE_ATS, .id =3D 3, }, }, +}; + +static struct arm_smmu_invs invs2 =3D { + .num_invs =3D 3, + .inv =3D { { .type =3D INV_TYPE_S2_VMID, .id =3D 1, }, /* duplicated */ + { .type =3D INV_TYPE_ATS, .id =3D 4, }, + { .type =3D INV_TYPE_ATS, .id =3D 5, }, }, +}; + +static struct arm_smmu_invs invs3 =3D { + .num_invs =3D 3, + .inv =3D { { .type =3D INV_TYPE_S2_VMID, .id =3D 1, }, /* duplicated */ + { .type =3D INV_TYPE_ATS, .id =3D 5, }, /* recover a trash */ + { .type =3D INV_TYPE_ATS, .id =3D 6, }, }, +}; + +static void arm_smmu_v3_invs_test(struct kunit *test) +{ + const int results1[2][3] =3D { { 1, 1, 3, }, { 1, 1, 1, }, }; + const int results2[2][5] =3D { { 1, 1, 3, 4, 5, }, { 2, 1, 1, 1, 1, }, }; + const int results3[2][3] =3D { { 1, 1, 3, }, { 1, 1, 1, }, }; + const int results4[2][5] =3D { { 1, 1, 3, 5, 6, }, { 2, 1, 1, 1, 1, }, }; + const int results5[2][5] =3D { { 1, 1, 3, 5, 6, }, { 1, 0, 0, 1, 1, }, }; + const int results6[2][3] =3D { { 1, 5, 6, }, { 1, 1, 1, }, }; + struct arm_smmu_invs *test_a, *test_b; + + /* New array */ + test_a =3D arm_smmu_invs_alloc(0); + KUNIT_EXPECT_EQ(test, test_a->num_invs, 0); + + /* Test1: merge invs1 (new array) */ + test_b =3D arm_smmu_invs_merge(test_a, &invs1); + kfree(test_a); + arm_smmu_v3_invs_test_verify(test, test_b, ARRAY_SIZE(results1[0]), 0, + results1[0], results1[1]); + + /* Test2: merge invs2 (new array) */ + test_a =3D arm_smmu_invs_merge(test_b, &invs2); + kfree(test_b); + arm_smmu_v3_invs_test_verify(test, test_a, ARRAY_SIZE(results2[0]), 0, + results2[0], results2[1]); + + /* Test3: unref invs2 (same array) */ + arm_smmu_invs_unref(test_a, &invs2); + arm_smmu_v3_invs_test_verify(test, test_a, ARRAY_SIZE(results3[0]), 0, + results3[0], results3[1]); + + /* Test4: merge invs3 (new array) */ + test_b =3D arm_smmu_invs_merge(test_a, &invs3); + kfree(test_a); + arm_smmu_v3_invs_test_verify(test, test_b, ARRAY_SIZE(results4[0]), 0, + results4[0], results4[1]); + + /* Test5: unref invs1 (same array) */ + arm_smmu_invs_unref(test_b, &invs1); + arm_smmu_v3_invs_test_verify(test, test_b, ARRAY_SIZE(results5[0]), 2, + results5[0], results5[1]); + + /* Test6: purge test_b (new array) */ + test_a =3D arm_smmu_invs_purge(test_b); + kfree(test_b); + arm_smmu_v3_invs_test_verify(test, test_a, ARRAY_SIZE(results6[0]), 0, + results6[0], results6[1]); + + /* Test7: unref invs3 (same array) */ + arm_smmu_invs_unref(test_a, &invs3); + KUNIT_EXPECT_EQ(test, test_a->num_invs, 0); + KUNIT_EXPECT_EQ(test, test_a->num_trashes, 0); + + kfree(test_a); +} + static struct kunit_case arm_smmu_v3_test_cases[] =3D { KUNIT_CASE(arm_smmu_v3_write_ste_test_bypass_to_abort), KUNIT_CASE(arm_smmu_v3_write_ste_test_abort_to_bypass), @@ -662,6 +753,7 @@ static struct kunit_case arm_smmu_v3_test_cases[] =3D { KUNIT_CASE(arm_smmu_v3_write_ste_test_nested_s1bypass_to_s1dssbypass), KUNIT_CASE(arm_smmu_v3_write_cd_test_sva_clear), KUNIT_CASE(arm_smmu_v3_write_cd_test_sva_release), + KUNIT_CASE(arm_smmu_v3_invs_test), {}, }; =20 diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/ar= m/arm-smmu-v3/arm-smmu-v3.c index 6beffa8e7c74..3f270c59f018 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -26,6 +26,7 @@ #include #include #include +#include #include #include #include @@ -1026,6 +1027,262 @@ static void arm_smmu_page_response(struct device *d= ev, struct iopf_fault *unused */ } =20 +/* Invalidation array manipulation functions */ +static inline struct arm_smmu_inv * +arm_smmu_invs_iter_next(struct arm_smmu_invs *invs, size_t next, size_t *i= dx) +{ + while (true) { + if (next >=3D invs->num_invs) { + *idx =3D next; + return NULL; + } + if (!READ_ONCE(invs->inv[next].users)) { + next++; + continue; + } + *idx =3D next; + return &invs->inv[next]; + } +} + +/** + * arm_smmu_invs_for_each_entry - Iterate over all non-trash entries in in= vs + * @invs: the base invalidation array + * @idx: a stack variable of 'size_t', to store the array index + * @cur: a stack variable of 'struct arm_smmu_inv *' + */ +#define arm_smmu_invs_for_each_entry(invs, idx, cur) = \ + for (cur =3D arm_smmu_invs_iter_next(invs, 0, &(idx)); cur; \ + cur =3D arm_smmu_invs_iter_next(invs, idx + 1, &(idx))) + +static int arm_smmu_inv_cmp(const struct arm_smmu_inv *inv_l, + const struct arm_smmu_inv *inv_r) +{ + if (inv_l->smmu !=3D inv_r->smmu) + return cmp_int((uintptr_t)inv_l->smmu, (uintptr_t)inv_r->smmu); + if (inv_l->type !=3D inv_r->type) + return cmp_int(inv_l->type, inv_r->type); + return cmp_int(inv_l->id, inv_r->id); +} + +static inline int arm_smmu_invs_iter_next_cmp(struct arm_smmu_invs *invs_l, + size_t next_l, size_t *idx_l, + struct arm_smmu_invs *invs_r, + size_t next_r, size_t *idx_r) +{ + struct arm_smmu_inv *cur_l =3D + arm_smmu_invs_iter_next(invs_l, next_l, idx_l); + + /* + * We have to update the idx_r manually, because the invs_r cannot call + * arm_smmu_invs_iter_next() as the invs_r never sets any users counter. + */ + *idx_r =3D next_r; + + /* + * Compare of two sorted arrays items. If one side is past the end of + * the array, return the other side to let it run out the iteration. + * + * If the left entry is empty, return 1 to pick the right entry. + * If the right entry is empty, return -1 to pick the left entry. + */ + if (!cur_l) + return 1; + if (next_r >=3D invs_r->num_invs) + return -1; + return arm_smmu_inv_cmp(cur_l, &invs_r->inv[next_r]); +} + +/** + * arm_smmu_invs_for_each_cmp - Iterate over two sorted arrays computing f= or + * arm_smmu_invs_merge() or arm_smmu_invs_unr= ef() + * @invs_l: the base invalidation array + * @idx_l: a stack variable of 'size_t', to store the base array index + * @invs_r: the build_invs array as to_merge or to_unref + * @idx_r: a stack variable of 'size_t', to store the build_invs index + * @cmp: a stack variable of 'int', to store return value (-1, 0, or 1) + */ +#define arm_smmu_invs_for_each_cmp(invs_l, idx_l, invs_r, idx_r, cmp) = \ + for (idx_l =3D idx_r =3D 0, = \ + cmp =3D arm_smmu_invs_iter_next_cmp(invs_l, 0, &(idx_l), \ + invs_r, 0, &(idx_r)); \ + idx_l < invs_l->num_invs || idx_r < invs_r->num_invs; \ + cmp =3D arm_smmu_invs_iter_next_cmp( \ + invs_l, idx_l + (cmp <=3D 0 ? 1 : 0), &(idx_l), \ + invs_r, idx_r + (cmp >=3D 0 ? 1 : 0), &(idx_r))) + +/** + * arm_smmu_invs_merge() - Merge @to_merge into @invs and generate a new a= rray + * @invs: the base invalidation array + * @to_merge: an array of invlidations to merge + * + * Return: a newly allocated array on success, or ERR_PTR + * + * This function must be locked and serialized with arm_smmu_invs_unref() = and + * arm_smmu_invs_purge(), but do not lockdep on any lock for KUNIT test. + * + * Both @invs and @to_merge must be sorted, to ensure the returned array w= ill be + * sorted as well. + * + * Caller is resposible for freeing the @invs and the returned new one. + * + * Entries marked as trash will be purged in the returned array. + */ +VISIBLE_IF_KUNIT +struct arm_smmu_invs *arm_smmu_invs_merge(struct arm_smmu_invs *invs, + struct arm_smmu_invs *to_merge) +{ + struct arm_smmu_invs *new_invs; + struct arm_smmu_inv *new; + size_t num_invs =3D 0; + size_t i, j; + int cmp; + + arm_smmu_invs_for_each_cmp(invs, i, to_merge, j, cmp) + num_invs++; + + new_invs =3D arm_smmu_invs_alloc(num_invs); + if (!new_invs) + return ERR_PTR(-ENOMEM); + + new =3D new_invs->inv; + arm_smmu_invs_for_each_cmp(invs, i, to_merge, j, cmp) { + if (cmp < 0) { + *new =3D invs->inv[i]; + } else if (cmp =3D=3D 0) { + *new =3D invs->inv[i]; + WRITE_ONCE(new->users, READ_ONCE(new->users) + 1); + } else { + *new =3D to_merge->inv[j]; + WRITE_ONCE(new->users, 1); + } + + /* + * Check that the new array is sorted. This also validates that + * to_merge is sorted. + */ + if (new !=3D new_invs->inv) + WARN_ON_ONCE(arm_smmu_inv_cmp(new - 1, new) =3D=3D 1); + if (arm_smmu_inv_is_ats(new)) + new_invs->has_ats =3D true; + new++; + } + + WARN_ON(new !=3D new_invs->inv + new_invs->num_invs); + + return new_invs; +} +EXPORT_SYMBOL_IF_KUNIT(arm_smmu_invs_merge); + +/** + * arm_smmu_invs_unref() - Find in @invs for all entries in @to_unref, dec= rease + * the user counts without deletions + * @invs: the base invalidation array + * @to_unref: an array of invlidations to decrease their user counts + * + * Return: the number of trash entries in the array, for arm_smmu_invs_pur= ge() + * + * This function will not fail. Any entry with users=3D0 will be marked as= trash, + * and caller will be notified about the trashed entry via @to_unref by se= tting + * a users=3D0. + * + * All tailing trash entries in the array will be dropped. And the size of= the + * array will be trimmed properly. All trash entries in-between will remai= n in + * the @invs until being completely deleted by the next arm_smmu_invs_merg= e() + * or an arm_smmu_invs_purge() function call. + * + * This function must be locked and serialized with arm_smmu_invs_merge() = and + * arm_smmu_invs_purge(), but do not lockdep on any mutex for KUNIT test. + * + * Note that the final @invs->num_invs might not reflect the actual number= of + * invalidations due to trash entries. Any reader should take the read loc= k to + * iterate each entry and check its users counter till the last entry. + */ +VISIBLE_IF_KUNIT +void arm_smmu_invs_unref(struct arm_smmu_invs *invs, + struct arm_smmu_invs *to_unref) +{ + unsigned long flags; + size_t num_invs =3D 0; + size_t i, j; + int cmp; + + arm_smmu_invs_for_each_cmp(invs, i, to_unref, j, cmp) { + if (cmp < 0) { + /* not found in to_unref, leave alone */ + WRITE_ONCE(to_unref->inv[j].users, 1); + num_invs =3D i + 1; + } else if (cmp =3D=3D 0) { + int users =3D READ_ONCE(invs->inv[i].users) - 1; + + if (WARN_ON(users < 0)) + continue; + + /* same item */ + WRITE_ONCE(invs->inv[i].users, users); + if (users) { + WRITE_ONCE(to_unref->inv[j].users, 1); + num_invs =3D i + 1; + continue; + } + + /* Notify the caller about the trash entry */ + WRITE_ONCE(to_unref->inv[j].users, 0); + invs->num_trashes++; + } else { + /* item in to_unref is not in invs or already a trash */ + WARN_ON(true); + } + } + + /* Exclude any tailing trash */ + invs->num_trashes -=3D invs->num_invs - num_invs; + + /* The lock is required to fence concurrent ATS operations. */ + write_lock_irqsave(&invs->rwlock, flags); + WRITE_ONCE(invs->num_invs, num_invs); /* Remove tailing trash entries */ + write_unlock_irqrestore(&invs->rwlock, flags); +} +EXPORT_SYMBOL_IF_KUNIT(arm_smmu_invs_unref); + +/** + * arm_smmu_invs_purge() - Purge all the trash entries in the @invs + * @invs: the base invalidation array + * + * Return: a newly allocated array on success removing all the trash entri= es, or + * NULL if there is no trash entry in the array or there is a bug + * + * This function must be locked and serialized with arm_smmu_invs_merge() = and + * arm_smmu_invs_unref(), but do not lockdep on any lock for KUNIT test. + * + * Caller is resposible for freeing the @invs and the returned new one. + */ +VISIBLE_IF_KUNIT +struct arm_smmu_invs *arm_smmu_invs_purge(struct arm_smmu_invs *invs) +{ + struct arm_smmu_invs *new_invs; + struct arm_smmu_inv *inv; + size_t i, num_invs =3D 0; + + if (WARN_ON(invs->num_invs < invs->num_trashes)) + return NULL; + if (!invs->num_invs || !invs->num_trashes) + return NULL; + + new_invs =3D arm_smmu_invs_alloc(invs->num_invs - invs->num_trashes); + if (!new_invs) + return ERR_PTR(-ENOMEM); + + arm_smmu_invs_for_each_entry(invs, i, inv) { + new_invs->inv[num_invs] =3D *inv; + num_invs++; + } + + WARN_ON(num_invs !=3D new_invs->num_invs); + return new_invs; +} +EXPORT_SYMBOL_IF_KUNIT(arm_smmu_invs_purge); + /* Context descriptor manipulation functions */ void arm_smmu_tlb_inv_asid(struct arm_smmu_device *smmu, u16 asid) { @@ -2530,13 +2787,21 @@ static bool arm_smmu_enforce_cache_coherency(struct= iommu_domain *domain) struct arm_smmu_domain *arm_smmu_domain_alloc(void) { struct arm_smmu_domain *smmu_domain; + struct arm_smmu_invs *new_invs; =20 smmu_domain =3D kzalloc(sizeof(*smmu_domain), GFP_KERNEL); if (!smmu_domain) return ERR_PTR(-ENOMEM); =20 + new_invs =3D arm_smmu_invs_alloc(0); + if (!new_invs) { + kfree(smmu_domain); + return ERR_PTR(-ENOMEM); + } + INIT_LIST_HEAD(&smmu_domain->devices); spin_lock_init(&smmu_domain->devices_lock); + rcu_assign_pointer(smmu_domain->invs, new_invs); =20 return smmu_domain; } --=20 2.43.0 From nobody Sat Feb 7 11:52:10 2026 Received: from CH4PR04CU002.outbound.protection.outlook.com (mail-northcentralusazon11013068.outbound.protection.outlook.com [40.107.201.68]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A8DD32FD7A3 for ; Tue, 27 Jan 2026 03:09:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.201.68 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1769483393; cv=fail; b=OWBGOhyEwAAX7+H2KsvmjdEhAsbxDdl80S49QNbnkbta1aqvY5NjaqYXALvALcGMA+FHEWXQBphMuIOZ248LcPiDeDuB+R+09YdYEQTNgohDiUvGue3b0JekBA+Cz81GQxSNPuJI7r/Oi8/GqQVIldQ4k1Yb18u370jO4ZRbvsE= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1769483393; c=relaxed/simple; bh=uwHejIAKgENMtgk31QVWC6hzcFZDiRqrtkB2HicB5zI=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=CskCXKtqV8VP0vS6zzJRpkJQ6O/r3sb0zZeopU0n3HkPi9NgDxP2rKlggnPHuCHRIFzGVLu2riqck6CT45c8LEsDrNRNXJgqARh/nYIf0RpvJCk6vYjvgUfWBsKvbBIYC364nGV7DGy9WvkPQviSqlLzS+Eqinh7j3e9epqDx28= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=ZxLsPqIp; arc=fail smtp.client-ip=40.107.201.68 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="ZxLsPqIp" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=kD1QNelG+IahQaNgqlOdeph4Ggvu/LyyiNA85QybFqWpEKz+J55V9Cxil+YaMKhoSro8FiKvTFgoBmGw0ddsk2ZyvxO6j2fwNSMRGmXCCuahb52z3u5Y4rga7HdoIxW3d+JklKT/U9p6Oh+Mj1g4FYoLY7oG1X9HZSEiBj6RQQ2QmjqXtJQ9qlOmgJcm6NMde1u0kkdDqaIDHLjK7ilN5jXJN/IOCbrHELZ79gfm4piBuoTMurGfd2nzgVAVTxaFh5FfhXp7GZrkC8TV7Q9DDMoc0XFxp1m4tF1Rzen8HvJSReJi80H4V7dhQElLz0vyqAIOOJJFenuxTaieblXSlQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=n461Kx0eJMocvVPeg5efp0FLTkulazUFstAKfq71wwQ=; b=uudSk62Riku1714aSiL0lLUtHp5vGvGdM4+dGNuh40hHusJD46IUDPhxzP3DmOQY1wnGBJxsYThE3xmR3ZGgEC8Py0dV7x4nFrUFreIRwoC5KeHM1RNwqvL7VtCUWoSFPigDDj6HJCN0DzL+6wg78MH63v2cnm9ScOs4NW9cj85hR9hRr6tx8hBSFJT1roVqTrtxynseSQZGWDqNRasQjZ/mKogn9RXX4RyHeKgr9B/buLIjgpuVgBvak36CMAYhUqGrO6YKnxaDcufXDP0k/Ai9D7W9cmbEvlHLd6LIWv+WEaBcuQxFz347CwASWaaJngyr/tAuPDp8z/KFQWlMHw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.232) smtp.rcpttodomain=kernel.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=n461Kx0eJMocvVPeg5efp0FLTkulazUFstAKfq71wwQ=; b=ZxLsPqIpgrIqaqfEw7zAX5qTaUlMBnGxSc2MFKxF/wX6MMb2ocIvWdjF9XZwBsq89Gb4TGaMw+LoKVo/HC4o4en5PotVxWUcLQsHch++A07o0NiF+qcYeAHXqDYs0KgU0e96+iQk4OPf+pGPvrBzgq08vbvkRFOxouVivqWgozJgCFE41N+YN9HAHVW8a7a6kqdRKHYjKvbP0mEsyMKPHB+JNyNdG4TRoqNBicXfH2FOuQbpuI/Liq1nlfvO+kOHJIuD6vs8aHwU6fOh3QNak5YfY5EpBuAtohyOOrIdItNpsI155uTKQV3rK5162MD7l2PQG8iax8hewjXptTLtnw== Received: from CH0PR03CA0414.namprd03.prod.outlook.com (2603:10b6:610:11b::32) by DM4PR12MB5796.namprd12.prod.outlook.com (2603:10b6:8:63::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9542.16; Tue, 27 Jan 2026 03:09:48 +0000 Received: from CH1PEPF0000AD83.namprd04.prod.outlook.com (2603:10b6:610:11b::4) by CH0PR03CA0414.outlook.office365.com (2603:10b6:610:11b::32) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.9542.16 via Frontend Transport; Tue, 27 Jan 2026 03:09:42 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.232) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.232 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.232; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.118.232) by CH1PEPF0000AD83.mail.protection.outlook.com (10.167.244.85) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9564.3 via Frontend Transport; Tue, 27 Jan 2026 03:09:47 +0000 Received: from drhqmail203.nvidia.com (10.126.190.182) by mail.nvidia.com (10.127.129.5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.20; Mon, 26 Jan 2026 19:09:38 -0800 Received: from drhqmail201.nvidia.com (10.126.190.180) by drhqmail203.nvidia.com (10.126.190.182) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.20; Mon, 26 Jan 2026 19:09:37 -0800 Received: from Asurada-Nvidia.nvidia.com (10.127.8.14) by mail.nvidia.com (10.126.190.180) with Microsoft SMTP Server id 15.2.2562.20 via Frontend Transport; Mon, 26 Jan 2026 19:09:37 -0800 From: Nicolin Chen To: CC: , , , , , , , , , , , Subject: [PATCH v10 5/8] iommu/arm-smmu-v3: Pre-allocate a per-master invalidation array Date: Mon, 26 Jan 2026 19:09:16 -0800 Message-ID: <5270d84e09276a8368b413a2d9ef69deea03474b.1769476588.git.nicolinc@nvidia.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH1PEPF0000AD83:EE_|DM4PR12MB5796:EE_ X-MS-Office365-Filtering-Correlation-Id: a77b1f4b-bd76-4926-55b4-08de5d51878d X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|7416014|36860700013|1800799024|376014|82310400026; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?mjDPT1iLFzcdP+Mgt9N1CWV1IMViVGoQ8BuLVouEFcpia1fjxjJBb9Z29BI+?= =?us-ascii?Q?a4QJk4OHOj8RwkGkKCvut3ecRVwPHEIhmB78VQyh+2csqwLVkBvsgUisC0MF?= =?us-ascii?Q?FpwkUVcI4PRFpnQANnLA+cMudwRpmG/V3K49KFzriLwOWlJx+oeFqPVZiIxv?= =?us-ascii?Q?Sy7HbMZcZJ+0fc8UofyX3ChPRZJ9e98L0iT0eot9Zr7gss+luHMoaI6olGay?= =?us-ascii?Q?Tl8Oi/SWtpblqPZRD8snRLafE2cef0VVQ+CrgDkwzArHN3+AiuBSXvXwB0DB?= =?us-ascii?Q?aCwWvkPO14TM1uBtz+E1m041Cj5+8klD2AnA2etXha7dZXpPCfCt5dTgf+yF?= =?us-ascii?Q?pVxYjiUckZfHZTa1LI5ZIQAwi8smCJ5aFLKz83PbMLqcboz9/0WFJuMhpLNm?= =?us-ascii?Q?llxvnjxPVB0W6g7KipG/lzLWT90ltnK13yqzov+wdgj4R/YlVjFARHRrW8kF?= =?us-ascii?Q?2YrnlT8mXU7IQp4eaIg+ZroUCy89IPEKsf/w92H3qD+rzh3HhuVyZTtOiurl?= =?us-ascii?Q?YYknM+4+x9KLp4G6re71zuydzyIGwadTKBY15DqHBxXqM4RosiLcIAdA/5vw?= =?us-ascii?Q?2+jsiBnOoxEGuo9+Wz12HD094TIzo4RPQaYLs9PMBytiZQ93zs9b33PcR7CR?= =?us-ascii?Q?XkMW+gnb+dtpCRBkmF4Tr4FXrmswmgwssa8DOHeuPwuVDCokFT4bHyKes2Ns?= =?us-ascii?Q?5MuqZeAz30/8RNAcsv9IS5jVJ9dCga57WybRHiVMIjSv+Nh4dwJH9CTx+Nnx?= =?us-ascii?Q?3OYoWtOT0GGLt7HtoNuF8to7qr+spfTS6T/YS7k5jo5pjW6NwSPpI7V7FLyL?= =?us-ascii?Q?FcTAx2PUDR8xIkzRGVPvLaLiDRMm+ey/8XLcSjH7tIKdoW1kdv+FNztHrxZO?= =?us-ascii?Q?VeUwwIrP8SgrkwgAucsG6V3gDKGb55AvSgEnxutaujoDPmEdaQQ6nme5wqzl?= =?us-ascii?Q?ef+sgERoeIG7ssr51cM6ulOLeQhQ0uckZf+PYW3Ody4MttW/2xJUsIYTr6Kx?= =?us-ascii?Q?eKy64lQ3wGI6huwHLaMQ5ioZLI8uDpBma69OPZq8CI52BxxwZiQrTxRJJbIV?= =?us-ascii?Q?BikF9yGOQZ+OnACtAtyjaDLig/KStTnsBGJ2+rqnhjxx8aEvGR7CuNZa9WHE?= =?us-ascii?Q?NcCHq8f7G76xZXOvOfG3SoHW5dbvYyZA4xZQL6seJtuz0d5tPRA8BpA5doJu?= =?us-ascii?Q?QOW8j73ITZCo8W7kBcB7fQ7ezaYIRk0lF0qxkh7RJ1JXdfza8y5nk2ggybhG?= =?us-ascii?Q?ZNMUo/tKUn5FTZFNkZr9PnG4hhSe3Ol3/vDi9brfUX4rK50uNhwRTYV9Agoy?= =?us-ascii?Q?4RizCdYftJ8swOdc+DhDzh0MvnqrQOakBW0yOYXwnew5cdg2O/ibi+DK6WxB?= =?us-ascii?Q?9VLJIjtn0jiOktIVd/mtoZY1PaSplj/IdN8nFPa9Ru+uY/VunTMJ5/e3wST7?= =?us-ascii?Q?FJpcFlYFG97iW4+EQCOm//6EWd+Kiizksn5kd5FsMUNhULF2cCmsw1WPod11?= =?us-ascii?Q?e+tQiuMHmMJhI1iiad3mXSkOsXzBh7dy36TpN7a4gQi6rwIQa99yM5bJ2xZ5?= =?us-ascii?Q?hyMfjmTOsHVXDrtBGChLwVB/2sN6cOOqycs2/syAYZPmlFQMSP9jnAA1kZ66?= =?us-ascii?Q?lWiYHFj+Z2PlxC9VjR+7Mf0PSTQN94dNrh7SQD/NTUO0WKA5mIoJo1nttL0x?= =?us-ascii?Q?tWwtAw=3D=3D?= X-Forefront-Antispam-Report: CIP:216.228.118.232;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc7edge1.nvidia.com;CAT:NONE;SFS:(13230040)(7416014)(36860700013)(1800799024)(376014)(82310400026);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Jan 2026 03:09:47.9629 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: a77b1f4b-bd76-4926-55b4-08de5d51878d X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.118.232];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CH1PEPF0000AD83.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM4PR12MB5796 Content-Type: text/plain; charset="utf-8" When a master is attached from an old domain to a new domain, it needs to build an invalidation array to delete and add the array entries from/onto the invalidation arrays of those two domains, passed via the to_merge and to_unref arguments into arm_smmu_invs_merge/unref() respectively. Since the master->num_streams might differ across masters, a memory would have to be allocated when building an to_merge/to_unref array which might fail with -ENOMEM. On the other hand, an attachment to arm_smmu_blocked_domain must not fail so it's the best to avoid any memory allocation in that path. Pre-allocate a fixed size invalidation array for every master. This array will be used as a scratch to fill dynamically when building a to_merge or to_unref invs array. Sort fwspec->ids in an ascending order to fit to the arm_smmu_invs_merge() function. Co-developed-by: Jason Gunthorpe Signed-off-by: Jason Gunthorpe Reviewed-by: Jason Gunthorpe Reviewed-by: Pranjal Shrivastava Signed-off-by: Nicolin Chen --- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 8 ++++ drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 41 +++++++++++++++++++-- 2 files changed, 45 insertions(+), 4 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/ar= m/arm-smmu-v3/arm-smmu-v3.h index ed8820f12ba3..5e0e5055af1e 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h @@ -928,6 +928,14 @@ struct arm_smmu_master { struct arm_smmu_device *smmu; struct device *dev; struct arm_smmu_stream *streams; + /* + * Scratch memory for a to_merge or to_unref array to build a per-domain + * invalidation array. It'll be pre-allocated with enough enries for all + * possible build scenarios. It can be used by only one caller at a time + * until the arm_smmu_invs_merge/unref() finishes. Must be locked by the + * iommu_group mutex. + */ + struct arm_smmu_invs *build_invs; struct arm_smmu_vmaster *vmaster; /* use smmu->streams_mutex */ /* Locked by the iommu core using the group mutex */ struct arm_smmu_ctx_desc_cfg cd_table; diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/ar= m/arm-smmu-v3/arm-smmu-v3.c index 3f270c59f018..5a0a8b136352 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -3784,12 +3784,22 @@ static int arm_smmu_init_sid_strtab(struct arm_smmu= _device *smmu, u32 sid) return 0; } =20 +static int arm_smmu_stream_id_cmp(const void *_l, const void *_r) +{ + const typeof_member(struct arm_smmu_stream, id) *l =3D _l; + const typeof_member(struct arm_smmu_stream, id) *r =3D _r; + + return cmp_int(*l, *r); +} + static int arm_smmu_insert_master(struct arm_smmu_device *smmu, struct arm_smmu_master *master) { int i; int ret =3D 0; struct iommu_fwspec *fwspec =3D dev_iommu_fwspec_get(master->dev); + bool ats_supported =3D dev_is_pci(master->dev) && + pci_ats_supported(to_pci_dev(master->dev)); =20 master->streams =3D kcalloc(fwspec->num_ids, sizeof(*master->streams), GFP_KERNEL); @@ -3797,14 +3807,35 @@ static int arm_smmu_insert_master(struct arm_smmu_d= evice *smmu, return -ENOMEM; master->num_streams =3D fwspec->num_ids; =20 - mutex_lock(&smmu->streams_mutex); + if (!ats_supported) { + /* Base case has 1 ASID entry or maximum 2 VMID entries */ + master->build_invs =3D arm_smmu_invs_alloc(2); + } else { + /* ATS case adds num_ids of entries, on top of the base case */ + master->build_invs =3D arm_smmu_invs_alloc(2 + fwspec->num_ids); + } + if (!master->build_invs) { + kfree(master->streams); + return -ENOMEM; + } + for (i =3D 0; i < fwspec->num_ids; i++) { struct arm_smmu_stream *new_stream =3D &master->streams[i]; - struct rb_node *existing; - u32 sid =3D fwspec->ids[i]; =20 - new_stream->id =3D sid; + new_stream->id =3D fwspec->ids[i]; new_stream->master =3D master; + } + + /* Put the ids into order for sorted to_merge/to_unref arrays */ + sort_nonatomic(master->streams, master->num_streams, + sizeof(master->streams[0]), arm_smmu_stream_id_cmp, + NULL); + + mutex_lock(&smmu->streams_mutex); + for (i =3D 0; i < fwspec->num_ids; i++) { + struct arm_smmu_stream *new_stream =3D &master->streams[i]; + struct rb_node *existing; + u32 sid =3D new_stream->id; =20 ret =3D arm_smmu_init_sid_strtab(smmu, sid); if (ret) @@ -3834,6 +3865,7 @@ static int arm_smmu_insert_master(struct arm_smmu_dev= ice *smmu, for (i--; i >=3D 0; i--) rb_erase(&master->streams[i].node, &smmu->streams); kfree(master->streams); + kfree(master->build_invs); } mutex_unlock(&smmu->streams_mutex); =20 @@ -3855,6 +3887,7 @@ static void arm_smmu_remove_master(struct arm_smmu_ma= ster *master) mutex_unlock(&smmu->streams_mutex); =20 kfree(master->streams); + kfree(master->build_invs); } =20 static struct iommu_device *arm_smmu_probe_device(struct device *dev) --=20 2.43.0 From nobody Sat Feb 7 11:52:10 2026 Received: from DM5PR21CU001.outbound.protection.outlook.com (mail-centralusazon11011035.outbound.protection.outlook.com [52.101.62.35]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1464530103F for ; Tue, 27 Jan 2026 03:09:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=52.101.62.35 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1769483398; cv=fail; b=PdVTRGEEPb3RSuw8whcDyvGZeef07RT8WSs9aXeDfgPW/vS/aFIA+5c7L2DLQioBvU5f6RLMG1COz5pMqKT6Bs2CDMXr65pQ9EunZqy0M+c6QAhl6ugPSp5dvM8jLtIo42cvcAQ7sNw01R9QfozIatLtUaRJqES6EJYRdiW/vKQ= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1769483398; c=relaxed/simple; bh=IkHFUbTnRo/zRIQgiPyQKeoZLCMo2+1mhh7MBzXvQMQ=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=pX3COksudnvPNFOJmBUsbP1GLM+UZFRZESPEjmIECe6vwuXt7E/BFfiNhR2j0smVEWdzz8i7QaEmg2oMnY5TnsHaRy2Pd19KVEOyyNozkfV5BahDtplDitxPNa/rErCGvO5/U+Eib4ipwgLopD/gE5vS0WruJgQTahkcSJrHfE4= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=Q6lm3uoB; arc=fail smtp.client-ip=52.101.62.35 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="Q6lm3uoB" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=DnWPMsBbC5ZPx3+c19zNBIyBycRAmcYICRjdSqR7ZYe0LDcDBWg9O0fiVi3c94kSrh9ID7Ngo2259Ifb29OvUt+y2viPcgtwk+Iu7ml6V/TXh2FRh5rvGjW5xeXuKNGcY3Ww2b1D52qRJ1lgCckcd5ZFqMvk3idJdgPnVyP5YS+bRQ00CyYAJGu/B0o/dr+QJAU7z/U+BOyvMT2n20zA3yiX0alH9tPWodPU1yUKVLDRVdHfXwFTu59sI1x8dHxYw+PY8u7Nw5/eDUCL6k5Z4X57D3H4CVqSGrMc3pgxCn0HcnR1fnhWmiAdaqjJqUkaUw95U+VxL+x1/LJMUdByjw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Qz3jKgI7Md+uqar7u2XPjC9ELAnk2UI/e5V+u4cvZuc=; b=Sa5ahsR/7MVaA23NNRkiWPvMWcZh/s2BXFSKiH3myw5Xrexhf1WBTkPQERuwbBzp0jpofKp69MC32xk5D0ID3EYbSu02X1Gp7IDfpjm6MUHLs4b5mz3aef0/T8AWYqv3oraXI7q+GKaJ5WUdworV2NiL4rBh/Z3/FnJLNQVXiWuEEVz9wbXTy3cx9xXjVXWOejP57r7qo1Q+VBbsC8hGFG84VHKYjbf7fTsZZZYbpg2KDqIJQDuKhaJsuIk7J/0I2BUwY0v9jAeAztgn82K6694OgTvhnrGUbhAHosfN59pJPa8LucXHvdrvuCDXQnaeuFLl3b+nPRid4VL9r7ms4w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.232) smtp.rcpttodomain=kernel.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Qz3jKgI7Md+uqar7u2XPjC9ELAnk2UI/e5V+u4cvZuc=; b=Q6lm3uoB9PJde8wDZAMdfpCWmNCy3EmhZD0LU1+BNE5rO26r3H+WNkn9fyBv2X11PBxG565pnRUoHqaMXY7qaRzpN0QV5h7hfZS4rrXKuUDztA3wh+YAbph+Tj2IQ3wSSiXWwN6l0aJZjTabYaqL6n0g7vujT+zeYsz3AQ62SuRZF4VPwbK8Ec38epyAd3yZrm3gOD2FrU2SF8UBVS5u2VNHTZbDMbWO9JraxOF0gekNLx7nwTQVRhmjS5IhJfZoF2W9NPMzkWbhpk5JtFq1Qjx+GBzC3MJhVf/Z2oxkfUKU5EX5OZLQLV+BCC1qxWj9ZtxzXlQ9MHQ7zahDWV2eGQ== Received: from CH0PR03CA0414.namprd03.prod.outlook.com (2603:10b6:610:11b::32) by DS7PR12MB5936.namprd12.prod.outlook.com (2603:10b6:8:7f::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9542.15; Tue, 27 Jan 2026 03:09:50 +0000 Received: from CH1PEPF0000AD83.namprd04.prod.outlook.com (2603:10b6:610:11b:cafe::10) by CH0PR03CA0414.outlook.office365.com (2603:10b6:610:11b::32) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.9542.16 via Frontend Transport; Tue, 27 Jan 2026 03:09:45 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.232) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.232 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.232; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.118.232) by CH1PEPF0000AD83.mail.protection.outlook.com (10.167.244.85) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9564.3 via Frontend Transport; Tue, 27 Jan 2026 03:09:50 +0000 Received: from drhqmail202.nvidia.com (10.126.190.181) by mail.nvidia.com (10.127.129.5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.20; Mon, 26 Jan 2026 19:09:39 -0800 Received: from drhqmail201.nvidia.com (10.126.190.180) by drhqmail202.nvidia.com (10.126.190.181) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.20; Mon, 26 Jan 2026 19:09:38 -0800 Received: from Asurada-Nvidia.nvidia.com (10.127.8.14) by mail.nvidia.com (10.126.190.180) with Microsoft SMTP Server id 15.2.2562.20 via Frontend Transport; Mon, 26 Jan 2026 19:09:38 -0800 From: Nicolin Chen To: CC: , , , , , , , , , , , Subject: [PATCH v10 6/8] iommu/arm-smmu-v3: Populate smmu_domain->invs when attaching masters Date: Mon, 26 Jan 2026 19:09:17 -0800 Message-ID: X-Mailer: git-send-email 2.43.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH1PEPF0000AD83:EE_|DS7PR12MB5936:EE_ X-MS-Office365-Filtering-Correlation-Id: b8e0ac1a-7e36-410e-4d13-08de5d5188d9 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|36860700013|1800799024|82310400026|376014|7416014; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?O7NLSQMlVJd12atEccgaCiDuKRqZ3CHDrB6+QY04IyKMlDfPorHQzFkKRT3w?= =?us-ascii?Q?cl38Bir6E5JzC/xmFCSCsdhL8q84Noa2PiyBzIUJnbVpxYV8L26JirDg2fQO?= =?us-ascii?Q?oEF5BI5zdFkgsdCQfeY3T7mMJqsZJ3WiMnJQJKsH1SBYvWnqvW191jIzL2WO?= =?us-ascii?Q?inF1LuYqzGfjROFvqfzieV1kLmnts2RRvtl5sCq3/XE/Q2J0GJG6yB1Vvks8?= =?us-ascii?Q?H3hdpEmQIzxZQaJ6alJ+Z3A9VVBtmnE/DWfMpinYFCdBPeHteS75dvSlTdlr?= =?us-ascii?Q?dl3sBkF+t6gu1b7HvyQLe9i0jdWb6RJLEsTktH4Dj7iULHO0i8GCqLU4FNLe?= =?us-ascii?Q?OIuTqdDCthXBr4wKxkTC9E+kDnnqdbHAPqbalt3zdHa66Cp1/Zcj0XUXy8ba?= =?us-ascii?Q?LLepFVjFQw1GTNLy0kimXIi5K8UNQdoFQ9d3dQefG4w/DBMeLrjtHiAxrtlg?= =?us-ascii?Q?wLHCXF4revNOLJ/aBVjZuUEJBXGmlx/Hky7J+WwfKtsS/AdcQeJWfk9sF/Kq?= =?us-ascii?Q?YQ6fUh8h2oru11ozMA6B3JaopRExo3/mWOHbvqrVauZP4v33IyeODMVOdmvU?= =?us-ascii?Q?dfyWBYgqMk6pJA7aMinRbBOu8qGNv4BV27GK0gopyPebVWa5oc9lS6W+d+dF?= =?us-ascii?Q?iNhbYq3V8u7CI+S8tZavmGwCe2DS3/26wSS/DJ4QNRZ/xM+hbvZt0bdJ3dHa?= =?us-ascii?Q?OCzp1qYJrSN0ZgKMxNTjjzeltn2kbrzdyhFkRSyVS1r5Dd6ykBmeU0W7FpLO?= =?us-ascii?Q?5Y41yMwosGfjvFGkM1shuZuPbM2dxRnWkXB8JJBU47gNlwphlu1YLUVZ9EJu?= =?us-ascii?Q?JPiQ/den5H2wVs54Wkem6/0VIwEay2nqjcHJZkowxhq21td5JIq1ER0UjgBK?= =?us-ascii?Q?j2BA9JtGUZlx7EHhQll2BS97YsWMOGx1GIiceOxsmh5XymTsCIjAXABNfRV5?= =?us-ascii?Q?2q8DkcNuInmUXnkHoUwd3OnoXg1EglWbbDWmmJMxXFKrMJ1e40Hf+pgiSqg3?= =?us-ascii?Q?4PW8l21xkceldeLRHxzZcr9r9BndR2qPD4ntMvTM56k3r0+Zj7PCxyAn9uRn?= =?us-ascii?Q?pOGUU3OhpUVvCClNsprtGWs/V4IoN6h4ZREMJWasD/03LU5+LFXaRb0PKMMA?= =?us-ascii?Q?3lwj+Rzqb4dFeHPyuiMQnaUiVo+4H0xK2gdEYbn5/zS48x3Dbqd2IWyXOB/Y?= =?us-ascii?Q?1zaafKt7cTAn0m66iixp5YZlShN2N+xCXHRL5KW0FVRFrpZrYRhY2GpWfPte?= =?us-ascii?Q?12/izHXi/WFYLcj8Iy7suFAhu/eMzHdFP+IfGLP/kUJcDyXQmRKx1tuRyTry?= =?us-ascii?Q?LEHEifkUvMEuTOqGb3OIMuu9ScMaOTYmlXy3tgFgMn/LJD0M7k3R5CiIlhCQ?= =?us-ascii?Q?tzQNjgsRNtoEu9k38xFQd0RCJwntIUIwdSHHiW16M5BPedyQppr3+uKxZ8mq?= =?us-ascii?Q?0gjInrX5384ka2BkdiG6FrnZCXgmTVyDQOVKZyaz3BwWk+8SQrZVHP5d6qgw?= =?us-ascii?Q?TzhO3NhvzivAl1eI44G3dl5+1Fw3gdQ1yIKsQI+37ofTvO0xYfEUS17aoFe+?= =?us-ascii?Q?iA7dL7F+Qo7G5f2ojkiLF7wrg4ydim2onjjjolvIf5J+BDsZo2MznaQ+g3AB?= =?us-ascii?Q?aIQn44f2s1I4X7+c6M3kVClGRCd09DyvLwu2Twqul2YrLkUQ7UHd/A3kDTBX?= =?us-ascii?Q?3Z8uoA=3D=3D?= X-Forefront-Antispam-Report: CIP:216.228.118.232;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc7edge1.nvidia.com;CAT:NONE;SFS:(13230040)(36860700013)(1800799024)(82310400026)(376014)(7416014);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Jan 2026 03:09:50.1438 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: b8e0ac1a-7e36-410e-4d13-08de5d5188d9 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.118.232];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CH1PEPF0000AD83.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS7PR12MB5936 Content-Type: text/plain; charset="utf-8" Update the invs array with the invalidations required by each domain type during attachment operations. Only an SVA domain or a paging domain will have an invs array: a. SVA domain will add an INV_TYPE_S1_ASID per SMMU and an INV_TYPE_ATS per SID b. Non-nesting-parent paging domain with no ATS-enabled master will add a single INV_TYPE_S1_ASID or INV_TYPE_S2_VMID per SMMU c. Non-nesting-parent paging domain with ATS-enabled master(s) will do (b) and add an INV_TYPE_ATS per SID d. Nesting-parent paging domain will add an INV_TYPE_S2_VMID followed by an INV_TYPE_S2_VMID_S1_CLEAR per vSMMU. For an ATS-enabled master, it will add an INV_TYPE_ATS_FULL per SID Note that case #d prepares for a future implementation of VMID allocation which requires a followup series for S2 domain sharing. So when a nesting parent domain is attached through a vSMMU instance using a nested domain. VMID will be allocated per vSMMU instance v.s. currectly per S2 domain. The per-domain invalidation is not needed until the domain is attached to a master (when it starts to possibly use TLB). This will make it possible to attach the domain to multiple SMMUs and avoid unnecessary invalidation overhead during teardown if no STEs/CDs refer to the domain. It also means that when the last device is detached, the old domain must flush its ASID or VMID, since any new iommu_unmap() call would not trigger invalidations given an empty domain->invs array. Introduce some arm_smmu_invs helper functions for building scratch arrays, preparing and installing old/new domain's invalidation arrays. Co-developed-by: Jason Gunthorpe Signed-off-by: Jason Gunthorpe Reviewed-by: Jason Gunthorpe Signed-off-by: Nicolin Chen --- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 17 ++ drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 260 +++++++++++++++++++- 2 files changed, 276 insertions(+), 1 deletion(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/ar= m/arm-smmu-v3/arm-smmu-v3.h index 5e0e5055af1e..83d7e4952dff 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h @@ -1102,6 +1102,21 @@ static inline bool arm_smmu_master_canwbs(struct arm= _smmu_master *master) IOMMU_FWSPEC_PCI_RC_CANWBS; } =20 +/** + * struct arm_smmu_inv_state - Per-domain invalidation array state + * @invs_ptr: points to the domain->invs (unwinding nesting/etc.) or is NU= LL if + * no change should be made + * @old_invs: the original invs array + * @new_invs: for new domain, this is the new invs array to update domain-= >invs; + * for old domain, this is the master->build_invs to pass in as= the + * to_unref argument to an arm_smmu_invs_unref() call + */ +struct arm_smmu_inv_state { + struct arm_smmu_invs __rcu **invs_ptr; + struct arm_smmu_invs *old_invs; + struct arm_smmu_invs *new_invs; +}; + struct arm_smmu_attach_state { /* Inputs */ struct iommu_domain *old_domain; @@ -1111,6 +1126,8 @@ struct arm_smmu_attach_state { ioasid_t ssid; /* Resulting state */ struct arm_smmu_vmaster *vmaster; + struct arm_smmu_inv_state old_domain_invst; + struct arm_smmu_inv_state new_domain_invst; bool ats_enabled; }; =20 diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/ar= m/arm-smmu-v3/arm-smmu-v3.c index 5a0a8b136352..4648d0aad693 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -3143,6 +3143,121 @@ static void arm_smmu_disable_iopf(struct arm_smmu_m= aster *master, iopf_queue_remove_device(master->smmu->evtq.iopf, master->dev); } =20 +static struct arm_smmu_inv * +arm_smmu_master_build_inv(struct arm_smmu_master *master, + enum arm_smmu_inv_type type, u32 id, ioasid_t ssid, + size_t pgsize) +{ + struct arm_smmu_invs *build_invs =3D master->build_invs; + struct arm_smmu_inv *cur, inv =3D { + .smmu =3D master->smmu, + .type =3D type, + .id =3D id, + .pgsize =3D pgsize, + }; + + if (WARN_ON(build_invs->num_invs >=3D build_invs->max_invs)) + return NULL; + cur =3D &build_invs->inv[build_invs->num_invs]; + build_invs->num_invs++; + + *cur =3D inv; + switch (type) { + case INV_TYPE_S1_ASID: + /* + * For S1 page tables the driver always uses VMID=3D0, and the + * invalidation logic for this type will set it as well. + */ + if (master->smmu->features & ARM_SMMU_FEAT_E2H) { + cur->size_opcode =3D CMDQ_OP_TLBI_EL2_VA; + cur->nsize_opcode =3D CMDQ_OP_TLBI_EL2_ASID; + } else { + cur->size_opcode =3D CMDQ_OP_TLBI_NH_VA; + cur->nsize_opcode =3D CMDQ_OP_TLBI_NH_ASID; + } + break; + case INV_TYPE_S2_VMID: + cur->size_opcode =3D CMDQ_OP_TLBI_S2_IPA; + cur->nsize_opcode =3D CMDQ_OP_TLBI_S12_VMALL; + break; + case INV_TYPE_S2_VMID_S1_CLEAR: + cur->size_opcode =3D cur->nsize_opcode =3D CMDQ_OP_TLBI_NH_ALL; + break; + case INV_TYPE_ATS: + case INV_TYPE_ATS_FULL: + cur->size_opcode =3D cur->nsize_opcode =3D CMDQ_OP_ATC_INV; + cur->ssid =3D ssid; + break; + } + + return cur; +} + +/* + * Use the preallocated scratch array at master->build_invs, to build a to= _merge + * or to_unref array, to pass into a following arm_smmu_invs_merge/unref()= call. + * + * Do not free the returned invs array. It is reused, and will be overwrit= ten by + * the next arm_smmu_master_build_invs() call. + */ +static struct arm_smmu_invs * +arm_smmu_master_build_invs(struct arm_smmu_master *master, bool ats_enable= d, + ioasid_t ssid, struct arm_smmu_domain *smmu_domain) +{ + const bool nesting =3D smmu_domain->nest_parent; + size_t pgsize =3D 0, i; + + iommu_group_mutex_assert(master->dev); + + master->build_invs->num_invs =3D 0; + + /* Range-based invalidation requires the leaf pgsize for calculation */ + if (master->smmu->features & ARM_SMMU_FEAT_RANGE_INV) + pgsize =3D __ffs(smmu_domain->domain.pgsize_bitmap); + + switch (smmu_domain->stage) { + case ARM_SMMU_DOMAIN_SVA: + case ARM_SMMU_DOMAIN_S1: + if (!arm_smmu_master_build_inv(master, INV_TYPE_S1_ASID, + smmu_domain->cd.asid, + IOMMU_NO_PASID, pgsize)) + return NULL; + break; + case ARM_SMMU_DOMAIN_S2: + if (!arm_smmu_master_build_inv(master, INV_TYPE_S2_VMID, + smmu_domain->s2_cfg.vmid, + IOMMU_NO_PASID, pgsize)) + return NULL; + break; + default: + WARN_ON(true); + return NULL; + } + + /* All the nested S1 ASIDs have to be flushed when S2 parent changes */ + if (nesting) { + if (!arm_smmu_master_build_inv( + master, INV_TYPE_S2_VMID_S1_CLEAR, + smmu_domain->s2_cfg.vmid, IOMMU_NO_PASID, 0)) + return NULL; + } + + for (i =3D 0; ats_enabled && i < master->num_streams; i++) { + /* + * If an S2 used as a nesting parent is changed we have no + * option but to completely flush the ATC. + */ + if (!arm_smmu_master_build_inv( + master, nesting ? INV_TYPE_ATS_FULL : INV_TYPE_ATS, + master->streams[i].id, ssid, 0)) + return NULL; + } + + /* Note this build_invs must have been sorted */ + + return master->build_invs; +} + static void arm_smmu_remove_master_domain(struct arm_smmu_master *master, struct iommu_domain *domain, ioasid_t ssid) @@ -3172,6 +3287,133 @@ static void arm_smmu_remove_master_domain(struct ar= m_smmu_master *master, kfree(master_domain); } =20 +/* + * During attachment, the updates of the two domain->invs arrays are seque= nced: + * 1. new domain updates its invs array, merging master->build_invs + * 2. new domain starts to include the master during its invalidation + * 3. master updates its STE switching from the old domain to the new dom= ain + * 4. old domain still includes the master during its invalidation + * 5. old domain updates its invs array, unreferencing master->build_invs + * + * For 1 and 5, prepare the two updated arrays in advance, handling any ch= anges + * that can possibly failure. So the actual update of either 1 or 5 won't = fail. + * arm_smmu_asid_lock ensures that the old invs in the domains are intact = while + * we are sequencing to update them. + */ +static int arm_smmu_attach_prepare_invs(struct arm_smmu_attach_state *stat= e, + struct arm_smmu_domain *new_smmu_domain) +{ + struct arm_smmu_domain *old_smmu_domain =3D + to_smmu_domain_devices(state->old_domain); + struct arm_smmu_master *master =3D state->master; + ioasid_t ssid =3D state->ssid; + + /* + * At this point a NULL domain indicates the domain doesn't use the + * IOTLB, see to_smmu_domain_devices(). + */ + if (new_smmu_domain) { + struct arm_smmu_inv_state *invst =3D &state->new_domain_invst; + struct arm_smmu_invs *build_invs; + + invst->invs_ptr =3D &new_smmu_domain->invs; + invst->old_invs =3D rcu_dereference_protected( + new_smmu_domain->invs, + lockdep_is_held(&arm_smmu_asid_lock)); + build_invs =3D arm_smmu_master_build_invs( + master, state->ats_enabled, ssid, new_smmu_domain); + if (!build_invs) + return -EINVAL; + + invst->new_invs =3D + arm_smmu_invs_merge(invst->old_invs, build_invs); + if (IS_ERR(invst->new_invs)) + return PTR_ERR(invst->new_invs); + } + + if (old_smmu_domain) { + struct arm_smmu_inv_state *invst =3D &state->old_domain_invst; + + invst->invs_ptr =3D &old_smmu_domain->invs; + /* A re-attach case might have a different ats_enabled state */ + if (new_smmu_domain =3D=3D old_smmu_domain) + invst->old_invs =3D state->new_domain_invst.new_invs; + else + invst->old_invs =3D rcu_dereference_protected( + old_smmu_domain->invs, + lockdep_is_held(&arm_smmu_asid_lock)); + /* For old_smmu_domain, new_invs points to master->build_invs */ + invst->new_invs =3D arm_smmu_master_build_invs( + master, master->ats_enabled, ssid, old_smmu_domain); + } + + return 0; +} + +/* Must be installed before arm_smmu_install_ste_for_dev() */ +static void +arm_smmu_install_new_domain_invs(struct arm_smmu_attach_state *state) +{ + struct arm_smmu_inv_state *invst =3D &state->new_domain_invst; + + if (!invst->invs_ptr) + return; + + rcu_assign_pointer(*invst->invs_ptr, invst->new_invs); + kfree_rcu(invst->old_invs, rcu); +} + +static void arm_smmu_inv_flush_iotlb_tag(struct arm_smmu_inv *inv) +{ + struct arm_smmu_cmdq_ent cmd =3D {}; + + switch (inv->type) { + case INV_TYPE_S1_ASID: + cmd.tlbi.asid =3D inv->id; + break; + case INV_TYPE_S2_VMID: + /* S2_VMID using nsize_opcode covers S2_VMID_S1_CLEAR */ + cmd.tlbi.vmid =3D inv->id; + break; + default: + return; + } + + cmd.opcode =3D inv->nsize_opcode; + arm_smmu_cmdq_issue_cmd_with_sync(inv->smmu, &cmd); +} + +/* Should be installed after arm_smmu_install_ste_for_dev() */ +static void +arm_smmu_install_old_domain_invs(struct arm_smmu_attach_state *state) +{ + struct arm_smmu_inv_state *invst =3D &state->old_domain_invst; + struct arm_smmu_invs *old_invs =3D invst->old_invs; + struct arm_smmu_invs *new_invs; + + lockdep_assert_held(&arm_smmu_asid_lock); + + if (!invst->invs_ptr) + return; + + arm_smmu_invs_unref(old_invs, invst->new_invs); + /* + * When an IOTLB tag (the first entry in invs->new_invs) is no longer use= d, + * it means the ASID or VMID will no longer be invalidated by map/unmap a= nd + * must be cleaned right now. The rule is that any ASID/VMID not in an in= vs + * array must be left cleared in the IOTLB. + */ + if (!READ_ONCE(invst->new_invs->inv[0].users)) + arm_smmu_inv_flush_iotlb_tag(&invst->new_invs->inv[0]); + + new_invs =3D arm_smmu_invs_purge(old_invs); + if (!new_invs) + return; + + rcu_assign_pointer(*invst->invs_ptr, new_invs); + kfree_rcu(old_invs, rcu); +} + /* * Start the sequence to attach a domain to a master. The sequence contain= s three * steps: @@ -3229,12 +3471,16 @@ int arm_smmu_attach_prepare(struct arm_smmu_attach_= state *state, arm_smmu_ats_supported(master); } =20 + ret =3D arm_smmu_attach_prepare_invs(state, smmu_domain); + if (ret) + return ret; + if (smmu_domain) { if (new_domain->type =3D=3D IOMMU_DOMAIN_NESTED) { ret =3D arm_smmu_attach_prepare_vmaster( state, to_smmu_nested_domain(new_domain)); if (ret) - return ret; + goto err_unprepare_invs; } =20 master_domain =3D kzalloc(sizeof(*master_domain), GFP_KERNEL); @@ -3282,6 +3528,8 @@ int arm_smmu_attach_prepare(struct arm_smmu_attach_st= ate *state, atomic_inc(&smmu_domain->nr_ats_masters); list_add(&master_domain->devices_elm, &smmu_domain->devices); spin_unlock_irqrestore(&smmu_domain->devices_lock, flags); + + arm_smmu_install_new_domain_invs(state); } =20 if (!state->ats_enabled && master->ats_enabled) { @@ -3301,6 +3549,8 @@ int arm_smmu_attach_prepare(struct arm_smmu_attach_st= ate *state, kfree(master_domain); err_free_vmaster: kfree(state->vmaster); +err_unprepare_invs: + kfree(state->new_domain_invst.new_invs); return ret; } =20 @@ -3332,6 +3582,7 @@ void arm_smmu_attach_commit(struct arm_smmu_attach_st= ate *state) } =20 arm_smmu_remove_master_domain(master, state->old_domain, state->ssid); + arm_smmu_install_old_domain_invs(state); master->ats_enabled =3D state->ats_enabled; } =20 @@ -3513,12 +3764,19 @@ static int arm_smmu_blocking_set_dev_pasid(struct i= ommu_domain *new_domain, { struct arm_smmu_domain *smmu_domain =3D to_smmu_domain(old_domain); struct arm_smmu_master *master =3D dev_iommu_priv_get(dev); + struct arm_smmu_attach_state state =3D { + .master =3D master, + .old_domain =3D old_domain, + .ssid =3D pasid, + }; =20 mutex_lock(&arm_smmu_asid_lock); + arm_smmu_attach_prepare_invs(&state, NULL); arm_smmu_clear_cd(master, pasid); if (master->ats_enabled) arm_smmu_atc_inv_master(master, pasid); arm_smmu_remove_master_domain(master, &smmu_domain->domain, pasid); + arm_smmu_install_old_domain_invs(&state); mutex_unlock(&arm_smmu_asid_lock); =20 /* --=20 2.43.0 From nobody Sat Feb 7 11:52:10 2026 Received: from PH0PR06CU001.outbound.protection.outlook.com (mail-westus3azon11011055.outbound.protection.outlook.com [40.107.208.55]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 25DB02FF64C for ; Tue, 27 Jan 2026 03:09:59 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.208.55 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1769483401; cv=fail; b=GRs7r0zldzaSmxUPSgdDh2K3YE3h1yNObdFIdA+SjGuYk7qlmLNRijlZpyLSiHJDTfethSkfzIBCyIfEZYjFeGv0XveA/w2SQr+6YbdhXkNsqERFYpVhcb1uTzd6K786ochhb98U+Di7eqZEE2P6+go9K/owpfa0W7bl3UM2qVc= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1769483401; c=relaxed/simple; bh=oq1zqY1qmNYw2jjr0o14CAUtR3Q6uQshRDsiwNDfN4Q=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=tJ2xz3H0P5++6a/v9qF3UYj0jYBzWyMvdQVeenI1BGTXCW0x2hV2Ymh+NGMuBgIoRzrq6szSyyaZ84E6T+Euo7ZWnrY+s66z8FWXPo3alIzIGHPSseGVkII7hfZ0q0geFDlsugcJacWLJEtPHAT5haXwOWB+2YzcebkrXOFw/c4= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=kLWEqyPv; arc=fail smtp.client-ip=40.107.208.55 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="kLWEqyPv" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=hlynqyk3jbgMRt2VlLcEsdEotESi7uTiDj2SyUykBQ3ntLCGGRs/U67KLmzcd246MlnYmWtnBLqCiBaRVAjVd0mpTRSQUzF/kEeZ8m8Sw4E7Oi7x8HF0F6nN1uv1CwifmeynqRtjp+ljOjGTyTnN80FBOjjgX0G2rt/WxvGvYPNj0diJuqEDwiuvY/cPROmdg/otyyNrHSxAePGBaopBH9j9khxx00ItkQ4Np01LQhqQmVfEPdOdTjiaQ0sYB61HOEYxoe9/XIhs2ZzNzvpBwd7RD/6U7WQ7cCHASbuswM5gqr3nl+gTpDSf1DKPCiAvq5l7B0bzeUUZ/kTwIjbmhw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=oJuzqIs5b5YweipQH8dpop+URN7F4iBqO9iV7u4rHp4=; b=cYB2/dTXV9GDA6sqLmCn7Nkls5nDkOGAHVU34ncr33MWzouw7kIYNAV7G3yuyh5gcnc59vvGge/d/h+BCNVP+mCyaMZPwZ2ez69OSEVJmMIzvFH07YFvDQ7wBz4Odsq8KJXK4sU1FmSU3zIq/omr5P7d7vD54IegelXfEx+jUVxQZAnGsmvYUd6mEu8gW+cYoPBWQK+UsMa+QDBfzznComYa9qfiNV1sp8FnYnYqneonMgAJDIR/c/lUZ8S4oPthSQX4lUu2KTiNHlyFAKZ/JGt/zSzYhv2tdfubyS+QfTDEg+RnMmRe8/6009nm5f1QSna0ah7GGQKQhEZub1+p8Q== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.233) smtp.rcpttodomain=kernel.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=oJuzqIs5b5YweipQH8dpop+URN7F4iBqO9iV7u4rHp4=; b=kLWEqyPv0303k+mA0+GavVJmWTsHnCqK5Qr+b4YGPY2yyGLDI+mBkJp1/JLeMfpkG2UaIIOCXO2S6SGd9Zy7gEshD++nW5LdemMYyhgx6WMZ2FLVwM5ciXoVmFqtWpHaRAXu5aFPzHSPnR89snTJbhnObqyKCCICj/JgZAZ6iihWZQ6l2ClMfw2KziuQtiA+OMOIMURFNqskoZhwpY70rGdFo34rBPcCuosBoReKeLc7UGbYHpTL2x0GuHYdIeYWCymo6oXTelTB+UyA2H/H8osl3+48lOroJMIaDeYb1q7r3uWhhBF8hFFPPrkSnmK9XRcWiBI/r1XCyuvfiACEGw== Received: from BN9PR03CA0156.namprd03.prod.outlook.com (2603:10b6:408:f4::11) by CH1PPF931B95D07.namprd12.prod.outlook.com (2603:10b6:61f:fc00::619) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9542.15; Tue, 27 Jan 2026 03:09:51 +0000 Received: from BN2PEPF000044A6.namprd04.prod.outlook.com (2603:10b6:408:f4:cafe::c2) by BN9PR03CA0156.outlook.office365.com (2603:10b6:408:f4::11) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.9542.16 via Frontend Transport; Tue, 27 Jan 2026 03:09:49 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.233) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.233 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.233; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.118.233) by BN2PEPF000044A6.mail.protection.outlook.com (10.167.243.100) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9564.3 via Frontend Transport; Tue, 27 Jan 2026 03:09:51 +0000 Received: from drhqmail203.nvidia.com (10.126.190.182) by mail.nvidia.com (10.127.129.6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.20; Mon, 26 Jan 2026 19:09:40 -0800 Received: from drhqmail201.nvidia.com (10.126.190.180) by drhqmail203.nvidia.com (10.126.190.182) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.20; Mon, 26 Jan 2026 19:09:39 -0800 Received: from Asurada-Nvidia.nvidia.com (10.127.8.14) by mail.nvidia.com (10.126.190.180) with Microsoft SMTP Server id 15.2.2562.20 via Frontend Transport; Mon, 26 Jan 2026 19:09:39 -0800 From: Nicolin Chen To: CC: , , , , , , , , , , , Subject: [PATCH v10 7/8] iommu/arm-smmu-v3: Add arm_smmu_invs based arm_smmu_domain_inv_range() Date: Mon, 26 Jan 2026 19:09:18 -0800 Message-ID: X-Mailer: git-send-email 2.43.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BN2PEPF000044A6:EE_|CH1PPF931B95D07:EE_ X-MS-Office365-Filtering-Correlation-Id: ff7f0882-52df-48d9-1d06-08de5d51897b X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|7416014|36860700013|1800799024|376014|82310400026|3613699012; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?kFaIfSjlmjyFa4yvTRBzrgs/wn8ETSAD35Ao+Qxr1r/QZdCgNetkZlJAn+Cf?= =?us-ascii?Q?JXDMlmisQ5fnWE9gVS6EwCMVrrqZIcr2qYiezKm/tCQRsSH9EtRaL7ToHGuB?= =?us-ascii?Q?3UPP7SzrIAPHqnoQc0zfA8RvEca8mNlI1RQpqKX5/1xOdqZJmNMVDViqsbvH?= =?us-ascii?Q?5MICOiFhnU7tuDoVJ9wIWiNqM6+P32qdNyn6kRNRtJV2Cp9fkv59rgwpnhdc?= =?us-ascii?Q?9jBbtcgZQaPMP/s2zOAb0M/jHq8xIb/ZrmWUPSWCzpP5zDAH1dlaytDkqL4m?= =?us-ascii?Q?5to597gGmwHgflx3TDsQm/DbzqvlB5gSIRXYUFrZ1HMNSFG60Wy1GepBnxvq?= =?us-ascii?Q?B7by4Hto1jq4x1cB+QasIkP9m//Pe+WgiLYkUc8YP/7fYJlvCsh9xikWtcHv?= =?us-ascii?Q?sBenQH1ikLqBaaMGs2TaOkRe6FaEioBi4wLxy4JSMe2gHDdLeSHg2de3nqbM?= =?us-ascii?Q?TjMVDT89EeEvEToLxnzmM1ThgyJUmMGe5t01txQs+N7IqUwMa+3ZwHeb2WQV?= =?us-ascii?Q?JnhIrt+g4xlQazrAJ3JOxc+RmPXE64wuMUPhN/Vd3pwExyzp8K0Vy5XMMt+w?= =?us-ascii?Q?DINPWkSUxU7IEojlGvSjGFJiz098WnfM39Ol3PDqobz0971PldkDjnJuGdVn?= =?us-ascii?Q?Elh08ayAXSJDGtVwBnuOAD2R8AAZgmVFj8zDR1sKWTIt135jzrG9ePBKhoAV?= =?us-ascii?Q?QNDQKTroyqYUqgw5O+XRe+3T4B8Yorq9+6j0rIs8nY6Jr93PWj8j4xTkVnyj?= =?us-ascii?Q?cYnpmdTpQqL0W4XBAIvEnmlwcQsY1kCv9vPDa6rmArT6xHgTFonkmepf26DB?= =?us-ascii?Q?4F0WOWUyeyGrMCM+SXjt5ePkUYRYQjwH2090gXnl8xND4hrCoN7SjTIXJ3qK?= =?us-ascii?Q?O3y9tT+8WnTvB+ik+NAxBGVAyEVy71rCitYn7l14c2oGwNiuFxe11AEP3omb?= =?us-ascii?Q?045pJfSfMe+6YONghGgVultj2r2ftn+3oCgSVikwDN31yCMTrIpdKq+VIXFH?= =?us-ascii?Q?oMGFUorCUWji+P6HjD8YBBSBKLoUl3bQkKItkqVqGhBkmE0MN9nhGJsVwbYG?= =?us-ascii?Q?gvkUuodNyFiq4vM6j/kY5FtKroB+bdyGQTBq9AcjYG8gVyywiGKMtWX01pqO?= =?us-ascii?Q?1f/+dluISUcRj/ClecTzqvZQk8FGUa8viaUwzwp/ekRWfhzlJLlLD2lpsl+i?= =?us-ascii?Q?a8OfC668F/HSDFOoOlk403c4E7SKPx3Qxn2w3NRmOOMkC2pv2cOZzADwj4U2?= =?us-ascii?Q?ZnGTjIR5naEU8/XpBtvmUx62qACaP3MYfFt1LnaVihh90nyh+9dRp1xZ57f5?= =?us-ascii?Q?sQSkroapHew5TIKHnyu1v2w+mGSSP6IMQt5tJmWevdxvxPPbiXe7+1im4ZQe?= =?us-ascii?Q?DZQW/ZIiS34JFlam6NhWetWtcaRsF2t4ktq/djTQuLNtEk+gBH9fogOYzZxd?= =?us-ascii?Q?Ay7h5IAyvUVA8CdwoMYPfzg8+3VY/PfAk8AgXqQCflcTBdEm6FQNgQ1h+Vo1?= =?us-ascii?Q?IV7tBP8bExstA2ykJMTE92xeRigDfmYMvQykrJC+tMG/VEJp/GK+gDEkWf6F?= =?us-ascii?Q?+qED2AOIAkASbkVa8XAtId8sTy7Kag3BoT9zyOldwMEHu1Y+/cvP3Q/FdxTM?= =?us-ascii?Q?NqBdtUXfvl3b4lRvkVhhyWZ/OwaKqyNR7S37qYOFH7C5zFy8QATzvA4zHCdr?= =?us-ascii?Q?wUue5tSarsckxxTfIhJIOXtJAIs=3D?= X-Forefront-Antispam-Report: CIP:216.228.118.233;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc7edge2.nvidia.com;CAT:NONE;SFS:(13230040)(7416014)(36860700013)(1800799024)(376014)(82310400026)(3613699012);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Jan 2026 03:09:51.1792 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: ff7f0882-52df-48d9-1d06-08de5d51897b X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.118.233];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BN2PEPF000044A6.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH1PPF931B95D07 Content-Type: text/plain; charset="utf-8" Each smmu_domain now has an arm_smmu_invs that specifies the invalidation steps to perform after any change the IOPTEs. This includes supports for basic ASID/VMID, the special case for nesting, and ATC invalidations. Introduce a new arm_smmu_domain_inv helper iterating smmu_domain->invs to convert the invalidation array to commands. Any invalidation request with no size specified means an entire flush over a range based one. Take advantage of the sorted array to compatible batch operations together to the same SMMU. For instance, ATC invaliations for multiple SIDs can be pushed as a batch. ATC invalidations must be completed before the driver disables ATS. Or the device is permitted to ignore any racing invalidation that would cause an SMMU timeout. The sequencing is done with a rwlock where holding the write side of the rwlock means that there are no outstanding ATC invalidations. If ATS is not used the rwlock is ignored, similar to the existing code. Co-developed-by: Jason Gunthorpe Signed-off-by: Jason Gunthorpe Reviewed-by: Jason Gunthorpe Signed-off-by: Nicolin Chen --- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 9 + drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 223 ++++++++++++++++++-- 2 files changed, 219 insertions(+), 13 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/ar= m/arm-smmu-v3/arm-smmu-v3.h index 83d7e4952dff..534e9a5ddca3 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h @@ -1087,6 +1087,15 @@ void arm_smmu_tlb_inv_range_asid(unsigned long iova,= size_t size, int asid, int arm_smmu_atc_inv_domain(struct arm_smmu_domain *smmu_domain, unsigned long iova, size_t size); =20 +void arm_smmu_domain_inv_range(struct arm_smmu_domain *smmu_domain, + unsigned long iova, size_t size, + unsigned int granule, bool leaf); + +static inline void arm_smmu_domain_inv(struct arm_smmu_domain *smmu_domain) +{ + arm_smmu_domain_inv_range(smmu_domain, 0, 0, 0, false); +} + void __arm_smmu_cmdq_skip_err(struct arm_smmu_device *smmu, struct arm_smmu_cmdq *cmdq); int arm_smmu_init_one_queue(struct arm_smmu_device *smmu, diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/ar= m/arm-smmu-v3/arm-smmu-v3.c index 4648d0aad693..d94bb290d190 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -2591,23 +2591,19 @@ static void arm_smmu_tlb_inv_context(void *cookie) arm_smmu_atc_inv_domain(smmu_domain, 0, 0); } =20 -static void __arm_smmu_tlb_inv_range(struct arm_smmu_cmdq_ent *cmd, - unsigned long iova, size_t size, - size_t granule, - struct arm_smmu_domain *smmu_domain) +static void arm_smmu_cmdq_batch_add_range(struct arm_smmu_device *smmu, + struct arm_smmu_cmdq_batch *cmds, + struct arm_smmu_cmdq_ent *cmd, + unsigned long iova, size_t size, + size_t granule, size_t pgsize) { - struct arm_smmu_device *smmu =3D smmu_domain->smmu; - unsigned long end =3D iova + size, num_pages =3D 0, tg =3D 0; + unsigned long end =3D iova + size, num_pages =3D 0, tg =3D pgsize; size_t inv_range =3D granule; - struct arm_smmu_cmdq_batch cmds; =20 if (!size) return; =20 if (smmu->features & ARM_SMMU_FEAT_RANGE_INV) { - /* Get the leaf page size */ - tg =3D __ffs(smmu_domain->domain.pgsize_bitmap); - num_pages =3D size >> tg; =20 /* Convert page size of 12,14,16 (log2) to 1,2,3 */ @@ -2627,8 +2623,6 @@ static void __arm_smmu_tlb_inv_range(struct arm_smmu_= cmdq_ent *cmd, num_pages++; } =20 - arm_smmu_cmdq_batch_init(smmu, &cmds, cmd); - while (iova < end) { if (smmu->features & ARM_SMMU_FEAT_RANGE_INV) { /* @@ -2656,9 +2650,26 @@ static void __arm_smmu_tlb_inv_range(struct arm_smmu= _cmdq_ent *cmd, } =20 cmd->tlbi.addr =3D iova; - arm_smmu_cmdq_batch_add(smmu, &cmds, cmd); + arm_smmu_cmdq_batch_add(smmu, cmds, cmd); iova +=3D inv_range; } +} + +static void __arm_smmu_tlb_inv_range(struct arm_smmu_cmdq_ent *cmd, + unsigned long iova, size_t size, + size_t granule, + struct arm_smmu_domain *smmu_domain) +{ + struct arm_smmu_device *smmu =3D smmu_domain->smmu; + struct arm_smmu_cmdq_batch cmds; + size_t pgsize; + + /* Get the leaf page size */ + pgsize =3D __ffs(smmu_domain->domain.pgsize_bitmap); + + arm_smmu_cmdq_batch_init(smmu, &cmds, cmd); + arm_smmu_cmdq_batch_add_range(smmu, &cmds, cmd, iova, size, granule, + pgsize); arm_smmu_cmdq_batch_submit(smmu, &cmds); } =20 @@ -2714,6 +2725,192 @@ void arm_smmu_tlb_inv_range_asid(unsigned long iova= , size_t size, int asid, __arm_smmu_tlb_inv_range(&cmd, iova, size, granule, smmu_domain); } =20 +static bool arm_smmu_inv_size_too_big(struct arm_smmu_device *smmu, size_t= size, + size_t granule) +{ + size_t max_tlbi_ops; + + /* 0 size means invalidate all */ + if (!size || size =3D=3D SIZE_MAX) + return true; + + if (smmu->features & ARM_SMMU_FEAT_RANGE_INV) + return false; + + /* + * Borrowed from the MAX_TLBI_OPS in arch/arm64/include/asm/tlbflush.h, + * this is used as a threshold to replace "size_opcode" commands with a + * single "nsize_opcode" command, when SMMU doesn't implement the range + * invalidation feature, where there can be too many per-granule TLBIs, + * resulting in a soft lockup. + */ + max_tlbi_ops =3D 1 << (ilog2(granule) - 3); + return size >=3D max_tlbi_ops * granule; +} + +/* Used by non INV_TYPE_ATS* invalidations */ +static void arm_smmu_inv_to_cmdq_batch(struct arm_smmu_inv *inv, + struct arm_smmu_cmdq_batch *cmds, + struct arm_smmu_cmdq_ent *cmd, + unsigned long iova, size_t size, + unsigned int granule) +{ + if (arm_smmu_inv_size_too_big(inv->smmu, size, granule)) { + cmd->opcode =3D inv->nsize_opcode; + arm_smmu_cmdq_batch_add(inv->smmu, cmds, cmd); + return; + } + + cmd->opcode =3D inv->size_opcode; + arm_smmu_cmdq_batch_add_range(inv->smmu, cmds, cmd, iova, size, granule, + inv->pgsize); +} + +static inline bool arm_smmu_invs_end_batch(struct arm_smmu_inv *cur, + struct arm_smmu_inv *next) +{ + /* Changing smmu means changing command queue */ + if (cur->smmu !=3D next->smmu) + return true; + /* The batch for S2 TLBI must be done before nested S1 ASIDs */ + if (cur->type !=3D INV_TYPE_S2_VMID_S1_CLEAR && + next->type =3D=3D INV_TYPE_S2_VMID_S1_CLEAR) + return true; + /* ATS must be after a sync of the S1/S2 invalidations */ + if (!arm_smmu_inv_is_ats(cur) && arm_smmu_inv_is_ats(next)) + return true; + return false; +} + +static void __arm_smmu_domain_inv_range(struct arm_smmu_invs *invs, + unsigned long iova, size_t size, + unsigned int granule, bool leaf) +{ + struct arm_smmu_cmdq_batch cmds =3D {}; + struct arm_smmu_inv *cur; + struct arm_smmu_inv *end; + + cur =3D invs->inv; + end =3D cur + READ_ONCE(invs->num_invs); + /* Skip any leading entry marked as a trash */ + for (; cur !=3D end; cur++) + if (READ_ONCE(cur->users)) + break; + while (cur !=3D end) { + struct arm_smmu_device *smmu =3D cur->smmu; + struct arm_smmu_cmdq_ent cmd =3D { + /* + * Pick size_opcode to run arm_smmu_get_cmdq(). This can + * be changed to nsize_opcode, which would result in the + * same CMDQ pointer. + */ + .opcode =3D cur->size_opcode, + }; + struct arm_smmu_inv *next; + + if (!cmds.num) + arm_smmu_cmdq_batch_init(smmu, &cmds, &cmd); + + switch (cur->type) { + case INV_TYPE_S1_ASID: + cmd.tlbi.asid =3D cur->id; + cmd.tlbi.leaf =3D leaf; + arm_smmu_inv_to_cmdq_batch(cur, &cmds, &cmd, iova, size, + granule); + break; + case INV_TYPE_S2_VMID: + cmd.tlbi.vmid =3D cur->id; + cmd.tlbi.leaf =3D leaf; + arm_smmu_inv_to_cmdq_batch(cur, &cmds, &cmd, iova, size, + granule); + break; + case INV_TYPE_S2_VMID_S1_CLEAR: + /* CMDQ_OP_TLBI_S12_VMALL already flushed S1 entries */ + if (arm_smmu_inv_size_too_big(cur->smmu, size, granule)) + continue; + cmd.tlbi.vmid =3D cur->id; + arm_smmu_cmdq_batch_add(smmu, &cmds, &cmd); + break; + case INV_TYPE_ATS: + arm_smmu_atc_inv_to_cmd(cur->ssid, iova, size, &cmd); + cmd.atc.sid =3D cur->id; + arm_smmu_cmdq_batch_add(smmu, &cmds, &cmd); + break; + case INV_TYPE_ATS_FULL: + arm_smmu_atc_inv_to_cmd(IOMMU_NO_PASID, 0, 0, &cmd); + cmd.atc.sid =3D cur->id; + arm_smmu_cmdq_batch_add(smmu, &cmds, &cmd); + break; + default: + WARN_ON_ONCE(1); + continue; + } + + /* Skip any trash entry in-between */ + for (next =3D cur + 1; next !=3D end; next++) + if (READ_ONCE(next->users)) + break; + + if (cmds.num && + (next =3D=3D end || arm_smmu_invs_end_batch(cur, next))) { + arm_smmu_cmdq_batch_submit(smmu, &cmds); + cmds.num =3D 0; + } + cur =3D next; + } +} + +void arm_smmu_domain_inv_range(struct arm_smmu_domain *smmu_domain, + unsigned long iova, size_t size, + unsigned int granule, bool leaf) +{ + struct arm_smmu_invs *invs; + + /* + * An invalidation request must follow some IOPTE change and then load + * an invalidation array. In the meantime, a domain attachment mutates + * the array and then stores an STE/CD asking SMMU HW to acquire those + * changed IOPTEs. + * + * When running alone, a domain attachment relies on the dma_wmb() in + * arm_smmu_write_entry() used by arm_smmu_install_ste_for_dev(). + * + * But in a race, these two can be interdependent, making it a special + * case requiring an additional smp_mb() for the write->read ordering. + * Pairing with the dma_wmb() in arm_smmu_install_ste_for_dev(), this + * makes sure that IOPTE update prior to this point is visable to SMMU + * hardware before we load the updated invalidation array. + * + * [CPU0] | [CPU1] + * change IOPTE on new domain: | + * arm_smmu_domain_inv_range() { | arm_smmu_install_new_domain_invs() + * smp_mb(); // ensures IOPTE | arm_smmu_install_ste_for_dev { + * // seen by SMMU | dma_wmb(); // ensures invs update + * // load the updated invs | // before updating STE + * invs =3D rcu_dereference(); | STE =3D TTB0; + * ... | ... + * } | } + */ + smp_mb(); + + rcu_read_lock(); + invs =3D rcu_dereference(smmu_domain->invs); + + /* + * Avoid locking unless ATS is being used. No ATC invalidation can be + * going on after a domain is detached. + */ + if (invs->has_ats) { + read_lock(&invs->rwlock); + __arm_smmu_domain_inv_range(invs, iova, size, granule, leaf); + read_unlock(&invs->rwlock); + } else { + __arm_smmu_domain_inv_range(invs, iova, size, granule, leaf); + } + + rcu_read_unlock(); +} + static void arm_smmu_tlb_inv_page_nosync(struct iommu_iotlb_gather *gather, unsigned long iova, size_t granule, void *cookie) --=20 2.43.0 From nobody Sat Feb 7 11:52:10 2026 Received: from CH4PR04CU002.outbound.protection.outlook.com (mail-northcentralusazon11013054.outbound.protection.outlook.com [40.107.201.54]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 304223033ED for ; Tue, 27 Jan 2026 03:09:59 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.201.54 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1769483403; cv=fail; b=qfUSpsxILvjKlaUR0L3Q/pSTiHgT4WvMPWJuDq0fNKynMLjbqwRFqP13knm/CTiy/tGgsbPwwK4F3gh6bsaeDGvFrEUT2O9zZoY/KnnE2gujqKoyySWP6gDbEZuXRBgkbiKGawaAJn/soBwreozZOfUYjvRZ9a+ATiuDBxlduW0= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1769483403; c=relaxed/simple; bh=wVjfs4Gdk+q4mka8L0KcDnc3L52xEsrvs58qOYoOBTE=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=TDCzKq2smf2P3Y8HFgarsql3/hi7TDKUO+4XqWaWqfLhEgVMrEna/y1C0xl1NFR2urWkhGY34/lKUh7sg/EsZgSXnKSLmvirrwXxNnbf5xceJ3XHmsES2v2LMeEnBe6TmDJDuuuoflwhKhLuEPmHIFc+PFVUL/YnFm8RkTqG2XQ= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=VIrh4Rkf; arc=fail smtp.client-ip=40.107.201.54 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="VIrh4Rkf" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=v0/6/2+VeWbnu9AI/snjpe7rv9/6tsG2GjQJKZbzDdYg6Gbgvo1AnQ9I4kgH3n+GzV81Q75dUhX7q04+wR/mMzshe5H2r6V/bHxkDH2l368Z2iLf3kpqwhx36aYO5v+pVCPHki94cMZhw22z9RZtcCUhN1fvHkL+JDwDRFYkzDV5o83u/bEa4OmB+npuw2s7wxTliJYpnArlikNzlf3Ktr/ostC7d2HVbxze6cxAdYO5nfMY+kNNca4SlZE/YpXmdIYT58AOis01qvYC5McvJMtZzCSwZP4oLhjlK4ll0pmBAw65yjOvJ2YHhnqDDVC8c2qQG906nTjz6t6ZKhYPfA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=yNSPyUtkiFUdmLV1Kxxux5qj18UQLDSaKpcgxsaoT6Y=; b=rTcTsnNq9BrDRc9+W24C5GgcXBWc7NLNIvcObgRFNuGoBcqOXrsPzFVgykt6E97gP2tz1YeQZ/olSfBCEBzRdydwZTI0mAK0jsxrd7olrZvUwAdqR74+9k7v5llSgr3HdZqUuqC4qh2RV3yhuZPz8JksEzHeSCWe/HRqULJsIElKpVMHaCTvQsbW/CuFEx9gxqE/U62B2yMUAilt6IKm5B2d7hqXBm+8Ax18stBWrXWFEneVpwypiHtyQ7uHxQ2WUuGzswtDuk78qD3KxGGjSMSReVXhzkQRor5KqvfazxguOgmDxiGaOrgJf2ODpyRW2gaXvDbd8HRIDwDXU9Xs3w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.233) smtp.rcpttodomain=kernel.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=yNSPyUtkiFUdmLV1Kxxux5qj18UQLDSaKpcgxsaoT6Y=; b=VIrh4Rkfo3VSNYLa/lIC0+miaCHU8ZlkVM+47AwM02ln7jxsalHoq4lp4J2EGcb5QMzdt8KWswFoC88zgvGVWwbokyYYX0TuGGpQn5Y7Sqjc1ogM10l3zt0zPfGU0ok8PDMCHhWbUfjYnSRF7o+A05rvJNEGw20XVxDcVwrp6nx0xyd4RpCFhf7JSjiopTkHchMZfPwqAsOhZiYsJqwlz2nXa7lYVS6XZxLfU3jpSkjF3+H4y0mQy0G6pYFVPnJUjggZS1W9ifbBxK+SFneOcnp0gA/NUQT8uq7iUyuN8KIv4/70pn2xVPivbtnr+jT9UyC3n51GSv5cBSLrJyCnxg== Received: from LV3P220CA0012.NAMP220.PROD.OUTLOOK.COM (2603:10b6:408:234::17) by BN7PPFB3F5C406F.namprd12.prod.outlook.com (2603:10b6:40f:fc02::6e0) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9542.16; Tue, 27 Jan 2026 03:09:54 +0000 Received: from BN2PEPF000044A8.namprd04.prod.outlook.com (2603:10b6:408:234:cafe::9a) by LV3P220CA0012.outlook.office365.com (2603:10b6:408:234::17) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.9542.16 via Frontend Transport; Tue, 27 Jan 2026 03:09:54 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.233) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.233 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.233; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.118.233) by BN2PEPF000044A8.mail.protection.outlook.com (10.167.243.102) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9564.3 via Frontend Transport; Tue, 27 Jan 2026 03:09:52 +0000 Received: from drhqmail203.nvidia.com (10.126.190.182) by mail.nvidia.com (10.127.129.6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.20; Mon, 26 Jan 2026 19:09:41 -0800 Received: from drhqmail201.nvidia.com (10.126.190.180) by drhqmail203.nvidia.com (10.126.190.182) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.20; Mon, 26 Jan 2026 19:09:40 -0800 Received: from Asurada-Nvidia.nvidia.com (10.127.8.14) by mail.nvidia.com (10.126.190.180) with Microsoft SMTP Server id 15.2.2562.20 via Frontend Transport; Mon, 26 Jan 2026 19:09:40 -0800 From: Nicolin Chen To: CC: , , , , , , , , , , , Subject: [PATCH v10 8/8] iommu/arm-smmu-v3: Perform per-domain invalidations using arm_smmu_invs Date: Mon, 26 Jan 2026 19:09:19 -0800 Message-ID: <1ad93ac5573ce2fa908a3c9cb0bc2ca53cb14ebb.1769476588.git.nicolinc@nvidia.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BN2PEPF000044A8:EE_|BN7PPFB3F5C406F:EE_ X-MS-Office365-Filtering-Correlation-Id: dc70ca0c-c933-46ba-91d9-08de5d5189fa X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|82310400026|376014|7416014|36860700013|1800799024; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?D8ZPDt+H27tc1PHPMwZbNuswliTO1P1FOTMsfp+K6E6nBLTKQeZ7tSWJoMGx?= =?us-ascii?Q?kiU8Gft4mBQz0db1+DfgubSbf2JOScm5jmgA+5PcMx1znfvmEu1kluctpd2X?= =?us-ascii?Q?3b5e8ZhNs1wZbCds+0pMZc6qXFrHBZI+9WThs3rnAi/SCr8/SigZCyDPyYPX?= =?us-ascii?Q?sz/ZMi64LOZAN5ewhiPydmJUuqFLxFlSZSmDfRydLTcOrT3VHN8sXQTEfI7L?= =?us-ascii?Q?TbRUfExvXIG3VQQq6kGWR2+czO6CxE+8FQ4tURjfvSS5U2O6XV0OaKqGTqR4?= =?us-ascii?Q?VCytcSA8KgJ3PJqXRvTciKA4bDCchIimINQZ+ZaRY07nFmS2rY1tQk/GGoJe?= =?us-ascii?Q?90aa7k7dFS9Mu779J0asvl27KvB+qqj/oOqLI85uaRhSf8KyjSFpa7ef8Tvs?= =?us-ascii?Q?FMtE9FqjVaisR+s1rqiN4/O7d2IwlqhpO6sx/SmYeBvrOP46tsxsyhqKP/WG?= =?us-ascii?Q?NW9aUVzD/VIVD796P2L/ci8v953Si/mhR/8/s2MVoOabD6AeV6et+acmPD4F?= =?us-ascii?Q?XWliZs5uJPjHUfeLn66QS8QpmthkIhtNRgfk2VSReDEuqrpMvRzT64Z4GY65?= =?us-ascii?Q?JCFrqjkTPNp0oSqpNSavtxRwHWZN50BTiBpJMTdz9ckvNnKlel7m+R08Y8DE?= =?us-ascii?Q?9M8CNkBa+hbO/2pkbWenb8lCBge8Eo9lhIdPnupFjkk3d3BgH/gHKX22Lm4u?= =?us-ascii?Q?CKilkwD2sSXA8R/YXNXNZdEtMgdjR6gG4dttNszv8NQ6DwD1F5434JoaqhdV?= =?us-ascii?Q?xeP0DJRGKNvN31Eqbc8JfmNhapPvbjVqMIdBpoNcjM7smDKEVC7vOrhgZggZ?= =?us-ascii?Q?XeAl3UouS+BkZ0xzw1EAaCQ7XaiGdCbILbpToiUvIWg+kBgVrII/9XURJZDq?= =?us-ascii?Q?pKLNtZkTb4/7VgYW1dELypiOg7Elupa9caGTwyd+zTyRCVHURjGxrHneE+LE?= =?us-ascii?Q?KHnHlEH5qWIlCMyjKkXdfg8HhFJb/tOf+fCMVmrfRglHrnyeZE6dNtFkmy66?= =?us-ascii?Q?3TEhdSjyhIT5Em4LoXHbBOdXV4nVQnUOrf9xUcTIZXBQFSrOQ1eaDGg1ki3t?= =?us-ascii?Q?6p1caYbAvzYlRvFUy9oP0O6Z9ftQ1e+Ae+Wb63rs8ZwAZCqjPRwO8JGLZ9Y3?= =?us-ascii?Q?2OhD7CyIBlEEL3n3d3bm/vw2IF/CRXTNuYEUSWI69vSKgog7ondXMa3iNiCZ?= =?us-ascii?Q?lRrfT78d28rXRHmykVhmaPuR9I23df6UDwv2ljE24vjUdDOsm+4jXC4M3ZOV?= =?us-ascii?Q?fbBuqwctwGzGfAx6TfxS3XPi7KYvI1njfOdjUkE9YWSAM0SjwzCYQH5Rgvd5?= =?us-ascii?Q?ojvgDFUckQHxPVLl5ASC8xfjdRtOOtIzvtr6/hOa9ehrUSauRR5uuA7deWq6?= =?us-ascii?Q?L+yTIA5elIHeaBeHXSFb1vOdu/iX3ebPBz0MwIgf5gTMzfzny56K9lzQEsT+?= =?us-ascii?Q?3XAFR6qEcriT0bjQcZ1/y4VOfm/f0w6d5/PQC3kk9SKyVrp40NbFzdCtXlvw?= =?us-ascii?Q?LDyRd5eiiV/Z4FXApQGSKGS4WpB5XsMe/rinN/Bq/RdD9r9dnzRBWvAmB3SO?= =?us-ascii?Q?bHxOZ+uNNJvOydjOzu3AdTMC7ZyvKmAcLkYSapSBtggq4AHuvS3+eLcbWbWS?= =?us-ascii?Q?cDVKKdw42KgGV8+IyI6IgwHsksiF7k/NNbRYjohQrrYHRiRi4B5oZjbyJu6n?= =?us-ascii?Q?C03t4A=3D=3D?= X-Forefront-Antispam-Report: CIP:216.228.118.233;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc7edge2.nvidia.com;CAT:NONE;SFS:(13230040)(82310400026)(376014)(7416014)(36860700013)(1800799024);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Jan 2026 03:09:52.0074 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: dc70ca0c-c933-46ba-91d9-08de5d5189fa X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.118.233];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BN2PEPF000044A8.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN7PPFB3F5C406F Content-Type: text/plain; charset="utf-8" Replace the old invalidation functions with arm_smmu_domain_inv_range() in all the existing invalidation routines. And deprecate the old functions. The new arm_smmu_domain_inv_range() handles the CMDQ_MAX_TLBI_OPS as well, so drop it in the SVA function. Since arm_smmu_cmdq_batch_add_range() has only one caller now, and it must be given a valid size, add a WARN_ON_ONCE to catch any missed case. Also update the comments in arm_smmu_tlb_inv_context() to clarify things with the new invalidation functions. Reviewed-by: Jason Gunthorpe Signed-off-by: Nicolin Chen --- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 7 - .../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 29 +-- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 183 ++---------------- 3 files changed, 24 insertions(+), 195 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/ar= m/arm-smmu-v3/arm-smmu-v3.h index 534e9a5ddca3..36de2b0b2ebe 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h @@ -1080,13 +1080,6 @@ int arm_smmu_set_pasid(struct arm_smmu_master *maste= r, struct arm_smmu_domain *smmu_domain, ioasid_t pasid, struct arm_smmu_cd *cd, struct iommu_domain *old); =20 -void arm_smmu_tlb_inv_asid(struct arm_smmu_device *smmu, u16 asid); -void arm_smmu_tlb_inv_range_asid(unsigned long iova, size_t size, int asid, - size_t granule, bool leaf, - struct arm_smmu_domain *smmu_domain); -int arm_smmu_atc_inv_domain(struct arm_smmu_domain *smmu_domain, - unsigned long iova, size_t size); - void arm_smmu_domain_inv_range(struct arm_smmu_domain *smmu_domain, unsigned long iova, size_t size, unsigned int granule, bool leaf); diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iomm= u/arm/arm-smmu-v3/arm-smmu-v3-sva.c index 440ad8cc07de..f1f8e01a7e91 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c @@ -122,15 +122,6 @@ void arm_smmu_make_sva_cd(struct arm_smmu_cd *target, } EXPORT_SYMBOL_IF_KUNIT(arm_smmu_make_sva_cd); =20 -/* - * Cloned from the MAX_TLBI_OPS in arch/arm64/include/asm/tlbflush.h, this - * is used as a threshold to replace per-page TLBI commands to issue in the - * command queue with an address-space TLBI command, when SMMU w/o a range - * invalidation feature handles too many per-page TLBI commands, which will - * otherwise result in a soft lockup. - */ -#define CMDQ_MAX_TLBI_OPS (1 << (PAGE_SHIFT - 3)) - static void arm_smmu_mm_arch_invalidate_secondary_tlbs(struct mmu_notifier= *mn, struct mm_struct *mm, unsigned long start, @@ -146,21 +137,8 @@ static void arm_smmu_mm_arch_invalidate_secondary_tlbs= (struct mmu_notifier *mn, * range. So do a simple translation here by calculating size correctly. */ size =3D end - start; - if (!(smmu_domain->smmu->features & ARM_SMMU_FEAT_RANGE_INV)) { - if (size >=3D CMDQ_MAX_TLBI_OPS * PAGE_SIZE) - size =3D 0; - } else { - if (size =3D=3D ULONG_MAX) - size =3D 0; - } - - if (!size) - arm_smmu_tlb_inv_asid(smmu_domain->smmu, smmu_domain->cd.asid); - else - arm_smmu_tlb_inv_range_asid(start, size, smmu_domain->cd.asid, - PAGE_SIZE, false, smmu_domain); =20 - arm_smmu_atc_inv_domain(smmu_domain, start, size); + arm_smmu_domain_inv_range(smmu_domain, start, size, PAGE_SIZE, false); } =20 static void arm_smmu_mm_release(struct mmu_notifier *mn, struct mm_struct = *mm) @@ -191,8 +169,7 @@ static void arm_smmu_mm_release(struct mmu_notifier *mn= , struct mm_struct *mm) } spin_unlock_irqrestore(&smmu_domain->devices_lock, flags); =20 - arm_smmu_tlb_inv_asid(smmu_domain->smmu, smmu_domain->cd.asid); - arm_smmu_atc_inv_domain(smmu_domain, 0, 0); + arm_smmu_domain_inv(smmu_domain); } =20 static void arm_smmu_mmu_notifier_free(struct mmu_notifier *mn) @@ -302,7 +279,7 @@ static void arm_smmu_sva_domain_free(struct iommu_domai= n *domain) /* * Ensure the ASID is empty in the iommu cache before allowing reuse. */ - arm_smmu_tlb_inv_asid(smmu_domain->smmu, smmu_domain->cd.asid); + arm_smmu_domain_inv(smmu_domain); =20 /* * Notice that the arm_smmu_mm_arch_invalidate_secondary_tlbs op can diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/ar= m/arm-smmu-v3/arm-smmu-v3.c index d94bb290d190..d0749577197d 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -1284,16 +1284,6 @@ struct arm_smmu_invs *arm_smmu_invs_purge(struct arm= _smmu_invs *invs) EXPORT_SYMBOL_IF_KUNIT(arm_smmu_invs_purge); =20 /* Context descriptor manipulation functions */ -void arm_smmu_tlb_inv_asid(struct arm_smmu_device *smmu, u16 asid) -{ - struct arm_smmu_cmdq_ent cmd =3D { - .opcode =3D smmu->features & ARM_SMMU_FEAT_E2H ? - CMDQ_OP_TLBI_EL2_ASID : CMDQ_OP_TLBI_NH_ASID, - .tlbi.asid =3D asid, - }; - - arm_smmu_cmdq_issue_cmd_with_sync(smmu, &cmd); -} =20 /* * Based on the value of ent report which bits of the STE the HW will acce= ss. It @@ -2505,90 +2495,27 @@ static int arm_smmu_atc_inv_master(struct arm_smmu_= master *master, return arm_smmu_cmdq_batch_submit(master->smmu, &cmds); } =20 -int arm_smmu_atc_inv_domain(struct arm_smmu_domain *smmu_domain, - unsigned long iova, size_t size) -{ - struct arm_smmu_master_domain *master_domain; - int i; - unsigned long flags; - struct arm_smmu_cmdq_ent cmd =3D { - .opcode =3D CMDQ_OP_ATC_INV, - }; - struct arm_smmu_cmdq_batch cmds; - - if (!(smmu_domain->smmu->features & ARM_SMMU_FEAT_ATS)) - return 0; - - /* - * Ensure that we've completed prior invalidation of the main TLBs - * before we read 'nr_ats_masters' in case of a concurrent call to - * arm_smmu_enable_ats(): - * - * // unmap() // arm_smmu_enable_ats() - * TLBI+SYNC atomic_inc(&nr_ats_masters); - * smp_mb(); [...] - * atomic_read(&nr_ats_masters); pci_enable_ats() // writel() - * - * Ensures that we always see the incremented 'nr_ats_masters' count if - * ATS was enabled at the PCI device before completion of the TLBI. - */ - smp_mb(); - if (!atomic_read(&smmu_domain->nr_ats_masters)) - return 0; - - arm_smmu_cmdq_batch_init(smmu_domain->smmu, &cmds, &cmd); - - spin_lock_irqsave(&smmu_domain->devices_lock, flags); - list_for_each_entry(master_domain, &smmu_domain->devices, - devices_elm) { - struct arm_smmu_master *master =3D master_domain->master; - - if (!master->ats_enabled) - continue; - - if (master_domain->nested_ats_flush) { - /* - * If a S2 used as a nesting parent is changed we have - * no option but to completely flush the ATC. - */ - arm_smmu_atc_inv_to_cmd(IOMMU_NO_PASID, 0, 0, &cmd); - } else { - arm_smmu_atc_inv_to_cmd(master_domain->ssid, iova, size, - &cmd); - } - - for (i =3D 0; i < master->num_streams; i++) { - cmd.atc.sid =3D master->streams[i].id; - arm_smmu_cmdq_batch_add(smmu_domain->smmu, &cmds, &cmd); - } - } - spin_unlock_irqrestore(&smmu_domain->devices_lock, flags); - - return arm_smmu_cmdq_batch_submit(smmu_domain->smmu, &cmds); -} - /* IO_PGTABLE API */ static void arm_smmu_tlb_inv_context(void *cookie) { struct arm_smmu_domain *smmu_domain =3D cookie; - struct arm_smmu_device *smmu =3D smmu_domain->smmu; - struct arm_smmu_cmdq_ent cmd; =20 /* - * NOTE: when io-pgtable is in non-strict mode, we may get here with - * PTEs previously cleared by unmaps on the current CPU not yet visible - * to the SMMU. We are relying on the dma_wmb() implicit during cmd - * insertion to guarantee those are observed before the TLBI. Do be - * careful, 007. + * If the DMA API is running in non-strict mode then another CPU could + * have changed the page table and not invoked any flush op. Instead the + * other CPU will do an atomic_read() and this CPU will have done an + * atomic_write(). That handshake is enough to acquire the page table + * writes from the other CPU. + * + * All command execution has a dma_wmb() to release all the in-memory + * structures written by this CPU, that barrier must also release the + * writes acquired from all the other CPUs too. + * + * There are other barriers and atomics on this path, but the above is + * the essential mechanism for ensuring that HW sees the page table + * writes from another CPU before it executes the IOTLB invalidation. */ - if (smmu_domain->stage =3D=3D ARM_SMMU_DOMAIN_S1) { - arm_smmu_tlb_inv_asid(smmu, smmu_domain->cd.asid); - } else { - cmd.opcode =3D CMDQ_OP_TLBI_S12_VMALL; - cmd.tlbi.vmid =3D smmu_domain->s2_cfg.vmid; - arm_smmu_cmdq_issue_cmd_with_sync(smmu, &cmd); - } - arm_smmu_atc_inv_domain(smmu_domain, 0, 0); + arm_smmu_domain_inv(smmu_domain); } =20 static void arm_smmu_cmdq_batch_add_range(struct arm_smmu_device *smmu, @@ -2600,7 +2527,7 @@ static void arm_smmu_cmdq_batch_add_range(struct arm_= smmu_device *smmu, unsigned long end =3D iova + size, num_pages =3D 0, tg =3D pgsize; size_t inv_range =3D granule; =20 - if (!size) + if (WARN_ON_ONCE(!size)) return; =20 if (smmu->features & ARM_SMMU_FEAT_RANGE_INV) { @@ -2655,76 +2582,6 @@ static void arm_smmu_cmdq_batch_add_range(struct arm= _smmu_device *smmu, } } =20 -static void __arm_smmu_tlb_inv_range(struct arm_smmu_cmdq_ent *cmd, - unsigned long iova, size_t size, - size_t granule, - struct arm_smmu_domain *smmu_domain) -{ - struct arm_smmu_device *smmu =3D smmu_domain->smmu; - struct arm_smmu_cmdq_batch cmds; - size_t pgsize; - - /* Get the leaf page size */ - pgsize =3D __ffs(smmu_domain->domain.pgsize_bitmap); - - arm_smmu_cmdq_batch_init(smmu, &cmds, cmd); - arm_smmu_cmdq_batch_add_range(smmu, &cmds, cmd, iova, size, granule, - pgsize); - arm_smmu_cmdq_batch_submit(smmu, &cmds); -} - -static void arm_smmu_tlb_inv_range_domain(unsigned long iova, size_t size, - size_t granule, bool leaf, - struct arm_smmu_domain *smmu_domain) -{ - struct arm_smmu_cmdq_ent cmd =3D { - .tlbi =3D { - .leaf =3D leaf, - }, - }; - - if (smmu_domain->stage =3D=3D ARM_SMMU_DOMAIN_S1) { - cmd.opcode =3D smmu_domain->smmu->features & ARM_SMMU_FEAT_E2H ? - CMDQ_OP_TLBI_EL2_VA : CMDQ_OP_TLBI_NH_VA; - cmd.tlbi.asid =3D smmu_domain->cd.asid; - } else { - cmd.opcode =3D CMDQ_OP_TLBI_S2_IPA; - cmd.tlbi.vmid =3D smmu_domain->s2_cfg.vmid; - } - __arm_smmu_tlb_inv_range(&cmd, iova, size, granule, smmu_domain); - - if (smmu_domain->nest_parent) { - /* - * When the S2 domain changes all the nested S1 ASIDs have to be - * flushed too. - */ - cmd.opcode =3D CMDQ_OP_TLBI_NH_ALL; - arm_smmu_cmdq_issue_cmd_with_sync(smmu_domain->smmu, &cmd); - } - - /* - * Unfortunately, this can't be leaf-only since we may have - * zapped an entire table. - */ - arm_smmu_atc_inv_domain(smmu_domain, iova, size); -} - -void arm_smmu_tlb_inv_range_asid(unsigned long iova, size_t size, int asid, - size_t granule, bool leaf, - struct arm_smmu_domain *smmu_domain) -{ - struct arm_smmu_cmdq_ent cmd =3D { - .opcode =3D smmu_domain->smmu->features & ARM_SMMU_FEAT_E2H ? - CMDQ_OP_TLBI_EL2_VA : CMDQ_OP_TLBI_NH_VA, - .tlbi =3D { - .asid =3D asid, - .leaf =3D leaf, - }, - }; - - __arm_smmu_tlb_inv_range(&cmd, iova, size, granule, smmu_domain); -} - static bool arm_smmu_inv_size_too_big(struct arm_smmu_device *smmu, size_t= size, size_t granule) { @@ -2924,7 +2781,9 @@ static void arm_smmu_tlb_inv_page_nosync(struct iommu= _iotlb_gather *gather, static void arm_smmu_tlb_inv_walk(unsigned long iova, size_t size, size_t granule, void *cookie) { - arm_smmu_tlb_inv_range_domain(iova, size, granule, false, cookie); + struct arm_smmu_domain *smmu_domain =3D cookie; + + arm_smmu_domain_inv_range(smmu_domain, iova, size, granule, false); } =20 static const struct iommu_flush_ops arm_smmu_flush_ops =3D { @@ -4192,9 +4051,9 @@ static void arm_smmu_iotlb_sync(struct iommu_domain *= domain, if (!gather->pgsize) return; =20 - arm_smmu_tlb_inv_range_domain(gather->start, - gather->end - gather->start + 1, - gather->pgsize, true, smmu_domain); + arm_smmu_domain_inv_range(smmu_domain, gather->start, + gather->end - gather->start + 1, + gather->pgsize, true); } =20 static phys_addr_t --=20 2.43.0