From nobody Sat Oct 4 17:29:51 2025 Received: from NAM12-BN8-obe.outbound.protection.outlook.com (mail-bn8nam12on2040.outbound.protection.outlook.com [40.107.237.40]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 23D5B1A238C for ; Thu, 14 Aug 2025 01:26:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.237.40 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755134784; cv=fail; b=sCoLL39J//XXMH9powt8twQCHDdOXpiE22o00I7rgd48Zqx0R+Uo+mLU6/SQfzzotZLwVZkyaZa2a+LARx8m4SQAhzd+cI1aoJoCrsl9q1bfVIsCUQjY879zwR80WrLgIegzyDUzKmrFH4RdtX8rD3VwNOkQWgsgXp1Hz6rtv6g= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755134784; c=relaxed/simple; bh=43ZMWBORDEpyjm+2IXDdkboOglfwTlt2zG1vUiJBlh0=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=Zjr0I7lmchsZ1wqicB0zQ+23if4+owRxlwYEWKr9Fno6wbbcrESS+xPxIRf0KIdHScjUOrOljNnZuFIn5fRYMgnUMJHhicXV3wqR57NQda5mnLIrgH7AzV7gpsV48JKo5dgs6U4BX8b/998m5r7SFG0NXrNJNFl0Mejs/2Xn2bQ= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=kPFDpa6K; arc=fail smtp.client-ip=40.107.237.40 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="kPFDpa6K" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=qcWWkLC8ISjiEcxOkJxua+qBRnOHZZcNRckHWiozcBUKjBT+Wxn+s1MNhuUgpO6YH6SQl08ET86DE75sWd9eAX/nO5iR4f3UWLAmF/BEPMT5pJqeP5dtkuGhaVBdIzzVcNWwELfwPKjmbfUbSHNXYCT4xIFOW+nv611N3gMkzl/VjeSfqsFQCv6vaDavnttVImhtdLOKa46CwTFJVy8tcponu9PZosGdqGGBFL3jqCEqrCoTDP8Xnjsd6CRJusXtxXcBsdT0X6gbeiJYARpaGn6AhyfN2N8nt6FBpH6RscnpIOWLChVrmEjhIRBflHCVVKu/6dxF97mjditIB/3H6g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=6dAYTIaV9Q0/Ly0dbrAP3ivsk4xFT1n521Lg7rIKLrg=; b=jCNefJDMsb8GHtAHqHCqgdopami4bVRISWynpRyhwEguKby72H9aXmNs5/ztZkWSvji7CE+5sGc+ArZWP4CaJXXzgCpaqs7ljymE+sP7oTbm2sL2sq5Ewx8GJf1P6v/Vy7jQxXs2G8jRSEtDzgihJd+JVU5ApQPynuzMBU3AyKfN91oLMYT+3Ahuwh/oW9ttw47t7v0A/rlKq/duqK1ZNeqe7jvKxHn3eHZ066TN1hG6GFXecb5sWUox0PmAqE/1z2rIt4SFi0R5ydufXylqn7lQL3gxpyr/24QMS1LDSoi8MlqozMZvEDc1Yg5X+mgS22QO3Du9OvNmWABY83f34A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.161) smtp.rcpttodomain=kernel.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=6dAYTIaV9Q0/Ly0dbrAP3ivsk4xFT1n521Lg7rIKLrg=; b=kPFDpa6KFmOUrvzvtXgJvJMAm8rikIHG5sUo5Cl9aQiDcCjXppMXYLeNGxOyjVda0/jc/TER47TsbZPsviG8d+2RyNgSaLngLtcbD/u/ipCCTpAx/e2Jc/OOa1N06vccl8S9x+qWpf+ehaIk9p2XUaAGEr5lnn6MBQgT2GPMK6X32WIPAzAMaIyI5gHCLjOSM/ZvdFK4p3A0FeCidtOnai2sUm8OQEZztBePx+tmTgvrSfFnkPbYtwjsgTI6KvCnHuiPvvsBH13d7Vk7U34s4ZjsGjmpWuBV0Q6560j/Bz7kfDLOoIwFyNb0CNASDZAsPSuIotcnHklSptxS3AYlvQ== Received: from BYAPR08CA0068.namprd08.prod.outlook.com (2603:10b6:a03:117::45) by PH8PR12MB6913.namprd12.prod.outlook.com (2603:10b6:510:1ca::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9031.13; Thu, 14 Aug 2025 01:26:18 +0000 Received: from SJ5PEPF000001F4.namprd05.prod.outlook.com (2603:10b6:a03:117:cafe::d8) by BYAPR08CA0068.outlook.office365.com (2603:10b6:a03:117::45) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.9031.16 via Frontend Transport; Thu, 14 Aug 2025 01:26:18 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.161) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.161 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.161; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.161) by SJ5PEPF000001F4.mail.protection.outlook.com (10.167.242.72) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9031.11 via Frontend Transport; Thu, 14 Aug 2025 01:26:17 +0000 Received: from rnnvmail205.nvidia.com (10.129.68.10) by mail.nvidia.com (10.129.200.67) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.14; Wed, 13 Aug 2025 18:26:02 -0700 Received: from rnnvmail205.nvidia.com (10.129.68.10) by rnnvmail205.nvidia.com (10.129.68.10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.14; Wed, 13 Aug 2025 18:26:02 -0700 Received: from Asurada-Nvidia.nvidia.com (10.127.8.9) by mail.nvidia.com (10.129.68.10) with Microsoft SMTP Server id 15.2.1544.14 via Frontend Transport; Wed, 13 Aug 2025 18:26:01 -0700 From: Nicolin Chen To: , , CC: , , , , , , , , , , , , Subject: [PATCH rfcv1 1/8] iommu/arm-smmu-v3: Clear cmds->num after arm_smmu_cmdq_batch_submit Date: Wed, 13 Aug 2025 18:25:32 -0700 Message-ID: X-Mailer: git-send-email 2.43.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SJ5PEPF000001F4:EE_|PH8PR12MB6913:EE_ X-MS-Office365-Filtering-Correlation-Id: 562d6e60-6821-41d8-ab08-08dddad1917a X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|36860700013|7416014|1800799024|376014|82310400026; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?3A8eDBpWUwWNnQ8WCI16Jxy9jqNic0VO3r4vVlQu4y5gXowxmNuJqAb3OrON?= =?us-ascii?Q?LfRISR9ocPO0cesOIzka87LCfasM8x9hsEcC3+PYu8dej4LtWePKSJa5f01c?= =?us-ascii?Q?seRB983fVHT3XxazndSQ6mdYQKAVUQHEPaz7AvNW/FoImImSLKDohlcZD/nv?= =?us-ascii?Q?DCtS8xQPNge0oWHmnY3FTmBXEB0f/U+7XBM9cggM4rW+x3zVe0ZrVZ+zPYtD?= =?us-ascii?Q?IZTUblvSANS6QcCJSw7aHYNMVLqCj+GEVXYxoxqEep1Y5VDooAgoFtznNYry?= =?us-ascii?Q?K29/umOkdvxeeKx68B4Drhh3oekZC/7JwP2hdyr2JGCNtqhoSzJhTAogiHDr?= =?us-ascii?Q?e/u6JPI1D+mho3bJH2KnM6D+bgGJ4BNLjXrFkw4hT4oBXZgygq/db+bxNFMS?= =?us-ascii?Q?FuKxUkIGnJjq6bY9QfreJn9elSBdPgfYJflQUNewaFGNWGkKyK3dY5eScNPY?= =?us-ascii?Q?pwQMmtMjGZZbsFY370FqEIAhmJAnReh1EHjydxvjFRPCxXsLG1Ln0163KaeM?= =?us-ascii?Q?7bCPzGfIffkL+rhc0moxD7KyBcy/AsK7+EM1Qx2OWsbizFTP/s+2bJGEJEkb?= =?us-ascii?Q?rofhyMpqMv9WO1El0PCLxx7GCqsL3HBuJNhhIhN5bCp2fMg2usOtn98fx9fe?= =?us-ascii?Q?5h42NZEwbQWCH9p4IDGm6zgWFck39c/qQctqdCWNBxwo6esV7CtvfufO2yiB?= =?us-ascii?Q?kiOrwwHECE5toBsiWFsZGbYJBJSndwNdbxCDTZ8ksIQ7ABQFwTfiGo3WihjY?= =?us-ascii?Q?nze4EuPLKb0g7Qs+YMTXyVMLffG2GucQHAE7CtbcB7jFxp59YW638v/Q4iCP?= =?us-ascii?Q?bS7tJZUF36/nuJDI2Kta2agefYqhFx6oYy/xfONz1jQR0iI5fk15tICTVbPt?= =?us-ascii?Q?0Dym3QvFBmMd6DCIVkM5Zds3qPQ75T/wM95JTqA7yXajyFHYYDPKjskjncne?= =?us-ascii?Q?UDVgVeewe3A0JS0+Kanjvt8naEuS5mIrtYfKpjZRaG7TgqEN1T0Ecx5W/b8+?= =?us-ascii?Q?Q2MnsRrRXy1RZrldG/sNvveulWoUsliMHF6Z0hivQj5aWFZslsM5rLf2h0eJ?= =?us-ascii?Q?01B60LfX+d0CfaZHet1t7Obd9zOSYl2bvABVD03cxBPnMT8VpxktxjgFpLwQ?= =?us-ascii?Q?cEz6bSys+sPVadK+pSQtyK7+iiKYsHl5eEdFTiz7pWDzqy0FVD5mD2LKvbsm?= =?us-ascii?Q?dB3xWC4+OnkgbQCd3mfR8FqsmyGuio+MnIEr0c+SzepeexaqThzsv0PrxdRT?= =?us-ascii?Q?h+Lz/QMeRnBCni0Tw6zm0t16zcYl0Br+m+/bNrbJEZkTMmm/DrR4dtxFYBR1?= =?us-ascii?Q?b7D3Zcl80ivgm5IZLsQjIxOEfvAPV/gXOTFrpy3rZnYr7PaQrLNV1xUhxiI/?= =?us-ascii?Q?ExYRvIlYqDgx+xqTt83nOpMWzWzY4ptvSUrhWVvGD3nxi2pnmZy2IBeHruse?= =?us-ascii?Q?jm08UEekaUGBB0+yi/h0se4SqR8aMz7Az4Ay3/pQB6HN1qELnJqecI4CcPIH?= =?us-ascii?Q?Ii1vnHHWQXmyiGvxKIQv0AY6iwAdDZ9l1e2V?= X-Forefront-Antispam-Report: CIP:216.228.117.161;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge2.nvidia.com;CAT:NONE;SFS:(13230040)(36860700013)(7416014)(1800799024)(376014)(82310400026);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Aug 2025 01:26:17.9578 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 562d6e60-6821-41d8-ab08-08dddad1917a X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.161];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: SJ5PEPF000001F4.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH8PR12MB6913 Content-Type: text/plain; charset="utf-8" None of the callers of arm_smmu_cmdq_batch_submit() cares about the batch after a submission. So, it'll be certainly safe to nuke the cmds->num, at least upon a successful one. This will ease a bit a new wrapper function. Signed-off-by: Nicolin Chen --- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/ar= m/arm-smmu-v3/arm-smmu-v3.c index 2a8b46b948f05..cccf8f52ee0d5 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -974,11 +974,17 @@ static void arm_smmu_cmdq_batch_add(struct arm_smmu_d= evice *smmu, cmds->num++; } =20 +/* Clears cmds->num after a successful submission */ static int arm_smmu_cmdq_batch_submit(struct arm_smmu_device *smmu, struct arm_smmu_cmdq_batch *cmds) { - return arm_smmu_cmdq_issue_cmdlist(smmu, cmds->cmdq, cmds->cmds, - cmds->num, true); + int ret; + + ret =3D arm_smmu_cmdq_issue_cmdlist(smmu, cmds->cmdq, cmds->cmds, + cmds->num, true); + if (!ret) + cmds->num =3D 0; + return ret; } =20 static void arm_smmu_page_response(struct device *dev, struct iopf_fault *= unused, --=20 2.43.0 From nobody Sat Oct 4 17:29:51 2025 Received: from NAM12-BN8-obe.outbound.protection.outlook.com (mail-bn8nam12on2056.outbound.protection.outlook.com [40.107.237.56]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5BA681A3BD7 for ; Thu, 14 Aug 2025 01:26:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.237.56 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755134787; cv=fail; b=KVElDhWvVOjjYaDycYsrD61acA6SUu2sKNvrPDSSXU+FoAIhK3UtdI7WzmBmlaSXaE5FEaYA5bKYr98Srclr7uOFW0+5DXP5Fh5XvzkVETnvhuWaAWEqo6+NEf75UM7aOHd7ArwGk2dTE6zfgGYfkelWxkDFlVjXTdw2o/tW3LM= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755134787; c=relaxed/simple; bh=DDogWgQw6MspgUUgMu6U1J6gzKIf8H1yrxuaRZXdpec=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=KwbUPYVmbBxQ8lgC56E/RfFgPczp3ML4IaKowRNv4bk65O4rcnZmhUE290PAg5ds3uk4I4XP7TkWHAn2FPgRyaHTDMMnWV3yWD/YMkyKyguXIfNwRbPSbnoygY6uJE2PwSEQ16+drrRvqNNoAy8yc+aYZoDRON70wk4gQaZPLCE= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=hrgB9KRS; arc=fail smtp.client-ip=40.107.237.56 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="hrgB9KRS" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=Qkm8FHgw5nHyKr8cXgkw997SkMBwxyEkxGx2Y2hLcBdim3T0l2jmhX3E/e7uUG6YuO7GGxTT1EpztjQcSm6JS/ALC4oZb58dtY+/RcqlMUAqSwUFYkb9vs4HS5+lRqgNxaWQLgxhW/BW5tgI2fm6/nxFyKW83yULZrJVT1kB5Ffvb6tP5husfzpDfIJZyx2pcD5/Hvfmo9fAvNiPlYeSLPf1/xawC0sVuFzQGH0RUxCQS8thXPzQu4ALAaiu9BVfkOSx4UNhTBitTuEzQYjY5zUkLbbY4fHr9n+5py3oTcIKPvMrsT6rkfaSKiBoxS+HSjk2mCygkfQHvJ4PYCWdRA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=8qqoeVnRECMfD1Pp4tZ2Fwc6QXLvQlgWCLGshn4ZE9I=; b=Qx0fdVuQ57eWknqg4rUsImDLIRqCeojfS7zQdUDGf3tUOwLPL9MGHiluJJvuo4MU67OqLo4Or77mpFSL8CkNjFGkoaHRl0U/sFCXa42Xl5wFrQZ5UFN5sQTLLGQnDRvx18OsjLKpxi4asNr9w5Nu3tczO6v/E8Mev1sDeexnuT7hz9LRFApLV+6awrbG0oaE7Pp8DBtDixsZBbmgMKDQCXbB01RIsb5XcOzRDkca7ndvpI6F7AkixjpqJwoHfOo1A6nm2IzR+kf+WadzLOCy1Ycl8BM0UjEGfo8P0GczLyLUIR6kdhHrlJPwtRDlgxdnKre/6VPQBA4LeYQspfDfdQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.160) smtp.rcpttodomain=kernel.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=8qqoeVnRECMfD1Pp4tZ2Fwc6QXLvQlgWCLGshn4ZE9I=; b=hrgB9KRSz2O+ebWtl+3ezqXDyZbFsOVeiKapSJvXOsirpFFDQyD5pf78gDqqhV9mhEFvc1kLIsoPkqxGRsTwyJTebIrf2jD1QfBlwP+/20+XMK3EMhK3pZ7MhPfk2IibpmB9dXMNwNWKP9s18xjRlxDeDXRHwt4bpdZbGrE3759QZWTbUTaR1JdGudXV1bqeGfNRkgJ19GvoLaSM5SzclAJDApCvKduZHMH6lDn9h7d06R5sHzQi2Pb24jnBABHs3UrgKBLspPTF1xnr3R/Pf5zgXo94VUDbDFRnxdXxvbPB9YX6ItfhosiOXuhvDh9OPZ6/GUZtQy3wVxh2tPoRIg== Received: from BN9PR03CA0642.namprd03.prod.outlook.com (2603:10b6:408:13b::17) by SJ0PR12MB7460.namprd12.prod.outlook.com (2603:10b6:a03:48d::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9031.16; Thu, 14 Aug 2025 01:26:19 +0000 Received: from BN2PEPF000044A2.namprd02.prod.outlook.com (2603:10b6:408:13b:cafe::bf) by BN9PR03CA0642.outlook.office365.com (2603:10b6:408:13b::17) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.9031.16 via Frontend Transport; Thu, 14 Aug 2025 01:26:18 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.160) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.160 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.160; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.160) by BN2PEPF000044A2.mail.protection.outlook.com (10.167.243.153) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9031.11 via Frontend Transport; Thu, 14 Aug 2025 01:26:18 +0000 Received: from rnnvmail205.nvidia.com (10.129.68.10) by mail.nvidia.com (10.129.200.66) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.14; Wed, 13 Aug 2025 18:26:04 -0700 Received: from rnnvmail205.nvidia.com (10.129.68.10) by rnnvmail205.nvidia.com (10.129.68.10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.14; Wed, 13 Aug 2025 18:26:03 -0700 Received: from Asurada-Nvidia.nvidia.com (10.127.8.9) by mail.nvidia.com (10.129.68.10) with Microsoft SMTP Server id 15.2.1544.14 via Frontend Transport; Wed, 13 Aug 2025 18:26:02 -0700 From: Nicolin Chen To: , , CC: , , , , , , , , , , , , Subject: [PATCH rfcv1 2/8] iommu/arm-smmu-v3: Explicitly set smmu_domain->stage for SVA Date: Wed, 13 Aug 2025 18:25:33 -0700 Message-ID: <826a71630caef33effbd82d918e7f7393fb4d377.1755131672.git.nicolinc@nvidia.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BN2PEPF000044A2:EE_|SJ0PR12MB7460:EE_ X-MS-Office365-Filtering-Correlation-Id: ef4e2337-2481-4c55-d84f-08dddad191c1 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|36860700013|82310400026|7416014|376014; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?Dg5a5askZmfSFLaUsasf03aqwwAvgxQ/OvGhkeAg6FVDpBTy0CKzoWv0ZqrF?= =?us-ascii?Q?wkbmk5zGSnDfQl4mswdGhnnmmAp+xc6AxoTvwx/fZHsjqN0eP2sYKlhDDwg4?= =?us-ascii?Q?mlYYD+fHIsFNM8xprHaqUrqf7bJdl2q+uw1oAdUZ1AL5UE5h8flIzbm1AIfc?= =?us-ascii?Q?8omLcfFFnnRYIDBZ7LxqoYMr+LJW3ubo6uyrmdBszQVTztOVorS95KgtH2Ii?= =?us-ascii?Q?shARMf6t7WNArF3ySeAZlZUQ42ugpLHpYmBeMjfjCqQ6z3xkUZkqaVv9LLIM?= =?us-ascii?Q?sW4IIoEs6NSPPEalBaBxmocu6k43f0TpfMKMpFmOr2eIu1w7wPzb29zuQ1C2?= =?us-ascii?Q?e0HpstyaYFoSa+BZ3qDdFYlTSLVm3t/9+qC/MDQHpbGukxdl2f77Cg9ZM6H3?= =?us-ascii?Q?nPDxiKle05AvWniUek4txbuswot0WKRoSbDkMbGjDcfMJFio6AIW2DSj306Z?= =?us-ascii?Q?kYViOeVW9YSS2Hle3kNlLTaTobsUOYqrwHbZ2RsosTkJNzNGy2kDqHtIctPj?= =?us-ascii?Q?yqHXRit1Mo/tGapR25j/UhU/mI1U/wATlOIXukqzScUH+KTFRRPfVlbL50nm?= =?us-ascii?Q?3k6fcDT3qFDch6RuIOKkpzV+026tyQ6xkwMhuXL1t4t3V2n1iYlBc8c4wydg?= =?us-ascii?Q?57S9A8/3lBsa/efqYC+SDUhkX2U0GD8XtmJB+sfAgmJe0ixPPxRWROsDepRh?= =?us-ascii?Q?q37A1nXbk26k/7kVn3KouwOc2xgYODQi7vjFKsCpGvvk9UdaWB0TyBPO8mT2?= =?us-ascii?Q?00LH7omWnS34ymDyyohsh8pjcOS5XV1IuV61u5ocisGbzD7zSadA+OAfCaXC?= =?us-ascii?Q?yy70BiCH1Aq7hAs+cEzaZeoQH/YuStMb9x5dphuytpcX8M/byBCLErgTLcqs?= =?us-ascii?Q?mSjP7qJqXoTx3R3ve8ZCQNR0Zv0FSeJ1IQfSxxVdTaZHZ17UKXd+A/0yvJfs?= =?us-ascii?Q?cRmKYJTYAmeEPFqlb9OnWbvleGqovMrjIxpK0ZL9oNsmbfWI0eadReHW8KIN?= =?us-ascii?Q?0EA557Kxo0vKZ2Wk7WvTv0YUQEvhaQ3/49r/J2jAmAjkqe8yIc5bwelstC2J?= =?us-ascii?Q?YM/jfQjw+uaCln8/Dy1Mp7WRGymXJio2pRmpEoynDjcg2MfLEchJFdys4lPR?= =?us-ascii?Q?ARA1eOldrd1IgHemEPfMFr177hCFz8qJdO4AVCT+Y49QMiBDlXKuRE2T7W9N?= =?us-ascii?Q?ZjyXSoIEy/H+s9nwahR7eqFxekO8GBHPg/twBFvWTzrisDoYFGuAqZclLgo0?= =?us-ascii?Q?j1m/oqsWINgQ+1BJzBCoJWdfYzhoU2BbINW6L8eu9s9tnVBwr5yaCxhqZyV6?= =?us-ascii?Q?znpwz3qy65zqWgYE6jdN9Bh2rz/buTSnn8lsza0rhFsPBxoEWjNrcEmZie6R?= =?us-ascii?Q?fOmxnGW5HW1yZhARwVqoz/igUCU+a+F8ffXMolw5bayQVI6LA2nCf3DB7ZRc?= =?us-ascii?Q?Td5LRg5txzQlHG3YckwJ+hLhHFdnCaGf1PFvVxeG6uCyKmzKohuRf4kzuu45?= =?us-ascii?Q?xJKbbMBTHgu4sNg6dfkleNxDC1DAOHlrWtBW?= X-Forefront-Antispam-Report: CIP:216.228.117.160;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge1.nvidia.com;CAT:NONE;SFS:(13230040)(1800799024)(36860700013)(82310400026)(7416014)(376014);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Aug 2025 01:26:18.2814 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: ef4e2337-2481-4c55-d84f-08dddad191c1 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.160];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BN2PEPF000044A2.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR12MB7460 Content-Type: text/plain; charset="utf-8" Both the ARM_SMMU_DOMAIN_S1 case and the SVA case use ASID, requiring ASID based invalidation commands to flush the TLB. Define an ARM_SMMU_DOMAIN_SVA to make the SVA case clear to share the same path with the ARM_SMMU_DOMAIN_S1 case, which will be a part of the routine to build a new per-domain invalidation array. There is no function change. Suggested-by: Jason Gunthorpe Signed-off-by: Nicolin Chen --- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 1 + drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 1 + drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 3 +++ 3 files changed, 5 insertions(+) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/ar= m/arm-smmu-v3/arm-smmu-v3.h index ae23aacc38402..5c0b38595d209 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h @@ -858,6 +858,7 @@ struct arm_smmu_master { enum arm_smmu_domain_stage { ARM_SMMU_DOMAIN_S1 =3D 0, ARM_SMMU_DOMAIN_S2, + ARM_SMMU_DOMAIN_SVA, }; =20 struct arm_smmu_domain { diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iomm= u/arm/arm-smmu-v3/arm-smmu-v3-sva.c index 59a480974d80f..6097f1f540d87 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c @@ -346,6 +346,7 @@ struct iommu_domain *arm_smmu_sva_domain_alloc(struct d= evice *dev, * ARM_SMMU_FEAT_RANGE_INV is present */ smmu_domain->domain.pgsize_bitmap =3D PAGE_SIZE; + smmu_domain->stage =3D ARM_SMMU_DOMAIN_SVA; smmu_domain->smmu =3D smmu; =20 ret =3D xa_alloc(&arm_smmu_asid_xa, &asid, smmu_domain, diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/ar= m/arm-smmu-v3/arm-smmu-v3.c index cccf8f52ee0d5..0016ec699acfe 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -3070,6 +3070,9 @@ static int arm_smmu_attach_dev(struct iommu_domain *d= omain, struct device *dev) arm_smmu_install_ste_for_dev(master, &target); arm_smmu_clear_cd(master, IOMMU_NO_PASID); break; + default: + WARN_ON(true); + break; } =20 arm_smmu_attach_commit(&state); --=20 2.43.0 From nobody Sat Oct 4 17:29:51 2025 Received: from NAM11-CO1-obe.outbound.protection.outlook.com (mail-co1nam11on2075.outbound.protection.outlook.com [40.107.220.75]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 72E811D5CED for ; Thu, 14 Aug 2025 01:26:27 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.220.75 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755134790; cv=fail; b=fJpIXwR7jCFb/Jf29g4ThFYktGyDLRtb2gt9R5++bpI+KqTOGoOvJIxHRQKfwm0W0cb5YLkk4AKSDmzhTzcb9MepumZ+jOg4T6Zj/PVCxHm1qGQPprlIZkfZ7qWYpQqJnWFFa2/m6I8ho98GSXuqEiV4Wpqds6a658rDGg6hI4c= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755134790; c=relaxed/simple; bh=blIQBQYJAfmmH54Xp/NOl8LRNtytQ6IN41ywkb1z5WQ=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=V2Jny4t3M2yIlkGt0VwuE6dK3XkkLye8+666q+uvVrNSNz32mN/qwUi69CQTYb7LceztXtQydvROT/MWHlGNDZNrlRESEZ/c3ys1OlFPY8BpBGGJz1yml+yGTpBNZtEW/JMzvW6uAutIn/s7GnCmnCDgOFjaOnekugfohhc7SQA= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=mtF16Thl; arc=fail smtp.client-ip=40.107.220.75 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="mtF16Thl" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=N9OifM/Omv1Iwa1FoJY638rgqq6yV8zhJkg+hes+ZeOwuJtQduh3SqhAMAWRXVn5taEcew51fKKoVOJLmBCK774Eg18W2QJt4EKOsOOPz1FBriNlD3cXdZKc6HmjeLxVJH/QgI7JEYx0J3CpdheIQzIcDqj63jIz5nBV8Z3AklbR/RVshIBph7MipJLeFTmFama2kmXOojXAg4WlJ9+asTsFaEKNufFOuTQ1u9XMGRwD17bfUcyIfhfNS8EDrbsk9muIeK7JBa7L345AlcejmnLGjqQ1E/bxkYWhO611AsnyzwbuiBeEX++1IukO3pwi/79Vu2XAkANDOccNaSxIWw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=n9tdY7vma3BZOIB/pf+qIbNyJWlUDjlUslTFVsNzYF4=; b=Ays3E6P8v1zuHSLQyAS8lH5Nrm2n3Q2FmO+VNq6nXbZz/nVAeSUaJPqb9wdSsIJf1FAW9G9v33HgtX+wwPfS0e91imnFxA1cWr6E/mlkVH8JOvAfoHHt/xv4sf/UnrmlJM0gLaxZS5dreqFLEprh9DXZHT6nOs8/IEcmAxnJR1N2mbFgfsSoFqNtNRbiZ+PQD/8H+1FCtAXGxqDvexCTsoFweqbBGRKvBvmfWelMlprJ/dpAzYLL5V1RSRpxZ3GqJCJ5RtVPUA6awCrz5ekvSOWz/VIFabfT61TY+eQhafHekRgJ6yIoSPIlZL/z6DW1CHvNxKBevaS49tdELj8Vig== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.161) smtp.rcpttodomain=kernel.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=n9tdY7vma3BZOIB/pf+qIbNyJWlUDjlUslTFVsNzYF4=; b=mtF16ThlgJzLkXxQ3Q3sGXCGzkEoEwnf4sRpQTsobyKln6V5Kkk5eq26DNhz7zNRVWs85sZT/mgxkTZtBOZ3naGHZYQsMZN0jgYqzvExnVeKuO2mZQZU1aw6XyVh3m+Vd/6WxmJLM8yv5MT8GbcUWYOYX9VMtbUhqT5eTC5euBRvM4Eqg3IJJYwKjX73bH4N8o0xn7ISVFSUxrrnBNGAXkBzNulNqNj/sCwHLyvElHxsowt6VLJXt63DAdsAz7Eg2xu04ao+xtUm1tGT6FHXudzIjYY8kuF9cWuOyLtP8o82YLfOwyaNQ5kTkZ2qC6Z/0sraZTHcrj9OL908UmVMGg== Received: from BYAPR08CA0071.namprd08.prod.outlook.com (2603:10b6:a03:117::48) by IA0PPF44635DB8D.namprd12.prod.outlook.com (2603:10b6:20f:fc04::bcc) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8964.21; Thu, 14 Aug 2025 01:26:23 +0000 Received: from SJ5PEPF000001F4.namprd05.prod.outlook.com (2603:10b6:a03:117:cafe::59) by BYAPR08CA0071.outlook.office365.com (2603:10b6:a03:117::48) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.9031.16 via Frontend Transport; Thu, 14 Aug 2025 01:26:21 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.161) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.161 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.161; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.161) by SJ5PEPF000001F4.mail.protection.outlook.com (10.167.242.72) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9031.11 via Frontend Transport; Thu, 14 Aug 2025 01:26:21 +0000 Received: from rnnvmail205.nvidia.com (10.129.68.10) by mail.nvidia.com (10.129.200.67) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.14; Wed, 13 Aug 2025 18:26:05 -0700 Received: from rnnvmail205.nvidia.com (10.129.68.10) by rnnvmail205.nvidia.com (10.129.68.10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.14; Wed, 13 Aug 2025 18:26:05 -0700 Received: from Asurada-Nvidia.nvidia.com (10.127.8.9) by mail.nvidia.com (10.129.68.10) with Microsoft SMTP Server id 15.2.1544.14 via Frontend Transport; Wed, 13 Aug 2025 18:26:04 -0700 From: Nicolin Chen To: , , CC: , , , , , , , , , , , , Subject: [PATCH rfcv1 3/8] iommu/arm-smmu-v3: Add an inline arm_smmu_domain_free() Date: Wed, 13 Aug 2025 18:25:34 -0700 Message-ID: X-Mailer: git-send-email 2.43.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SJ5PEPF000001F4:EE_|IA0PPF44635DB8D:EE_ X-MS-Office365-Filtering-Correlation-Id: e7c10e49-456d-430b-8e5d-08dddad193b7 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|7416014|36860700013|82310400026|1800799024|376014; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?uHJgskRqArwd7LY2NWF2dkQHv125ZIuIONnD+XOTQOFmitjQ1nd1Oeu5LyzU?= =?us-ascii?Q?OXtIxMFdnFayvv4ezVodzUvj2kXaGzYpxTpW4OT69QV0DShOOFDDpuy1Q+TL?= =?us-ascii?Q?0v1FazDfSaSisTkMBShYhxB1TzXfz4KiqxPHqYeQFS+NikQQg9ZXN7ocwB7t?= =?us-ascii?Q?jj/pb9SMSRM+HaQ4afsPDKZIww+4AcnYpLqO3mxsc6dUSpEmAQGzQJ2An4CI?= =?us-ascii?Q?nXkhiVeRSjlxjvDUu+yqQX4NX3rMtf/8UyenNkv/BuXgBHP5OFFklfLYRBl+?= =?us-ascii?Q?s4LaaYBzsZe2BY+yfsJ1m2BzfY40WfLsIm03CTdmw8vdeWxPyeSeAWZXSCno?= =?us-ascii?Q?17hXJcyOjb2K2W78tyZuxoQwWa1eE4tvfTg9fz5d2QT05LndffoYkknEMiGR?= =?us-ascii?Q?fm0qgZqvW62f5VIr94MBrhAHmu9KEmAujnVdS7wg1CZCQesKUJO4Bc3ZhtC+?= =?us-ascii?Q?CuWSf835Sy99l1N+MX+PR/eB1ElvKJqu9/IcZn1iG4zwQ7PR9BHwNJyRybPs?= =?us-ascii?Q?WLGDoHoskoHWUkBS8e2581Mz2+gM9EbC3iT78ItEh3/o7XULTonRzhD4FzTW?= =?us-ascii?Q?JfcKzzil+VecYv/+MXIgwksyHcLKT69HDicZpuQWfmaCbWEn1VrDiJAhQUJd?= =?us-ascii?Q?FUPaf6i/VScFHxNfAciRM6lhcaX6f65r+q5qM2YAJvzdJckW1Z7+syAFRNQY?= =?us-ascii?Q?MstqOHoE9EUEKh0Qh/dBuB+N4PdOKU0fabNfwAdBNQa1pleiaKF75TQp8Rd4?= =?us-ascii?Q?uk7R5bFmAqgzLfD9YphnmygY8cDM6F2IhS1b0iptsrpsPnrdZBDMp703WUx+?= =?us-ascii?Q?bZXSQb3SRywkt4lRQuKQQ+X54sOwRe7TTpa4PeuWKfrwjy4TLKGYmm38KbO4?= =?us-ascii?Q?PPSIOIdQzjfb3SVCkof5sadXwcGu/+jHqNKQY6idAsXpiWgjBml8sZzNVc1G?= =?us-ascii?Q?GzBf+bRwRBjHg3TpZjlsgjmJ7DJFpoNtvH0b74YstIr/lQVTqwdWFDE3TQMG?= =?us-ascii?Q?9Ew5SCk03//EhH4cXBh4zUPLQhUB+EI1BdhrKflDI1E1r9+1bcOgGEFo6QAs?= =?us-ascii?Q?ySV9Lz/k7OZoSYJ42Pn9L8qZTbS2PIl6BkMWwoik0g7XMbE6LULyxAdgjPiv?= =?us-ascii?Q?zFNi9fv3qaodyYB4z4yRn1FdAjiQWITgmQwzuMtwL7zbb2E00aUVwNXXDfPR?= =?us-ascii?Q?OFBcTM/3hSqRAgVujuBOaeQGIU0KgPoh6qdC7WDP5L6hFaynKIAmz7Hyq4ij?= =?us-ascii?Q?6y2unRHwOMw+RB4xYfLINq/mUZpFZjUF+7PUAWdbOjjp8xPqFHIQV97VvaFA?= =?us-ascii?Q?auDIgCetOqWyF2ZeK+fLf9Qvd0XfykGaa1K4eHzVXVYEexIzcPtg1zebb+lH?= =?us-ascii?Q?x9C4uEhOHy9zCY5Vja+zgCgtpbXOI+L25sfYDhh9Syu2/icY1VnD0PajqgKe?= =?us-ascii?Q?pOH3/qOsGy1WoAnQMRtR8zLqXE6l72muZGzR+4BsT3h0SmQegf6TkDH+DgD0?= =?us-ascii?Q?rrzQBc0kZZ3eyENqbyZrNbVYLrMDEqTYNtJx?= X-Forefront-Antispam-Report: CIP:216.228.117.161;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge2.nvidia.com;CAT:NONE;SFS:(13230040)(7416014)(36860700013)(82310400026)(1800799024)(376014);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Aug 2025 01:26:21.6454 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: e7c10e49-456d-430b-8e5d-08dddad193b7 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.161];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: SJ5PEPF000001F4.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA0PPF44635DB8D Content-Type: text/plain; charset="utf-8" There will be a bit more things to free than smmu_domain itself. So keep a simple inline function in the header to share aross files. Suggested-by: Jason Gunthorpe Signed-off-by: Nicolin Chen --- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 5 +++++ drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 2 +- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 4 ++-- 3 files changed, 8 insertions(+), 3 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/ar= m/arm-smmu-v3/arm-smmu-v3.h index 5c0b38595d209..96a23ca633cb6 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h @@ -954,6 +954,11 @@ extern struct mutex arm_smmu_asid_lock; =20 struct arm_smmu_domain *arm_smmu_domain_alloc(void); =20 +static inline void arm_smmu_domain_free(struct arm_smmu_domain *smmu_domai= n) +{ + kfree(smmu_domain); +} + void arm_smmu_clear_cd(struct arm_smmu_master *master, ioasid_t ssid); struct arm_smmu_cd *arm_smmu_get_cd_ptr(struct arm_smmu_master *master, u32 ssid); diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iomm= u/arm/arm-smmu-v3/arm-smmu-v3-sva.c index 6097f1f540d87..fc601b494e0af 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c @@ -365,6 +365,6 @@ struct iommu_domain *arm_smmu_sva_domain_alloc(struct d= evice *dev, err_asid: xa_erase(&arm_smmu_asid_xa, smmu_domain->cd.asid); err_free: - kfree(smmu_domain); + arm_smmu_domain_free(smmu_domain); return ERR_PTR(ret); } diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/ar= m/arm-smmu-v3/arm-smmu-v3.c index 0016ec699acfe..08af5f2d1235a 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -2498,7 +2498,7 @@ static void arm_smmu_domain_free_paging(struct iommu_= domain *domain) ida_free(&smmu->vmid_map, cfg->vmid); } =20 - kfree(smmu_domain); + arm_smmu_domain_free(smmu_domain); } =20 static int arm_smmu_domain_finalise_s1(struct arm_smmu_device *smmu, @@ -3359,7 +3359,7 @@ arm_smmu_domain_alloc_paging_flags(struct device *dev= , u32 flags, return &smmu_domain->domain; =20 err_free: - kfree(smmu_domain); + arm_smmu_domain_free(smmu_domain); return ERR_PTR(ret); } =20 --=20 2.43.0 From nobody Sat Oct 4 17:29:51 2025 Received: from NAM12-BN8-obe.outbound.protection.outlook.com (mail-bn8nam12on2085.outbound.protection.outlook.com [40.107.237.85]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4927819755B for ; Thu, 14 Aug 2025 01:26:30 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.237.85 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755134796; cv=fail; b=eQdqzEsXKbW5HsBB/YCcgNq59uVwnWCzJZCq7MlAvSXTkEJQm+LdRdbiATRJVMO1n9ZK9+jupOU0NrXmdgTkWCLRDuaD159mcVe0Hhh8jADM9gB8xIWvRBBjLFedRyk9UNtJBl7eU8/U5ygPON8NEb4rk1fcsp18+ZeRjDy5Hhs= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755134796; c=relaxed/simple; bh=ozp33+2gloR6AiJ19PMkq7LcwscI3WW8eijdK+Vs1TI=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=MjiqKENUBCWPb4he1aQ83J++bhJ9BJrXpR0Ms3aXHfb/gAhF2hqaKsffObiU4HGZUpHfVTlhZEIQ73FygMK39N7NIQHN/oSh2WocZg1qUF7SQxgSZ9SWEtrhOKjQOl0Y1yhGAJs8pKR1kFAaccE/PTQA5YLvQiQXusGP+5PSgHo= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=qC9vert7; arc=fail smtp.client-ip=40.107.237.85 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="qC9vert7" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=qDeM3X2Pnb+sN1XiOhpwQbPtDvbKLGhHMGyV/ssmKsUzFRa+NHEYdUZQIuCRLsABZC9p8GscQZ7A5K3gvtcQ4E4rBLPZYjUQsMZtn4vPU/bFGyxkrx1zvTnQdOrjtUVLnBHXvPgt6rvN0JPrFFPkbTpbg1qyTmqvdNa+216hwcDOL6CKpTYfnJ/8fV0Nnpy6Kago0PQ1v5JX5ErmpMxey73JI+v7ITIdOKXvJ2+nA+J4DpDQS2Uzh5/mANCoUT8ZwoDPFLiMuL3r75cJAMyFC5K0dSUqriJKtKrZv7171yMgByn9nF3snWtRUlUxoR/6Mm+0dcT6ALiR8/VhIBMm3w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=X7+DOaHzvoBkZ9ieUqsGvU4RjP2MgFaQhBiQNV17Gi8=; b=ZwTxBgivUMDSVjEXVSVDnlyJ89Z2XP828hsDejrMGHYCPvPEceHzKZ+ecaI2nPxjc5Pfa44zpiUnUEk5F8YFs+kgPGgKYmN4mXMcsBNqDETtSngkW3Nncok6gEzfyGkYQcHoCGAiAqu6KrezXQtKhziuorwm4oQMfYmeOiSmarKU7gU6D2qQAukF2kqDysTOIcxZuMF/mytENyEUaLlRT4DUCD9KMLGxnlr9GqknovrpbDaf0NWhXFbWaKYZAyDqFb3fCsTO6YH3jGmY3Hk4vuhjv1dTZOKUsz2baa9UzjEpqmx58oTg4AiJkzYeVZYYU9GCErGPO608GbB0OvwHCQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.161) smtp.rcpttodomain=kernel.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=X7+DOaHzvoBkZ9ieUqsGvU4RjP2MgFaQhBiQNV17Gi8=; b=qC9vert79MlqZdzbRis3h2M149m1WrcKEE5ez9kdKhJfwNbcFxIdS4bRxZUt3b07Cwi4UE6YSSMQH0koKecwPce6/CV+RWN11F67DUZl9tZHiqa4scC7oH6GZSORbtUOPZasVkoOr5WcnmZFP/g31WsbcNnFiQBBN1O35OgCKyXD1LEI4JToLjaiNA+fdizoUMW4x1dX/Fv2f1v1ePUivg4utxl2xYUZxglYmY45sL4q0OFY4epvpN91PIXoLECzNmvvyjzcrcBn2IUjaE35h0Ju5xkraJB7bwvecoT0vBG2ZgO6UiBC+bupbxz/zuiauMz66PCWaTwqkQXXi+thiA== Received: from BYAPR08CA0016.namprd08.prod.outlook.com (2603:10b6:a03:100::29) by MW4PR12MB7288.namprd12.prod.outlook.com (2603:10b6:303:223::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9009.22; Thu, 14 Aug 2025 01:26:23 +0000 Received: from SJ5PEPF000001F6.namprd05.prod.outlook.com (2603:10b6:a03:100:cafe::61) by BYAPR08CA0016.outlook.office365.com (2603:10b6:a03:100::29) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.9031.17 via Frontend Transport; Thu, 14 Aug 2025 01:26:23 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.161) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.161 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.161; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.161) by SJ5PEPF000001F6.mail.protection.outlook.com (10.167.242.74) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9031.11 via Frontend Transport; Thu, 14 Aug 2025 01:26:23 +0000 Received: from rnnvmail204.nvidia.com (10.129.68.6) by mail.nvidia.com (10.129.200.67) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.14; Wed, 13 Aug 2025 18:26:07 -0700 Received: from rnnvmail205.nvidia.com (10.129.68.10) by rnnvmail204.nvidia.com (10.129.68.6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.14; Wed, 13 Aug 2025 18:26:06 -0700 Received: from Asurada-Nvidia.nvidia.com (10.127.8.9) by mail.nvidia.com (10.129.68.10) with Microsoft SMTP Server id 15.2.1544.14 via Frontend Transport; Wed, 13 Aug 2025 18:26:05 -0700 From: Nicolin Chen To: , , CC: , , , , , , , , , , , , Subject: [PATCH rfcv1 4/8] iommu/arm-smmu-v3: Introduce a per-domain arm_smmu_invs array Date: Wed, 13 Aug 2025 18:25:35 -0700 Message-ID: X-Mailer: git-send-email 2.43.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SJ5PEPF000001F6:EE_|MW4PR12MB7288:EE_ X-MS-Office365-Filtering-Correlation-Id: eb17f76f-a973-4496-060c-08dddad194b0 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|7416014|376014|82310400026|1800799024|36860700013; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?11jGr5FRQ+lmntcaiQ2dcOH5GXZJUS5OUkqxhLNFUes6Vsk9jr8Ux34iAzCi?= =?us-ascii?Q?mqfip1yz0DxH+JwaDc+yWXtoX6A8GaGuDvQwXZHfaZAwyMyl9K1Jq8U/I/5s?= =?us-ascii?Q?9VWzqJmrGEHCQN2ce7E6pJIKxKk5WCIVJSqjACEa0tOA6+uc+s7pn8/RJ3rL?= =?us-ascii?Q?arNm4HEK9VfT2PGC1kjH+6pW1VntRLChnsuKjQzx6YyzFaXs0ylybjmMbfw/?= =?us-ascii?Q?JUTt4SjmLKecvBwpE3D32OLaZH6G/sPmnI36h3paLGYNotPRLsfCoTuf6eVs?= =?us-ascii?Q?uOrVFqXWmX35YgRjc4zbEtiJLBpcWH07ruzT5GpdA6Ie4JEOJv1BnE2KYYrF?= =?us-ascii?Q?qo+CEYaoVPnlaCwFxv/b8wUGUnrvgTzwZL9bkqxFqYrv32VdvfgEwk0Qry6d?= =?us-ascii?Q?GRj2LsLV87fbQiP+NfWuuxCupZXQWYMeBwitkexbyC5L0J9TvecbN6lcoMDt?= =?us-ascii?Q?p4FU7M6V/PZWJg0bwh8kheETjaROplI9KERXSQbP+X8eHJ3lq16sPVQTa1LZ?= =?us-ascii?Q?gZKr8xWEUfZQs0HaRvq6bhrtamPMvO1Ee89bJ6ppRTf3T8oFXU06cqE9TakU?= =?us-ascii?Q?NnSDreha97oAklIE1BpTTxIobvVXxj0IJFiqNFLrCBWZzKe100zROya5RBKn?= =?us-ascii?Q?HGmc1YIk7H4eBdqZ4F2p0fc0KA2mZHxP3sl0Es1QUeo8QmbLNoPjX2m6t6CD?= =?us-ascii?Q?tERj0ATO88zEPAt6Jbm1/yGyQRbNWvjclNsp2e7zfkqIRpDwg8wDGhUO1EXt?= =?us-ascii?Q?5t1oSPhV4AsxipjyKvmVRkdT7W/MX6r0u8QABGMLPQ1p1VsSM7ARwKqzbE6H?= =?us-ascii?Q?9oI9paH8giOYk5Isi+edIVuodobEk8HgWz+Se48EIqERxk6Ij46EcPwfO6Xm?= =?us-ascii?Q?fzBasYhgJqYqyVbuvEmVnTCPVzwBmuIZyHCxjFx1KC/rweda5hx/6tC2/zw0?= =?us-ascii?Q?X3uMVzmFZL8SfX8B5l777FwQ+PuHUdip99DsVxyDsjVDWbgq5O2ZdTIcv0gU?= =?us-ascii?Q?art6kjF9u19P/yTvBGG+ahmqF94n6tJgdBOUQhaTwpm29NJCrC0WT8b8k9HJ?= =?us-ascii?Q?FzlREzZGzGhhFjuMeVbTmg/ur0bM5YP+YYCX8b5qewZbeqkzCE17oWWVtVFC?= =?us-ascii?Q?vnRLYA/aGtCHDembFujLJ2Z2nBDCkdxCZcLgnzS8995AxgPj+WC7WTRXfHgK?= =?us-ascii?Q?mw2wc8yy3GyI/TFMilc5yuxkFvNMHrG9vLC4teM+wmQ4wdlX8DabqYU1op2g?= =?us-ascii?Q?HwpuTQDM7w9ZhdR3oyg9SvC/2C3fsJC/TNmNLhLbKlZ3QitUp+wh/JGgzv5Z?= =?us-ascii?Q?qHBAb8+AYk/oyBjy0CgfWk86F4BiM8KnCeyFv+eKtfdQBkvIUY+ahclzz4xN?= =?us-ascii?Q?pgO4glN0z1Mya8Wb6gErD0nYOggZDibYuoWNGZmcKdLb7BzMNAXmmVkw44rG?= =?us-ascii?Q?rnWCYLZnpgoTUjr71Z5clnYaL+dcRMTL/UYcd9JIgNv6eOuAoPQd9o3g0/8z?= =?us-ascii?Q?Gte08vEcxxLveLKUuiosh9k0igZE+lerFo7i?= X-Forefront-Antispam-Report: CIP:216.228.117.161;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge2.nvidia.com;CAT:NONE;SFS:(13230040)(7416014)(376014)(82310400026)(1800799024)(36860700013);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Aug 2025 01:26:23.3072 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: eb17f76f-a973-4496-060c-08dddad194b0 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.161];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: SJ5PEPF000001F6.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW4PR12MB7288 Content-Type: text/plain; charset="utf-8" From: Jason Gunthorpe Create a new data structure to hold an array of invalidations that need to be performed for the domain based on what masters are attached, to replace the single smmu pointer and linked list of masters in the current design. Each array entry holds one of the invalidation actions - S1_ASID, S2_VMID, ATS or their variant with information to feed invalidation commands to HW. It is structured so that multiple SMMUs can participate in the same array, removing one key limitation of the current system. To maximize performance, a sorted array is used as the data structure. It allows grouping SYNCs together to parallelize invalidations. For instance, it will group all the ATS entries after the ASID/VMID entry, so they will all be pushed to the PCI devices in parallel with one SYNC. To minimize the locking cost on the invalidation fast path (reader of the invalidation array), the array is managed with RCU. Provide a set of APIs to add/delete entries to/from an array, including a special no-fail helper function for a cannot-fail case, e.g. attaching to arm_smmu_blocked_domain. Also, add kunit coverage for those APIs. Signed-off-by: Jason Gunthorpe Co-developed-by: Nicolin Chen Signed-off-by: Nicolin Chen --- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 79 ++++++ .../iommu/arm/arm-smmu-v3/arm-smmu-v3-test.c | 85 ++++++ drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 266 ++++++++++++++++++ 3 files changed, 430 insertions(+) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/ar= m/arm-smmu-v3/arm-smmu-v3.h index 96a23ca633cb6..d7421b56e3598 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h @@ -649,6 +649,82 @@ struct arm_smmu_cmdq_batch { int num; }; =20 +enum arm_smmu_inv_type { + INV_TYPE_S1_ASID, + INV_TYPE_S2_VMID, + INV_TYPE_S2_VMID_S1_CLEAR, + INV_TYPE_ATS, + INV_TYPE_ATS_FULL, +}; + +struct arm_smmu_inv { + /* invalidation items */ + struct arm_smmu_device *smmu; + u8 type; + u8 size_opcode; + u8 nsize_opcode; + u32 id; /* ASID or VMID or SID */ + union { + size_t pgsize; /* ARM_SMMU_FEAT_RANGE_INV */ + u32 ssid; /* INV_TYPE_ATS */ + }; + + /* infrastructure items */ + refcount_t users; /* users=3D0 to mark as a trash */ + bool todel : 1; /* set for a pending deletion */ +}; + +/** + * struct arm_smmu_invs - Per-domain invalidation array + * @num_invs: number of invalidations in @invs + * @rwlock: Optional rwlock to fench arm_smmu_invs_dec() + * @rcu: rcu head for kfree_rcu() + * @inv: flexible invalidation array + * + * The arm_smmu_invs is an RCU data structure. During a ->attach_dev callb= ack, + * arm_smmu_invs_add() and arm_smmu_invs_del() will be used to allocate a = new + * copy of an old array for addition and deletion. + * + * The arm_smmu_invs_dec() is a special function to mutate a given array, = by + * internally reducing the users counts of some given entries. This exists= to + * support a no-fail routine like attaching to an IOMMU_DOMAIN_BLOCKED. Us= e it + * carefully as it can impact performance from the extra rwlock. + * + * Concurrent invalidation thread will push all the invalidations describe= d on + * the array into the command queue for each invalidation event. It is des= igned + * like this to optimize the invalidation fast path by avoiding locks. + * + * A domain can be shared across SMMU instances. When an instance gets rem= oved + * it would delete all the entries that belong to that SMMU instance. Then= , a + * synchronize_rcu() would have to be called to sync the array, to prevent= any + * concurrent invalidation thread accessing the old array from issuing com= mands + * to the command queue of a removed SMMU instance. + */ +struct arm_smmu_invs { + size_t num_invs; + rwlock_t rwlock; + struct rcu_head rcu; + struct arm_smmu_inv inv[]; +}; + +static inline struct arm_smmu_invs *arm_smmu_invs_alloc(size_t num_invs) +{ + struct arm_smmu_invs *new_invs; + + new_invs =3D kzalloc(struct_size(new_invs, inv, num_invs), GFP_KERNEL); + if (!new_invs) + return ERR_PTR(-ENOMEM); + rwlock_init(&new_invs->rwlock); + return new_invs; +} + +struct arm_smmu_invs *arm_smmu_invs_add(struct arm_smmu_invs *old_invs, + struct arm_smmu_invs *add_invs); +struct arm_smmu_invs *arm_smmu_invs_del(struct arm_smmu_invs *old_invs, + struct arm_smmu_invs *del_invs); +size_t arm_smmu_invs_dec(struct arm_smmu_invs *invs, + struct arm_smmu_invs *dec_invs); + struct arm_smmu_evtq { struct arm_smmu_queue q; struct iopf_queue *iopf; @@ -875,6 +951,8 @@ struct arm_smmu_domain { =20 struct iommu_domain domain; =20 + struct arm_smmu_invs *invs; + /* List of struct arm_smmu_master_domain */ struct list_head devices; spinlock_t devices_lock; @@ -956,6 +1034,7 @@ struct arm_smmu_domain *arm_smmu_domain_alloc(void); =20 static inline void arm_smmu_domain_free(struct arm_smmu_domain *smmu_domai= n) { + kfree_rcu(smmu_domain->invs, rcu); kfree(smmu_domain); } =20 diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-test.c b/drivers/iom= mu/arm/arm-smmu-v3/arm-smmu-v3-test.c index d2671bfd37981..2008a4b55ef70 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-test.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-test.c @@ -567,6 +567,90 @@ static void arm_smmu_v3_write_cd_test_sva_release(stru= ct kunit *test) NUM_EXPECTED_SYNCS(2)); } =20 +static void arm_smmu_v3_invs_test_verify(struct kunit *test, + struct arm_smmu_invs *invs, int num, + const int *ids, const int *users) +{ + KUNIT_EXPECT_EQ(test, invs->num_invs, num); + while (num--) { + KUNIT_EXPECT_EQ(test, invs->inv[num].id, ids[num]); + KUNIT_EXPECT_EQ(test, refcount_read(&invs->inv[num].users), + users[num]); + } +} + +static struct arm_smmu_invs invs1 =3D { + .num_invs =3D 3, + .inv =3D { { .type =3D INV_TYPE_S2_VMID, .id =3D 1, }, + { .type =3D INV_TYPE_S2_VMID, .id =3D 2, }, + { .type =3D INV_TYPE_S2_VMID, .id =3D 3, }, }, +}; + +static struct arm_smmu_invs invs2 =3D { + .num_invs =3D 3, + .inv =3D { { .type =3D INV_TYPE_S2_VMID, .id =3D 1, }, /* duplicated */ + { .type =3D INV_TYPE_ATS, .id =3D 5, }, + { .type =3D INV_TYPE_ATS, .id =3D 4, }, }, +}; + +static struct arm_smmu_invs invs3 =3D { + .num_invs =3D 3, + .inv =3D { { .type =3D INV_TYPE_S2_VMID, .id =3D 1, }, /* duplicated */ + { .type =3D INV_TYPE_ATS, .id =3D 7, }, + { .type =3D INV_TYPE_ATS, .id =3D 6, }, }, +}; + +static void arm_smmu_v3_invs_test(struct kunit *test) +{ + const int results1[2][3] =3D { { 1, 2, 3, }, { 1, 1, 1, }, }; + const int results2[2][5] =3D { { 1, 2, 3, 4, 5, }, { 2, 1, 1, 1, 1, }, }; + const int results3[2][5] =3D { { 1, 2, 3, 4, 5, }, { 1, 1, 1, 0, 0, }, }; + const int results4[2][5] =3D { { 1, 2, 3, 6, 7, }, { 2, 1, 1, 1, 1, }, }; + const int results5[2][5] =3D { { 1, 2, 3, 6, 7, }, { 1, 0, 0, 1, 1, }, }; + struct arm_smmu_invs *test_a, *test_b; + size_t num_invs; + + /* New array */ + test_a =3D arm_smmu_invs_alloc(0); + KUNIT_EXPECT_EQ(test, test_a->num_invs, 0); + + /* Test1: add invs1 (new array) */ + test_b =3D arm_smmu_invs_add(test_a, &invs1); + arm_smmu_v3_invs_test_verify(test, test_b, ARRAY_SIZE(results1[0]), + results1[0], results1[1]); + kfree(test_a); + + /* Test2: add invs2 (new array) */ + test_a =3D arm_smmu_invs_add(test_b, &invs2); + arm_smmu_v3_invs_test_verify(test, test_a, ARRAY_SIZE(results2[0]), + results2[0], results2[1]); + kfree(test_b); + + /* Test3: decrease invs2 (same array) */ + num_invs =3D arm_smmu_invs_dec(test_a, &invs2); + arm_smmu_v3_invs_test_verify(test, test_a, ARRAY_SIZE(results3[0]), + results3[0], results3[1]); + KUNIT_EXPECT_EQ(test, num_invs, 3); + + /* Test4: add invs3 (new array) */ + test_b =3D arm_smmu_invs_add(test_a, &invs3); + arm_smmu_v3_invs_test_verify(test, test_b, ARRAY_SIZE(results4[0]), + results4[0], results4[1]); + kfree(test_a); + + /* Test5: decrease invs1 (same array) */ + num_invs =3D arm_smmu_invs_dec(test_b, &invs1); + arm_smmu_v3_invs_test_verify(test, test_b, ARRAY_SIZE(results5[0]), + results5[0], results5[1]); + KUNIT_EXPECT_EQ(test, num_invs, 3); + + /* Test6: delete invs3 (new array) */ + test_a =3D arm_smmu_invs_del(test_b, &invs3); + KUNIT_EXPECT_EQ(test, test_a->num_invs, 0); + kfree(test_b); + kfree(test_a); +} + static struct kunit_case arm_smmu_v3_test_cases[] =3D { KUNIT_CASE(arm_smmu_v3_write_ste_test_bypass_to_abort), KUNIT_CASE(arm_smmu_v3_write_ste_test_abort_to_bypass), @@ -590,6 +674,7 @@ static struct kunit_case arm_smmu_v3_test_cases[] =3D { KUNIT_CASE(arm_smmu_v3_write_ste_test_s2_to_s1_stall), KUNIT_CASE(arm_smmu_v3_write_cd_test_sva_clear), KUNIT_CASE(arm_smmu_v3_write_cd_test_sva_release), + KUNIT_CASE(arm_smmu_v3_invs_test), {}, }; =20 diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/ar= m/arm-smmu-v3/arm-smmu-v3.c index 08af5f2d1235a..73f3b411ff7ef 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -26,6 +26,7 @@ #include #include #include +#include #include #include #include @@ -1033,6 +1034,263 @@ void arm_smmu_tlb_inv_asid(struct arm_smmu_device *= smmu, u16 asid) arm_smmu_cmdq_issue_cmd_with_sync(smmu, &cmd); } =20 +static int arm_smmu_invs_cmp(const void *_l, const void *_r) +{ + const struct arm_smmu_inv *l =3D _l; + const struct arm_smmu_inv *r =3D _r; + + if (l->smmu !=3D r->smmu) + return cmp_int((uintptr_t)l->smmu, (uintptr_t)r->smmu); + if (l->type !=3D r->type) + return cmp_int(l->type, r->type); + return cmp_int(l->id, r->id); +} + +static inline bool same_op(const struct arm_smmu_inv *a, + const struct arm_smmu_inv *b) +{ + return a->smmu =3D=3D b->smmu && a->type =3D=3D b->type && a->id =3D=3D b= ->id; +} + +/** + * arm_smmu_invs_add() - Combine @old_invs with @add_invs to a new array + * @old_invs: the old invalidation array + * @add_invs: an array of invlidations to add + * + * Return: a newly allocated and sorted invalidation array on success, or = an + * ERR_PTR. + * + * This function must be locked and serialized with arm_smmu_invs_del/dec(= ), + * but do not lockdep on any lock for KUNIT test. + * + * Caller is resposible for freeing the @old_invs and the returned one. + * + * Entries marked as trash can be resued if @add_invs wants to add them ba= ck. + * Otherwise, they will be completely removed in the returned array. + */ +VISIBLE_IF_KUNIT +struct arm_smmu_invs *arm_smmu_invs_add(struct arm_smmu_invs *old_invs, + struct arm_smmu_invs *add_invs) +{ + size_t need =3D old_invs->num_invs + add_invs->num_invs; + struct arm_smmu_invs *new_invs; + size_t deletes =3D 0, i, j; + u64 existed =3D 0; + + /* Max of add_invs->num_invs is 64 */ + if (WARN_ON(add_invs->num_invs > sizeof(existed) * 8)) + return ERR_PTR(-EINVAL); + + for (i =3D 0; i !=3D old_invs->num_invs; i++) { + struct arm_smmu_inv *cur =3D &old_invs->inv[i]; + /* Count the trash entries to deletes */ + if (cur->todel) { + WARN_ON_ONCE(refcount_read(&cur->users)); + deletes++; + } + for (j =3D 0; j !=3D add_invs->num_invs; j++) { + if (!same_op(cur, &add_invs->inv[j])) + continue; + /* Found duplicated entries in add_invs */ + if (WARN_ON_ONCE(existed & BIT_ULL(j))) + continue; + /* Revert the todel marker for reuse */ + if (cur->todel) { + cur->todel =3D false; + deletes--; + } + /* Store the new location of this existing op in id */ + add_invs->inv[j].id =3D i - deletes; + existed |=3D BIT_ULL(j); + need--; + break; + } + } + + need -=3D deletes; + + new_invs =3D arm_smmu_invs_alloc(need); + if (IS_ERR(new_invs)) { + /* Don't forget to revert all the todel markers */ + for (i =3D 0; i !=3D old_invs->num_invs; i++) { + if (refcount_read(&old_invs->inv[i].users) =3D=3D 0) + old_invs->inv[i].todel =3D true; + } + return new_invs; + } + + /* Copy the entire array less all the todel entries */ + for (i =3D 0; i !=3D old_invs->num_invs; i++) { + if (old_invs->inv[i].todel) + continue; + new_invs->inv[new_invs->num_invs++] =3D old_invs->inv[i]; + } + + for (j =3D 0; j !=3D add_invs->num_invs; j++) { + if (existed & BIT_ULL(j)) { + unsigned int idx =3D add_invs->inv[j].id; + + refcount_inc(&new_invs->inv[idx].users); + + /* Restore the id of the passed in add_invs->inv[j] */ + add_invs->inv[j].id =3D new_invs->inv[idx].id; + } else { + unsigned int idx =3D new_invs->num_invs; + + new_invs->inv[idx] =3D add_invs->inv[j]; + refcount_set(&new_invs->inv[idx].users, 1); + new_invs->num_invs++; + } + } + + WARN_ON(new_invs->num_invs !=3D need); + + /* + * A sorted array allows batching invalidations together for fewer SYNCs. + * Also, ATS must follow the ASID/VMID invalidation SYNC. + */ + sort_nonatomic(new_invs->inv, new_invs->num_invs, + sizeof(add_invs->inv[0]), arm_smmu_invs_cmp, NULL); + return new_invs; +} +EXPORT_SYMBOL_IF_KUNIT(arm_smmu_invs_add); + +/** + * arm_smmu_invs_del() - Remove @del_invs from @old_invs + * @old_invs: the old invalidation array + * @del_invs: an array of invlidations to delete + * + * Return: a newly allocated and sorted invalidation array on success, or = an + * ERR_PTR. + * + * This function must be locked and serialized with arm_smmu_invs_add/dec(= ), + * but do not lockdep on any lock for KUNIT test. + * + * Caller is resposible for freeing the @old_invs and the returned one. + * + * Entries marked as trash will be completely removed in the returned arra= y. + */ +VISIBLE_IF_KUNIT +struct arm_smmu_invs *arm_smmu_invs_del(struct arm_smmu_invs *old_invs, + struct arm_smmu_invs *del_invs) +{ + size_t need =3D old_invs->num_invs; + struct arm_smmu_invs *new_invs; + size_t i, j; + + if (WARN_ON(old_invs->num_invs < del_invs->num_invs)) + return ERR_PTR(-EINVAL); + + for (i =3D 0; i !=3D old_invs->num_invs; i++) { + struct arm_smmu_inv *cur =3D &old_invs->inv[i]; + /* Skip any trash entry */ + if (cur->todel) { + WARN_ON_ONCE(refcount_read(&cur->users)); + need--; + continue; + } + for (j =3D 0; j !=3D del_invs->num_invs; j++) { + if (!same_op(cur, &del_invs->inv[j])) + continue; + /* Found duplicated entries in del_invs */ + if (WARN_ON_ONCE(cur->todel)) + continue; + /* Mark todel. The deletion part will take care of it */ + cur->todel =3D true; + if (refcount_read(&cur->users) =3D=3D 1) + need--; + } + } + + new_invs =3D arm_smmu_invs_alloc(need); + if (IS_ERR(new_invs)) { + /* Don't forget to revert all the todel markers */ + for (i =3D 0; i !=3D old_invs->num_invs; i++) { + if (refcount_read(&old_invs->inv[i].users) !=3D 0) + old_invs->inv[i].todel =3D false; + } + return new_invs; + } + + for (i =3D 0; i !=3D old_invs->num_invs; i++) { + struct arm_smmu_inv *cur =3D &old_invs->inv[i]; + unsigned int idx =3D new_invs->num_invs; + + /* Either a trash entry or a matched entry for a dec-and-test */ + if (cur->todel) { + /* Can't do refcount_dec_and_test() on a trash entry */ + if (refcount_read(&cur->users) <=3D 1) + continue; + refcount_dec(&cur->users); + cur->todel =3D false; + } + new_invs->inv[idx] =3D *cur; + new_invs->num_invs++; + } + + WARN_ON(new_invs->num_invs !=3D need); + + /* Still sorted */ + return new_invs; +} +EXPORT_SYMBOL_IF_KUNIT(arm_smmu_invs_del); + +/** + * arm_smmu_invs_dec() - Find in @invs for all entries in @del_invs, decre= ase + * the user counts without deletions + * @invs: a given invalidation array + * @dec_invs: an array of invlidations to decrease their user counts + * + * Return: the actual number of invs in the array, excluding all trash ent= ries + * + * This function will not fail. Any entry with users=3D0 will be marked as= trash. + * All trash entries will remain in the @invs until being completely delet= ed by + * the next arm_smmu_invs_add() or arm_smmu_invs_del() function call. + * + * This function must be locked and serialized with arm_smmu_invs_add/del(= ), but + * do not lockdep on any lock for KUNIT test. + * + * Note that the @invs->num_invs will not be updated, even if the actual n= umber + * of invalidations are decreased. Readers should take the read lock to it= erate + * each entry and check its users counter until @inv->num_invs. + */ +VISIBLE_IF_KUNIT +size_t arm_smmu_invs_dec(struct arm_smmu_invs *invs, + struct arm_smmu_invs *dec_invs) +{ + size_t num_invs =3D 0, i, j; + unsigned long flags; + + /* Driver bug. Must fix rather, but do not fail here */ + if (WARN_ON(invs->num_invs < dec_invs->num_invs)) { + for (i =3D 0; i !=3D invs->num_invs; i++) { + if (!invs->inv[i].todel) + num_invs++; + } + return num_invs; + } + + /* We have no choice but to lock the array while editing it in place */ + write_lock_irqsave(&invs->rwlock, flags); + + for (i =3D 0; i !=3D invs->num_invs; i++) { + for (j =3D 0; j !=3D dec_invs->num_invs; j++) { + if (same_op(&invs->inv[i], &dec_invs->inv[j]) && + refcount_dec_and_test(&invs->inv[i].users)) { + /* Set the todel marker for deletion */ + invs->inv[i].todel =3D true; + break; + } + } + if (!invs->inv[i].todel) + num_invs++; + } + + write_unlock_irqrestore(&invs->rwlock, flags); + return num_invs; +} +EXPORT_SYMBOL_IF_KUNIT(arm_smmu_invs_dec); + /* * Based on the value of ent report which bits of the STE the HW will acce= ss. It * would be nice if this was complete according to the spec, but minimally= it @@ -2468,13 +2726,21 @@ static bool arm_smmu_enforce_cache_coherency(struct= iommu_domain *domain) struct arm_smmu_domain *arm_smmu_domain_alloc(void) { struct arm_smmu_domain *smmu_domain; + struct arm_smmu_invs *new_invs; =20 smmu_domain =3D kzalloc(sizeof(*smmu_domain), GFP_KERNEL); if (!smmu_domain) return ERR_PTR(-ENOMEM); =20 + new_invs =3D arm_smmu_invs_alloc(0); + if (IS_ERR(new_invs)) { + kfree(smmu_domain); + return ERR_CAST(new_invs); + } + INIT_LIST_HEAD(&smmu_domain->devices); spin_lock_init(&smmu_domain->devices_lock); + rcu_assign_pointer(smmu_domain->invs, new_invs); =20 return smmu_domain; } --=20 2.43.0 From nobody Sat Oct 4 17:29:51 2025 Received: from NAM12-MW2-obe.outbound.protection.outlook.com (mail-mw2nam12on2087.outbound.protection.outlook.com [40.107.244.87]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 683B71CD215 for ; Thu, 14 Aug 2025 01:26:27 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.244.87 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755134788; cv=fail; b=eWWX/IAxlEdHR+CCw+dMxLYhESi4tQNLUOZegHGnyRzsY37z5Y5kpeWELuZNhRo7egJ470C7520k/NG0Gy7j8feJTJdlreoHZ4wzCy374gXSaFk5PgdpOc56VB1VrrpzypsVQ+Ivhk0qDeljVFcOi0wteQAseMTwmcy9TzPznOA= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755134788; c=relaxed/simple; bh=nH0ZCGFMbLVKN5l3omEWdqjqkMt6NCla4oFmOoWZM0U=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=gBc0x2yqy18HevO/Gye1gYibE0cx2loPJCRqMNtluX+d2GyevnJYgXh5QGewS5tKXaCkXJMNl+pAhY9Caj88zjX85oepSspHC43V96XTR9s7F658beULv0qJQVp/CEDKGN02Jz/Q4i2us+HyI3JKwr9Vl6u0IFCJnsZJGe8sB8Y= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=lci6uVBP; arc=fail smtp.client-ip=40.107.244.87 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="lci6uVBP" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=MHEk/wG7uWO48VY0AYZPEkU9RA687s5jMM6vMzrckqbjdaDMxSYQkVNgPEq29srpdAczQvftzCWdeemIkCUrlLIebwl8BdNxcVaKF9EeXL7gOEiiJj6QtwQM9wBvUl29XbUsIKFU62KnB2wOqIF54J5pYpQIDdi/Wr54FIw6pGaQa9ousn/hvowlmJJPVqvJs5ZRjRMxXWssXV+Q3iU08I/RGVqNwqEAVspQnm8TzdZ/heUJvTdFpeB5v8os4U/stHWAQxR34j+EKj9LES4O0ygwUbe0lhhA74C83Y0yv9o+n5QUhMJC6rxLmlt+ntAd8Ykh92Cbgb2bJPJJt1vhPw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=W4MrySLSyWxepi/Xp8MMnwcDI2CQDt0kK96IuuLqhZg=; b=dcXrBHQqPAxJiMDUmCfVwhSsg63qKNMVj5c+HkW+heiAU7eVMN7P9L8tpdk9vHudcq+fCm9QGOjX8aCZ6z/HslGmTv5U0X3XN4gda23jP3yEkyavyz17H8UjeNdBxaFS+Oo+sHrW/M0BjsYLoW0uDETZP+l8pEuIFkMCyJylwdla8IcNhx/uQKW4LUnLHwU/gW8kn5VqYw3OISX4CAJ+B11ZjZR3VhBdTqS2tGXzrwkrPT2iZECuxxDXW3ZWtK0u5ZG/RDAzbCXGBgmBG5k9h6VTg5MlAqzPJLZkwznM1XsWqJcS0pGgNO1+4FW2KYN/h/1TW5aDABie3WGjJI55wQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.160) smtp.rcpttodomain=kernel.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=W4MrySLSyWxepi/Xp8MMnwcDI2CQDt0kK96IuuLqhZg=; b=lci6uVBPT2MpgoaDG2bv090zNAJWh99rjT0mgfSUq/gZ1El+8C/UFdL99JVc/O+uJvjLGRoWa0m0iL+XxMqd1xNa0BpJ6H+Ip6LkdNe8ioO32nxGTZygBuzOHSekPF7G93gRMvozTUik9mKOTkNZuWPCdMFObjlxqTL8xYinZI4UPZA4xsAT5oTeLlAQgBfi/XdQ8fiRup6OsEeORf0wc0UT27FAJ+094i+p9YnC4s0qaYD91gQiimp+XwEvii+aB9/x0/Wc/FYcmdHJlThE6jkTsedWa7pGdxCkxP4SfB295pVDl+Tsiwjl+kM8lrj8r8Q4OLRu+m5F7UKCZAzcGg== Received: from BN9PR03CA0653.namprd03.prod.outlook.com (2603:10b6:408:13b::28) by PH7PR12MB8794.namprd12.prod.outlook.com (2603:10b6:510:27d::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9031.15; Thu, 14 Aug 2025 01:26:23 +0000 Received: from BN2PEPF000044A2.namprd02.prod.outlook.com (2603:10b6:408:13b:cafe::6f) by BN9PR03CA0653.outlook.office365.com (2603:10b6:408:13b::28) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.9031.16 via Frontend Transport; Thu, 14 Aug 2025 01:26:22 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.160) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.160 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.160; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.160) by BN2PEPF000044A2.mail.protection.outlook.com (10.167.243.153) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9031.11 via Frontend Transport; Thu, 14 Aug 2025 01:26:22 +0000 Received: from rnnvmail205.nvidia.com (10.129.68.10) by mail.nvidia.com (10.129.200.66) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.14; Wed, 13 Aug 2025 18:26:08 -0700 Received: from rnnvmail205.nvidia.com (10.129.68.10) by rnnvmail205.nvidia.com (10.129.68.10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.14; Wed, 13 Aug 2025 18:26:07 -0700 Received: from Asurada-Nvidia.nvidia.com (10.127.8.9) by mail.nvidia.com (10.129.68.10) with Microsoft SMTP Server id 15.2.1544.14 via Frontend Transport; Wed, 13 Aug 2025 18:26:06 -0700 From: Nicolin Chen To: , , CC: , , , , , , , , , , , , Subject: [PATCH rfcv1 5/8] iommu/arm-smmu-v3: Pre-allocate a per-master invalidation array Date: Wed, 13 Aug 2025 18:25:36 -0700 Message-ID: <2b55869f698340d5791210b9a7e542f81ad20bcb.1755131672.git.nicolinc@nvidia.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BN2PEPF000044A2:EE_|PH7PR12MB8794:EE_ X-MS-Office365-Filtering-Correlation-Id: df1fcaaa-59fa-4ec1-b041-08dddad19439 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|7416014|376014|1800799024|36860700013|82310400026; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?D7mFmEh4UERTsC9hSWeyjy7JNljFs/+u0wmazcfk2NzMVTjLmDM+zZWj+9Ma?= =?us-ascii?Q?NGe38KP3iCgMgWNRlanVI+8xwlwdTxbaCtQDRqpGKgGkZnzGAozK8M8uJqN0?= =?us-ascii?Q?MsY4PIEHFev3ub11VdZ2OJoX0m0jGHsBWhRq+NSi7wRxj+VTkTbRteUk3cwk?= =?us-ascii?Q?wsbZT/z3BTWVIRdAwex14JBUlLlK88+mBnGIHoQHW8iDfAWAX3mzem08LUt6?= =?us-ascii?Q?euv5VbOoCfdairFTtNjexnx4ts6bTTNL6pz2uQ9ZMWlvlKe5+1dnxxSiQGIv?= =?us-ascii?Q?hczBVWH+08Nj1j0RT3W3ACaLfWj8Hkj9Y1I/5oYE3QbnQekavYKxxt5ORf8Z?= =?us-ascii?Q?SW1bZbVfIbuMqFjfgqOTKmfzuRIf6rFBFipGA2Iui5NRmJiUIjD6EGwqLZcj?= =?us-ascii?Q?t++zvlYK/kLEMDt7OgxPeZRSxOeBgVHP5aeOeMZ0927iVLHv1U98TiKBMu7B?= =?us-ascii?Q?ANxTrw0kv8tViNVuiXpyCHcwHrRGCJy/80FeurM06DSLPJ6zzQeN7idOG4hl?= =?us-ascii?Q?bgK3RSULl/r8onFN5/pdzNXUq0B8HBtR5eQJfGnidVaTmjzYHbK8NGfageg7?= =?us-ascii?Q?SEFP1J10GJs/JLMAW6SKE5QUqI5HaT0XvZiNy/xEY7+l8toc0ovgJrVQBAgI?= =?us-ascii?Q?tT2mCtkMy5Rcop1vZnngl/eZnsUtFesfeWCnIjyCvWpPSPRut/LekIF3JfrB?= =?us-ascii?Q?6LySO7mQED0V+HuySpQ1QGmpPKASiaZbAjwrHnV3Nxa4DpkIPNvBD2HBpJ0D?= =?us-ascii?Q?+HQUL//eh5fWxbWTiM2+lY9XeHXbyWdok12XzHLoMpDlSzVloIwtAeHRj0am?= =?us-ascii?Q?nMEF1vw1ERMWslw6oqA1ucnE8pgk7cPKeKn0JDY4lNbZFUjCUMGpzz9ss1Go?= =?us-ascii?Q?r5/fsluVymSVW+JelyfNwB8z2mzXR9Sa8QN85wzwTxEfZxSFxHfdw3UAz85l?= =?us-ascii?Q?1n0TpO1YFf9mOJvI2H+3Bf5BjucAGL72Q4ZFjkbRsaK6xzp2ATG9jmaEiWG0?= =?us-ascii?Q?RK0JyUET4lKsLG2s/WonG65J6vy5l8apBdDugS8L1S4EJUSZOCVIx3xoLWVz?= =?us-ascii?Q?1Y2kAVs3PPMuZ7u50NpOWgxJ6pJWde1RGqn6L/bqz5WIDU2R74ZfxPsBcLAz?= =?us-ascii?Q?T2h1p533zzMqzQD1Mzc4WmnSiRTCBFEoLqMmhla3NvWB43P64Dk4D7vHlrAr?= =?us-ascii?Q?J91QUcG23pft2R20H7Hs0vr12Q+Ubm5Ir6tgxt9lJhj0NsIUNyTVPvzmXOLA?= =?us-ascii?Q?O6u6P7a5i623QYZTj+FCpyRrth3VWn+48yr2VtPb5jDD/lx0zOEGJpJPKlkr?= =?us-ascii?Q?VZTbiYtaaTlg4IZbV4Tr7EqUXUhx5u2si9iMf7M1YU0Xsb8OAikwvxk/yXI2?= =?us-ascii?Q?iVcadlxqxl3/cusWgIRKdduzZUKgpaQwig5CyLvXABOx1YXQxb7gzrxHsukH?= =?us-ascii?Q?2JkSjE9WsP02A6s3DP0r9dkRjei4W0Wsf29vIJW6dHi2ogCPg+EiuVduOtmu?= =?us-ascii?Q?b9cwI0X0y2FgP4eMQgCtJJMLWXP8aQCu9DGw?= X-Forefront-Antispam-Report: CIP:216.228.117.160;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge1.nvidia.com;CAT:NONE;SFS:(13230040)(7416014)(376014)(1800799024)(36860700013)(82310400026);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Aug 2025 01:26:22.4411 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: df1fcaaa-59fa-4ec1-b041-08dddad19439 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.160];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BN2PEPF000044A2.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR12MB8794 Content-Type: text/plain; charset="utf-8" When a master is attached from an old domain to a new domain, it needs to build an invalidation array to delete and add the array entries from/onto the invalidation arrays of those two domains, passed via the del_invs and add_invs arguments in to arm_smmu_invs_del/add() respectively. Since the master->num_streams might differ across masters, a memory would have to be allocated when building an add_invs/del_invs array which might fail with -ENOMEM. On the other hand, an attachment to arm_smmu_blocked_domain must not fail so it's the best to avoid any memory allocation in that path. Pre-allocate a fixed size invalidation array for every master. This array will be filled dynamically when building an add_invs or del_invs array to attach or detach an smmu_domain. Signed-off-by: Nicolin Chen --- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 1 + drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 10 ++++++++++ 2 files changed, 11 insertions(+) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/ar= m/arm-smmu-v3/arm-smmu-v3.h index d7421b56e3598..0330444bef45f 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h @@ -919,6 +919,7 @@ struct arm_smmu_master { struct arm_smmu_device *smmu; struct device *dev; struct arm_smmu_stream *streams; + struct arm_smmu_invs *invs; struct arm_smmu_vmaster *vmaster; /* use smmu->streams_mutex */ /* Locked by the iommu core using the group mutex */ struct arm_smmu_ctx_desc_cfg cd_table; diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/ar= m/arm-smmu-v3/arm-smmu-v3.c index 73f3b411ff7ef..fb5429d8ebb29 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -3723,6 +3723,7 @@ static int arm_smmu_insert_master(struct arm_smmu_dev= ice *smmu, int i; int ret =3D 0; struct iommu_fwspec *fwspec =3D dev_iommu_fwspec_get(master->dev); + size_t num_ats =3D dev_is_pci(master->dev) ? master->num_streams : 0; =20 master->streams =3D kcalloc(fwspec->num_ids, sizeof(*master->streams), GFP_KERNEL); @@ -3730,6 +3731,13 @@ static int arm_smmu_insert_master(struct arm_smmu_de= vice *smmu, return -ENOMEM; master->num_streams =3D fwspec->num_ids; =20 + /* Max possible num_invs: two for ASID/VMIDs and num_ats for ATC_INVs */ + master->invs =3D arm_smmu_invs_alloc(2 + num_ats); + if (IS_ERR(master->invs)) { + kfree(master->streams); + return PTR_ERR(master->invs); + } + mutex_lock(&smmu->streams_mutex); for (i =3D 0; i < fwspec->num_ids; i++) { struct arm_smmu_stream *new_stream =3D &master->streams[i]; @@ -3767,6 +3775,7 @@ static int arm_smmu_insert_master(struct arm_smmu_dev= ice *smmu, for (i--; i >=3D 0; i--) rb_erase(&master->streams[i].node, &smmu->streams); kfree(master->streams); + kfree(master->invs); } mutex_unlock(&smmu->streams_mutex); =20 @@ -3788,6 +3797,7 @@ static void arm_smmu_remove_master(struct arm_smmu_ma= ster *master) mutex_unlock(&smmu->streams_mutex); =20 kfree(master->streams); + kfree(master->invs); } =20 static struct iommu_device *arm_smmu_probe_device(struct device *dev) --=20 2.43.0 From nobody Sat Oct 4 17:29:51 2025 Received: from NAM12-BN8-obe.outbound.protection.outlook.com (mail-bn8nam12on2059.outbound.protection.outlook.com [40.107.237.59]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 543B41DED64 for ; Thu, 14 Aug 2025 01:26:31 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.237.59 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755134793; cv=fail; b=ac+pKK4d1w9cnthDKPeeTfyAN+/U395Bm+uewmz+9qbxth3zGLSRGH/gpMYg2nn3vfBv6rbhD8FFXHY1J8A7fEQIDNDuNcsooN3Peggf5eDv+KDH03rUyYbrn7fcuMoWNUBdkJVBrcL0GrLXtAVwHrsH8k4yBq69CVh64+MHQng= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755134793; c=relaxed/simple; bh=n68IzJHPPZYsHEpPUoc7Tfm+zcw9E3wWbzLtQVXZf04=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=kzoafPVXUn1dtuWZZVjQZSebnzM3+vwFW0I49qEw0FwHqUhwptJKY1EL0R6aZ+7h0EIA0MJnxENLizh+QvtV1EuKHINkYJdAtxMl8ThGbIrKvcsLEyI2QW9TDjJU3rOWTi+62kuZfc3ib5p93Z35dDsVQUWINAJmg9tdj1SiKBQ= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=eWPHOuW+; arc=fail smtp.client-ip=40.107.237.59 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="eWPHOuW+" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=RYLmVkADWcr0FczSDQD9QtyW/HJKstZ5NbfCJLAH5x9SAMIabm3Zl6JGte4h7stQf1kCZzMpsY29ZL0Q54JoM5ZWAFRXYjUhZP42L4MBvXjeKHdGpRW6qV/kak0RnmcoIUUlLE3P7bzj7vL2jzr3TkVFOj6Lq5pNyre8Lz4WmGUUV7MbRFCLUqn5GnY5gGKExj9R4NLc00KS2Qq2jmiBnQERrWR8Tp5soluoLBgpDyhEgbu++x1p7EBzKWGXbee38+J8jxhf6JJ94WnK4PwgQaUEa7f9909mBBK3hVJO4ov2x8IXaR7LgQHW19ByUpzwyMKf6pf+pDZo8nUrg9O3vA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Q58sfs3tok8ob+rxiYVQH6vmTOamD0GfCbmlcVi3DK8=; b=ABdEMrE+btHaQgTBU/4mqI+XkewJkJa6fG0LU3Nezl86Ixmxpf60qWO2O5u3kJ6bNjHlw1ogZzXm4KZrL6xbeEc82DY9kpQ/7GT40jTHOdCLD+079Wz2O9spRJ+zgolpA5LiQ0QxjD3Tllcn0ioOIuazTXJldMehwGEn4xNumL3xJdJWZjbMCAU130mEVY4pi+Sxo6MKbmDlxUw6iUj/5HMGIVc9upBI6/kPrjhB59za0YtrOarSyGGwXoFc5b+JFZM2uGupIs/d2BZ4aXJCuiMprQTCH4NQAPljWeyXEJhYg7kVW4OkbhcfsQRDqic87YJ+AUrNTgLd2nSkqgnBbQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.161) smtp.rcpttodomain=kernel.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Q58sfs3tok8ob+rxiYVQH6vmTOamD0GfCbmlcVi3DK8=; b=eWPHOuW+rYk4qBNF7AJXppnkbk8201MYzNbQ+IKLDljFbM0aiikgXARPTv3PFx3E/yOE4OI6yvnqB7H/TYDn4U4eNVddRJz1a6pVrD54Ou7+uhyVk4mdscxw7faYlG7aRVm57QCS815q+QpKi+qMPRKAR7LOug99tHl96SLuuQPFr55f+wXIDaMpUP9T/0slDXdMnkOP5H7I5fIDVt/isTOmdg3xCtw/kLrC16SkNOmLYQpKdw27Ir6yMKj8gUyQQuHYuOydIHRPx9gMcz7sXh6Rr/KaHgjBTF0jo9qwbp4PfYE4gs53eXqKOooZUkwhqtB768ZMp20P18Eq7ORI7g== Received: from BYAPR08CA0059.namprd08.prod.outlook.com (2603:10b6:a03:117::36) by IA0PR12MB8279.namprd12.prod.outlook.com (2603:10b6:208:40c::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9031.15; Thu, 14 Aug 2025 01:26:25 +0000 Received: from SJ5PEPF000001F4.namprd05.prod.outlook.com (2603:10b6:a03:117:cafe::46) by BYAPR08CA0059.outlook.office365.com (2603:10b6:a03:117::36) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.9031.16 via Frontend Transport; Thu, 14 Aug 2025 01:26:25 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.161) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.161 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.161; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.161) by SJ5PEPF000001F4.mail.protection.outlook.com (10.167.242.72) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9031.11 via Frontend Transport; Thu, 14 Aug 2025 01:26:25 +0000 Received: from rnnvmail205.nvidia.com (10.129.68.10) by mail.nvidia.com (10.129.200.67) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.14; Wed, 13 Aug 2025 18:26:09 -0700 Received: from rnnvmail205.nvidia.com (10.129.68.10) by rnnvmail205.nvidia.com (10.129.68.10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.14; Wed, 13 Aug 2025 18:26:09 -0700 Received: from Asurada-Nvidia.nvidia.com (10.127.8.9) by mail.nvidia.com (10.129.68.10) with Microsoft SMTP Server id 15.2.1544.14 via Frontend Transport; Wed, 13 Aug 2025 18:26:08 -0700 From: Nicolin Chen To: , , CC: , , , , , , , , , , , , Subject: [PATCH rfcv1 6/8] iommu/arm-smmu-v3: Populate smmu_domain->invs when attaching masters Date: Wed, 13 Aug 2025 18:25:37 -0700 Message-ID: X-Mailer: git-send-email 2.43.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SJ5PEPF000001F4:EE_|IA0PR12MB8279:EE_ X-MS-Office365-Filtering-Correlation-Id: 2f22efe3-a07c-4905-a58a-08dddad195ba X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|7416014|1800799024|36860700013|82310400026; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?+/XJJajsT3lm50DMZ2ic/JIdAjwxpLuICBmFbZbnRypplXY4XKZQyQULJNvx?= =?us-ascii?Q?5k8SSfhMtbOJiaip93anLF98FqWXQypTozDiLO/OoxoJiCqoj43dkUsMB2PR?= =?us-ascii?Q?nNBnwI6PKGwFFBeNN56jRcTm5sb8sriYKcSREfyV8F9+U/vzBVcz7XEf2adg?= =?us-ascii?Q?VGLl18kUtdeRbX9svvx1IG5kCikybY1iAmWVGhX0Tgj2DlJ2aettZgG3y3Ii?= =?us-ascii?Q?bEDvmdzVdANJH+DSNoQNb1ymck5AnOwlyci6/MaD0Lo3//IAIdPVyKh1dzQ2?= =?us-ascii?Q?Jhua2ep5NhQ/KpTV/QjOvrqho4ohkLruOsrZsGZVmT6zE8/75jxdySGkiEVm?= =?us-ascii?Q?jPlnyLozfG7mUdQqEOjxGoijIjzug6IT0ZNPczma/gX3B7Fc0zCP7/nEz73C?= =?us-ascii?Q?999yDvJTta3AbBkCw6UbzilioNLSqS9byrWnf0e+FKYuoZ2RLoQ7eHZYYmwZ?= =?us-ascii?Q?NrEoi623UAvRK0fhhjZKrM6TVq2t/lK9VngG6ErbweuvKtENSnmNNABIZjpo?= =?us-ascii?Q?CxW7Z7libBfRn3P/tL2Ix0jDKtdacy+x+PJj6WAYwYqFWLM8GZG19md17Gi7?= =?us-ascii?Q?zqJNxYdQ6w3rAyQUiMKVbvreNr4+pY4wkBrs+Pe1r2Pu1K54Vg0X7kyk/UO7?= =?us-ascii?Q?USAEE2/DHhmJ0SRXJvLFuUqe+UScb0BtkFJVdHe6HfzAJ36DrFG4WdGzyCUN?= =?us-ascii?Q?mi0tw2bPE7+wvRDVNn3bclZJ0sXjvBZT8wplGw91jE6e6oZ4hjjJQpHHHXt1?= =?us-ascii?Q?OR8YdHwdZEfd7HVtei+b/nqPudniX0dmRSVadehXfl051d14NhsVH79sDqQ8?= =?us-ascii?Q?SvQbLvt+TNw/4XjCI1AfI8BIcm84hAvgldTT9cC4nh2FvMdtWjzdbyz74rSO?= =?us-ascii?Q?O5Wcve3+kMBQq+2WjgV3YTEE1bA6ZGhtyNB5PXieezZS+fC9TWPGhBHBUv1A?= =?us-ascii?Q?RVDLUP6ciKztdpjNEULpLDytJ2co/Na89//Dw0/wfx4xuZ7D5pLPDJOGs/Hu?= =?us-ascii?Q?kg7O6FAtO2lwa85asJnhfbtW9sqZQkvu342l18mUYIbkPecCcR3Rtz7ou9Hy?= =?us-ascii?Q?kDtLHSZ6BFNAsjJzWZFm4oRwlndQyFzUA7MuaNU6Ir35XpL0jwpFSZ+yrPJD?= =?us-ascii?Q?Q+0zvP9TVYsccyJlvdEEjxBrtWiRt7kpcUJCKgxky/smjfXP9Kmj42SSdyDe?= =?us-ascii?Q?vE//E1c+DaJUgY5MQftbvoxtap/U7zpYkmnGNaZHF7SkQQtQD9NigArMi00v?= =?us-ascii?Q?Wd6yavN86K2RpWICkWd3yfuYJAeOQHCekR2K0w3cLv1Lx3EEIrD2u1KALUmM?= =?us-ascii?Q?9seTJI3cnHf8J5CcA6f79IRJwAmCyeIfcaDREFaGg9Ednjl+7lF6E3PXs8jS?= =?us-ascii?Q?8ElMP0AthLxbNokQpKTNdVYDz7+pN9tET5rW4tRQmqzX5QsQTQycQGhtEakf?= =?us-ascii?Q?SI74OL9zxysT4HpSz1it85Cbh4neYyDnsdELBNDe9ZdYkXjvwSNsxs/QXcRS?= =?us-ascii?Q?vhe+4PPDUca2/9WHoQi3mAtXDIgNWZBd20Lw?= X-Forefront-Antispam-Report: CIP:216.228.117.161;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge2.nvidia.com;CAT:NONE;SFS:(13230040)(376014)(7416014)(1800799024)(36860700013)(82310400026);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Aug 2025 01:26:25.0829 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 2f22efe3-a07c-4905-a58a-08dddad195ba X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.161];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: SJ5PEPF000001F4.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA0PR12MB8279 Content-Type: text/plain; charset="utf-8" Update the invs array with the invalidations required by each domain type during attachment operations. Only an SVA domain or a paging domain will have an invs array: a. SVA domain will add an INV_TYPE_S1_ASID per SMMU and an INV_TYPE_ATS per SID b. Non-nesting-parent paging domain with no ATS-enabled master will add a single INV_TYPE_S1_ASID or INV_TYPE_S2_VMID per SMMU c. Non-nesting-parent paging domain with ATS-enabled master(s) will do (b) and add an INV_TYPE_ATS per SID d. Nesting-parent paging domain will add an INV_TYPE_S2_VMID followed by an INV_TYPE_S2_VMID_S1_CLEAR per vSMMU. For an ATS-enabled master, it will add an INV_TYPE_ATS_FULL per SID The per-domain invalidation is not needed, until the domain is attached to a master, i.e. a possible translation request. Giving this clears a way to allowing the domain to be attached to many SMMUs, and avoids any pointless invalidation overheads during a teardown if there are no STE/CDs referring to the domain. This also means, when the last device is detached, the old domain must flush its ASID or VMID because any iommu_unmap() call after it wouldn't initiate any invalidation given an empty domain invs array. Co-developed-by: Jason Gunthorpe Signed-off-by: Jason Gunthorpe Signed-off-by: Nicolin Chen --- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 6 + drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 220 +++++++++++++++++++- 2 files changed, 225 insertions(+), 1 deletion(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/ar= m/arm-smmu-v3/arm-smmu-v3.h index 0330444bef45f..715179249eced 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h @@ -1084,6 +1084,12 @@ struct arm_smmu_attach_state { ioasid_t ssid; /* Resulting state */ struct arm_smmu_vmaster *vmaster; + struct arm_smmu_invs **old_domain_invs; + struct arm_smmu_invs *old_domain_oinvs; + struct arm_smmu_invs *old_domain_ninvs; + struct arm_smmu_invs **new_domain_invs; + struct arm_smmu_invs *new_domain_oinvs; + struct arm_smmu_invs *new_domain_ninvs; bool ats_enabled; }; =20 diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/ar= m/arm-smmu-v3/arm-smmu-v3.c index fb5429d8ebb29..95615525b0ab8 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -3082,6 +3082,76 @@ static void arm_smmu_disable_iopf(struct arm_smmu_ma= ster *master, iopf_queue_remove_device(master->smmu->evtq.iopf, master->dev); } =20 +typedef struct arm_smmu_invs *(*invs_fn)(struct arm_smmu_invs *old_invs, + struct arm_smmu_invs *invs); + +static struct arm_smmu_invs *arm_smmu_build_invs( + struct arm_smmu_invs *old_invs, struct arm_smmu_domain *smmu_domain, + struct arm_smmu_master *master, bool ats, ioasid_t ssid, invs_fn fn) +{ + const bool e2h =3D master->smmu->features & ARM_SMMU_FEAT_E2H; + const bool nesting =3D smmu_domain->nest_parent; + struct arm_smmu_inv *cur =3D master->invs->inv; + size_t num_invs =3D 1; + size_t i; + + switch (smmu_domain->stage) { + case ARM_SMMU_DOMAIN_SVA: + case ARM_SMMU_DOMAIN_S1: + cur->smmu =3D master->smmu; + cur->type =3D INV_TYPE_S1_ASID; + cur->id =3D smmu_domain->cd.asid; + cur->size_opcode =3D e2h ? CMDQ_OP_TLBI_EL2_VA : + CMDQ_OP_TLBI_NH_VA; + cur->nsize_opcode =3D e2h ? CMDQ_OP_TLBI_EL2_ASID : + CMDQ_OP_TLBI_NH_ASID; + break; + case ARM_SMMU_DOMAIN_S2: + cur->smmu =3D master->smmu; + cur->type =3D INV_TYPE_S2_VMID; + cur->id =3D smmu_domain->s2_cfg.vmid; + cur->size_opcode =3D CMDQ_OP_TLBI_S2_IPA; + cur->nsize_opcode =3D CMDQ_OP_TLBI_S12_VMALL; + break; + default: + WARN_ON(true); + return old_invs; + } + + /* Range-based invalidation requires the leaf pgsize for calculation */ + if (master->smmu->features & ARM_SMMU_FEAT_RANGE_INV) + cur->pgsize =3D __ffs(smmu_domain->domain.pgsize_bitmap); + + /* All the nested S1 ASIDs have to be flushed when S2 parent changes */ + if (nesting) { + cur =3D &master->invs->inv[num_invs++]; + cur->smmu =3D master->smmu; + cur->type =3D INV_TYPE_S2_VMID_S1_CLEAR; + cur->id =3D smmu_domain->s2_cfg.vmid; + cur->size_opcode =3D CMDQ_OP_TLBI_NH_ALL; + cur->nsize_opcode =3D CMDQ_OP_TLBI_NH_ALL; + } + + if (ats) { + for (i =3D 0, cur++; i < master->num_streams; i++) { + cur->smmu =3D master->smmu; + /* + * If an S2 used as a nesting parent is changed we have + * no option but to completely flush the ATC. + */ + cur->type =3D nesting ? INV_TYPE_ATS_FULL : INV_TYPE_ATS; + cur->id =3D master->streams[i].id; + cur->ssid =3D ssid; + cur->size_opcode =3D CMDQ_OP_ATC_INV; + cur->nsize_opcode =3D CMDQ_OP_ATC_INV; + } + num_invs +=3D master->num_streams; + } + + master->invs->num_invs =3D num_invs; + return fn(old_invs, master->invs); +} + static void arm_smmu_remove_master_domain(struct arm_smmu_master *master, struct iommu_domain *domain, ioasid_t ssid) @@ -3111,6 +3181,144 @@ static void arm_smmu_remove_master_domain(struct ar= m_smmu_master *master, kfree(master_domain); } =20 +static int arm_smmu_attach_prepare_invs(struct arm_smmu_attach_state *stat= e, + struct arm_smmu_domain *new_smmu_domain) +{ + struct arm_smmu_domain *old_smmu_domain =3D + to_smmu_domain_devices(state->old_domain); + struct arm_smmu_master *master =3D state->master; + bool blocking =3D false; + + /* A re-attach case doesn't need to update invs array */ + if (new_smmu_domain =3D=3D old_smmu_domain) + return 0; + + if (new_smmu_domain) { + state->new_domain_oinvs =3D rcu_dereference_protected( + new_smmu_domain->invs, + lockdep_is_held(&arm_smmu_asid_lock)); + state->new_domain_ninvs =3D arm_smmu_build_invs( + state->new_domain_oinvs, new_smmu_domain, master, + state->ats_enabled, state->ssid, arm_smmu_invs_add); + if (IS_ERR(state->new_domain_ninvs)) + return PTR_ERR(state->new_domain_ninvs); + state->new_domain_invs =3D &new_smmu_domain->invs; + blocking =3D new_smmu_domain->domain.type =3D=3D IOMMU_DOMAIN_BLOCKED; + } + + if (old_smmu_domain) { + state->old_domain_oinvs =3D rcu_dereference_protected( + old_smmu_domain->invs, + lockdep_is_held(&arm_smmu_asid_lock)); + state->old_domain_ninvs =3D arm_smmu_build_invs( + state->old_domain_oinvs, old_smmu_domain, master, + master->ats_enabled, state->ssid, arm_smmu_invs_del); + if (IS_ERR(state->old_domain_ninvs)) { + /* An attachment to the blocked_domain must not fail */ + if (blocking) { + state->old_domain_ninvs =3D NULL; + } else { + kfree(state->new_domain_ninvs); + return PTR_ERR(state->old_domain_ninvs); + } + } + state->old_domain_invs =3D &old_smmu_domain->invs; + /* master->invs is retaining the del_invs for the old domain */ + } + + return 0; +} + +/* Must be installed before arm_smmu_install_ste_for_dev() */ +static void +arm_smmu_install_new_domain_invs(struct arm_smmu_attach_state *state) +{ + if (!state->new_domain_invs) + return; + + rcu_assign_pointer(*state->new_domain_invs, state->new_domain_ninvs); + /* + * Committed to updating the STE, using the new invalidation array, and + * acquiring any racing IOPTE updates. + */ + smp_mb(); + kfree_rcu(state->new_domain_oinvs, rcu); +} + +/* Should be installed after arm_smmu_install_ste_for_dev() */ +static void +arm_smmu_install_old_domain_invs(struct arm_smmu_attach_state *state) +{ + struct arm_smmu_invs *old_domain_oinvs =3D state->old_domain_oinvs; + struct arm_smmu_invs *old_domain_ninvs =3D state->old_domain_ninvs; + struct arm_smmu_master *master =3D state->master; + unsigned long flags; + size_t num_invs; + + if (!state->old_domain_invs) + return; + + /* Activate the no-fail protocol upon an allocation failure */ + if (!old_domain_ninvs) { + /* + * Notes: + * - The array will be edited in place while holding its rwlock + * which has a tradeoff that any concurrent invalidation will + * fail at read_trylock() until arm_smmu_invs_dec() returns. + * - arm_smmu_invs_dec() doesn't update the array's num_invs as + * if only decrease users counters. So, get num_invs from the + * returned value. + * - The master->invs retains the del_invs for the old domain. + */ + num_invs =3D arm_smmu_invs_dec(old_domain_oinvs, master->invs); + } else { + rcu_assign_pointer(*state->old_domain_invs, old_domain_ninvs); + /* + * Fake an empty old array that a concurrent invalidation thread + * races at. It either lets the reader quickly respin for a new + * array with fewer num_invs (avoiding deleted invalidations) or + * blocks the writer till the reader flushes the array (avoiding + * ATC invalidation timeouts for ATS invalidations being sent to + * a resetting PCI device). + */ + write_lock_irqsave(&old_domain_oinvs->rwlock, flags); + old_domain_oinvs->num_invs =3D 0; + write_unlock_irqrestore(&old_domain_oinvs->rwlock, flags); + + kfree_rcu(old_domain_oinvs, rcu); + num_invs =3D state->old_domain_ninvs->num_invs; + } + + /* + * The domain invs array was filled when the first device attaches to it + * and emptied when the last device detaches. So, the invs array doesn't + * syncrhonize with iommu_unmap() calls, which might come after the last + * detach and end up with a NOP. This would result in missing a critical + * TLB maintanance. Thus, when the last device is detached (indicated by + * an empty invs array), flush all TLBs using the removed ASID or VMID. + */ + if (!num_invs) { + struct arm_smmu_inv *inv =3D &master->invs->inv[0]; + struct arm_smmu_cmdq_ent cmd =3D { + .opcode =3D inv->nsize_opcode, + }; + + switch (inv->type) { + case INV_TYPE_S1_ASID: + cmd.tlbi.asid =3D inv->id; + arm_smmu_cmdq_issue_cmd_with_sync(inv->smmu, &cmd); + break; + case INV_TYPE_S2_VMID: + cmd.tlbi.vmid =3D inv->id; + arm_smmu_cmdq_issue_cmd_with_sync(inv->smmu, &cmd); + break; + default: + WARN_ON(true); + break; + } + } +} + /* * Start the sequence to attach a domain to a master. The sequence contain= s three * steps: @@ -3168,12 +3376,16 @@ int arm_smmu_attach_prepare(struct arm_smmu_attach_= state *state, arm_smmu_ats_supported(master); } =20 + ret =3D arm_smmu_attach_prepare_invs(state, smmu_domain); + if (ret) + return ret; + if (smmu_domain) { if (new_domain->type =3D=3D IOMMU_DOMAIN_NESTED) { ret =3D arm_smmu_attach_prepare_vmaster( state, to_smmu_nested_domain(new_domain)); if (ret) - return ret; + goto err_unprepare_invs; } =20 master_domain =3D kzalloc(sizeof(*master_domain), GFP_KERNEL); @@ -3221,6 +3433,8 @@ int arm_smmu_attach_prepare(struct arm_smmu_attach_st= ate *state, atomic_inc(&smmu_domain->nr_ats_masters); list_add(&master_domain->devices_elm, &smmu_domain->devices); spin_unlock_irqrestore(&smmu_domain->devices_lock, flags); + + arm_smmu_install_new_domain_invs(state); } =20 if (!state->ats_enabled && master->ats_enabled) { @@ -3240,6 +3454,9 @@ int arm_smmu_attach_prepare(struct arm_smmu_attach_st= ate *state, kfree(master_domain); err_free_vmaster: kfree(state->vmaster); +err_unprepare_invs: + kfree(state->old_domain_ninvs); + kfree(state->new_domain_ninvs); return ret; } =20 @@ -3271,6 +3488,7 @@ void arm_smmu_attach_commit(struct arm_smmu_attach_st= ate *state) } =20 arm_smmu_remove_master_domain(master, state->old_domain, state->ssid); + arm_smmu_install_old_domain_invs(state); master->ats_enabled =3D state->ats_enabled; } =20 --=20 2.43.0 From nobody Sat Oct 4 17:29:51 2025 Received: from NAM12-BN8-obe.outbound.protection.outlook.com (mail-bn8nam12on2082.outbound.protection.outlook.com [40.107.237.82]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A893E1991D2 for ; Thu, 14 Aug 2025 01:26:33 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.237.82 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755134795; cv=fail; b=D3w4swx9PYkGeHMPu1A3fgTwSsZenvKG4Qw40zZPKUQQ2d+onWzHDAnjjXAFSK12jZr5NQC1TnTFyPfEtxjxms5fif3zFymeiQcep5xvFMHNmqjcC+WAVioEW1S0OKfN5V7n1d7T07dJlre/LY0VDrFrIH496Fyz6IAHWcvl7YY= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755134795; c=relaxed/simple; bh=YO2iAHObku2jzCZuK7wxk6nK98fCCt7xO6P4tp4aPG4=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=m11k6WxU8mFpseEjKhV/euK0jpByTEnTl7zIjtL2bvi6PvXlkd8rbKUbG92kB4ul5fBRRGIFTSpMIWjzzDLlKHA7k3P3MIjsO87t3s2R1DEQNrnyRssoZDjdOxYmm6ExPjizzFJc3dZOeqbcPcYvBZdTH4g317AyqDFXqlzPM8E= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=sGY9M5Aw; arc=fail smtp.client-ip=40.107.237.82 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="sGY9M5Aw" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=FYh5OgYkgB9wiqd+8a+WaXlvQSgsFjuTon735NOOMjKexBDKTj0relkklN47nG7sCWcorndBNXtPhAw/v/du07wYYZF9FIDVj5dMV43W/hmm4O8dl+6R/8rGI6rO2bAhlsL+rMQoqTnnOIKLqM1GswVf4EzqU5Hw2A4t8uSUWVVH3xIUIOhJk7qzLzPggv3IDdg7cpqddFxBriQtt6CV+1ywDSRSaUf+TtbyeHNfHVUAL+mfmTaTini0JuFbZpHEunNt79RSrGsFHN+qqntq8wyDLgFbQmRy6c1lKbiKCkfGgLWMz0S2VLUHwZsoT/D+YBzU6Yc70KLLvbkY6COJew== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=zuulPnsBVTp/JuY0yLnOEEusgqIj2mp8ETgz6UjsfUE=; b=VcxxdGE2vOKo6zCxtb7dhoqD400DXThOpAanWihsj1tkMr91iQ0Osk646WeJG9SMCqmXmyquxBx+5vAFyt7MZdLAAdXl0DRKHXVahapWajB/zTCKhmsP9Nt/IEuRLgu1tWSpyhSH6lQ3Yu4gKQBFYWJWdWfy6EmYPnNPMsREX3o8KUkLl2n5njnQc3bxPSKrJ5esIoQ+F+YP4RPAx/DDTOF/5Q491hVzjWxrjdCleWHcj+fMdObNa1s2qdPSp2iUtgF8AvW+zjQDDzLf/OLW217X2UBxoSCbipBCDok0RbvHckiawaQ8Mm4bHPnrN/ZM/23MEDsqjpelQIYhf7VHiQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.161) smtp.rcpttodomain=kernel.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=zuulPnsBVTp/JuY0yLnOEEusgqIj2mp8ETgz6UjsfUE=; b=sGY9M5AwDyiJLzeWinzMyaiUF94eRyOCSNyFBvHDrFcBdPymSprqirBsx/kPOyays53e16e8B6yqkyhXnfQ69dd9rjVzrkjmmZRI8si5l8G2i8SdpPXs2mlElRQpaN3vOK4/L/mbuWpUVd8Ygz4fkwTe4CQG4GT28916Hv9xQDQmTpId+b77EqgDUPM6DpawaMlKDkjl4gttKTi2dndBc1EgaKZdQONg786a6/dKCGicgFz3MfzuY5owk/FWX92WqQFaKkyknmTtcJw2wlzR3U2n/CFNnkCbM7CHlqjB1y4yARzA+l0R2I+ZZwDlLNIhLLtF2XgR917MFBkRlaDFeA== Received: from SJ0PR13CA0217.namprd13.prod.outlook.com (2603:10b6:a03:2c1::12) by LV3PR12MB9165.namprd12.prod.outlook.com (2603:10b6:408:19f::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9009.18; Thu, 14 Aug 2025 01:26:27 +0000 Received: from SJ5PEPF000001F1.namprd05.prod.outlook.com (2603:10b6:a03:2c1:cafe::83) by SJ0PR13CA0217.outlook.office365.com (2603:10b6:a03:2c1::12) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.9031.15 via Frontend Transport; Thu, 14 Aug 2025 01:26:27 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.161) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.161 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.161; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.161) by SJ5PEPF000001F1.mail.protection.outlook.com (10.167.242.69) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9031.11 via Frontend Transport; Thu, 14 Aug 2025 01:26:27 +0000 Received: from rnnvmail204.nvidia.com (10.129.68.6) by mail.nvidia.com (10.129.200.67) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.14; Wed, 13 Aug 2025 18:26:11 -0700 Received: from rnnvmail205.nvidia.com (10.129.68.10) by rnnvmail204.nvidia.com (10.129.68.6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.14; Wed, 13 Aug 2025 18:26:10 -0700 Received: from Asurada-Nvidia.nvidia.com (10.127.8.9) by mail.nvidia.com (10.129.68.10) with Microsoft SMTP Server id 15.2.1544.14 via Frontend Transport; Wed, 13 Aug 2025 18:26:09 -0700 From: Nicolin Chen To: , , CC: , , , , , , , , , , , , Subject: [PATCH rfcv1 7/8] iommu/arm-smmu-v3: Add arm_smmu_invs based arm_smmu_domain_inv_range() Date: Wed, 13 Aug 2025 18:25:38 -0700 Message-ID: <8c4c5aec144f65cfd1fcef2eafb395876dac97ec.1755131672.git.nicolinc@nvidia.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SJ5PEPF000001F1:EE_|LV3PR12MB9165:EE_ X-MS-Office365-Filtering-Correlation-Id: 1364b41d-bacf-497f-3794-08dddad1972a X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|7416014|82310400026|36860700013|376014|1800799024|3613699012; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?6CynS+78chb6OeEf5C5MGr32AcnxiEP9zbUfVxsr/Yf8/ai8uZYJSe2dIo1J?= =?us-ascii?Q?7pbjr+YzbwEHrWxNqUaTCiiejWppA7gUYvd98hwAmu/xWN/2c9O+lXJZQRFB?= =?us-ascii?Q?lzJEQuMmbNrvjLL/apNEcnQR+FFFIac8ePad1CsLFB74+smaXU3+oE6ovyGG?= =?us-ascii?Q?lFO2qfxhSTb/txwICnQnzjPhlTNaD5trtvq4ETYDAzkqM6+LLg7UJNpfIHm0?= =?us-ascii?Q?7lJl0nZJClVacOEhhInIEebBnCUGpa7IfhQjnHg2AfpLpV3+nT4CtS65x9JL?= =?us-ascii?Q?Xwd/v61mqwjS958hYTPTV7f2ZTx3D5Cx7ksrAd9zARayrqKTlKN2irpaXOT6?= =?us-ascii?Q?+tX3E35cnCDoC5qZpyeNLXVw0Ha5cY64To6xocfoRl4hgpMadhG6Vi/d72lJ?= =?us-ascii?Q?tGCnID3GKvYT1aK0w93EIjHf0MYAyl27kC4fqFJspdKLga6hxqMfGuerOMYV?= =?us-ascii?Q?3QN2nUZd5Kbvn5q0BQfPx/MeMhFbYuUFnHP817C0Ka+DPxBJlaFERWnnyPKh?= =?us-ascii?Q?eYWcm1qGnCy2xKCmvLgB9fzp+n59iG10U6wW9jU94a5cNJV6qRVimwWecyRH?= =?us-ascii?Q?pvJQ2KPPTh374/Wet4qPzd59PgQGHKdReLq8hGF28+AQUCsdTi+KUKxVo05u?= =?us-ascii?Q?L9q/RQGda4xbuFOW90Qyykwmf0Qb38FOJMDwAIWO/LWb3OGcYK8vgx9P7lFs?= =?us-ascii?Q?mA6mKzQqoa868FNxI22eAmuVhstU8PRnSKcKxdz1swxr1A8EIlSordrmS7GO?= =?us-ascii?Q?e4ZD0eBTG4AbXCs6nBALKpgjlZaxLNmu2gIBIqjSCc6E66ClsuFglV+aNlz9?= =?us-ascii?Q?er73hgH9RicayvNJR+UpSPORbV5SrVUq+miR4LSfMST4nqen+rCCv+QWDKv8?= =?us-ascii?Q?4E3SiUnYNv8cSe1mwxDZwG6+S1CEiXwlGJH56/QyAlbZ9kGI/1dLw0DiGwYt?= =?us-ascii?Q?5eWubbaRl9nOvXzEhbF0+pLsb+ELZU9INYG9CFsdJd7nQuqxtFuCS45TXWIv?= =?us-ascii?Q?vXLKonRqcievv0BvJMyouEC8m++F9lK4okYkEV+nDdH+fDjNEGaukF60LB0y?= =?us-ascii?Q?DVhqQFDoTVjQtx6qMrOEVTa9QrQGVZhMJ31F+zUzgvjw3j9IprbyGwzh2VJ+?= =?us-ascii?Q?o1SyVyA5FDUXuJdXnJ68q6VbdmBY0VZHkmKnUDwnHUnHQh6wU5EmAGPAk3ep?= =?us-ascii?Q?B0UIIlLBcdsELHRYXzhOz2yw6g718FMmGJaBv4aK0fbmxcH8Ps0uHZlCaa6v?= =?us-ascii?Q?0pthNCnzH43C1n2o8tNCPo7aCQVu1aMQoZYSjH8b07jGN3GIOjUqnG40ltYL?= =?us-ascii?Q?yhFPYSCDVAMPtdJIPCt+W4YROXoZxnZor4weGJvfvNKDkO/08KCMSAcHuNtm?= =?us-ascii?Q?GoMqjOKvlyx+VttmO4wUoz2Z7orQMd4XdM7QKEZt+BpQBqrqEOJzlS52efWr?= =?us-ascii?Q?GisRtMiTZVs570DHo7/yhfVHYGDUQvHGGemRT9rAj7SGZ41jLPqkQYkcBfPX?= =?us-ascii?Q?+rQ0ZSppsmSIbbCoiSZCVlKzbRv6KHqkeqL0sQaN/SkFMq24wAsr+G1iXyR/?= =?us-ascii?Q?y2CqiATsygT3Dqjawiw=3D?= X-Forefront-Antispam-Report: CIP:216.228.117.161;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge2.nvidia.com;CAT:NONE;SFS:(13230040)(7416014)(82310400026)(36860700013)(376014)(1800799024)(3613699012);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Aug 2025 01:26:27.4283 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 1364b41d-bacf-497f-3794-08dddad1972a X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.161];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: SJ5PEPF000001F1.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: LV3PR12MB9165 Content-Type: text/plain; charset="utf-8" Now, each smmu_domain is built with an invs array that keeps all the IDs (asid/vmid) and its attached device SIDs, following the exact pattern of all the existing invalidation functions. Introduce a new arm_smmu_domain_inv helper iterating smmu_domain->invs, to convert the invalidation array to commands. Any invalidation request with no size specified means an entire flush over a range based one. Co-developed-by: Jason Gunthorpe Signed-off-by: Jason Gunthorpe Signed-off-by: Nicolin Chen --- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 9 + drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 215 ++++++++++++++++++-- 2 files changed, 211 insertions(+), 13 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/ar= m/arm-smmu-v3/arm-smmu-v3.h index 715179249eced..69271beb54527 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h @@ -1060,6 +1060,15 @@ void arm_smmu_tlb_inv_range_asid(unsigned long iova,= size_t size, int asid, int arm_smmu_atc_inv_domain(struct arm_smmu_domain *smmu_domain, unsigned long iova, size_t size); =20 +void arm_smmu_domain_inv_range(struct arm_smmu_domain *smmu_domain, + unsigned long iova, size_t size, + unsigned int granule, bool leaf); + +static inline void arm_smmu_domain_inv(struct arm_smmu_domain *smmu_domain) +{ + arm_smmu_domain_inv_range(smmu_domain, 0, 0, 0, false); +} + void __arm_smmu_cmdq_skip_err(struct arm_smmu_device *smmu, struct arm_smmu_cmdq *cmdq); int arm_smmu_init_one_queue(struct arm_smmu_device *smmu, diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/ar= m/arm-smmu-v3/arm-smmu-v3.c index 95615525b0ab8..aa770275029e2 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -2530,23 +2530,19 @@ static void arm_smmu_tlb_inv_context(void *cookie) arm_smmu_atc_inv_domain(smmu_domain, 0, 0); } =20 -static void __arm_smmu_tlb_inv_range(struct arm_smmu_cmdq_ent *cmd, - unsigned long iova, size_t size, - size_t granule, - struct arm_smmu_domain *smmu_domain) +static void arm_smmu_cmdq_batch_add_range(struct arm_smmu_device *smmu, + struct arm_smmu_cmdq_batch *cmds, + struct arm_smmu_cmdq_ent *cmd, + unsigned long iova, size_t size, + size_t granule, size_t pgsize) { - struct arm_smmu_device *smmu =3D smmu_domain->smmu; - unsigned long end =3D iova + size, num_pages =3D 0, tg =3D 0; + unsigned long end =3D iova + size, num_pages =3D 0, tg =3D pgsize; size_t inv_range =3D granule; - struct arm_smmu_cmdq_batch cmds; =20 if (!size) return; =20 if (smmu->features & ARM_SMMU_FEAT_RANGE_INV) { - /* Get the leaf page size */ - tg =3D __ffs(smmu_domain->domain.pgsize_bitmap); - num_pages =3D size >> tg; =20 /* Convert page size of 12,14,16 (log2) to 1,2,3 */ @@ -2566,8 +2562,6 @@ static void __arm_smmu_tlb_inv_range(struct arm_smmu_= cmdq_ent *cmd, num_pages++; } =20 - arm_smmu_cmdq_batch_init(smmu, &cmds, cmd); - while (iova < end) { if (smmu->features & ARM_SMMU_FEAT_RANGE_INV) { /* @@ -2595,9 +2589,26 @@ static void __arm_smmu_tlb_inv_range(struct arm_smmu= _cmdq_ent *cmd, } =20 cmd->tlbi.addr =3D iova; - arm_smmu_cmdq_batch_add(smmu, &cmds, cmd); + arm_smmu_cmdq_batch_add(smmu, cmds, cmd); iova +=3D inv_range; } +} + +static void __arm_smmu_tlb_inv_range(struct arm_smmu_cmdq_ent *cmd, + unsigned long iova, size_t size, + size_t granule, + struct arm_smmu_domain *smmu_domain) +{ + struct arm_smmu_device *smmu =3D smmu_domain->smmu; + struct arm_smmu_cmdq_batch cmds; + size_t pgsize; + + /* Get the leaf page size */ + pgsize =3D __ffs(smmu_domain->domain.pgsize_bitmap); + + arm_smmu_cmdq_batch_init(smmu, &cmds, cmd); + arm_smmu_cmdq_batch_add_range(smmu, &cmds, cmd, iova, size, granule, + pgsize); arm_smmu_cmdq_batch_submit(smmu, &cmds); } =20 @@ -2653,6 +2664,184 @@ void arm_smmu_tlb_inv_range_asid(unsigned long iova= , size_t size, int asid, __arm_smmu_tlb_inv_range(&cmd, iova, size, granule, smmu_domain); } =20 +static bool arm_smmu_inv_size_too_big(struct arm_smmu_device *smmu, size_t= size, + size_t granule) +{ + size_t max_tlbi_ops; + + /* 0 size means invalidate all */ + if (!size || size =3D=3D SIZE_MAX) + return true; + + if (smmu->features & ARM_SMMU_FEAT_RANGE_INV) + return false; + + /* + * Borrowed from the MAX_TLBI_OPS in arch/arm64/include/asm/tlbflush.h, + * this is used as a threshold to replace "size_opcode" commands with a + * single "nsize_opcode" command, when SMMU doesn't implement the range + * invalidation feature, where there can be too many per-granule TLBIs, + * resulting in a soft lockup. + */ + max_tlbi_ops =3D 1 << (ilog2(granule) - 3); + return size >=3D max_tlbi_ops * granule; +} + +/* Used by non INV_TYPE_ATS* invalidations */ +static void arm_smmu_inv_to_cmdq_batch(struct arm_smmu_inv *inv, + struct arm_smmu_cmdq_batch *cmds, + struct arm_smmu_cmdq_ent *cmd, + unsigned long iova, size_t size, + unsigned int granule) +{ + if (arm_smmu_inv_size_too_big(inv->smmu, size, granule)) { + cmd->opcode =3D inv->nsize_opcode; + /* nsize_opcode always needs a sync, no batching */ + arm_smmu_cmdq_issue_cmd_with_sync(inv->smmu, cmd); + return; + } + + cmd->opcode =3D inv->size_opcode; + arm_smmu_cmdq_batch_add_range(inv->smmu, cmds, cmd, iova, size, granule, + inv->pgsize); +} + +static bool arm_smmu_invs_end_batch(struct arm_smmu_invs *invs, size_t idx) +{ + struct arm_smmu_inv *cur =3D &invs->inv[idx]; + struct arm_smmu_inv *next; + + /* Last array entry ends */ + if (idx + 1 =3D=3D invs->num_invs) + return true; + + next =3D &cur[1]; + /* Changing smmu means changing command queue */ + if (cur->smmu !=3D next->smmu) + return true; + /* The batch for S2 TLBI must be done before nested S1 ASIDs */ + if (next->type =3D=3D INV_TYPE_S2_VMID_S1_CLEAR) + return true; + /* ATS must be after a sync of the S1/S2 invalidations */ + if (cur->type !=3D INV_TYPE_ATS && cur->type !=3D INV_TYPE_ATS_FULL && + (next->type =3D=3D INV_TYPE_ATS || next->type =3D=3D INV_TYPE_ATS_FUL= L)) + return true; + return false; +} + +void arm_smmu_domain_inv_range(struct arm_smmu_domain *smmu_domain, + unsigned long iova, size_t size, + unsigned int granule, bool leaf) +{ + struct arm_smmu_cmdq_batch cmds =3D {}; + struct arm_smmu_invs *invs; + bool retried =3D false; + size_t i; + + /* + * An invalidation request must follow some IOPTE change and then load + * the invalidation array In the meantime, a domain attachment mutates + * the array and then stores an STE/CD asking SMMU HW to acquire those + * changed IOPTEs. In other word, these two are interdependent and can + * race. + * + * In a race, the RCU design (with its underlying memory barriers) can + * ensure the invalidations array to always get updated before loaded. + * + * smp_mb() is used here, paired with the smp_mb() following the array + * update in a concurrent attach, to ensure: + * - HW sees the new IOPTEs if it walks after STE installation + * - Invalidation thread sees the updated array with the new ASID. + * + * [CPU0] | [CPU1] + * | + * change IOPTEs and TLB flush: | + * arm_smmu_domain_inv_range() { | arm_smmu_install_new_domain_invs { + * ... | rcu_assign_pointer(new_invs); + * smp_mb(); // ensure IOPTEs | smp_mb(); // ensure new_invs + * ... | kfree_rcu(old_invs, rcu); + * // load invalidation array | } + * invs =3D rcu_dereference(); | arm_smmu_install_ste_for_dev { + * | STE =3D TTB0 // read new IOPTEs + */ + smp_mb(); + + rcu_read_lock(); +again: + invs =3D rcu_dereference(smmu_domain->invs); + + /* A concurrent attachment might have changed the array. Do a respin */ + if (unlikely(!read_trylock(&invs->rwlock))) + goto again; + /* Only one retry. Otherwise, it would soft lockup on an empty array */ + if (!retried && unlikely(!invs->num_invs)) { + read_unlock(&invs->rwlock); + retried =3D true; + goto again; + } + + for (i =3D 0; i < invs->num_invs; i++) { + struct arm_smmu_inv *cur =3D &invs->inv[i]; + struct arm_smmu_device *smmu =3D cur->smmu; + struct arm_smmu_cmdq_ent cmd =3D { + /* + * Pick size_opcode to run arm_smmu_get_cmdq(). This can + * be changed to nsize_opcode, which would result in the + * same CMDQ pointer. + */ + .opcode =3D cur->size_opcode, + }; + + /* Do not throw any trash to the command queue */ + if (refcount_read(&cur->users) =3D=3D 0) + continue; + + if (!cmds.num) + arm_smmu_cmdq_batch_init(smmu, &cmds, &cmd); + + switch (cur->type) { + case INV_TYPE_S1_ASID: + cmd.tlbi.asid =3D cur->id; + cmd.tlbi.leaf =3D leaf; + arm_smmu_inv_to_cmdq_batch(cur, &cmds, &cmd, iova, size, + granule); + break; + case INV_TYPE_S2_VMID: + cmd.tlbi.vmid =3D cur->id; + cmd.tlbi.leaf =3D leaf; + arm_smmu_inv_to_cmdq_batch(cur, &cmds, &cmd, iova, size, + granule); + break; + case INV_TYPE_S2_VMID_S1_CLEAR: + /* CMDQ_OP_TLBI_S12_VMALL already flushed S1 entries */ + if (arm_smmu_inv_size_too_big(cur->smmu, size, granule)) + continue; + /* Just a single CMDQ_OP_TLBI_NH_ALL, no batching */ + cmd.tlbi.vmid =3D cur->id; + arm_smmu_cmdq_issue_cmd_with_sync(cur->smmu, &cmd); + continue; + case INV_TYPE_ATS: + arm_smmu_atc_inv_to_cmd(cur->ssid, iova, size, &cmd); + cmd.atc.sid =3D cur->id; + arm_smmu_cmdq_batch_add(smmu, &cmds, &cmd); + break; + case INV_TYPE_ATS_FULL: + arm_smmu_atc_inv_to_cmd(IOMMU_NO_PASID, 0, 0, &cmd); + cmd.atc.sid =3D cur->id; + arm_smmu_cmdq_batch_add(smmu, &cmds, &cmd); + break; + default: + WARN_ON_ONCE(1); + continue; + } + + if (cmds.num && arm_smmu_invs_end_batch(invs, i)) + arm_smmu_cmdq_batch_submit(smmu, &cmds); + } + read_unlock(&invs->rwlock); + rcu_read_unlock(); +} + static void arm_smmu_tlb_inv_page_nosync(struct iommu_iotlb_gather *gather, unsigned long iova, size_t granule, void *cookie) --=20 2.43.0 From nobody Sat Oct 4 17:29:51 2025 Received: from NAM12-DM6-obe.outbound.protection.outlook.com (mail-dm6nam12on2052.outbound.protection.outlook.com [40.107.243.52]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3680A1E98FB for ; Thu, 14 Aug 2025 01:26:33 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.243.52 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755134796; cv=fail; b=G/1vCkuuEfanS0Tx9Cdmo9YJpR0zzFsepaGOzoFGWgy8AccRd0JwpeJMoH6sg23p2jDNX0TuqkiOaPtrDCNtRzkxYEULyDq9zdYMvNt5w8nT5aBshm5hK4ktotF0i9rw5XPvhhw2J1H/wrAoTM1Vr2qwFoM+Yk6LWaumA1K6nYU= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755134796; c=relaxed/simple; bh=gu1b9lydvZRvXZ2GYX/kHfUBQiJYA0gPorRMrm7BQHQ=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=n00mVGZum0u+FRe+mJbUwdBNYPUc3C8mSO9Hc+Y+WEcvLDQ5cQnBpBOghH+eQTWUwsuvO4dBEfjFndCw6+bDEAZp6+FghbvFObDopqOwbI/bLr6kNBU3DRnL15b1sdM5ewQvMKJZW3P7l2fXJwk0/byafalHkFXVOKRysIHGoSs= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=JClj7FO6; arc=fail smtp.client-ip=40.107.243.52 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="JClj7FO6" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=L9lty397IA+civDfYb7+TT/LSJNG3ByEI+exHryltFFl6OaITBYk2rMT35bXitQ39POBobsSFFxmrseLA6piKUkvR1daFbiZ5NhjzISt8aZh7gobd4/wnFYYIFileAoOCeVqFgR7wLknaUx94SpPRVobgJ96Zjjj7JssoAg9eW9euJ6Tp2pti+Y7l39OCUoHtDHDgrZU80My1roFKGjR39BoJ0yZW+rdNCy3LTlShJ8sjKF/HfsGp8UzY0+XWI3crdenjzTcyd/n9r6gpdtx2id+AMUqc7ttCt4b5yGUpgxVz/8O8EmE4svtyAPTKLR+iSepLd2KRx7padaKqANQmA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=pcWvSIyq0a0uu6CGR9CTlc+66BEUW8FvGiGckjIBPqs=; b=KHj6YVS7BvDgcNqZaqSqBKcUX+Q8pd3ssDaBM9t/O5NACYSorSgeitcLtJi5ZOPS2qI5pP2EK3w0oNMpmVLwf/gSi2ogGYRIevz8n72GHYtfNKiHMnOTtC6vaLd5O9FMtAeXM2+5jKzYWVv2AuqOHLlx+QmkyP1WYNfzfA5BbGLxLjgM5nNr77mmTjPwfgWRd/leXs6A/nv/q3Ior1JhByifW1iN20tmPMa1fLCaFHrMedNBOtT3FnK1cjo2LhcoTjOezGR26vs1c+WYIEst8j0KaQ2y93ddQKlqAcJ1E/qd64XMauoQ6wgeVoLhIOt0BJHhIUy7rgifzZ/j0bLliQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.160) smtp.rcpttodomain=kernel.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=pcWvSIyq0a0uu6CGR9CTlc+66BEUW8FvGiGckjIBPqs=; b=JClj7FO64h597s9Aq0ZaBERhjIvTUy95HZVnEpnwr3XxnDEaR2jWko4OHUWcK5cWX5tgSDmYIDd9/ri4ysNQIC4HTj/n/JQEUBE5zVlYXjeBcbDxAmva9aSiiJn7zZEVnv9S9byBqeTxiWq+0sNuYfbp5EUX+fl4CyysJfxzyCKn4dKfDJ1mt2s3cNP9fPBkmiYS9cPmOm0H9UeenuvUgo8lATdjH2DJKXbB1E0Ciid7TMrPHrqNr9HvA3hNODqHJiHlrPwkz+tGJJDlk5cG2qOvFKrsA/KlYtA5noLuSuUIMMfpe2y+hpvSLVEttuUKOcqvq9I75Cwij4intyuO/Q== Received: from BN9PR03CA0636.namprd03.prod.outlook.com (2603:10b6:408:13b::11) by PH7PR12MB6718.namprd12.prod.outlook.com (2603:10b6:510:1b1::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9031.13; Thu, 14 Aug 2025 01:26:27 +0000 Received: from BN2PEPF000044A2.namprd02.prod.outlook.com (2603:10b6:408:13b:cafe::69) by BN9PR03CA0636.outlook.office365.com (2603:10b6:408:13b::11) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.9031.16 via Frontend Transport; Thu, 14 Aug 2025 01:26:26 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.160) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.160 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.160; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.160) by BN2PEPF000044A2.mail.protection.outlook.com (10.167.243.153) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9031.11 via Frontend Transport; Thu, 14 Aug 2025 01:26:26 +0000 Received: from rnnvmail204.nvidia.com (10.129.68.6) by mail.nvidia.com (10.129.200.66) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.14; Wed, 13 Aug 2025 18:26:12 -0700 Received: from rnnvmail205.nvidia.com (10.129.68.10) by rnnvmail204.nvidia.com (10.129.68.6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.14; Wed, 13 Aug 2025 18:26:12 -0700 Received: from Asurada-Nvidia.nvidia.com (10.127.8.9) by mail.nvidia.com (10.129.68.10) with Microsoft SMTP Server id 15.2.1544.14 via Frontend Transport; Wed, 13 Aug 2025 18:26:11 -0700 From: Nicolin Chen To: , , CC: , , , , , , , , , , , , Subject: [PATCH rfcv1 8/8] iommu/arm-smmu-v3: Perform per-domain invalidations using arm_smmu_invs Date: Wed, 13 Aug 2025 18:25:39 -0700 Message-ID: X-Mailer: git-send-email 2.43.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BN2PEPF000044A2:EE_|PH7PR12MB6718:EE_ X-MS-Office365-Filtering-Correlation-Id: 686e6f54-863a-4be0-0f74-08dddad196b6 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|36860700013|82310400026|7416014|1800799024; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?cHrKSq9iXvAyDcjhzsN18BWqb32qCKcRQwIsr44Zeq6vzm0vAQC+xiWdUiBN?= =?us-ascii?Q?nd+3bqRFs27Ageo6MaFdVeZq+pXgHPToyvymFcp3+nHRgB7xkMFhqwcYqZXP?= =?us-ascii?Q?6nxJqIkakskMJhGqIFlytwvspiv21CYwOiINl91ki/ubNB0vjXocJI9ngM9c?= =?us-ascii?Q?MlOITHxXPGUkk2h0ALu/aGak5oTeHzR8Z7A5/dA6/SaAfKRTz+0/XUJHiKmE?= =?us-ascii?Q?8VXIf57uNhwAzfQmJOYyl5v+47STpI2sgY3BahyI1Cty1kkRgDn/R4FhwXvM?= =?us-ascii?Q?QY31bTOvQ0+aGNvwxU0eOSAq67j+aeAYT5Vbe/o4pYOlLla3RTQmrUReKRrM?= =?us-ascii?Q?/PTsXn4QU1rxVMjpM7DJvwZucjul+G1ffHIF1ArMOAEk/xnkl8NydpnEoIHo?= =?us-ascii?Q?9UdXsd1VwQmLzWqIv3XTfL+MEEeINwnReba2v4VWB5pQO0KyGwAhu72d2bH3?= =?us-ascii?Q?JKwmvuTdHfoIu68QKsUQRqUydum37kLzx3jWKrAO9LwGMsR/yoXj+MDNueEF?= =?us-ascii?Q?WZUf0O6BocFl9ZxKT3YDNNZboOgpCZUrGyzOiMPzmkuH4H8nlK1nVe1qOvO4?= =?us-ascii?Q?JHvjCiJzpAPUotyBj47u8I1BlW0jNwXaA1quqEDImpC5vXBrVCS0FWOsLySk?= =?us-ascii?Q?QabJB3yodxttcxglW280/yAd1JcS7RgZBL6g119MwQRJr1pwjYU9B1Sy68iL?= =?us-ascii?Q?QnE9kpnOwf/HS8JDH4KpZ9W5k9UrWy6mE1b6DOKJlLukGIGqMFhx6TPrfdyJ?= =?us-ascii?Q?BJ5hkLUARxlK1B6FfpohiA6nouw7A4LkIfOpJQ6VS7LVywWIdL1tWesC0ZSp?= =?us-ascii?Q?zn/85l/8G6CYn0YTP6wtZD+tx8d9RLNxnQigwZFTVDGvtrOU93LKuw0ilAuW?= =?us-ascii?Q?COqT2Xpr2ONQchO+RwSZMN2RDUFKg6T4+RLvfKjaRDjbmlN+RzScPedBVbPs?= =?us-ascii?Q?7t1LRocDWJ3pcjd8ZAOI4AM7465SPckWh8XhficXz2tyoqbF86T0a79b9yYB?= =?us-ascii?Q?Ft0n6XM3c9LWO0pURAkhfT8MsILWiWIQ17gtKStqVHrlZkPv9Z2dO2nwsM2F?= =?us-ascii?Q?LHei3L+wfvA0T6Qe4KC7SYaASbJO7AojCSrXTfqf280hObezqy0IEN2K1M2U?= =?us-ascii?Q?Ar39/wZClqNl3+4m4nYBcfuvvWbD1AMXou1ZkxsIsGOscFbu7p9QjPWzPL8A?= =?us-ascii?Q?1WTEpHWFXoOAsMppt1+1QOoYEjhfjoU2H6eQGUeFpjTv7C/R1lQx7OlBl8X8?= =?us-ascii?Q?x2CLvtQg7RFPPUWbPrJNx1kIdyQXJCHW7qNVPwbW07adIOwU3i/6/kXjf2jh?= =?us-ascii?Q?oVzWWy3fhxofAK2xNyN4o62tbi4jGHH+ApYhIQLTs7zw99OPWojPJXSCLPRj?= =?us-ascii?Q?5uq+/VGfULA6QUxbCCa8s85NrJBTjixyPkBojqM9X/cbhdp+mCH4W9WnL1Ko?= =?us-ascii?Q?xHaGEe1W5yX2HQBNrwx/YFEECZhSkbzgwc3KXX3xJEafH5LeJ7C5U+SsnmZF?= =?us-ascii?Q?31Fw60ztVZcN3OQ8eWZ3qnO/IQThfM/yl++P?= X-Forefront-Antispam-Report: CIP:216.228.117.160;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge1.nvidia.com;CAT:NONE;SFS:(13230040)(376014)(36860700013)(82310400026)(7416014)(1800799024);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Aug 2025 01:26:26.6183 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 686e6f54-863a-4be0-0f74-08dddad196b6 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.160];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BN2PEPF000044A2.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR12MB6718 Content-Type: text/plain; charset="utf-8" Replace the old invalidation functions with arm_smmu_domain_inv_range() in all the existing invalidation routines. And deprecate the old functions. The new arm_smmu_domain_inv_range() handles the CMDQ_MAX_TLBI_OPS as well, so drop it in the SVA function. Since arm_smmu_cmdq_batch_add_range() has only one caller now, and it must be given a valid size, add a WARN_ON_ONCE to catch any missed case. Signed-off-by: Nicolin Chen --- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 7 - .../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 29 +-- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 165 +----------------- 3 files changed, 11 insertions(+), 190 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/ar= m/arm-smmu-v3/arm-smmu-v3.h index 69271beb54527..4ccb03c4a69d0 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h @@ -1053,13 +1053,6 @@ int arm_smmu_set_pasid(struct arm_smmu_master *maste= r, struct arm_smmu_domain *smmu_domain, ioasid_t pasid, struct arm_smmu_cd *cd, struct iommu_domain *old); =20 -void arm_smmu_tlb_inv_asid(struct arm_smmu_device *smmu, u16 asid); -void arm_smmu_tlb_inv_range_asid(unsigned long iova, size_t size, int asid, - size_t granule, bool leaf, - struct arm_smmu_domain *smmu_domain); -int arm_smmu_atc_inv_domain(struct arm_smmu_domain *smmu_domain, - unsigned long iova, size_t size); - void arm_smmu_domain_inv_range(struct arm_smmu_domain *smmu_domain, unsigned long iova, size_t size, unsigned int granule, bool leaf); diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iomm= u/arm/arm-smmu-v3/arm-smmu-v3-sva.c index fc601b494e0af..048b53f79b144 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c @@ -122,15 +122,6 @@ void arm_smmu_make_sva_cd(struct arm_smmu_cd *target, } EXPORT_SYMBOL_IF_KUNIT(arm_smmu_make_sva_cd); =20 -/* - * Cloned from the MAX_TLBI_OPS in arch/arm64/include/asm/tlbflush.h, this - * is used as a threshold to replace per-page TLBI commands to issue in the - * command queue with an address-space TLBI command, when SMMU w/o a range - * invalidation feature handles too many per-page TLBI commands, which will - * otherwise result in a soft lockup. - */ -#define CMDQ_MAX_TLBI_OPS (1 << (PAGE_SHIFT - 3)) - static void arm_smmu_mm_arch_invalidate_secondary_tlbs(struct mmu_notifier= *mn, struct mm_struct *mm, unsigned long start, @@ -146,21 +137,8 @@ static void arm_smmu_mm_arch_invalidate_secondary_tlbs= (struct mmu_notifier *mn, * range. So do a simple translation here by calculating size correctly. */ size =3D end - start; - if (!(smmu_domain->smmu->features & ARM_SMMU_FEAT_RANGE_INV)) { - if (size >=3D CMDQ_MAX_TLBI_OPS * PAGE_SIZE) - size =3D 0; - } else { - if (size =3D=3D ULONG_MAX) - size =3D 0; - } - - if (!size) - arm_smmu_tlb_inv_asid(smmu_domain->smmu, smmu_domain->cd.asid); - else - arm_smmu_tlb_inv_range_asid(start, size, smmu_domain->cd.asid, - PAGE_SIZE, false, smmu_domain); =20 - arm_smmu_atc_inv_domain(smmu_domain, start, size); + arm_smmu_domain_inv_range(smmu_domain, start, size, PAGE_SIZE, false); } =20 static void arm_smmu_mm_release(struct mmu_notifier *mn, struct mm_struct = *mm) @@ -191,8 +169,7 @@ static void arm_smmu_mm_release(struct mmu_notifier *mn= , struct mm_struct *mm) } spin_unlock_irqrestore(&smmu_domain->devices_lock, flags); =20 - arm_smmu_tlb_inv_asid(smmu_domain->smmu, smmu_domain->cd.asid); - arm_smmu_atc_inv_domain(smmu_domain, 0, 0); + arm_smmu_domain_inv(smmu_domain); } =20 static void arm_smmu_mmu_notifier_free(struct mmu_notifier *mn) @@ -301,7 +278,7 @@ static void arm_smmu_sva_domain_free(struct iommu_domai= n *domain) /* * Ensure the ASID is empty in the iommu cache before allowing reuse. */ - arm_smmu_tlb_inv_asid(smmu_domain->smmu, smmu_domain->cd.asid); + arm_smmu_domain_inv(smmu_domain); =20 /* * Notice that the arm_smmu_mm_arch_invalidate_secondary_tlbs op can diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/ar= m/arm-smmu-v3/arm-smmu-v3.c index aa770275029e2..8aec471af4316 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -1023,16 +1023,6 @@ static void arm_smmu_page_response(struct device *de= v, struct iopf_fault *unused } =20 /* Context descriptor manipulation functions */ -void arm_smmu_tlb_inv_asid(struct arm_smmu_device *smmu, u16 asid) -{ - struct arm_smmu_cmdq_ent cmd =3D { - .opcode =3D smmu->features & ARM_SMMU_FEAT_E2H ? - CMDQ_OP_TLBI_EL2_ASID : CMDQ_OP_TLBI_NH_ASID, - .tlbi.asid =3D asid, - }; - - arm_smmu_cmdq_issue_cmd_with_sync(smmu, &cmd); -} =20 static int arm_smmu_invs_cmp(const void *_l, const void *_r) { @@ -2444,74 +2434,10 @@ static int arm_smmu_atc_inv_master(struct arm_smmu_= master *master, return arm_smmu_cmdq_batch_submit(master->smmu, &cmds); } =20 -int arm_smmu_atc_inv_domain(struct arm_smmu_domain *smmu_domain, - unsigned long iova, size_t size) -{ - struct arm_smmu_master_domain *master_domain; - int i; - unsigned long flags; - struct arm_smmu_cmdq_ent cmd =3D { - .opcode =3D CMDQ_OP_ATC_INV, - }; - struct arm_smmu_cmdq_batch cmds; - - if (!(smmu_domain->smmu->features & ARM_SMMU_FEAT_ATS)) - return 0; - - /* - * Ensure that we've completed prior invalidation of the main TLBs - * before we read 'nr_ats_masters' in case of a concurrent call to - * arm_smmu_enable_ats(): - * - * // unmap() // arm_smmu_enable_ats() - * TLBI+SYNC atomic_inc(&nr_ats_masters); - * smp_mb(); [...] - * atomic_read(&nr_ats_masters); pci_enable_ats() // writel() - * - * Ensures that we always see the incremented 'nr_ats_masters' count if - * ATS was enabled at the PCI device before completion of the TLBI. - */ - smp_mb(); - if (!atomic_read(&smmu_domain->nr_ats_masters)) - return 0; - - arm_smmu_cmdq_batch_init(smmu_domain->smmu, &cmds, &cmd); - - spin_lock_irqsave(&smmu_domain->devices_lock, flags); - list_for_each_entry(master_domain, &smmu_domain->devices, - devices_elm) { - struct arm_smmu_master *master =3D master_domain->master; - - if (!master->ats_enabled) - continue; - - if (master_domain->nested_ats_flush) { - /* - * If a S2 used as a nesting parent is changed we have - * no option but to completely flush the ATC. - */ - arm_smmu_atc_inv_to_cmd(IOMMU_NO_PASID, 0, 0, &cmd); - } else { - arm_smmu_atc_inv_to_cmd(master_domain->ssid, iova, size, - &cmd); - } - - for (i =3D 0; i < master->num_streams; i++) { - cmd.atc.sid =3D master->streams[i].id; - arm_smmu_cmdq_batch_add(smmu_domain->smmu, &cmds, &cmd); - } - } - spin_unlock_irqrestore(&smmu_domain->devices_lock, flags); - - return arm_smmu_cmdq_batch_submit(smmu_domain->smmu, &cmds); -} - /* IO_PGTABLE API */ static void arm_smmu_tlb_inv_context(void *cookie) { struct arm_smmu_domain *smmu_domain =3D cookie; - struct arm_smmu_device *smmu =3D smmu_domain->smmu; - struct arm_smmu_cmdq_ent cmd; =20 /* * NOTE: when io-pgtable is in non-strict mode, we may get here with @@ -2520,14 +2446,7 @@ static void arm_smmu_tlb_inv_context(void *cookie) * insertion to guarantee those are observed before the TLBI. Do be * careful, 007. */ - if (smmu_domain->stage =3D=3D ARM_SMMU_DOMAIN_S1) { - arm_smmu_tlb_inv_asid(smmu, smmu_domain->cd.asid); - } else { - cmd.opcode =3D CMDQ_OP_TLBI_S12_VMALL; - cmd.tlbi.vmid =3D smmu_domain->s2_cfg.vmid; - arm_smmu_cmdq_issue_cmd_with_sync(smmu, &cmd); - } - arm_smmu_atc_inv_domain(smmu_domain, 0, 0); + arm_smmu_domain_inv(smmu_domain); } =20 static void arm_smmu_cmdq_batch_add_range(struct arm_smmu_device *smmu, @@ -2539,7 +2458,7 @@ static void arm_smmu_cmdq_batch_add_range(struct arm_= smmu_device *smmu, unsigned long end =3D iova + size, num_pages =3D 0, tg =3D pgsize; size_t inv_range =3D granule; =20 - if (!size) + if (WARN_ON_ONCE(!size)) return; =20 if (smmu->features & ARM_SMMU_FEAT_RANGE_INV) { @@ -2594,76 +2513,6 @@ static void arm_smmu_cmdq_batch_add_range(struct arm= _smmu_device *smmu, } } =20 -static void __arm_smmu_tlb_inv_range(struct arm_smmu_cmdq_ent *cmd, - unsigned long iova, size_t size, - size_t granule, - struct arm_smmu_domain *smmu_domain) -{ - struct arm_smmu_device *smmu =3D smmu_domain->smmu; - struct arm_smmu_cmdq_batch cmds; - size_t pgsize; - - /* Get the leaf page size */ - pgsize =3D __ffs(smmu_domain->domain.pgsize_bitmap); - - arm_smmu_cmdq_batch_init(smmu, &cmds, cmd); - arm_smmu_cmdq_batch_add_range(smmu, &cmds, cmd, iova, size, granule, - pgsize); - arm_smmu_cmdq_batch_submit(smmu, &cmds); -} - -static void arm_smmu_tlb_inv_range_domain(unsigned long iova, size_t size, - size_t granule, bool leaf, - struct arm_smmu_domain *smmu_domain) -{ - struct arm_smmu_cmdq_ent cmd =3D { - .tlbi =3D { - .leaf =3D leaf, - }, - }; - - if (smmu_domain->stage =3D=3D ARM_SMMU_DOMAIN_S1) { - cmd.opcode =3D smmu_domain->smmu->features & ARM_SMMU_FEAT_E2H ? - CMDQ_OP_TLBI_EL2_VA : CMDQ_OP_TLBI_NH_VA; - cmd.tlbi.asid =3D smmu_domain->cd.asid; - } else { - cmd.opcode =3D CMDQ_OP_TLBI_S2_IPA; - cmd.tlbi.vmid =3D smmu_domain->s2_cfg.vmid; - } - __arm_smmu_tlb_inv_range(&cmd, iova, size, granule, smmu_domain); - - if (smmu_domain->nest_parent) { - /* - * When the S2 domain changes all the nested S1 ASIDs have to be - * flushed too. - */ - cmd.opcode =3D CMDQ_OP_TLBI_NH_ALL; - arm_smmu_cmdq_issue_cmd_with_sync(smmu_domain->smmu, &cmd); - } - - /* - * Unfortunately, this can't be leaf-only since we may have - * zapped an entire table. - */ - arm_smmu_atc_inv_domain(smmu_domain, iova, size); -} - -void arm_smmu_tlb_inv_range_asid(unsigned long iova, size_t size, int asid, - size_t granule, bool leaf, - struct arm_smmu_domain *smmu_domain) -{ - struct arm_smmu_cmdq_ent cmd =3D { - .opcode =3D smmu_domain->smmu->features & ARM_SMMU_FEAT_E2H ? - CMDQ_OP_TLBI_EL2_VA : CMDQ_OP_TLBI_NH_VA, - .tlbi =3D { - .asid =3D asid, - .leaf =3D leaf, - }, - }; - - __arm_smmu_tlb_inv_range(&cmd, iova, size, granule, smmu_domain); -} - static bool arm_smmu_inv_size_too_big(struct arm_smmu_device *smmu, size_t= size, size_t granule) { @@ -2855,7 +2704,9 @@ static void arm_smmu_tlb_inv_page_nosync(struct iommu= _iotlb_gather *gather, static void arm_smmu_tlb_inv_walk(unsigned long iova, size_t size, size_t granule, void *cookie) { - arm_smmu_tlb_inv_range_domain(iova, size, granule, false, cookie); + struct arm_smmu_domain *smmu_domain =3D cookie; + + arm_smmu_domain_inv_range(smmu_domain, iova, size, granule, false); } =20 static const struct iommu_flush_ops arm_smmu_flush_ops =3D { @@ -4077,9 +3928,9 @@ static void arm_smmu_iotlb_sync(struct iommu_domain *= domain, if (!gather->pgsize) return; =20 - arm_smmu_tlb_inv_range_domain(gather->start, - gather->end - gather->start + 1, - gather->pgsize, true, smmu_domain); + arm_smmu_domain_inv_range(smmu_domain, gather->start, + gather->end - gather->start + 1, + gather->pgsize, true); } =20 static phys_addr_t --=20 2.43.0