From nobody Fri Sep 25 22:19:11 2026 Received: from DM5PR21CU001.outbound.protection.outlook.com (mail-centralusazon11011001.outbound.protection.outlook.com [52.101.62.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id ABE1239FCCD; Tue, 8 Sep 2026 07:48:26 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=52.101.62.1 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788853708; cv=fail; b=UJ/EzEzlPY8hGDTPUt1rQEZbopwYG47JImCxFc+ThlEftKalOH0LN8e7MKaKALtfy9m9L9XYrh7JJSAqJJcykVCV0jKa2E1xRedU5QcVOmn5DE+ck39CNiZPRMeUfBCmN4mp9hXJDt4dBQ1vQ/tZfJKzoId0MhaKfCV7wT14ZBs= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788853708; c=relaxed/simple; bh=pQwCmIzCVgaXoEaFIz+Ol66f8UjODTgXkjr5JL27TRQ=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=X6tJQOC1NQkYry8/w9pnIUm1B5VVMZkvKJ/wpWUfzcwd/OPcZ/V/L4Cv5Z/eaLePbbwbeD+iEHdVrgEjb0zBGxU8KaXM3FknsU2s38tVM4QNkHKxQdLHaPlNhYX0+7jvzfcQPwa6lZtFGrr/PToHOw1KyZxZhz4+aBGZoHul50A= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=AbBo9reX; arc=fail smtp.client-ip=52.101.62.1 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="AbBo9reX" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=VqR0NnGFj73wjzuk0h/bILjER/u6C422J/z934uvpGW3zZ1D0HMM/Y3NSLiRsUE/jPa6/LqZvqkGsxkqS6be7hr/KopOHpSgA1X84owz1xuF9/w5CoTYoAAstl/RvbHvTQxhCJAzGnvODIl9T2N/1UbfgLG8xPwSY12bKUDnOrSp1MgkuQRPsiC5a6y5IGqTkZ8bsOSLtkyquPeWPv5jr0u8afcbLbU8B46gLKKmik4qe+ARy1aRUE1P3+5SndrQn70hNLJubbMEJ4n4Av6u+dPl11dhsomyha/J0iydHsz+m5Vjlw4a7IZ3FS5TgkS0agPzInSRdofKYfMcCsLQIw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=FohchD1g/5xVdBrUXpDGCAzle6Fmxr5lQdON8S7B+qQ=; b=PpA1FbtDddm4SKAXzee/U9KdV8DW2VVKrrevmKeKqJNtxDiYAd/kpnzAW0R4zQAzCZQ10BZtCdlv9hzZUW0OE62NNu4EGDPqVr1ypQgNefMppSyTvUIWVaHJ3hjtivHwFRaMArqeCcHLR9ohOlPoXrvNLEIv0/zevFipSvdDXY7JHPiXMGIvbBtyp/SVZmGlkgRJB2VPlBuihNSTDSYPJPB9mga0DRqbXsTk88tL2KLWfJpW4SwjswhzzHk7/tM1TnqIMtdXfbeQcDmI6BBNFsXq1/VSQkdWrwQoKLwoueCuabiAtu9aycBvzMH+baO3p6nOuQQWBymDGqLmI1cGGg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=zytor.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=FohchD1g/5xVdBrUXpDGCAzle6Fmxr5lQdON8S7B+qQ=; b=AbBo9reXYtwheLLQIejCivy6gMOogkR6cqRicj+cnqWc8KWKFYhfSWvLg+v3PTI61zXVKniNh7IkdhMagyOH8yRGSs0e/z154aNCKqwVCgZptOldfGrANPx9bVZME0loag7oS3EDjeu88I4tdm9PbPpFP2ttRr5RKMuUyxoF/to= Received: from CY5PR15CA0205.namprd15.prod.outlook.com (2603:10b6:930:82::23) by IA1PR12MB7565.namprd12.prod.outlook.com (2603:10b6:208:42f::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.382.15; Tue, 8 Sep 2026 07:48:21 +0000 Received: from CY4PEPF0000EE30.namprd05.prod.outlook.com (2603:10b6:930:82:cafe::28) by CY5PR15CA0205.outlook.office365.com (2603:10b6:930:82::23) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.382.15 via Frontend Transport; Tue, 8 Sep 2026 07:48:21 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=satlexmb07.amd.com; pr=C Received: from satlexmb07.amd.com (165.204.84.17) by CY4PEPF0000EE30.mail.protection.outlook.com (10.167.242.36) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.406.5 via Frontend Transport; Tue, 8 Sep 2026 07:48:20 +0000 Received: from BLR-L1-SARUNKOD.amd.com (10.180.168.240) by satlexmb07.amd.com (10.181.42.216) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.46; Tue, 8 Sep 2026 02:48:15 -0500 From: Sairaj Kodilkar To: "H. Peter Anvin" , "Peter Zijlstra (Intel)" , Borislav Petkov , Dave Hansen , Ingo Molnar , "Mathieu Desnoyers" , Paolo Bonzini , Sairaj Kodilkar , "Sean Christopherson" , Thomas Gleixner , "Uros Bizjak" , , , CC: , Subject: [PATCH v4 1/2] x86/uaccess: Extend CMPXCHG user helpers to 128-bit operands Date: Tue, 8 Sep 2026 13:17:38 +0530 Message-ID: <20260908074739.10748-2-sarunkod@amd.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20260908074739.10748-1-sarunkod@amd.com> References: <20260908074739.10748-1-sarunkod@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: satlexmb07.amd.com (10.181.42.216) To satlexmb07.amd.com (10.181.42.216) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CY4PEPF0000EE30:EE_|IA1PR12MB7565:EE_ X-MS-Office365-Filtering-Correlation-Id: 1f44e31a-48ed-4d35-f204-08df0d7d8da8 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|36860700016|1800799024|7416014|376014|82310400026|23010399003|921020|3023799007|10067099003|18002099003|11063799006|56012099006|22082099003; X-Microsoft-Antispam-Message-Info: CQwgG9Z5VCovOTLrLOcUdJaT9A0BAuClSZM/F0vvRYNU/peSODXhB5yIKm2wnZS/tR4/7RTUPW0V4WNdutSGvgxPlZLYWGCSTZgZ8iPFBOtHBqrj8Au+tc+Z8zMMyb0/BjVYbfz7Sx/H08EUgUWGceUDYquxbF2t+NYkZnXYYNuIa3Td47D15RQDDCAqg2mGwm3YrrzFBmde5s836PbpyYCH34/CJiKBZlJGfEpS+UqE1GVPCbcrehPwyPqzVhap794oAYYuVaDuLBt1UWES4MSXL4mWIX45hnUbZaSFH+j2KXhrTYZFmavw42NHhiUoViIlGHLMrgilpy2P0CQQa55wSQAHNj5JBRjcjv1HPpOpOx7QsW+vA+haufe8+j0+raCiE0yswXgXle5rtNWS7raJgb+roDe9vQyxSgOIsUPING3w/3wiUMWpde2Ju0FbCYEMhCuUlAZc5i72FDZkiIiQvD4JnMpMPJGwjCbojAxM0aOWCvtb7+I5Szqq+FOAkRQ9u5aUspGRUu77gwI4YnSXi5VA7uHD00AwhPqDf3Pq+KZighV38/09kl2c9pxACtRYgW07eAWNIwOVe41CofhkhTZN/A16rQmDxN4Uxp2IzMt9cqlalyGGJU/aA4AGGc0r6XoTnkR4Qv2Efxdat02XbV16fXPVHR1Xx8ZZ4RFHEPlCQLTOuJee4F4FL60iih+FaCFdSVAN8XqB7XNbqXwDQDvxc3N3iZLSYPo509dP0hkF+65ouFMC3tLIM3A+ X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:satlexmb07.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(36860700016)(1800799024)(7416014)(376014)(82310400026)(23010399003)(921020)(3023799007)(10067099003)(18002099003)(11063799006)(56012099006)(22082099003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: x/jxf11XZ3ieiIDasTe0dleONMePJb/TkwUi7aEhMs0nC33uaWe9T6H4EhCO/An+ayW0VTQ64Jvmsu9uS5RYSc2zw2ylgaKE3MUm1XBXzd/GJaQbzikPkXKa6kuHORy8drH5Jl/U/xJBrsGlZTLrUWeEZ+DiQk+i3adJ5BxZ7RBSOWhRLHqfqIIrqKl4JErF9APvXGseXxdQg+jq8ufgI+H24W++f+zUodp+AQZRIYHK5jvj4jehCQzkmee8Xq0vtWm32ZQoPdPkwBX36flpV2UppCElHp3lkXGayQXo4+gRD+R2onpkVfOkDKaWYaxFnJk6e8VwGGebwy0G3qECKSJGtcA9EaTv1RjTt+TZidDA2GjaxNTZCHy/Loi7V5af4MSN92YPQZUWWs+SkZnSss6NVxgGbXOi510efSx7wou9mPkpOA2kzfb5SyIRwIam X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 08 Sep 2026 07:48:20.7745 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 1f44e31a-48ed-4d35-f204-08df0d7d8da8 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[satlexmb07.amd.com] X-MS-Exchange-CrossTenant-AuthSource: CY4PEPF0000EE30.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR12MB7565 Content-Type: text/plain; charset="utf-8" Extend the existing user CMPXCHG helpers to support 16-byte operands on x86-64, using LOCK_PREFIX "cmpxchg16b". This mirrors the existing __try_cmpxchg64_user_asm() / cmpxchg8b path provided for 32-bit kernels, where KVM needs an atomic compare-exchange wider than the generic cmpxchg helper can provide. On 32-bit kernels, stub the helper to generate build failure because cmpxchg16b requires 64-bit GPRs and is not available. KVM uses this to atomically emulate guest cmpxchg16b on guest RAM mapped via userspace addresses. Signed-off-by: Sairaj Kodilkar --- arch/x86/include/asm/uaccess.h | 64 +++++++++++++++++++++++++++++++++- 1 file changed, 63 insertions(+), 1 deletion(-) diff --git a/arch/x86/include/asm/uaccess.h b/arch/x86/include/asm/uaccess.h index 3a0dd3c2b233..6e10a8a3ba4d 100644 --- a/arch/x86/include/asm/uaccess.h +++ b/arch/x86/include/asm/uaccess.h @@ -407,6 +407,27 @@ do { \ if (unlikely(!success)) \ *_old =3D __old; \ likely(success); }) +#else // !CONFIG_X86_32 +#define __try_cmpxchg128_user_asm(_ptr, _pold, _new, label) ({ \ + bool success; \ + __typeof__(_ptr) _old =3D (__typeof__(_ptr))(_pold); \ + u64 __old_low =3D (u64)*_old; \ + u64 __old_high =3D (u64)(*_old >> 64); \ + __typeof__(*(_ptr)) __new =3D (_new); \ + asm_goto_output("\n" \ + "1: " LOCK_PREFIX "cmpxchg16b %[ptr]\n" \ + _ASM_EXTABLE_UA(1b, %l[label]) \ + : "=3D@ccz" (success), \ + "+a" (__old_low), \ + "+d" (__old_high), \ + [ptr] "+m" (*_ptr) \ + : "b" ((u64)__new), \ + "c" ((u64)((u128)__new >> 64)) \ + : "memory" \ + : label); \ + if (unlikely(!success)) \ + *_old =3D ((u128)__old_high << 64) | __old_low; \ + likely(success); }) #endif // CONFIG_X86_32 #else // !CONFIG_CC_HAS_ASM_GOTO_TIED_OUTPUT #define __try_cmpxchg_user_asm(itype, ltype, _ptr, _pold, _new, label) ({ \ @@ -463,6 +484,32 @@ do { \ if (unlikely(!__result)) \ *_old =3D __old; \ likely(__result); }) +#else //!CONFIG_X86_32 +#define __try_cmpxchg128_user_asm(_ptr, _pold, _new, label) ({ \ + int __result; \ + __typeof__(_ptr) _old =3D (__typeof__(_ptr))(_pold); \ + u64 __old_low =3D (u64)*_old; \ + u64 __old_high =3D (u64)(*_old >> 64); \ + __typeof__(*(_ptr)) __new =3D (_new); \ + asm volatile("\n" \ + "1: " LOCK_PREFIX "cmpxchg16b %[ptr]\n" \ + "mov $0, %[result]\n\t" \ + "setz %b[result]\n" \ + "2:\n" \ + _ASM_EXTABLE_TYPE_REG(1b, 2b, EX_TYPE_EFAULT_REG, \ + %[result]) \ + : [result] "=3Dq" (__result), \ + "+a" (__old_low), \ + "+d" (__old_high), \ + [ptr] "+m" (*_ptr) \ + : "b" ((u64)__new), \ + "c" ((u64)((u128)__new >> 64)) \ + : "memory", "cc"); \ + if (unlikely(__result < 0)) \ + goto label; \ + if (unlikely(!__result)) \ + *_old =3D ((u128)__old_high << 64) | __old_low; \ + likely(__result); }) #endif // CONFIG_X86_32 #endif // CONFIG_CC_HAS_ASM_GOTO_TIED_OUTPUT =20 @@ -551,11 +598,18 @@ do { \ =20 extern void __try_cmpxchg_user_wrong_size(void); =20 -#ifndef CONFIG_X86_32 +#if defined(CONFIG_X86_32) || !defined(X86_FEATURE_CX16) +/* + * Always fail on 32 bit arch or 64 arch without 128 bit cmpxchg support + */ +#define __try_cmpxchg128_user_asm(_ptr, _pold, _new, label) ({ BUILD_BUG_O= N(1); 0; }) +#else #define __try_cmpxchg64_user_asm(_ptr, _oldp, _nval, _label) \ __try_cmpxchg_user_asm("q", "r", (_ptr), (_oldp), (_nval), _label) + #endif =20 + /* * Force the pointer to u to match the size expected by the asm help= er. * clang/LLVM compiles all cases and only discards the unused paths after @@ -580,6 +634,14 @@ extern void __try_cmpxchg_user_wrong_size(void); case 8: __ret =3D __try_cmpxchg64_user_asm((__force u64 *)(_ptr), (_oldp)= ,\ (_nval), _label); \ break; \ + case 16: \ + if (boot_cpu_has(X86_FEATURE_CX16)) \ + __ret =3D __try_cmpxchg128_user_asm( \ + (__force u128 *)(_ptr), \ + (_oldp), (_nval), _label); \ + else \ + __ret =3D 0; \ + break; \ default: __try_cmpxchg_user_wrong_size(); \ } \ __ret; }) --=20 2.34.1 From nobody Fri Sep 25 22:19:11 2026 Received: from CO1PR03CU002.outbound.protection.outlook.com (mail-westus2azon11010070.outbound.protection.outlook.com [52.101.46.70]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C133139FCCD; Tue, 8 Sep 2026 07:48:43 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=52.101.46.70 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788853725; cv=fail; b=RmsMeW9eZxAi0hxM/yCsS+i/DMZa5B06xo/cE1GgiJAwZuEFAX+pWFrHw8JRR0CHhwpfG/MCYnf2Eye2Hhk68zyDGq/koKaL+mGTALK8QItfwxEjjtvL+l81z6UbUa9Fnl/0mRsxm98q7qRTSnYeTKOfd5Finm173KbG6mVxVXY= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788853725; c=relaxed/simple; bh=65r6mz9u21XnMm2Qr8EvNQDGidZSD4jrHhsD/Bs6oHI=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=Bd7veeDVSrmlzfQssBGrc7gFkV/QB7pVfd0XQMffTlpeqAOPnE+VMAaxDST0DP2FpMk75n0ybzbEVFn5Z1vihnU+1HCoYZpY6DOzZrrVoKYMk8r2XvOAczY3+kQH34FRhaYZ7XgYRxyX5fykcHUQlN2tga9AMfAsK7MXXVZ5YsQ= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=IMrlP0SV; arc=fail smtp.client-ip=52.101.46.70 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="IMrlP0SV" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=YvjU/SQzM+232eO8pZbp2cAWh9V2+L/IOcoAtcaQ7iB5XMVBf0FE4NcjgV5be1uvCB3/By3LoC8mxCehCU8mrR9YIdqH8+7061qph/IpJZWUAkDqnVdYGfXjtbOgALiaanZJF0c9vKRMBXbaZEZgMLLb6H+lIZqWcaAja/55flCkIF4vqEDlrmXiNADo6IyEsCETyvLxajQz3ammeq4E1cDiuoUXGq+S0oxXqG0MoJ9vkeoQJeped2YXeN3uQYgfRX19qj3f7OUGxmKaLJTBu8o48DjZMt1tf3GyVl2uCkoJUbA4egnMxuQXPZaaf7PFLphnVqxJIdt1l25jEfZPow== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=tdK3zbvfFAvLFJ8N2wgUjRoDe6bN6DLIL192uGMz0jg=; b=wLrZOQ2IuTG73FdeRyfmSmTAHKChUQYaJ+Vjf1BW+0k+rhfk/mGkUHy/NUo9w7di4vigvq7d4S6EjGfN5wkig5VDfTpxvFjB2qlNUgdvwupIxkYjpqImrE6Lb3UiOcR21fH2oi+DpuIsBhZkoAGdqoBSqFdR4Ft5c/qH0gppuWZLeY2Li5NJDMGbB0Q7ymESxPSwsEamXLKg0CZVMxZxgGitRHPE8QIDhAR6gkAS+saccix+j09TYKOinwUnWyBze2gqyHyyR54Gv58tvPYano7NGK5l/xQciC3c7XugJ7caMh5+3mUCrQRwEhUmWssUgNFJ+9ehHVU0g5XGzdUc/A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=zytor.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=tdK3zbvfFAvLFJ8N2wgUjRoDe6bN6DLIL192uGMz0jg=; b=IMrlP0SVfbnCXggl30YRXS/Q5oslbn9gBhMdob0H9zmQrENh5m4kFpMvAM8DAGlR+9Ag4TIJSjH9DF/ACqdXmVBiOBMB8IatwOT26TZuRNO4AfRkdaZZYsabmHr+rqUvoFpYaN6l67zegE7yVgeMLIHJvj+0shPQ7rLFqbhk8xQ= Received: from CY5PR15CA0188.namprd15.prod.outlook.com (2603:10b6:930:82::9) by DM4PR12MB6159.namprd12.prod.outlook.com (2603:10b6:8:a8::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.382.15; Tue, 8 Sep 2026 07:48:40 +0000 Received: from CY4PEPF0000EE30.namprd05.prod.outlook.com (2603:10b6:930:82:cafe::72) by CY5PR15CA0188.outlook.office365.com (2603:10b6:930:82::9) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.382.15 via Frontend Transport; Tue, 8 Sep 2026 07:48:39 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=satlexmb07.amd.com; pr=C Received: from satlexmb07.amd.com (165.204.84.17) by CY4PEPF0000EE30.mail.protection.outlook.com (10.167.242.36) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.406.5 via Frontend Transport; Tue, 8 Sep 2026 07:48:39 +0000 Received: from BLR-L1-SARUNKOD.amd.com (10.180.168.240) by satlexmb07.amd.com (10.181.42.216) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.46; Tue, 8 Sep 2026 02:48:34 -0500 From: Sairaj Kodilkar To: "H. Peter Anvin" , "Peter Zijlstra (Intel)" , Borislav Petkov , Dave Hansen , Ingo Molnar , "Mathieu Desnoyers" , Paolo Bonzini , Sairaj Kodilkar , "Sean Christopherson" , Thomas Gleixner , "Uros Bizjak" , , , CC: , Subject: [PATCH v4 2/2] KVM: x86: Add support for cmpxchg16b emulation Date: Tue, 8 Sep 2026 13:17:39 +0530 Message-ID: <20260908074739.10748-3-sarunkod@amd.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20260908074739.10748-1-sarunkod@amd.com> References: <20260908074739.10748-1-sarunkod@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: satlexmb07.amd.com (10.181.42.216) To satlexmb07.amd.com (10.181.42.216) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CY4PEPF0000EE30:EE_|DM4PR12MB6159:EE_ X-MS-Office365-Filtering-Correlation-Id: a6f2781c-6ef9-466d-e09b-08df0d7d98f3 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|36860700016|23010399003|1800799024|82310400026|7416014|10067099003|3023799007|56012099006|11063799006|22082099003|18002099003|13003099007|921020; X-Microsoft-Antispam-Message-Info: WgGCv8k5eP03t8f+PiUmjk2HwaeekSTMxKRUlmt57/HJg5pWnf6i+E1y/SWZHD4ICaGRi2OHA3G1Y5DdpiVnU15HjG9XVEj7dGey5+K64m602Nzh8ToQZm18azLY/O5Bq7UQJ9s82mglC8IsdRY7PR6DMxD78j86rqS4rhigfGCdbgM5g1EFxO+qQZfOf15t4AaThTtKAoh9UsPj4ofCwQ4MTgTAtWDMNDdZ/hjHMB9clNtATZmCWNktS7f8E+NslVSoNtTYQsmxyJmhPGOWg3xAD+TysFvk1K6rw0pKEhMWFnF21qyhBOBU9v6fn7KLK7iI5o3bUHQp+rwn6AMEqzjzziZmbhJAffd8PEFn6FPBv+Z3YNP/kQKu3F0gjcuxVxP8XxMN2cuaNBPgMOEryDeh7REggbnATwzVE8mmqrtalRY83rbgDzoCFC0HFVrTBgdZ2Yo4327kUz6eo4hbC8IbDeNuA1uo79YaDIzET1TerWGA2HUeCLE6jMfRoHUCi6ruqwz3YFa1x96Ke8kP+yIAiFK6B3C/MaaFHsT5UWDiMYgD0BkKjBraSop2LI+EC0xfO+qE0eFWxxc55q6ls8VEjl8pg26GVjHp92tejtPsIhdxZzxfrK8LVnnAdJDPN9REqmDljm84dy2W4/K5fhRzl1y6I+ni8HIdKTKdNGZXfB1Cw8tMeHpIZboQ4QERUS1SB8EylJLPuWfCRhv5xOYPByykMdCkQ+JMZZGPmbe6Kt0RYslcG+aPETRwkpMP X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:satlexmb07.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(376014)(36860700016)(23010399003)(1800799024)(82310400026)(7416014)(10067099003)(3023799007)(56012099006)(11063799006)(22082099003)(18002099003)(13003099007)(921020);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: I0zNvnX9P07TXljs7iDPSZuLyeIMYvJ7ILmNHMmOrvLYan/0rys2jLQAMFC5j7GlkSxOfcdo8Sw8+EQSw5MRfjEN3ht3yTUeP5ANpUIiXm2YHGC2bkmbQg45QZNdt/XmDhi1P9cwTw8+cFLccfaSwg1ELlG/flR4kJtwcCvePbo8/K7ftGUzNe7gUXJElk0KC4Swe7QVhU3o/KU+jD9dEFcKZj0b+81KbeLABQcK51iDl/jb/rBUNfYKS3/uZWmpY/ZqhhHRxlSkkE8imtH0YVfPw4bsekB/8Iu51eXchl4r5Yi201kSea4/1U1oKyJs/tWH7iekx78kRY4/mnhxFnUCUwTUickcjL8eN7+X+MkJKdUWAo5CdoXP3oiqCuXEqRV9dsrzf60F1dGfc+ZMSethFR2YEWANTM+rOPbY0DIQVD2jY23Hl7VsXLGCT+v0 X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 08 Sep 2026 07:48:39.7205 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: a6f2781c-6ef9-466d-e09b-08df0d7d98f3 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[satlexmb07.amd.com] X-MS-Exchange-CrossTenant-AuthSource: CY4PEPF0000EE30.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM4PR12MB6159 Content-Type: text/plain; charset="utf-8" AMD and Intel both provides support for 128 bit cmpxchg operands using cmpxchg8b/cmpxchg16b instructions (opcode 0FC7). However, kvm does not support emulating cmpxchg16b (i.e when destination memory is 128 bit and REX.W =3D 1) which causes emulation failure when QEMU guest performs a cmpxchg16b on a memory region setup as a IO. This failure is seen on the AMD IOMMU driver which writes 256-bit device table entries with two 128-bit cmpxchg operations. For guests using hardware-accelerated vIOMMU, QEMU traps device table accesses to set up nested page tables (see [1]). Without 128-bit cmpxchg emulation, KVM cannot handle these traps and DTE access emulation fails. Hence extend cmpxchg8b to perform cmpxchg16b when the destination memory is 128 bit. [1] https://github.com/AMDESE/qemu-iommu/blob/wip/for_iommufd_hw_queue-v8_a= md_viommu_20260106/hw/i386/amd_viommu.c#L517 Signed-off-by: Sairaj Kodilkar --- arch/x86/include/asm/uaccess.h | 2 +- arch/x86/kvm/emulate.c | 50 +++++++++++++++++++++++----------- arch/x86/kvm/kvm_emulate.h | 6 ++++ arch/x86/kvm/x86.c | 7 ++++- 4 files changed, 47 insertions(+), 18 deletions(-) diff --git a/arch/x86/include/asm/uaccess.h b/arch/x86/include/asm/uaccess.h index 6e10a8a3ba4d..948937a24cc5 100644 --- a/arch/x86/include/asm/uaccess.h +++ b/arch/x86/include/asm/uaccess.h @@ -598,7 +598,7 @@ do { \ =20 extern void __try_cmpxchg_user_wrong_size(void); =20 -#if defined(CONFIG_X86_32) || !defined(X86_FEATURE_CX16) +#ifdef CONFIG_X86_32 /* * Always fail on 32 bit arch or 64 arch without 128 bit cmpxchg support */ diff --git a/arch/x86/kvm/emulate.c b/arch/x86/kvm/emulate.c index c1b21282187f..535842336d76 100644 --- a/arch/x86/kvm/emulate.c +++ b/arch/x86/kvm/emulate.c @@ -2184,24 +2184,36 @@ static int em_call_near_abs(struct x86_emulate_ctxt= *ctxt) return rc; } =20 +#define em_cmpxchg8b_16b(__c, rbits, mbits)\ +do { \ + u##mbits old =3D __c->dst.orig_val##mbits; \ + \ + BUILD_BUG_ON(rbits * 2 !=3D mbits); \ + \ + if (((u##rbits) (old >> 0) !=3D (u##rbits) reg_read(__c, VCPU_REGS_RAX)) = || \ + ((u##rbits) (old >> rbits) !=3D (u##rbits) reg_read(__c, VCPU_REGS_RD= X))) { \ + *reg_write(__c, VCPU_REGS_RAX) =3D (u##rbits) (old >> 0); \ + *reg_write(__c, VCPU_REGS_RDX) =3D (u##rbits) (old >> rbits); \ + __c->eflags &=3D ~X86_EFLAGS_ZF; \ + } else { \ + __c->dst.val##mbits =3D ((u##mbits)reg_read(__c, VCPU_REGS_RCX) << rbits= ) | \ + (u##rbits) reg_read(__c, VCPU_REGS_RBX); \ + \ + __c->eflags |=3D X86_EFLAGS_ZF; \ + } \ +} while (0) + static int em_cmpxchg8b(struct x86_emulate_ctxt *ctxt) { - u64 old =3D ctxt->dst.orig_val64; - - if (ctxt->dst.bytes =3D=3D 16) + if (WARN_ON_ONCE(8 + !!(ctxt->rex_bits & REX_W) * 8 !=3D ctxt->dst.bytes)) return X86EMUL_UNHANDLEABLE; =20 - if (((u32) (old >> 0) !=3D (u32) reg_read(ctxt, VCPU_REGS_RAX)) || - ((u32) (old >> 32) !=3D (u32) reg_read(ctxt, VCPU_REGS_RDX))) { - *reg_write(ctxt, VCPU_REGS_RAX) =3D (u32) (old >> 0); - *reg_write(ctxt, VCPU_REGS_RDX) =3D (u32) (old >> 32); - ctxt->eflags &=3D ~X86_EFLAGS_ZF; - } else { - ctxt->dst.val64 =3D ((u64)reg_read(ctxt, VCPU_REGS_RCX) << 32) | - (u32) reg_read(ctxt, VCPU_REGS_RBX); - - ctxt->eflags |=3D X86_EFLAGS_ZF; - } + if (!(ctxt->rex_bits & REX_W)) + em_cmpxchg8b_16b(ctxt, 32, 64); +#ifdef CONFIG_X86_64 + else + em_cmpxchg8b_16b(ctxt, 64, 128); +#endif return X86EMUL_CONTINUE; } =20 @@ -5414,8 +5426,14 @@ int x86_emulate_insn(struct x86_emulate_ctxt *ctxt, = bool check_intercepts) goto done; } } - /* Copy full 64-bit value for CMPXCHG8B. */ - ctxt->dst.orig_val64 =3D ctxt->dst.val64; + /* Copy full 64/128-bit value for CMPXCHG8B. */ + +#ifdef CONFIG_X86_64 + if (ctxt->dst.bytes =3D=3D 16) + ctxt->dst.orig_val128 =3D ctxt->dst.val128; + else +#endif + ctxt->dst.orig_val64 =3D ctxt->dst.val64; =20 special_insn: =20 diff --git a/arch/x86/kvm/kvm_emulate.h b/arch/x86/kvm/kvm_emulate.h index 3e375af15c03..89911845233d 100644 --- a/arch/x86/kvm/kvm_emulate.h +++ b/arch/x86/kvm/kvm_emulate.h @@ -263,6 +263,9 @@ struct operand { union { unsigned long orig_val; u64 orig_val64; +#ifdef CONFIG_X86_64 + u128 orig_val128; +#endif }; union { unsigned long *reg; @@ -276,6 +279,9 @@ struct operand { union { unsigned long val; u64 val64; +#ifdef CONFIG_X86_64 + u128 val128; +#endif char valptr[sizeof(avx256_t)]; sse128_t vec_val; avx256_t vec_val2; diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index 79468ddfe473..60be16f05d79 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -5271,7 +5271,7 @@ static int emulator_cmpxchg_emulated(struct x86_emula= te_ctxt *ctxt, int r; =20 /* guests cmpxchg8b have to be emulated atomically */ - if (bytes > 8 || (bytes & (bytes - 1))) + if (bytes > 2 * sizeof(unsigned long) || (bytes & (bytes - 1))) goto emul_write; =20 gpa =3D kvm_mmu_gva_to_gpa_write(vcpu, addr, NULL); @@ -5311,6 +5311,11 @@ static int emulator_cmpxchg_emulated(struct x86_emul= ate_ctxt *ctxt, case 8: r =3D emulator_try_cmpxchg_user(u64, hva, old, new); break; +#ifdef CONFIG_X86_64 + case 16: + r =3D emulator_try_cmpxchg_user(u128, hva, old, new); + break; +#endif default: BUG(); } --=20 2.34.1