From nobody Mon Jun 29 23:24:05 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B3DF5C433FE for ; Tue, 1 Feb 2022 01:09:01 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231631AbiBABJA (ORCPT ); Mon, 31 Jan 2022 20:09:00 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43068 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231558AbiBABI5 (ORCPT ); Mon, 31 Jan 2022 20:08:57 -0500 Received: from mail-pl1-x64a.google.com (mail-pl1-x64a.google.com [IPv6:2607:f8b0:4864:20::64a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E23A1C061714 for ; Mon, 31 Jan 2022 17:08:57 -0800 (PST) Received: by mail-pl1-x64a.google.com with SMTP id a9-20020a170902710900b0014c8132e8b8so5816141pll.10 for ; Mon, 31 Jan 2022 17:08:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=reply-to:date:in-reply-to:message-id:mime-version:references :subject:from:to:cc; bh=+cGTUlncoKI8qOYBohRERc3y0sAWcqimeQBl8Ab9RTI=; b=e+kmca8jtulDGdN54BEXwgAQTzwVvQmOdxghFymavue4yUbH/JhbYwHShmg1mknAC6 TGWIGxmq/jFX2ptfeKHdKOfDRmgEQ5v4CL5i0MZStw7TyXz0CKikehanAZ6q80gNqdkb Ln5rWelPLY+WQKzPp2vu9zFOh71C0LYNRNranTXX76/y2XvUFmFfGmEf/iVuidZzaps1 CjSQiklb9jr+iJ0X5DmoXGzzY6x4s3D4XjdeLZJcY33tvhR/KBRMFbbVerGg86YRHTIL osbxNrDIlYlZl+1rA1VTn8E3fc1/sAuiWV221P8fbNWyWHM3ryGiW4oTC8AqpbxI3xiT Avcg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:reply-to:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=+cGTUlncoKI8qOYBohRERc3y0sAWcqimeQBl8Ab9RTI=; b=p6vyUmuVlFSQcXFOMrxSb4Iorjfs2BvfPQY77aw72BAovhExwpvASV3eV4QjLkIgQV D9XOByDNfKYIaBFqkOA97Wsd0x0wg0dqrzlttDY6AcmOhZ8/plmwFlnlOIs/9R+uFbpj VZXdOvezACfj+o0ZP1Y9ISyNbRgw4umHdd/uH0LAL3APTcGu9OPOkp/SmY2puYTXlEH2 Lg65CW4SXfZopeHsOpI7lu8z/toR3t63LoUK3pf5cDlmBbxD+o28NUSOTRtRl/WoBwgN r0p2/9yn0ZUGGRGSEdrHH+Edhvy8UMLsdh4pIlzYKEDFLOxXMPywzrywefXBzobpQMFK LckQ== X-Gm-Message-State: AOAM533W/B4H3mviIMgtA/R96d7YFFkNGRL8wLOyAM2Tpj/MDcREB604 ZE/n/JgI0nV11LsM7DYWdVKZH1EOa9w= X-Google-Smtp-Source: ABdhPJwCxixihN3pWoO3t60Sv6arP9CVtFwc3hJkZSrBPQ+/8LfdsTga0XZrersZPuDddjA//zRW0Ms6kc8= X-Received: from seanjc.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e5]) (user=seanjc job=sendgmr) by 2002:a17:902:c086:: with SMTP id j6mr22888340pld.101.1643677736737; Mon, 31 Jan 2022 17:08:56 -0800 (PST) Reply-To: Sean Christopherson Date: Tue, 1 Feb 2022 01:08:34 +0000 In-Reply-To: <20220201010838.1494405-1-seanjc@google.com> Message-Id: <20220201010838.1494405-2-seanjc@google.com> Mime-Version: 1.0 References: <20220201010838.1494405-1-seanjc@google.com> X-Mailer: git-send-email 2.35.0.rc2.247.g8bbb082509-goog Subject: [PATCH 1/5] Kconfig: Add option for asm goto w/ tied outputs to workaround clang-13 bug From: Sean Christopherson To: Paolo Bonzini , Nathan Chancellor , Nick Desaulniers Cc: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , kvm@vger.kernel.org, llvm@lists.linux.dev, linux-kernel@vger.kernel.org, Peter Zijlstra , syzbot+6cde2282daa792c49ab8@syzkaller.appspotmail.com Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add a config option to guard (future) usage of asm_volatile_goto() that includes "tied outputs", i.e. "+" constraints that specify both an input and output parameter. clang-13 has a bug[1] that causes compilation of such inline asm to fail, and KVM wants to use a "+m" constraint to implement a uaccess form of CMPXCHG[2]. E.g. the test code fails with :1:29: error: invalid operand in inline asm: '.long (${1:l}) - .' int foo(int *x) { asm goto (".long (%l[bar]) - .\n": "+m"(*x) ::: bar); r= eturn *x; bar: return 0; } ^ :1:29: error: unknown token in expression :1:9: note: instantiated into assembly here .long () - . ^ 2 errors generated. on clang-13, but passes on gcc (with appropriate asm goto support). The bug is fixed in clang-14, but won't be backported to clang-13 as the changes are too invasive/risky. [1] https://github.com/ClangBuiltLinux/linux/issues/1512 [2] https://lore.kernel.org/all/YfMruK8%2F1izZ2VHS@google.com Suggested-by: Nick Desaulniers Cc: stable@vger.kernel.org Signed-off-by: Sean Christopherson Tested-by: Tadeusz Struk --- init/Kconfig | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/init/Kconfig b/init/Kconfig index e9119bf54b1f..a206b21703be 100644 --- a/init/Kconfig +++ b/init/Kconfig @@ -77,6 +77,10 @@ config CC_HAS_ASM_GOTO_OUTPUT depends on CC_HAS_ASM_GOTO def_bool $(success,echo 'int foo(int x) { asm goto ("": "=3Dr"(x) ::: bar= ); return x; bar: return 0; }' | $(CC) -x c - -c -o /dev/null) =20 +config CC_HAS_ASM_GOTO_TIED_OUTPUT + depends on CC_HAS_ASM_GOTO_OUTPUT + def_bool $(success,echo 'int foo(int *x) { asm goto (".long (%l[bar]) - .= \n": "+m"(*x) ::: bar); return *x; bar: return 0; }' | $CC -x c - -c -o /de= v/null) + config TOOLS_SUPPORT_RELR def_bool $(success,env "CC=3D$(CC)" "LD=3D$(LD)" "NM=3D$(NM)" "OBJCOPY=3D= $(OBJCOPY)" $(srctree)/scripts/tools-support-relr.sh) =20 --=20 2.35.0.rc2.247.g8bbb082509-goog From nobody Mon Jun 29 23:24:05 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A1EB9C4332F for ; Tue, 1 Feb 2022 01:09:03 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231682AbiBABJC (ORCPT ); Mon, 31 Jan 2022 20:09:02 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43078 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231592AbiBABI7 (ORCPT ); Mon, 31 Jan 2022 20:08:59 -0500 Received: from mail-pl1-x649.google.com (mail-pl1-x649.google.com [IPv6:2607:f8b0:4864:20::649]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2048BC06173B for ; Mon, 31 Jan 2022 17:08:59 -0800 (PST) Received: by mail-pl1-x649.google.com with SMTP id p16-20020a170902a41000b0014992c5d56bso6108719plq.19 for ; Mon, 31 Jan 2022 17:08:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=reply-to:date:in-reply-to:message-id:mime-version:references :subject:from:to:cc; bh=o19M2/2rEEC9XlqyRHCtm92cgxOv/iqljRv4RIXkdoQ=; b=Thb1uNHwmhRE1IUrtC3LxdwFZMcqizXJg2fsLS1x19IBdPVMlthwhCef0wccwNj8RT PNoHxvbb16l/5xcIUSN7yGlmUmdALNHYWuNGqkIcaFK+HthmCxHEiuYnGxjhqz6D5c6/ flIeR+UcZydZuaTx4+cL76NU6enAqzQaOyCe09RiRefHF0ULIM3xhk4vuj24pbWJZhc7 jzBFX2Kqdq1t52wYob2iPGxcb7lBdF8wKst0SsSwKZ2Ffy5XcVZV9PoO63qaTz4Ppuvb XsLtYR54PRtVZtaswjvWz9EpHUEw4c1aK+n3bbe56twI0/LLbBssLOBvRUwJkuLYnnCr UbXA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:reply-to:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=o19M2/2rEEC9XlqyRHCtm92cgxOv/iqljRv4RIXkdoQ=; b=LNGlpCn3RxvnD8E6J9KApsLlrSJV5KHjZuxlJ9SO50sZnTa3pGJ0v8N7r1Vhdpih5r usCYL/4TTcaUO6BnpGsn4FG5lwrdZItdO29RaYcY+FJhkKQuv3HXd+eE9hjMGt+M3ooe 0YO8VaN3Lzg4BYf56fLWXjXfd+lf1cKpUF1Mbi93v+jv/G7SdvknJH4uMIA6pFg7BR25 JZowC9SsarGsqRr3PyAbs7Ha2BWmAAJtjZrXFohLQK/wIBCoGiRMMWDKWvy8kB6YJ5bL a2+6Z68cq3doH5y/npMtveuSxe6wjvXpn4qqo9OqYbH5eoPi6h8iyTfo8A5aUSLvrUT/ gd5Q== X-Gm-Message-State: AOAM533mVJ2DomZlJJDTxFViTKUl+goR5F8f3/RXBOgFtq1Jtvvz/PfN GqsxQ53Qa3E3hO/QQTBwoCEU1lDReGI= X-Google-Smtp-Source: ABdhPJypx2tPaOAnNKGjQcJspqm3ZE/EERH60WoXE1+w/w/BrbrsLS6tT4VIx/uoDkZeVXXDHO8aA4fDgyI= X-Received: from seanjc.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e5]) (user=seanjc job=sendgmr) by 2002:a65:6182:: with SMTP id c2mr19079339pgv.95.1643677738626; Mon, 31 Jan 2022 17:08:58 -0800 (PST) Reply-To: Sean Christopherson Date: Tue, 1 Feb 2022 01:08:35 +0000 In-Reply-To: <20220201010838.1494405-1-seanjc@google.com> Message-Id: <20220201010838.1494405-3-seanjc@google.com> Mime-Version: 1.0 References: <20220201010838.1494405-1-seanjc@google.com> X-Mailer: git-send-email 2.35.0.rc2.247.g8bbb082509-goog Subject: [PATCH 2/5] x86/uaccess: Implement macros for CMPXCHG on user addresses From: Sean Christopherson To: Paolo Bonzini , Nathan Chancellor , Nick Desaulniers Cc: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , kvm@vger.kernel.org, llvm@lists.linux.dev, linux-kernel@vger.kernel.org, Peter Zijlstra , syzbot+6cde2282daa792c49ab8@syzkaller.appspotmail.com Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Peter Zijlstra Add support for CMPXCHG loops on userspace addresses. Provide both an "unsafe" version for tight loops that do their own uaccess begin/end, as well as a "safe" version for use cases where the CMPXCHG is not buried in a loop, e.g. KVM will resume the guest instead of looping when emulation of a guest atomic accesses fails the CMPXCHG. Provide 8-byte versions for 32-bit kernels so that KVM can do CMPXCHG on guest PAE PTEs, which are accessed via userspace addresses. Guard the asm_volatile_goto() variation with CC_HAS_ASM_GOTO_TIED_OUTPUT, the "+m" constraint fails on some compilers that otherwise support CC_HAS_ASM_GOTO_OUTPUT. Cc: stable@vger.kernel.org Signed-off-by: Peter Zijlstra (Intel) Co-developed-by: Sean Christopherson Signed-off-by: Sean Christopherson Tested-by: Tadeusz Struk --- arch/x86/include/asm/uaccess.h | 131 +++++++++++++++++++++++++++++++++ 1 file changed, 131 insertions(+) diff --git a/arch/x86/include/asm/uaccess.h b/arch/x86/include/asm/uaccess.h index ac96f9b2d64b..423bfcc1ec4b 100644 --- a/arch/x86/include/asm/uaccess.h +++ b/arch/x86/include/asm/uaccess.h @@ -409,6 +409,98 @@ do { \ =20 #endif // CONFIG_CC_HAS_ASM_GOTO_OUTPUT =20 +#ifdef CONFIG_CC_HAS_ASM_GOTO_TIED_OUTPUT +#define __try_cmpxchg_user_asm(itype, ltype, _ptr, _pold, _new, label) ({ \ + bool success; \ + __typeof__(_ptr) _old =3D (__typeof__(_ptr))(_pold); \ + __typeof__(*(_ptr)) __old =3D *_old; \ + __typeof__(*(_ptr)) __new =3D (_new); \ + asm_volatile_goto("\n" \ + "1: " LOCK_PREFIX "cmpxchg"itype" %[new], %[ptr]\n"\ + _ASM_EXTABLE_UA(1b, %l[label]) \ + : CC_OUT(z) (success), \ + [ptr] "+m" (*_ptr), \ + [old] "+a" (__old) \ + : [new] ltype (__new) \ + : "memory" \ + : label); \ + if (unlikely(!success)) \ + *_old =3D __old; \ + likely(success); }) + +#ifdef CONFIG_X86_32 +#define __try_cmpxchg64_user_asm(_ptr, _pold, _new, label) ({ \ + bool success; \ + __typeof__(_ptr) _old =3D (__typeof__(_ptr))(_pold); \ + __typeof__(*(_ptr)) __old =3D *_old; \ + __typeof__(*(_ptr)) __new =3D (_new); \ + asm_volatile_goto("\n" \ + "1: " LOCK_PREFIX "cmpxchg8b %[ptr]\n" \ + _ASM_EXTABLE_UA(1b, %l[label]) \ + : CC_OUT(z) (success), \ + "+A" (__old), \ + [ptr] "+m" (*_ptr) \ + : "b" ((u32)__new), \ + "c" ((u32)((u64)__new >> 32)) \ + : "memory" \ + : label); \ + if (unlikely(!success)) \ + *_old =3D __old; \ + likely(success); }) +#endif // CONFIG_X86_32 +#else // !CONFIG_CC_HAS_ASM_GOTO_TIED_OUTPUT +#define __try_cmpxchg_user_asm(itype, ltype, _ptr, _pold, _new, label) ({ \ + int __err =3D 0; \ + bool success; \ + __typeof__(_ptr) _old =3D (__typeof__(_ptr))(_pold); \ + __typeof__(*(_ptr)) __old =3D *_old; \ + __typeof__(*(_ptr)) __new =3D (_new); \ + asm volatile("\n" \ + "1: " LOCK_PREFIX "cmpxchg"itype" %[new], %[ptr]\n"\ + CC_SET(z) \ + "2:\n" \ + _ASM_EXTABLE_TYPE_REG(1b, 2b, EX_TYPE_EFAULT_REG, \ + %[errout]) \ + : CC_OUT(z) (success), \ + [errout] "+r" (__err), \ + [ptr] "+m" (*_ptr), \ + [old] "+a" (__old) \ + : [new] ltype (__new) \ + : "memory", "cc"); \ + if (unlikely(__err)) \ + goto label; \ + if (unlikely(!success)) \ + *_old =3D __old; \ + likely(success); }) + +#ifdef CONFIG_X86_32 +#define __try_cmpxchg64_user_asm(_ptr, _pold, _new, label) ({ \ + int __err =3D 0; \ + bool success; \ + __typeof__(_ptr) _old =3D (__typeof__(_ptr))(_pold); \ + __typeof__(*(_ptr)) __old =3D *_old; \ + __typeof__(*(_ptr)) __new =3D (_new); \ + asm volatile("\n" \ + "1: " LOCK_PREFIX "cmpxchg8b %[ptr]\n" \ + CC_SET(z) \ + "2:\n" \ + _ASM_EXTABLE_TYPE_REG(1b, 2b, EX_TYPE_EFAULT_REG, \ + %[errout]) \ + : CC_OUT(z) (success), \ + [errout] "+r" (__err), \ + "+A" (__old), \ + [ptr] "+m" (*_ptr) \ + : "b" ((u32)__new), \ + "c" ((u32)((u64)__new >> 32)) \ + : "memory", "cc"); \ + if (unlikely(__err)) \ + goto label; \ + if (unlikely(!success)) \ + *_old =3D __old; \ + likely(success); }) +#endif // CONFIG_X86_32 +#endif // CONFIG_CC_HAS_ASM_GOTO_TIED_OUTPUT + /* FIXME: this hack is definitely wrong -AK */ struct __large_struct { unsigned long buf[100]; }; #define __m(x) (*(struct __large_struct __user *)(x)) @@ -501,6 +593,45 @@ do { \ } while (0) #endif // CONFIG_CC_HAS_ASM_GOTO_OUTPUT =20 +extern void __try_cmpxchg_user_wrong_size(void); + +#ifndef CONFIG_X86_32 +#define __try_cmpxchg64_user_asm(_ptr, _oldp, _nval, _label) \ + __try_cmpxchg_user_asm("q", "r", (_ptr), (_oldp), (_nval), _label) +#endif + +#define unsafe_try_cmpxchg_user(_ptr, _oldp, _nval, _label) ({ \ + bool __ret; \ + switch (sizeof(*(_ptr))) { \ + case 1: __ret =3D __try_cmpxchg_user_asm("b", "q", \ + (_ptr), (_oldp), \ + (_nval), _label); \ + break; \ + case 2: __ret =3D __try_cmpxchg_user_asm("w", "r", \ + (_ptr), (_oldp), \ + (_nval), _label); \ + break; \ + case 4: __ret =3D __try_cmpxchg_user_asm("l", "r", \ + (_ptr), (_oldp), \ + (_nval), _label); \ + break; \ + case 8: __ret =3D __try_cmpxchg64_user_asm((_ptr), (_oldp), \ + (_nval), _label); \ + break; \ + default: __try_cmpxchg_user_wrong_size(); \ + } \ + __ret; }) + +/* "Returns" 0 on success, 1 on failure, -EFAULT if the access faults. */ +#define __try_cmpxchg_user(_ptr, _oldp, _nval, _label) ({ \ + int __ret =3D -EFAULT; \ + __uaccess_begin_nospec(); \ + __ret =3D !unsafe_try_cmpxchg_user(_ptr, _oldp, _nval, _label); \ +_label: \ + __uaccess_end(); \ + __ret; \ + }) + /* * We want the unsafe accessors to always be inlined and use * the error labels - thus the macro games. --=20 2.35.0.rc2.247.g8bbb082509-goog From nobody Mon Jun 29 23:24:05 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 980F8C433EF for ; Tue, 1 Feb 2022 01:09:05 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231669AbiBABJE (ORCPT ); Mon, 31 Jan 2022 20:09:04 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43088 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231633AbiBABJA (ORCPT ); Mon, 31 Jan 2022 20:09:00 -0500 Received: from mail-pj1-x1049.google.com (mail-pj1-x1049.google.com [IPv6:2607:f8b0:4864:20::1049]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AD245C06173B for ; Mon, 31 Jan 2022 17:09:00 -0800 (PST) Received: by mail-pj1-x1049.google.com with SMTP id 62-20020a17090a09c400b001b80b0742b0so1223724pjo.8 for ; Mon, 31 Jan 2022 17:09:00 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=reply-to:date:in-reply-to:message-id:mime-version:references :subject:from:to:cc; bh=4kaP3FlUpQFdehq8pWYdxVNwRbxH7i/N3SsKaNgWUGc=; b=h3EL5uywFEwrVZHjm51EZtEEJMlwQaV1z88Z7DeuzixINWb3XSD5/WyLhozYxe4LMx sct2ZuwT0YBcjticdqdu3PPzXlEma6RygOCKDK1odDKv0cVT7Kis7SLSnACsVuUx6mcc wMBnqr1/FPewELU4IkhgvXGP6E4JeniG02IJsQxq4HBwo4rLuxDQBwTqNvODxXDo4P6G yMiN3bsXYAj6ENO8uK9KZ5Q78slE35XhaFJN7NofJ1MP/uY8Q2tfFFOck9Ry8ukeDDIz EvY53YPpv4Es1Xf+/BzOl7qEEv6kREOkqoaY8CZ5m0M6ocDkqkBTe2+GFe4lXgtjRjHf Sesw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:reply-to:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=4kaP3FlUpQFdehq8pWYdxVNwRbxH7i/N3SsKaNgWUGc=; b=cmOFgIRr6jUevU32OCph26VbC+8f9Mp00tin2RK7ZS1bGKgyZAhoN0R9hYDEsb4PUI lPCH6+iJTsE9Zuvf+KKthqmNTYvpRW5w4Yom3HgYCM+uT9IILOgWfn0is5EG3H8mHbw2 EEfM8ZpO0MKSekZ8ATqOTsYHilbDFBnW+W2oQvm5ZC0oYquiabB0binblIZ3kpQfsqj5 2Xsx8GrNAlNj3BHjwLyPUCXtZ5xGB86TFY5Fh2WPiBhknFdv8E/F7/VN32SpmshlBmol WVqFQgMtBOskKi38zfyVWxhqL+TgQDal5U5z1d1Xg8Kjph0kU3Ui0AhBOtpyvMe4Ijtz C0cA== X-Gm-Message-State: AOAM532A155MwacpghXlVwZJKVWWh463oxFEOko7/0zGOiBwMgCU5kTL wcYl3SbLfJXWbIJ0aGyAIKGf8vmcR4M= X-Google-Smtp-Source: ABdhPJylCV3aOjYEjhVRX1NvX5dqLtIPXCTzy0xsVbRBa2f5cIttpJ71kIFQFrbLJbe3ibfEaVbQypjJEnU= X-Received: from seanjc.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e5]) (user=seanjc job=sendgmr) by 2002:a17:902:c24d:: with SMTP id 13mr20607003plg.24.1643677740209; Mon, 31 Jan 2022 17:09:00 -0800 (PST) Reply-To: Sean Christopherson Date: Tue, 1 Feb 2022 01:08:36 +0000 In-Reply-To: <20220201010838.1494405-1-seanjc@google.com> Message-Id: <20220201010838.1494405-4-seanjc@google.com> Mime-Version: 1.0 References: <20220201010838.1494405-1-seanjc@google.com> X-Mailer: git-send-email 2.35.0.rc2.247.g8bbb082509-goog Subject: [PATCH 3/5] KVM: x86: Use __try_cmpxchg_user() to update guest PTE A/D bits From: Sean Christopherson To: Paolo Bonzini , Nathan Chancellor , Nick Desaulniers Cc: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , kvm@vger.kernel.org, llvm@lists.linux.dev, linux-kernel@vger.kernel.org, Peter Zijlstra , syzbot+6cde2282daa792c49ab8@syzkaller.appspotmail.com Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Use the recently introduced __try_cmpxchg_user() to update guest PTE A/D bits instead of mapping the PTE into kernel address space. The VM_PFNMAP path is broken as it assumes that vm_pgoff is the base pfn of the mapped VMA range, which is conceptually wrong as vm_pgoff is the offset relative to the file and has nothing to do with the pfn. The horrific hack worked for the original use case (backing guest memory with /dev/mem), but leads to accessing "random" pfns for pretty much any other VM_PFNMAP case. Fixes: bd53cb35a3e9 ("X86/KVM: Handle PFNs outside of kernel reach when tou= ching GPTEs") Debugged-by: Tadeusz Struk Reported-by: syzbot+6cde2282daa792c49ab8@syzkaller.appspotmail.com Cc: stable@vger.kernel.org Signed-off-by: Sean Christopherson Tested-by: Tadeusz Struk --- arch/x86/kvm/mmu/paging_tmpl.h | 45 +--------------------------------- 1 file changed, 1 insertion(+), 44 deletions(-) diff --git a/arch/x86/kvm/mmu/paging_tmpl.h b/arch/x86/kvm/mmu/paging_tmpl.h index 5b5bdac97c7b..551de15f342f 100644 --- a/arch/x86/kvm/mmu/paging_tmpl.h +++ b/arch/x86/kvm/mmu/paging_tmpl.h @@ -143,49 +143,6 @@ static bool FNAME(is_rsvd_bits_set)(struct kvm_mmu *mm= u, u64 gpte, int level) FNAME(is_bad_mt_xwr)(&mmu->guest_rsvd_check, gpte); } =20 -static int FNAME(cmpxchg_gpte)(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu, - pt_element_t __user *ptep_user, unsigned index, - pt_element_t orig_pte, pt_element_t new_pte) -{ - int npages; - pt_element_t ret; - pt_element_t *table; - struct page *page; - - npages =3D get_user_pages_fast((unsigned long)ptep_user, 1, FOLL_WRITE, &= page); - if (likely(npages =3D=3D 1)) { - table =3D kmap_atomic(page); - ret =3D CMPXCHG(&table[index], orig_pte, new_pte); - kunmap_atomic(table); - - kvm_release_page_dirty(page); - } else { - struct vm_area_struct *vma; - unsigned long vaddr =3D (unsigned long)ptep_user & PAGE_MASK; - unsigned long pfn; - unsigned long paddr; - - mmap_read_lock(current->mm); - vma =3D find_vma_intersection(current->mm, vaddr, vaddr + PAGE_SIZE); - if (!vma || !(vma->vm_flags & VM_PFNMAP)) { - mmap_read_unlock(current->mm); - return -EFAULT; - } - pfn =3D ((vaddr - vma->vm_start) >> PAGE_SHIFT) + vma->vm_pgoff; - paddr =3D pfn << PAGE_SHIFT; - table =3D memremap(paddr, PAGE_SIZE, MEMREMAP_WB); - if (!table) { - mmap_read_unlock(current->mm); - return -EFAULT; - } - ret =3D CMPXCHG(&table[index], orig_pte, new_pte); - memunmap(table); - mmap_read_unlock(current->mm); - } - - return (ret !=3D orig_pte); -} - static bool FNAME(prefetch_invalid_gpte)(struct kvm_vcpu *vcpu, struct kvm_mmu_page *sp, u64 *spte, u64 gpte) @@ -284,7 +241,7 @@ static int FNAME(update_accessed_dirty_bits)(struct kvm= _vcpu *vcpu, if (unlikely(!walker->pte_writable[level - 1])) continue; =20 - ret =3D FNAME(cmpxchg_gpte)(vcpu, mmu, ptep_user, index, orig_pte, pte); + ret =3D __try_cmpxchg_user(ptep_user, &orig_pte, pte, fault); if (ret) return ret; =20 --=20 2.35.0.rc2.247.g8bbb082509-goog From nobody Mon Jun 29 23:24:05 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0A59FC4332F for ; Tue, 1 Feb 2022 01:09:11 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231834AbiBABJK (ORCPT ); Mon, 31 Jan 2022 20:09:10 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43098 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231680AbiBABJC (ORCPT ); Mon, 31 Jan 2022 20:09:02 -0500 Received: from mail-pg1-x549.google.com (mail-pg1-x549.google.com [IPv6:2607:f8b0:4864:20::549]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5C951C061714 for ; Mon, 31 Jan 2022 17:09:02 -0800 (PST) Received: by mail-pg1-x549.google.com with SMTP id u133-20020a63798b000000b0034c0630b044so9490841pgc.3 for ; Mon, 31 Jan 2022 17:09:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=reply-to:date:in-reply-to:message-id:mime-version:references :subject:from:to:cc; bh=LcukT3aVWQNuFlO5tp4KbgwXT1VI3YuMTdLXdDHGgw4=; b=L5WVkpDBAaqAGX7sBQ9aM3BYEJZZItsnHILJyVw9anPCW9GBY6gMrJI/qsNxUDKi/b tVx/WXxQ8dpcYEP1PyI5F3A0LkrX7NlEchmWKjxbmmhItMO1VPhqL3SXKPc2GKJT7gHF +MueL52xjMxeZDZ79vz8mCrvikzwKG6i6rncgx3/3NWKPEWXvRLJd3F8+AI8TD093WC3 fy1eBEB50+5Spbs0Z7cyviGeT2+7SixtFXWSLNt7/xq8TG68bsmbOSjVWmBBXVRLRpNF uqsM9E1016Ul0cWQQ93GlHN5rp3UH5pyOer1I+J5viYD6/BjVDbSiG6usGovwNBRvfZJ zkBg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:reply-to:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=LcukT3aVWQNuFlO5tp4KbgwXT1VI3YuMTdLXdDHGgw4=; b=OQAowKsOPBDpZLufF9+3dMoH5p6PyTDxCq1qzsR4pEqm9xz8++iNr7PJMj62cjm6u7 okhiyX7mdTv9Pq9GQthugvhNDocANaryE/G/XkPNWAdNFiDgL7bMUXaM4RjkhvHr+GXA BGsGo3hEZ2l2ahC9LjLTU6uftOUZp4erThe23DRo593Epk6aRHd0NIVzuXXK5vm7M0eX eq4aHVksOhrMnl73oBTNHTJwH5BrCJ7OhkZaG2supdmt6V7H8++hgFejfEwgOCKdr+Vo 3La+J+UOlm7TO/P5erFGrcuqMtd6b8RGtzD7Sv4hx27OU+x7pvjKJ2Zs2fXuz+q6QXQ+ yvZg== X-Gm-Message-State: AOAM531vGJ4dr2XQj8t5c5YJFs0pqvc2iJBXc0FcctUlDQ/7LBfoS6LQ QeNfs0xm8XieczfE4ZaMTfBa/9TZlhA= X-Google-Smtp-Source: ABdhPJxukxPNZ3VcpmMc2eg8v3Edq+8quTgFxnPmvAUmlOB6In41l4WjFVxqrNN/Oh2K/QwpEJn1488nuA8= X-Received: from seanjc.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e5]) (user=seanjc job=sendgmr) by 2002:a17:90a:f198:: with SMTP id bv24mr25807269pjb.32.1643677741876; Mon, 31 Jan 2022 17:09:01 -0800 (PST) Reply-To: Sean Christopherson Date: Tue, 1 Feb 2022 01:08:37 +0000 In-Reply-To: <20220201010838.1494405-1-seanjc@google.com> Message-Id: <20220201010838.1494405-5-seanjc@google.com> Mime-Version: 1.0 References: <20220201010838.1494405-1-seanjc@google.com> X-Mailer: git-send-email 2.35.0.rc2.247.g8bbb082509-goog Subject: [PATCH 4/5] KVM: x86: Use __try_cmpxchg_user() to emulate atomic accesses From: Sean Christopherson To: Paolo Bonzini , Nathan Chancellor , Nick Desaulniers Cc: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , kvm@vger.kernel.org, llvm@lists.linux.dev, linux-kernel@vger.kernel.org, Peter Zijlstra , syzbot+6cde2282daa792c49ab8@syzkaller.appspotmail.com Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Use the recently introduce __try_cmpxchg_user() to emulate atomic guest accesses via the associated userspace address instead of mapping the backing pfn into kernel address space. Using kvm_vcpu_map() is unsafe as it does not coordinate with KVM's mmu_notifier to ensure the hva=3D>pfn translation isn't changed/unmapped in the memremap() path, i.e. when there's no struct page and thus no elevated refcount. Fixes: 42e35f8072c3 ("KVM/X86: Use kvm_vcpu_map in emulator_cmpxchg_emulate= d") Cc: stable@vger.kernel.org Signed-off-by: Sean Christopherson Tested-by: Tadeusz Struk --- arch/x86/kvm/x86.c | 35 ++++++++++++++--------------------- 1 file changed, 14 insertions(+), 21 deletions(-) diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index 74b53a16f38a..37064d565bbc 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -7155,15 +7155,8 @@ static int emulator_write_emulated(struct x86_emulat= e_ctxt *ctxt, exception, &write_emultor); } =20 -#define CMPXCHG_TYPE(t, ptr, old, new) \ - (cmpxchg((t *)(ptr), *(t *)(old), *(t *)(new)) =3D=3D *(t *)(old)) - -#ifdef CONFIG_X86_64 -# define CMPXCHG64(ptr, old, new) CMPXCHG_TYPE(u64, ptr, old, new) -#else -# define CMPXCHG64(ptr, old, new) \ - (cmpxchg64((u64 *)(ptr), *(u64 *)(old), *(u64 *)(new)) =3D=3D *(u64 *)(ol= d)) -#endif +#define emulator_try_cmpxchg_user(t, ptr, old, new) \ + (__try_cmpxchg_user((t *)(ptr), (t *)(old), *(t *)(new), efault ## t)) =20 static int emulator_cmpxchg_emulated(struct x86_emulate_ctxt *ctxt, unsigned long addr, @@ -7172,12 +7165,11 @@ static int emulator_cmpxchg_emulated(struct x86_emu= late_ctxt *ctxt, unsigned int bytes, struct x86_exception *exception) { - struct kvm_host_map map; struct kvm_vcpu *vcpu =3D emul_to_vcpu(ctxt); u64 page_line_mask; + unsigned long hva; gpa_t gpa; - char *kaddr; - bool exchanged; + int r; =20 /* guests cmpxchg8b have to be emulated atomically */ if (bytes > 8 || (bytes & (bytes - 1))) @@ -7201,31 +7193,32 @@ static int emulator_cmpxchg_emulated(struct x86_emu= late_ctxt *ctxt, if (((gpa + bytes - 1) & page_line_mask) !=3D (gpa & page_line_mask)) goto emul_write; =20 - if (kvm_vcpu_map(vcpu, gpa_to_gfn(gpa), &map)) + hva =3D kvm_vcpu_gfn_to_hva(vcpu, gpa_to_gfn(gpa)); + if (kvm_is_error_hva(addr)) goto emul_write; =20 - kaddr =3D map.hva + offset_in_page(gpa); + hva +=3D offset_in_page(gpa); =20 switch (bytes) { case 1: - exchanged =3D CMPXCHG_TYPE(u8, kaddr, old, new); + r =3D emulator_try_cmpxchg_user(u8, hva, old, new); break; case 2: - exchanged =3D CMPXCHG_TYPE(u16, kaddr, old, new); + r =3D emulator_try_cmpxchg_user(u16, hva, old, new); break; case 4: - exchanged =3D CMPXCHG_TYPE(u32, kaddr, old, new); + r =3D emulator_try_cmpxchg_user(u32, hva, old, new); break; case 8: - exchanged =3D CMPXCHG64(kaddr, old, new); + r =3D emulator_try_cmpxchg_user(u64, hva, old, new); break; default: BUG(); } =20 - kvm_vcpu_unmap(vcpu, &map, true); - - if (!exchanged) + if (r < 0) + goto emul_write; + if (r) return X86EMUL_CMPXCHG_FAILED; =20 kvm_page_track_write(vcpu, gpa, new, bytes); --=20 2.35.0.rc2.247.g8bbb082509-goog From nobody Mon Jun 29 23:24:05 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9F6C9C433F5 for ; Tue, 1 Feb 2022 01:09:09 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231801AbiBABJI (ORCPT ); Mon, 31 Jan 2022 20:09:08 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43112 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231699AbiBABJE (ORCPT ); Mon, 31 Jan 2022 20:09:04 -0500 Received: from mail-pf1-x449.google.com (mail-pf1-x449.google.com [IPv6:2607:f8b0:4864:20::449]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EF0ABC06173E for ; Mon, 31 Jan 2022 17:09:03 -0800 (PST) Received: by mail-pf1-x449.google.com with SMTP id z20-20020aa791d4000000b004bd024eaf19so8234401pfa.16 for ; Mon, 31 Jan 2022 17:09:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=reply-to:date:in-reply-to:message-id:mime-version:references :subject:from:to:cc; bh=bL3FT8k/Xb3BcGI4WIfdGnBaIJMFJqmCvBuj1gJGH44=; b=qfjP1xUS1KlVS/IMykXefKvK1qQTncFnnytcKJMuNz7cZ6izT1HFdUafvswOgJug6w fUu2gRE/sPO/gMGaQJHy1cu/gikzlTFd+dwYsScIQ2aSEr19oNQfPZGlCSgrgT+A/ptb /BmWw9Q/5FLUgVtaJXFmF+ufSdVc0dVArOlkXmwriEDJ5vmsdUA1krlC7d3M4dgj/hAk ZH38ahZRKwAFuq+7rfQkwCiebPXcQlBJDlblbvkGBTV3eebX1oiryLzUoQsHW5mONR+9 1uIdR6azhY2x3J/pkEG8jDjFnokQdpdQEl+iLKEku7fanF+2DkTtHnGCKT2OvwN8+DmO kjOA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:reply-to:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=bL3FT8k/Xb3BcGI4WIfdGnBaIJMFJqmCvBuj1gJGH44=; b=4yPGtJCK2JUnnkZINpIfBUBQdtQoSfQG+AhiszIAfVedUaglBVzUyVsxJrfUEhOVBY D2aWfW5iCjkd08nh6owij3pVUQGLeJf0BzNadRD9h+e2BOtpOJ0PvDqrVj41IYGNr518 EVgC05E86pDE4bwrz/H9WrDbUIDyRf5gu+4M1L2Ib0gdoaUFlZxAj1+a+4YBQTNDaVd3 0idTBnTLoLbCJI0QhELijL+k9mNAATBiGvLuX0zISY8Ev1MhCM+TifZGUtdWApGslBjJ In4POO6sm4lFwEO9zax4va9/fu1npP359X8NLr3A0LObiei4nQ3Cjygzab1n3nvc58vZ +I4w== X-Gm-Message-State: AOAM532HBiTq5mpZ0JYATxy7uzU6spC3HMebi52/3yUIPDLg2xP/r255 ztb6MVFp6BebXRzl7qpP2bz2oYLkpq0= X-Google-Smtp-Source: ABdhPJyr1xQ8ILXqOSdU52v8l3fst3RJPIJ4yAP+ddUL0kFfLBBG3IGhmHWreArnTTL96e8JdujXax4R5iE= X-Received: from seanjc.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e5]) (user=seanjc job=sendgmr) by 2002:a17:902:7784:: with SMTP id o4mr23247435pll.173.1643677743484; Mon, 31 Jan 2022 17:09:03 -0800 (PST) Reply-To: Sean Christopherson Date: Tue, 1 Feb 2022 01:08:38 +0000 In-Reply-To: <20220201010838.1494405-1-seanjc@google.com> Message-Id: <20220201010838.1494405-6-seanjc@google.com> Mime-Version: 1.0 References: <20220201010838.1494405-1-seanjc@google.com> X-Mailer: git-send-email 2.35.0.rc2.247.g8bbb082509-goog Subject: [PATCH 5/5] KVM: x86: Bail to userspace if emulation of atomic user access faults From: Sean Christopherson To: Paolo Bonzini , Nathan Chancellor , Nick Desaulniers Cc: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , kvm@vger.kernel.org, llvm@lists.linux.dev, linux-kernel@vger.kernel.org, Peter Zijlstra , syzbot+6cde2282daa792c49ab8@syzkaller.appspotmail.com Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Exit to userspace when emulating an atomic guest access if the CMPXCHG on the userspace address faults. Emulating the access as a write and thus likely treating it as emulated MMIO is wrong, as KVM has already confirmed there is a valid, writable memslot. Signed-off-by: Sean Christopherson Tested-by: Tadeusz Struk --- arch/x86/kvm/x86.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index 37064d565bbc..66c5410dd4c3 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -7217,7 +7217,7 @@ static int emulator_cmpxchg_emulated(struct x86_emula= te_ctxt *ctxt, } =20 if (r < 0) - goto emul_write; + return X86EMUL_UNHANDLEABLE; if (r) return X86EMUL_CMPXCHG_FAILED; =20 --=20 2.35.0.rc2.247.g8bbb082509-goog