From nobody Mon Jun 29 22:17:28 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 83840C433EF for ; Wed, 2 Feb 2022 00:50:03 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S243549AbiBBAuC (ORCPT ); Tue, 1 Feb 2022 19:50:02 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55842 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238381AbiBBAuA (ORCPT ); Tue, 1 Feb 2022 19:50:00 -0500 Received: from mail-pf1-x44a.google.com (mail-pf1-x44a.google.com [IPv6:2607:f8b0:4864:20::44a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B5BC2C06173B for ; Tue, 1 Feb 2022 16:50:00 -0800 (PST) Received: by mail-pf1-x44a.google.com with SMTP id 16-20020a621710000000b004c81f7ea48aso9899677pfx.17 for ; Tue, 01 Feb 2022 16:50:00 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=reply-to:date:in-reply-to:message-id:mime-version:references :subject:from:to:cc; bh=4ANd1NUt8XUnXAXLJfyLyXEO2qzCf9R+Cd1MDd6nvE0=; b=cV8caTREkXIyZNXOZWEuBAuUILEs8niIQmAdqOwqy9Sp8eEqynTK7+QQfLP6gyuS4b lqJ+DmA3d4N9rmH4qlFiXq9ZuJxlNYcx2kzL8EMQN7wwCcTuYDNhXC0oMfxv6WRZ/erF Lal5TrRX8YxKFQeXudAvv8ctHBsDTv+un7qkUeHJGEXqf27hSEW0kx1YhXzvy5C/MLDV tAyqnNXXEtJt2xM5K/gjXytZFv9025spz7gL0TxZW3v7ZP5RADc78YtJvh84rAFpIYlT 49wpkearPsmuurMmxnW0FX6oQKa/Sm2/F2QHNrOKGvlpFcTNQh7ZSMt3LbVk8EMMpIaE lF+w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:reply-to:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=4ANd1NUt8XUnXAXLJfyLyXEO2qzCf9R+Cd1MDd6nvE0=; b=P5/erkALxSC0VXUDae4Vj3MI/J0OT8uV6qXrl6ezIMVTUyImNwmX7xO2nvtNpOTVr/ HV0ijlHOSJMyn31EDCOAJDu4ryMOHCvwD1lbKUHsid8UAMwf72/cDmILk3CgdtWuGGUa p1UZMrZeM8booapkaGbQjs5FmB2pU5yXD01+xT8uWMzcP/RQPefE9fRuzxMHFjSCw4eE N7Rka9xkF13fIR5kjtrvkdB9NozpURanV5FME1RYoPolczKrEqGkGUE0509c6o//Z4c8 +R06ebvt6HB6B0yYKoDVpwJpnZ62M6pkF5RCDWzrK2ZIcPPg8WaEjwqGWdgX9a8GzXlu udZA== X-Gm-Message-State: AOAM532qBSsKBjeyOUkGzJt96LRL37OUeHezall5bzrAy5jpNa8TKaET 8kI6XzK+1NmX0jUAg1SCbHJJPzdHKYA= X-Google-Smtp-Source: ABdhPJzT4Dxs+EfukbNZysiCeyomNEphZ+rxnsI/uMF3ouGE9elNSk58GrrmSDabXQRLq7ZfIXwPvIRUsYI= X-Received: from seanjc.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e5]) (user=seanjc job=sendgmr) by 2002:a05:6a00:1989:: with SMTP id d9mr27233932pfl.14.1643763000201; Tue, 01 Feb 2022 16:50:00 -0800 (PST) Reply-To: Sean Christopherson Date: Wed, 2 Feb 2022 00:49:41 +0000 In-Reply-To: <20220202004945.2540433-1-seanjc@google.com> Message-Id: <20220202004945.2540433-2-seanjc@google.com> Mime-Version: 1.0 References: <20220202004945.2540433-1-seanjc@google.com> X-Mailer: git-send-email 2.35.0.rc2.247.g8bbb082509-goog Subject: [PATCH v2 1/5] Kconfig: Add option for asm goto w/ tied outputs to workaround clang-13 bug From: Sean Christopherson To: Paolo Bonzini , Nathan Chancellor , Nick Desaulniers Cc: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , kvm@vger.kernel.org, llvm@lists.linux.dev, linux-kernel@vger.kernel.org, Peter Zijlstra , Tadeusz Struk , syzbot+6cde2282daa792c49ab8@syzkaller.appspotmail.com Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add a config option to guard (future) usage of asm_volatile_goto() that includes "tied outputs", i.e. "+" constraints that specify both an input and output parameter. clang-13 has a bug[1] that causes compilation of such inline asm to fail, and KVM wants to use a "+m" constraint to implement a uaccess form of CMPXCHG[2]. E.g. the test code fails with :1:29: error: invalid operand in inline asm: '.long (${1:l}) - .' int foo(int *x) { asm goto (".long (%l[bar]) - .\n": "+m"(*x) ::: bar); r= eturn *x; bar: return 0; } ^ :1:29: error: unknown token in expression :1:9: note: instantiated into assembly here .long () - . ^ 2 errors generated. on clang-13, but passes on gcc (with appropriate asm goto support). The bug is fixed in clang-14, but won't be backported to clang-13 as the changes are too invasive/risky. gcc also had a similar bug[3], fixed in gcc-11, where gcc failed to account for its behavior of assigning two numbers to tied outputs (one for input, one for output) when evaluating symbolic references. [1] https://github.com/ClangBuiltLinux/linux/issues/1512 [2] https://lore.kernel.org/all/YfMruK8%2F1izZ2VHS@google.com [3] https://gcc.gnu.org/bugzilla/show_bug.cgi?id=3D98096 Suggested-by: Nick Desaulniers Reviewed-by: Nick Desaulniers Cc: stable@vger.kernel.org Signed-off-by: Sean Christopherson --- init/Kconfig | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/init/Kconfig b/init/Kconfig index e9119bf54b1f..4a7a569706c5 100644 --- a/init/Kconfig +++ b/init/Kconfig @@ -77,6 +77,11 @@ config CC_HAS_ASM_GOTO_OUTPUT depends on CC_HAS_ASM_GOTO def_bool $(success,echo 'int foo(int x) { asm goto ("": "=3Dr"(x) ::: bar= ); return x; bar: return 0; }' | $(CC) -x c - -c -o /dev/null) =20 +config CC_HAS_ASM_GOTO_TIED_OUTPUT + depends on CC_HAS_ASM_GOTO_OUTPUT + # Detect buggy gcc and clang, fixed in gcc-11 clang-14. + def_bool $(success,echo 'int foo(int *x) { asm goto (".long (%l[bar]) - .= \n": "+m"(*x) ::: bar); return *x; bar: return 0; }' | $CC -x c - -c -o /de= v/null) + config TOOLS_SUPPORT_RELR def_bool $(success,env "CC=3D$(CC)" "LD=3D$(LD)" "NM=3D$(NM)" "OBJCOPY=3D= $(OBJCOPY)" $(srctree)/scripts/tools-support-relr.sh) =20 --=20 2.35.0.rc2.247.g8bbb082509-goog From nobody Mon Jun 29 22:17:28 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8781AC433F5 for ; Wed, 2 Feb 2022 00:50:07 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S243578AbiBBAuG (ORCPT ); Tue, 1 Feb 2022 19:50:06 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55856 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S243544AbiBBAuC (ORCPT ); Tue, 1 Feb 2022 19:50:02 -0500 Received: from mail-pl1-x64a.google.com (mail-pl1-x64a.google.com [IPv6:2607:f8b0:4864:20::64a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 73EC0C06173D for ; Tue, 1 Feb 2022 16:50:02 -0800 (PST) Received: by mail-pl1-x64a.google.com with SMTP id j1-20020a170903028100b0014b1f9e0068so7684082plr.8 for ; Tue, 01 Feb 2022 16:50:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=reply-to:date:in-reply-to:message-id:mime-version:references :subject:from:to:cc; bh=zqC7ErNhTDTy1v1hLoDrXbuxVMv5IRrOrkRyOKcUaeg=; b=XKb4zPuXIIwctcSOmCDmgJfg8NTQ92lvtVXEJu/4qhXFU2LKeL42KXV03TYMTCs379 8jXzoYm/94swbvUY5Yg0Lg8i2wfq2DciirhQjJKTV0wsndl1BybrIiAh71gb7u9tCo1k dwFSXfs4lpOeoZM3HVz0ZJpLOadK8AQM2fzyfb7JWWnxZFIOdlNqo+7ptVmHZE7ovoLs /hN4L1bzTHmRpRhrNvh9qdo/RDzN6WhEleS5YKjMBTYDm5d/ymorEkf3y3+ZycMpvFa0 H8jmh01pHfFQUFK4r4r4yV/EbZYPMn2DV5E3leaXgCMzaWZ/8qSaeKJa+6RLnvIjEZm8 imdg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:reply-to:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=zqC7ErNhTDTy1v1hLoDrXbuxVMv5IRrOrkRyOKcUaeg=; b=tZmwnwbNlSLxQ3Q1FSGXncOj/snH1acK7iZuPblEP6LyigjrRYzIencpRasLByZiGy z3BoLi0wc5WZVLEF8p+nTXWNtcZ57+cnB99/HlofiyYEPXWM3AQmoCzwXA1tZmi9kHTL HdG2aa6jAS2Ko/95aURwz5d26euDpi3FpFcoYW9pWLqQXsl0dFCkOwNxv3JREsYJ+Ri1 063fb2btXxSFpgafyPrVHrFAQ9cVY2FroAjGKuhPYazYPp64TBXstBauX2p2YEr7vHec WeQ318NBaCs0SrZmQtgoK8/T2IE9TBw7Bv+5PRtVksDWZTuE/GoSQXcgCsCofzRAeQCM I30w== X-Gm-Message-State: AOAM532tXDYhiFvRQEApAydEJ3/ZzmZfIaJpBSENCEhRNXrSnxcjhDU8 mZM3Ep3nSrh8zivRZapEHtQrYL6eUpE= X-Google-Smtp-Source: ABdhPJyK+/wekACSBU7YDtI0MDyZWi14PbNwtr0sorptRCs82INgfWofplxqnj/Mp3fClTt/ZVZdVz23RM0= X-Received: from seanjc.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e5]) (user=seanjc job=sendgmr) by 2002:a17:902:da90:: with SMTP id j16mr28314536plx.101.1643763001854; Tue, 01 Feb 2022 16:50:01 -0800 (PST) Reply-To: Sean Christopherson Date: Wed, 2 Feb 2022 00:49:42 +0000 In-Reply-To: <20220202004945.2540433-1-seanjc@google.com> Message-Id: <20220202004945.2540433-3-seanjc@google.com> Mime-Version: 1.0 References: <20220202004945.2540433-1-seanjc@google.com> X-Mailer: git-send-email 2.35.0.rc2.247.g8bbb082509-goog Subject: [PATCH v2 2/5] x86/uaccess: Implement macros for CMPXCHG on user addresses From: Sean Christopherson To: Paolo Bonzini , Nathan Chancellor , Nick Desaulniers Cc: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , kvm@vger.kernel.org, llvm@lists.linux.dev, linux-kernel@vger.kernel.org, Peter Zijlstra , Tadeusz Struk , syzbot+6cde2282daa792c49ab8@syzkaller.appspotmail.com Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Peter Zijlstra Add support for CMPXCHG loops on userspace addresses. Provide both an "unsafe" version for tight loops that do their own uaccess begin/end, as well as a "safe" version for use cases where the CMPXCHG is not buried in a loop, e.g. KVM will resume the guest instead of looping when emulation of a guest atomic accesses fails the CMPXCHG. Provide 8-byte versions for 32-bit kernels so that KVM can do CMPXCHG on guest PAE PTEs, which are accessed via userspace addresses. Guard the asm_volatile_goto() variation with CC_HAS_ASM_GOTO_TIED_OUTPUT, the "+m" constraint fails on some compilers that otherwise support CC_HAS_ASM_GOTO_OUTPUT. Cc: stable@vger.kernel.org Signed-off-by: Peter Zijlstra (Intel) Co-developed-by: Sean Christopherson Signed-off-by: Sean Christopherson --- arch/x86/include/asm/uaccess.h | 142 +++++++++++++++++++++++++++++++++ 1 file changed, 142 insertions(+) diff --git a/arch/x86/include/asm/uaccess.h b/arch/x86/include/asm/uaccess.h index ac96f9b2d64b..1c14bcce88f2 100644 --- a/arch/x86/include/asm/uaccess.h +++ b/arch/x86/include/asm/uaccess.h @@ -409,6 +409,103 @@ do { \ =20 #endif // CONFIG_CC_HAS_ASM_GOTO_OUTPUT =20 +#ifdef CONFIG_CC_HAS_ASM_GOTO_TIED_OUTPUT +#define __try_cmpxchg_user_asm(itype, ltype, _ptr, _pold, _new, label) ({ \ + bool success; \ + __typeof__(_ptr) _old =3D (__typeof__(_ptr))(_pold); \ + __typeof__(*(_ptr)) __old =3D *_old; \ + __typeof__(*(_ptr)) __new =3D (_new); \ + asm_volatile_goto("\n" \ + "1: " LOCK_PREFIX "cmpxchg"itype" %[new], %[ptr]\n"\ + _ASM_EXTABLE_UA(1b, %l[label]) \ + : CC_OUT(z) (success), \ + [ptr] "+m" (*_ptr), \ + [old] "+a" (__old) \ + : [new] ltype (__new) \ + : "memory" \ + : label); \ + if (unlikely(!success)) \ + *_old =3D __old; \ + likely(success); }) + +#ifdef CONFIG_X86_32 +#define __try_cmpxchg64_user_asm(_ptr, _pold, _new, label) ({ \ + bool success; \ + __typeof__(_ptr) _old =3D (__typeof__(_ptr))(_pold); \ + __typeof__(*(_ptr)) __old =3D *_old; \ + __typeof__(*(_ptr)) __new =3D (_new); \ + asm_volatile_goto("\n" \ + "1: " LOCK_PREFIX "cmpxchg8b %[ptr]\n" \ + _ASM_EXTABLE_UA(1b, %l[label]) \ + : CC_OUT(z) (success), \ + "+A" (__old), \ + [ptr] "+m" (*_ptr) \ + : "b" ((u32)__new), \ + "c" ((u32)((u64)__new >> 32)) \ + : "memory" \ + : label); \ + if (unlikely(!success)) \ + *_old =3D __old; \ + likely(success); }) +#endif // CONFIG_X86_32 +#else // !CONFIG_CC_HAS_ASM_GOTO_TIED_OUTPUT +#define __try_cmpxchg_user_asm(itype, ltype, _ptr, _pold, _new, label) ({ \ + int __err =3D 0; \ + bool success; \ + __typeof__(_ptr) _old =3D (__typeof__(_ptr))(_pold); \ + __typeof__(*(_ptr)) __old =3D *_old; \ + __typeof__(*(_ptr)) __new =3D (_new); \ + asm volatile("\n" \ + "1: " LOCK_PREFIX "cmpxchg"itype" %[new], %[ptr]\n"\ + CC_SET(z) \ + "2:\n" \ + _ASM_EXTABLE_TYPE_REG(1b, 2b, EX_TYPE_EFAULT_REG, \ + %[errout]) \ + : CC_OUT(z) (success), \ + [errout] "+r" (__err), \ + [ptr] "+m" (*_ptr), \ + [old] "+a" (__old) \ + : [new] ltype (__new) \ + : "memory", "cc"); \ + if (unlikely(__err)) \ + goto label; \ + if (unlikely(!success)) \ + *_old =3D __old; \ + likely(success); }) + +#ifdef CONFIG_X86_32 +/* + * Unlike the normal CMPXCHG, hardcode ECX for both success/fail and error. + * There are only six GPRs available and four (EAX, EBX, ECX, and EDX) are + * hardcoded by CMPXCHG8B, leaving only ESI and EDI. If the compiler uses + * both ESI and EDI for the memory operand, compilation will fail if the e= rror + * is an input+output as there will be no register available for input. + */ +#define __try_cmpxchg64_user_asm(_ptr, _pold, _new, label) ({ \ + int __result; \ + __typeof__(_ptr) _old =3D (__typeof__(_ptr))(_pold); \ + __typeof__(*(_ptr)) __old =3D *_old; \ + __typeof__(*(_ptr)) __new =3D (_new); \ + asm volatile("\n" \ + "1: " LOCK_PREFIX "cmpxchg8b %[ptr]\n" \ + "mov $0, %%ecx\n\t" \ + "setz %%cl\n" \ + "2:\n" \ + _ASM_EXTABLE_TYPE_REG(1b, 2b, EX_TYPE_EFAULT_REG, %%ecx) \ + : [result]"=3Dc" (__result), \ + "+A" (__old), \ + [ptr] "+m" (*_ptr) \ + : "b" ((u32)__new), \ + "c" ((u32)((u64)__new >> 32)) \ + : "memory", "cc"); \ + if (unlikely(__result < 0)) \ + goto label; \ + if (unlikely(!__result)) \ + *_old =3D __old; \ + likely(__result); }) +#endif // CONFIG_X86_32 +#endif // CONFIG_CC_HAS_ASM_GOTO_TIED_OUTPUT + /* FIXME: this hack is definitely wrong -AK */ struct __large_struct { unsigned long buf[100]; }; #define __m(x) (*(struct __large_struct __user *)(x)) @@ -501,6 +598,51 @@ do { \ } while (0) #endif // CONFIG_CC_HAS_ASM_GOTO_OUTPUT =20 +extern void __try_cmpxchg_user_wrong_size(void); + +#ifndef CONFIG_X86_32 +#define __try_cmpxchg64_user_asm(_ptr, _oldp, _nval, _label) \ + __try_cmpxchg_user_asm("q", "r", (_ptr), (_oldp), (_nval), _label) +#endif + +/* + * Force the pointer to u to match the size expected by the asm help= er. + * clang/LLVM compiles all cases and only discards the unused paths after + * processing errors, which breaks i386 if the pointer is an 8-byte value. + */ +#define unsafe_try_cmpxchg_user(_ptr, _oldp, _nval, _label) ({ \ + bool __ret; \ + __chk_user_ptr(_ptr); \ + switch (sizeof(*(_ptr))) { \ + case 1: __ret =3D __try_cmpxchg_user_asm("b", "q", \ + (__force u8 *)(_ptr), (_oldp), \ + (_nval), _label); \ + break; \ + case 2: __ret =3D __try_cmpxchg_user_asm("w", "r", \ + (__force u16 *)(_ptr), (_oldp), \ + (_nval), _label); \ + break; \ + case 4: __ret =3D __try_cmpxchg_user_asm("l", "r", \ + (__force u32 *)(_ptr), (_oldp), \ + (_nval), _label); \ + break; \ + case 8: __ret =3D __try_cmpxchg64_user_asm((__force u64 *)(_ptr), (_oldp)= ,\ + (_nval), _label); \ + break; \ + default: __try_cmpxchg_user_wrong_size(); \ + } \ + __ret; }) + +/* "Returns" 0 on success, 1 on failure, -EFAULT if the access faults. */ +#define __try_cmpxchg_user(_ptr, _oldp, _nval, _label) ({ \ + int __ret =3D -EFAULT; \ + __uaccess_begin_nospec(); \ + __ret =3D !unsafe_try_cmpxchg_user(_ptr, _oldp, _nval, _label); \ +_label: \ + __uaccess_end(); \ + __ret; \ + }) + /* * We want the unsafe accessors to always be inlined and use * the error labels - thus the macro games. --=20 2.35.0.rc2.247.g8bbb082509-goog From nobody Mon Jun 29 22:17:28 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id DFCAEC433F5 for ; Wed, 2 Feb 2022 00:50:10 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S243573AbiBBAuK (ORCPT ); Tue, 1 Feb 2022 19:50:10 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55868 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S243564AbiBBAuE (ORCPT ); Tue, 1 Feb 2022 19:50:04 -0500 Received: from mail-pf1-x449.google.com (mail-pf1-x449.google.com [IPv6:2607:f8b0:4864:20::449]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 192E7C061714 for ; Tue, 1 Feb 2022 16:50:04 -0800 (PST) Received: by mail-pf1-x449.google.com with SMTP id t69-20020a627848000000b004cb24c27d5aso9867474pfc.21 for ; Tue, 01 Feb 2022 16:50:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=reply-to:date:in-reply-to:message-id:mime-version:references :subject:from:to:cc; bh=LW0CGWCdKGNybf5BvgByl5slEYTpalX7lGLuyFTmzzE=; b=KF/N+QYr4iO95cCpL1G4uzy3+ivkEMfpIpF7KmjMsjT6UFuAiLOkDkuFPC+4P2sUlY ihpRqaIjOz4kN6/89V2YCtvtnEp6CgO2lgyYbLPfV92eWmU/hBojGwPjCs3FwKyZsXrX yuoY9faMjNWzQpgzPtwNASEsZCoeIVq40P6nCQwX4KrJmIivrRIFkVMNC4gBhgu7QdQK nzJ5R1NlpEyjMBzZKMCYYmm8a4J05RK4Y1u2dvI4M0nezvomCbEtKSi+IA/jtC0k80d9 fXLvxpGVAcj1/8GzA0YN4YJezqs8Fe3DTGszxuuV5bau90Ty1COT15DCtwmt2tWIfHVv EWiQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:reply-to:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=LW0CGWCdKGNybf5BvgByl5slEYTpalX7lGLuyFTmzzE=; b=ggVbW38rdwooX2rBaAhq9/zgv9dvwuPk59R9mTTRMvBdMp/mQLmfYOn4GkU6/nKtjm jyaFeIlRWMCsMMS+3mdgHXvg1/xQrrkV6y2CIdFYTiJDPVUgu4JLCYQ+8u5xha7EjE9q S0zuCS9UApJU7cd/jhy4Pz+O8Fd6U+IErf97hsF+i0hPXoUJrcR4ULb5a8g/6uFPwe3Y tTDAeTdP4Bx2XDzHvcZ6ArV565aMGYjbK2coPBohw7Mljxz0ci+Ulsx8zGfWo7G45xMi fyBHhVNNyH8tl/RC9ILsHFCYtjkmbK+f7Emn7fYUoWJOY7aFQRL6yPv9cYQVcKyA5XNz rdfA== X-Gm-Message-State: AOAM532A0mtlBYVP2iCbiZn16CG3c/253kMKx3MpBuSy5ytpPLjsStNe qGWH+mAq1zhZhPjWgKLXvzhdC4fGOHU= X-Google-Smtp-Source: ABdhPJz2wdcFwYzdK82hva9pUCt+iTHlqo2myaKtJAqY+WJziu6dFA2uGVEY5NWktaR6JfhrV1Ro8gjVAm8= X-Received: from seanjc.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e5]) (user=seanjc job=sendgmr) by 2002:a05:6a00:2cf:: with SMTP id b15mr13593957pft.0.1643763003513; Tue, 01 Feb 2022 16:50:03 -0800 (PST) Reply-To: Sean Christopherson Date: Wed, 2 Feb 2022 00:49:43 +0000 In-Reply-To: <20220202004945.2540433-1-seanjc@google.com> Message-Id: <20220202004945.2540433-4-seanjc@google.com> Mime-Version: 1.0 References: <20220202004945.2540433-1-seanjc@google.com> X-Mailer: git-send-email 2.35.0.rc2.247.g8bbb082509-goog Subject: [PATCH v2 3/5] KVM: x86: Use __try_cmpxchg_user() to update guest PTE A/D bits From: Sean Christopherson To: Paolo Bonzini , Nathan Chancellor , Nick Desaulniers Cc: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , kvm@vger.kernel.org, llvm@lists.linux.dev, linux-kernel@vger.kernel.org, Peter Zijlstra , Tadeusz Struk , syzbot+6cde2282daa792c49ab8@syzkaller.appspotmail.com Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Use the recently introduced __try_cmpxchg_user() to update guest PTE A/D bits instead of mapping the PTE into kernel address space. The VM_PFNMAP path is broken as it assumes that vm_pgoff is the base pfn of the mapped VMA range, which is conceptually wrong as vm_pgoff is the offset relative to the file and has nothing to do with the pfn. The horrific hack worked for the original use case (backing guest memory with /dev/mem), but leads to accessing "random" pfns for pretty much any other VM_PFNMAP case. Fixes: bd53cb35a3e9 ("X86/KVM: Handle PFNs outside of kernel reach when tou= ching GPTEs") Debugged-by: Tadeusz Struk Tested-by: Tadeusz Struk Reported-by: syzbot+6cde2282daa792c49ab8@syzkaller.appspotmail.com Cc: stable@vger.kernel.org Signed-off-by: Sean Christopherson --- arch/x86/kvm/mmu/paging_tmpl.h | 45 +--------------------------------- 1 file changed, 1 insertion(+), 44 deletions(-) diff --git a/arch/x86/kvm/mmu/paging_tmpl.h b/arch/x86/kvm/mmu/paging_tmpl.h index 5b5bdac97c7b..551de15f342f 100644 --- a/arch/x86/kvm/mmu/paging_tmpl.h +++ b/arch/x86/kvm/mmu/paging_tmpl.h @@ -143,49 +143,6 @@ static bool FNAME(is_rsvd_bits_set)(struct kvm_mmu *mm= u, u64 gpte, int level) FNAME(is_bad_mt_xwr)(&mmu->guest_rsvd_check, gpte); } =20 -static int FNAME(cmpxchg_gpte)(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu, - pt_element_t __user *ptep_user, unsigned index, - pt_element_t orig_pte, pt_element_t new_pte) -{ - int npages; - pt_element_t ret; - pt_element_t *table; - struct page *page; - - npages =3D get_user_pages_fast((unsigned long)ptep_user, 1, FOLL_WRITE, &= page); - if (likely(npages =3D=3D 1)) { - table =3D kmap_atomic(page); - ret =3D CMPXCHG(&table[index], orig_pte, new_pte); - kunmap_atomic(table); - - kvm_release_page_dirty(page); - } else { - struct vm_area_struct *vma; - unsigned long vaddr =3D (unsigned long)ptep_user & PAGE_MASK; - unsigned long pfn; - unsigned long paddr; - - mmap_read_lock(current->mm); - vma =3D find_vma_intersection(current->mm, vaddr, vaddr + PAGE_SIZE); - if (!vma || !(vma->vm_flags & VM_PFNMAP)) { - mmap_read_unlock(current->mm); - return -EFAULT; - } - pfn =3D ((vaddr - vma->vm_start) >> PAGE_SHIFT) + vma->vm_pgoff; - paddr =3D pfn << PAGE_SHIFT; - table =3D memremap(paddr, PAGE_SIZE, MEMREMAP_WB); - if (!table) { - mmap_read_unlock(current->mm); - return -EFAULT; - } - ret =3D CMPXCHG(&table[index], orig_pte, new_pte); - memunmap(table); - mmap_read_unlock(current->mm); - } - - return (ret !=3D orig_pte); -} - static bool FNAME(prefetch_invalid_gpte)(struct kvm_vcpu *vcpu, struct kvm_mmu_page *sp, u64 *spte, u64 gpte) @@ -284,7 +241,7 @@ static int FNAME(update_accessed_dirty_bits)(struct kvm= _vcpu *vcpu, if (unlikely(!walker->pte_writable[level - 1])) continue; =20 - ret =3D FNAME(cmpxchg_gpte)(vcpu, mmu, ptep_user, index, orig_pte, pte); + ret =3D __try_cmpxchg_user(ptep_user, &orig_pte, pte, fault); if (ret) return ret; =20 --=20 2.35.0.rc2.247.g8bbb082509-goog From nobody Mon Jun 29 22:17:28 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B6465C433EF for ; Wed, 2 Feb 2022 00:50:12 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S243627AbiBBAuL (ORCPT ); Tue, 1 Feb 2022 19:50:11 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55880 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S243574AbiBBAuF (ORCPT ); Tue, 1 Feb 2022 19:50:05 -0500 Received: from mail-pg1-x549.google.com (mail-pg1-x549.google.com [IPv6:2607:f8b0:4864:20::549]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C6A5FC061401 for ; Tue, 1 Feb 2022 16:50:05 -0800 (PST) Received: by mail-pg1-x549.google.com with SMTP id i23-20020a635417000000b00364c29f39aaso4643748pgb.8 for ; Tue, 01 Feb 2022 16:50:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=reply-to:date:in-reply-to:message-id:mime-version:references :subject:from:to:cc; bh=YlCl+ohirKd09vv0wcllKf/IW1dSS26lbTtAtOAdLYg=; b=Qctxu1B8NhsO5um4y7KJJHlRD72php90lxuzKdp6hG5en0y0USYIVlfvIob1Axm8Nb 6STElTyHIexmGhrBQEpkqzbKFDOpCVzflM2HphSEsSkOrbX4CDLKcr90FGgiXg0APt64 8kVvBHzJqrsDrIftmybBme8IGhVQnO2UA2C93yfqw4yQdipU6RahqIzRpgUrNzOyPbRh RVhuPRgRVkUChtu1JW6GEO0VfHYgs3ipc8aRJ300voV5AOlPEKIrAlI10r6mcf0pcvUZ NsApodqde8pLGBWv/m7ZNwhbWsiD4Vm3+re04yGZ/ffBvWYOgp1s3D7Udp0d1+VH9lOU 096w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:reply-to:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=YlCl+ohirKd09vv0wcllKf/IW1dSS26lbTtAtOAdLYg=; b=OLpd+gm6zheuHRCiGxfW3MRvp15aLiX/n05iOBLuriYk+0MQWQmDsp2w47ioZrA1yt x7/dkhtxfieoN1wkuY2mZGw3XP5gTNxJ6IJ0qpoWt2aJKnVHgG27SYqcOFL6RAiHP/kR 7SzsLcXR0Ik/YB7Q9uov0Yd/oIAD+Ta0HDT8EZxt7GpYEuaEJtNf5rdAb6c/PrMPD0vD n4FBuDjS6+0IfC1nNqsC0UQd2hYiyynWss1ILUcTfsCPJIpOFtlzlvunWF0lna+MDwwc qsXztt9Lzwc4l9681BII/OuEfYdFaaHU/9Oj9r5fOxKkLaEQ+S86XtZDyZgiWBshXjlm /fXA== X-Gm-Message-State: AOAM531i3Wuo40MiA1UhQPutZuLbn6IpXDPg0P0CD7mZ7gAzICx4zu/F /OtzADiBeb+d87Kz+nVsHUqtFeWfmbM= X-Google-Smtp-Source: ABdhPJz2X2rTMuFJJEYa6E10XzsJthSh3qvVQfRKYL+RLkE4rRhykPoFGY/gKPXyjSkfJpiwXyTqUs0zF2U= X-Received: from seanjc.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e5]) (user=seanjc job=sendgmr) by 2002:a05:6a00:1992:: with SMTP id d18mr26971386pfl.85.1643763005290; Tue, 01 Feb 2022 16:50:05 -0800 (PST) Reply-To: Sean Christopherson Date: Wed, 2 Feb 2022 00:49:44 +0000 In-Reply-To: <20220202004945.2540433-1-seanjc@google.com> Message-Id: <20220202004945.2540433-5-seanjc@google.com> Mime-Version: 1.0 References: <20220202004945.2540433-1-seanjc@google.com> X-Mailer: git-send-email 2.35.0.rc2.247.g8bbb082509-goog Subject: [PATCH v2 4/5] KVM: x86: Use __try_cmpxchg_user() to emulate atomic accesses From: Sean Christopherson To: Paolo Bonzini , Nathan Chancellor , Nick Desaulniers Cc: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , kvm@vger.kernel.org, llvm@lists.linux.dev, linux-kernel@vger.kernel.org, Peter Zijlstra , Tadeusz Struk , syzbot+6cde2282daa792c49ab8@syzkaller.appspotmail.com Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Use the recently introduce __try_cmpxchg_user() to emulate atomic guest accesses via the associated userspace address instead of mapping the backing pfn into kernel address space. Using kvm_vcpu_map() is unsafe as it does not coordinate with KVM's mmu_notifier to ensure the hva=3D>pfn translation isn't changed/unmapped in the memremap() path, i.e. when there's no struct page and thus no elevated refcount. Fixes: 42e35f8072c3 ("KVM/X86: Use kvm_vcpu_map in emulator_cmpxchg_emulate= d") Cc: stable@vger.kernel.org Signed-off-by: Sean Christopherson --- arch/x86/kvm/x86.c | 35 ++++++++++++++--------------------- 1 file changed, 14 insertions(+), 21 deletions(-) diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index 74b53a16f38a..c9cac3100f77 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -7155,15 +7155,8 @@ static int emulator_write_emulated(struct x86_emulat= e_ctxt *ctxt, exception, &write_emultor); } =20 -#define CMPXCHG_TYPE(t, ptr, old, new) \ - (cmpxchg((t *)(ptr), *(t *)(old), *(t *)(new)) =3D=3D *(t *)(old)) - -#ifdef CONFIG_X86_64 -# define CMPXCHG64(ptr, old, new) CMPXCHG_TYPE(u64, ptr, old, new) -#else -# define CMPXCHG64(ptr, old, new) \ - (cmpxchg64((u64 *)(ptr), *(u64 *)(old), *(u64 *)(new)) =3D=3D *(u64 *)(ol= d)) -#endif +#define emulator_try_cmpxchg_user(t, ptr, old, new) \ + (__try_cmpxchg_user((t __user *)(ptr), (t *)(old), *(t *)(new), efault ##= t)) =20 static int emulator_cmpxchg_emulated(struct x86_emulate_ctxt *ctxt, unsigned long addr, @@ -7172,12 +7165,11 @@ static int emulator_cmpxchg_emulated(struct x86_emu= late_ctxt *ctxt, unsigned int bytes, struct x86_exception *exception) { - struct kvm_host_map map; struct kvm_vcpu *vcpu =3D emul_to_vcpu(ctxt); u64 page_line_mask; + unsigned long hva; gpa_t gpa; - char *kaddr; - bool exchanged; + int r; =20 /* guests cmpxchg8b have to be emulated atomically */ if (bytes > 8 || (bytes & (bytes - 1))) @@ -7201,31 +7193,32 @@ static int emulator_cmpxchg_emulated(struct x86_emu= late_ctxt *ctxt, if (((gpa + bytes - 1) & page_line_mask) !=3D (gpa & page_line_mask)) goto emul_write; =20 - if (kvm_vcpu_map(vcpu, gpa_to_gfn(gpa), &map)) + hva =3D kvm_vcpu_gfn_to_hva(vcpu, gpa_to_gfn(gpa)); + if (kvm_is_error_hva(addr)) goto emul_write; =20 - kaddr =3D map.hva + offset_in_page(gpa); + hva +=3D offset_in_page(gpa); =20 switch (bytes) { case 1: - exchanged =3D CMPXCHG_TYPE(u8, kaddr, old, new); + r =3D emulator_try_cmpxchg_user(u8, hva, old, new); break; case 2: - exchanged =3D CMPXCHG_TYPE(u16, kaddr, old, new); + r =3D emulator_try_cmpxchg_user(u16, hva, old, new); break; case 4: - exchanged =3D CMPXCHG_TYPE(u32, kaddr, old, new); + r =3D emulator_try_cmpxchg_user(u32, hva, old, new); break; case 8: - exchanged =3D CMPXCHG64(kaddr, old, new); + r =3D emulator_try_cmpxchg_user(u64, hva, old, new); break; default: BUG(); } =20 - kvm_vcpu_unmap(vcpu, &map, true); - - if (!exchanged) + if (r < 0) + goto emul_write; + if (r) return X86EMUL_CMPXCHG_FAILED; =20 kvm_page_track_write(vcpu, gpa, new, bytes); --=20 2.35.0.rc2.247.g8bbb082509-goog From nobody Mon Jun 29 22:17:28 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 82882C433F5 for ; Wed, 2 Feb 2022 00:50:16 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S243624AbiBBAuP (ORCPT ); Tue, 1 Feb 2022 19:50:15 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55892 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S243598AbiBBAuH (ORCPT ); Tue, 1 Feb 2022 19:50:07 -0500 Received: from mail-pg1-x549.google.com (mail-pg1-x549.google.com [IPv6:2607:f8b0:4864:20::549]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 77932C06173D for ; Tue, 1 Feb 2022 16:50:07 -0800 (PST) Received: by mail-pg1-x549.google.com with SMTP id t1-20020a6564c1000000b002e7f31cf59fso11297243pgv.14 for ; Tue, 01 Feb 2022 16:50:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=reply-to:date:in-reply-to:message-id:mime-version:references :subject:from:to:cc; bh=3/7OuVceBn0EleUU7nySvspzg5VyH7dxhNHRLM/3NIE=; b=EtaUGeB0crakORjKPe7ruh7EgNS85rRKIo1vZQSAgHFcEudyPUz4RuiIq+KtjJYiZR 1OqL25OxkYDNrCXKWKLz2YOtA4ckbOSJ3+YP4TfziGxQnMIEELSoi06HE/9CeAUSLKXU kSy5TvR9DjgowQ6RUPus4vXm0kcKZIN0W/4GBqjk0cAptuQUniCap/nPA6FskhIFtUeQ TTChArhpBpKfLOJe2aEt4MAxjHdkfSWQ++Di3jyS/VRH2ieCQD9GR20lU+klsKYnE5iY Fk4wRG/2OAgDLpvQ3U/NmazfqzqkzoGGPPqm8cy9lzbAKtPwG/yzEg1DGV4eFnlG7ZaS qHGw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:reply-to:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=3/7OuVceBn0EleUU7nySvspzg5VyH7dxhNHRLM/3NIE=; b=FYg29m1IbEuqyY7zdTiOtx5c4LHEMzo2S3VQ0d4H/DZyo2n+S9SgNjNivtWVqn9FO5 zVmt4QstQQMWxRoKviVkmpY2oFWKmNhHc8WKMtGIhNbGbntPrRTsVPlhbvTUNITtKp2t gcfCd/L2fJZHeVeuHW5sWxYN27Lsgl28+mx/28AzOMZ8+nHW1fHVzIK5sgTmBky5kPDX 4XQwhdRS06BPnnKSw94Z273mRX8XXBm0zollXv8k0RlbSWERunx53BdudtIx7GIDlgZC g77uj3QNHZ77Q7S9fVCAhBQ10vaM3cEoPGPXo1vhPr4Z+PYoOk+STWL7toU5aK6h7bNU OOag== X-Gm-Message-State: AOAM530eSPhCtRj8lqvH9gFQzuPEgaZLK8NwBScqsZyh6gcm16D/BpTs x8BpaVrComKaPYo/XDZB6kqouXW4QYg= X-Google-Smtp-Source: ABdhPJxaOUM0XxNVSnap4U+XQvTqFWJQ1EEDHMwv7ezGfJzzItoGIpIVtAqpcjtk78QdP1E/SRU2o8215xU= X-Received: from seanjc.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e5]) (user=seanjc job=sendgmr) by 2002:a17:902:b212:: with SMTP id t18mr13193260plr.80.1643763007012; Tue, 01 Feb 2022 16:50:07 -0800 (PST) Reply-To: Sean Christopherson Date: Wed, 2 Feb 2022 00:49:45 +0000 In-Reply-To: <20220202004945.2540433-1-seanjc@google.com> Message-Id: <20220202004945.2540433-6-seanjc@google.com> Mime-Version: 1.0 References: <20220202004945.2540433-1-seanjc@google.com> X-Mailer: git-send-email 2.35.0.rc2.247.g8bbb082509-goog Subject: [PATCH v2 5/5] KVM: x86: Bail to userspace if emulation of atomic user access faults From: Sean Christopherson To: Paolo Bonzini , Nathan Chancellor , Nick Desaulniers Cc: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , kvm@vger.kernel.org, llvm@lists.linux.dev, linux-kernel@vger.kernel.org, Peter Zijlstra , Tadeusz Struk , syzbot+6cde2282daa792c49ab8@syzkaller.appspotmail.com Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Exit to userspace when emulating an atomic guest access if the CMPXCHG on the userspace address faults. Emulating the access as a write and thus likely treating it as emulated MMIO is wrong, as KVM has already confirmed there is a valid, writable memslot. Signed-off-by: Sean Christopherson --- arch/x86/kvm/x86.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index c9cac3100f77..24e0981816c0 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -7217,7 +7217,7 @@ static int emulator_cmpxchg_emulated(struct x86_emula= te_ctxt *ctxt, } =20 if (r < 0) - goto emul_write; + return X86EMUL_UNHANDLEABLE; if (r) return X86EMUL_CMPXCHG_FAILED; =20 --=20 2.35.0.rc2.247.g8bbb082509-goog