From nobody Mon May 11 04:11:52 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 37362C433FE for ; Fri, 15 Apr 2022 00:43:55 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1348130AbiDOAqT (ORCPT ); Thu, 14 Apr 2022 20:46:19 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37106 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1348099AbiDOAqP (ORCPT ); Thu, 14 Apr 2022 20:46:15 -0400 Received: from mail-pg1-x54a.google.com (mail-pg1-x54a.google.com [IPv6:2607:f8b0:4864:20::54a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E414C1D0CA for ; Thu, 14 Apr 2022 17:43:48 -0700 (PDT) Received: by mail-pg1-x54a.google.com with SMTP id w3-20020a639343000000b003821fb4a367so3459207pgm.21 for ; Thu, 14 Apr 2022 17:43:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=reply-to:date:in-reply-to:message-id:mime-version:references :subject:from:to:cc; bh=WHJorEYqdPsyoiS8LBeb1G8S4S7kzQkyLWZGD9aUrjs=; b=bZwTwRaSjFo3UIyEnG1Se3GCFh+RY6IvOG3VlupdDpfNfI6GX6hkJeLI81eGmOGX6G 1cHm7fTg8TqH04RjFv10Of5DvAYAHAM8tiEPrQRiuCNBDJRSzCUDQUG0BMRWHtD+LdNa OXpZzAre9wRSs/2YHv0GHmqXz+Pqefc9fuZxtMcl8GYafBb9wbzvPCjbT/nzQhtpz2Wn fyyyAL63cPcu44vOIB0qfBetcyUX/9l8dZ2b05Q4AfOcPl4SLNf/Fbj41CZv9b/yvg1F Aer1piomJTMTT3b0SNhbaUMWvPS5/iUeOlL+Pvad6lLyGugPdHlz8JZmH21Rl+8vgLjA mBcA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:reply-to:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=WHJorEYqdPsyoiS8LBeb1G8S4S7kzQkyLWZGD9aUrjs=; b=irc5sfjs+uzjEYgTApHbxYmXUBzBSP0sGiksKdX8nm0Rf3NeRdFV8kIqIjmpWF/C2W /lVkcAm+fV0ptnKvonP648timQeQTBkvOLL5EDnD7jd63bR5hLurvxfdSvPU1pqstAS5 NbVtFEjcWuHbtyK2kbjEnELcEaut5Bn+yVVfWxJXvKUmzB7DaqZG6HGdqnnQPbewI3rf lP0lcvtk3Pw2zOl/ET/L2h/A6Amzh+ck0VYyfcxZTRZwrWuVUPJmhxGtj+/k+yhNNb+i alDplSqK16QR+nwyz1kF1xhvWi7C805yHWI7GkaExfb2YohmKQOJVHwkA8e5emZdA/at JyVw== X-Gm-Message-State: AOAM531+vzzij85AsdAMT/ZkN9YH8rINLxCDY+ye71wSQhRZfQBl74IO 1piUMSYZgFCImF0QXMyeC5YOtqEXQLg= X-Google-Smtp-Source: ABdhPJzt0tscL0kcshOCrjwd8G3ruQcGXCkqq7xMYex2fkXBeIa/R8tC7xq7T2RAxBd1S9ySsMfZwzJ54UU= X-Received: from seanjc.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e5]) (user=seanjc job=sendgmr) by 2002:a17:90b:4b84:b0:1cb:6cf5:d2ff with SMTP id lr4-20020a17090b4b8400b001cb6cf5d2ffmr1311935pjb.41.1649983428416; Thu, 14 Apr 2022 17:43:48 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 15 Apr 2022 00:43:41 +0000 In-Reply-To: <20220415004343.2203171-1-seanjc@google.com> Message-Id: <20220415004343.2203171-2-seanjc@google.com> Mime-Version: 1.0 References: <20220415004343.2203171-1-seanjc@google.com> X-Mailer: git-send-email 2.36.0.rc0.470.gd361397f0d-goog Subject: [PATCH 1/3] KVM: x86: Don't re-acquire SRCU lock in complete_emulated_io() From: Sean Christopherson To: Anup Patel , Paul Walmsley , Palmer Dabbelt , Albert Ou , Christian Borntraeger , Janosch Frank , Claudio Imbrenda , Paolo Bonzini Cc: Atish Patra , David Hildenbrand , Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , linuxppc-dev@lists.ozlabs.org, kvm@vger.kernel.org, kvm-riscv@lists.infradead.org, linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Don't re-acquire SRCU in complete_emulated_io() now that KVM acquires the lock in kvm_arch_vcpu_ioctl_run(). More importantly, don't overwrite vcpu->srcu_idx. If the index acquired by complete_emulated_io() differs from the one acquired by kvm_arch_vcpu_ioctl_run(), KVM will effectively leak a lock and hang if/when synchronize_srcu() is invoked for the relevant grace period. Fixes: 8d25b7beca7e ("KVM: x86: pull kvm->srcu read-side to kvm_arch_vcpu_i= octl_run") Cc: stable@vger.kernel.org Signed-off-by: Sean Christopherson Reviewed-by: Maxim Levitsky --- arch/x86/kvm/x86.c | 7 +------ 1 file changed, 1 insertion(+), 6 deletions(-) diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index ab336f7c82e4..f35fe09de59d 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -10450,12 +10450,7 @@ static int vcpu_run(struct kvm_vcpu *vcpu) =20 static inline int complete_emulated_io(struct kvm_vcpu *vcpu) { - int r; - - vcpu->srcu_idx =3D srcu_read_lock(&vcpu->kvm->srcu); - r =3D kvm_emulate_instruction(vcpu, EMULTYPE_NO_DECODE); - srcu_read_unlock(&vcpu->kvm->srcu, vcpu->srcu_idx); - return r; + return kvm_emulate_instruction(vcpu, EMULTYPE_NO_DECODE); } =20 static int complete_emulated_pio(struct kvm_vcpu *vcpu) --=20 2.36.0.rc0.470.gd361397f0d-goog From nobody Mon May 11 04:11:52 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 809B9C433EF for ; Fri, 15 Apr 2022 00:43:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1348146AbiDOAqW (ORCPT ); Thu, 14 Apr 2022 20:46:22 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37130 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1348110AbiDOAqQ (ORCPT ); Thu, 14 Apr 2022 20:46:16 -0400 Received: from mail-pf1-x449.google.com (mail-pf1-x449.google.com [IPv6:2607:f8b0:4864:20::449]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A383996832 for ; Thu, 14 Apr 2022 17:43:50 -0700 (PDT) Received: by mail-pf1-x449.google.com with SMTP id o190-20020a62cdc7000000b0050835e37396so1285031pfg.14 for ; Thu, 14 Apr 2022 17:43:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=reply-to:date:in-reply-to:message-id:mime-version:references :subject:from:to:cc; bh=VDBBxbRymwn99ifXkPvrfvP8JeLntCySB5qG2FLdHGc=; b=FuTYBhDITxI66Qxt5+87xMlO2q0C9iJXI9s7fddQAntJKNcZhy4i5CX3Vpr1K7b1gG lpGYjjhgm/CKQjYxIZBLOAlu6Wf9zZOr5BTnS1Mm46MrfyyFviEAqnWwqxLbNvwWcsyf JvdoNGNXJ/6sDa03DP7PftstcK+dADaHLntSvUANgU/0B4isfM6BZs4hWPzvL2xkAlhB CUMjlksohwVrJztO8x/MgujxI4iPE/ysUfovmABWThXxb2g+rlV8zXfjr6a4wKFGQQ8A 2BioePglRBUYrzqDp/34X5DBOonLRTI2L6fETCpuSbqRp5Wtv6N/vgi4/Y0xhSy1jDZu aMxQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:reply-to:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=VDBBxbRymwn99ifXkPvrfvP8JeLntCySB5qG2FLdHGc=; b=CJlk63/TZiKTlzw/pS7DlO5BPHmQMyn2thBddZsMUaaxKR4P8f4aawxLRU9bxG4JK1 D94/7mwUAnUWMO2p2PhoabrW+60TH/j0y7Tfjv9TC7DwXKQ0a3mmJNKag7EYj6ZfKMOu XJ+EVpGxpcLX2PIu6tNYjcO2qrOhHvJWxZwSZrKwe+cZWWpoNq/Wbg850VmvcNXyI69T 3S5RJlrShQexNX8BeaIU1E+G7mrRF6DL2+UsBJTLHbtfZVytxN27GdUyJ4YqOM7365NX n2C460UWRAkkDVM9OFN9oepyMAWry0/MUkzRz84fzn3JYsCkYxmJGiToV93yBXTq7D+L TmRw== X-Gm-Message-State: AOAM531CekqAnEUpj2ye4zzA5GeB8SJ0Yh+XRTWguzyf7plo8tbC3qEz zgwmmJ5PX1WJEApuD8JO3/aMDu57adE= X-Google-Smtp-Source: ABdhPJxWvnh9vGPdu8WwNvP/XothzcdXxz8DVteSxV4BWZ39Cw9eF09ZeKITExpxjd/oejtHjB373C4/RJY= X-Received: from seanjc.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e5]) (user=seanjc job=sendgmr) by 2002:a05:6a00:1a90:b0:506:1e27:a579 with SMTP id e16-20020a056a001a9000b005061e27a579mr6426109pfv.46.1649983430118; Thu, 14 Apr 2022 17:43:50 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 15 Apr 2022 00:43:42 +0000 In-Reply-To: <20220415004343.2203171-1-seanjc@google.com> Message-Id: <20220415004343.2203171-3-seanjc@google.com> Mime-Version: 1.0 References: <20220415004343.2203171-1-seanjc@google.com> X-Mailer: git-send-email 2.36.0.rc0.470.gd361397f0d-goog Subject: [PATCH 2/3] KVM: RISC-V: Use kvm_vcpu.srcu_idx, drop RISC-V's unnecessary copy From: Sean Christopherson To: Anup Patel , Paul Walmsley , Palmer Dabbelt , Albert Ou , Christian Borntraeger , Janosch Frank , Claudio Imbrenda , Paolo Bonzini Cc: Atish Patra , David Hildenbrand , Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , linuxppc-dev@lists.ozlabs.org, kvm@vger.kernel.org, kvm-riscv@lists.infradead.org, linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Use the generic kvm_vcpu's srcu_idx instead of using an indentical field in RISC-V's version of kvm_vcpu_arch. Generic KVM very intentionally does not touch vcpu->srcu_idx, i.e. there's zero chance of running afoul of common code. Signed-off-by: Sean Christopherson --- arch/riscv/include/asm/kvm_host.h | 3 --- arch/riscv/kvm/vcpu.c | 16 ++++++++-------- arch/riscv/kvm/vcpu_exit.c | 4 ++-- 3 files changed, 10 insertions(+), 13 deletions(-) diff --git a/arch/riscv/include/asm/kvm_host.h b/arch/riscv/include/asm/kvm= _host.h index 78da839657e5..cd4bbcecb0fb 100644 --- a/arch/riscv/include/asm/kvm_host.h +++ b/arch/riscv/include/asm/kvm_host.h @@ -193,9 +193,6 @@ struct kvm_vcpu_arch { =20 /* Don't run the VCPU (blocked) */ bool pause; - - /* SRCU lock index for in-kernel run loop */ - int srcu_idx; }; =20 static inline void kvm_arch_hardware_unsetup(void) {} diff --git a/arch/riscv/kvm/vcpu.c b/arch/riscv/kvm/vcpu.c index 6785aef4cbd4..2f1caf23eed4 100644 --- a/arch/riscv/kvm/vcpu.c +++ b/arch/riscv/kvm/vcpu.c @@ -724,13 +724,13 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu) /* Mark this VCPU ran at least once */ vcpu->arch.ran_atleast_once =3D true; =20 - vcpu->arch.srcu_idx =3D srcu_read_lock(&vcpu->kvm->srcu); + vcpu->srcu_idx =3D srcu_read_lock(&vcpu->kvm->srcu); =20 /* Process MMIO value returned from user-space */ if (run->exit_reason =3D=3D KVM_EXIT_MMIO) { ret =3D kvm_riscv_vcpu_mmio_return(vcpu, vcpu->run); if (ret) { - srcu_read_unlock(&vcpu->kvm->srcu, vcpu->arch.srcu_idx); + srcu_read_unlock(&vcpu->kvm->srcu, vcpu->srcu_idx); return ret; } } @@ -739,13 +739,13 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu) if (run->exit_reason =3D=3D KVM_EXIT_RISCV_SBI) { ret =3D kvm_riscv_vcpu_sbi_return(vcpu, vcpu->run); if (ret) { - srcu_read_unlock(&vcpu->kvm->srcu, vcpu->arch.srcu_idx); + srcu_read_unlock(&vcpu->kvm->srcu, vcpu->srcu_idx); return ret; } } =20 if (run->immediate_exit) { - srcu_read_unlock(&vcpu->kvm->srcu, vcpu->arch.srcu_idx); + srcu_read_unlock(&vcpu->kvm->srcu, vcpu->srcu_idx); return -EINTR; } =20 @@ -784,7 +784,7 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu) */ vcpu->mode =3D IN_GUEST_MODE; =20 - srcu_read_unlock(&vcpu->kvm->srcu, vcpu->arch.srcu_idx); + srcu_read_unlock(&vcpu->kvm->srcu, vcpu->srcu_idx); smp_mb__after_srcu_read_unlock(); =20 /* @@ -802,7 +802,7 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu) vcpu->mode =3D OUTSIDE_GUEST_MODE; local_irq_enable(); preempt_enable(); - vcpu->arch.srcu_idx =3D srcu_read_lock(&vcpu->kvm->srcu); + vcpu->srcu_idx =3D srcu_read_lock(&vcpu->kvm->srcu); continue; } =20 @@ -846,7 +846,7 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu) =20 preempt_enable(); =20 - vcpu->arch.srcu_idx =3D srcu_read_lock(&vcpu->kvm->srcu); + vcpu->srcu_idx =3D srcu_read_lock(&vcpu->kvm->srcu); =20 ret =3D kvm_riscv_vcpu_exit(vcpu, run, &trap); } @@ -855,7 +855,7 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu) =20 vcpu_put(vcpu); =20 - srcu_read_unlock(&vcpu->kvm->srcu, vcpu->arch.srcu_idx); + srcu_read_unlock(&vcpu->kvm->srcu, vcpu->srcu_idx); =20 return ret; } diff --git a/arch/riscv/kvm/vcpu_exit.c b/arch/riscv/kvm/vcpu_exit.c index aa8af129e4bb..2d56faddb9d1 100644 --- a/arch/riscv/kvm/vcpu_exit.c +++ b/arch/riscv/kvm/vcpu_exit.c @@ -456,9 +456,9 @@ static int stage2_page_fault(struct kvm_vcpu *vcpu, str= uct kvm_run *run, void kvm_riscv_vcpu_wfi(struct kvm_vcpu *vcpu) { if (!kvm_arch_vcpu_runnable(vcpu)) { - srcu_read_unlock(&vcpu->kvm->srcu, vcpu->arch.srcu_idx); + srcu_read_unlock(&vcpu->kvm->srcu, vcpu->srcu_idx); kvm_vcpu_halt(vcpu); - vcpu->arch.srcu_idx =3D srcu_read_lock(&vcpu->kvm->srcu); + vcpu->srcu_idx =3D srcu_read_lock(&vcpu->kvm->srcu); kvm_clear_request(KVM_REQ_UNHALT, vcpu); } } --=20 2.36.0.rc0.470.gd361397f0d-goog From nobody Mon May 11 04:11:52 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B3F5DC43217 for ; Fri, 15 Apr 2022 00:44:07 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1348179AbiDOAqb (ORCPT ); Thu, 14 Apr 2022 20:46:31 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37196 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1348099AbiDOAqT (ORCPT ); Thu, 14 Apr 2022 20:46:19 -0400 Received: from mail-pl1-x649.google.com (mail-pl1-x649.google.com [IPv6:2607:f8b0:4864:20::649]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 87CF4972E3 for ; Thu, 14 Apr 2022 17:43:52 -0700 (PDT) Received: by mail-pl1-x649.google.com with SMTP id f6-20020a170902ab8600b0015895212d23so3553688plr.6 for ; Thu, 14 Apr 2022 17:43:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=reply-to:date:in-reply-to:message-id:mime-version:references :subject:from:to:cc; bh=J6d3I3cnZrxLwCCfWif9ZhGI1oj0wUvU2P/xwVxpp7E=; b=CuSw2eeS58tcsTk9P9jmAslUqLdVK+ojiMGCsZ4mnRsJM68tbU/9liTnKHoLbh/zkB CET5HLBaxyV3NzCIGyi18Y9MsYS26wH5TgRQTwP+gZ5f4Ar+VKgHDYKroN/6WsiRj7RQ eYBX18ZmYNcQc016hYZ/tn8O9erQLGiZhEtG50W9HsvPGIl0DO3yiGcUlFKDtZvgaiVI NI2olXANvvBr5xbPXB5K7JCvA/atk44/If4Jg6Ey7vg9mNW0gNp3IHxnApukaQEBDfQB oL1lNDBwghjRrkX+QxsS4PYwTmDT6W7IGfpDDgw25D5r7VVZWAXSCZhjJlXhjA7edeTe gQag== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:reply-to:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=J6d3I3cnZrxLwCCfWif9ZhGI1oj0wUvU2P/xwVxpp7E=; b=4WC8c9Yb51Kj0VT+Um0eb4TCwHu80GkUts6LSHfNPtc1G+CNF53ZafPlycRNurkBOC bynXrcX2SXUPX7eu7wtqCezwDJMqhxwzBFwKT55ZZ6Wy/TK/iyjvUmwhB6ImHMUx+6sa O7xBCSxdrBc23c9J8KIOuFZ322Yqh993z9G64uKw0I1v4bfpzX5H/BlZQLgOrm5FrQnz X9A/SYxs3sXalGtSsA+JmdEUEEfdUWns8Q9bZnL2nzgnyhULlH+TAgxlDVQ5WiNcPM8S IkOXE+CYjtkHz6Q+RVy4dsEuXQnd/k7N8IjkTHe8P9fqaLoVTHtgn4zmPJAl/+2ugme8 u7yg== X-Gm-Message-State: AOAM532OFEf9fc28bgY0nw7aDyfLBZVX8yAjnXAVnCQ0E/nngdOldRNR bUDJ415s3cm+8ejsajY/xi0WUdXqk9c= X-Google-Smtp-Source: ABdhPJwyjaEfxslH43Y0Hgw/F956yYwPWZ3LOHItRcDaHW9++RXdFSnPeuESTWLSl5iBgzGpFgGfSb9Cceo= X-Received: from seanjc.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e5]) (user=seanjc job=sendgmr) by 2002:a05:6a00:a26:b0:4fd:f9dd:549c with SMTP id p38-20020a056a000a2600b004fdf9dd549cmr6441264pfh.42.1649983431844; Thu, 14 Apr 2022 17:43:51 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 15 Apr 2022 00:43:43 +0000 In-Reply-To: <20220415004343.2203171-1-seanjc@google.com> Message-Id: <20220415004343.2203171-4-seanjc@google.com> Mime-Version: 1.0 References: <20220415004343.2203171-1-seanjc@google.com> X-Mailer: git-send-email 2.36.0.rc0.470.gd361397f0d-goog Subject: [PATCH 3/3] KVM: Add helpers to wrap vcpu->srcu_idx and yell if it's abused From: Sean Christopherson To: Anup Patel , Paul Walmsley , Palmer Dabbelt , Albert Ou , Christian Borntraeger , Janosch Frank , Claudio Imbrenda , Paolo Bonzini Cc: Atish Patra , David Hildenbrand , Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , linuxppc-dev@lists.ozlabs.org, kvm@vger.kernel.org, kvm-riscv@lists.infradead.org, linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add wrappers to acquire/release KVM's SRCU lock when stashing the index in vcpu->src_idx, along with rudimentary detection of illegal usage, e.g. re-acquiring SRCU and thus overwriting vcpu->src_idx. Because the SRCU index is (currently) either 0 or 1, illegal nesting bugs can go unnoticed for quite some time and only cause problems when the nested lock happens to get a different index. Wrap the WARNs in PROVE_RCU=3Dy, and make them ONCE, otherwise KVM will likely yell so loudly that it will bring the kernel to its knees. Signed-off-by: Sean Christopherson Tested-by: Fabiano Rosas --- arch/powerpc/kvm/book3s_64_mmu_radix.c | 9 +++++---- arch/powerpc/kvm/book3s_hv_nested.c | 16 +++++++-------- arch/powerpc/kvm/book3s_rtas.c | 4 ++-- arch/powerpc/kvm/powerpc.c | 4 ++-- arch/riscv/kvm/vcpu.c | 16 +++++++-------- arch/riscv/kvm/vcpu_exit.c | 4 ++-- arch/s390/kvm/interrupt.c | 4 ++-- arch/s390/kvm/kvm-s390.c | 8 ++++---- arch/s390/kvm/vsie.c | 4 ++-- arch/x86/kvm/x86.c | 28 ++++++++++++-------------- include/linux/kvm_host.h | 24 +++++++++++++++++++++- 11 files changed, 71 insertions(+), 50 deletions(-) diff --git a/arch/powerpc/kvm/book3s_64_mmu_radix.c b/arch/powerpc/kvm/book= 3s_64_mmu_radix.c index e4ce2a35483f..42851c32ff3b 100644 --- a/arch/powerpc/kvm/book3s_64_mmu_radix.c +++ b/arch/powerpc/kvm/book3s_64_mmu_radix.c @@ -168,9 +168,10 @@ int kvmppc_mmu_walk_radix_tree(struct kvm_vcpu *vcpu, = gva_t eaddr, return -EINVAL; /* Read the entry from guest memory */ addr =3D base + (index * sizeof(rpte)); - vcpu->srcu_idx =3D srcu_read_lock(&kvm->srcu); + + kvm_vcpu_srcu_read_lock(vcpu); ret =3D kvm_read_guest(kvm, addr, &rpte, sizeof(rpte)); - srcu_read_unlock(&kvm->srcu, vcpu->srcu_idx); + kvm_vcpu_srcu_read_unlock(vcpu); if (ret) { if (pte_ret_p) *pte_ret_p =3D addr; @@ -246,9 +247,9 @@ int kvmppc_mmu_radix_translate_table(struct kvm_vcpu *v= cpu, gva_t eaddr, =20 /* Read the table to find the root of the radix tree */ ptbl =3D (table & PRTB_MASK) + (table_index * sizeof(entry)); - vcpu->srcu_idx =3D srcu_read_lock(&kvm->srcu); + kvm_vcpu_srcu_read_lock(vcpu); ret =3D kvm_read_guest(kvm, ptbl, &entry, sizeof(entry)); - srcu_read_unlock(&kvm->srcu, vcpu->srcu_idx); + kvm_vcpu_srcu_read_unlock(vcpu); if (ret) return ret; =20 diff --git a/arch/powerpc/kvm/book3s_hv_nested.c b/arch/powerpc/kvm/book3s_= hv_nested.c index 9d373f8963ee..c943a051c6e7 100644 --- a/arch/powerpc/kvm/book3s_hv_nested.c +++ b/arch/powerpc/kvm/book3s_hv_nested.c @@ -306,10 +306,10 @@ long kvmhv_enter_nested_guest(struct kvm_vcpu *vcpu) /* copy parameters in */ hv_ptr =3D kvmppc_get_gpr(vcpu, 4); regs_ptr =3D kvmppc_get_gpr(vcpu, 5); - vcpu->srcu_idx =3D srcu_read_lock(&vcpu->kvm->srcu); + kvm_vcpu_srcu_read_lock(vcpu); err =3D kvmhv_read_guest_state_and_regs(vcpu, &l2_hv, &l2_regs, hv_ptr, regs_ptr); - srcu_read_unlock(&vcpu->kvm->srcu, vcpu->srcu_idx); + kvm_vcpu_srcu_read_unlock(vcpu); if (err) return H_PARAMETER; =20 @@ -410,10 +410,10 @@ long kvmhv_enter_nested_guest(struct kvm_vcpu *vcpu) byteswap_hv_regs(&l2_hv); byteswap_pt_regs(&l2_regs); } - vcpu->srcu_idx =3D srcu_read_lock(&vcpu->kvm->srcu); + kvm_vcpu_srcu_read_lock(vcpu); err =3D kvmhv_write_guest_state_and_regs(vcpu, &l2_hv, &l2_regs, hv_ptr, regs_ptr); - srcu_read_unlock(&vcpu->kvm->srcu, vcpu->srcu_idx); + kvm_vcpu_srcu_read_unlock(vcpu); if (err) return H_AUTHORITY; =20 @@ -600,16 +600,16 @@ long kvmhv_copy_tofrom_guest_nested(struct kvm_vcpu *= vcpu) goto not_found; =20 /* Write what was loaded into our buffer back to the L1 guest */ - vcpu->srcu_idx =3D srcu_read_lock(&vcpu->kvm->srcu); + kvm_vcpu_srcu_read_lock(vcpu); rc =3D kvm_vcpu_write_guest(vcpu, gp_to, buf, n); - srcu_read_unlock(&vcpu->kvm->srcu, vcpu->srcu_idx); + kvm_vcpu_srcu_read_unlock(vcpu); if (rc) goto not_found; } else { /* Load the data to be stored from the L1 guest into our buf */ - vcpu->srcu_idx =3D srcu_read_lock(&vcpu->kvm->srcu); + kvm_vcpu_srcu_read_lock(vcpu); rc =3D kvm_vcpu_read_guest(vcpu, gp_from, buf, n); - srcu_read_unlock(&vcpu->kvm->srcu, vcpu->srcu_idx); + kvm_vcpu_srcu_read_unlock(vcpu); if (rc) goto not_found; =20 diff --git a/arch/powerpc/kvm/book3s_rtas.c b/arch/powerpc/kvm/book3s_rtas.c index 0f847f1e5ddd..6808bda0dbc1 100644 --- a/arch/powerpc/kvm/book3s_rtas.c +++ b/arch/powerpc/kvm/book3s_rtas.c @@ -229,9 +229,9 @@ int kvmppc_rtas_hcall(struct kvm_vcpu *vcpu) */ args_phys =3D kvmppc_get_gpr(vcpu, 4) & KVM_PAM; =20 - vcpu->srcu_idx =3D srcu_read_lock(&vcpu->kvm->srcu); + kvm_vcpu_srcu_read_lock(vcpu); rc =3D kvm_read_guest(vcpu->kvm, args_phys, &args, sizeof(args)); - srcu_read_unlock(&vcpu->kvm->srcu, vcpu->srcu_idx); + kvm_vcpu_srcu_read_unlock(vcpu); if (rc) goto fail; =20 diff --git a/arch/powerpc/kvm/powerpc.c b/arch/powerpc/kvm/powerpc.c index 9772b176e406..fd88c2412e83 100644 --- a/arch/powerpc/kvm/powerpc.c +++ b/arch/powerpc/kvm/powerpc.c @@ -425,9 +425,9 @@ int kvmppc_ld(struct kvm_vcpu *vcpu, ulong *eaddr, int = size, void *ptr, return EMULATE_DONE; } =20 - vcpu->srcu_idx =3D srcu_read_lock(&vcpu->kvm->srcu); + kvm_vcpu_srcu_read_lock(vcpu); rc =3D kvm_read_guest(vcpu->kvm, pte.raddr, ptr, size); - srcu_read_unlock(&vcpu->kvm->srcu, vcpu->srcu_idx); + kvm_vcpu_srcu_read_unlock(vcpu); if (rc) return EMULATE_DO_MMIO; =20 diff --git a/arch/riscv/kvm/vcpu.c b/arch/riscv/kvm/vcpu.c index 2f1caf23eed4..256cf04ec01e 100644 --- a/arch/riscv/kvm/vcpu.c +++ b/arch/riscv/kvm/vcpu.c @@ -724,13 +724,13 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu) /* Mark this VCPU ran at least once */ vcpu->arch.ran_atleast_once =3D true; =20 - vcpu->srcu_idx =3D srcu_read_lock(&vcpu->kvm->srcu); + kvm_vcpu_srcu_read_lock(vcpu); =20 /* Process MMIO value returned from user-space */ if (run->exit_reason =3D=3D KVM_EXIT_MMIO) { ret =3D kvm_riscv_vcpu_mmio_return(vcpu, vcpu->run); if (ret) { - srcu_read_unlock(&vcpu->kvm->srcu, vcpu->srcu_idx); + kvm_vcpu_srcu_read_unlock(vcpu); return ret; } } @@ -739,13 +739,13 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu) if (run->exit_reason =3D=3D KVM_EXIT_RISCV_SBI) { ret =3D kvm_riscv_vcpu_sbi_return(vcpu, vcpu->run); if (ret) { - srcu_read_unlock(&vcpu->kvm->srcu, vcpu->srcu_idx); + kvm_vcpu_srcu_read_unlock(vcpu); return ret; } } =20 if (run->immediate_exit) { - srcu_read_unlock(&vcpu->kvm->srcu, vcpu->srcu_idx); + kvm_vcpu_srcu_read_unlock(vcpu); return -EINTR; } =20 @@ -784,7 +784,7 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu) */ vcpu->mode =3D IN_GUEST_MODE; =20 - srcu_read_unlock(&vcpu->kvm->srcu, vcpu->srcu_idx); + kvm_vcpu_srcu_read_unlock(vcpu); smp_mb__after_srcu_read_unlock(); =20 /* @@ -802,7 +802,7 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu) vcpu->mode =3D OUTSIDE_GUEST_MODE; local_irq_enable(); preempt_enable(); - vcpu->srcu_idx =3D srcu_read_lock(&vcpu->kvm->srcu); + kvm_vcpu_srcu_read_lock(vcpu); continue; } =20 @@ -846,7 +846,7 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu) =20 preempt_enable(); =20 - vcpu->srcu_idx =3D srcu_read_lock(&vcpu->kvm->srcu); + kvm_vcpu_srcu_read_lock(vcpu); =20 ret =3D kvm_riscv_vcpu_exit(vcpu, run, &trap); } @@ -855,7 +855,7 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu) =20 vcpu_put(vcpu); =20 - srcu_read_unlock(&vcpu->kvm->srcu, vcpu->srcu_idx); + kvm_vcpu_srcu_read_unlock(vcpu); =20 return ret; } diff --git a/arch/riscv/kvm/vcpu_exit.c b/arch/riscv/kvm/vcpu_exit.c index 2d56faddb9d1..a72c15d4b42a 100644 --- a/arch/riscv/kvm/vcpu_exit.c +++ b/arch/riscv/kvm/vcpu_exit.c @@ -456,9 +456,9 @@ static int stage2_page_fault(struct kvm_vcpu *vcpu, str= uct kvm_run *run, void kvm_riscv_vcpu_wfi(struct kvm_vcpu *vcpu) { if (!kvm_arch_vcpu_runnable(vcpu)) { - srcu_read_unlock(&vcpu->kvm->srcu, vcpu->srcu_idx); + kvm_vcpu_srcu_read_unlock(vcpu); kvm_vcpu_halt(vcpu); - vcpu->srcu_idx =3D srcu_read_lock(&vcpu->kvm->srcu); + kvm_vcpu_srcu_read_lock(vcpu); kvm_clear_request(KVM_REQ_UNHALT, vcpu); } } diff --git a/arch/s390/kvm/interrupt.c b/arch/s390/kvm/interrupt.c index 9b30beac904d..af96dc0549a4 100644 --- a/arch/s390/kvm/interrupt.c +++ b/arch/s390/kvm/interrupt.c @@ -1334,11 +1334,11 @@ int kvm_s390_handle_wait(struct kvm_vcpu *vcpu) hrtimer_start(&vcpu->arch.ckc_timer, sltime, HRTIMER_MODE_REL); VCPU_EVENT(vcpu, 4, "enabled wait: %llu ns", sltime); no_timer: - srcu_read_unlock(&vcpu->kvm->srcu, vcpu->srcu_idx); + kvm_vcpu_srcu_read_unlock(vcpu); kvm_vcpu_halt(vcpu); vcpu->valid_wakeup =3D false; __unset_cpu_idle(vcpu); - vcpu->srcu_idx =3D srcu_read_lock(&vcpu->kvm->srcu); + kvm_vcpu_srcu_read_lock(vcpu); =20 hrtimer_cancel(&vcpu->arch.ckc_timer); return 0; diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c index 156d1c25a3c1..da3dabda1a12 100644 --- a/arch/s390/kvm/kvm-s390.c +++ b/arch/s390/kvm/kvm-s390.c @@ -4237,14 +4237,14 @@ static int __vcpu_run(struct kvm_vcpu *vcpu) * We try to hold kvm->srcu during most of vcpu_run (except when run- * ning the guest), so that memslots (and other stuff) are protected */ - vcpu->srcu_idx =3D srcu_read_lock(&vcpu->kvm->srcu); + kvm_vcpu_srcu_read_lock(vcpu); =20 do { rc =3D vcpu_pre_run(vcpu); if (rc) break; =20 - srcu_read_unlock(&vcpu->kvm->srcu, vcpu->srcu_idx); + kvm_vcpu_srcu_read_unlock(vcpu); /* * As PF_VCPU will be used in fault handler, between * guest_enter and guest_exit should be no uaccess. @@ -4281,12 +4281,12 @@ static int __vcpu_run(struct kvm_vcpu *vcpu) __enable_cpu_timer_accounting(vcpu); guest_exit_irqoff(); local_irq_enable(); - vcpu->srcu_idx =3D srcu_read_lock(&vcpu->kvm->srcu); + kvm_vcpu_srcu_read_lock(vcpu); =20 rc =3D vcpu_post_run(vcpu, exit_reason); } while (!signal_pending(current) && !guestdbg_exit_pending(vcpu) && !rc); =20 - srcu_read_unlock(&vcpu->kvm->srcu, vcpu->srcu_idx); + kvm_vcpu_srcu_read_unlock(vcpu); return rc; } =20 diff --git a/arch/s390/kvm/vsie.c b/arch/s390/kvm/vsie.c index acda4b6fc851..dada78b92691 100644 --- a/arch/s390/kvm/vsie.c +++ b/arch/s390/kvm/vsie.c @@ -1091,7 +1091,7 @@ static int do_vsie_run(struct kvm_vcpu *vcpu, struct = vsie_page *vsie_page) =20 handle_last_fault(vcpu, vsie_page); =20 - srcu_read_unlock(&vcpu->kvm->srcu, vcpu->srcu_idx); + kvm_vcpu_srcu_read_unlock(vcpu); =20 /* save current guest state of bp isolation override */ guest_bp_isolation =3D test_thread_flag(TIF_ISOLATE_BP_GUEST); @@ -1133,7 +1133,7 @@ static int do_vsie_run(struct kvm_vcpu *vcpu, struct = vsie_page *vsie_page) if (!guest_bp_isolation) clear_thread_flag(TIF_ISOLATE_BP_GUEST); =20 - vcpu->srcu_idx =3D srcu_read_lock(&vcpu->kvm->srcu); + kvm_vcpu_srcu_read_lock(vcpu); =20 if (rc =3D=3D -EINTR) { VCPU_EVENT(vcpu, 3, "%s", "machine check"); diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index f35fe09de59d..bc24b35c4e80 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -10157,7 +10157,7 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu) /* Store vcpu->apicv_active before vcpu->mode. */ smp_store_release(&vcpu->mode, IN_GUEST_MODE); =20 - srcu_read_unlock(&vcpu->kvm->srcu, vcpu->srcu_idx); + kvm_vcpu_srcu_read_unlock(vcpu); =20 /* * 1) We should set ->mode before checking ->requests. Please see @@ -10188,7 +10188,7 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu) smp_wmb(); local_irq_enable(); preempt_enable(); - vcpu->srcu_idx =3D srcu_read_lock(&vcpu->kvm->srcu); + kvm_vcpu_srcu_read_lock(vcpu); r =3D 1; goto cancel_injection; } @@ -10314,7 +10314,7 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu) local_irq_enable(); preempt_enable(); =20 - vcpu->srcu_idx =3D srcu_read_lock(&vcpu->kvm->srcu); + kvm_vcpu_srcu_read_lock(vcpu); =20 /* * Profile KVM exit RIPs: @@ -10344,7 +10344,7 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu) } =20 /* Called within kvm->srcu read side. */ -static inline int vcpu_block(struct kvm *kvm, struct kvm_vcpu *vcpu) +static inline int vcpu_block(struct kvm_vcpu *vcpu) { bool hv_timer; =20 @@ -10360,12 +10360,12 @@ static inline int vcpu_block(struct kvm *kvm, str= uct kvm_vcpu *vcpu) if (hv_timer) kvm_lapic_switch_to_sw_timer(vcpu); =20 - srcu_read_unlock(&kvm->srcu, vcpu->srcu_idx); + kvm_vcpu_srcu_read_unlock(vcpu); if (vcpu->arch.mp_state =3D=3D KVM_MP_STATE_HALTED) kvm_vcpu_halt(vcpu); else kvm_vcpu_block(vcpu); - vcpu->srcu_idx =3D srcu_read_lock(&kvm->srcu); + kvm_vcpu_srcu_read_lock(vcpu); =20 if (hv_timer) kvm_lapic_switch_to_hv_timer(vcpu); @@ -10407,7 +10407,6 @@ static inline bool kvm_vcpu_running(struct kvm_vcpu= *vcpu) static int vcpu_run(struct kvm_vcpu *vcpu) { int r; - struct kvm *kvm =3D vcpu->kvm; =20 vcpu->arch.l1tf_flush_l1d =3D true; =20 @@ -10415,7 +10414,7 @@ static int vcpu_run(struct kvm_vcpu *vcpu) if (kvm_vcpu_running(vcpu)) { r =3D vcpu_enter_guest(vcpu); } else { - r =3D vcpu_block(kvm, vcpu); + r =3D vcpu_block(vcpu); } =20 if (r <=3D 0) @@ -10437,9 +10436,9 @@ static int vcpu_run(struct kvm_vcpu *vcpu) } =20 if (__xfer_to_guest_mode_work_pending()) { - srcu_read_unlock(&kvm->srcu, vcpu->srcu_idx); + kvm_vcpu_srcu_read_unlock(vcpu); r =3D xfer_to_guest_mode_handle_work(vcpu); - vcpu->srcu_idx =3D srcu_read_lock(&kvm->srcu); + kvm_vcpu_srcu_read_lock(vcpu); if (r) return r; } @@ -10542,7 +10541,6 @@ static void kvm_put_guest_fpu(struct kvm_vcpu *vcpu) int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu) { struct kvm_run *kvm_run =3D vcpu->run; - struct kvm *kvm =3D vcpu->kvm; int r; =20 vcpu_load(vcpu); @@ -10550,7 +10548,7 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu) kvm_run->flags =3D 0; kvm_load_guest_fpu(vcpu); =20 - vcpu->srcu_idx =3D srcu_read_lock(&vcpu->kvm->srcu); + kvm_vcpu_srcu_read_lock(vcpu); if (unlikely(vcpu->arch.mp_state =3D=3D KVM_MP_STATE_UNINITIALIZED)) { if (kvm_run->immediate_exit) { r =3D -EINTR; @@ -10562,9 +10560,9 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu) */ WARN_ON_ONCE(kvm_lapic_hv_timer_in_use(vcpu)); =20 - srcu_read_unlock(&kvm->srcu, vcpu->srcu_idx); + kvm_vcpu_srcu_read_unlock(vcpu); kvm_vcpu_block(vcpu); - vcpu->srcu_idx =3D srcu_read_lock(&kvm->srcu); + kvm_vcpu_srcu_read_lock(vcpu); =20 if (kvm_apic_accept_events(vcpu) < 0) { r =3D 0; @@ -10625,7 +10623,7 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu) if (kvm_run->kvm_valid_regs) store_regs(vcpu); post_kvm_run_save(vcpu); - srcu_read_unlock(&kvm->srcu, vcpu->srcu_idx); + kvm_vcpu_srcu_read_unlock(vcpu); =20 kvm_sigset_deactivate(vcpu); vcpu_put(vcpu); diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h index 252ee4a61b58..76fc7233bd6a 100644 --- a/include/linux/kvm_host.h +++ b/include/linux/kvm_host.h @@ -315,7 +315,10 @@ struct kvm_vcpu { int cpu; int vcpu_id; /* id given by userspace at creation */ int vcpu_idx; /* index in kvm->vcpus array */ - int srcu_idx; + int ____srcu_idx; /* Don't use this directly. You've been warned. */ +#ifdef CONFIG_PROVE_RCU + int srcu_depth; +#endif int mode; u64 requests; unsigned long guest_debug; @@ -841,6 +844,25 @@ static inline void kvm_vm_bugged(struct kvm *kvm) unlikely(__ret); \ }) =20 +static inline void kvm_vcpu_srcu_read_lock(struct kvm_vcpu *vcpu) +{ +#ifdef CONFIG_PROVE_RCU + WARN_ONCE(vcpu->srcu_depth++, + "KVM: Illegal vCPU srcu_idx LOCK, depth=3D%d", vcpu->srcu_depth - 1); +#endif + vcpu->____srcu_idx =3D srcu_read_lock(&vcpu->kvm->srcu); +} + +static inline void kvm_vcpu_srcu_read_unlock(struct kvm_vcpu *vcpu) +{ + srcu_read_unlock(&vcpu->kvm->srcu, vcpu->____srcu_idx); + +#ifdef CONFIG_PROVE_RCU + WARN_ONCE(--vcpu->srcu_depth, + "KVM: Illegal vCPU srcu_idx UNLOCK, depth=3D%d", vcpu->srcu_depth); +#endif +} + static inline bool kvm_dirty_log_manual_protect_and_init_set(struct kvm *k= vm) { return !!(kvm->manual_dirty_log_protect & KVM_DIRTY_LOG_INITIALLY_SET); --=20 2.36.0.rc0.470.gd361397f0d-goog