From nobody Tue Apr 7 12:07:50 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C92E2C0502F for ; Sun, 28 Aug 2022 22:25:56 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229561AbiH1WZy (ORCPT ); Sun, 28 Aug 2022 18:25:54 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40318 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229527AbiH1WZu (ORCPT ); Sun, 28 Aug 2022 18:25:50 -0400 Received: from mail-pl1-x64a.google.com (mail-pl1-x64a.google.com [IPv6:2607:f8b0:4864:20::64a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D2EF213FB7 for ; Sun, 28 Aug 2022 15:25:48 -0700 (PDT) Received: by mail-pl1-x64a.google.com with SMTP id d6-20020a170902cec600b00174be1616c4so1031523plg.22 for ; Sun, 28 Aug 2022 15:25:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc; bh=bJzhK9SKVOQc70YSCOsmEBRTVrANtgpX4ADnFqlNU4E=; b=CWAFJqwl6pgZp8h7EYlQRaYnt/lP0OKs0B8JSfoq0n4dQD7VKjtlhXPBHwkb3kP5rm fsbrh66CcP9AAtVteRxv8bhFS619geeXipmihzDvR+k/7ICaJ06CaOhK1G1jsfQJuIsO Z/zCU1fb9+iPNg5vBeSBNWolfdla/C7X/1FA9Sv2s18eXVV5yg9IkpTgglr9vMKpRnUx VrzEBtVcdrkEeuYvQ+XnM4xNBAHgNVta/ClOwEREQhaIHimQ/NXLzjshOteCBjHsG98o ZCzHVpDInEm4rVfQRw/Itg0jloMDWCUp2fqgKveNcMGmQO45FsJzijLD3Oh7WqW/mAPv N12g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc; bh=bJzhK9SKVOQc70YSCOsmEBRTVrANtgpX4ADnFqlNU4E=; b=UHMfoPFzM766g7vAvG4mwc0sbM5+4BiRPf9NBFPgHF9EzE4WHx1plk4hfbhoP3QoXx uhM/tTrtDONztCCasRWu23Y96/5kWrIvqsYL48Ps++G3uY2+UM1ENC7FuPLyfb9AtQgl Hd3xnHybtdYtQGf5egyDWNZ2IxDAC5d3UCHF1IcAmyKwAQOvX7hOwMGCoWekIZg/AWGc z7tVVlK5emzZI9L+5FUgowZv22cR9W9CzuKiqjpRpfWIo2ORdch/CK33h/sICri8V9EA IxcaxTuItVU6s3Rmhb+/jDHA1F/yblLTzopGHovQWYK/lZCNNpyFss5Utpy2+4oLHmsR zOJA== X-Gm-Message-State: ACgBeo11sw/N3kmbTNmiONaEiM0ALDdYM868SURMTGAG54q6ZJ9r91Ej k6JM0s67fELvkOryoUKzuuxUp3dwQIru X-Google-Smtp-Source: AA6agR6aj2JFtODDYyHrp7CN/D0UPfsbQsmThliJrQuKeU+iU4k3Pg9cYu+ZkfDelt/t/B/ejt2Cao/HT200 X-Received: from mizhang-super.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:1071]) (user=mizhang job=sendgmr) by 2002:a63:81c6:0:b0:42b:c3fa:3a0 with SMTP id t189-20020a6381c6000000b0042bc3fa03a0mr5326593pgd.72.1661725548439; Sun, 28 Aug 2022 15:25:48 -0700 (PDT) Reply-To: Mingwei Zhang Date: Sun, 28 Aug 2022 22:25:41 +0000 In-Reply-To: <20220828222544.1964917-1-mizhang@google.com> Mime-Version: 1.0 References: <20220828222544.1964917-1-mizhang@google.com> X-Mailer: git-send-email 2.37.2.672.g94769d06f0-goog Message-ID: <20220828222544.1964917-2-mizhang@google.com> Subject: [PATCH v2 1/4] KVM: x86: move the event handling of KVM_REQ_GET_VMCS12_PAGES into a common function From: Mingwei Zhang To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Maxim Levitsky , Vitaly Kuznetsov , Oliver Upton , Mingwei Zhang , Jim Mattson Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Create a common function to handle kvm request in the vcpu_run loop. KVM implicitly assumes the virtual APIC page being present + mapped into the kernel address space when executing vmx_guest_apic_has_interrupts(). However, with demand paging KVM breaks the assumption, as the KVM_REQ_GET_VMCS12_PAGES event isn't assessed before entering vcpu_block. Fix this by getting vmcs12 pages before inspecting the guest's APIC page. Because of this fix, the event handling code of KVM_REQ_GET_NESTED_STATE_PAGES becomes a common code path for both vcpu_enter_guest() and vcpu_block(). Thus, put this code snippet into a common helper function to avoid code duplication. Cc: Maxim Levitsky Cc: Vitaly Kuznetsov Originally-by: Oliver Upton Signed-off-by: Oliver Upton Signed-off-by: Mingwei Zhang --- arch/x86/kvm/x86.c | 29 +++++++++++++++++++++++------ 1 file changed, 23 insertions(+), 6 deletions(-) diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index d7374d768296..3dcaac8f0584 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -10261,12 +10261,6 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu) r =3D -EIO; goto out; } - if (kvm_check_request(KVM_REQ_GET_NESTED_STATE_PAGES, vcpu)) { - if (unlikely(!kvm_x86_ops.nested_ops->get_nested_state_pages(vcpu))) { - r =3D 0; - goto out; - } - } if (kvm_check_request(KVM_REQ_MMU_FREE_OBSOLETE_ROOTS, vcpu)) kvm_mmu_free_obsolete_roots(vcpu); if (kvm_check_request(KVM_REQ_MIGRATE_TIMER, vcpu)) @@ -10666,6 +10660,23 @@ static inline bool kvm_vcpu_running(struct kvm_vcp= u *vcpu) !vcpu->arch.apf.halted); } =20 +static int kvm_vcpu_handle_common_requests(struct kvm_vcpu *vcpu) +{ + if (kvm_request_pending(vcpu)) { + /* + * Get the vmcs12 pages before checking for interrupts that + * might unblock the guest if L1 is using virtual-interrupt + * delivery. + */ + if (kvm_check_request(KVM_REQ_GET_NESTED_STATE_PAGES, vcpu)) { + if (unlikely(!kvm_x86_ops.nested_ops->get_nested_state_pages(vcpu))) + return 0; + } + } + + return 1; +} + /* Called within kvm->srcu read side. */ static int vcpu_run(struct kvm_vcpu *vcpu) { @@ -10681,6 +10692,12 @@ static int vcpu_run(struct kvm_vcpu *vcpu) * this point can start executing an instruction. */ vcpu->arch.at_instruction_boundary =3D false; + + /* Process common request regardless of vcpu state. */ + r =3D kvm_vcpu_handle_common_requests(vcpu); + if (r <=3D 0) + break; + if (kvm_vcpu_running(vcpu)) { r =3D vcpu_enter_guest(vcpu); } else { --=20 2.37.2.672.g94769d06f0-goog From nobody Tue Apr 7 12:07:50 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id EC3D8C0502C for ; Sun, 28 Aug 2022 22:26:00 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229494AbiH1WZ6 (ORCPT ); Sun, 28 Aug 2022 18:25:58 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40450 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229509AbiH1WZw (ORCPT ); Sun, 28 Aug 2022 18:25:52 -0400 Received: from mail-pj1-x1049.google.com (mail-pj1-x1049.google.com [IPv6:2607:f8b0:4864:20::1049]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CBFD814085 for ; Sun, 28 Aug 2022 15:25:50 -0700 (PDT) Received: by mail-pj1-x1049.google.com with SMTP id d16-20020a17090ad3d000b001fb42eb7467so2371497pjw.9 for ; Sun, 28 Aug 2022 15:25:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc; bh=+0YcUxl6w5S43rp5xSvLoFsoOQeX/6nXvUKDRMGuY+4=; b=lc30ffSB8YQZjEbZDqmcgbW3ZeNzpXVKwnW2YhSZLRRf+SIgS+ivUaMd7Vhc4TnTOe btrHLR5JniiAprJhu+IXljXcomK14NYed3aaauIkPwiAm2OSKgOZokFUYFoSCFI4sM27 SRoJNeIjth14/1aBxa9a4WE0zUNbDhb00IPoRdn+hvotJJ3fGctGjjkrogEgqdv1gDsQ mDjATNizqZ/tBQD28YiT8ry7U+dweH2IyZzZNeuPEoZlYVJUvefancxIZiRDsagXctp7 I+k+CHDYXmYO45zK8xNn0dATkQNHqRb09Q9ka0aeuvwcuILbza9w8evzMPTP6h0ufDyL k53A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc; bh=+0YcUxl6w5S43rp5xSvLoFsoOQeX/6nXvUKDRMGuY+4=; b=JvuxLGaCSX508u5jad/678jVHRGgHDR0WYCSpZ6YN5SNtmB9SeTdk7zDDguvFE3Wxz 8fzjjF6I+ppW0jsb/8M3C0swJlukKHg40AWve9eM/o66JZ6u8uJNo/1yltiMJ41QKl7S Hu0W9IfwFVOnOjwu/m3LAizn1/vHu5slgEYFShBeXbbnIpZlx36BmLhlQc0M40H9R7lS CTAtsH0j2drFrL4vu20A0bgUkCJ03ekmRK26W9/ear+NUdQnZz3momMkFUzXtuUXZpRd 3s+gJTYg4vw0Y8i1vFRinfpxwNq/URtudLdrB9sctb3c63VEIHKNqxT+4K9gN7V0NeXK tJ5Q== X-Gm-Message-State: ACgBeo0ovvqLMzwo+Ab2My2yuASS/G4a8vsO5U5ysLkEfB5o3UvHEAmO IzQHEe5BhDlCxZFesTCKtP/xanr9CY9i X-Google-Smtp-Source: AA6agR6VtT2c2EjZ/labJtCIac4RDvO7XzQLXT1mkuGrZeZDh8HGBq2GIyvbrQmwulc75D7KaUM48UUYry/k X-Received: from mizhang-super.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:1071]) (user=mizhang job=sendgmr) by 2002:a17:90b:3807:b0:1f4:ecf7:5987 with SMTP id mq7-20020a17090b380700b001f4ecf75987mr15014728pjb.13.1661725549891; Sun, 28 Aug 2022 15:25:49 -0700 (PDT) Reply-To: Mingwei Zhang Date: Sun, 28 Aug 2022 22:25:42 +0000 In-Reply-To: <20220828222544.1964917-1-mizhang@google.com> Mime-Version: 1.0 References: <20220828222544.1964917-1-mizhang@google.com> X-Mailer: git-send-email 2.37.2.672.g94769d06f0-goog Message-ID: <20220828222544.1964917-3-mizhang@google.com> Subject: [PATCH v2 2/4] KVM: selftests: Save/restore vAPIC state in migration tests From: Mingwei Zhang To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Maxim Levitsky , Vitaly Kuznetsov , Oliver Upton , Mingwei Zhang , Jim Mattson Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Save/restore vAPIC state as part of vCPU save/load so that it is preserved across VM (copyless) migration. This wil allow testing the posted interrupts are properly handled across VM migration. Cc: Jim Mattson Signed-off-by: Mingwei Zhang --- tools/testing/selftests/kvm/include/kvm_util_base.h | 10 ++++++++++ tools/testing/selftests/kvm/include/x86_64/processor.h | 1 + tools/testing/selftests/kvm/lib/x86_64/processor.c | 2 ++ 3 files changed, 13 insertions(+) diff --git a/tools/testing/selftests/kvm/include/kvm_util_base.h b/tools/te= sting/selftests/kvm/include/kvm_util_base.h index 24fde97f6121..ac883b8eab57 100644 --- a/tools/testing/selftests/kvm/include/kvm_util_base.h +++ b/tools/testing/selftests/kvm/include/kvm_util_base.h @@ -457,6 +457,16 @@ static inline void vcpu_fpu_set(struct kvm_vcpu *vcpu,= struct kvm_fpu *fpu) vcpu_ioctl(vcpu, KVM_SET_FPU, fpu); } =20 +static inline void vcpu_apic_get(struct kvm_vcpu *vcpu, struct kvm_lapic_s= tate *apic) +{ + vcpu_ioctl(vcpu, KVM_GET_LAPIC, apic); +} + +static inline void vcpu_apic_set(struct kvm_vcpu *vcpu, struct kvm_lapic_s= tate *apic) +{ + vcpu_ioctl(vcpu, KVM_SET_LAPIC, apic); +} + static inline int __vcpu_get_reg(struct kvm_vcpu *vcpu, uint64_t id, void = *addr) { struct kvm_one_reg reg =3D { .id =3D id, .addr =3D (uint64_t)addr }; diff --git a/tools/testing/selftests/kvm/include/x86_64/processor.h b/tools= /testing/selftests/kvm/include/x86_64/processor.h index 0cbc71b7af50..102a56a60652 100644 --- a/tools/testing/selftests/kvm/include/x86_64/processor.h +++ b/tools/testing/selftests/kvm/include/x86_64/processor.h @@ -225,6 +225,7 @@ struct kvm_x86_state { struct kvm_nested_state nested; char nested_[16384]; }; + struct kvm_lapic_state apic; struct kvm_msrs msrs; }; =20 diff --git a/tools/testing/selftests/kvm/lib/x86_64/processor.c b/tools/tes= ting/selftests/kvm/lib/x86_64/processor.c index 2e6e61bbe81b..e22b4f0e24f1 100644 --- a/tools/testing/selftests/kvm/lib/x86_64/processor.c +++ b/tools/testing/selftests/kvm/lib/x86_64/processor.c @@ -980,6 +980,7 @@ struct kvm_x86_state *vcpu_save_state(struct kvm_vcpu *= vcpu) vcpu_msrs_get(vcpu, &state->msrs); =20 vcpu_debugregs_get(vcpu, &state->debugregs); + vcpu_apic_get(vcpu, &state->apic); =20 return state; } @@ -997,6 +998,7 @@ void vcpu_load_state(struct kvm_vcpu *vcpu, struct kvm_= x86_state *state) vcpu_mp_state_set(vcpu, &state->mp_state); vcpu_debugregs_set(vcpu, &state->debugregs); vcpu_regs_set(vcpu, &state->regs); + vcpu_apic_set(vcpu, &state->apic); =20 if (state->nested.size) vcpu_nested_state_set(vcpu, &state->nested); --=20 2.37.2.672.g94769d06f0-goog From nobody Tue Apr 7 12:07:50 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B1DBEECAAA2 for ; Sun, 28 Aug 2022 22:26:04 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229587AbiH1W0D (ORCPT ); Sun, 28 Aug 2022 18:26:03 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40488 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229544AbiH1WZw (ORCPT ); Sun, 28 Aug 2022 18:25:52 -0400 Received: from mail-pl1-x64a.google.com (mail-pl1-x64a.google.com [IPv6:2607:f8b0:4864:20::64a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A1A0E140BB for ; Sun, 28 Aug 2022 15:25:51 -0700 (PDT) Received: by mail-pl1-x64a.google.com with SMTP id a13-20020a170902eccd00b001730da9d40fso5037703plh.10 for ; Sun, 28 Aug 2022 15:25:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc; bh=O9kmISZGgihA4T8t28vEwYfV0dJn2c/+twbqczubmvE=; b=odN/wZEzKPDjrYtsqZpqQlo5DyO9yPziqUidIynlB4MzlR/AbwMMPfknzajSUH6HNN mrqpnrQLzHkNGzSGpAYvBoeMhpzSY3k0kOhV783lhGcKKFp/BbwngUm6DH6td6KOGU0Q Ri0FLTaKIIyyZAKTB3pexllFa7Kiv6mLBJf8Ju4W/W+L1pHV3dLn/t5mUMuHjQda+OkN sL5Q06Ga2C2Ik90dee0/qM7Aiw5AmmfTZt3qzEwcWAlI02tof5ncdNxfojTZheRk2MW6 cO2LtinBWxi887wwh0EZBH/vN0tI0K7Xamm5gUrk/sogCaelnm1/K1dGKvrnV1KWvRgx 2Xhw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc; bh=O9kmISZGgihA4T8t28vEwYfV0dJn2c/+twbqczubmvE=; b=aYfXU1jrvEGvoxN1miM8UK74s5abNoHT2WKhFAdC/0SNeMD8ApLXd0yeS86XZAQO0s 83HNdEX9oMesLBHueMADPuyuaGrrIYRaR2SkAvHD83GZj2+bFw4DegmROZ8nfh54kIhh OGM0dmH98GXKZYdF+OpMZEbRmKjD9cjfo+ywKUvivAlVAc8Npy7zAqOfzXlEUtPEB28N cMSFTAEjHISrvOQWEV8jrsInqcU1OhmW4eZcvbhDGTqf1dxaZKFjcvsxelmb0NIUEgw4 X7gO4yu9i3ldSfS6aoFCbKPj9cXOEEWpuQbKy4FqBvNJ3oHpRxKlUt9d9Y9zioZu6Izm KwTg== X-Gm-Message-State: ACgBeo28eAwd2kkF6Bhvb91mh4vyhPNyM8yltf2Vpxzeg2GrHBKLiX+9 46RbQVLfJ9ednQo0/4j64E/THY4JA2mH X-Google-Smtp-Source: AA6agR5A7jAZuGYFUVtouKXe1LOefqg+qPyGJ5Bwh+pnBY3tczIAGKNOk2pmuuyUa2uP/OofQYh6rQccmXGO X-Received: from mizhang-super.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:1071]) (user=mizhang job=sendgmr) by 2002:a05:6a00:1345:b0:536:d73a:4392 with SMTP id k5-20020a056a00134500b00536d73a4392mr13739160pfu.6.1661725551230; Sun, 28 Aug 2022 15:25:51 -0700 (PDT) Reply-To: Mingwei Zhang Date: Sun, 28 Aug 2022 22:25:43 +0000 In-Reply-To: <20220828222544.1964917-1-mizhang@google.com> Mime-Version: 1.0 References: <20220828222544.1964917-1-mizhang@google.com> X-Mailer: git-send-email 2.37.2.672.g94769d06f0-goog Message-ID: <20220828222544.1964917-4-mizhang@google.com> Subject: [PATCH v2 3/4] KVM: selftests: Add support for posted interrupt handling in L2 From: Mingwei Zhang To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Maxim Levitsky , Vitaly Kuznetsov , Oliver Upton , Mingwei Zhang , Jim Mattson Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add support for posted interrupt handling in L2. This is done by adding needed data structures in vmx_pages and APIs to allow an L2 receive posted interrupts. Cc: Jim Mattson Signed-off-by: Mingwei Zhang --- tools/testing/selftests/kvm/include/x86_64/vmx.h | 10 ++++++++++ tools/testing/selftests/kvm/lib/x86_64/vmx.c | 14 ++++++++++++++ 2 files changed, 24 insertions(+) diff --git a/tools/testing/selftests/kvm/include/x86_64/vmx.h b/tools/testi= ng/selftests/kvm/include/x86_64/vmx.h index 99fa1410964c..69784fc71bce 100644 --- a/tools/testing/selftests/kvm/include/x86_64/vmx.h +++ b/tools/testing/selftests/kvm/include/x86_64/vmx.h @@ -577,6 +577,14 @@ struct vmx_pages { void *apic_access_hva; uint64_t apic_access_gpa; void *apic_access; + + void *virtual_apic_hva; + uint64_t virtual_apic_gpa; + void *virtual_apic; + + void *posted_intr_desc_hva; + uint64_t posted_intr_desc_gpa; + void *posted_intr_desc; }; =20 union vmx_basic { @@ -620,5 +628,7 @@ void nested_identity_map_1g(struct vmx_pages *vmx, stru= ct kvm_vm *vm, void prepare_eptp(struct vmx_pages *vmx, struct kvm_vm *vm, uint32_t eptp_memslot); void prepare_virtualize_apic_accesses(struct vmx_pages *vmx, struct kvm_vm= *vm); +void prepare_virtual_apic(struct vmx_pages *vmx, struct kvm_vm *vm); +void prepare_posted_intr_desc(struct vmx_pages *vmx, struct kvm_vm *vm); =20 #endif /* SELFTEST_KVM_VMX_H */ diff --git a/tools/testing/selftests/kvm/lib/x86_64/vmx.c b/tools/testing/s= elftests/kvm/lib/x86_64/vmx.c index 80a568c439b8..47ae419d7eb1 100644 --- a/tools/testing/selftests/kvm/lib/x86_64/vmx.c +++ b/tools/testing/selftests/kvm/lib/x86_64/vmx.c @@ -556,3 +556,17 @@ void prepare_virtualize_apic_accesses(struct vmx_pages= *vmx, struct kvm_vm *vm) vmx->apic_access_hva =3D addr_gva2hva(vm, (uintptr_t)vmx->apic_access); vmx->apic_access_gpa =3D addr_gva2gpa(vm, (uintptr_t)vmx->apic_access); } + +void prepare_virtual_apic(struct vmx_pages *vmx, struct kvm_vm *vm) +{ + vmx->virtual_apic =3D (void *)vm_vaddr_alloc_page(vm); + vmx->virtual_apic_hva =3D addr_gva2hva(vm, (uintptr_t)vmx->virtual_apic); + vmx->virtual_apic_gpa =3D addr_gva2gpa(vm, (uintptr_t)vmx->virtual_apic); +} + +void prepare_posted_intr_desc(struct vmx_pages *vmx, struct kvm_vm *vm) +{ + vmx->posted_intr_desc =3D (void *)vm_vaddr_alloc_page(vm); + vmx->posted_intr_desc_hva =3D addr_gva2hva(vm, vmx->posted_intr_desc); + vmx->posted_intr_desc_gpa =3D addr_gva2gpa(vm, vmx->posted_intr_desc); +} --=20 2.37.2.672.g94769d06f0-goog From nobody Tue Apr 7 12:07:50 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 39A26ECAAA2 for ; Sun, 28 Aug 2022 22:26:08 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229619AbiH1W0G (ORCPT ); Sun, 28 Aug 2022 18:26:06 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40744 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229577AbiH1WZ5 (ORCPT ); Sun, 28 Aug 2022 18:25:57 -0400 Received: from mail-pj1-x1049.google.com (mail-pj1-x1049.google.com [IPv6:2607:f8b0:4864:20::1049]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8205E140BC for ; Sun, 28 Aug 2022 15:25:53 -0700 (PDT) Received: by mail-pj1-x1049.google.com with SMTP id lb5-20020a17090b4a4500b001fd605b6e82so2359554pjb.3 for ; Sun, 28 Aug 2022 15:25:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc; bh=VvwS1hKq6yqPBAQMZ/469wEe25X0TK/1Sfxh6oNBXNI=; b=D1bOzxTxYBwv0//uzSojw76NeI27fzkhmpi/ONn1ji473Sck6tjlqdKQbNvXgC5x9r fCyCA4rmRySzvdGwbr1piTtEfUpKhuDHvNtyI+I5gq6xXeuIIpukgRQ7zWyG4ziSxXSP ty9pEuFWk5v65wvNSnHycdbVKgG0S3XNcRIrpETtFEl2Hm/AbQQAWv6lhBk9/qetp03n 3r9DQDjlOUCyvZN8hT9snGR39qeEWpRA4K8tZDEyOhIawom5woQwmO2JoM6Qff8gAZq6 QLugN6lQGq6zKVC5csQmoU5nQFhQKVLnGGDqvYNsqSUW625Sf9E72w7RtqAe1/4s85vR 5WYQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc; bh=VvwS1hKq6yqPBAQMZ/469wEe25X0TK/1Sfxh6oNBXNI=; b=bS7+TQVlcXBIbfQJDfqbZrbTXcDcZCrh5PQSk4yCuiJUvz/mJXdOUm+ei/M+xeGZcP w89MfpW6D8km6CanJX3sD5qMA5g7ugxPDtTX+Y6y6EkmOjj4503v237+XTHdaQ3BS9tu cLi8KBCtMQSu+J57VaCKQnOFDjSxQbWdAiiG9qjt36mto4uJHuZMxBE9EeKkPNgBA+eO MKwcQVP4sVb7MQVD0f1i/qY3wvR8oq+MxUwO/aZe+koLcDrGmYiCookPFE+8qONjmWW7 uiUMW/gJVlSjT4eZtgpyhRAGUQth/Z1eQ/rz0sn/RG99IirmG7Hhg2NzRd5tsxHqhTCG +F3Q== X-Gm-Message-State: ACgBeo358WstGQBU+RdxanAXyaBhlzqLNKcBlheAcXjUWFRPSn2UiCoM qBoyLkdnpjBbfmNI2BckQ6ozzpV3+zuw X-Google-Smtp-Source: AA6agR54uhN6vlDiV0Ks9R1JiJyZpt2hjsw5CAE5/JqnOlYTVp0ssKihJ15GpKXRvNWqBb1L3qCXUnGVmQ6h X-Received: from mizhang-super.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:1071]) (user=mizhang job=sendgmr) by 2002:a17:90b:4c4a:b0:1fb:3c1b:b7f8 with SMTP id np10-20020a17090b4c4a00b001fb3c1bb7f8mr15345016pjb.98.1661725552893; Sun, 28 Aug 2022 15:25:52 -0700 (PDT) Reply-To: Mingwei Zhang Date: Sun, 28 Aug 2022 22:25:44 +0000 In-Reply-To: <20220828222544.1964917-1-mizhang@google.com> Mime-Version: 1.0 References: <20220828222544.1964917-1-mizhang@google.com> X-Mailer: git-send-email 2.37.2.672.g94769d06f0-goog Message-ID: <20220828222544.1964917-5-mizhang@google.com> Subject: [PATCH v2 4/4] KVM: selftests: Test if posted interrupt delivery race with migration From: Mingwei Zhang To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Maxim Levitsky , Vitaly Kuznetsov , Oliver Upton , Mingwei Zhang , Jim Mattson Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Jim Mattson Test if posted interrupt delivery race with migration. Add a selftest to demonstrate a race condition between migration and posted interrupt for a nested VM. The consequence of this race condition causes the loss of a posted interrupt for a nested vCPU after migration and triggers a warning for unpatched kernel. The selftest demonstrates that if a L2 vCPU is in halted state before migration, then after migration, it is not able to receive a posted interrupt from another vCPU within the same VM. The fundamental problem is deeply buried in the kernel logic where vcpu_block() will directly check vmcs12 related mappings before having a valid vmcs12 ready. Because of that, it fails to process the posted interrupt and triggers the warning in vmx_guest_apic_has_interrupt() static bool vmx_guest_apic_has_interrupt(struct kvm_vcpu *vcpu) { ... if (WARN_ON_ONCE(!is_guest_mode(vcpu)) || !nested_cpu_has_vid(get_vmcs12(vcpu)) || WARN_ON_ONCE(!vmx->nested.virtual_apic_map.gfn)) <=3D HERE return false; ... } Signed-off-by: Jim Mattson Co-developed-by: Mingwei Zhang Signed-off-by: Mingwei Zhang --- tools/testing/selftests/kvm/.gitignore | 1 + tools/testing/selftests/kvm/Makefile | 1 + .../kvm/x86_64/vmx_migrate_pi_pending.c | 291 ++++++++++++++++++ 3 files changed, 293 insertions(+) create mode 100644 tools/testing/selftests/kvm/x86_64/vmx_migrate_pi_pendi= ng.c diff --git a/tools/testing/selftests/kvm/.gitignore b/tools/testing/selftes= ts/kvm/.gitignore index d625a3f83780..749b2be5b23c 100644 --- a/tools/testing/selftests/kvm/.gitignore +++ b/tools/testing/selftests/kvm/.gitignore @@ -51,6 +51,7 @@ /x86_64/vmx_exception_with_invalid_guest_state /x86_64/vmx_invalid_nested_guest_state /x86_64/vmx_msrs_test +/x86_64/vmx_migrate_pi_pending /x86_64/vmx_preemption_timer_test /x86_64/vmx_set_nested_state_test /x86_64/vmx_tsc_adjust_test diff --git a/tools/testing/selftests/kvm/Makefile b/tools/testing/selftests= /kvm/Makefile index 4c122f1b1737..5e830c65c068 100644 --- a/tools/testing/selftests/kvm/Makefile +++ b/tools/testing/selftests/kvm/Makefile @@ -110,6 +110,7 @@ TEST_GEN_PROGS_x86_64 +=3D x86_64/vmx_dirty_log_test TEST_GEN_PROGS_x86_64 +=3D x86_64/vmx_exception_with_invalid_guest_state TEST_GEN_PROGS_x86_64 +=3D x86_64/vmx_msrs_test TEST_GEN_PROGS_x86_64 +=3D x86_64/vmx_invalid_nested_guest_state +TEST_GEN_PROGS_x86_64 +=3D x86_64/vmx_migrate_pi_pending TEST_GEN_PROGS_x86_64 +=3D x86_64/vmx_set_nested_state_test TEST_GEN_PROGS_x86_64 +=3D x86_64/vmx_tsc_adjust_test TEST_GEN_PROGS_x86_64 +=3D x86_64/vmx_nested_tsc_scaling_test diff --git a/tools/testing/selftests/kvm/x86_64/vmx_migrate_pi_pending.c b/= tools/testing/selftests/kvm/x86_64/vmx_migrate_pi_pending.c new file mode 100644 index 000000000000..6f1a9f284754 --- /dev/null +++ b/tools/testing/selftests/kvm/x86_64/vmx_migrate_pi_pending.c @@ -0,0 +1,291 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * vmx_migrate_pi_pending + * + * Copyright (C) 2022, Google, LLC. + * + * Deliver a nested posted interrupt between migration and the first + * KVM_RUN on the target. + */ + +#include "test_util.h" +#include "kvm_util.h" +#include "processor.h" +#include "vmx.h" + +#include +#include +#include +#include +#include + +#include "kselftest.h" + +#define VCPU_ID0 0 +#define VCPU_ID1 1 +#define PI_ON_BIT 256 +#define PI_NV 0x42 +#define L2_INTR 0x71 + +enum { + PORT_L0_EXIT =3D 0x2000, +}; + +/* The virtual machine object. */ +struct vmx_pages *vmx; + +static struct kvm_vm *vm; +static struct kvm_vcpu *vcpu0; +static struct kvm_vcpu *vcpu1; +bool vcpu0_can_run =3D true; +bool vcpu0_running; +bool pi_executed; +pthread_t pthread_cpu0; +pthread_t pthread_cpu1; + +static void vcpu0_ipi_handler(struct ex_regs *regs) +{ + asm volatile("inb %%dx, %%al" + : : [port] "d" (PORT_L0_EXIT) : "rax"); + asm volatile("vmcall"); +} + +static void l2_vcpu0_guest_code(void) +{ + asm volatile("cli"); + asm volatile("sti; nop; hlt"); +} + +static void l1_vcpu0_guest_code(struct vmx_pages *vmx_pages) +{ +#define L2_GUEST_STACK_SIZE 64 + unsigned long l2_vcpu0_guest_stack[L2_GUEST_STACK_SIZE]; + uint32_t control; + + x2apic_enable(); + + GUEST_ASSERT(prepare_for_vmx_operation(vmx_pages)); + GUEST_ASSERT(load_vmcs(vmx_pages)); + + /* Prepare the VMCS for L2 execution. */ + prepare_vmcs(vmx_pages, l2_vcpu0_guest_code, + &l2_vcpu0_guest_stack[L2_GUEST_STACK_SIZE]); + control =3D vmreadz(PIN_BASED_VM_EXEC_CONTROL); + control |=3D (PIN_BASED_EXT_INTR_MASK | + PIN_BASED_POSTED_INTR); + vmwrite(PIN_BASED_VM_EXEC_CONTROL, control); + control =3D vmreadz(CPU_BASED_VM_EXEC_CONTROL); + control |=3D (CPU_BASED_TPR_SHADOW | + CPU_BASED_ACTIVATE_SECONDARY_CONTROLS); + vmwrite(CPU_BASED_VM_EXEC_CONTROL, control); + control =3D vmreadz(SECONDARY_VM_EXEC_CONTROL); + control |=3D (SECONDARY_EXEC_VIRTUALIZE_X2APIC_MODE | + SECONDARY_EXEC_VIRTUAL_INTR_DELIVERY); + vmwrite(SECONDARY_VM_EXEC_CONTROL, control); + control =3D vmreadz(VM_EXIT_CONTROLS); + control |=3D VM_EXIT_ACK_INTR_ON_EXIT; + vmwrite(VM_EXIT_CONTROLS, control); + vmwrite(VIRTUAL_APIC_PAGE_ADDR, vmx_pages->virtual_apic_gpa); + vmwrite(POSTED_INTR_DESC_ADDR, vmx_pages->posted_intr_desc_gpa); + vmwrite(POSTED_INTR_NV, PI_NV); + + GUEST_ASSERT(!vmlaunch()); + GUEST_ASSERT(vmreadz(VM_EXIT_REASON) =3D=3D EXIT_REASON_VMCALL); + GUEST_ASSERT(!test_bit(PI_ON_BIT, (void *)vmx_pages->posted_intr_desc)); + GUEST_DONE(); +} + +static void post_intr(u8 vector, void *pi_desc) +{ + set_bit(vector, pi_desc); + set_bit(PI_ON_BIT, pi_desc); +} + +static void l1_vcpu1_guest_code(void *vcpu0_pi_desc) +{ + post_intr(L2_INTR, vcpu0_pi_desc); + x2apic_enable(); + x2apic_write_reg(APIC_ICR, ((u64)VCPU_ID0 << 32) | + APIC_DEST_PHYSICAL | APIC_DM_FIXED | PI_NV); + GUEST_DONE(); +} + +static void save_restore_vm(struct kvm_vm *vm) +{ + struct kvm_regs regs1 =3D {}, regs2 =3D {}; + struct kvm_x86_state *state; + + state =3D vcpu_save_state(vcpu0); + vcpu_regs_get(vcpu0, ®s1); + + kvm_vm_release(vm); + + /* Restore state in a new VM. */ + vcpu0 =3D vm_recreate_with_one_vcpu(vm); + vcpu_load_state(vcpu0, state); + kvm_x86_state_cleanup(state); + + vcpu_regs_get(vcpu0, ®s2); + TEST_ASSERT(!memcmp(®s1, ®s2, sizeof(regs2)), + "vcpu0: Unexpected register values after vcpu_load_state; rdi: %lx r= si: %lx", + (ulong) regs2.rdi, (ulong) regs2.rsi); +} + +void *create_and_run_vcpu1(void *arg) +{ + struct ucall uc; + struct kvm_run *run; + struct kvm_mp_state vcpu0_mp_state; + + pthread_cpu1 =3D pthread_self(); + + /* Keep trying to kick out vcpu0 until it is in halted state. */ + for (;;) { + WRITE_ONCE(vcpu0_can_run, true); + sleep(1); + WRITE_ONCE(vcpu0_can_run, false); + pthread_kill(pthread_cpu0, SIGUSR1); + printf("vcpu1: Sent SIGUSR1 to vcpu0\n"); + + while (READ_ONCE(vcpu0_running)) + ; + + vcpu_mp_state_get(vcpu0, &vcpu0_mp_state); + if (vcpu0_mp_state.mp_state =3D=3D KVM_MP_STATE_HALTED) + break; + } + + printf("vcpu1: Kicked out vcpu0 and ensure vcpu0 is halted\n"); + + /* Use save_restore_vm() to simulate a VM migration. */ + save_restore_vm(vm); + + printf("vcpu1: Finished save and restore vm.\n"); + vcpu1 =3D vm_vcpu_add(vm, VCPU_ID1, l1_vcpu1_guest_code); + vcpu_args_set(vcpu1, 1, vmx->posted_intr_desc); + + /* Start an L1 in vcpu1 and send a posted interrupt to halted L2 in vcpu0= . */ + for (;;) { + run =3D vcpu1->run; + vcpu_run(vcpu1); + + TEST_ASSERT(run->exit_reason =3D=3D KVM_EXIT_IO, + "vcpu1: Got exit_reason other than KVM_EXIT_IO: %u (%s)\n", + run->exit_reason, + exit_reason_str(run->exit_reason)); + + switch (get_ucall(vcpu1, &uc)) { + case UCALL_ABORT: + TEST_FAIL("%s", (const char *)uc.args[0]); + /* NOT REACHED */ + case UCALL_DONE: + printf("vcpu1: Successfully send a posted interrupt to vcpu0\n"); + goto done; + default: + TEST_FAIL("vcpu1: Unknown ucall %lu", uc.cmd); + } + } + +done: + /* + * Allow vcpu0 resume execution from L0 userspace and check if the + * posted interrupt get executed. + */ + WRITE_ONCE(vcpu0_can_run, true); + sleep(1); + TEST_ASSERT(READ_ONCE(pi_executed), + "vcpu0 did not execute the posted interrupt.\n"); + + return NULL; +} + +void sig_handler(int signum, siginfo_t *info, void *context) +{ + TEST_ASSERT(pthread_self() =3D=3D pthread_cpu0, + "Incorrect receiver of the signal, expect pthread_cpu0: " + "%lu, but get: %lu\n", pthread_cpu0, pthread_self()); + printf("vcpu0: Execute sighandler for signal: %d\n", signum); +} + +int main(int argc, char *argv[]) +{ + vm_vaddr_t vmx_pages_gva; + struct sigaction sig_action; + struct sigaction old_action; + + memset(&sig_action, 0, sizeof(sig_action)); + sig_action.sa_sigaction =3D sig_handler; + sig_action.sa_flags =3D SA_RESTART | SA_SIGINFO; + sigemptyset(&sig_action.sa_mask); + sigaction(SIGUSR1, &sig_action, &old_action); + + pthread_cpu0 =3D pthread_self(); + printf("vcpu0: Finish setup signal handler for SIGUSR1\n"); + + TEST_REQUIRE(kvm_has_cap(KVM_CAP_NESTED_STATE)); + TEST_REQUIRE(kvm_cpu_has(X86_FEATURE_VMX)); + + vm =3D vm_create_with_one_vcpu(&vcpu0, (void *)l1_vcpu0_guest_code); + + vm_init_descriptor_tables(vm); + vcpu_init_descriptor_tables(vcpu0); + vm_install_exception_handler(vm, L2_INTR, vcpu0_ipi_handler); + + /* Allocate VMX pages and shared descriptors (vmx_pages). */ + vmx =3D vcpu_alloc_vmx(vm, &vmx_pages_gva); + prepare_virtual_apic(vmx, vm); + prepare_posted_intr_desc(vmx, vm); + vcpu_args_set(vcpu0, 1, vmx_pages_gva); + + pthread_create(&pthread_cpu1, NULL, create_and_run_vcpu1, NULL); + + for (;;) { + struct kvm_run *run =3D vcpu0->run; + struct ucall uc; + int rc; + + while (!READ_ONCE(vcpu0_can_run)) + ; + + WRITE_ONCE(vcpu0_running, true); + + rc =3D __vcpu_run(vcpu0); + + vcpu0->run->immediate_exit =3D 0; + + /* + * When vCPU is kicked out by a signal, ensure a consistent vCPU + * state to prepare for migration before setting the + * vcpu_running flag to false. + */ + if (rc =3D=3D -1 && run->exit_reason =3D=3D KVM_EXIT_INTR) { + vcpu_run_complete_io(vcpu0); + + WRITE_ONCE(vcpu0_running, false); + + continue; + } + + WRITE_ONCE(vcpu0_running, false); + + if (run->io.port =3D=3D PORT_L0_EXIT) { + printf("vcpu0: Executed the posted interrupt\n"); + WRITE_ONCE(pi_executed, true); + continue; + } + + switch (get_ucall(vcpu0, &uc)) { + case UCALL_ABORT: + TEST_FAIL("%s", (const char *)uc.args[0]); + /* NOT REACHED */ + case UCALL_DONE: + goto done; + default: + TEST_FAIL("vcpu0: Unknown ucall %lu", uc.cmd); + } + } + +done: + kvm_vm_free(vm); + return 0; +} --=20 2.37.2.672.g94769d06f0-goog