From nobody Tue Jun 23 21:22:54 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5B1C1C433EF for ; Sat, 26 Feb 2022 00:16:11 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240171AbiBZAQl (ORCPT ); Fri, 25 Feb 2022 19:16:41 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49632 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240069AbiBZAQg (ORCPT ); Fri, 25 Feb 2022 19:16:36 -0500 Received: from mail-pj1-x1049.google.com (mail-pj1-x1049.google.com [IPv6:2607:f8b0:4864:20::1049]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 762512118FA for ; Fri, 25 Feb 2022 16:16:03 -0800 (PST) Received: by mail-pj1-x1049.google.com with SMTP id j22-20020a17090aeb1600b001bc32977e07so6453060pjz.7 for ; Fri, 25 Feb 2022 16:16:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=reply-to:date:in-reply-to:message-id:mime-version:references :subject:from:to:cc; bh=ZZBR7JFqEgpZm8jnS4/juRTRiIa97p0dVPrcdd7xFkE=; b=mUvLUVTIJXmMVDDZlI7ha2WbQOI90g3pkRHk/HCDPswvyM6eoRW7bWE8/cjKeH5LTQ 8crsCHf5SZ9IVsljfPwQRCzOgmPq0QxaeQUmE0CwVLjfgFoq6lmV8CWUYDBIInO6YmRY U99LRb+0Ju9wDjYyV8Pu5p/N6ItE+U07M8Qma1L5UexQ7dn+BfqlYHP0ObcgZvMiDnr5 6C/oDYC8QGiHdjvp8URsrDQYR1XoAxjg1xUu9h835fwzi5DWg4Z4lz2z+M/oxpGWvhuD tnfV0wpbU5JDuphYCRQwGDW9KCCQXUVj0mjOfihWKQc3is3uUFawDsvAb0rAn2YE64bK QS7w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:reply-to:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=ZZBR7JFqEgpZm8jnS4/juRTRiIa97p0dVPrcdd7xFkE=; b=h84m+d+x/Q54j+5t/xrgQoAag0SsUYwJ0dBOJWy3W/oBZO3iosUg6QNxdqKkPse2Qp N1G7dfA9bDQ2rxqNslLZQXFlgZDf6VPi9j3F+tKM4OzICo+YW9nxDkYqJ7oup5Xxr2CI 8KjyQcmZXdBq3y6OAhvISTo7rtnaspIyZNDpViETfuKdYgv4aRI41OogHsLv13pJKjP1 WVWU1g/EcQPUC0oD8yzpGWkQVnuHz7NUDtJntZ6M6rY0K/72iVXc3YcQda0eylAOe8lN ++V7O3rSqeGo/UOefndUQSOS1kZnbzIMk6xZ2YxRCqTOpIMSlpasSEoTw1OdXyh0RXeu 0UFw== X-Gm-Message-State: AOAM531FQBgYO3IK37SeUgcawY0aJkOMckmrOB9H2t0tjjFDFBrQ6Mcx ekqpJ4CORu7Yp6ZMsPvsTTZdtChTCD4= X-Google-Smtp-Source: ABdhPJz9WJWelElLxPoSbSX0aEe4XxeFTg2wwVfmdEUYCN4ODT/RvQ3HZRw9l9bEfE7UbG6g/T3mTDbOp1M= X-Received: from seanjc.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e5]) (user=seanjc job=sendgmr) by 2002:a62:e317:0:b0:4ca:25ee:d633 with SMTP id g23-20020a62e317000000b004ca25eed633mr10229857pfh.23.1645834562766; Fri, 25 Feb 2022 16:16:02 -0800 (PST) Reply-To: Sean Christopherson Date: Sat, 26 Feb 2022 00:15:19 +0000 In-Reply-To: <20220226001546.360188-1-seanjc@google.com> Message-Id: <20220226001546.360188-2-seanjc@google.com> Mime-Version: 1.0 References: <20220226001546.360188-1-seanjc@google.com> X-Mailer: git-send-email 2.35.1.574.g5d30c73bfb-goog Subject: [PATCH v3 01/28] KVM: x86/mmu: Use common iterator for walking invalid TDP MMU roots From: Sean Christopherson To: Paolo Bonzini , Christian Borntraeger , Janosch Frank , Claudio Imbrenda Cc: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , David Hildenbrand , kvm@vger.kernel.org, linux-kernel@vger.kernel.org, David Matlack , Ben Gardon , Mingwei Zhang Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Now that tdp_mmu_next_root() can process both valid and invalid roots, extend it to be able to process _only_ invalid roots, add yet another iterator macro for walking invalid roots, and use the new macro in kvm_tdp_mmu_zap_invalidated_roots(). No functional change intended. Reviewed-by: David Matlack Signed-off-by: Sean Christopherson --- arch/x86/kvm/mmu/tdp_mmu.c | 74 ++++++++++++++------------------------ 1 file changed, 26 insertions(+), 48 deletions(-) diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c index debf08212f12..25148e8b711d 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.c +++ b/arch/x86/kvm/mmu/tdp_mmu.c @@ -98,6 +98,12 @@ void kvm_tdp_mmu_put_root(struct kvm *kvm, struct kvm_mm= u_page *root, call_rcu(&root->rcu_head, tdp_mmu_free_sp_rcu_callback); } =20 +enum tdp_mmu_roots_iter_type { + ALL_ROOTS =3D -1, + VALID_ROOTS =3D 0, + INVALID_ROOTS =3D 1, +}; + /* * Returns the next root after @prev_root (or the first root if @prev_root= is * NULL). A reference to the returned root is acquired, and the reference= to @@ -110,10 +116,16 @@ void kvm_tdp_mmu_put_root(struct kvm *kvm, struct kvm= _mmu_page *root, */ static struct kvm_mmu_page *tdp_mmu_next_root(struct kvm *kvm, struct kvm_mmu_page *prev_root, - bool shared, bool only_valid) + bool shared, + enum tdp_mmu_roots_iter_type type) { struct kvm_mmu_page *next_root; =20 + kvm_lockdep_assert_mmu_lock_held(kvm, shared); + + /* Ensure correctness for the below comparison against role.invalid. */ + BUILD_BUG_ON(!!VALID_ROOTS || !INVALID_ROOTS); + rcu_read_lock(); =20 if (prev_root) @@ -125,7 +137,7 @@ static struct kvm_mmu_page *tdp_mmu_next_root(struct kv= m *kvm, typeof(*next_root), link); =20 while (next_root) { - if ((!only_valid || !next_root->role.invalid) && + if ((type =3D=3D ALL_ROOTS || (type =3D=3D !!next_root->role.invalid)) && kvm_tdp_mmu_get_root(next_root)) break; =20 @@ -151,18 +163,21 @@ static struct kvm_mmu_page *tdp_mmu_next_root(struct = kvm *kvm, * mode. In the unlikely event that this thread must free a root, the lock * will be temporarily dropped and reacquired in write mode. */ -#define __for_each_tdp_mmu_root_yield_safe(_kvm, _root, _as_id, _shared, _= only_valid)\ - for (_root =3D tdp_mmu_next_root(_kvm, NULL, _shared, _only_valid); \ +#define __for_each_tdp_mmu_root_yield_safe(_kvm, _root, _as_id, _shared, _= type) \ + for (_root =3D tdp_mmu_next_root(_kvm, NULL, _shared, _type); \ _root; \ - _root =3D tdp_mmu_next_root(_kvm, _root, _shared, _only_valid)) \ - if (kvm_mmu_page_as_id(_root) !=3D _as_id) { \ + _root =3D tdp_mmu_next_root(_kvm, _root, _shared, _type)) \ + if (_as_id > 0 && kvm_mmu_page_as_id(_root) !=3D _as_id) { \ } else =20 +#define for_each_invalid_tdp_mmu_root_yield_safe(_kvm, _root) \ + __for_each_tdp_mmu_root_yield_safe(_kvm, _root, -1, true, INVALID_ROOTS) + #define for_each_valid_tdp_mmu_root_yield_safe(_kvm, _root, _as_id, _share= d) \ - __for_each_tdp_mmu_root_yield_safe(_kvm, _root, _as_id, _shared, true) + __for_each_tdp_mmu_root_yield_safe(_kvm, _root, _as_id, _shared, VALID_RO= OTS) =20 #define for_each_tdp_mmu_root_yield_safe(_kvm, _root, _as_id, _shared) \ - __for_each_tdp_mmu_root_yield_safe(_kvm, _root, _as_id, _shared, false) + __for_each_tdp_mmu_root_yield_safe(_kvm, _root, _as_id, _shared, ALL_ROOT= S) =20 #define for_each_tdp_mmu_root(_kvm, _root, _as_id) \ list_for_each_entry_rcu(_root, &_kvm->arch.tdp_mmu_roots, link, \ @@ -810,28 +825,6 @@ void kvm_tdp_mmu_zap_all(struct kvm *kvm) kvm_flush_remote_tlbs(kvm); } =20 -static struct kvm_mmu_page *next_invalidated_root(struct kvm *kvm, - struct kvm_mmu_page *prev_root) -{ - struct kvm_mmu_page *next_root; - - if (prev_root) - next_root =3D list_next_or_null_rcu(&kvm->arch.tdp_mmu_roots, - &prev_root->link, - typeof(*prev_root), link); - else - next_root =3D list_first_or_null_rcu(&kvm->arch.tdp_mmu_roots, - typeof(*next_root), link); - - while (next_root && !(next_root->role.invalid && - refcount_read(&next_root->tdp_mmu_root_count))) - next_root =3D list_next_or_null_rcu(&kvm->arch.tdp_mmu_roots, - &next_root->link, - typeof(*next_root), link); - - return next_root; -} - /* * Since kvm_tdp_mmu_zap_all_fast has acquired a reference to each * invalidated root, they will not be freed until this function drops the @@ -842,36 +835,21 @@ static struct kvm_mmu_page *next_invalidated_root(str= uct kvm *kvm, */ void kvm_tdp_mmu_zap_invalidated_roots(struct kvm *kvm) { - struct kvm_mmu_page *next_root; struct kvm_mmu_page *root; bool flush =3D false; =20 lockdep_assert_held_read(&kvm->mmu_lock); =20 - rcu_read_lock(); - - root =3D next_invalidated_root(kvm, NULL); - - while (root) { - next_root =3D next_invalidated_root(kvm, root); - - rcu_read_unlock(); - + for_each_invalid_tdp_mmu_root_yield_safe(kvm, root) { flush =3D zap_gfn_range(kvm, root, 0, -1ull, true, flush, true); =20 /* - * Put the reference acquired in - * kvm_tdp_mmu_invalidate_roots + * Put the reference acquired in kvm_tdp_mmu_invalidate_roots(). + * Note, the iterator holds its own reference. */ kvm_tdp_mmu_put_root(kvm, root, true); - - root =3D next_root; - - rcu_read_lock(); } =20 - rcu_read_unlock(); - if (flush) kvm_flush_remote_tlbs(kvm); } --=20 2.35.1.574.g5d30c73bfb-goog From nobody Tue Jun 23 21:22:54 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 52E1AC433FE for ; Sat, 26 Feb 2022 00:16:23 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240333AbiBZAQy (ORCPT ); Fri, 25 Feb 2022 19:16:54 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49672 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240091AbiBZAQi (ORCPT ); Fri, 25 Feb 2022 19:16:38 -0500 Received: from mail-yb1-xb49.google.com (mail-yb1-xb49.google.com [IPv6:2607:f8b0:4864:20::b49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6A410214F86 for ; Fri, 25 Feb 2022 16:16:05 -0800 (PST) Received: by mail-yb1-xb49.google.com with SMTP id d17-20020a253611000000b006244e94b7b4so4990998yba.4 for ; Fri, 25 Feb 2022 16:16:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=reply-to:date:in-reply-to:message-id:mime-version:references :subject:from:to:cc; bh=W0dsOEq/PIsQRXnhmEzN4z4vDCVezWtZyWCTAFD0mm4=; b=kkTCjC4tVIj7enFaFMKIYHTANMNAYxjkHrt4FVwFXDe4UpZ9MJGM/QS0YDczVjf7zS xyCmw6fI2EpbOhAcLNPaVAm6RBmzabO5vXpzITVrlUpoTf1wQw4PbIxstYS2SyGOBdd8 ObaNGe64fwLi3o87QsWZ21hi0Zu0f1Knbx5jNrinn0D46zmVKOUU2/5VpUlnm2EybEEa fZ/TaCH4VDiAo0gmuhj9G1xesKLSJ2KCRPDK6t1kNJrbMcld3k6Y6hjYi12nyVC7Hbt6 jMvWvSfnhrHp49CLfuCKLksw6MoPezPBJHQQQyC97WTVDmrWDX/xDhXIWsCx7+7fauCA Ebdg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:reply-to:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=W0dsOEq/PIsQRXnhmEzN4z4vDCVezWtZyWCTAFD0mm4=; b=T0BexUPmxmKAFIOjT5SW0tU/Oy+7uYeQ8z1At29t5rfa9B3RguLbgfzMuh5hjMk3J5 1QyVgikWDpKg1YiTv2xvsQJXgwYsc3+35Pofu5IoqhkM6fsZ320i9BynmhZYEhMuEZGg OvzEpvg7JAFwN5NSfXeyiKwB+9QIdKReiHmHsBdiaaAcblntsGUKL6MTn9Wz8jKQRETW e4b0cKm1+FodULVz0fOWUjmF45dxgD1j5zFBcRKNZDQzvnZm9wCPWSVjFN3M3COMgDNy /oVIQ7qadwrjelVmpBmoee5Vjc5uBNtZZ3YiuO76lHSF+VQbwdcpVqkkV/LbO+xlgjNV XLeA== X-Gm-Message-State: AOAM533ZcHPakFwI93/ZvbmNi5fxoc4yS23c6Xfdqz43NeW4vIizONh0 EVoASu/ilCxW+rWNq/yzMnAV0kcBda8= X-Google-Smtp-Source: ABdhPJwyWL5owexPrfMNeXvwM0V0pC6xHluXdGFWOU2XldPBciQnXTg/AL4B9+xNVnXtV72T/fRBvSj0OJg= X-Received: from seanjc.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e5]) (user=seanjc job=sendgmr) by 2002:a05:690c:3a1:b0:2d6:9324:cb4f with SMTP id bh33-20020a05690c03a100b002d69324cb4fmr10270352ywb.399.1645834564647; Fri, 25 Feb 2022 16:16:04 -0800 (PST) Reply-To: Sean Christopherson Date: Sat, 26 Feb 2022 00:15:20 +0000 In-Reply-To: <20220226001546.360188-1-seanjc@google.com> Message-Id: <20220226001546.360188-3-seanjc@google.com> Mime-Version: 1.0 References: <20220226001546.360188-1-seanjc@google.com> X-Mailer: git-send-email 2.35.1.574.g5d30c73bfb-goog Subject: [PATCH v3 02/28] KVM: x86/mmu: Check for present SPTE when clearing dirty bit in TDP MMU From: Sean Christopherson To: Paolo Bonzini , Christian Borntraeger , Janosch Frank , Claudio Imbrenda Cc: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , David Hildenbrand , kvm@vger.kernel.org, linux-kernel@vger.kernel.org, David Matlack , Ben Gardon , Mingwei Zhang Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Explicitly check for present SPTEs when clearing dirty bits in the TDP MMU. This isn't strictly required for correctness, as setting the dirty bit in a defunct SPTE will not change the SPTE from !PRESENT to PRESENT. However, the guarded MMU_WARN_ON() in spte_ad_need_write_protect() would complain if anyone actually turned on KVM's MMU debugging. Fixes: a6a0b05da9f3 ("kvm: x86/mmu: Support dirty logging for the TDP MMU") Cc: Ben Gardon Signed-off-by: Sean Christopherson Reviewed-by: Ben Gardon --- arch/x86/kvm/mmu/tdp_mmu.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c index 25148e8b711d..9357780ec28f 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.c +++ b/arch/x86/kvm/mmu/tdp_mmu.c @@ -1446,6 +1446,9 @@ static bool clear_dirty_gfn_range(struct kvm *kvm, st= ruct kvm_mmu_page *root, if (tdp_mmu_iter_cond_resched(kvm, &iter, false, true)) continue; =20 + if (!is_shadow_present_pte(iter.old_spte)) + continue; + if (spte_ad_need_write_protect(iter.old_spte)) { if (is_writable_pte(iter.old_spte)) new_spte =3D iter.old_spte & ~PT_WRITABLE_MASK; --=20 2.35.1.574.g5d30c73bfb-goog From nobody Tue Jun 23 21:22:54 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 05EFAC433FE for ; Sat, 26 Feb 2022 00:16:16 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240251AbiBZAQq (ORCPT ); Fri, 25 Feb 2022 19:16:46 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49732 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240106AbiBZAQk (ORCPT ); Fri, 25 Feb 2022 19:16:40 -0500 Received: from mail-pj1-x104a.google.com (mail-pj1-x104a.google.com [IPv6:2607:f8b0:4864:20::104a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2BA6E2177E8 for ; Fri, 25 Feb 2022 16:16:07 -0800 (PST) Received: by mail-pj1-x104a.google.com with SMTP id 97-20020a17090a09ea00b001bc54996b62so6480519pjo.3 for ; Fri, 25 Feb 2022 16:16:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=reply-to:date:in-reply-to:message-id:mime-version:references :subject:from:to:cc; bh=Jj10j2aUg8MzEq8hNDXHAtFe0jk7/OZgYIYqdt1cOoc=; b=D2bVuMswqCIu+UG+EbEQzN6gM9KXJVT/zxc3AY9+iTQqtXVpHhdM9v0rQKNQOneKYn TQuXm3NDvUYi2SmHd4dr0u+ZMKGEOtqN798Hl830qR4uSuTKj6SwrOv/iG1EmfH95473 zSN+CrNkkuFMFH2/evxrT9A2Gy7Mdsg++tsrtvD/M9yNK75LEI2fqokwA85WII7OOmHL 6FLI7TpHmCd2tEVHkUq5XxG92BPqrfEifCe1XZZ5u1Fe0M3idj3nPxZCN2KAvAehEgf9 ZzBbOryoXNMqmhCpVH9edTem6LJAoVZ6xV2yHTV429B9uQ1qT/NiXinAvUBsEh0yng7h Tw6g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:reply-to:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=Jj10j2aUg8MzEq8hNDXHAtFe0jk7/OZgYIYqdt1cOoc=; b=095x+wGLwFY9GMeW4ernhC1k0HffXFHDSxvV3OWBDpEIrs4VUdpjKh+v+WE21OJjIC dwEg0CQqr01LsPTZ8BpGTQAgVqlIjityf3DKgGaZoIOvmvp6UkQFac8ypJPZvsHcQ7HI nJonOYH9a2ygzfFC6Nf5/rHYuAe4uJHh5AFyNysr+8BLqPNYWitHNOhufJd52R2fAjJH PTWWinbBu63UhSG0bnKLZzgiZEs0n4OXVYJ+c6PqeWikFxWetGjLW6jcGBUGjrx7TbjJ +VzLr/uK9/iJjs5uv9v/1FLSAs9KHi0L/JgInIFr7NYLoqeFS7EzNNF/NcgLV5qb2qZ6 UyHw== X-Gm-Message-State: AOAM532FJcAuX0VEVvxBuOlxGZznsl8fR9rCmnuZqGcHmQDCQtqLGIt3 vubDamN404tyJrjBNUXzFQ7KMr0MA1c= X-Google-Smtp-Source: ABdhPJxTsqEjDeM7T6Ba1EZBe2JaFy1Xue5S2foiQNfqnVpKVKsbF7n7QWvcOf2FFnbszq/jivwVk86lxIs= X-Received: from seanjc.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e5]) (user=seanjc job=sendgmr) by 2002:a17:90b:4b52:b0:1bc:b208:dc5c with SMTP id mi18-20020a17090b4b5200b001bcb208dc5cmr761796pjb.1.1645834566189; Fri, 25 Feb 2022 16:16:06 -0800 (PST) Reply-To: Sean Christopherson Date: Sat, 26 Feb 2022 00:15:21 +0000 In-Reply-To: <20220226001546.360188-1-seanjc@google.com> Message-Id: <20220226001546.360188-4-seanjc@google.com> Mime-Version: 1.0 References: <20220226001546.360188-1-seanjc@google.com> X-Mailer: git-send-email 2.35.1.574.g5d30c73bfb-goog Subject: [PATCH v3 03/28] KVM: x86/mmu: Fix wrong/misleading comments in TDP MMU fast zap From: Sean Christopherson To: Paolo Bonzini , Christian Borntraeger , Janosch Frank , Claudio Imbrenda Cc: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , David Hildenbrand , kvm@vger.kernel.org, linux-kernel@vger.kernel.org, David Matlack , Ben Gardon , Mingwei Zhang Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Fix misleading and arguably wrong comments in the TDP MMU's fast zap flow. The comments, and the fact that actually zapping invalid roots was added separately, strongly suggests that zapping invalid roots is an optimization and not required for correctness. That is a lie. KVM _must_ zap invalid roots before returning from kvm_mmu_zap_all_fast(), because when it's called from kvm_mmu_invalidate_zap_pages_in_memslot(), KVM is relying on it to fully remove all references to the memslot. Once the memslot is gone, KVM's mmu_notifier hooks will be unable to find the stale references as the hva=3D>gfn translation is done via the memslots. If KVM doesn't immediately zap SPTEs and userspace unmaps a range after deleting a memslot, KVM will fail to zap in response to the mmu_notifier due to not finding a memslot corresponding to the notifier's range, which leads to a variation of use-after-free. The other misleading comment (and code) explicitly states that roots without a reference should be skipped. While that's technically true, it's also extremely misleading as it should be impossible for KVM to encounter a defunct root on the list while holding mmu_lock for write. Opportunstically add a WARN to enforce that invariant. Fixes: b7cccd397f31 ("KVM: x86/mmu: Fast invalidation for TDP MMU") Fixes: 4c6654bd160d ("KVM: x86/mmu: Tear down roots before kvm_mmu_zap_all_= fast returns") Signed-off-by: Sean Christopherson Reviewed-by: Ben Gardon --- arch/x86/kvm/mmu/mmu.c | 8 +++++++ arch/x86/kvm/mmu/tdp_mmu.c | 46 +++++++++++++++++++++----------------- 2 files changed, 33 insertions(+), 21 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index b2c1c4eb6007..80607513a1f2 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -5662,6 +5662,14 @@ static void kvm_mmu_zap_all_fast(struct kvm *kvm) =20 write_unlock(&kvm->mmu_lock); =20 + /* + * Zap the invalidated TDP MMU roots, all SPTEs must be dropped before + * returning to the caller, e.g. if the zap is in response to a memslot + * deletion, mmu_notifier callbacks will be unable to reach the SPTEs + * associated with the deleted memslot once the update completes, and + * Deferring the zap until the final reference to the root is put would + * lead to use-after-free. + */ if (is_tdp_mmu_enabled(kvm)) { read_lock(&kvm->mmu_lock); kvm_tdp_mmu_zap_invalidated_roots(kvm); diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c index 9357780ec28f..12866113fb4f 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.c +++ b/arch/x86/kvm/mmu/tdp_mmu.c @@ -826,12 +826,11 @@ void kvm_tdp_mmu_zap_all(struct kvm *kvm) } =20 /* - * Since kvm_tdp_mmu_zap_all_fast has acquired a reference to each - * invalidated root, they will not be freed until this function drops the - * reference. Before dropping that reference, tear down the paging - * structure so that whichever thread does drop the last reference - * only has to do a trivial amount of work. Since the roots are invalid, - * no new SPTEs should be created under them. + * Zap all invalidated roots to ensure all SPTEs are dropped before the "f= ast + * zap" completes. Since kvm_tdp_mmu_invalidate_all_roots() has acquired a + * reference to each invalidated root, roots will not be freed until after= this + * function drops the gifted reference, e.g. so that vCPUs don't get stuck= with + * tearing paging structures. */ void kvm_tdp_mmu_zap_invalidated_roots(struct kvm *kvm) { @@ -855,21 +854,25 @@ void kvm_tdp_mmu_zap_invalidated_roots(struct kvm *kv= m) } =20 /* - * Mark each TDP MMU root as invalid so that other threads - * will drop their references and allow the root count to - * go to 0. + * Mark each TDP MMU root as invalid to prevent vCPUs from reusing a root = that + * is about to be zapped, e.g. in response to a memslots update. The call= er is + * responsible for invoking kvm_tdp_mmu_zap_invalidated_roots() to the act= ual + * zapping. * - * Also take a reference on all roots so that this thread - * can do the bulk of the work required to free the roots - * once they are invalidated. Without this reference, a - * vCPU thread might drop the last reference to a root and - * get stuck with tearing down the entire paging structure. + * Take a reference on all roots to prevent the root from being freed befo= re it + * is zapped by this thread. Freeing a root is not a correctness issue, b= ut if + * a vCPU drops the last reference to a root prior to the root being zappe= d, it + * will get stuck with tearing down the entire paging structure. * - * Roots which have a zero refcount should be skipped as - * they're already being torn down. - * Already invalid roots should be referenced again so that - * they aren't freed before kvm_tdp_mmu_zap_all_fast is - * done with them. + * Get a reference even if the root is already invalid, + * kvm_tdp_mmu_zap_invalidated_roots() assumes it was gifted a reference t= o all + * invalid roots, e.g. there's no epoch to identify roots that were invali= dated + * by a previous call. Roots stay on the list until the last reference is + * dropped, so even though all invalid roots are zapped, a root may not go= away + * for quite some time, e.g. if a vCPU blocks across multiple memslot upda= tes. + * + * Because mmu_lock is held for write, it should be impossible to observe a + * root with zero refcount, i.e. the list of roots cannot be stale. * * This has essentially the same effect for the TDP MMU * as updating mmu_valid_gen does for the shadow MMU. @@ -879,9 +882,10 @@ void kvm_tdp_mmu_invalidate_all_roots(struct kvm *kvm) struct kvm_mmu_page *root; =20 lockdep_assert_held_write(&kvm->mmu_lock); - list_for_each_entry(root, &kvm->arch.tdp_mmu_roots, link) - if (refcount_inc_not_zero(&root->tdp_mmu_root_count)) + list_for_each_entry(root, &kvm->arch.tdp_mmu_roots, link) { + if (!WARN_ON_ONCE(!kvm_tdp_mmu_get_root(root))) root->role.invalid =3D true; + } } =20 /* --=20 2.35.1.574.g5d30c73bfb-goog From nobody Tue Jun 23 21:22:54 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id F0B37C433F5 for ; Sat, 26 Feb 2022 00:16:19 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240275AbiBZAQv (ORCPT ); Fri, 25 Feb 2022 19:16:51 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49780 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240165AbiBZAQl (ORCPT ); Fri, 25 Feb 2022 19:16:41 -0500 Received: from mail-pj1-x104a.google.com (mail-pj1-x104a.google.com [IPv6:2607:f8b0:4864:20::104a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C26A221CD14 for ; Fri, 25 Feb 2022 16:16:08 -0800 (PST) Received: by mail-pj1-x104a.google.com with SMTP id j22-20020a17090aeb1600b001bc32977e07so6453188pjz.7 for ; Fri, 25 Feb 2022 16:16:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=reply-to:date:in-reply-to:message-id:mime-version:references :subject:from:to:cc; bh=1LwlOnokaTZ070XNVC9qwt/b9mpGNAxjLSfPQpUGhzg=; b=QsEqpFIthDCOl+CW8X6crKs61Os+2aEmwyiqRUlI/TlvRi59OYvHUqW1xn12sx6TVB 737oQCm+hfLDVRTd2EOV0rFDNcHYhX9QzNSaO/IBh/uazLWPCWVojkHoxp05AVNZ1OTT n2uwJVBotnOsITVbELerw/16BFamFz+QmMJZEgoW0f/Y95Aoofqfm4hvmSsp8CH3grGC 7LV2OMHiPwHrnKaPuEUdsjLKK9EIPHTQvcbQ85epOW+obRncKRQToDjsCF6h/+pLc5Wh AMV+VJsmzBWE6J9f4CgntUBYnQKp8iCBRWEZ+O8kd8IsYbLfbqLilQfTCOrVQVvQxffz /x5Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:reply-to:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=1LwlOnokaTZ070XNVC9qwt/b9mpGNAxjLSfPQpUGhzg=; b=7hpK3Kayh9LB1Y+jTb4WxjFRQBWXOWUIsEZKVJ+C00a5mRWP0HL61pbj8Vut6KfkYJ HmPJ7x1j8aX3lRUdAcgQn+ywz2eoAeSfMJ+mgdufFZONDRSgIplJUBBuepGG4BmN0O2I Fg9k5lMEfqcUPDiD0zSznt7IXGfk79x2aBjSxm23NSR0ayZzu6ft80lFRtHgy1QAJ3bl FJBpDWyRD3r2Rf+YGjV+RUCoAfBEX1gxWGxYvzYbRPkxUXoLWB5onp6dWZdheMDeqab7 by69mFdYYQd8kdSfCP6KXtQdfz/eAFv+FdoEyK3HYHjnc467u1AHkz1ASowgCBMM2KIz 2W/w== X-Gm-Message-State: AOAM533wj5kkV4ZuxN/IXJuEsF3HOtb9FV/Gn4b+YtplN+BCdu4zGxeq cm1FxGxVP6yasONvSFw5MgDwP3XC2Ck= X-Google-Smtp-Source: ABdhPJzVf+Iyo3hR1ZTnjDaOKvm9FKY4DHDhSBKkXQndAg3+GsOOkeZ67NqLYNm7sq1hJeioUCFCZeKk2z8= X-Received: from seanjc.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e5]) (user=seanjc job=sendgmr) by 2002:a05:6a00:148f:b0:4bc:fb2d:4b6f with SMTP id v15-20020a056a00148f00b004bcfb2d4b6fmr10113984pfu.62.1645834568211; Fri, 25 Feb 2022 16:16:08 -0800 (PST) Reply-To: Sean Christopherson Date: Sat, 26 Feb 2022 00:15:22 +0000 In-Reply-To: <20220226001546.360188-1-seanjc@google.com> Message-Id: <20220226001546.360188-5-seanjc@google.com> Mime-Version: 1.0 References: <20220226001546.360188-1-seanjc@google.com> X-Mailer: git-send-email 2.35.1.574.g5d30c73bfb-goog Subject: [PATCH v3 04/28] KVM: x86/mmu: Formalize TDP MMU's (unintended?) deferred TLB flush logic From: Sean Christopherson To: Paolo Bonzini , Christian Borntraeger , Janosch Frank , Claudio Imbrenda Cc: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , David Hildenbrand , kvm@vger.kernel.org, linux-kernel@vger.kernel.org, David Matlack , Ben Gardon , Mingwei Zhang Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Explicitly ignore the result of zap_gfn_range() when putting the last reference to a TDP MMU root, and add a pile of comments to formalize the TDP MMU's behavior of deferring TLB flushes to alloc/reuse. Note, this only affects the !shared case, as zap_gfn_range() subtly never returns true for "flush" as the flush is handled by tdp_mmu_zap_spte_atomic(). Putting the root without a flush is ok because even if there are stale references to the root in the TLB, they are unreachable because KVM will not run the guest with the same ASID without first flushing (where ASID in this context refers to both SVM's explicit ASID and Intel's implicit ASID that is constructed from VPID+PCID+EPT4A+etc...). Signed-off-by: Sean Christopherson Reviewed-by: Mingwei Zhang --- arch/x86/kvm/mmu/mmu.c | 8 ++++++++ arch/x86/kvm/mmu/tdp_mmu.c | 10 +++++++++- 2 files changed, 17 insertions(+), 1 deletion(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index 80607513a1f2..5a931c89d27b 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -5069,6 +5069,14 @@ int kvm_mmu_load(struct kvm_vcpu *vcpu) kvm_mmu_sync_roots(vcpu); =20 kvm_mmu_load_pgd(vcpu); + + /* + * Flush any TLB entries for the new root, the provenance of the root + * is unknown. In theory, even if KVM ensures there are no stale TLB + * entries for a freed root, in theory, an out-of-tree hypervisor could + * have left stale entries. Flushing on alloc also allows KVM to skip + * the TLB flush when freeing a root (see kvm_tdp_mmu_put_root()). + */ static_call(kvm_x86_flush_tlb_current)(vcpu); out: return r; diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c index 12866113fb4f..e35bd88d92fd 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.c +++ b/arch/x86/kvm/mmu/tdp_mmu.c @@ -93,7 +93,15 @@ void kvm_tdp_mmu_put_root(struct kvm *kvm, struct kvm_mm= u_page *root, list_del_rcu(&root->link); spin_unlock(&kvm->arch.tdp_mmu_pages_lock); =20 - zap_gfn_range(kvm, root, 0, -1ull, false, false, shared); + /* + * A TLB flush is not necessary as KVM performs a local TLB flush when + * allocating a new root (see kvm_mmu_load()), and when migrating vCPU + * to a different pCPU. Note, the local TLB flush on reuse also + * invalidates any paging-structure-cache entries, i.e. TLB entries for + * intermediate paging structures, that may be zapped, as such entries + * are associated with the ASID on both VMX and SVM. + */ + (void)zap_gfn_range(kvm, root, 0, -1ull, false, false, shared); =20 call_rcu(&root->rcu_head, tdp_mmu_free_sp_rcu_callback); } --=20 2.35.1.574.g5d30c73bfb-goog From nobody Tue Jun 23 21:22:54 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id BC66BC433F5 for ; Sat, 26 Feb 2022 00:16:29 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240102AbiBZAQ6 (ORCPT ); Fri, 25 Feb 2022 19:16:58 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50250 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240292AbiBZAQv (ORCPT ); Fri, 25 Feb 2022 19:16:51 -0500 Received: from mail-pg1-x549.google.com (mail-pg1-x549.google.com [IPv6:2607:f8b0:4864:20::549]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7B222214F86 for ; Fri, 25 Feb 2022 16:16:10 -0800 (PST) Received: by mail-pg1-x549.google.com with SMTP id b5-20020a631b05000000b00373bd90134dso3462181pgb.22 for ; Fri, 25 Feb 2022 16:16:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=reply-to:date:in-reply-to:message-id:mime-version:references :subject:from:to:cc; bh=zez4Gs30dxckwmE+XcZFyy9DVdQB//tN93t4s3+TKo8=; b=GqRA+N2Xdwf0MftL4IZRlc9dGRi5XcOybgeQMHWIYaYTTz6Hx8v2f47ehCxN/l/R9M kotSaU4HDuDWT5tALQFjZhRpfwcjnlxxivgLf2wCd5sEECtGOVrWWSu0r2rWXsyrNp3u 4kjtaVXDixoCG8jke13IHkMA8j4gSxD7ioBSj8yGRVaQXp9sIJmzq8yGgy50q+0UOQib 5sW88eKK0j2TDfwRPwjzzV8XJSA7IUyWpebMlxF9WIRudTDECqhWPwkEdpDoOzUcPL3f /KXhHnJJUBVsiuk67rQp7rnPQFUqSJpSR242fcjiM7/lQm8dGASJE6QUaGjD2KMzjQeW QomQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:reply-to:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=zez4Gs30dxckwmE+XcZFyy9DVdQB//tN93t4s3+TKo8=; b=lnYmwNTxloeauXKkavhksyKghMwFbpFXtXYTl93eeGDeImRFRlxF7UMx7sBMle88Tc xs3vYeJT4lgb73+8bAfVyilaiq905FGjqt4Au5zy36mzjMM3ywsr19qpdWXoeMCBmggi NglD7Zn8dDlBMqddq/00p0l+PT701NA0zGafSL6IgwLy0hRxIzAnOPCBrrB04X2xorxz PM2i1UbecrzE4+bFSnLS0qtvC7QTE4HUMV+xP+URPIINExeH1reT+bAc63mB0eaFGbap 47sRZr3r1UD8datNHwZmrZmu75bVaAGw9QvYSIAm+K5c9Yf2ZiY0WIUYFX6xf0OLIsvT V0Hg== X-Gm-Message-State: AOAM532xZStGEPLHAnaRg/5bTj3+0zOVqXtKMyagLWdTcLVjmhSkdud7 /lT6ZtIznR9P6zoY9LW5QcuCBC/7dSI= X-Google-Smtp-Source: ABdhPJyMKt5K1hday9KJX2febmO2RNxhZGslMxjGB3MrHRn5iFHo+/U2JORNEGrCHZPM5iVC7xM4YIWEgGI= X-Received: from seanjc.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e5]) (user=seanjc job=sendgmr) by 2002:a17:902:f68b:b0:14f:c84c:ad6d with SMTP id l11-20020a170902f68b00b0014fc84cad6dmr9734294plg.155.1645834569885; Fri, 25 Feb 2022 16:16:09 -0800 (PST) Reply-To: Sean Christopherson Date: Sat, 26 Feb 2022 00:15:23 +0000 In-Reply-To: <20220226001546.360188-1-seanjc@google.com> Message-Id: <20220226001546.360188-6-seanjc@google.com> Mime-Version: 1.0 References: <20220226001546.360188-1-seanjc@google.com> X-Mailer: git-send-email 2.35.1.574.g5d30c73bfb-goog Subject: [PATCH v3 05/28] KVM: x86/mmu: Document that zapping invalidated roots doesn't need to flush From: Sean Christopherson To: Paolo Bonzini , Christian Borntraeger , Janosch Frank , Claudio Imbrenda Cc: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , David Hildenbrand , kvm@vger.kernel.org, linux-kernel@vger.kernel.org, David Matlack , Ben Gardon , Mingwei Zhang Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Remove the misleading flush "handling" when zapping invalidated TDP MMU roots, and document that flushing is unnecessary for all flavors of MMUs when zapping invalid/obsolete roots/pages. The "handling" in the TDP MMU is dead code, as zap_gfn_range() is called with shared=3Dtrue, in which case it will never return true due to the flushing being handled by tdp_mmu_zap_spte_atomic(). No functional change intended. Signed-off-by: Sean Christopherson Reviewed-by: Ben Gardon --- arch/x86/kvm/mmu/mmu.c | 10 +++++++--- arch/x86/kvm/mmu/tdp_mmu.c | 15 ++++++++++----- 2 files changed, 17 insertions(+), 8 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index 5a931c89d27b..1c4b84e80841 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -5615,9 +5615,13 @@ static void kvm_zap_obsolete_pages(struct kvm *kvm) } =20 /* - * Trigger a remote TLB flush before freeing the page tables to ensure - * KVM is not in the middle of a lockless shadow page table walk, which - * may reference the pages. + * Kick all vCPUs (via remote TLB flush) before freeing the page tables + * to ensure KVM is not in the middle of a lockless shadow page table + * walk, which may reference the pages. The remote TLB flush itself is + * not required and is simply a convenient way to kick vCPUs as needed. + * KVM performs a local TLB flush when allocating a new root (see + * kvm_mmu_load()), and the reload in the caller ensure no vCPUs are + * running with an obsolete MMU. */ kvm_mmu_commit_zap_page(kvm, &kvm->arch.zapped_obsolete_pages); } diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c index e35bd88d92fd..5994db5d5226 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.c +++ b/arch/x86/kvm/mmu/tdp_mmu.c @@ -843,12 +843,20 @@ void kvm_tdp_mmu_zap_all(struct kvm *kvm) void kvm_tdp_mmu_zap_invalidated_roots(struct kvm *kvm) { struct kvm_mmu_page *root; - bool flush =3D false; =20 lockdep_assert_held_read(&kvm->mmu_lock); =20 for_each_invalid_tdp_mmu_root_yield_safe(kvm, root) { - flush =3D zap_gfn_range(kvm, root, 0, -1ull, true, flush, true); + /* + * A TLB flush is unnecessary, invalidated roots are guaranteed + * to be unreachable by the guest (see kvm_tdp_mmu_put_root() + * for more details), and unlike the legacy MMU, no vCPU kick + * is needed to play nice with lockless shadow walks as the TDP + * MMU protects its paging structures via RCU. Note, zapping + * will still flush on yield, but that's a minor performance + * blip and not a functional issue. + */ + (void)zap_gfn_range(kvm, root, 0, -1ull, true, false, true); =20 /* * Put the reference acquired in kvm_tdp_mmu_invalidate_roots(). @@ -856,9 +864,6 @@ void kvm_tdp_mmu_zap_invalidated_roots(struct kvm *kvm) */ kvm_tdp_mmu_put_root(kvm, root, true); } - - if (flush) - kvm_flush_remote_tlbs(kvm); } =20 /* --=20 2.35.1.574.g5d30c73bfb-goog From nobody Tue Jun 23 21:22:54 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2CCCDC433EF for ; Sat, 26 Feb 2022 00:16:51 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240662AbiBZARW (ORCPT ); Fri, 25 Feb 2022 19:17:22 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50284 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240153AbiBZAQw (ORCPT ); Fri, 25 Feb 2022 19:16:52 -0500 Received: from mail-pf1-x44a.google.com (mail-pf1-x44a.google.com [IPv6:2607:f8b0:4864:20::44a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 039AA22559E for ; Fri, 25 Feb 2022 16:16:12 -0800 (PST) Received: by mail-pf1-x44a.google.com with SMTP id z28-20020aa79f9c000000b004e10449d919so3967795pfr.4 for ; Fri, 25 Feb 2022 16:16:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=reply-to:date:in-reply-to:message-id:mime-version:references :subject:from:to:cc; bh=CkcWm9TOTaG84AakiNdIByFyddSdpVR6qJn4me3qHQU=; b=SrM/ryVBa/JLHQKNP3iNMNNVSSk4v5Ddwyv7M6e0IUPaBHea0ca2RgWivk1Shp4eg2 uCxS3N6bsurRUuP3f6of53M5aE0Jm8DfWQ3bPnNp6v5bLxJsYaGsyYLDKohJUWq1UyCH JJ//QzYD/LPHCpuq81wIXJGyR1pBt4umkqf11JnylOtcmv7VoMh3zMdnchl1KxnFIcb+ vjymo1seRffIdQdUjCrQvKvayjF6qiIX9HCWLbg7pSoUGiQsraZ61xX2TmydRLJuzAjp qLjkn1imbV0HaXP3mdWyAQqCZmRXh2kHITNGLlxhsmWGarGmArbPXKQHFezRF/B587wE zz6w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:reply-to:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=CkcWm9TOTaG84AakiNdIByFyddSdpVR6qJn4me3qHQU=; b=mhKRE8LOxAJlpmoNUJK/cmDqbFCqUaf9xFVuYs1Lj+DGa4bQ97zCUD2skylYMhmKri KlbcuSFZsanesNqi+0bWq8wsAabrJsJ02pxnin03MZRA4bonY+9xl68IRV1NvZzQqfTr B4c+ZCohJGR70dQIa4Brhu8qw2jwt50MUrtfQPZO+lxbjJX6WgtYvLNObfUnJfbSPb/8 tfxKoUmCDgTW+fzBlq80+8MZxdmxiJvEBCBNwR5useNTdnACLvWsFsSAawGbC4KsYSpy Akn2z11TyhpsYedWTbN9iFI7kebxUeY7jaPGobfImdTSctqZmbmC29CKiVgWIZ2CBHql Xj7Q== X-Gm-Message-State: AOAM533B5fPhWbXp7FwCVusADh/iXxiRhUdU17LBECNZ1aqapTPqCPrb C7PKrZ0WH6UtNO7sZbQ5p2mzGSfBHpc= X-Google-Smtp-Source: ABdhPJwDFDIV7OblSikTqfHjNLInNVnmlGYEWyoM69GLtFDl5Kxsr2oAbsXo65Axu1YJx3fSWE3oLmQMlEc= X-Received: from seanjc.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e5]) (user=seanjc job=sendgmr) by 2002:a17:902:a404:b0:14b:1100:aebc with SMTP id p4-20020a170902a40400b0014b1100aebcmr9483781plq.133.1645834571470; Fri, 25 Feb 2022 16:16:11 -0800 (PST) Reply-To: Sean Christopherson Date: Sat, 26 Feb 2022 00:15:24 +0000 In-Reply-To: <20220226001546.360188-1-seanjc@google.com> Message-Id: <20220226001546.360188-7-seanjc@google.com> Mime-Version: 1.0 References: <20220226001546.360188-1-seanjc@google.com> X-Mailer: git-send-email 2.35.1.574.g5d30c73bfb-goog Subject: [PATCH v3 06/28] KVM: x86/mmu: Require mmu_lock be held for write in unyielding root iter From: Sean Christopherson To: Paolo Bonzini , Christian Borntraeger , Janosch Frank , Claudio Imbrenda Cc: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , David Hildenbrand , kvm@vger.kernel.org, linux-kernel@vger.kernel.org, David Matlack , Ben Gardon , Mingwei Zhang Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Assert that mmu_lock is held for write by users of the yield-unfriendly TDP iterator. The nature of a shared walk means that the caller needs to play nice with other tasks modifying the page tables, which is more or less the same thing as playing nice with yielding. Theoretically, KVM could gain a flow where it could legitimately take mmu_lock for read in a non-preemptible context, but that's highly unlikely and any such case should be viewed with a fair amount of scrutiny. Signed-off-by: Sean Christopherson Reviewed-by: Ben Gardon --- arch/x86/kvm/mmu/tdp_mmu.c | 21 +++++++++++++++------ 1 file changed, 15 insertions(+), 6 deletions(-) diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c index 5994db5d5226..189f21e71c36 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.c +++ b/arch/x86/kvm/mmu/tdp_mmu.c @@ -29,13 +29,16 @@ bool kvm_mmu_init_tdp_mmu(struct kvm *kvm) return true; } =20 -static __always_inline void kvm_lockdep_assert_mmu_lock_held(struct kvm *k= vm, +/* Arbitrarily returns true so that this may be used in if statements. */ +static __always_inline bool kvm_lockdep_assert_mmu_lock_held(struct kvm *k= vm, bool shared) { if (shared) lockdep_assert_held_read(&kvm->mmu_lock); else lockdep_assert_held_write(&kvm->mmu_lock); + + return true; } =20 void kvm_mmu_uninit_tdp_mmu(struct kvm *kvm) @@ -187,11 +190,17 @@ static struct kvm_mmu_page *tdp_mmu_next_root(struct = kvm *kvm, #define for_each_tdp_mmu_root_yield_safe(_kvm, _root, _as_id, _shared) \ __for_each_tdp_mmu_root_yield_safe(_kvm, _root, _as_id, _shared, ALL_ROOT= S) =20 -#define for_each_tdp_mmu_root(_kvm, _root, _as_id) \ - list_for_each_entry_rcu(_root, &_kvm->arch.tdp_mmu_roots, link, \ - lockdep_is_held_type(&kvm->mmu_lock, 0) || \ - lockdep_is_held(&kvm->arch.tdp_mmu_pages_lock)) \ - if (kvm_mmu_page_as_id(_root) !=3D _as_id) { \ +/* + * Iterate over all TDP MMU roots. Requires that mmu_lock be held for wri= te, + * the implication being that any flow that holds mmu_lock for read is + * inherently yield-friendly and should use the yielf-safe variant above. + * Holding mmu_lock for write obviates the need for RCU protection as the = list + * is guaranteed to be stable. + */ +#define for_each_tdp_mmu_root(_kvm, _root, _as_id) \ + list_for_each_entry(_root, &_kvm->arch.tdp_mmu_roots, link) \ + if (kvm_lockdep_assert_mmu_lock_held(_kvm, false) && \ + kvm_mmu_page_as_id(_root) !=3D _as_id) { \ } else =20 static struct kvm_mmu_page *tdp_mmu_alloc_sp(struct kvm_vcpu *vcpu) --=20 2.35.1.574.g5d30c73bfb-goog From nobody Tue Jun 23 21:22:54 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5610BC433EF for ; Sat, 26 Feb 2022 00:16:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240344AbiBZARF (ORCPT ); Fri, 25 Feb 2022 19:17:05 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50260 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240354AbiBZAQw (ORCPT ); Fri, 25 Feb 2022 19:16:52 -0500 Received: from mail-pf1-x449.google.com (mail-pf1-x449.google.com [IPv6:2607:f8b0:4864:20::449]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BC9B8214F99 for ; Fri, 25 Feb 2022 16:16:13 -0800 (PST) Received: by mail-pf1-x449.google.com with SMTP id b6-20020a621b06000000b004e1453487efso3948209pfb.22 for ; Fri, 25 Feb 2022 16:16:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=reply-to:date:in-reply-to:message-id:mime-version:references :subject:from:to:cc; bh=Woj2Bw+oml/iasKr9ZaRbac+/ftGLH0xOnzbgL1A7dk=; b=Qc+tUkr6gYwZ0XsOSuNXU7xpgIing1CxAj5rs++waMqJEzVClUQdpUouf6Qt2rl1ol vOWyv9mL3UgLXppAyVZC96Yc/Lrw/ldLrVMXmjVLugkwxdOXizRLOv1CMgfTRMprPriF LEEYfdtQbBo2m2mh86pMjWbhd282X+lOMJ/GSWqjH9CpyoGKiSsUJk5vcIf6V8YO7TdK LK2FRoMOm2vFXsKXYmgueEECx5xv+zJuInoHleR13aEgu6lz5OhzcsOzDSGUj/NTrtLp 2h8JtOzzt4KAcE8dZmU7wjrpQrp6j8k53yXXzQ6CFEa0l0mZRmqAWirOD9wVur1toO7L i4ug== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:reply-to:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=Woj2Bw+oml/iasKr9ZaRbac+/ftGLH0xOnzbgL1A7dk=; b=ESm85T0GKPm8ojf8CWizBxCgjH+yoIo0mxFwrYklDRS47bi7CJwiIkWXZuL//2cgZH s8mc56yK3vDbVQ8CDgAEdsdZwS9E6PE0zA9BHGDQbBMy8+MR9xEwZum/9LB+eF3pZlVV Yupz4oCwk1Omhtiib5ZakUrHEYIiOfwphSidpLn4O57Fdoylpkq1ieOrpl8ETdo25MQN m2N6YxozTjNjAqcdAauQLhDc1JtjFqqeqVZ/pZt1GQlIdhflvfOB+3jGj5PrAPIPsU+g lQfxGJmPeAnGHsSOiXZrglF4weacvLBHaMpKXDuSSgekBhnSHgAn1etQDilfHcoOAXuw KVkg== X-Gm-Message-State: AOAM532+6qBnRwpjQvA55/xBdU0xGgci4m8AxpnWTTUkZHr4fo6lOpsS eKLYbr/r5uZqEOUSaVyCIrAJAOCtcoY= X-Google-Smtp-Source: ABdhPJyUX5t0pyzj0U7YymUOkbyXueWsC8h5DQRNC1MHthdGoUFXbAdU/nG2klZ1iOz8qYIOT4ryBLb1KLQ= X-Received: from seanjc.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e5]) (user=seanjc job=sendgmr) by 2002:aa7:85c2:0:b0:4cb:b95a:887f with SMTP id z2-20020aa785c2000000b004cbb95a887fmr10094149pfn.74.1645834573211; Fri, 25 Feb 2022 16:16:13 -0800 (PST) Reply-To: Sean Christopherson Date: Sat, 26 Feb 2022 00:15:25 +0000 In-Reply-To: <20220226001546.360188-1-seanjc@google.com> Message-Id: <20220226001546.360188-8-seanjc@google.com> Mime-Version: 1.0 References: <20220226001546.360188-1-seanjc@google.com> X-Mailer: git-send-email 2.35.1.574.g5d30c73bfb-goog Subject: [PATCH v3 07/28] KVM: x86/mmu: Check for !leaf=>leaf, not PFN change, in TDP MMU SP removal From: Sean Christopherson To: Paolo Bonzini , Christian Borntraeger , Janosch Frank , Claudio Imbrenda Cc: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , David Hildenbrand , kvm@vger.kernel.org, linux-kernel@vger.kernel.org, David Matlack , Ben Gardon , Mingwei Zhang Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Look for a !leaf=3D>leaf conversion instead of a PFN change when checking if a SPTE change removed a TDP MMU shadow page. Convert the PFN check into a WARN, as KVM should never change the PFN of a shadow page (except when its being zapped or replaced). From a purely theoretical perspective, it's not illegal to replace a SP with a hugepage pointing at the same PFN. In practice, it's impossible as that would require mapping guest memory overtop a kernel-allocated SP. Either way, the check is odd. Signed-off-by: Sean Christopherson Reviewed-by: Ben Gardon Reviewed-by: Mingwei Zhang --- arch/x86/kvm/mmu/tdp_mmu.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c index 189f21e71c36..848448b65703 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.c +++ b/arch/x86/kvm/mmu/tdp_mmu.c @@ -505,9 +505,12 @@ static void __handle_changed_spte(struct kvm *kvm, int= as_id, gfn_t gfn, =20 /* * Recursively handle child PTs if the change removed a subtree from - * the paging structure. + * the paging structure. Note the WARN on the PFN changing without the + * SPTE being converted to a hugepage (leaf) or being zapped. Shadow + * pages are kernel allocations and should never be migrated. */ - if (was_present && !was_leaf && (pfn_changed || !is_present)) + if (was_present && !was_leaf && + (is_leaf || !is_present || WARN_ON_ONCE(pfn_changed))) handle_removed_pt(kvm, spte_to_child_pt(old_spte, level), shared); } =20 --=20 2.35.1.574.g5d30c73bfb-goog From nobody Tue Jun 23 21:22:54 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 07C9FC433EF for ; Sat, 26 Feb 2022 00:16:48 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240597AbiBZART (ORCPT ); Fri, 25 Feb 2022 19:17:19 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50292 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240463AbiBZAQy (ORCPT ); Fri, 25 Feb 2022 19:16:54 -0500 Received: from mail-pg1-x54a.google.com (mail-pg1-x54a.google.com [IPv6:2607:f8b0:4864:20::54a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CCB972255BF for ; Fri, 25 Feb 2022 16:16:15 -0800 (PST) Received: by mail-pg1-x54a.google.com with SMTP id g31-20020a63521f000000b003783582a261so1439648pgb.5 for ; Fri, 25 Feb 2022 16:16:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=reply-to:date:in-reply-to:message-id:mime-version:references :subject:from:to:cc; bh=G398P7RcaSAEnVDGuXyKG1MtPaHrsCLmNhGjwb/vYpg=; b=R8L3GXb2iyMDkg0JQmqrX2X7ARQEO+Z7DOJsS5/5XHBLGcLkRCgSn+ANc7wRoFYg99 ziYt2seRoQuDB2HvG+EaZ9MHLF2GdsHB4o6qqxE6aBd5DNyFAXDZVzz1EYDnRcxhC2r4 S2geRnlh62kLBJUVLBy6yVJwLY3RQyPv7AMJNmGocuZKnW1uRQsNZwh94q4Q4XbXlmKL foeaFq/xbag68suV0gOwJO0JXHPCBFEy+Mo35Qyr75jDB+S25pbbpHqdxicOZ+5rd6Op QPt3fLxyXF2hsSO7/EYM6J+XV840P5D+ooewjyfq+2Hyw3hak7qvX8oJmmD6hR5m0xAC 2U/w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:reply-to:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=G398P7RcaSAEnVDGuXyKG1MtPaHrsCLmNhGjwb/vYpg=; b=0RcgO3YIiVSG0Me6rBvnTFfZp2LnWR3kMcd87rMrZ9YGNu1LAIq/tvN0MUujwM4lSL gWZxGyLuEwYeCSCpzhU7Ngq1s0Hgxb6cy65ZmEM+DtPWH/ATNI6ux//VHsov4EIJPGKf jZzv80c/74GVXqQNr4zfQ6tKOR3i67RO/PEie6hJuNoH9m76v6UQ1Etu8gHkXCUtLWTI KkuiJ+IJ0qzK/bHHBNeqSEGIEGctkGmGwtT9uRXeiK90Qih2qcp06SY8K/P33mKaavHb hmm7KfKApXw7lT9GQBeFiYnTocf3B4V/khCEUqhAOB84ph8SRQHrkQwmDCZtIgt9Bwht Ew3A== X-Gm-Message-State: AOAM530XHMQDH8+XSKNRTQddJOAcnfitRr3tuJgs0zSYktQoyAYlX1Zj mejX3de7+egyDOxCJrKW2p56Kr7JNNE= X-Google-Smtp-Source: ABdhPJy7lHJWDyxwAWHN4q2rpPYcO/QcoMfFh0Asba4foDlLtm+DAdHJ17610fWaI/B/snIanvtPDxy+PBc= X-Received: from seanjc.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e5]) (user=seanjc job=sendgmr) by 2002:a17:90b:4b52:b0:1bc:b208:dc5c with SMTP id mi18-20020a17090b4b5200b001bcb208dc5cmr761907pjb.1.1645834574884; Fri, 25 Feb 2022 16:16:14 -0800 (PST) Reply-To: Sean Christopherson Date: Sat, 26 Feb 2022 00:15:26 +0000 In-Reply-To: <20220226001546.360188-1-seanjc@google.com> Message-Id: <20220226001546.360188-9-seanjc@google.com> Mime-Version: 1.0 References: <20220226001546.360188-1-seanjc@google.com> X-Mailer: git-send-email 2.35.1.574.g5d30c73bfb-goog Subject: [PATCH v3 08/28] KVM: x86/mmu: Batch TLB flushes from TDP MMU for MMU notifier change_spte From: Sean Christopherson To: Paolo Bonzini , Christian Borntraeger , Janosch Frank , Claudio Imbrenda Cc: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , David Hildenbrand , kvm@vger.kernel.org, linux-kernel@vger.kernel.org, David Matlack , Ben Gardon , Mingwei Zhang Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Batch TLB flushes (with other MMUs) when handling ->change_spte() notifications in the TDP MMU. The MMU notifier path in question doesn't allow yielding and correcty flushes before dropping mmu_lock. Signed-off-by: Sean Christopherson Reviewed-by: Ben Gardon Reviewed-by: Mingwei Zhang --- arch/x86/kvm/mmu/tdp_mmu.c | 13 ++++++------- 1 file changed, 6 insertions(+), 7 deletions(-) diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c index 848448b65703..634a2838e117 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.c +++ b/arch/x86/kvm/mmu/tdp_mmu.c @@ -1203,13 +1203,12 @@ static bool set_spte_gfn(struct kvm *kvm, struct td= p_iter *iter, */ bool kvm_tdp_mmu_set_spte_gfn(struct kvm *kvm, struct kvm_gfn_range *range) { - bool flush =3D kvm_tdp_mmu_handle_gfn(kvm, range, set_spte_gfn); - - /* FIXME: return 'flush' instead of flushing here. */ - if (flush) - kvm_flush_remote_tlbs_with_address(kvm, range->start, 1); - - return false; + /* + * No need to handle the remote TLB flush under RCU protection, the + * target SPTE _must_ be a leaf SPTE, i.e. cannot result in freeing a + * shadow page. See the WARN on pfn_changed in __handle_changed_spte(). + */ + return kvm_tdp_mmu_handle_gfn(kvm, range, set_spte_gfn); } =20 /* --=20 2.35.1.574.g5d30c73bfb-goog From nobody Tue Jun 23 21:22:54 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8DAA1C433F5 for ; Sat, 26 Feb 2022 00:16:55 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240701AbiBZAR0 (ORCPT ); Fri, 25 Feb 2022 19:17:26 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50448 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240524AbiBZAQy (ORCPT ); Fri, 25 Feb 2022 19:16:54 -0500 Received: from mail-pg1-x54a.google.com (mail-pg1-x54a.google.com [IPv6:2607:f8b0:4864:20::54a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 21F1A2261D2 for ; Fri, 25 Feb 2022 16:16:17 -0800 (PST) Received: by mail-pg1-x54a.google.com with SMTP id t7-20020a634447000000b0036579648983so3492393pgk.3 for ; Fri, 25 Feb 2022 16:16:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=reply-to:date:in-reply-to:message-id:mime-version:references :subject:from:to:cc; bh=GLAMaXO4BLnbAIX6/AcgYNLqdWbaIIgMTgsuYxMQrpk=; b=IPBFBbCu7NaqWZPAwKKolZZ7oYeFq3VbCvxdfwxTJg7sDDIHVVNvQ+JzTDvbEWoaSm JPyFzrWUNkecgHpj8QiukTXlwmFWIN6NsPOuVRTdiMBxdLhExWwS54vP8lvV9wxlLzAk crDpVn+H2Gu4Jf5TVQTru59EVW2Ikn1TJUE9ESs1Kzrw4HQbGm24VbOGAU0i8ovY7ov7 dC15CQL3p9ur70HdDnvJZTi8QNCO2/AsxHOqOj49EXETuI33UMOH1MnrAcLf0RKXiwU9 dxu6skrBMSqR5SD2fAf+nOXqMJkKBu4qPwfBoYRfAxioYxRze7VllMVXgpicU9bJJC+l f7Bw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:reply-to:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=GLAMaXO4BLnbAIX6/AcgYNLqdWbaIIgMTgsuYxMQrpk=; b=PFJgbJBkTUpXC6nou+SN/voTqVZj2pq76f8Vv73M2BtjgM0OMuUaJBh9s1q8vyF8Vk z/e7Dm+p6+ChL1aRbfY7sEtZ0xDKPPhI85QCAFZhTb6knoDsj3kj5qed/GT364k2LUfC 7wIt1dUheTlaeGIpakcmQfHXmSwNmmYGngagl+qmIiBp/Zke9uexSlx2799XoJmr+j3F wK255IbJC+UshfxcpREeUyN5QKCEKB6p/e/DVXrjbbR3hHrDJDzGO+6KEwmOSK+mhrX3 P8edSsN7xYNoMS2Eb3HcahYirEU1kbqoNwYVGsuysETW95ouG2Hmg/R6gQ2W6mbANHSc Y7+g== X-Gm-Message-State: AOAM532d8RbHXndq5fbHoj4YcLRiwqDmGD5eZ3IH+Eftq3syFhmrLGqY 8MNieAAx7unWqotjr0tJiT0BT7yMkMg= X-Google-Smtp-Source: ABdhPJx8rYJ83MXizyFpwRKKPP/3gcYAoJ2ByXAk92IyTDKirx3ywRF4qjmtmIJKkIa8SfBeZxMvIoxCRF0= X-Received: from seanjc.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e5]) (user=seanjc job=sendgmr) by 2002:a05:6a00:3018:b0:4e1:de9a:a5a3 with SMTP id ay24-20020a056a00301800b004e1de9aa5a3mr10262616pfb.80.1645834576574; Fri, 25 Feb 2022 16:16:16 -0800 (PST) Reply-To: Sean Christopherson Date: Sat, 26 Feb 2022 00:15:27 +0000 In-Reply-To: <20220226001546.360188-1-seanjc@google.com> Message-Id: <20220226001546.360188-10-seanjc@google.com> Mime-Version: 1.0 References: <20220226001546.360188-1-seanjc@google.com> X-Mailer: git-send-email 2.35.1.574.g5d30c73bfb-goog Subject: [PATCH v3 09/28] KVM: x86/mmu: Drop RCU after processing each root in MMU notifier hooks From: Sean Christopherson To: Paolo Bonzini , Christian Borntraeger , Janosch Frank , Claudio Imbrenda Cc: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , David Hildenbrand , kvm@vger.kernel.org, linux-kernel@vger.kernel.org, David Matlack , Ben Gardon , Mingwei Zhang Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Drop RCU protection after processing each root when handling MMU notifier hooks that aren't the "unmap" path, i.e. aren't zapping. Temporarily drop RCU to let RCU do its thing between roots, and to make it clear that there's no special behavior that relies on holding RCU across all roots. Currently, the RCU protection is completely superficial, it's necessary only to make rcu_dereference() of SPTE pointers happy. A future patch will rely on holding RCU as a proxy for vCPUs in the guest, e.g. to ensure shadow pages aren't freed before all vCPUs do a TLB flush (or rather, acknowledge the need for a flush), but in that case RCU needs to be held until the flush is complete if and only if the flush is needed because a shadow page may have been removed. And except for the "unmap" path, MMU notifier events cannot remove SPs (don't toggle PRESENT bit, and can't change the PFN for a SP). Signed-off-by: Sean Christopherson Reviewed-by: Ben Gardon Reviewed-by: Mingwei Zhang --- arch/x86/kvm/mmu/tdp_mmu.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c index 634a2838e117..4f460782a848 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.c +++ b/arch/x86/kvm/mmu/tdp_mmu.c @@ -1100,18 +1100,18 @@ static __always_inline bool kvm_tdp_mmu_handle_gfn(= struct kvm *kvm, struct tdp_iter iter; bool ret =3D false; =20 - rcu_read_lock(); - /* * Don't support rescheduling, none of the MMU notifiers that funnel * into this helper allow blocking; it'd be dead, wasteful code. */ for_each_tdp_mmu_root(kvm, root, range->slot->as_id) { + rcu_read_lock(); + tdp_root_for_each_leaf_pte(iter, root, range->start, range->end) ret |=3D handler(kvm, &iter, range); - } =20 - rcu_read_unlock(); + rcu_read_unlock(); + } =20 return ret; } --=20 2.35.1.574.g5d30c73bfb-goog From nobody Tue Jun 23 21:22:54 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B0857C433F5 for ; Sat, 26 Feb 2022 00:17:10 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240461AbiBZARm (ORCPT ); Fri, 25 Feb 2022 19:17:42 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50392 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240323AbiBZAQz (ORCPT ); Fri, 25 Feb 2022 19:16:55 -0500 Received: from mail-pg1-x54a.google.com (mail-pg1-x54a.google.com [IPv6:2607:f8b0:4864:20::54a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 024542261E4 for ; Fri, 25 Feb 2022 16:16:18 -0800 (PST) Received: by mail-pg1-x54a.google.com with SMTP id t68-20020a635f47000000b003732348b971so3493527pgb.7 for ; Fri, 25 Feb 2022 16:16:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=reply-to:date:in-reply-to:message-id:mime-version:references :subject:from:to:cc; bh=Lu5Zb5K5z9Ao4faecJWogQLcUl62itC1pDHyKWB4Szo=; b=rAH+CawAA7QAlMyJN3NdqP1kqf3643HSBwkEoDU38fZX3HhmSZHXCjwkA6eTZF9zeM h8uoj6K3E+4qSYaXDmhoZPe3RDLUrIVTd6CuVu4fLZgkqYqHd6GXX9i6aaRrZ4hkmOhX X6xr9MHYI9covtPmNWnB5Acxyd6uc03ZbiMN3JIduDeI9A0To20OtTKKlgERreCy45Hv 6+f9s6fhKfE7Ds/+l8W8wMtjwKTZEqLSTNU6Phr9jtqjAi2jTIJV4bEymOSZyhh1HixD X5dPTEYpbbpdSN9TH+JMi227hrXI6mQpe2OPvmLz4NEv3v/nwi8wX3DUqKhh7fQi89FD JYcw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:reply-to:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=Lu5Zb5K5z9Ao4faecJWogQLcUl62itC1pDHyKWB4Szo=; b=HtTij8xNpQo+yeJU++tLoXNjrBGhf8XCVPG6FxRTh4lpLfPCgtmLkZlgsOhXNdaT/+ 7VKQqXF/vxvcnj4mlexrMHkDDOrTGUqrdbTcRRDwbbn+bTrOdXT5OfTIuO2rpkiRyqE6 4VKR63fBrJvNqF6I2jyRZcBowa+kNetZkxaa/J3yZQy1H6BDXemeaomZQWu5Tgq7U8Wz vg8nRAD5FlXWPLfc4BsEqeVUEyb1uO1FShCgsvmhhW0kiBU1bJLC0cOAPIGtt0IeGSfn EMxddLuiueWEg18ulrLhHqorermPtbIx/KHvZrtkkfeSul2OTpqoXcjkJs+OnW0vEEDN ji+A== X-Gm-Message-State: AOAM530xR0VchJVN4vI5OucS0vG8z+qNfQTTduFBEAMiY71lNDahvhLJ 9OBPmVkQR9fj24VLMGFe8oz5QG3JO/k= X-Google-Smtp-Source: ABdhPJz14fr8CmwWpHoBUZQ5h7RoKy3te0//NmcR2dmx3Olc8psLMchJrSZrtQlTY0WAg1eKXW9jY43YI/8= X-Received: from seanjc.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e5]) (user=seanjc job=sendgmr) by 2002:a62:7554:0:b0:4e1:5898:4fbb with SMTP id q81-20020a627554000000b004e158984fbbmr10185233pfc.2.1645834578388; Fri, 25 Feb 2022 16:16:18 -0800 (PST) Reply-To: Sean Christopherson Date: Sat, 26 Feb 2022 00:15:28 +0000 In-Reply-To: <20220226001546.360188-1-seanjc@google.com> Message-Id: <20220226001546.360188-11-seanjc@google.com> Mime-Version: 1.0 References: <20220226001546.360188-1-seanjc@google.com> X-Mailer: git-send-email 2.35.1.574.g5d30c73bfb-goog Subject: [PATCH v3 10/28] KVM: x86/mmu: Add helpers to read/write TDP MMU SPTEs and document RCU From: Sean Christopherson To: Paolo Bonzini , Christian Borntraeger , Janosch Frank , Claudio Imbrenda Cc: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , David Hildenbrand , kvm@vger.kernel.org, linux-kernel@vger.kernel.org, David Matlack , Ben Gardon , Mingwei Zhang Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add helpers to read and write TDP MMU SPTEs instead of open coding rcu_dereference() all over the place, and to provide a convenient location to document why KVM doesn't exempt holding mmu_lock for write from having to hold RCU (and any future changes to the rules). No functional change intended. Signed-off-by: Sean Christopherson Reviewed-by: Ben Gardon Reviewed-by: Mingwei Zhang --- arch/x86/kvm/mmu/tdp_iter.c | 6 +++--- arch/x86/kvm/mmu/tdp_iter.h | 16 ++++++++++++++++ arch/x86/kvm/mmu/tdp_mmu.c | 6 +++--- 3 files changed, 22 insertions(+), 6 deletions(-) diff --git a/arch/x86/kvm/mmu/tdp_iter.c b/arch/x86/kvm/mmu/tdp_iter.c index be3f096db2eb..6d3b3e5a5533 100644 --- a/arch/x86/kvm/mmu/tdp_iter.c +++ b/arch/x86/kvm/mmu/tdp_iter.c @@ -12,7 +12,7 @@ static void tdp_iter_refresh_sptep(struct tdp_iter *iter) { iter->sptep =3D iter->pt_path[iter->level - 1] + SHADOW_PT_INDEX(iter->gfn << PAGE_SHIFT, iter->level); - iter->old_spte =3D READ_ONCE(*rcu_dereference(iter->sptep)); + iter->old_spte =3D kvm_tdp_mmu_read_spte(iter->sptep); } =20 static gfn_t round_gfn_for_level(gfn_t gfn, int level) @@ -89,7 +89,7 @@ static bool try_step_down(struct tdp_iter *iter) * Reread the SPTE before stepping down to avoid traversing into page * tables that are no longer linked from this entry. */ - iter->old_spte =3D READ_ONCE(*rcu_dereference(iter->sptep)); + iter->old_spte =3D kvm_tdp_mmu_read_spte(iter->sptep); =20 child_pt =3D spte_to_child_pt(iter->old_spte, iter->level); if (!child_pt) @@ -123,7 +123,7 @@ static bool try_step_side(struct tdp_iter *iter) iter->gfn +=3D KVM_PAGES_PER_HPAGE(iter->level); iter->next_last_level_gfn =3D iter->gfn; iter->sptep++; - iter->old_spte =3D READ_ONCE(*rcu_dereference(iter->sptep)); + iter->old_spte =3D kvm_tdp_mmu_read_spte(iter->sptep); =20 return true; } diff --git a/arch/x86/kvm/mmu/tdp_iter.h b/arch/x86/kvm/mmu/tdp_iter.h index 216ebbe76ddd..bb9b581f1ee4 100644 --- a/arch/x86/kvm/mmu/tdp_iter.h +++ b/arch/x86/kvm/mmu/tdp_iter.h @@ -9,6 +9,22 @@ =20 typedef u64 __rcu *tdp_ptep_t; =20 +/* + * TDP MMU SPTEs are RCU protected to allow paging structures (non-leaf SP= TEs) + * to be zapped while holding mmu_lock for read. Holding RCU isn't requir= ed for + * correctness if mmu_lock is held for write, but plumbing "struct kvm" do= wn to + * the lower depths of the TDP MMU just to make lockdep happy is a nightma= re, so + * all accesses to SPTEs are done under RCU protection. + */ +static inline u64 kvm_tdp_mmu_read_spte(tdp_ptep_t sptep) +{ + return READ_ONCE(*rcu_dereference(sptep)); +} +static inline void kvm_tdp_mmu_write_spte(tdp_ptep_t sptep, u64 val) +{ + WRITE_ONCE(*rcu_dereference(sptep), val); +} + /* * A TDP iterator performs a pre-order walk over a TDP paging structure. */ diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c index 4f460782a848..8fbf3364f116 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.c +++ b/arch/x86/kvm/mmu/tdp_mmu.c @@ -609,7 +609,7 @@ static inline int tdp_mmu_zap_spte_atomic(struct kvm *k= vm, * here since the SPTE is going from non-present * to non-present. */ - WRITE_ONCE(*rcu_dereference(iter->sptep), 0); + kvm_tdp_mmu_write_spte(iter->sptep, 0); =20 return 0; } @@ -648,7 +648,7 @@ static inline void __tdp_mmu_set_spte(struct kvm *kvm, = struct tdp_iter *iter, */ WARN_ON(is_removed_spte(iter->old_spte)); =20 - WRITE_ONCE(*rcu_dereference(iter->sptep), new_spte); + kvm_tdp_mmu_write_spte(iter->sptep, new_spte); =20 __handle_changed_spte(kvm, iter->as_id, iter->gfn, iter->old_spte, new_spte, iter->level, false); @@ -1046,7 +1046,7 @@ int kvm_tdp_mmu_map(struct kvm_vcpu *vcpu, struct kvm= _page_fault *fault) * because the new value informs the !present * path below. */ - iter.old_spte =3D READ_ONCE(*rcu_dereference(iter.sptep)); + iter.old_spte =3D kvm_tdp_mmu_read_spte(iter.sptep); } =20 if (!is_shadow_present_pte(iter.old_spte)) { --=20 2.35.1.574.g5d30c73bfb-goog From nobody Tue Jun 23 21:22:54 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B281DC433F5 for ; Sat, 26 Feb 2022 00:17:00 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S241057AbiBZARb (ORCPT ); Fri, 25 Feb 2022 19:17:31 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50664 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240359AbiBZARA (ORCPT ); Fri, 25 Feb 2022 19:17:00 -0500 Received: from mail-pg1-x54a.google.com (mail-pg1-x54a.google.com [IPv6:2607:f8b0:4864:20::54a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9B3CE2261D1 for ; Fri, 25 Feb 2022 16:16:20 -0800 (PST) Received: by mail-pg1-x54a.google.com with SMTP id t18-20020a63dd12000000b00342725203b5so3458097pgg.16 for ; Fri, 25 Feb 2022 16:16:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=reply-to:date:in-reply-to:message-id:mime-version:references :subject:from:to:cc; bh=mkmQ6MFrlZy4eKd/Q3wbeOobNp7CN8W6giifpouEK3Q=; b=HNIMRDG1beTnwyz26v1dtXmdSZhkkNIT50ZAUvd6nR9l9tMgyyLfGhzlV+OqJCPEHw TiL7s2zfof/Z0d7TYvcEljJ0Z5/Rlzqge6Cu2y6WB6ZEdqWGgH9Del7NJBRv9vdYGZlG ROSoidAZmK6YDct2cNe23Qz5WbKtSXyHAPHDu4a1VCAgcib73hMUvWeBlkIljaSMzrGe kePNhRjQ0u4Z+bNIpQUXw5KynJ8w7BCfpUeXOeq6brTm1zXLiRPEgsDhf4QiDt7kx+So cJyGEZ46LlQfjFZ+eUlOlAveINLeZxzBEKSBRJlkE06aDrtbOo5uGcxLK97/LpUa6ped 0dlA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:reply-to:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=mkmQ6MFrlZy4eKd/Q3wbeOobNp7CN8W6giifpouEK3Q=; b=3e8G9m+eSHE6lp0A1hg+YHZWJgJtUfxxwbcjRpJpQrzcX5+jgByiUB1D2bfdmBaunu 9HL+v4HAUB/t3yreuxFml1MUJMqnrjibOL8YV5i07A6MxOTDsdJ1ZVIbXY4zEzB+Ex0L XucTjf3bNCUVlfL7pRdESwCsdihfSKkS0GlSL6on8NojttDQvyC1OwAaGM/DGm9ymJi4 5AC3BaX+iGNBH7ZOrXyL1DZfUOqSEC1wLj82d9WRS1Q6HFfb1BE4HgxYWSS6opWW4O4O SM69HFjwEa/LkDT6efqHP9Ar9Dawhz5MMDDWZj2VQk/b8xvnVf5OlMm6wrtQArzdNAhY aoBQ== X-Gm-Message-State: AOAM531ieuEqrdt+NV4zKYVlGM2v2VvSsbwCemI+0JnhKJODov7RUt57 b9XwQdAjQWknJigkTHLlrJtQ9sN3MKk= X-Google-Smtp-Source: ABdhPJyWCyvm2sgSjm0RwiuidMtkOSbJaGbyT+NiwBP55uKTx1afDWW6MyIdn7+BA//anx1BPN3Dk9lol4Y= X-Received: from seanjc.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e5]) (user=seanjc job=sendgmr) by 2002:a05:6a00:124f:b0:4c0:6242:c14e with SMTP id u15-20020a056a00124f00b004c06242c14emr10089164pfi.83.1645834580082; Fri, 25 Feb 2022 16:16:20 -0800 (PST) Reply-To: Sean Christopherson Date: Sat, 26 Feb 2022 00:15:29 +0000 In-Reply-To: <20220226001546.360188-1-seanjc@google.com> Message-Id: <20220226001546.360188-12-seanjc@google.com> Mime-Version: 1.0 References: <20220226001546.360188-1-seanjc@google.com> X-Mailer: git-send-email 2.35.1.574.g5d30c73bfb-goog Subject: [PATCH v3 11/28] KVM: x86/mmu: WARN if old _or_ new SPTE is REMOVED in non-atomic path From: Sean Christopherson To: Paolo Bonzini , Christian Borntraeger , Janosch Frank , Claudio Imbrenda Cc: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , David Hildenbrand , kvm@vger.kernel.org, linux-kernel@vger.kernel.org, David Matlack , Ben Gardon , Mingwei Zhang Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" WARN if the new_spte being set by __tdp_mmu_set_spte() is a REMOVED_SPTE, which is called out by the comment as being disallowed but not actually checked. Keep the WARN on the old_spte as well, because overwriting a REMOVED_SPTE in the non-atomic path is also disallowed (as evidence by lack of splats with the existing WARN). Fixes: 08f07c800e9d ("KVM: x86/mmu: Flush TLBs after zap in TDP MMU PF hand= ler") Cc: Ben Gardon Signed-off-by: Sean Christopherson Reviewed-by: Ben Gardon --- arch/x86/kvm/mmu/tdp_mmu.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c index 8fbf3364f116..1dcdf1a4fcc1 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.c +++ b/arch/x86/kvm/mmu/tdp_mmu.c @@ -640,13 +640,13 @@ static inline void __tdp_mmu_set_spte(struct kvm *kvm= , struct tdp_iter *iter, lockdep_assert_held_write(&kvm->mmu_lock); =20 /* - * No thread should be using this function to set SPTEs to the + * No thread should be using this function to set SPTEs to or from the * temporary removed SPTE value. * If operating under the MMU lock in read mode, tdp_mmu_set_spte_atomic * should be used. If operating under the MMU lock in write mode, the * use of the removed SPTE should not be necessary. */ - WARN_ON(is_removed_spte(iter->old_spte)); + WARN_ON(is_removed_spte(iter->old_spte) || is_removed_spte(new_spte)); =20 kvm_tdp_mmu_write_spte(iter->sptep, new_spte); =20 --=20 2.35.1.574.g5d30c73bfb-goog From nobody Tue Jun 23 21:22:54 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C2FB8C433EF for ; Sat, 26 Feb 2022 00:17:06 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235340AbiBZARg (ORCPT ); Fri, 25 Feb 2022 19:17:36 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50784 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240663AbiBZARC (ORCPT ); Fri, 25 Feb 2022 19:17:02 -0500 Received: from mail-pg1-x549.google.com (mail-pg1-x549.google.com [IPv6:2607:f8b0:4864:20::549]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E6F0722559D for ; Fri, 25 Feb 2022 16:16:21 -0800 (PST) Received: by mail-pg1-x549.google.com with SMTP id d192-20020a6336c9000000b00372eb4c4bf4so3477406pga.8 for ; Fri, 25 Feb 2022 16:16:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=reply-to:date:in-reply-to:message-id:mime-version:references :subject:from:to:cc; bh=ENO0x4uKs4rqsjN1xB0qFJMoRAxQ1GMRSi+kdo7qAvw=; b=lhfMZMr/HcUZRGRAalYnJGHqVZsOjXMz94FZ2392zvT3nXG8ZEgYlnAfpVuqMVoG4s vQnfMTJMlL4JHX/+6hN/2my3McSMWbIfCGT6gZ8CcdMU2jt55aPfoOq+mJ537gXev2b/ VzD9jYFmAZzINuPpAeNPRZTzzRu2CyVxgpdgd29mOXSqrGSYjtxI21NR5V1R1UsOcg++ L8W60IWSB0jZ/uevf/LPKOAKmxPlXBOWyTJ8JdcedPlKLPqlnjMs3IXhYm/wKnASI0P0 ML2/OiOmwfAyxlR4JPds616MmFjqFn7uhffuWzWSo0pMZRWbtdME4x5w/8KPEc16P4vE hNdA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:reply-to:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=ENO0x4uKs4rqsjN1xB0qFJMoRAxQ1GMRSi+kdo7qAvw=; b=ArZ++U9qiExCtbcv4X0wDU0n/8vz2+ixIXgtLVSzX7ehQVnRXEPCS0RCAwjOwJhHbR qQim1lf903mQgQEIKgXfkH3QpDn/rMAFbmgwJEQq8sBht7/UVv2uQMf0h+eGHQjhSZws DGabwrL/AF7MmHAkVuaNHXwfDbnxp+SadYnW5/QfBZBH6YkgbGPPOu24OqaLDWw9+4dY cVM3eS2A0XlnqO7HTxr5n//gbDrbbmoTZ/tzwYQVcqooqSnbhKtGpZC94KtVpFaCoEO/ WtrQzrt6kUxSHXKYfMsu+GyrZrVLvCb9FTd0DyYd9R3W0PF/b0SRqN82sb8Fz4mi2wCI cvrg== X-Gm-Message-State: AOAM532kJuikOwJ8+cemO42SIkYTaIO7kymvyAsgM4lqTXfhpYXOvxDP g9jZEboWCNzSIoMoudpJcIBm6eJJ5as= X-Google-Smtp-Source: ABdhPJzrbxQn7NgpNV/g9aLZwydofMo3rhOO2xpfsf/UPmMpU4CrBDA6znpW6k2RNcYcQVgoM9BmqlkXkHA= X-Received: from seanjc.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e5]) (user=seanjc job=sendgmr) by 2002:a17:90a:f02:b0:1bd:ab6:42ec with SMTP id 2-20020a17090a0f0200b001bd0ab642ecmr2190693pjy.18.1645834581467; Fri, 25 Feb 2022 16:16:21 -0800 (PST) Reply-To: Sean Christopherson Date: Sat, 26 Feb 2022 00:15:30 +0000 In-Reply-To: <20220226001546.360188-1-seanjc@google.com> Message-Id: <20220226001546.360188-13-seanjc@google.com> Mime-Version: 1.0 References: <20220226001546.360188-1-seanjc@google.com> X-Mailer: git-send-email 2.35.1.574.g5d30c73bfb-goog Subject: [PATCH v3 12/28] KVM: x86/mmu: Refactor low-level TDP MMU set SPTE helper to take raw vals From: Sean Christopherson To: Paolo Bonzini , Christian Borntraeger , Janosch Frank , Claudio Imbrenda Cc: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , David Hildenbrand , kvm@vger.kernel.org, linux-kernel@vger.kernel.org, David Matlack , Ben Gardon , Mingwei Zhang Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Refactor __tdp_mmu_set_spte() to work with raw values instead of a tdp_iter objects so that a future patch can modify SPTEs without doing a walk, and without having to synthesize a tdp_iter. No functional change intended. Signed-off-by: Sean Christopherson Reviewed-by: Ben Gardon Reviewed-by: Mingwei Zhang --- arch/x86/kvm/mmu/tdp_mmu.c | 51 +++++++++++++++++++++++--------------- 1 file changed, 31 insertions(+), 20 deletions(-) diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c index 1dcdf1a4fcc1..9e8ba6f12ebf 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.c +++ b/arch/x86/kvm/mmu/tdp_mmu.c @@ -617,9 +617,13 @@ static inline int tdp_mmu_zap_spte_atomic(struct kvm *= kvm, =20 /* * __tdp_mmu_set_spte - Set a TDP MMU SPTE and handle the associated bookk= eeping - * @kvm: kvm instance - * @iter: a tdp_iter instance currently on the SPTE that should be set - * @new_spte: The value the SPTE should be set to + * @kvm: KVM instance + * @as_id: Address space ID, i.e. regular vs. SMM + * @sptep: Pointer to the SPTE + * @old_spte: The current value of the SPTE + * @new_spte: The new value that will be set for the SPTE + * @gfn: The base GFN that was (or will be) mapped by the SPTE + * @level: The level _containing_ the SPTE (its parent PT's level) * @record_acc_track: Notify the MM subsystem of changes to the accessed s= tate * of the page. Should be set unless handling an MMU * notifier for access tracking. Leaving record_acc_track @@ -631,12 +635,10 @@ static inline int tdp_mmu_zap_spte_atomic(struct kvm = *kvm, * Leaving record_dirty_log unset in that case prevents page * writes from being double counted. */ -static inline void __tdp_mmu_set_spte(struct kvm *kvm, struct tdp_iter *it= er, - u64 new_spte, bool record_acc_track, - bool record_dirty_log) +static void __tdp_mmu_set_spte(struct kvm *kvm, int as_id, tdp_ptep_t spte= p, + u64 old_spte, u64 new_spte, gfn_t gfn, int level, + bool record_acc_track, bool record_dirty_log) { - WARN_ON_ONCE(iter->yielded); - lockdep_assert_held_write(&kvm->mmu_lock); =20 /* @@ -646,39 +648,48 @@ static inline void __tdp_mmu_set_spte(struct kvm *kvm= , struct tdp_iter *iter, * should be used. If operating under the MMU lock in write mode, the * use of the removed SPTE should not be necessary. */ - WARN_ON(is_removed_spte(iter->old_spte) || is_removed_spte(new_spte)); + WARN_ON(is_removed_spte(old_spte) || is_removed_spte(new_spte)); =20 - kvm_tdp_mmu_write_spte(iter->sptep, new_spte); + kvm_tdp_mmu_write_spte(sptep, new_spte); + + __handle_changed_spte(kvm, as_id, gfn, old_spte, new_spte, level, false); =20 - __handle_changed_spte(kvm, iter->as_id, iter->gfn, iter->old_spte, - new_spte, iter->level, false); if (record_acc_track) - handle_changed_spte_acc_track(iter->old_spte, new_spte, - iter->level); + handle_changed_spte_acc_track(old_spte, new_spte, level); if (record_dirty_log) - handle_changed_spte_dirty_log(kvm, iter->as_id, iter->gfn, - iter->old_spte, new_spte, - iter->level); + handle_changed_spte_dirty_log(kvm, as_id, gfn, old_spte, + new_spte, level); +} + +static inline void _tdp_mmu_set_spte(struct kvm *kvm, struct tdp_iter *ite= r, + u64 new_spte, bool record_acc_track, + bool record_dirty_log) +{ + WARN_ON_ONCE(iter->yielded); + + __tdp_mmu_set_spte(kvm, iter->as_id, iter->sptep, iter->old_spte, + new_spte, iter->gfn, iter->level, + record_acc_track, record_dirty_log); } =20 static inline void tdp_mmu_set_spte(struct kvm *kvm, struct tdp_iter *iter, u64 new_spte) { - __tdp_mmu_set_spte(kvm, iter, new_spte, true, true); + _tdp_mmu_set_spte(kvm, iter, new_spte, true, true); } =20 static inline void tdp_mmu_set_spte_no_acc_track(struct kvm *kvm, struct tdp_iter *iter, u64 new_spte) { - __tdp_mmu_set_spte(kvm, iter, new_spte, false, true); + _tdp_mmu_set_spte(kvm, iter, new_spte, false, true); } =20 static inline void tdp_mmu_set_spte_no_dirty_log(struct kvm *kvm, struct tdp_iter *iter, u64 new_spte) { - __tdp_mmu_set_spte(kvm, iter, new_spte, true, false); + _tdp_mmu_set_spte(kvm, iter, new_spte, true, false); } =20 #define tdp_root_for_each_pte(_iter, _root, _start, _end) \ --=20 2.35.1.574.g5d30c73bfb-goog From nobody Tue Jun 23 21:22:54 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E8874C433EF for ; Sat, 26 Feb 2022 00:17:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236228AbiBZARr (ORCPT ); Fri, 25 Feb 2022 19:17:47 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50808 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240726AbiBZARC (ORCPT ); Fri, 25 Feb 2022 19:17:02 -0500 Received: from mail-pg1-x549.google.com (mail-pg1-x549.google.com [IPv6:2607:f8b0:4864:20::549]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CDC3A2255AB for ; Fri, 25 Feb 2022 16:16:23 -0800 (PST) Received: by mail-pg1-x549.google.com with SMTP id b5-20020a631b05000000b00373bd90134dso3462449pgb.22 for ; Fri, 25 Feb 2022 16:16:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=reply-to:date:in-reply-to:message-id:mime-version:references :subject:from:to:cc; bh=Nb5T6qukJfhOngauD2ESxpCO0BeOqZQialSaNjnnS68=; b=YRhZoUpI1WKzeeuxXZCk7f2N6l4lfLKrp/cTyzEUwa85H3cFJoLfqi83eCctqMW7Sy PO4FOeZtACnJq/1EfPBDGPHhsxlusMOAKKNVJuWVwid/8z0LbaWU7d4wuSNrwSaELNTa dOTtF/gQ8jz+CyzwdFWdxOd7guqCQ/4EDBxXvC8t8EwLmJRPzShQcN6DWGkv70B6rmR+ UcWCqS6GfPRrvpwUAEuv97Bso93hOkxN5wZIxlyKbW3TuxXHgGHpUHBzpkmK0jh6QjKf sbjhpKJiGl1Vl4mbfn4bBNm/pWlUJ+3tK8rJR135lKfwhAKh7/0BejsFrgr9/86gJOvR gXRQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:reply-to:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=Nb5T6qukJfhOngauD2ESxpCO0BeOqZQialSaNjnnS68=; b=wVIoKWrms5Mizqg6ZNR2VpNWtlZmoK0qCVamUdBjhQPkOMsJAmDU93MZ10msdh977O UiacxVFW6boCZdO4oKZRWt/jspTFTuQjBn3+eXfNr7EzKF4X6N3mln/00hH6iQgtDQhO jgezTdo6ObSZb/1I9V4438FNsNOtGMRBlqMaM38uFry2zgo1dmGojzHPcfuKIVFml4Hg udQ/bgSM39XeLIUEMMoEhg19Ro7u0kJKdxkH2xVH4ZVH+n1CsViRK6yfLybKfHCcQAGT x3l52RJsHgWVSkNXbrvXXS6TyBukFTSO3Z0VNqmeYN2/BU9Pv8Pf6kY/pWFXyAK0m+f0 sBqw== X-Gm-Message-State: AOAM5313BC06jfq5I8p6KcFczH/4eokL1ZqpphtqB3n3yOuuF54U/DRm /lpKFVSan6yTNCD9q/vX/OnmYYIjTiI= X-Google-Smtp-Source: ABdhPJwCvauD91w5CnudMRn251U0c/3mR0mjM4DYE6FTyCQ8OHHKIg0SuJEnTbT2acnNwjL5J5yrQTyQP5o= X-Received: from seanjc.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e5]) (user=seanjc job=sendgmr) by 2002:a17:902:be10:b0:14d:66b5:7065 with SMTP id r16-20020a170902be1000b0014d66b57065mr9466370pls.81.1645834583158; Fri, 25 Feb 2022 16:16:23 -0800 (PST) Reply-To: Sean Christopherson Date: Sat, 26 Feb 2022 00:15:31 +0000 In-Reply-To: <20220226001546.360188-1-seanjc@google.com> Message-Id: <20220226001546.360188-14-seanjc@google.com> Mime-Version: 1.0 References: <20220226001546.360188-1-seanjc@google.com> X-Mailer: git-send-email 2.35.1.574.g5d30c73bfb-goog Subject: [PATCH v3 13/28] KVM: x86/mmu: Zap only the target TDP MMU shadow page in NX recovery From: Sean Christopherson To: Paolo Bonzini , Christian Borntraeger , Janosch Frank , Claudio Imbrenda Cc: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , David Hildenbrand , kvm@vger.kernel.org, linux-kernel@vger.kernel.org, David Matlack , Ben Gardon , Mingwei Zhang Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" When recovering a potential hugepage that was shattered for the iTLB multihit workaround, precisely zap only the target page instead of iterating over the TDP MMU to find the SP that was passed in. This will allow future simplification of zap_gfn_range() by having it zap only leaf SPTEs. Signed-off-by: Sean Christopherson --- arch/x86/kvm/mmu/mmu_internal.h | 7 ++++++- arch/x86/kvm/mmu/tdp_iter.h | 2 -- arch/x86/kvm/mmu/tdp_mmu.c | 36 +++++++++++++++++++++++++++++---- arch/x86/kvm/mmu/tdp_mmu.h | 18 +---------------- 4 files changed, 39 insertions(+), 24 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu_internal.h b/arch/x86/kvm/mmu/mmu_interna= l.h index da6166b5c377..be063b6c91b7 100644 --- a/arch/x86/kvm/mmu/mmu_internal.h +++ b/arch/x86/kvm/mmu/mmu_internal.h @@ -30,6 +30,8 @@ extern bool dbg; #define INVALID_PAE_ROOT 0 #define IS_VALID_PAE_ROOT(x) (!!(x)) =20 +typedef u64 __rcu *tdp_ptep_t; + struct kvm_mmu_page { /* * Note, "link" through "spt" fit in a single 64 byte cache line on @@ -59,7 +61,10 @@ struct kvm_mmu_page { refcount_t tdp_mmu_root_count; }; unsigned int unsync_children; - struct kvm_rmap_head parent_ptes; /* rmap pointers to parent sptes */ + union { + struct kvm_rmap_head parent_ptes; /* rmap pointers to parent sptes */ + tdp_ptep_t ptep; + }; DECLARE_BITMAP(unsync_child_bitmap, 512); =20 struct list_head lpage_disallowed_link; diff --git a/arch/x86/kvm/mmu/tdp_iter.h b/arch/x86/kvm/mmu/tdp_iter.h index bb9b581f1ee4..e2a7e267a77d 100644 --- a/arch/x86/kvm/mmu/tdp_iter.h +++ b/arch/x86/kvm/mmu/tdp_iter.h @@ -7,8 +7,6 @@ =20 #include "mmu.h" =20 -typedef u64 __rcu *tdp_ptep_t; - /* * TDP MMU SPTEs are RCU protected to allow paging structures (non-leaf SP= TEs) * to be zapped while holding mmu_lock for read. Holding RCU isn't requir= ed for diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c index 9e8ba6f12ebf..c231b60e1726 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.c +++ b/arch/x86/kvm/mmu/tdp_mmu.c @@ -213,13 +213,14 @@ static struct kvm_mmu_page *tdp_mmu_alloc_sp(struct k= vm_vcpu *vcpu) return sp; } =20 -static void tdp_mmu_init_sp(struct kvm_mmu_page *sp, gfn_t gfn, - union kvm_mmu_page_role role) +static void tdp_mmu_init_sp(struct kvm_mmu_page *sp, tdp_ptep_t sptep, + gfn_t gfn, union kvm_mmu_page_role role) { set_page_private(virt_to_page(sp->spt), (unsigned long)sp); =20 sp->role =3D role; sp->gfn =3D gfn; + sp->ptep =3D sptep; sp->tdp_mmu_page =3D true; =20 trace_kvm_mmu_get_page(sp, true); @@ -236,7 +237,7 @@ static void tdp_mmu_init_child_sp(struct kvm_mmu_page *= child_sp, role =3D parent_sp->role; role.level--; =20 - tdp_mmu_init_sp(child_sp, iter->gfn, role); + tdp_mmu_init_sp(child_sp, iter->sptep, iter->gfn, role); } =20 hpa_t kvm_tdp_mmu_get_vcpu_root_hpa(struct kvm_vcpu *vcpu) @@ -258,7 +259,7 @@ hpa_t kvm_tdp_mmu_get_vcpu_root_hpa(struct kvm_vcpu *vc= pu) } =20 root =3D tdp_mmu_alloc_sp(vcpu); - tdp_mmu_init_sp(root, 0, role); + tdp_mmu_init_sp(root, NULL, 0, role); =20 refcount_set(&root->tdp_mmu_root_count, 1); =20 @@ -750,6 +751,33 @@ static inline bool __must_check tdp_mmu_iter_cond_resc= hed(struct kvm *kvm, return iter->yielded; } =20 +bool kvm_tdp_mmu_zap_sp(struct kvm *kvm, struct kvm_mmu_page *sp) +{ + u64 old_spte; + + /* + * This helper intentionally doesn't allow zapping a root shadow page, + * which doesn't have a parent page table and thus no associated entry. + */ + if (WARN_ON_ONCE(!sp->ptep)) + return false; + + rcu_read_lock(); + + old_spte =3D kvm_tdp_mmu_read_spte(sp->ptep); + if (WARN_ON_ONCE(!is_shadow_present_pte(old_spte))) { + rcu_read_unlock(); + return false; + } + + __tdp_mmu_set_spte(kvm, kvm_mmu_page_as_id(sp), sp->ptep, old_spte, 0, + sp->gfn, sp->role.level + 1, true, true); + + rcu_read_unlock(); + + return true; +} + /* * Tears down the mappings for the range of gfns, [start, end), and frees = the * non-root pages mapping GFNs strictly within that range. Returns true if diff --git a/arch/x86/kvm/mmu/tdp_mmu.h b/arch/x86/kvm/mmu/tdp_mmu.h index 57c73d8f76ce..5e5ef2576c81 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.h +++ b/arch/x86/kvm/mmu/tdp_mmu.h @@ -22,24 +22,8 @@ static inline bool kvm_tdp_mmu_zap_gfn_range(struct kvm = *kvm, int as_id, { return __kvm_tdp_mmu_zap_gfn_range(kvm, as_id, start, end, true, flush); } -static inline bool kvm_tdp_mmu_zap_sp(struct kvm *kvm, struct kvm_mmu_page= *sp) -{ - gfn_t end =3D sp->gfn + KVM_PAGES_PER_HPAGE(sp->role.level + 1); - - /* - * Don't allow yielding, as the caller may have a flush pending. Note, - * if mmu_lock is held for write, zapping will never yield in this case, - * but explicitly disallow it for safety. The TDP MMU does not yield - * until it has made forward progress (steps sideways), and when zapping - * a single shadow page that it's guaranteed to see (thus the mmu_lock - * requirement), its "step sideways" will always step beyond the bounds - * of the shadow page's gfn range and stop iterating before yielding. - */ - lockdep_assert_held_write(&kvm->mmu_lock); - return __kvm_tdp_mmu_zap_gfn_range(kvm, kvm_mmu_page_as_id(sp), - sp->gfn, end, false, false); -} =20 +bool kvm_tdp_mmu_zap_sp(struct kvm *kvm, struct kvm_mmu_page *sp); void kvm_tdp_mmu_zap_all(struct kvm *kvm); void kvm_tdp_mmu_invalidate_all_roots(struct kvm *kvm); void kvm_tdp_mmu_zap_invalidated_roots(struct kvm *kvm); --=20 2.35.1.574.g5d30c73bfb-goog From nobody Tue Jun 23 21:22:54 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 71CC1C433EF for ; Sat, 26 Feb 2022 00:17:42 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234801AbiBZASK (ORCPT ); Fri, 25 Feb 2022 19:18:10 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50826 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240459AbiBZARC (ORCPT ); Fri, 25 Feb 2022 19:17:02 -0500 Received: from mail-pf1-x44a.google.com (mail-pf1-x44a.google.com [IPv6:2607:f8b0:4864:20::44a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 835AC2261DE for ; Fri, 25 Feb 2022 16:16:25 -0800 (PST) Received: by mail-pf1-x44a.google.com with SMTP id t134-20020a62788c000000b004e1367caccaso3948321pfc.14 for ; Fri, 25 Feb 2022 16:16:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=reply-to:date:in-reply-to:message-id:mime-version:references :subject:from:to:cc; bh=JDThRSqN0byFLKsQYI89Co7AKmuBCHnME5v6WomODDw=; b=KnNEuHSDYY9Qpv8Sc7pMTFoKsQ1o1irShtKNr/STHSxYwJLaYyi5K1uzyhNvsRIB+t GmneOFlCNqt6+D5k8U9urB6oYaTV70HwxQqPey79vYIYg2VXQBvVhwTPKie/2bAcGEen yw0Omtgx2d7PvTH+1z372LtMBDGvi4yvNRQ3516OqK89vTJfmn98v81FAVz13FI/gcb0 FMOYaSUSX8sDJ4F/5NY/WaHzV00pF8Aw3qeWDHuBihAGUcaxx+SaNk6vPwupm8BgvMUt YT9pjc7hdVJFH6fBi0ci5quzkscOsEQIKLLvKn6WqvEuBmzO92jF8ncfBNTNQJv6r1rA c1fQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:reply-to:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=JDThRSqN0byFLKsQYI89Co7AKmuBCHnME5v6WomODDw=; b=1MuTGzLSHzllG89cTQ/JijHzkbluiFX9kc7wjCh4HXD0P4gjogp04bPLR7resKjXqA 579kwLibJG0NILt+kwlzNdVfNlwkidiepaaILsp98m4/uuCaWDv5wEvIpfExy5Lh3Vf7 gXVvK5OwzDJyXUgeSjcSNdzLCjPtUCs0TZxqAQ243APz+3+vocl93y3Y9aApfOtQPd5k l3fF5sXJh/Pa2smszh6aIGUCnYnmIksGtp74OlewXOguoTPgb1PdCxneK/39v1qXw2P+ CO5NblzfPioykmM0uAa2a2LODGJWe3jJW4hb3yPyuv4+e5gSqsFxynRK6nMAsAU6PygL Mnjg== X-Gm-Message-State: AOAM5307PA1SKaZNR3GEVgb7bli9QfjsO0KguGNejMnHcCcUAOr1s6KL q35D8t3MXAvTTmVgH1mLYz2MgjrvyEo= X-Google-Smtp-Source: ABdhPJyDXjmJG5ubpBBt2IByek7X49XHlU8cpZNg0E33OFPw7Jkh1SIZ8sqLHR0uUVnkCt9UJDZ4mM68s6g= X-Received: from seanjc.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e5]) (user=seanjc job=sendgmr) by 2002:a17:902:b906:b0:14f:76a0:ad48 with SMTP id bf6-20020a170902b90600b0014f76a0ad48mr9902830plb.79.1645834584804; Fri, 25 Feb 2022 16:16:24 -0800 (PST) Reply-To: Sean Christopherson Date: Sat, 26 Feb 2022 00:15:32 +0000 In-Reply-To: <20220226001546.360188-1-seanjc@google.com> Message-Id: <20220226001546.360188-15-seanjc@google.com> Mime-Version: 1.0 References: <20220226001546.360188-1-seanjc@google.com> X-Mailer: git-send-email 2.35.1.574.g5d30c73bfb-goog Subject: [PATCH v3 14/28] KVM: x86/mmu: Skip remote TLB flush when zapping all of TDP MMU From: Sean Christopherson To: Paolo Bonzini , Christian Borntraeger , Janosch Frank , Claudio Imbrenda Cc: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , David Hildenbrand , kvm@vger.kernel.org, linux-kernel@vger.kernel.org, David Matlack , Ben Gardon , Mingwei Zhang Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Don't flush the TLBs when zapping all TDP MMU pages, as the only time KVM uses the slow version of "zap everything" is when the VM is being destroyed or the owning mm has exited. In either case, KVM_RUN is unreachable for the VM, i.e. the guest TLB entries cannot be consumed. Signed-off-by: Sean Christopherson Reviewed-by: Ben Gardon Reviewed-by: Mingwei Zhang --- arch/x86/kvm/mmu/tdp_mmu.c | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c index c231b60e1726..87706e9cc6f3 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.c +++ b/arch/x86/kvm/mmu/tdp_mmu.c @@ -874,14 +874,15 @@ bool __kvm_tdp_mmu_zap_gfn_range(struct kvm *kvm, int= as_id, gfn_t start, =20 void kvm_tdp_mmu_zap_all(struct kvm *kvm) { - bool flush =3D false; int i; =20 + /* + * A TLB flush is unnecessary, KVM zaps everything if and only the VM + * is being destroyed or the userspace VMM has exited. In both cases, + * KVM_RUN is unreachable, i.e. no vCPUs will ever service the request. + */ for (i =3D 0; i < KVM_ADDRESS_SPACE_NUM; i++) - flush =3D kvm_tdp_mmu_zap_gfn_range(kvm, i, 0, -1ull, flush); - - if (flush) - kvm_flush_remote_tlbs(kvm); + (void)kvm_tdp_mmu_zap_gfn_range(kvm, i, 0, -1ull, false); } =20 /* --=20 2.35.1.574.g5d30c73bfb-goog From nobody Tue Jun 23 21:22:54 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id AD940C4332F for ; Sat, 26 Feb 2022 00:17:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S241395AbiBZARy (ORCPT ); Fri, 25 Feb 2022 19:17:54 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50992 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240794AbiBZARG (ORCPT ); Fri, 25 Feb 2022 19:17:06 -0500 Received: from mail-pj1-x1049.google.com (mail-pj1-x1049.google.com [IPv6:2607:f8b0:4864:20::1049]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 31AC02261FC for ; Fri, 25 Feb 2022 16:16:27 -0800 (PST) Received: by mail-pj1-x1049.google.com with SMTP id f4-20020a17090ac28400b001bc40aa09fbso6462289pjt.6 for ; Fri, 25 Feb 2022 16:16:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=reply-to:date:in-reply-to:message-id:mime-version:references :subject:from:to:cc; bh=ws25+vfJGJ9kiCAG7dqzkUUc7DAFp/oKX41ZhFLe5UQ=; b=EDX5h2qsUQnUe+gIppEHl+tdDc1N++5/Bs6ytSHiECgi17QgtD/4XIdw27t7GVxpVP NbyTk3PmRYbE5hwuM5x98Rr/5UJFbP1mbGP1Y9VXkdXx4hO9OJEwesSpenA3jlVVFRAy ZYyZ4C9R5kOM7pah1pA5yTMcyyGzXCMPoYLTKklta6BTT9zSaQtX/Tx+tGw3pq6zxIaG fHBlyaNaOZthq7/x+pMGnBlPsZ7UBWxKwJSSil4cjqrQOt329s0sY0DQ4LP8sIXYTN6i eLmTCnZsKdEvoKtb3Wo5G3iVfWjnU0xF0bHkI0hjq36ibWemxhiWR2cX5ZPyU733Vc5W /TVQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:reply-to:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=ws25+vfJGJ9kiCAG7dqzkUUc7DAFp/oKX41ZhFLe5UQ=; b=J99O+cUsPRl/5BpLBnAWlXE/I2rMcOsZBqwQvIuCLF42PgVE26G5V+5JFMV/LBYusp AZJz7O/QldQZ6yila7aGuT8dEfQe11lrSSNhY8Gr+Xt21u9zJS/twyPXj4/xcOfetn+n Wo8oVdqDF1BemgTkJ+CluUyzeWaCXAko1/oHoGXQliL9KVZbdW7LhG2z5xEVDPVLrIKq wTdW5P59u2qaRgGiU2TiO2MmJcZRTxBuxXT2xbuhb2IGRZs07DWDRMz54t3ooQ2nSMf6 Fg2MdYIIPOD6iIUxLZX72OURp4RKSsvOgRRmeBtKHosIvSXCEdJG3if4RCrqhhH7pshI YKVA== X-Gm-Message-State: AOAM532rQqU1dOmcMSA7N8/HYsZ++82FnHVDTNlaXEBLt7vJ7wn6vhLe BFvRwYQXrA4WHCtEwnIueuYd1dzBV9E= X-Google-Smtp-Source: ABdhPJz93wb0tPC1EcHKqcvhBStFIe6bqLyXA3MKbqEkEWEDp1vZtIrsJbtsTBuwKMPFXGwOuYuznq6YWTw= X-Received: from seanjc.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e5]) (user=seanjc job=sendgmr) by 2002:aa7:88d1:0:b0:4f3:6132:42e1 with SMTP id k17-20020aa788d1000000b004f3613242e1mr10048864pff.67.1645834586595; Fri, 25 Feb 2022 16:16:26 -0800 (PST) Reply-To: Sean Christopherson Date: Sat, 26 Feb 2022 00:15:33 +0000 In-Reply-To: <20220226001546.360188-1-seanjc@google.com> Message-Id: <20220226001546.360188-16-seanjc@google.com> Mime-Version: 1.0 References: <20220226001546.360188-1-seanjc@google.com> X-Mailer: git-send-email 2.35.1.574.g5d30c73bfb-goog Subject: [PATCH v3 15/28] KVM: x86/mmu: Add dedicated helper to zap TDP MMU root shadow page From: Sean Christopherson To: Paolo Bonzini , Christian Borntraeger , Janosch Frank , Claudio Imbrenda Cc: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , David Hildenbrand , kvm@vger.kernel.org, linux-kernel@vger.kernel.org, David Matlack , Ben Gardon , Mingwei Zhang Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add a dedicated helper for zapping a TDP MMU root, and use it in the three flows that do "zap_all" and intentionally do not do a TLB flush if SPTEs are zapped (zapping an entire root is safe if and only if it cannot be in use by any vCPU). Because a TLB flush is never required, unconditionally pass "false" to tdp_mmu_iter_cond_resched() when potentially yielding. Opportunistically document why KVM must not yield when zapping roots that are being zapped by kvm_tdp_mmu_put_root(), i.e. roots whose refcount has reached zero, and further harden the flow to detect improper KVM behavior with respect to roots that are supposed to be unreachable. In addition to hardening zapping of roots, isolating zapping of roots will allow future simplification of zap_gfn_range() by having it zap only leaf SPTEs, and by removing its tricky "zap all" heuristic. By having all paths that truly need to free _all_ SPs flow through the dedicated root zapper, the generic zapper can be freed of those concerns. Signed-off-by: Sean Christopherson Reviewed-by: Ben Gardon --- arch/x86/kvm/mmu/tdp_mmu.c | 98 +++++++++++++++++++++++++++++++------- 1 file changed, 82 insertions(+), 16 deletions(-) diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c index 87706e9cc6f3..c5df9a552470 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.c +++ b/arch/x86/kvm/mmu/tdp_mmu.c @@ -56,10 +56,6 @@ void kvm_mmu_uninit_tdp_mmu(struct kvm *kvm) rcu_barrier(); } =20 -static bool zap_gfn_range(struct kvm *kvm, struct kvm_mmu_page *root, - gfn_t start, gfn_t end, bool can_yield, bool flush, - bool shared); - static void tdp_mmu_free_sp(struct kvm_mmu_page *sp) { free_page((unsigned long)sp->spt); @@ -82,6 +78,9 @@ static void tdp_mmu_free_sp_rcu_callback(struct rcu_head = *head) tdp_mmu_free_sp(sp); } =20 +static void tdp_mmu_zap_root(struct kvm *kvm, struct kvm_mmu_page *root, + bool shared); + void kvm_tdp_mmu_put_root(struct kvm *kvm, struct kvm_mmu_page *root, bool shared) { @@ -104,7 +103,7 @@ void kvm_tdp_mmu_put_root(struct kvm *kvm, struct kvm_m= mu_page *root, * intermediate paging structures, that may be zapped, as such entries * are associated with the ASID on both VMX and SVM. */ - (void)zap_gfn_range(kvm, root, 0, -1ull, false, false, shared); + tdp_mmu_zap_root(kvm, root, shared); =20 call_rcu(&root->rcu_head, tdp_mmu_free_sp_rcu_callback); } @@ -751,6 +750,76 @@ static inline bool __must_check tdp_mmu_iter_cond_resc= hed(struct kvm *kvm, return iter->yielded; } =20 +static inline gfn_t tdp_mmu_max_gfn_host(void) +{ + /* + * Bound TDP MMU walks at host.MAXPHYADDR, guest accesses beyond that + * will hit a #PF(RSVD) and never hit an EPT Violation/Misconfig / #NPF, + * and so KVM will never install a SPTE for such addresses. + */ + return 1ULL << (shadow_phys_bits - PAGE_SHIFT); +} + +static void tdp_mmu_zap_root(struct kvm *kvm, struct kvm_mmu_page *root, + bool shared) +{ + bool root_is_unreachable =3D !refcount_read(&root->tdp_mmu_root_count); + struct tdp_iter iter; + + gfn_t end =3D tdp_mmu_max_gfn_host(); + gfn_t start =3D 0; + + kvm_lockdep_assert_mmu_lock_held(kvm, shared); + + rcu_read_lock(); + + /* + * No need to try to step down in the iterator when zapping an entire + * root, zapping an upper-level SPTE will recurse on its children. + */ + for_each_tdp_pte_min_level(iter, root, root->role.level, start, end) { +retry: + /* + * Yielding isn't allowed when zapping an unreachable root as + * the root won't be processed by mmu_notifier callbacks. When + * handling an unmap/release mmu_notifier command, KVM must + * drop all references to relevant pages prior to completing + * the callback. Dropping mmu_lock can result in zapping SPTEs + * for an unreachable root after a relevant callback completes, + * which leads to use-after-free as zapping a SPTE triggers + * "writeback" of dirty/accessed bits to the SPTE's associated + * struct page. + */ + if (!root_is_unreachable && + tdp_mmu_iter_cond_resched(kvm, &iter, false, shared)) + continue; + + if (!is_shadow_present_pte(iter.old_spte)) + continue; + + if (!shared) { + tdp_mmu_set_spte(kvm, &iter, 0); + } else if (tdp_mmu_set_spte_atomic(kvm, &iter, 0)) { + /* + * cmpxchg() shouldn't fail if the root is unreachable. + * Retry so as not to leak the page and its children. + */ + WARN_ONCE(root_is_unreachable, + "Contended TDP MMU SPTE in unreachable root."); + goto retry; + } + + /* + * WARN if the root is invalid and is unreachable, all SPTEs + * should've been zapped by kvm_tdp_mmu_zap_invalidated_roots(), + * and inserting new SPTEs under an invalid root is a KVM bug. + */ + WARN_ON_ONCE(root_is_unreachable && root->role.invalid); + } + + rcu_read_unlock(); +} + bool kvm_tdp_mmu_zap_sp(struct kvm *kvm, struct kvm_mmu_page *sp) { u64 old_spte; @@ -799,8 +868,7 @@ static bool zap_gfn_range(struct kvm *kvm, struct kvm_m= mu_page *root, gfn_t start, gfn_t end, bool can_yield, bool flush, bool shared) { - gfn_t max_gfn_host =3D 1ULL << (shadow_phys_bits - PAGE_SHIFT); - bool zap_all =3D (start =3D=3D 0 && end >=3D max_gfn_host); + bool zap_all =3D (start =3D=3D 0 && end >=3D tdp_mmu_max_gfn_host()); struct tdp_iter iter; =20 /* @@ -809,12 +877,7 @@ static bool zap_gfn_range(struct kvm *kvm, struct kvm_= mmu_page *root, */ int min_level =3D zap_all ? root->role.level : PG_LEVEL_4K; =20 - /* - * Bound the walk at host.MAXPHYADDR, guest accesses beyond that will - * hit a #PF(RSVD) and never get to an EPT Violation/Misconfig / #NPF, - * and so KVM will never install a SPTE for such addresses. - */ - end =3D min(end, max_gfn_host); + end =3D min(end, tdp_mmu_max_gfn_host()); =20 kvm_lockdep_assert_mmu_lock_held(kvm, shared); =20 @@ -874,6 +937,7 @@ bool __kvm_tdp_mmu_zap_gfn_range(struct kvm *kvm, int a= s_id, gfn_t start, =20 void kvm_tdp_mmu_zap_all(struct kvm *kvm) { + struct kvm_mmu_page *root; int i; =20 /* @@ -881,8 +945,10 @@ void kvm_tdp_mmu_zap_all(struct kvm *kvm) * is being destroyed or the userspace VMM has exited. In both cases, * KVM_RUN is unreachable, i.e. no vCPUs will ever service the request. */ - for (i =3D 0; i < KVM_ADDRESS_SPACE_NUM; i++) - (void)kvm_tdp_mmu_zap_gfn_range(kvm, i, 0, -1ull, false); + for (i =3D 0; i < KVM_ADDRESS_SPACE_NUM; i++) { + for_each_tdp_mmu_root_yield_safe(kvm, root, i, false) + tdp_mmu_zap_root(kvm, root, false); + } } =20 /* @@ -908,7 +974,7 @@ void kvm_tdp_mmu_zap_invalidated_roots(struct kvm *kvm) * will still flush on yield, but that's a minor performance * blip and not a functional issue. */ - (void)zap_gfn_range(kvm, root, 0, -1ull, true, false, true); + tdp_mmu_zap_root(kvm, root, true); =20 /* * Put the reference acquired in kvm_tdp_mmu_invalidate_roots(). --=20 2.35.1.574.g5d30c73bfb-goog From nobody Tue Jun 23 21:22:54 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 969F6C433F5 for ; Sat, 26 Feb 2022 00:17:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S241153AbiBZARv (ORCPT ); Fri, 25 Feb 2022 19:17:51 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51038 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240817AbiBZARH (ORCPT ); Fri, 25 Feb 2022 19:17:07 -0500 Received: from mail-pj1-x1049.google.com (mail-pj1-x1049.google.com [IPv6:2607:f8b0:4864:20::1049]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 738EE227595 for ; Fri, 25 Feb 2022 16:16:28 -0800 (PST) Received: by mail-pj1-x1049.google.com with SMTP id f33-20020a17090a702400b001bc822ccbacso6496037pjk.0 for ; Fri, 25 Feb 2022 16:16:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=reply-to:date:in-reply-to:message-id:mime-version:references :subject:from:to:cc; bh=jE7hvTIwynzoEhqj9gg0C8wmmkSrwJFvj0OAsRaPAlA=; b=srFn6/Gu6kkdg0tWw8AP8f0ag5lnSoNgNzSyKt42ty/ArJ0sDsbGxDiqG9F8WTo4+i s8o3gb4AqaCFm5tB68JP2sp3xMK4tDhMaRXTCNoW8ye0qMuuOB+k/sXSxj4EHbzDw9XD bLZHLWHYvbyp245DKxFfD5ART5g0xwZkpyxaH9wDt07/SvRGs4J/TWTClLP+tHywhJYF q43b5YGS3wGKDHqz5KPZCJ/ALfjcWBtccT0kMd+eNLQAwPZEXJywwx0pZrC8+S7NNd+b 7BPpblxaes9aNc98ZZfjhcFwaCXUOtgodIKlDTiIRS4KASmSRSRgrz8o5RNiE5w/QZna amTA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:reply-to:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=jE7hvTIwynzoEhqj9gg0C8wmmkSrwJFvj0OAsRaPAlA=; b=PQen6Tp3Ro4oUOsyn4RgN3hA374sT51rOYSiLqJgZ8RhqKSbTTmiHCi4r0f441I1f+ lxLgym0IfUQQITGQnSMtT57UfhGpDuJ6LiJlEMBIWzrcgk2vC72cWS0WzXCZh7W5JiLu vOHwy3Unh2kora14Cm1sWmcCQBpDDjznIGwBl211+w3bXfbmWgGc4smXiHxtjc5zl/e6 xSl5ANUMgFj8vjuHGPNi9BTm9aZOCyWYo0KXYOvtC6PU/BbLG2t1rlc3FpY9L69033Vq Y923JPIpDig7NQIpOlKcFSX0XTcVJ/Rwit9nm+3yYdoU6ILxcSfgE7dGVhcReBUbV484 KX2w== X-Gm-Message-State: AOAM53035oEBCj35fvUAUOvKBIvxFQat7/ZQVh4k9oHAb8NpwzglU3t6 B0Si+1X8sBBnTuu6W/htvg7t7WpUeq4= X-Google-Smtp-Source: ABdhPJwJp2sMRKVTOQNmfwDXuQFGza9n04onJWCaorDfzPz3xBNBNE+5W7ARgHLaTP78NXYR+Qw1NgKOM9Q= X-Received: from seanjc.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e5]) (user=seanjc job=sendgmr) by 2002:a63:ad47:0:b0:373:4c14:35e2 with SMTP id y7-20020a63ad47000000b003734c1435e2mr8128596pgo.67.1645834588279; Fri, 25 Feb 2022 16:16:28 -0800 (PST) Reply-To: Sean Christopherson Date: Sat, 26 Feb 2022 00:15:34 +0000 In-Reply-To: <20220226001546.360188-1-seanjc@google.com> Message-Id: <20220226001546.360188-17-seanjc@google.com> Mime-Version: 1.0 References: <20220226001546.360188-1-seanjc@google.com> X-Mailer: git-send-email 2.35.1.574.g5d30c73bfb-goog Subject: [PATCH v3 16/28] KVM: x86/mmu: Require mmu_lock be held for write to zap TDP MMU range From: Sean Christopherson To: Paolo Bonzini , Christian Borntraeger , Janosch Frank , Claudio Imbrenda Cc: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , David Hildenbrand , kvm@vger.kernel.org, linux-kernel@vger.kernel.org, David Matlack , Ben Gardon , Mingwei Zhang Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Now that all callers of zap_gfn_range() hold mmu_lock for write, drop support for zapping with mmu_lock held for read. That all callers hold mmu_lock for write isn't a random coincedence; now that the paths that need to zap _everything_ have their own path, the only callers left are those that need to zap for functional correctness. And when zapping is required for functional correctness, mmu_lock must be held for write, otherwise the caller has no guarantees about the state of the TDP MMU page tables after it has run, e.g. the SPTE(s) it zapped can be immediately replaced by a vCPU faulting in a page. Signed-off-by: Sean Christopherson Reviewed-by: Ben Gardon --- arch/x86/kvm/mmu/tdp_mmu.c | 24 ++++++------------------ 1 file changed, 6 insertions(+), 18 deletions(-) diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c index c5df9a552470..b55eb7bec308 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.c +++ b/arch/x86/kvm/mmu/tdp_mmu.c @@ -858,15 +858,9 @@ bool kvm_tdp_mmu_zap_sp(struct kvm *kvm, struct kvm_mm= u_page *sp) * function cannot yield, it will not release the MMU lock or reschedule a= nd * the caller must ensure it does not supply too large a GFN range, or the * operation can cause a soft lockup. - * - * If shared is true, this thread holds the MMU lock in read mode and must - * account for the possibility that other threads are modifying the paging - * structures concurrently. If shared is false, this thread should hold the - * MMU lock in write mode. */ static bool zap_gfn_range(struct kvm *kvm, struct kvm_mmu_page *root, - gfn_t start, gfn_t end, bool can_yield, bool flush, - bool shared) + gfn_t start, gfn_t end, bool can_yield, bool flush) { bool zap_all =3D (start =3D=3D 0 && end >=3D tdp_mmu_max_gfn_host()); struct tdp_iter iter; @@ -879,14 +873,13 @@ static bool zap_gfn_range(struct kvm *kvm, struct kvm= _mmu_page *root, =20 end =3D min(end, tdp_mmu_max_gfn_host()); =20 - kvm_lockdep_assert_mmu_lock_held(kvm, shared); + lockdep_assert_held_write(&kvm->mmu_lock); =20 rcu_read_lock(); =20 for_each_tdp_pte_min_level(iter, root, min_level, start, end) { -retry: if (can_yield && - tdp_mmu_iter_cond_resched(kvm, &iter, flush, shared)) { + tdp_mmu_iter_cond_resched(kvm, &iter, flush, false)) { flush =3D false; continue; } @@ -905,12 +898,8 @@ static bool zap_gfn_range(struct kvm *kvm, struct kvm_= mmu_page *root, !is_last_spte(iter.old_spte, iter.level)) continue; =20 - if (!shared) { - tdp_mmu_set_spte(kvm, &iter, 0); - flush =3D true; - } else if (tdp_mmu_zap_spte_atomic(kvm, &iter)) { - goto retry; - } + tdp_mmu_set_spte(kvm, &iter, 0); + flush =3D true; } =20 rcu_read_unlock(); @@ -929,8 +918,7 @@ bool __kvm_tdp_mmu_zap_gfn_range(struct kvm *kvm, int a= s_id, gfn_t start, struct kvm_mmu_page *root; =20 for_each_tdp_mmu_root_yield_safe(kvm, root, as_id, false) - flush =3D zap_gfn_range(kvm, root, start, end, can_yield, flush, - false); + flush =3D zap_gfn_range(kvm, root, start, end, can_yield, flush); =20 return flush; } --=20 2.35.1.574.g5d30c73bfb-goog From nobody Tue Jun 23 21:22:54 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id CD2FBC433F5 for ; Sat, 26 Feb 2022 00:17:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240931AbiBZASS (ORCPT ); Fri, 25 Feb 2022 19:18:18 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51208 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240875AbiBZARN (ORCPT ); Fri, 25 Feb 2022 19:17:13 -0500 Received: from mail-pj1-x104a.google.com (mail-pj1-x104a.google.com [IPv6:2607:f8b0:4864:20::104a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 536E0214F8B for ; Fri, 25 Feb 2022 16:16:30 -0800 (PST) Received: by mail-pj1-x104a.google.com with SMTP id f4-20020a17090ac28400b001bc40aa09fbso6462366pjt.6 for ; Fri, 25 Feb 2022 16:16:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=reply-to:date:in-reply-to:message-id:mime-version:references :subject:from:to:cc; bh=tRw/deY2VUJH9uZ2P/hJN+/3kIiv3vn+Rqob7NUnZN4=; b=PaaSV5IRPBmtEyd0+pvviWtBTpUOYzerXq4v5mPU1O/7behXkjfV+J9JAFmS98y3PQ ok8BXbQ/aDwJtX5HGEtG1w/zh/2qwN7uxpJvigr49COt9DKOG4fl0kZRN9JM+9bLuQv1 SUux0tSM6dHFd6wAH/c/c+9f9Kx1jnEry9GEKQ+gjno4Yn9BnM93z1tj+/IHjoW0QvV9 IxfMhtrUZUB5nGC0c2J5hkopkVzO4HFC1vDTg3VWnOxZAK/5WCLy3O0+BynDb3GExLSM wacKrzlB5LLFtiIWDU/5ILRyHEXm9xTnqaQj6r9cB4rmajwPQIDBIUw+S8czycZ5IMXP 5cHg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:reply-to:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=tRw/deY2VUJH9uZ2P/hJN+/3kIiv3vn+Rqob7NUnZN4=; b=ofVhVJ/GouKtqbX2NQktNm9J/hOg0CqGitQ4rE/L4qZLj98Ul/+N303hS9ExBFaYE2 QLOMysZvKZsmUoZ3sLwgh8TcIBnKHYd7cI6sUF/cYc7EtF85QrF+bvYGIc/8n2WO9qVB Mc+wG1OjsSsXAWSm/5ytyxsXYRMMZYneS69RelyHFxBLrYeLwDmJJEutmjDxNjVUDJn2 XBIP+1hCQ6GqinhKlR5VnA+Jp2PWkGlgFQQmwZYqQPlUEcnfIxf/TEyNyd268Aj0l/Ae Up3wTgUZpolelQU66ELAnVOtwT+hsjeb8AE4Xd/MtFWxeK1iX3WW4KQRstgef1+eQzwK NVkA== X-Gm-Message-State: AOAM531f8HpRWimlykRD2EViG7JwXYRcs1cJArt/DVpz4YTT0V7Pcnpv YzRk7ggMsoeWF4U0Ro09ewOOcMbbkz8= X-Google-Smtp-Source: ABdhPJwsueKl4GHgV3xPfatFXXxxYt7VznGdNycV8rXaEF320vIs/okAqFaQ8bX7p+ARIye8G2egzOfWTDI= X-Received: from seanjc.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e5]) (user=seanjc job=sendgmr) by 2002:a17:903:11cf:b0:14d:654b:b559 with SMTP id q15-20020a17090311cf00b0014d654bb559mr9854802plh.164.1645834589988; Fri, 25 Feb 2022 16:16:29 -0800 (PST) Reply-To: Sean Christopherson Date: Sat, 26 Feb 2022 00:15:35 +0000 In-Reply-To: <20220226001546.360188-1-seanjc@google.com> Message-Id: <20220226001546.360188-18-seanjc@google.com> Mime-Version: 1.0 References: <20220226001546.360188-1-seanjc@google.com> X-Mailer: git-send-email 2.35.1.574.g5d30c73bfb-goog Subject: [PATCH v3 17/28] KVM: x86/mmu: Zap only TDP MMU leafs in kvm_zap_gfn_range() From: Sean Christopherson To: Paolo Bonzini , Christian Borntraeger , Janosch Frank , Claudio Imbrenda Cc: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , David Hildenbrand , kvm@vger.kernel.org, linux-kernel@vger.kernel.org, David Matlack , Ben Gardon , Mingwei Zhang Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Zap only leaf SPTEs in the TDP MMU's zap_gfn_range(), and rename various functions accordingly. When removing mappings for functional correctness (except for the stupid VFIO GPU passthrough memslots bug), zapping the leaf SPTEs is sufficient as the paging structures themselves do not point at guest memory and do not directly impact the final translation (in the TDP MMU). Note, this aligns the TDP MMU with the legacy/full MMU, which zaps only the rmaps, a.k.a. leaf SPTEs, in kvm_zap_gfn_range() and kvm_unmap_gfn_range(). Signed-off-by: Sean Christopherson Reviewed-by: Ben Gardon --- arch/x86/kvm/mmu/mmu.c | 4 ++-- arch/x86/kvm/mmu/tdp_mmu.c | 41 ++++++++++---------------------------- arch/x86/kvm/mmu/tdp_mmu.h | 8 +------- 3 files changed, 14 insertions(+), 39 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index 1c4b84e80841..92fe1ba27213 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -5775,8 +5775,8 @@ void kvm_zap_gfn_range(struct kvm *kvm, gfn_t gfn_sta= rt, gfn_t gfn_end) =20 if (is_tdp_mmu_enabled(kvm)) { for (i =3D 0; i < KVM_ADDRESS_SPACE_NUM; i++) - flush =3D kvm_tdp_mmu_zap_gfn_range(kvm, i, gfn_start, - gfn_end, flush); + flush =3D kvm_tdp_mmu_zap_leafs(kvm, i, gfn_start, + gfn_end, true, flush); } =20 if (flush) diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c index b55eb7bec308..42803d5dbbf9 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.c +++ b/arch/x86/kvm/mmu/tdp_mmu.c @@ -848,10 +848,8 @@ bool kvm_tdp_mmu_zap_sp(struct kvm *kvm, struct kvm_mm= u_page *sp) } =20 /* - * Tears down the mappings for the range of gfns, [start, end), and frees = the - * non-root pages mapping GFNs strictly within that range. Returns true if - * SPTEs have been cleared and a TLB flush is needed before releasing the - * MMU lock. + * Zap leafs SPTEs for the range of gfns, [start, end). Returns true if SP= TEs + * have been cleared and a TLB flush is needed before releasing the MMU lo= ck. * * If can_yield is true, will release the MMU lock and reschedule if the * scheduler needs the CPU or there is contention on the MMU lock. If this @@ -859,42 +857,25 @@ bool kvm_tdp_mmu_zap_sp(struct kvm *kvm, struct kvm_m= mu_page *sp) * the caller must ensure it does not supply too large a GFN range, or the * operation can cause a soft lockup. */ -static bool zap_gfn_range(struct kvm *kvm, struct kvm_mmu_page *root, - gfn_t start, gfn_t end, bool can_yield, bool flush) +static bool tdp_mmu_zap_leafs(struct kvm *kvm, struct kvm_mmu_page *root, + gfn_t start, gfn_t end, bool can_yield, bool flush) { - bool zap_all =3D (start =3D=3D 0 && end >=3D tdp_mmu_max_gfn_host()); struct tdp_iter iter; =20 - /* - * No need to try to step down in the iterator when zapping all SPTEs, - * zapping the top-level non-leaf SPTEs will recurse on their children. - */ - int min_level =3D zap_all ? root->role.level : PG_LEVEL_4K; - end =3D min(end, tdp_mmu_max_gfn_host()); =20 lockdep_assert_held_write(&kvm->mmu_lock); =20 rcu_read_lock(); =20 - for_each_tdp_pte_min_level(iter, root, min_level, start, end) { + for_each_tdp_pte_min_level(iter, root, PG_LEVEL_4K, start, end) { if (can_yield && tdp_mmu_iter_cond_resched(kvm, &iter, flush, false)) { flush =3D false; continue; } =20 - if (!is_shadow_present_pte(iter.old_spte)) - continue; - - /* - * If this is a non-last-level SPTE that covers a larger range - * than should be zapped, continue, and zap the mappings at a - * lower level, except when zapping all SPTEs. - */ - if (!zap_all && - (iter.gfn < start || - iter.gfn + KVM_PAGES_PER_HPAGE(iter.level) > end) && + if (!is_shadow_present_pte(iter.old_spte) || !is_last_spte(iter.old_spte, iter.level)) continue; =20 @@ -912,13 +893,13 @@ static bool zap_gfn_range(struct kvm *kvm, struct kvm= _mmu_page *root, * SPTEs have been cleared and a TLB flush is needed before releasing the * MMU lock. */ -bool __kvm_tdp_mmu_zap_gfn_range(struct kvm *kvm, int as_id, gfn_t start, - gfn_t end, bool can_yield, bool flush) +bool kvm_tdp_mmu_zap_leafs(struct kvm *kvm, int as_id, gfn_t start, gfn_t = end, + bool can_yield, bool flush) { struct kvm_mmu_page *root; =20 for_each_tdp_mmu_root_yield_safe(kvm, root, as_id, false) - flush =3D zap_gfn_range(kvm, root, start, end, can_yield, flush); + flush =3D tdp_mmu_zap_leafs(kvm, root, start, end, can_yield, false); =20 return flush; } @@ -1179,8 +1160,8 @@ int kvm_tdp_mmu_map(struct kvm_vcpu *vcpu, struct kvm= _page_fault *fault) bool kvm_tdp_mmu_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *ra= nge, bool flush) { - return __kvm_tdp_mmu_zap_gfn_range(kvm, range->slot->as_id, range->start, - range->end, range->may_block, flush); + return kvm_tdp_mmu_zap_leafs(kvm, range->slot->as_id, range->start, + range->end, range->may_block, flush); } =20 typedef bool (*tdp_handler_t)(struct kvm *kvm, struct tdp_iter *iter, diff --git a/arch/x86/kvm/mmu/tdp_mmu.h b/arch/x86/kvm/mmu/tdp_mmu.h index 5e5ef2576c81..54bc8118c40a 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.h +++ b/arch/x86/kvm/mmu/tdp_mmu.h @@ -15,14 +15,8 @@ __must_check static inline bool kvm_tdp_mmu_get_root(str= uct kvm_mmu_page *root) void kvm_tdp_mmu_put_root(struct kvm *kvm, struct kvm_mmu_page *root, bool shared); =20 -bool __kvm_tdp_mmu_zap_gfn_range(struct kvm *kvm, int as_id, gfn_t start, +bool kvm_tdp_mmu_zap_leafs(struct kvm *kvm, int as_id, gfn_t start, gfn_t end, bool can_yield, bool flush); -static inline bool kvm_tdp_mmu_zap_gfn_range(struct kvm *kvm, int as_id, - gfn_t start, gfn_t end, bool flush) -{ - return __kvm_tdp_mmu_zap_gfn_range(kvm, as_id, start, end, true, flush); -} - bool kvm_tdp_mmu_zap_sp(struct kvm *kvm, struct kvm_mmu_page *sp); void kvm_tdp_mmu_zap_all(struct kvm *kvm); void kvm_tdp_mmu_invalidate_all_roots(struct kvm *kvm); --=20 2.35.1.574.g5d30c73bfb-goog From nobody Tue Jun 23 21:22:54 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A4574C433FE for ; Sat, 26 Feb 2022 00:17:54 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234799AbiBZASY (ORCPT ); Fri, 25 Feb 2022 19:18:24 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50826 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240494AbiBZARa (ORCPT ); Fri, 25 Feb 2022 19:17:30 -0500 Received: from mail-pg1-x549.google.com (mail-pg1-x549.google.com [IPv6:2607:f8b0:4864:20::549]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B5F1422A294 for ; Fri, 25 Feb 2022 16:16:32 -0800 (PST) Received: by mail-pg1-x549.google.com with SMTP id t7-20020a634447000000b0036579648983so3492693pgk.3 for ; Fri, 25 Feb 2022 16:16:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=reply-to:date:in-reply-to:message-id:mime-version:references :subject:from:to:cc; bh=YcD14cPLYHRqCUN9XMqsA1N9iI/pU6jOe1s6Rx79mLg=; b=g2WIVIjucKcc3FeIMYEg1JqP8sH7orBfOFiRQ1HLRZYZiypl1HiH1GM7aL3Mn0jfc3 Zfk2B0RPCqztAdRLIp61/bnAGdzjSAuoGODazTgygTETjUDBQLRTAppOIA6z7Zt/nCyf T4JO0+PjzV1XHbQAqzIn/57gBeaUkfrDWwsyxOvG5bvcUcRIfq4oqmHxE/b087WnJgRi vAprnAM5+Gwdv4cPPtbCt72aVuIkMLh/p0OneGBIQL6DCwxl5BWu8bLBjQ278loJa+1e ORLiFR6OU0prTd2CtEi8zG1iwoxdGN3cs/S5KYMLoE4j3sPu2SihnHhoI+rwZZp20oIH VlKA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:reply-to:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=YcD14cPLYHRqCUN9XMqsA1N9iI/pU6jOe1s6Rx79mLg=; b=Twx9ZMvQWnoCOyMe501eOg5PBy2eRd9I1sawkdvPC4raVGa0glPAPS6LCShpRjdzY3 Ea1eRNhIos+4nZ0awrYKD8yiCteUv2eq3VBClrUDV4t2b2xQ8Okw+d4BCwvONeMzu6dp 6bfzg3hAWx7mPrxNpMkO7Y3ay/VztbvFbb/IP7AUBv7GJTjwsoMBhw74u00DaXuTM3zL v3zgx9UxNp5nsOMM6q7TZQd7o8tNC365oC9i0oheKIMpkA6XRfJEHXCroljsxU0bBjbG F+0y4S6uGRRmnQ+J+u82qYFleKAoNfx0GNhFogWfS6q+Wb+gY+u3peYPXPdKPnFMQ4ki WuEg== X-Gm-Message-State: AOAM532PtcqvlMYc8obRLqZGEnWpeeNmMuZ/pNbPjbvqqvGi4CIoO0Q4 UuWha3Bs0UMTrTe/IDzwnUW2YLoBaGE= X-Google-Smtp-Source: ABdhPJyHiBIYgUmO1kHJu6ywpURyknS4uCXFRG4yfpCyMmM+FKVa0coTEtdb7K6OSSSwzo4ko7n3RPKzjxc= X-Received: from seanjc.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e5]) (user=seanjc job=sendgmr) by 2002:aa7:90d5:0:b0:4e1:307c:d94a with SMTP id k21-20020aa790d5000000b004e1307cd94amr10184465pfk.38.1645834591703; Fri, 25 Feb 2022 16:16:31 -0800 (PST) Reply-To: Sean Christopherson Date: Sat, 26 Feb 2022 00:15:36 +0000 In-Reply-To: <20220226001546.360188-1-seanjc@google.com> Message-Id: <20220226001546.360188-19-seanjc@google.com> Mime-Version: 1.0 References: <20220226001546.360188-1-seanjc@google.com> X-Mailer: git-send-email 2.35.1.574.g5d30c73bfb-goog Subject: [PATCH v3 18/28] KVM: x86/mmu: Do remote TLB flush before dropping RCU in TDP MMU resched From: Sean Christopherson To: Paolo Bonzini , Christian Borntraeger , Janosch Frank , Claudio Imbrenda Cc: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , David Hildenbrand , kvm@vger.kernel.org, linux-kernel@vger.kernel.org, David Matlack , Ben Gardon , Mingwei Zhang Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" When yielding in the TDP MMU iterator, service any pending TLB flush before dropping RCU protections in anticipation of using the caller's RCU "lock" as a proxy for vCPUs in the guest. Signed-off-by: Sean Christopherson Reviewed-by: Ben Gardon --- arch/x86/kvm/mmu/tdp_mmu.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c index 42803d5dbbf9..ef594af246f5 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.c +++ b/arch/x86/kvm/mmu/tdp_mmu.c @@ -730,11 +730,11 @@ static inline bool __must_check tdp_mmu_iter_cond_res= ched(struct kvm *kvm, return false; =20 if (need_resched() || rwlock_needbreak(&kvm->mmu_lock)) { - rcu_read_unlock(); - if (flush) kvm_flush_remote_tlbs(kvm); =20 + rcu_read_unlock(); + if (shared) cond_resched_rwlock_read(&kvm->mmu_lock); else --=20 2.35.1.574.g5d30c73bfb-goog From nobody Tue Jun 23 21:22:54 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id F2A2DC433F5 for ; Sat, 26 Feb 2022 00:17:59 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240728AbiBZAS3 (ORCPT ); Fri, 25 Feb 2022 19:18:29 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50476 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240994AbiBZARa (ORCPT ); Fri, 25 Feb 2022 19:17:30 -0500 Received: from mail-pl1-x649.google.com (mail-pl1-x649.google.com [IPv6:2607:f8b0:4864:20::649]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 82AB122B961 for ; Fri, 25 Feb 2022 16:16:34 -0800 (PST) Received: by mail-pl1-x649.google.com with SMTP id j3-20020a170902da8300b0014fdd4e979cso3522064plx.17 for ; Fri, 25 Feb 2022 16:16:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=reply-to:date:in-reply-to:message-id:mime-version:references :subject:from:to:cc; bh=tmkPc6mVqQlJh3WMOsL8fy8nyqdjVjzoO3UnpmYrMsg=; b=C6b9FT0/F6rJ98H5T5jjPa2bRoFq8Lx2Wuvtu/Tm0E8maOHWxUvPUndwuFtwRf66wN U52X1vYZH6X+nfmgIeWR2HVnNAqUmJtr0gk6GJbcR0rPhdlTy8m9DXxczJ0Jd9EMhPvV wBY12F590v8mOrKx4Dou2alNQe2SgbmLz5rQsPWTl0P85AE1B0S737inYfaQmVAiSM7/ dRe/4982nWWybeOn8qiZukkWM3I+/uq+81RZq9O0Ulz3Z22n9m0F6gOQvBINx7DR4k8U 770QhageE44FtYBWUvQ3YL8eFPK+ojjqjhShwy+TLmmcaKLIiKcEzI26Y3Wj516Nw3hi czQw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:reply-to:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=tmkPc6mVqQlJh3WMOsL8fy8nyqdjVjzoO3UnpmYrMsg=; b=sB6D675kbqYy4guuQB4RmEcfuKX4eavcgsblN/nGqKjV9uToR49NzrzLD6ezGK/SF0 b4WdajPR6+P5t3kgFGVC11xZfy9XsaWFMMv4bG4Y1mBScekjWFRqtpEYFrosuXR0QB5y Oy1y0sFchPyco7LbwH3yf7+IAQBgh9Lpw+geFACoaJKx9UT0qLiC4voCUu5c+lVT4Dqn pT0JKvKcITh86euVWPMr9uJ+4jaqlfKd7KFtPPWcS8B9KZbTCuSlw7YWFK79i42iFd4D 0CeHcUCMGKbhydRQR8sgHqzt0IvXsu0cvqjKQqVpGLiwd7RCYmHuZg+WcbcJU/Kz+THy OdxA== X-Gm-Message-State: AOAM531ELUXDnU/yHRPqlftuhiXK4Rl9ejoRVMvCWpgYTlZaOvwpGT39 270zvV+lXJ2iF4/yKC+obiOOC5A1jb8= X-Google-Smtp-Source: ABdhPJzCziF3izciZ/LhfwlHVySkBsqfM2Mexof/5DjEEPPRZwwrv6yEi04D0pIlOxkKRmlhaG66XuwfZFs= X-Received: from seanjc.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e5]) (user=seanjc job=sendgmr) by 2002:a17:902:ea81:b0:14f:a4ce:ef79 with SMTP id x1-20020a170902ea8100b0014fa4ceef79mr9692264plb.136.1645834593398; Fri, 25 Feb 2022 16:16:33 -0800 (PST) Reply-To: Sean Christopherson Date: Sat, 26 Feb 2022 00:15:37 +0000 In-Reply-To: <20220226001546.360188-1-seanjc@google.com> Message-Id: <20220226001546.360188-20-seanjc@google.com> Mime-Version: 1.0 References: <20220226001546.360188-1-seanjc@google.com> X-Mailer: git-send-email 2.35.1.574.g5d30c73bfb-goog Subject: [PATCH v3 19/28] KVM: x86/mmu: Defer TLB flush to caller when freeing TDP MMU shadow pages From: Sean Christopherson To: Paolo Bonzini , Christian Borntraeger , Janosch Frank , Claudio Imbrenda Cc: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , David Hildenbrand , kvm@vger.kernel.org, linux-kernel@vger.kernel.org, David Matlack , Ben Gardon , Mingwei Zhang Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Defer TLB flushes to the caller when freeing TDP MMU shadow pages instead of immediately flushing. Because the shadow pages are freed in an RCU callback, so long as at least one CPU holds RCU, all CPUs are protected. For vCPUs running in the guest, i.e. consuming TLB entries, KVM only needs to ensure the caller services the pending TLB flush before dropping its RCU protections. I.e. use the caller's RCU as a proxy for all vCPUs running in the guest. Deferring the flushes allows batching flushes, e.g. when installing a 1gb hugepage and zapping a pile of SPs. And when zapping an entire root, deferring flushes allows skipping the flush entirely (because flushes are not needed in that case). Avoiding flushes when zapping an entire root is especially important as synchronizing with other CPUs via IPI after zapping every shadow page can cause significant performance issues for large VMs. The issue is exacerbated by KVM zapping entire top-level entries without dropping RCU protection, which can lead to RCU stalls even when zapping roots backing relatively "small" amounts of guest memory, e.g. 2tb. Removing the IPI bottleneck largely mitigates the RCU issues, though it's likely still a problem for 5-level paging. A future patch will further address the problem by zapping roots in multiple passes to avoid holding RCU for an extended duration. Reviewed-by: Ben Gardon Signed-off-by: Sean Christopherson --- arch/x86/kvm/mmu/mmu.c | 12 ++++++++++++ arch/x86/kvm/mmu/tdp_iter.h | 7 +++---- arch/x86/kvm/mmu/tdp_mmu.c | 20 ++++++++++---------- 3 files changed, 25 insertions(+), 14 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index 92fe1ba27213..c1deaec795c2 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -6290,6 +6290,12 @@ static void kvm_recover_nx_lpages(struct kvm *kvm) rcu_idx =3D srcu_read_lock(&kvm->srcu); write_lock(&kvm->mmu_lock); =20 + /* + * Zapping TDP MMU shadow pages, including the remote TLB flush, must + * be done under RCU protection, the pages are freed via RCU callback. + */ + rcu_read_lock(); + ratio =3D READ_ONCE(nx_huge_pages_recovery_ratio); to_zap =3D ratio ? DIV_ROUND_UP(nx_lpage_splits, ratio) : 0; for ( ; to_zap; --to_zap) { @@ -6314,12 +6320,18 @@ static void kvm_recover_nx_lpages(struct kvm *kvm) =20 if (need_resched() || rwlock_needbreak(&kvm->mmu_lock)) { kvm_mmu_remote_flush_or_zap(kvm, &invalid_list, flush); + rcu_read_unlock(); + cond_resched_rwlock_write(&kvm->mmu_lock); flush =3D false; + + rcu_read_lock(); } } kvm_mmu_remote_flush_or_zap(kvm, &invalid_list, flush); =20 + rcu_read_unlock(); + write_unlock(&kvm->mmu_lock); srcu_read_unlock(&kvm->srcu, rcu_idx); } diff --git a/arch/x86/kvm/mmu/tdp_iter.h b/arch/x86/kvm/mmu/tdp_iter.h index e2a7e267a77d..b1eaf6ec0e0b 100644 --- a/arch/x86/kvm/mmu/tdp_iter.h +++ b/arch/x86/kvm/mmu/tdp_iter.h @@ -9,10 +9,9 @@ =20 /* * TDP MMU SPTEs are RCU protected to allow paging structures (non-leaf SP= TEs) - * to be zapped while holding mmu_lock for read. Holding RCU isn't requir= ed for - * correctness if mmu_lock is held for write, but plumbing "struct kvm" do= wn to - * the lower depths of the TDP MMU just to make lockdep happy is a nightma= re, so - * all accesses to SPTEs are done under RCU protection. + * to be zapped while holding mmu_lock for read, and to allow TLB flushes = to be + * batched without having to collect the list of zapped SPs. Flows that c= an + * remove SPs must service pending TLB flushes prior to dropping RCU prote= ction. */ static inline u64 kvm_tdp_mmu_read_spte(tdp_ptep_t sptep) { diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c index ef594af246f5..3031b42c27a6 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.c +++ b/arch/x86/kvm/mmu/tdp_mmu.c @@ -405,9 +405,6 @@ static void handle_removed_pt(struct kvm *kvm, tdp_ptep= _t pt, bool shared) shared); } =20 - kvm_flush_remote_tlbs_with_address(kvm, base_gfn, - KVM_PAGES_PER_HPAGE(level + 1)); - call_rcu(&sp->rcu_head, tdp_mmu_free_sp_rcu_callback); } =20 @@ -831,19 +828,13 @@ bool kvm_tdp_mmu_zap_sp(struct kvm *kvm, struct kvm_m= mu_page *sp) if (WARN_ON_ONCE(!sp->ptep)) return false; =20 - rcu_read_lock(); - old_spte =3D kvm_tdp_mmu_read_spte(sp->ptep); - if (WARN_ON_ONCE(!is_shadow_present_pte(old_spte))) { - rcu_read_unlock(); + if (WARN_ON_ONCE(!is_shadow_present_pte(old_spte))) return false; - } =20 __tdp_mmu_set_spte(kvm, kvm_mmu_page_as_id(sp), sp->ptep, old_spte, 0, sp->gfn, sp->role.level + 1, true, true); =20 - rcu_read_unlock(); - return true; } =20 @@ -884,6 +875,11 @@ static bool tdp_mmu_zap_leafs(struct kvm *kvm, struct = kvm_mmu_page *root, } =20 rcu_read_unlock(); + + /* + * Because this flow zaps _only_ leaf SPTEs, the caller doesn't need + * to provide RCU protection as no 'struct kvm_mmu_page' will be freed. + */ return flush; } =20 @@ -1013,6 +1009,10 @@ static int tdp_mmu_map_handle_target_level(struct kv= m_vcpu *vcpu, ret =3D RET_PF_SPURIOUS; else if (tdp_mmu_set_spte_atomic(vcpu->kvm, iter, new_spte)) return RET_PF_RETRY; + else if (is_shadow_present_pte(iter->old_spte) && + !is_last_spte(iter->old_spte, iter->level)) + kvm_flush_remote_tlbs_with_address(vcpu->kvm, sp->gfn, + KVM_PAGES_PER_HPAGE(iter->level + 1)); =20 /* * If the page fault was caused by a write but the page is write --=20 2.35.1.574.g5d30c73bfb-goog From nobody Tue Jun 23 21:22:54 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E041BC433F5 for ; Sat, 26 Feb 2022 00:19:24 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240373AbiBZATy (ORCPT ); Fri, 25 Feb 2022 19:19:54 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51208 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240652AbiBZARs (ORCPT ); Fri, 25 Feb 2022 19:17:48 -0500 Received: from mail-pl1-x64a.google.com (mail-pl1-x64a.google.com [IPv6:2607:f8b0:4864:20::64a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 933FE22BE92 for ; Fri, 25 Feb 2022 16:16:42 -0800 (PST) Received: by mail-pl1-x64a.google.com with SMTP id e13-20020a17090301cd00b00150145346f9so3345070plh.23 for ; Fri, 25 Feb 2022 16:16:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=reply-to:date:in-reply-to:message-id:mime-version:references :subject:from:to:cc; bh=MtFjInkw6IfHM8qigYcwgTznLrkAOMo13iKHQwnE+RY=; b=UHd0RZGwQM9tLWquYv40cKi4O+zpI8kl9GYujD7+NRNRf/rOM3orWVXz6hbm9DPwqn G2yVyOwRj8r0I91Balg/eUXT4Vssf1yySR+LnqtVeDfsOK0TqyPJCjVD1j7UZIrzS+H2 XgafyuCMW1zFpaDC/BuznK+9/sIINwetRnZ6GDAARzpqzz1Wmev0MYlL/J7X1SGHQ9gq fWeUYpGUxw7peSaIWNxspqeegQQGcW/VpMyL37FCggG6jWhj013Elqk9ugfloYEND583 mFH+M+fy7wvJNgIP3EYqRpLSf3NGs7L6UDxtkyGjqKhdHEu0xqBoQqN6c8SmQU0sBMaR s6Sg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:reply-to:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=MtFjInkw6IfHM8qigYcwgTznLrkAOMo13iKHQwnE+RY=; b=Ve7AoIzs0gJ/965QdoKIg3UG9/O+kHxZy5DjwNpcAHMizMXCvDRhIlKIZZTM0I8kNO ZUK/w3ImwdJxiBJfO/kGPQmtugL5y7w0t8tkeseybmC+6j/JWmlrVfgDyccLw5yV5wUe bXkJMolROTwRcSnxBuH+iHCg+mE9Z6cLVMHmK9q75G8B0rZik4FJ2tLGu2uUQ3HDmClc D8XzpykKLxaPyK5uf0zWgeNGbRW4ejcpIupTmvhRHCHo4+lD1e4RzaDd97v48oa6yLUR OXOO2lK/UlKfqFFBR1fP0VpoKLOOVVW5vKG9STe9ZZfXxPR6wsGxNdZwS+dCIUh6GvMh WmcQ== X-Gm-Message-State: AOAM5334jiTSozfFt8gIPQ3EmUHrf+BaWsWR+l0jJJBt90QbfVUye8mb kOVYaS3SYQbHqbb8SAZJWyp39XGP89I= X-Google-Smtp-Source: ABdhPJyBaElSz5P3eL5Msmfyo8Q6hiVLJEgTjKZw7rh4UamsOV8nO7GCL/BZGus/ptWCadyPC/4cDm/RtOs= X-Received: from seanjc.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e5]) (user=seanjc job=sendgmr) by 2002:a05:6a00:244f:b0:4df:82ad:447 with SMTP id d15-20020a056a00244f00b004df82ad0447mr10149467pfj.49.1645834595122; Fri, 25 Feb 2022 16:16:35 -0800 (PST) Reply-To: Sean Christopherson Date: Sat, 26 Feb 2022 00:15:38 +0000 In-Reply-To: <20220226001546.360188-1-seanjc@google.com> Message-Id: <20220226001546.360188-21-seanjc@google.com> Mime-Version: 1.0 References: <20220226001546.360188-1-seanjc@google.com> X-Mailer: git-send-email 2.35.1.574.g5d30c73bfb-goog Subject: [PATCH v3 20/28] KVM: x86/mmu: Allow yielding when zapping GFNs for defunct TDP MMU root From: Sean Christopherson To: Paolo Bonzini , Christian Borntraeger , Janosch Frank , Claudio Imbrenda Cc: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , David Hildenbrand , kvm@vger.kernel.org, linux-kernel@vger.kernel.org, David Matlack , Ben Gardon , Mingwei Zhang Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Allow yielding when zapping SPTEs after the last reference to a valid root is put. Because KVM must drop all SPTEs in response to relevant mmu_notifier events, mark defunct roots invalid and reset their refcount prior to zapping the root. Keeping the refcount elevated while the zap is in-progress ensures the root is reachable via mmu_notifier until the zap completes and the last reference to the invalid, defunct root is put. Allowing kvm_tdp_mmu_put_root() to yield fixes soft lockup issues if the root in being put has a massive paging structure, e.g. zapping a root that is backed entirely by 4kb pages for a guest with 32tb of memory can take hundreds of seconds to complete. watchdog: BUG: soft lockup - CPU#49 stuck for 485s! [max_guest_memor:5236= 8] RIP: 0010:kvm_set_pfn_dirty+0x30/0x50 [kvm] __handle_changed_spte+0x1b2/0x2f0 [kvm] handle_removed_tdp_mmu_page+0x1a7/0x2b8 [kvm] __handle_changed_spte+0x1f4/0x2f0 [kvm] handle_removed_tdp_mmu_page+0x1a7/0x2b8 [kvm] __handle_changed_spte+0x1f4/0x2f0 [kvm] tdp_mmu_zap_root+0x307/0x4d0 [kvm] kvm_tdp_mmu_put_root+0x7c/0xc0 [kvm] kvm_mmu_free_roots+0x22d/0x350 [kvm] kvm_mmu_reset_context+0x20/0x60 [kvm] kvm_arch_vcpu_ioctl_set_sregs+0x5a/0xc0 [kvm] kvm_vcpu_ioctl+0x5bd/0x710 [kvm] __se_sys_ioctl+0x77/0xc0 __x64_sys_ioctl+0x1d/0x20 do_syscall_64+0x44/0xa0 entry_SYSCALL_64_after_hwframe+0x44/0xae KVM currently doesn't put a root from a non-preemptible context, so other than the mmu_notifier wrinkle, yielding when putting a root is safe. Yield-unfriendly iteration uses for_each_tdp_mmu_root(), which doesn't take a reference to each root (it requires mmu_lock be held for the entire duration of the walk). tdp_mmu_next_root() is used only by the yield-friendly iterator. kvm_tdp_mmu_zap_invalidated_roots() is explicitly yield friendly. kvm_mmu_free_roots() =3D> mmu_free_root_page() is a much bigger fan-out, but is still yield-friendly in all call sites, as all callers can be traced back to some combination of vcpu_run(), kvm_destroy_vm(), and/or kvm_create_vm(). Signed-off-by: Sean Christopherson --- arch/x86/kvm/mmu/tdp_mmu.c | 122 ++++++++++++++++++++++++------------- 1 file changed, 81 insertions(+), 41 deletions(-) diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c index 3031b42c27a6..b838cfa984ad 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.c +++ b/arch/x86/kvm/mmu/tdp_mmu.c @@ -91,21 +91,66 @@ void kvm_tdp_mmu_put_root(struct kvm *kvm, struct kvm_m= mu_page *root, =20 WARN_ON(!root->tdp_mmu_page); =20 - spin_lock(&kvm->arch.tdp_mmu_pages_lock); - list_del_rcu(&root->link); - spin_unlock(&kvm->arch.tdp_mmu_pages_lock); + /* + * Ensure root->role.invalid is read after the refcount reaches zero to + * avoid zapping the root multiple times, e.g. if a different task + * acquires a reference (after the root was marked invalid) and puts + * the last reference, all while holding mmu_lock for read. Pairs + * with the smp_mb__before_atomic() below. + */ + smp_mb__after_atomic(); + + /* + * Free the root if it's already invalid. Invalid roots must be zapped + * before their last reference is put, i.e. there's no work to be done, + * and all roots must be invalidated (see below) before they're freed. + * Re-zapping invalid roots would put KVM into an infinite loop (again, + * see below). + */ + if (root->role.invalid) { + spin_lock(&kvm->arch.tdp_mmu_pages_lock); + list_del_rcu(&root->link); + spin_unlock(&kvm->arch.tdp_mmu_pages_lock); + + call_rcu(&root->rcu_head, tdp_mmu_free_sp_rcu_callback); + return; + } + + /* + * Invalidate the root to prevent it from being reused by a vCPU, and + * so that KVM doesn't re-zap the root when its last reference is put + * again (see above). + */ + root->role.invalid =3D true; + + /* + * Ensure role.invalid is visible if a concurrent reader acquires a + * reference after the root's refcount is reset. Pairs with the + * smp_mb__after_atomic() above. + */ + smp_mb__before_atomic(); + + /* + * Note, if mmu_lock is held for read this can race with other readers, + * e.g. they may acquire a reference without seeing the root as invalid, + * and the refcount may be reset after the root is skipped. Both races + * are benign, as flows that must visit all roots, e.g. need to zap + * SPTEs for correctness, must take mmu_lock for write to block page + * faults, and the only flow that must not consume an invalid root is + * allocating a new root for a vCPU, which also takes mmu_lock for write. + */ + refcount_set(&root->tdp_mmu_root_count, 1); =20 /* - * A TLB flush is not necessary as KVM performs a local TLB flush when - * allocating a new root (see kvm_mmu_load()), and when migrating vCPU - * to a different pCPU. Note, the local TLB flush on reuse also - * invalidates any paging-structure-cache entries, i.e. TLB entries for - * intermediate paging structures, that may be zapped, as such entries - * are associated with the ASID on both VMX and SVM. + * Zap the root, then put the refcount "acquired" above. Recursively + * call kvm_tdp_mmu_put_root() to test the above logic for avoiding an + * infinite loop by freeing invalid roots. By design, the root is + * reachable while it's being zapped, thus a different task can put its + * last reference, i.e. flowing through kvm_tdp_mmu_put_root() for a + * defunct root is unavoidable. */ tdp_mmu_zap_root(kvm, root, shared); - - call_rcu(&root->rcu_head, tdp_mmu_free_sp_rcu_callback); + kvm_tdp_mmu_put_root(kvm, root, shared); } =20 enum tdp_mmu_roots_iter_type { @@ -760,12 +805,23 @@ static inline gfn_t tdp_mmu_max_gfn_host(void) static void tdp_mmu_zap_root(struct kvm *kvm, struct kvm_mmu_page *root, bool shared) { - bool root_is_unreachable =3D !refcount_read(&root->tdp_mmu_root_count); struct tdp_iter iter; =20 gfn_t end =3D tdp_mmu_max_gfn_host(); gfn_t start =3D 0; =20 + /* + * The root must have an elevated refcount so that it's reachable via + * mmu_notifier callbacks, which allows this path to yield and drop + * mmu_lock. When handling an unmap/release mmu_notifier command, KVM + * must drop all references to relevant pages prior to completing the + * callback. Dropping mmu_lock with an unreachable root would result + * in zapping SPTEs after a relevant mmu_notifier callback completes + * and lead to use-after-free as zapping a SPTE triggers "writeback" of + * dirty accessed bits to the SPTE's associated struct page. + */ + WARN_ON_ONCE(!refcount_read(&root->tdp_mmu_root_count)); + kvm_lockdep_assert_mmu_lock_held(kvm, shared); =20 rcu_read_lock(); @@ -776,42 +832,16 @@ static void tdp_mmu_zap_root(struct kvm *kvm, struct = kvm_mmu_page *root, */ for_each_tdp_pte_min_level(iter, root, root->role.level, start, end) { retry: - /* - * Yielding isn't allowed when zapping an unreachable root as - * the root won't be processed by mmu_notifier callbacks. When - * handling an unmap/release mmu_notifier command, KVM must - * drop all references to relevant pages prior to completing - * the callback. Dropping mmu_lock can result in zapping SPTEs - * for an unreachable root after a relevant callback completes, - * which leads to use-after-free as zapping a SPTE triggers - * "writeback" of dirty/accessed bits to the SPTE's associated - * struct page. - */ - if (!root_is_unreachable && - tdp_mmu_iter_cond_resched(kvm, &iter, false, shared)) + if (tdp_mmu_iter_cond_resched(kvm, &iter, false, shared)) continue; =20 if (!is_shadow_present_pte(iter.old_spte)) continue; =20 - if (!shared) { + if (!shared) tdp_mmu_set_spte(kvm, &iter, 0); - } else if (tdp_mmu_set_spte_atomic(kvm, &iter, 0)) { - /* - * cmpxchg() shouldn't fail if the root is unreachable. - * Retry so as not to leak the page and its children. - */ - WARN_ONCE(root_is_unreachable, - "Contended TDP MMU SPTE in unreachable root."); + else if (tdp_mmu_set_spte_atomic(kvm, &iter, 0)) goto retry; - } - - /* - * WARN if the root is invalid and is unreachable, all SPTEs - * should've been zapped by kvm_tdp_mmu_zap_invalidated_roots(), - * and inserting new SPTEs under an invalid root is a KVM bug. - */ - WARN_ON_ONCE(root_is_unreachable && root->role.invalid); } =20 rcu_read_unlock(); @@ -906,6 +936,9 @@ void kvm_tdp_mmu_zap_all(struct kvm *kvm) int i; =20 /* + * Zap all roots, including invalid roots, as all SPTEs must be dropped + * before returning to the caller. + * * A TLB flush is unnecessary, KVM zaps everything if and only the VM * is being destroyed or the userspace VMM has exited. In both cases, * KVM_RUN is unreachable, i.e. no vCPUs will ever service the request. @@ -931,6 +964,13 @@ void kvm_tdp_mmu_zap_invalidated_roots(struct kvm *kvm) =20 for_each_invalid_tdp_mmu_root_yield_safe(kvm, root) { /* + * Zap the root regardless of what marked it invalid, e.g. even + * if the root was marked invalid by kvm_tdp_mmu_put_root() due + * to its last reference being put. All SPTEs must be dropped + * before returning to the caller, e.g. if a memslot is deleted + * or moved, the memslot's associated SPTEs are unreachable via + * the mmu_notifier once the memslot update completes. + * * A TLB flush is unnecessary, invalidated roots are guaranteed * to be unreachable by the guest (see kvm_tdp_mmu_put_root() * for more details), and unlike the legacy MMU, no vCPU kick --=20 2.35.1.574.g5d30c73bfb-goog From nobody Tue Jun 23 21:22:54 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C694AC433EF for ; Sat, 26 Feb 2022 00:19:29 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240742AbiBZAT7 (ORCPT ); Fri, 25 Feb 2022 19:19:59 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50808 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S241106AbiBZARu (ORCPT ); Fri, 25 Feb 2022 19:17:50 -0500 Received: from mail-yb1-xb4a.google.com (mail-yb1-xb4a.google.com [IPv6:2607:f8b0:4864:20::b4a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 53C3722D650 for ; Fri, 25 Feb 2022 16:16:44 -0800 (PST) Received: by mail-yb1-xb4a.google.com with SMTP id w1-20020a05690204e100b006244315a721so5032851ybs.0 for ; Fri, 25 Feb 2022 16:16:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=reply-to:date:in-reply-to:message-id:mime-version:references :subject:from:to:cc; bh=8vGylyelB/7s2Iu/cezFfDz4H+G0SXAJDoFtwhKJJ28=; b=dlXNILOL0RcgYaj/SOWVb/B+cFvaL+/RGu/bd2ecjAYaKJn1zHuPEJbHsfwU502SK2 +FPPiaiOVMxsUIrbOJ+sFuFZQplYzNHew/+TexiqluBvFAtJm5BOdpx6Dyf6PLamreU+ hNU6CzIJO6mRakhEXLnBbgxZvH297CHWiYPQxLFgrjRHVwdn3KMGXQbJE9fY5v2rlaPS xCQ3NSaXqx4Lj0B3QRVeXDEfNXLnUlDVvWk7kzFqOVIAbjhdCXNeMCJrNwKFT6My9RFG v+b9azLwXSHba017DB8cTjbwtzh12saWyoLZoBJXmekNmr6CKoNAi8Z/xXgdovpR015D alzw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:reply-to:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=8vGylyelB/7s2Iu/cezFfDz4H+G0SXAJDoFtwhKJJ28=; b=CD9h5TVM0cKR0hHWYomCIdxwFFE/L0oVBwayPb8MSbDrtnz1PJQMgjU2iMBE04IYrE KETdzL0Z4fKRAokSoJ/0r08IrsaU2WvYtOLOtmVOBhXocJPfrSEliYZPfb88Tfzuc0Rg VQbiFKammr+SErK1jXmLm4QMVUqVIugDUVyaxicUuU4tM1DvB1SYx6XTTgKiWurANad6 DxYkm9xC92ojVgYRF3tcVFC2Dcgh4gz3Mevp2Bv25lP+dWMNkob6HrGBtJANJQQa3HNe bYBlj2gz7wKgnMPJ0QKUvDY8vOl94cb/3vEoCy/63/IGVToW8epn1IDud/K5Q1LfWXa8 WJYQ== X-Gm-Message-State: AOAM531vvRroqS8BJPPhD3ut4ptFF6JD1KMPFoYnFvuw39ypvFMk4v28 vQ55/zIsGJ7ZUP1Yvq1ici4WNq7zzBY= X-Google-Smtp-Source: ABdhPJwwopT5Z1DLLtP8f3p7Bz63Hhro6EnAkKtt/I4Q68E+irs1imu/xJTXCCS7diEoreI04+8r4bs7KM4= X-Received: from seanjc.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e5]) (user=seanjc job=sendgmr) by 2002:a25:e014:0:b0:619:a368:c3b5 with SMTP id x20-20020a25e014000000b00619a368c3b5mr9542866ybg.383.1645834596968; Fri, 25 Feb 2022 16:16:36 -0800 (PST) Reply-To: Sean Christopherson Date: Sat, 26 Feb 2022 00:15:39 +0000 In-Reply-To: <20220226001546.360188-1-seanjc@google.com> Message-Id: <20220226001546.360188-22-seanjc@google.com> Mime-Version: 1.0 References: <20220226001546.360188-1-seanjc@google.com> X-Mailer: git-send-email 2.35.1.574.g5d30c73bfb-goog Subject: [PATCH v3 21/28] KVM: x86/mmu: Zap roots in two passes to avoid inducing RCU stalls From: Sean Christopherson To: Paolo Bonzini , Christian Borntraeger , Janosch Frank , Claudio Imbrenda Cc: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , David Hildenbrand , kvm@vger.kernel.org, linux-kernel@vger.kernel.org, David Matlack , Ben Gardon , Mingwei Zhang Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" When zapping a TDP MMU root, perform the zap in two passes to avoid zapping an entire top-level SPTE while holding RCU, which can induce RCU stalls. In the first pass, zap SPTEs at PG_LEVEL_1G, and then zap top-level entries in the second pass. With 4-level paging, zapping a PGD that is fully populated with 4kb leaf SPTEs take up to ~7 or so seconds (time varies based on kernel config, number of (v)CPUs, etc...). With 5-level paging, that time can balloon well into hundreds of seconds. Before remote TLB flushes were omitted, the problem was even worse as waiting for all active vCPUs to respond to the IPI introduced significant overhead for VMs with large numbers of vCPUs. By zapping 1gb SPTEs (both shadow pages and hugepages) in the first pass, the amount of work that is done without dropping RCU protection is strictly bounded, with the worst case latency for a single operation being less than 100ms. Zapping at 1gb in the first pass is not arbitrary. First and foremost, KVM relies on being able to zap 1gb shadow pages in a single shot when when repacing a shadow page with a hugepage. Zapping a 1gb shadow page that is fully populated with 4kb dirty SPTEs also triggers the worst case latency due writing back the struct page accessed/dirty bits for each 4kb page, i.e. the two-pass approach is guaranteed to work so long as KVM can cleany zap a 1gb shadow page. rcu: INFO: rcu_sched self-detected stall on CPU rcu: 52-....: (20999 ticks this GP) idle=3D7be/1/0x4000000000000000 softirq=3D15759/15759 fqs=3D5058 (t=3D21016 jiffies g=3D66453 q=3D238577) NMI backtrace for cpu 52 Call Trace: ... mark_page_accessed+0x266/0x2f0 kvm_set_pfn_accessed+0x31/0x40 handle_removed_tdp_mmu_page+0x259/0x2e0 __handle_changed_spte+0x223/0x2c0 handle_removed_tdp_mmu_page+0x1c1/0x2e0 __handle_changed_spte+0x223/0x2c0 handle_removed_tdp_mmu_page+0x1c1/0x2e0 __handle_changed_spte+0x223/0x2c0 zap_gfn_range+0x141/0x3b0 kvm_tdp_mmu_zap_invalidated_roots+0xc8/0x130 kvm_mmu_zap_all_fast+0x121/0x190 kvm_mmu_invalidate_zap_pages_in_memslot+0xe/0x10 kvm_page_track_flush_slot+0x5c/0x80 kvm_arch_flush_shadow_memslot+0xe/0x10 kvm_set_memslot+0x172/0x4e0 __kvm_set_memory_region+0x337/0x590 kvm_vm_ioctl+0x49c/0xf80 Reported-by: David Matlack Cc: Ben Gardon Cc: Mingwei Zhang Signed-off-by: Sean Christopherson Reviewed-by: Ben Gardon --- arch/x86/kvm/mmu/tdp_mmu.c | 51 +++++++++++++++++++++++++------------- 1 file changed, 34 insertions(+), 17 deletions(-) diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c index b838cfa984ad..ec28a88c6376 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.c +++ b/arch/x86/kvm/mmu/tdp_mmu.c @@ -802,14 +802,36 @@ static inline gfn_t tdp_mmu_max_gfn_host(void) return 1ULL << (shadow_phys_bits - PAGE_SHIFT); } =20 -static void tdp_mmu_zap_root(struct kvm *kvm, struct kvm_mmu_page *root, - bool shared) +static void __tdp_mmu_zap_root(struct kvm *kvm, struct kvm_mmu_page *root, + bool shared, int zap_level) { struct tdp_iter iter; =20 gfn_t end =3D tdp_mmu_max_gfn_host(); gfn_t start =3D 0; =20 + for_each_tdp_pte_min_level(iter, root, zap_level, start, end) { +retry: + if (tdp_mmu_iter_cond_resched(kvm, &iter, false, shared)) + continue; + + if (!is_shadow_present_pte(iter.old_spte)) + continue; + + if (iter.level > zap_level) + continue; + + if (!shared) + tdp_mmu_set_spte(kvm, &iter, 0); + else if (tdp_mmu_set_spte_atomic(kvm, &iter, 0)) + goto retry; + } +} + +static void tdp_mmu_zap_root(struct kvm *kvm, struct kvm_mmu_page *root, + bool shared) +{ + /* * The root must have an elevated refcount so that it's reachable via * mmu_notifier callbacks, which allows this path to yield and drop @@ -827,22 +849,17 @@ static void tdp_mmu_zap_root(struct kvm *kvm, struct = kvm_mmu_page *root, rcu_read_lock(); =20 /* - * No need to try to step down in the iterator when zapping an entire - * root, zapping an upper-level SPTE will recurse on its children. + * To avoid RCU stalls due to recursively removing huge swaths of SPs, + * split the zap into two passes. On the first pass, zap at the 1gb + * level, and then zap top-level SPs on the second pass. "1gb" is not + * arbitrary, as KVM must be able to zap a 1gb shadow page without + * inducing a stall to allow in-place replacement with a 1gb hugepage. + * + * Because zapping a SP recurses on its children, stepping down to + * PG_LEVEL_4K in the iterator itself is unnecessary. */ - for_each_tdp_pte_min_level(iter, root, root->role.level, start, end) { -retry: - if (tdp_mmu_iter_cond_resched(kvm, &iter, false, shared)) - continue; - - if (!is_shadow_present_pte(iter.old_spte)) - continue; - - if (!shared) - tdp_mmu_set_spte(kvm, &iter, 0); - else if (tdp_mmu_set_spte_atomic(kvm, &iter, 0)) - goto retry; - } + __tdp_mmu_zap_root(kvm, root, shared, PG_LEVEL_1G); + __tdp_mmu_zap_root(kvm, root, shared, root->role.level); =20 rcu_read_unlock(); } --=20 2.35.1.574.g5d30c73bfb-goog From nobody Tue Jun 23 21:22:54 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 06A11C433F5 for ; Sat, 26 Feb 2022 00:18:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239046AbiBZATI (ORCPT ); Fri, 25 Feb 2022 19:19:08 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50392 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S241262AbiBZARw (ORCPT ); Fri, 25 Feb 2022 19:17:52 -0500 Received: from mail-pj1-x1049.google.com (mail-pj1-x1049.google.com [IPv6:2607:f8b0:4864:20::1049]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EC68022D672 for ; Fri, 25 Feb 2022 16:16:47 -0800 (PST) Received: by mail-pj1-x1049.google.com with SMTP id t7-20020a17090a3b4700b001bc366c58faso6469966pjf.4 for ; Fri, 25 Feb 2022 16:16:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=reply-to:date:in-reply-to:message-id:mime-version:references :subject:from:to:cc; bh=MilK8zUZX7tWJl2GgjJY8oNeKvoyoiy4aBCti+o2Dno=; b=azOeWuW+lWy8w8HIL49LW9KhszQbW+YNWD8IHPgIc79FX9F9n3TXODYKi2HAmjESdo HOXPis4YQybgWPgSa31snGdX4dlIAVkB9Yva2qwEEiqK/z6DI1r5iw8PVKGyV8sZFp26 NKY9hJKEKL7NHbwOIm4FOe+6io3nDecTa17lwWjUGSyRR08BEvTKr9V1c2u071oaOFZD bC+F2p0SAaU18Xfrju7A1ThpQzKfyc7V0pHszBilaXsXz+TjfNJ1ZPCi7/KX24B4qLX4 v6o/Zqfmx6jE5Ve7LSoUfC4V2CUgRoY05MqIfyWwuNZzJn1fAx39O7gmmUpXmE+sUR+h 8GNA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:reply-to:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=MilK8zUZX7tWJl2GgjJY8oNeKvoyoiy4aBCti+o2Dno=; b=Xvies2fbQp5tAR+H8oUIg4sXGLwgcGYws7FWmu6b2+HhbnfQQa2JJ+ZQqdwidvzeTR g+/FhQwBs51docH/G99/SbOq0iSgW8HFMd0y3yQEiOB0AZDGqdrVCLx5MIb5/dj+Cbv6 NayV7b5sdzOLgx25MCvqSYjbazuoZp63/+OtSBKcAJMGUnUXUIBdzqLr8JLUubtm9zL4 rQ6y2BGX9SwBwOkHiSzMSy0lvjitZisQUGbQ1pz6Rpqk1YhObx8LUt6sqXxR34uNVXlR zYezMGjVMibdIeqAX9Srngi10uGd3kLFNEDG7S5/E3PH787Dcqj0nzowPw49XZ1FKqkh f5sA== X-Gm-Message-State: AOAM53078Mh/BijqyLL5S3Ie0PoR1lrnjASibyMa91FROMhRzGsH8QHB NDKIlhJNFowIT8xubzPl35fipeFsVDI= X-Google-Smtp-Source: ABdhPJwQC0seS+f5AjDyQB4suBW4N+vPljL91EEY6uoAI/tRdabfBdxDkAHZXw+1PFBpgG1HwP5YTKG451I= X-Received: from seanjc.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e5]) (user=seanjc job=sendgmr) by 2002:a65:5bcc:0:b0:378:4b73:4fe9 with SMTP id o12-20020a655bcc000000b003784b734fe9mr2939194pgr.533.1645834598449; Fri, 25 Feb 2022 16:16:38 -0800 (PST) Reply-To: Sean Christopherson Date: Sat, 26 Feb 2022 00:15:40 +0000 In-Reply-To: <20220226001546.360188-1-seanjc@google.com> Message-Id: <20220226001546.360188-23-seanjc@google.com> Mime-Version: 1.0 References: <20220226001546.360188-1-seanjc@google.com> X-Mailer: git-send-email 2.35.1.574.g5d30c73bfb-goog Subject: [PATCH v3 22/28] KVM: x86/mmu: Zap defunct roots via asynchronous worker From: Sean Christopherson To: Paolo Bonzini , Christian Borntraeger , Janosch Frank , Claudio Imbrenda Cc: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , David Hildenbrand , kvm@vger.kernel.org, linux-kernel@vger.kernel.org, David Matlack , Ben Gardon , Mingwei Zhang Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Zap defunct roots, a.k.a. roots that have been invalidated after their last reference was initially dropped, asynchronously via the system work queue instead of forcing the work upon the unfortunate task that happened to drop the last reference. If a vCPU task drops the last reference, the vCPU is effectively blocked by the host for the entire duration of the zap. If the root being zapped happens be fully populated with 4kb leaf SPTEs, e.g. due to dirty logging being active, the zap can take several hundred seconds. Unsurprisingly, most guests are unhappy if a vCPU disappears for hundreds of seconds. E.g. running a synthetic selftest that triggers a vCPU root zap with ~64tb of guest memory and 4kb SPTEs blocks the vCPU for 900+ seconds. Offloading the zap to a worker drops the block time to <100ms. Signed-off-by: Sean Christopherson Reviewed-by: Ben Gardon --- arch/x86/kvm/mmu/mmu_internal.h | 8 +++- arch/x86/kvm/mmu/tdp_mmu.c | 65 ++++++++++++++++++++++++++++----- 2 files changed, 63 insertions(+), 10 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu_internal.h b/arch/x86/kvm/mmu/mmu_interna= l.h index be063b6c91b7..1bff453f7cbe 100644 --- a/arch/x86/kvm/mmu/mmu_internal.h +++ b/arch/x86/kvm/mmu/mmu_internal.h @@ -65,7 +65,13 @@ struct kvm_mmu_page { struct kvm_rmap_head parent_ptes; /* rmap pointers to parent sptes */ tdp_ptep_t ptep; }; - DECLARE_BITMAP(unsync_child_bitmap, 512); + union { + DECLARE_BITMAP(unsync_child_bitmap, 512); + struct { + struct work_struct tdp_mmu_async_work; + void *tdp_mmu_async_data; + }; + }; =20 struct list_head lpage_disallowed_link; #ifdef CONFIG_X86_32 diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c index ec28a88c6376..4151e61245a7 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.c +++ b/arch/x86/kvm/mmu/tdp_mmu.c @@ -81,6 +81,38 @@ static void tdp_mmu_free_sp_rcu_callback(struct rcu_head= *head) static void tdp_mmu_zap_root(struct kvm *kvm, struct kvm_mmu_page *root, bool shared); =20 +static void tdp_mmu_zap_root_async(struct work_struct *work) +{ + struct kvm_mmu_page *root =3D container_of(work, struct kvm_mmu_page, + tdp_mmu_async_work); + struct kvm *kvm =3D root->tdp_mmu_async_data; + + read_lock(&kvm->mmu_lock); + + /* + * A TLB flush is not necessary as KVM performs a local TLB flush when + * allocating a new root (see kvm_mmu_load()), and when migrating vCPU + * to a different pCPU. Note, the local TLB flush on reuse also + * invalidates any paging-structure-cache entries, i.e. TLB entries for + * intermediate paging structures, that may be zapped, as such entries + * are associated with the ASID on both VMX and SVM. + */ + tdp_mmu_zap_root(kvm, root, true); + + /* + * Drop the refcount using kvm_tdp_mmu_put_root() to test its logic for + * avoiding an infinite loop. By design, the root is reachable while + * it's being asynchronously zapped, thus a different task can put its + * last reference, i.e. flowing through kvm_tdp_mmu_put_root() for an + * asynchronously zapped root is unavoidable. + */ + kvm_tdp_mmu_put_root(kvm, root, true); + + read_unlock(&kvm->mmu_lock); + + kvm_put_kvm(kvm); +} + void kvm_tdp_mmu_put_root(struct kvm *kvm, struct kvm_mmu_page *root, bool shared) { @@ -142,15 +174,26 @@ void kvm_tdp_mmu_put_root(struct kvm *kvm, struct kvm= _mmu_page *root, refcount_set(&root->tdp_mmu_root_count, 1); =20 /* - * Zap the root, then put the refcount "acquired" above. Recursively - * call kvm_tdp_mmu_put_root() to test the above logic for avoiding an - * infinite loop by freeing invalid roots. By design, the root is - * reachable while it's being zapped, thus a different task can put its - * last reference, i.e. flowing through kvm_tdp_mmu_put_root() for a - * defunct root is unavoidable. + * Attempt to acquire a reference to KVM itself. If KVM is alive, then + * zap the root asynchronously in a worker, otherwise it must be zapped + * directly here. Wait to do this check until after the refcount is + * reset so that tdp_mmu_zap_root() can safely yield. + * + * In both flows, zap the root, then put the refcount "acquired" above. + * When putting the reference, use kvm_tdp_mmu_put_root() to test the + * above logic for avoiding an infinite loop by freeing invalid roots. + * By design, the root is reachable while it's being zapped, thus a + * different task can put its last reference, i.e. flowing through + * kvm_tdp_mmu_put_root() for a defunct root is unavoidable. */ - tdp_mmu_zap_root(kvm, root, shared); - kvm_tdp_mmu_put_root(kvm, root, shared); + if (kvm_get_kvm_safe(kvm)) { + root->tdp_mmu_async_data =3D kvm; + INIT_WORK(&root->tdp_mmu_async_work, tdp_mmu_zap_root_async); + schedule_work(&root->tdp_mmu_async_work); + } else { + tdp_mmu_zap_root(kvm, root, shared); + kvm_tdp_mmu_put_root(kvm, root, shared); + } } =20 enum tdp_mmu_roots_iter_type { @@ -954,7 +997,11 @@ void kvm_tdp_mmu_zap_all(struct kvm *kvm) =20 /* * Zap all roots, including invalid roots, as all SPTEs must be dropped - * before returning to the caller. + * before returning to the caller. Zap directly even if the root is + * also being zapped by a worker. Walking zapped top-level SPTEs isn't + * all that expensive and mmu_lock is already held, which means the + * worker has yielded, i.e. flushing the work instead of zapping here + * isn't guaranteed to be any faster. * * A TLB flush is unnecessary, KVM zaps everything if and only the VM * is being destroyed or the userspace VMM has exited. In both cases, --=20 2.35.1.574.g5d30c73bfb-goog From nobody Tue Jun 23 21:22:54 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C3CCCC433F5 for ; Sat, 26 Feb 2022 00:18:04 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S241234AbiBZASe (ORCPT ); Fri, 25 Feb 2022 19:18:34 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50516 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S241354AbiBZARy (ORCPT ); Fri, 25 Feb 2022 19:17:54 -0500 Received: from mail-pg1-x549.google.com (mail-pg1-x549.google.com [IPv6:2607:f8b0:4864:20::549]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B594A22559E for ; Fri, 25 Feb 2022 16:16:50 -0800 (PST) Received: by mail-pg1-x549.google.com with SMTP id bm16-20020a656e90000000b00372932b1d83so3474383pgb.10 for ; Fri, 25 Feb 2022 16:16:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=reply-to:date:in-reply-to:message-id:mime-version:references :subject:from:to:cc; bh=EjwrK4ZJChvc6iOJgG1AWxHnLn4z532gyWXr0rVG/jQ=; b=lDvo9S5DXUSc+JKq46AkxVaccqJHq5C6rWdkLNnSG8RB7M0LjPk1aSTbjzHeUBF2SK LdC8PARFCyh+yPqzV5tav9taOV6Oxa+HPg0xnKCj8GKro1Ihjk0vStC60ejtziH7Kp22 hHSwQoe7biVA931GQdyCO+VJFwkH8jxSdWXJcPS3F3/jgFHejbPhBKfjtliBSkBeeDe1 XTSXxP4elMA42o+QiK1nVBNWnT11jiKN3/lT6obqM9bkRm5JT9VYMLqruY0A9EvvX79K jiQkAs77DMMXN1I5PgcKZxVhI/ZWz5mf4sza4V1Bg3oo9vG+KmZ3q66uyPOB4qwbxkLG bxfw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:reply-to:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=EjwrK4ZJChvc6iOJgG1AWxHnLn4z532gyWXr0rVG/jQ=; b=fathtZhHX+Mt1Ln0EMBagdDMVDymsXglmUPPv9oPs15obD/vq90yiXUCIfcQcEA1ec Z2yWQ5UT+GSvGj2WF4q2E96Iwi2MIsIw4HM6xifbEEHmIULndGljH+MGBgOBaFZXJLRr eyBe04Z18NwBeuPw6Cr6xL6tYJCqzsIbw8y3aa5h47NXOJ5lSEebHrNpKO31Pxgb54hT 1IxZCbAz7Tiny3Y63PeshjuaWRXb5omlcby8iUD6ZXhOHUY66nmFo2aLFMwbX+uSqVP7 oQa0L6cG/djDWOWBuWIA9DvNJgH4wMudq5DQY9dQBosTjKeZtrKUSaDIrBA7Q2VgBIJK Pr6Q== X-Gm-Message-State: AOAM533c8sH58gvQsJ+DoSlm0sfplHXNzpCXsW1vJbzG0yvJfzA9jeN5 9xwodpGmLt/rb1W0PY+yKS8Fh16hDr4= X-Google-Smtp-Source: ABdhPJzhwOAheOAxV2q4FzmEjanJlp4ncefsNjpgHyJ+GuhwmxFDKtdwHVUgHSS+KmKk/lVsS3qcCikIfn8= X-Received: from seanjc.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e5]) (user=seanjc job=sendgmr) by 2002:a17:90a:7f81:b0:1b9:4485:3f01 with SMTP id m1-20020a17090a7f8100b001b944853f01mr5573009pjl.120.1645834599991; Fri, 25 Feb 2022 16:16:39 -0800 (PST) Reply-To: Sean Christopherson Date: Sat, 26 Feb 2022 00:15:41 +0000 In-Reply-To: <20220226001546.360188-1-seanjc@google.com> Message-Id: <20220226001546.360188-24-seanjc@google.com> Mime-Version: 1.0 References: <20220226001546.360188-1-seanjc@google.com> X-Mailer: git-send-email 2.35.1.574.g5d30c73bfb-goog Subject: [PATCH v3 23/28] KVM: x86/mmu: Check for a REMOVED leaf SPTE before making the SPTE From: Sean Christopherson To: Paolo Bonzini , Christian Borntraeger , Janosch Frank , Claudio Imbrenda Cc: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , David Hildenbrand , kvm@vger.kernel.org, linux-kernel@vger.kernel.org, David Matlack , Ben Gardon , Mingwei Zhang Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Explicitly check for a REMOVED leaf SPTE prior to attempting to map the final SPTE when handling a TDP MMU fault. Functionally, this is a nop as tdp_mmu_set_spte_atomic() will eventually detect the frozen SPTE. Pre-checking for a REMOVED SPTE is a minor optmization, but the real goal is to allow tdp_mmu_set_spte_atomic() to have an invariant that the "old" SPTE is never a REMOVED SPTE. Signed-off-by: Sean Christopherson Reviewed-by: Ben Gardon --- arch/x86/kvm/mmu/tdp_mmu.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c index 4151e61245a7..1acd12bf309f 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.c +++ b/arch/x86/kvm/mmu/tdp_mmu.c @@ -1250,7 +1250,11 @@ int kvm_tdp_mmu_map(struct kvm_vcpu *vcpu, struct kv= m_page_fault *fault) } } =20 - if (iter.level !=3D fault->goal_level) { + /* + * Force the guest to retry the access if the upper level SPTEs aren't + * in place, or if the target leaf SPTE is frozen by another CPU. + */ + if (iter.level !=3D fault->goal_level || is_removed_spte(iter.old_spte)) { rcu_read_unlock(); return RET_PF_RETRY; } --=20 2.35.1.574.g5d30c73bfb-goog From nobody Tue Jun 23 21:22:54 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C3C5FC433F5 for ; Sat, 26 Feb 2022 00:18:32 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S241535AbiBZATA (ORCPT ); Fri, 25 Feb 2022 19:19:00 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51190 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S241437AbiBZARz (ORCPT ); Fri, 25 Feb 2022 19:17:55 -0500 Received: from mail-yb1-xb49.google.com (mail-yb1-xb49.google.com [IPv6:2607:f8b0:4864:20::b49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0E8282325DF for ; Fri, 25 Feb 2022 16:16:52 -0800 (PST) Received: by mail-yb1-xb49.google.com with SMTP id b64-20020a256743000000b0061e169a5f19so4968428ybc.11 for ; Fri, 25 Feb 2022 16:16:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=reply-to:date:in-reply-to:message-id:mime-version:references :subject:from:to:cc; bh=jDVD0FfZcUe5uWry8KBPjcFndvGRfOS1F52KZePQv64=; b=T9WXpGyEnZLrxkitFkBsPgYBLyCW7XwcE1VwCSGjV9s4WFQysreorHCGoMFsh8hrIM IzmblvucT4xzigG87YTxcYTRcNqbOFtd84wqGH3mYBWL8a73fTxpTd+W72Gk2DkVjLFg JTJBwaqj7alTS2R7hhU6fahZsJbsBIQDLo0mqHAJCulNVxA/aCYexv+WDODnzFHMSGa7 pAHZE38TLzNkxPPyJa5T+aGqLLcmsCY+Cg4SNmMxQPznkQVu8MZ4gMTg2zrMmBu6hrvO lGcxqdBxGELdEzHXgOoVmjk10z8+WTpCQe/Q0MSpH3qsaKxNeja84KgfseJx+eFOzU/V 0tzw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:reply-to:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=jDVD0FfZcUe5uWry8KBPjcFndvGRfOS1F52KZePQv64=; b=nwLqDZmWa7kV2mH7mQIYv4VrQgNva38scQqHgI+fTbEit0OWuopGZg3XKwgtLuOKWA GNoEDF1VXCqkV0PkTyN/4HeyNsqPh5je33MS+2t0/MKh/JCOEITUvtpri7qCKhWYil7g fK5PpjffcBQMIgY3RouPrmPUWE5Xyx2/crtXGvWtf3X9oD8ymOR/f9uwAXdeWJMDLxt9 xblUmWEhqpJQK+5Ql7+sjPMmQknwPxbtLGaq16TvvSW92+OQh4U2YehMFtGXbhMwcYCX 5bgWlmO6QI1le4dNPchEFyHUQSxnLuUjyTvpVRHJdDdjHXMuMGBfmYzodsiulCO31Qer L5kA== X-Gm-Message-State: AOAM531azqVou/ppRZq2i9AkY3q2wN/gOrm9dklWIjKlh/uLo6pU2dbb NMe5ZJ8oN8ohIejTD0oQbuOnu/dmY5g= X-Google-Smtp-Source: ABdhPJxBK2MG0ah/gTntZ+O9t2UxhQVSELyR+UzzKfx3xqejvGSGlTsiwTsqCrtgJgDW8NhWeejikSqR4IM= X-Received: from seanjc.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e5]) (user=seanjc job=sendgmr) by 2002:a25:c2c4:0:b0:615:ea99:43bc with SMTP id s187-20020a25c2c4000000b00615ea9943bcmr9759670ybf.283.1645834602019; Fri, 25 Feb 2022 16:16:42 -0800 (PST) Reply-To: Sean Christopherson Date: Sat, 26 Feb 2022 00:15:42 +0000 In-Reply-To: <20220226001546.360188-1-seanjc@google.com> Message-Id: <20220226001546.360188-25-seanjc@google.com> Mime-Version: 1.0 References: <20220226001546.360188-1-seanjc@google.com> X-Mailer: git-send-email 2.35.1.574.g5d30c73bfb-goog Subject: [PATCH v3 24/28] KVM: x86/mmu: WARN on any attempt to atomically update REMOVED SPTE From: Sean Christopherson To: Paolo Bonzini , Christian Borntraeger , Janosch Frank , Claudio Imbrenda Cc: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , David Hildenbrand , kvm@vger.kernel.org, linux-kernel@vger.kernel.org, David Matlack , Ben Gardon , Mingwei Zhang Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Disallow calling tdp_mmu_set_spte_atomic() with a REMOVED "old" SPTE. This solves a conundrum introduced by commit 3255530ab191 ("KVM: x86/mmu: Automatically update iter->old_spte if cmpxchg fails"); if the helper doesn't update old_spte in the REMOVED case, then theoretically the caller could get stuck in an infinite loop as it will fail indefinitely on the REMOVED SPTE. E.g. until recently, clear_dirty_gfn_range() didn't check for a present SPTE and would have spun until getting rescheduled. In practice, only the page fault path should "create" a new SPTE, all other paths should only operate on existing, a.k.a. shadow present, SPTEs. Now that the page fault path pre-checks for a REMOVED SPTE in all cases, require all other paths to indirectly pre-check by verifying the target SPTE is a shadow-present SPTE. Note, this does not guarantee the actual SPTE isn't REMOVED, nor is that scenario disallowed. The invariant is only that the caller mustn't invoke tdp_mmu_set_spte_atomic() if the SPTE was REMOVED when last observed by the caller. Cc: David Matlack Signed-off-by: Sean Christopherson --- arch/x86/kvm/mmu/tdp_mmu.c | 15 +++++++-------- 1 file changed, 7 insertions(+), 8 deletions(-) diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c index 1acd12bf309f..d223870b3790 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.c +++ b/arch/x86/kvm/mmu/tdp_mmu.c @@ -634,16 +634,15 @@ static inline int tdp_mmu_set_spte_atomic(struct kvm = *kvm, u64 *sptep =3D rcu_dereference(iter->sptep); u64 old_spte; =20 - WARN_ON_ONCE(iter->yielded); - - lockdep_assert_held_read(&kvm->mmu_lock); - /* - * Do not change removed SPTEs. Only the thread that froze the SPTE - * may modify it. + * The caller is responsible for ensuring the old SPTE is not a REMOVED + * SPTE. KVM should never attempt to zap or manipulate a REMOVED SPTE, + * and pre-checking before inserting a new SPTE is advantageous as it + * avoids unnecessary work. */ - if (is_removed_spte(iter->old_spte)) - return -EBUSY; + WARN_ON_ONCE(iter->yielded || is_removed_spte(iter->old_spte)); + + lockdep_assert_held_read(&kvm->mmu_lock); =20 /* * Note, fast_pf_fix_direct_spte() can also modify TDP MMU SPTEs and --=20 2.35.1.574.g5d30c73bfb-goog From nobody Tue Jun 23 21:22:54 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 108C8C433F5 for ; Sat, 26 Feb 2022 00:18:24 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S241506AbiBZASy (ORCPT ); Fri, 25 Feb 2022 19:18:54 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51230 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S241443AbiBZARz (ORCPT ); Fri, 25 Feb 2022 19:17:55 -0500 Received: from mail-pg1-x54a.google.com (mail-pg1-x54a.google.com [IPv6:2607:f8b0:4864:20::54a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A1E1222559D for ; Fri, 25 Feb 2022 16:16:54 -0800 (PST) Received: by mail-pg1-x54a.google.com with SMTP id u17-20020a63a911000000b0037491401c44so3473955pge.17 for ; Fri, 25 Feb 2022 16:16:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=reply-to:date:in-reply-to:message-id:mime-version:references :subject:from:to:cc; bh=+ihvawenlM+RVaDEQQYPtZrNBiUrDBlH+9+s6Q7Lpgg=; b=gi+AK6xiPwg8yu9CBcTyfHYef7XWZNnVoQKdY8/aXjUXsERCE0nm9lclBLqqNsXyiN BxgwOHKiIJ6xXSFIMIdy0dJynk2AfLQcrOVwVN+9xCU6DhB7++i99o/ZVpat7qEINVP/ dOf5yjb3vUeCKhWb3Ot0z0Uq/Yq8Y53XbcNFZXRuArHSyqLPhx9yu6WugYFGdCmESrqa piy8r4w9N51N7fmM6D39bBLP/0PoXTHA//5BOUoXupZrzOIjeiS8yFter7hyVp36xdvO csPPbNPwXsFJOt3B1JZ2p9EQ8e1hKjJOkeatusfWeQpfXRr2hI8WjJLIWNhKMaMGzDHk QtLg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:reply-to:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=+ihvawenlM+RVaDEQQYPtZrNBiUrDBlH+9+s6Q7Lpgg=; b=qoKtQpV7gA5kloV3t3d19sqXdPd+h5+sH78eD56MI1IxbXNp1EWXTP/FGr+sBPESTn zIXaIv0Kv7qhqrE53W/fPCye9cF82Oodmvu37RMqWNvx6kYMzX6noWxX4MOiMle8nI4O 6yyitgE3vjkcxiXdFRX1phEG+NEPHy7Zmm8vfv4uGBagNIUNcCsqED3vLOOBhJ5Og+F9 GtWHV0xR4vCSb2kwP5+4zwAlkk1GITj1JCT27Tsd9Tgkm8hNZeRvBYXUnyisomvOLmer 8/lbnljggijIMtUvqogPnoXHz0goyKmEW8dUsxyT29omMYvEMXDK0nI45iB+Wj4vSLNQ jPKg== X-Gm-Message-State: AOAM531HuZaUkeyF0i1jqIAsJ2zqtIBP+VrxHP6GAEx5oHF517HBxo2T i991ZzF4FkemUPq84UFSKdHvhn1W/Pg= X-Google-Smtp-Source: ABdhPJx+UIaYHh+lMWdP364BFZ+D4/nfCyFwVwf964cgqgQ6fWSrhGA4Dl7N8WsaGSdET2qADpF3LMCjOjg= X-Received: from seanjc.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e5]) (user=seanjc job=sendgmr) by 2002:aa7:874b:0:b0:4df:808f:2a1d with SMTP id g11-20020aa7874b000000b004df808f2a1dmr10259646pfo.68.1645834603632; Fri, 25 Feb 2022 16:16:43 -0800 (PST) Reply-To: Sean Christopherson Date: Sat, 26 Feb 2022 00:15:43 +0000 In-Reply-To: <20220226001546.360188-1-seanjc@google.com> Message-Id: <20220226001546.360188-26-seanjc@google.com> Mime-Version: 1.0 References: <20220226001546.360188-1-seanjc@google.com> X-Mailer: git-send-email 2.35.1.574.g5d30c73bfb-goog Subject: [PATCH v3 25/28] KVM: selftests: Move raw KVM_SET_USER_MEMORY_REGION helper to utils From: Sean Christopherson To: Paolo Bonzini , Christian Borntraeger , Janosch Frank , Claudio Imbrenda Cc: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , David Hildenbrand , kvm@vger.kernel.org, linux-kernel@vger.kernel.org, David Matlack , Ben Gardon , Mingwei Zhang Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Move set_memory_region_test's KVM_SET_USER_MEMORY_REGION helper to KVM's utils so that it can be used by other tests. Provide a raw version as well as an assert-success version to reduce the amount of boilerplate code need for basic usage. No functional change intended. Signed-off-by: Sean Christopherson --- .../selftests/kvm/include/kvm_util_base.h | 4 +++ tools/testing/selftests/kvm/lib/kvm_util.c | 24 +++++++++++++ .../selftests/kvm/set_memory_region_test.c | 35 +++++-------------- 3 files changed, 36 insertions(+), 27 deletions(-) diff --git a/tools/testing/selftests/kvm/include/kvm_util_base.h b/tools/te= sting/selftests/kvm/include/kvm_util_base.h index f987cf7c0d2e..573de0354175 100644 --- a/tools/testing/selftests/kvm/include/kvm_util_base.h +++ b/tools/testing/selftests/kvm/include/kvm_util_base.h @@ -147,6 +147,10 @@ void vcpu_dump(FILE *stream, struct kvm_vm *vm, uint32= _t vcpuid, =20 void vm_create_irqchip(struct kvm_vm *vm); =20 +void vm_set_user_memory_region(struct kvm_vm *vm, uint32_t slot, uint32_t = flags, + uint64_t gpa, uint64_t size, void *hva); +int __vm_set_user_memory_region(struct kvm_vm *vm, uint32_t slot, uint32_t= flags, + uint64_t gpa, uint64_t size, void *hva); void vm_userspace_mem_region_add(struct kvm_vm *vm, enum vm_mem_backing_src_type src_type, uint64_t guest_paddr, uint32_t slot, uint64_t npages, diff --git a/tools/testing/selftests/kvm/lib/kvm_util.c b/tools/testing/sel= ftests/kvm/lib/kvm_util.c index 64618032aa58..dcb8e96c6a54 100644 --- a/tools/testing/selftests/kvm/lib/kvm_util.c +++ b/tools/testing/selftests/kvm/lib/kvm_util.c @@ -839,6 +839,30 @@ static void vm_userspace_mem_region_hva_insert(struct = rb_root *hva_tree, rb_insert_color(®ion->hva_node, hva_tree); } =20 + +int __vm_set_user_memory_region(struct kvm_vm *vm, uint32_t slot, uint32_t= flags, + uint64_t gpa, uint64_t size, void *hva) +{ + struct kvm_userspace_memory_region region =3D { + .slot =3D slot, + .flags =3D flags, + .guest_phys_addr =3D gpa, + .memory_size =3D size, + .userspace_addr =3D (uintptr_t)hva, + }; + + return ioctl(vm->fd, KVM_SET_USER_MEMORY_REGION, ®ion); +} + +void vm_set_user_memory_region(struct kvm_vm *vm, uint32_t slot, uint32_t = flags, + uint64_t gpa, uint64_t size, void *hva) +{ + int ret =3D __vm_set_user_memory_region(vm, slot, flags, gpa, size, hva); + + TEST_ASSERT(!ret, "KVM_SET_USER_MEMORY_REGION failed, errno =3D %d (%s)", + errno, strerror(errno)); +} + /* * VM Userspace Memory Region Add * diff --git a/tools/testing/selftests/kvm/set_memory_region_test.c b/tools/t= esting/selftests/kvm/set_memory_region_test.c index 72a1c9b4882c..73bc297dabe6 100644 --- a/tools/testing/selftests/kvm/set_memory_region_test.c +++ b/tools/testing/selftests/kvm/set_memory_region_test.c @@ -329,22 +329,6 @@ static void test_zero_memory_regions(void) } #endif /* __x86_64__ */ =20 -static int test_memory_region_add(struct kvm_vm *vm, void *mem, uint32_t s= lot, - uint32_t size, uint64_t guest_addr) -{ - struct kvm_userspace_memory_region region; - int ret; - - region.slot =3D slot; - region.flags =3D 0; - region.guest_phys_addr =3D guest_addr; - region.memory_size =3D size; - region.userspace_addr =3D (uintptr_t) mem; - ret =3D ioctl(vm_get_fd(vm), KVM_SET_USER_MEMORY_REGION, ®ion); - - return ret; -} - /* * Test it can be added memory slots up to KVM_CAP_NR_MEMSLOTS, then any * tentative to add further slots should fail. @@ -382,23 +366,20 @@ static void test_add_max_memory_regions(void) TEST_ASSERT(mem !=3D MAP_FAILED, "Failed to mmap() host"); mem_aligned =3D (void *)(((size_t) mem + alignment - 1) & ~(alignment - 1= )); =20 - for (slot =3D 0; slot < max_mem_slots; slot++) { - ret =3D test_memory_region_add(vm, mem_aligned + - ((uint64_t)slot * MEM_REGION_SIZE), - slot, MEM_REGION_SIZE, - (uint64_t)slot * MEM_REGION_SIZE); - TEST_ASSERT(ret =3D=3D 0, "KVM_SET_USER_MEMORY_REGION IOCTL failed,\n" - " rc: %i errno: %i slot: %i\n", - ret, errno, slot); - } + for (slot =3D 0; slot < max_mem_slots; slot++) + vm_set_user_memory_region(vm, slot, 0, + ((uint64_t)slot * MEM_REGION_SIZE), + MEM_REGION_SIZE, + mem_aligned + (uint64_t)slot * MEM_REGION_SIZE); =20 /* Check it cannot be added memory slots beyond the limit */ mem_extra =3D mmap(NULL, MEM_REGION_SIZE, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); TEST_ASSERT(mem_extra !=3D MAP_FAILED, "Failed to mmap() host"); =20 - ret =3D test_memory_region_add(vm, mem_extra, max_mem_slots, MEM_REGION_S= IZE, - (uint64_t)max_mem_slots * MEM_REGION_SIZE); + ret =3D __vm_set_user_memory_region(vm, max_mem_slots, 0, + (uint64_t)max_mem_slots * MEM_REGION_SIZE, + MEM_REGION_SIZE, mem_extra); TEST_ASSERT(ret =3D=3D -1 && errno =3D=3D EINVAL, "Adding one more memory slot should fail with EINVAL"); =20 --=20 2.35.1.574.g5d30c73bfb-goog From nobody Tue Jun 23 21:22:54 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 72585C433F5 for ; Sat, 26 Feb 2022 00:18:08 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S241485AbiBZASj (ORCPT ); Fri, 25 Feb 2022 19:18:39 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50466 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S241471AbiBZARz (ORCPT ); Fri, 25 Feb 2022 19:17:55 -0500 Received: from mail-pf1-x44a.google.com (mail-pf1-x44a.google.com [IPv6:2607:f8b0:4864:20::44a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D0940225595 for ; Fri, 25 Feb 2022 16:16:55 -0800 (PST) Received: by mail-pf1-x44a.google.com with SMTP id h128-20020a625386000000b004f10a219a98so3988247pfb.0 for ; Fri, 25 Feb 2022 16:16:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=reply-to:date:in-reply-to:message-id:mime-version:references :subject:from:to:cc; bh=ZivJE8MVPu2+0VJQHIQDwCaX4QVLks+DVDCy+33e+r8=; b=mMU4YwGqseNPZMpoNYmxFxG4jtTHTrOwwNMMtFmvF5gU3KuAfnYwOTnZjRQMZCj6LL 2ge/6ljDTmTYz3JDy0ObU/z3MurRyM4S2ySEh38mtjBgB2XRzbyeuZTTOp3RhFmupm4L hqWf+267+WCsN3maMGLfrNA5cJadn3cyBsIsP//FOmbObzbe2kwKkbhShHvXyFYHGtIB gzcBXLaCiu4Fs3dVHuhYG49cqGvgJS09YjKP7bL3eyZvuADi+p3cNlB8sYDMxemp4whB 7XNLvyCKA6a6OtGvEBjRJKc6U4Io5PQjLJxHYjZencIXDE5DItvB0wddFu726/RY/Bp8 b3Jw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:reply-to:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=ZivJE8MVPu2+0VJQHIQDwCaX4QVLks+DVDCy+33e+r8=; b=yKSNX9D5GZTpkmYb4d97wZchi1Wn483vXbQesB90FJjqpeHeTMKQGvsWa4DzB89+du 9bdZj/euFTp2fb3srUqriG0U5picZNCnoSVmWkWm82CWMz1OVl8/zxuu96VSr5DM1slg cU4An7mnlpWIJ50Pvn0+xmdHfRoN4gBAH3k2ypF6D9Q/RjzrEVoa3Bd5OWkle/qM6HK8 w5APAqG02sDON1IT0H7EefgE6zoDVOnzvYKsI35aBiw5E5pEpssQiIuZhOTHfCog7QEf S1mgEK35gA1A28Xu7Cb/NIC8SImaaZktrMGGpKBF6AfNVpH2oDDvUC5UsFfttxd9xwRF uShQ== X-Gm-Message-State: AOAM530BXc2pzuWmpH2Oc2zt0hk+/V9prbMojT23Bdp+cIXIiubjAM5L yYDzDKpVK924RXgzpCjruexPRE8pSPY= X-Google-Smtp-Source: ABdhPJy1aQwG7FMFmonazVOqZZykIl+8MuBlSYKEV1ch4e6+SOFabB6zEO+fnjaBZuxd70dpPtyEts027Es= X-Received: from seanjc.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e5]) (user=seanjc job=sendgmr) by 2002:a62:1d42:0:b0:4c7:f78d:6f62 with SMTP id d63-20020a621d42000000b004c7f78d6f62mr9946549pfd.33.1645834605317; Fri, 25 Feb 2022 16:16:45 -0800 (PST) Reply-To: Sean Christopherson Date: Sat, 26 Feb 2022 00:15:44 +0000 In-Reply-To: <20220226001546.360188-1-seanjc@google.com> Message-Id: <20220226001546.360188-27-seanjc@google.com> Mime-Version: 1.0 References: <20220226001546.360188-1-seanjc@google.com> X-Mailer: git-send-email 2.35.1.574.g5d30c73bfb-goog Subject: [PATCH v3 26/28] KVM: selftests: Split out helper to allocate guest mem via memfd From: Sean Christopherson To: Paolo Bonzini , Christian Borntraeger , Janosch Frank , Claudio Imbrenda Cc: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , David Hildenbrand , kvm@vger.kernel.org, linux-kernel@vger.kernel.org, David Matlack , Ben Gardon , Mingwei Zhang Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Extract the code for allocating guest memory via memfd out of vm_userspace_mem_region_add() and into a new helper, kvm_memfd_alloc(). A future selftest to populate a guest with the maximum amount of guest memory will abuse KVM's memslots to alias guest memory regions to a single memfd-backed host region, i.e. needs to back a guest with memfd memory without a 1:1 association between a memslot and a memfd instance. No functional change intended. Signed-off-by: Sean Christopherson --- .../selftests/kvm/include/kvm_util_base.h | 1 + tools/testing/selftests/kvm/lib/kvm_util.c | 42 +++++++++++-------- 2 files changed, 25 insertions(+), 18 deletions(-) diff --git a/tools/testing/selftests/kvm/include/kvm_util_base.h b/tools/te= sting/selftests/kvm/include/kvm_util_base.h index 573de0354175..92cef0ffb19e 100644 --- a/tools/testing/selftests/kvm/include/kvm_util_base.h +++ b/tools/testing/selftests/kvm/include/kvm_util_base.h @@ -123,6 +123,7 @@ int kvm_memcmp_hva_gva(void *hva, struct kvm_vm *vm, co= nst vm_vaddr_t gva, size_t len); =20 void kvm_vm_elf_load(struct kvm_vm *vm, const char *filename); +int kvm_memfd_alloc(size_t size, bool hugepages); =20 void vm_dump(FILE *stream, struct kvm_vm *vm, uint8_t indent); =20 diff --git a/tools/testing/selftests/kvm/lib/kvm_util.c b/tools/testing/sel= ftests/kvm/lib/kvm_util.c index dcb8e96c6a54..1665a220abcb 100644 --- a/tools/testing/selftests/kvm/lib/kvm_util.c +++ b/tools/testing/selftests/kvm/lib/kvm_util.c @@ -718,6 +718,27 @@ void kvm_vm_free(struct kvm_vm *vmp) free(vmp); } =20 +int kvm_memfd_alloc(size_t size, bool hugepages) +{ + int memfd_flags =3D MFD_CLOEXEC; + int fd, r; + + if (hugepages) + memfd_flags |=3D MFD_HUGETLB; + + fd =3D memfd_create("kvm_selftest", memfd_flags); + TEST_ASSERT(fd !=3D -1, "memfd_create() failed, errno: %i (%s)", + errno, strerror(errno)); + + r =3D ftruncate(fd, size); + TEST_ASSERT(!r, "ftruncate() failed, errno: %i (%s)", errno, strerror(err= no)); + + r =3D fallocate(fd, FALLOC_FL_PUNCH_HOLE | FALLOC_FL_KEEP_SIZE, 0, size); + TEST_ASSERT(!r, "fallocate() failed, errno: %i (%s)", errno, strerror(err= no)); + + return fd; +} + /* * Memory Compare, host virtual to guest virtual * @@ -970,24 +991,9 @@ void vm_userspace_mem_region_add(struct kvm_vm *vm, region->mmap_size +=3D alignment; =20 region->fd =3D -1; - if (backing_src_is_shared(src_type)) { - int memfd_flags =3D MFD_CLOEXEC; - - if (src_type =3D=3D VM_MEM_SRC_SHARED_HUGETLB) - memfd_flags |=3D MFD_HUGETLB; - - region->fd =3D memfd_create("kvm_selftest", memfd_flags); - TEST_ASSERT(region->fd !=3D -1, - "memfd_create failed, errno: %i", errno); - - ret =3D ftruncate(region->fd, region->mmap_size); - TEST_ASSERT(ret =3D=3D 0, "ftruncate failed, errno: %i", errno); - - ret =3D fallocate(region->fd, - FALLOC_FL_PUNCH_HOLE | FALLOC_FL_KEEP_SIZE, 0, - region->mmap_size); - TEST_ASSERT(ret =3D=3D 0, "fallocate failed, errno: %i", errno); - } + if (backing_src_is_shared(src_type)) + region->fd =3D kvm_memfd_alloc(region->mmap_size, + src_type =3D=3D VM_MEM_SRC_SHARED_HUGETLB); =20 region->mmap_start =3D mmap(NULL, region->mmap_size, PROT_READ | PROT_WRITE, --=20 2.35.1.574.g5d30c73bfb-goog From nobody Tue Jun 23 21:22:54 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 69F07C433EF for ; Sat, 26 Feb 2022 00:18:19 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240910AbiBZASt (ORCPT ); Fri, 25 Feb 2022 19:18:49 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50440 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S241490AbiBZARz (ORCPT ); Fri, 25 Feb 2022 19:17:55 -0500 Received: from mail-pf1-x449.google.com (mail-pf1-x449.google.com [IPv6:2607:f8b0:4864:20::449]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E44CD234039 for ; Fri, 25 Feb 2022 16:16:56 -0800 (PST) Received: by mail-pf1-x449.google.com with SMTP id k130-20020a628488000000b004f362b45f28so3967041pfd.9 for ; Fri, 25 Feb 2022 16:16:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=reply-to:date:in-reply-to:message-id:mime-version:references :subject:from:to:cc; bh=eZ6FrXYp5R1CiB/zutWVqNTtHVIMJ9iK/usaw7re+ZM=; b=hl4rpFhiL+jXYPutqrx0t6H9bf3KWGfYlj27fcEIS4FHVk2BLJKj160C755dQet0Om 5wKQ0jzn+T5sVODiQ3ZVUFHquVzV4gTyXE1RAGwfmiNCujdwofwFLe67wnXAwY3ijDky 6SOhAc6xWzGXY2bLPyVm02FJWPuCr90UPEKlq6cMCQPphEEo9LBJWScK66QVyzou7qc5 4ZmdsrFTu6j72r1Bz7uS+S46URV1CD9jrUh3nSS5KFhoixty4Wr8AgwFKpu5RV+sqjz+ u1EMjoVbVh8eUyq40oxulkA2xXnztTkZBRfq7UC5YNTfkYaybbE9PZdge8FPKML8hwGy x+Ug== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:reply-to:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=eZ6FrXYp5R1CiB/zutWVqNTtHVIMJ9iK/usaw7re+ZM=; b=Kv4PpF3ZXoK20UtiZQA8PJLDQEbSg8bKwxN+/oc6GHD52D2CLL8jEiWCP6kQZHnuMK svA2Zvujd49pCQnu4ADZ/2UGw5ftfhyCP0GODzq++zT+56dSi/nmZy5KSWczX3/vMHjd hRUpFExS6DCHbNfu1srtOIK0L/ayGu0U74Q89DAYkreKY3M6u9zXVsLpUf2WnP4TlLVW gh82PSXRC/JUbiEI2LYJ93rRA2/czjGl9knJzJTQ5Q2+Zm8USjMTfhxVkSMRSdzxuSYS mk2NO+ZZ/RvUmTSOZjtkEiNEZtYUTI1TIwashiqMMQNr5Cy401ofDjBou53tEOo+Dt5p Eu5A== X-Gm-Message-State: AOAM530wZaeLK5sv5ZchdFL2NyP96ywJhmOZarQJKYDcF749ubSqzqIT eObnNG4qkwULMfvhs9pFJ/T15bgRemw= X-Google-Smtp-Source: ABdhPJwZkbulWxQN1QcjvFOpJ/+q7sB6nKIs0wpPywx0Zluyg3uCsp5kDjezDFiO3Pz0lKzYCwPrRRNdG08= X-Received: from seanjc.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e5]) (user=seanjc job=sendgmr) by 2002:a17:902:bc42:b0:14f:e6d0:fb7b with SMTP id t2-20020a170902bc4200b0014fe6d0fb7bmr9397561plz.127.1645834606838; Fri, 25 Feb 2022 16:16:46 -0800 (PST) Reply-To: Sean Christopherson Date: Sat, 26 Feb 2022 00:15:45 +0000 In-Reply-To: <20220226001546.360188-1-seanjc@google.com> Message-Id: <20220226001546.360188-28-seanjc@google.com> Mime-Version: 1.0 References: <20220226001546.360188-1-seanjc@google.com> X-Mailer: git-send-email 2.35.1.574.g5d30c73bfb-goog Subject: [PATCH v3 27/28] KVM: selftests: Define cpu_relax() helpers for s390 and x86 From: Sean Christopherson To: Paolo Bonzini , Christian Borntraeger , Janosch Frank , Claudio Imbrenda Cc: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , David Hildenbrand , kvm@vger.kernel.org, linux-kernel@vger.kernel.org, David Matlack , Ben Gardon , Mingwei Zhang Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add cpu_relax() for s390 and x86 for use in arch-agnostic tests. arm64 already defines its own version. Signed-off-by: Sean Christopherson --- tools/testing/selftests/kvm/include/s390x/processor.h | 8 ++++++++ tools/testing/selftests/kvm/include/x86_64/processor.h | 5 +++++ 2 files changed, 13 insertions(+) diff --git a/tools/testing/selftests/kvm/include/s390x/processor.h b/tools/= testing/selftests/kvm/include/s390x/processor.h index e0e96a5f608c..255c9b990f4c 100644 --- a/tools/testing/selftests/kvm/include/s390x/processor.h +++ b/tools/testing/selftests/kvm/include/s390x/processor.h @@ -5,6 +5,8 @@ #ifndef SELFTEST_KVM_PROCESSOR_H #define SELFTEST_KVM_PROCESSOR_H =20 +#include + /* Bits in the region/segment table entry */ #define REGION_ENTRY_ORIGIN ~0xfffUL /* region/segment table origin */ #define REGION_ENTRY_PROTECT 0x200 /* region protection bit */ @@ -19,4 +21,10 @@ #define PAGE_PROTECT 0x200 /* HW read-only bit */ #define PAGE_NOEXEC 0x100 /* HW no-execute bit */ =20 +/* Is there a portable way to do this? */ +static inline void cpu_relax(void) +{ + barrier(); +} + #endif diff --git a/tools/testing/selftests/kvm/include/x86_64/processor.h b/tools= /testing/selftests/kvm/include/x86_64/processor.h index 8a470da7b71a..37db341d4cc5 100644 --- a/tools/testing/selftests/kvm/include/x86_64/processor.h +++ b/tools/testing/selftests/kvm/include/x86_64/processor.h @@ -363,6 +363,11 @@ static inline unsigned long get_xmm(int n) return 0; } =20 +static inline void cpu_relax(void) +{ + asm volatile("rep; nop" ::: "memory"); +} + bool is_intel_cpu(void); bool is_amd_cpu(void); =20 --=20 2.35.1.574.g5d30c73bfb-goog From nobody Tue Jun 23 21:22:54 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 13663C433EF for ; Sat, 26 Feb 2022 00:18:14 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240724AbiBZASo (ORCPT ); Fri, 25 Feb 2022 19:18:44 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50336 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S241547AbiBZAR4 (ORCPT ); Fri, 25 Feb 2022 19:17:56 -0500 Received: from mail-pf1-x44a.google.com (mail-pf1-x44a.google.com [IPv6:2607:f8b0:4864:20::44a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3D2282255AE for ; Fri, 25 Feb 2022 16:16:58 -0800 (PST) Received: by mail-pf1-x44a.google.com with SMTP id h3-20020a628303000000b004e12f44a262so3936463pfe.21 for ; Fri, 25 Feb 2022 16:16:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=reply-to:date:in-reply-to:message-id:mime-version:references :subject:from:to:cc; bh=4glvYTec6CA1tuDy3vxBzyG1mlZFaJtlRRQjIc6IoPY=; b=Twv3kahbzIAS1tYSMTJWcj4ijW8tGV2GhRiX0mW0dWmRt2wph0Cdfq8VwHdr+X0akN B12+XKYO4iEoipIVwZeMclOROhqYSXPZEzYNG//MV4GPIdmKVHrkp1OZEa4J9qKLMQg5 VDrlearuWGdX/pOL1eldQUl0aSlqMoaH8jh4+aECbUiyl4g579rVdIv3RWgknoPQg0EL BnL2RAh+eNMuQt9oAHSDbYWs3wknyX1z0Ux1f54nOQckVKQZ4j5TeQ6Mkrl4A/briWX1 ZJadsR04afKwBj/XYtN6SKen2K5JleCca2FgroFZk9vlAiQDSiQ1EPb0cNIR5gtEOjGP EEVg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:reply-to:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=4glvYTec6CA1tuDy3vxBzyG1mlZFaJtlRRQjIc6IoPY=; b=JiyzF7Btbh5pjWb11lkScNGdT1gJyudjcUjLy+xQTNBE8RND/fwfg9xkgAThnVdy6h WK813/VaO3uoW17j1/0rnYu7LKOW0cpj0m1bPvXwqw4UxOqTDQt9r8B3zavnXwYQgwEb d1UywgF5Q36Q743ZktUZF7K7f0Yad7M09SbY9m//DbsPHHm1Jhkw/T0+H2owDJqVEXeH AlbU96TEV61WBG6eF/DYMlTTeaaH2sWvHGg2BzN89CeJMLm+JsNfdku8dg75Q3sdcEt6 1dZLwcgFgOFmn8mpVywsRh+UxYh0N9ZlJOsKhxQ9/7n//lLaA0NkLjfoUYPMh84LWBl6 obtw== X-Gm-Message-State: AOAM5301Qyc3doBBlpdFNfGFizPcZfx4kerPvTiWdMwGzuEf1EdhAiCA u6nb9DeMj6t0v8txxZMKTayvfczQ2ao= X-Google-Smtp-Source: ABdhPJy5PD0psRtqptbNhjQl940FK4qZ4jRgORmJxcApmp+iaGtPQHg7HZow9kgaSdeLvzAPRkFCuuPAoEE= X-Received: from seanjc.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e5]) (user=seanjc job=sendgmr) by 2002:a63:7150:0:b0:372:e0e0:f1a4 with SMTP id b16-20020a637150000000b00372e0e0f1a4mr8100838pgn.507.1645834608396; Fri, 25 Feb 2022 16:16:48 -0800 (PST) Reply-To: Sean Christopherson Date: Sat, 26 Feb 2022 00:15:46 +0000 In-Reply-To: <20220226001546.360188-1-seanjc@google.com> Message-Id: <20220226001546.360188-29-seanjc@google.com> Mime-Version: 1.0 References: <20220226001546.360188-1-seanjc@google.com> X-Mailer: git-send-email 2.35.1.574.g5d30c73bfb-goog Subject: [PATCH v3 28/28] KVM: selftests: Add test to populate a VM with the max possible guest mem From: Sean Christopherson To: Paolo Bonzini , Christian Borntraeger , Janosch Frank , Claudio Imbrenda Cc: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , David Hildenbrand , kvm@vger.kernel.org, linux-kernel@vger.kernel.org, David Matlack , Ben Gardon , Mingwei Zhang Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add a selftest that enables populating a VM with the maximum amount of guest memory allowed by the underlying architecture. Abuse KVM's memslots by mapping a single host memory region into multiple memslots so that the selftest doesn't require a system with terabytes of RAM. Default to 512gb of guest memory, which isn't all that interesting, but should work on all MMUs and doesn't take an exorbitant amount of memory or time. E.g. testing with ~64tb of guest memory takes the better part of an hour, and requires 200gb of memory for KVM's page tables when using 4kb pages. To inflicit maximum abuse on KVM' MMU, default to 4kb pages (or whatever the not-hugepage size is) in the backing store (memfd). Use memfd for the host backing store to ensure that hugepages are guaranteed when requested, and to give the user explicit control of the size of hugepage being tested. By default, spin up as many vCPUs as there are available to the selftest, and distribute the work of dirtying each 4kb chunk of memory across all vCPUs. Dirtying guest memory forces KVM to populate its page tables, and also forces KVM to write back accessed/dirty information to struct page when the guest memory is freed. On x86, perform two passes with a MMU context reset between each pass to coerce KVM into dropping all references to the MMU root, e.g. to emulate a vCPU dropping the last reference. Perform both passes and all rendezvous on all architectures in the hope that arm64 and s390x can gain similar shenanigans in the future. Measure and report the duration of each operation, which is helpful not only to verify the test is working as intended, but also to easily evaluate the performance differences different page sizes. Provide command line options to limit the amount of guest memory, set the size of each slot (i.e. of the host memory region), set the number of vCPUs, and to enable usage of hugepages. Signed-off-by: Sean Christopherson --- tools/testing/selftests/kvm/.gitignore | 1 + tools/testing/selftests/kvm/Makefile | 3 + .../selftests/kvm/max_guest_memory_test.c | 292 ++++++++++++++++++ 3 files changed, 296 insertions(+) create mode 100644 tools/testing/selftests/kvm/max_guest_memory_test.c diff --git a/tools/testing/selftests/kvm/.gitignore b/tools/testing/selftes= ts/kvm/.gitignore index 052ddfe4b23a..9b67343dc4ab 100644 --- a/tools/testing/selftests/kvm/.gitignore +++ b/tools/testing/selftests/kvm/.gitignore @@ -58,6 +58,7 @@ /hardware_disable_test /kvm_create_max_vcpus /kvm_page_table_test +/max_guest_memory_test /memslot_modification_stress_test /memslot_perf_test /rseq_test diff --git a/tools/testing/selftests/kvm/Makefile b/tools/testing/selftests= /kvm/Makefile index f7fa5655e535..c06b1f8bc649 100644 --- a/tools/testing/selftests/kvm/Makefile +++ b/tools/testing/selftests/kvm/Makefile @@ -93,6 +93,7 @@ TEST_GEN_PROGS_x86_64 +=3D dirty_log_perf_test TEST_GEN_PROGS_x86_64 +=3D hardware_disable_test TEST_GEN_PROGS_x86_64 +=3D kvm_create_max_vcpus TEST_GEN_PROGS_x86_64 +=3D kvm_page_table_test +TEST_GEN_PROGS_x86_64 +=3D max_guest_memory_test TEST_GEN_PROGS_x86_64 +=3D memslot_modification_stress_test TEST_GEN_PROGS_x86_64 +=3D memslot_perf_test TEST_GEN_PROGS_x86_64 +=3D rseq_test @@ -112,6 +113,7 @@ TEST_GEN_PROGS_aarch64 +=3D dirty_log_test TEST_GEN_PROGS_aarch64 +=3D dirty_log_perf_test TEST_GEN_PROGS_aarch64 +=3D kvm_create_max_vcpus TEST_GEN_PROGS_aarch64 +=3D kvm_page_table_test +TEST_GEN_PROGS_aarch64 +=3D max_guest_memory_test TEST_GEN_PROGS_aarch64 +=3D memslot_modification_stress_test TEST_GEN_PROGS_aarch64 +=3D memslot_perf_test TEST_GEN_PROGS_aarch64 +=3D rseq_test @@ -127,6 +129,7 @@ TEST_GEN_PROGS_s390x +=3D demand_paging_test TEST_GEN_PROGS_s390x +=3D dirty_log_test TEST_GEN_PROGS_s390x +=3D kvm_create_max_vcpus TEST_GEN_PROGS_s390x +=3D kvm_page_table_test +TEST_GEN_PROGS_s390x +=3D max_guest_memory_test TEST_GEN_PROGS_s390x +=3D rseq_test TEST_GEN_PROGS_s390x +=3D set_memory_region_test TEST_GEN_PROGS_s390x +=3D kvm_binary_stats_test diff --git a/tools/testing/selftests/kvm/max_guest_memory_test.c b/tools/te= sting/selftests/kvm/max_guest_memory_test.c new file mode 100644 index 000000000000..360c88288295 --- /dev/null +++ b/tools/testing/selftests/kvm/max_guest_memory_test.c @@ -0,0 +1,292 @@ +// SPDX-License-Identifier: GPL-2.0 +#define _GNU_SOURCE + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "kvm_util.h" +#include "test_util.h" +#include "guest_modes.h" +#include "processor.h" + +static void guest_code(uint64_t start_gpa, uint64_t end_gpa, uint64_t stri= de) +{ + uint64_t gpa; + + for (gpa =3D start_gpa; gpa < end_gpa; gpa +=3D stride) + *((volatile uint64_t *)gpa) =3D gpa; + + GUEST_DONE(); +} + +struct vcpu_info { + struct kvm_vm *vm; + uint32_t id; + uint64_t start_gpa; + uint64_t end_gpa; +}; + +static int nr_vcpus; +static atomic_t rendezvous; + +static void rendezvous_with_boss(void) +{ + int orig =3D atomic_read(&rendezvous); + + if (orig > 0) { + atomic_dec_and_test(&rendezvous); + while (atomic_read(&rendezvous) > 0) + cpu_relax(); + } else { + atomic_inc(&rendezvous); + while (atomic_read(&rendezvous) < 0) + cpu_relax(); + } +} + +static void run_vcpu(struct kvm_vm *vm, uint32_t vcpu_id) +{ + vcpu_run(vm, vcpu_id); + ASSERT_EQ(get_ucall(vm, vcpu_id, NULL), UCALL_DONE); +} + +static void *vcpu_worker(void *data) +{ + struct vcpu_info *vcpu =3D data; + struct kvm_vm *vm =3D vcpu->vm; + struct kvm_sregs sregs; + struct kvm_regs regs; + + vcpu_args_set(vm, vcpu->id, 3, vcpu->start_gpa, vcpu->end_gpa, + vm_get_page_size(vm)); + + /* Snapshot regs before the first run. */ + vcpu_regs_get(vm, vcpu->id, ®s); + rendezvous_with_boss(); + + run_vcpu(vm, vcpu->id); + rendezvous_with_boss(); + vcpu_regs_set(vm, vcpu->id, ®s); + vcpu_sregs_get(vm, vcpu->id, &sregs); +#ifdef __x86_64__ + /* Toggle CR0.WP to trigger a MMU context reset. */ + sregs.cr0 ^=3D X86_CR0_WP; +#endif + vcpu_sregs_set(vm, vcpu->id, &sregs); + rendezvous_with_boss(); + + run_vcpu(vm, vcpu->id); + rendezvous_with_boss(); + + return NULL; +} + +static pthread_t *spawn_workers(struct kvm_vm *vm, uint64_t start_gpa, + uint64_t end_gpa) +{ + struct vcpu_info *info; + uint64_t gpa, nr_bytes; + pthread_t *threads; + int i; + + threads =3D malloc(nr_vcpus * sizeof(*threads)); + TEST_ASSERT(threads, "Failed to allocate vCPU threads"); + + info =3D malloc(nr_vcpus * sizeof(*info)); + TEST_ASSERT(info, "Failed to allocate vCPU gpa ranges"); + + nr_bytes =3D ((end_gpa - start_gpa) / nr_vcpus) & + ~((uint64_t)vm_get_page_size(vm) - 1); + TEST_ASSERT(nr_bytes, "C'mon, no way you have %d CPUs", nr_vcpus); + + for (i =3D 0, gpa =3D start_gpa; i < nr_vcpus; i++, gpa +=3D nr_bytes) { + info[i].vm =3D vm; + info[i].id =3D i; + info[i].start_gpa =3D gpa; + info[i].end_gpa =3D gpa + nr_bytes; + pthread_create(&threads[i], NULL, vcpu_worker, &info[i]); + } + return threads; +} + +static void rendezvous_with_vcpus(struct timespec *time, const char *name) +{ + int i, rendezvoused; + + pr_info("Waiting for vCPUs to finish %s...\n", name); + + rendezvoused =3D atomic_read(&rendezvous); + for (i =3D 0; abs(rendezvoused) !=3D 1; i++) { + usleep(100); + if (!(i & 0x3f)) + pr_info("\r%d vCPUs haven't rendezvoused...", + abs(rendezvoused) - 1); + rendezvoused =3D atomic_read(&rendezvous); + } + + clock_gettime(CLOCK_MONOTONIC, time); + + /* Release the vCPUs after getting the time of the previous action. */ + pr_info("\rAll vCPUs finished %s, releasing...\n", name); + if (rendezvoused > 0) + atomic_set(&rendezvous, -nr_vcpus - 1); + else + atomic_set(&rendezvous, nr_vcpus + 1); +} + +static void calc_default_nr_vcpus(void) +{ + cpu_set_t possible_mask; + int r; + + r =3D sched_getaffinity(0, sizeof(possible_mask), &possible_mask); + TEST_ASSERT(!r, "sched_getaffinity failed, errno =3D %d (%s)", + errno, strerror(errno)); + + nr_vcpus =3D CPU_COUNT(&possible_mask); + TEST_ASSERT(nr_vcpus > 0, "Uh, no CPUs?"); +} + +int main(int argc, char *argv[]) +{ + /* + * Skip the first 4gb and slot0. slot0 maps <1gb and is used to back + * the guest's code, stack, and page tables. Because selftests creates + * an IRQCHIP, a.k.a. a local APIC, KVM creates an internal memslot + * just below the 4gb boundary. This test could create memory at + * 1gb-3gb,but it's simpler to skip straight to 4gb. + */ + const uint64_t size_1gb =3D (1 << 30); + const uint64_t start_gpa =3D (4ull * size_1gb); + const int first_slot =3D 1; + + struct timespec time_start, time_run1, time_reset, time_run2; + uint64_t max_gpa, gpa, slot_size, max_mem, i; + int max_slots, slot, opt, fd; + bool hugepages =3D false; + pthread_t *threads; + struct kvm_vm *vm; + void *mem; + + /* + * Default to 2gb so that maxing out systems with MAXPHADDR=3D46, which + * are quite common for x86, requires changing only max_mem (KVM allows + * 32k memslots, 32k * 2gb =3D=3D ~64tb of guest memory). + */ + slot_size =3D 2 * size_1gb; + + max_slots =3D kvm_check_cap(KVM_CAP_NR_MEMSLOTS); + TEST_ASSERT(max_slots > first_slot, "KVM is broken"); + + /* All KVM MMUs should be able to survive a 512gb guest. */ + max_mem =3D 512 * size_1gb; + + calc_default_nr_vcpus(); + + while ((opt =3D getopt(argc, argv, "c:h:m:s:u")) !=3D -1) { + switch (opt) { + case 'c': + nr_vcpus =3D atoi(optarg); + TEST_ASSERT(nr_vcpus, "#DE"); + break; + case 'm': + max_mem =3D atoi(optarg) * size_1gb; + TEST_ASSERT(max_mem, "#DE"); + break; + case 's': + slot_size =3D atoi(optarg) * size_1gb; + TEST_ASSERT(slot_size, "#DE"); + break; + case 'u': + hugepages =3D true; + break; + case 'h': + default: + printf("usage: %s [-c nr_vcpus] [-m max_mem_in_gb] [-s slot_size_in_gb]= [-u [huge_page_size]]\n", argv[0]); + exit(1); + } + } + + vm =3D vm_create_default_with_vcpus(nr_vcpus, 0, 0, guest_code, NULL); + + max_gpa =3D vm_get_max_gfn(vm) << vm_get_page_shift(vm); + TEST_ASSERT(max_gpa > (4 * slot_size), "MAXPHYADDR <4gb "); + + fd =3D kvm_memfd_alloc(slot_size, hugepages); + mem =3D mmap(NULL, slot_size, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0); + TEST_ASSERT(mem !=3D MAP_FAILED, "mmap() failed"); + + TEST_ASSERT(!madvise(mem, slot_size, MADV_NOHUGEPAGE), "madvise() failed"= ); + + /* Pre-fault the memory to avoid taking mmap_sem on guest page faults. */ + for (i =3D 0; i < slot_size; i +=3D vm_get_page_size(vm)) + ((uint8_t *)mem)[i] =3D 0xaa; + + gpa =3D 0; + for (slot =3D first_slot; slot < max_slots; slot++) { + gpa =3D start_gpa + ((slot - first_slot) * slot_size); + if (gpa + slot_size > max_gpa) + break; + + if ((gpa - start_gpa) >=3D max_mem) + break; + + vm_set_user_memory_region(vm, slot, 0, gpa, slot_size, mem); + +#ifdef __x86_64__ + /* Identity map memory in the guest using 1gb pages. */ + for (i =3D 0; i < slot_size; i +=3D size_1gb) + __virt_pg_map(vm, gpa + i, gpa + i, X86_PAGE_SIZE_1G); +#else + for (i =3D 0; i < slot_size; i +=3D vm_get_page_size(vm)) + virt_pg_map(vm, gpa + i, gpa + i); +#endif + } + + atomic_set(&rendezvous, nr_vcpus + 1); + threads =3D spawn_workers(vm, start_gpa, gpa); + + pr_info("Running with %lugb of guest memory and %u vCPUs\n", + (gpa - start_gpa) / size_1gb, nr_vcpus); + + rendezvous_with_vcpus(&time_start, "spawning"); + rendezvous_with_vcpus(&time_run1, "run 1"); + rendezvous_with_vcpus(&time_reset, "reset"); + rendezvous_with_vcpus(&time_run2, "run 2"); + + time_run2 =3D timespec_sub(time_run2, time_reset); + time_reset =3D timespec_sub(time_reset, time_run1); + time_run1 =3D timespec_sub(time_run1, time_start); + + pr_info("run1 =3D %ld.%.9lds, reset =3D %ld.%.9lds, run2 =3D %ld.%.9lds\= n", + time_run1.tv_sec, time_run1.tv_nsec, + time_reset.tv_sec, time_reset.tv_nsec, + time_run2.tv_sec, time_run2.tv_nsec); + + /* + * Delete even numbered slots (arbitrary) and unmap the first half of + * the backing (also arbitrary) to verify KVM correctly drops all + * references to the removed regions. + */ + for (slot =3D (slot - 1) & ~1ull; slot >=3D first_slot; slot -=3D 2) + vm_set_user_memory_region(vm, slot, 0, 0, 0, NULL); + + munmap(mem, slot_size / 2); + + /* Sanity check that the vCPUs actually ran. */ + for (i =3D 0; i < nr_vcpus; i++) + pthread_join(threads[i], NULL); + + /* + * Deliberately exit without deleting the remaining memslots or closing + * kvm_fd to test cleanup via mmu_notifier.release. + */ +} --=20 2.35.1.574.g5d30c73bfb-goog