From nobody Fri May 15 09:18:34 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C189FC433FE for ; Fri, 29 Apr 2022 21:00:46 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1381166AbiD2VD4 (ORCPT ); Fri, 29 Apr 2022 17:03:56 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36992 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1381154AbiD2VDx (ORCPT ); Fri, 29 Apr 2022 17:03:53 -0400 Received: from mail-pf1-x449.google.com (mail-pf1-x449.google.com [IPv6:2607:f8b0:4864:20::449]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0C813D3AD4 for ; Fri, 29 Apr 2022 14:00:32 -0700 (PDT) Received: by mail-pf1-x449.google.com with SMTP id p18-20020aa78612000000b0050d1c170018so4679021pfn.15 for ; Fri, 29 Apr 2022 14:00:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=reply-to:date:in-reply-to:message-id:mime-version:references :subject:from:to:cc; bh=LPVkLBPwEAjA8SToO1l83Vg1lu2tQY9qN9/CiKwVlGU=; b=IsuclB30S32MomEEZnx7eKe3YnpywIDDJ3ugO+zUmBmA0U7FjNOrRXhxz8mEqiCI/4 Tvz5jcIVNNULVi9NPLdtTQzJbS4NAYQnN4+IRtqufEnNf+EfYLGD0lGWhGwFpwedrjxY JjfxOH70aDEO18Jn7H4WZt3EV7hHpdrNhK6zCU2927NNekHZu7AgRNdLuivppHKPDzPY 8OX8/4WD6vHuhbsyGd+PUGXmeh7NZSG9coxaq7+8+lpMMN8EuMZud7fdzW90EgukBLlU vU/w7Q0j4A0HYxsvEfW9Cc1qkMpolziiuUAV8PV6pxmcTici3d3wQ7DFOOUzywTX2EkU Eqng== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:reply-to:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=LPVkLBPwEAjA8SToO1l83Vg1lu2tQY9qN9/CiKwVlGU=; b=MpDleNVSqSPhJzvmWmQnL1hFXCu/CfXD8dMz6xuIcix387HsLtHttGOF9RtnJHpQMI ccPuqm91S0D2HLZNJKeCB1o59OFnmIg/3OH/R9VOI/qEiwftZje5BYhUhjMBnHW+9/wV M/inguRP7JFnzbpQOua2zu3BMf6s1jZhD4dRVNjuUKMjNEklUAgBFwTGLLQd8r9Cs5g2 fiG+aq02cq/BdIqDR9y5xO1esTcD5+3dx8KKbjyh2VSh3u7wpW7iEwAWzgYZNwa0OFQD a34/ytBRalxCRr4fxc6e9EpFzJAKG7kErjt4H4fbbzNmJwe40uhGBJcPbCmQ+PekLqYH GODg== X-Gm-Message-State: AOAM531cnWHOu2Ow8eI2c/YRmxUZvoOkVlcpNWmLLSCGtmKdh4UOus2G IRNYD/hiQwJEmvSN2RuzgSklUwEI9ck= X-Google-Smtp-Source: ABdhPJwF72ZyuLedAAJsbXXnQhkqmpfiH1j4YWQDOOIF9AkF4nAk9MZU1II2RY3xDZCxYwYOWXB0pVgjTr4= X-Received: from seanjc.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e5]) (user=seanjc job=sendgmr) by 2002:a63:5917:0:b0:39c:c450:3143 with SMTP id n23-20020a635917000000b0039cc4503143mr861303pgb.531.1651266031469; Fri, 29 Apr 2022 14:00:31 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 29 Apr 2022 21:00:18 +0000 In-Reply-To: <20220429210025.3293691-1-seanjc@google.com> Message-Id: <20220429210025.3293691-2-seanjc@google.com> Mime-Version: 1.0 References: <20220429210025.3293691-1-seanjc@google.com> X-Mailer: git-send-email 2.36.0.464.gb9c8b46e94-goog Subject: [PATCH v3 1/8] Revert "KVM: Do not speculatively mark pfn cache valid to "fix" race" From: Sean Christopherson To: Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Sean Christopherson , Lai Jiangshan , David Woodhouse , Mingwei Zhang Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This reverts commit 55111927df1cd140aa7b7ea3f33f524b87776381. Signed-off-by: Sean Christopherson --- virt/kvm/pfncache.c | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/virt/kvm/pfncache.c b/virt/kvm/pfncache.c index 72eee096a7cd..71c84a43024c 100644 --- a/virt/kvm/pfncache.c +++ b/virt/kvm/pfncache.c @@ -81,8 +81,6 @@ bool kvm_gfn_to_pfn_cache_check(struct kvm *kvm, struct g= fn_to_pfn_cache *gpc, { struct kvm_memslots *slots =3D kvm_memslots(kvm); =20 - lockdep_assert_held_read(&gpc->lock); - if ((gpa & ~PAGE_MASK) + len > PAGE_SIZE) return false; =20 @@ -228,6 +226,11 @@ int kvm_gfn_to_pfn_cache_refresh(struct kvm *kvm, stru= ct gfn_to_pfn_cache *gpc, if (!old_valid || old_uhva !=3D gpc->uhva) { void *new_khva =3D NULL; =20 + /* Placeholders for "hva is valid but not yet mapped" */ + gpc->pfn =3D KVM_PFN_ERR_FAULT; + gpc->khva =3D NULL; + gpc->valid =3D true; + new_pfn =3D hva_to_pfn_retry(kvm, gpc); if (is_error_noslot_pfn(new_pfn)) { ret =3D -EFAULT; @@ -256,7 +259,7 @@ int kvm_gfn_to_pfn_cache_refresh(struct kvm *kvm, struc= t gfn_to_pfn_cache *gpc, gpc->pfn =3D KVM_PFN_ERR_FAULT; gpc->khva =3D NULL; } else { - gpc->valid =3D true; + /* At this point, gpc->valid may already have been cleared */ gpc->pfn =3D new_pfn; gpc->khva =3D new_khva; } --=20 2.36.0.464.gb9c8b46e94-goog From nobody Fri May 15 09:18:34 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A1033C433FE for ; Fri, 29 Apr 2022 21:00:49 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1381195AbiD2VEG (ORCPT ); Fri, 29 Apr 2022 17:04:06 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36996 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1381156AbiD2VDx (ORCPT ); Fri, 29 Apr 2022 17:03:53 -0400 Received: from mail-pf1-x44a.google.com (mail-pf1-x44a.google.com [IPv6:2607:f8b0:4864:20::44a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 77AE8D3AD7 for ; Fri, 29 Apr 2022 14:00:33 -0700 (PDT) Received: by mail-pf1-x44a.google.com with SMTP id k14-20020aa790ce000000b0050d3b201122so4672259pfk.20 for ; Fri, 29 Apr 2022 14:00:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=reply-to:date:in-reply-to:message-id:mime-version:references :subject:from:to:cc; bh=sYmSDs9vemF3/Z3s2wsijCq+GxxHmSayON1stTWk5n0=; b=axe09GMhmDkuVVX0wJP6zD6p/9eeAdfzoaERIJvgaCYAJ/25lJeRvMtNtT8DshPc22 Ojo2DaWzd0GfyAeYzSjc9RkXYxXseXe7XlvEWLDTHiWs+Y0YQBzcsy8B2Wcc8gQ/3Qdd 7Y2gG4sCtcemlR0Z7Hyhsp9obwKZw2RWUJ2Bx2nB9eMSKnvfeM3xPvUV1z2r3i4py7Ti 1pAqbEinMDC9JRVd9C5W1p6DEDVnXKVBC+e1E9T5Dr2xwGSiF6dV7D9uRYWyWky6eceH INIcv4oUHP8fv7aYlJsX/StGDfHmCRQ817gSS/hNaX4VRyj1Eg1dacIqLTXu8lGBgrPa BiZA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:reply-to:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=sYmSDs9vemF3/Z3s2wsijCq+GxxHmSayON1stTWk5n0=; b=hAw/blDP0FG6h4wCLBxrFGkmO2NktaqYqCh7cSJziN+u4AlkuB83eSIxK5SK+jV2Cf dypO/LvQKm/C2ujWsJ63SwRDBL417PXA50/ejHCCBIM2eDQCT5fLmAUkoP65jO3sTbsD DJn2v6vnyMecta/Jee8X003N0/z0l8Q+doxRs9CAvnp6i/Lge8Slq+nNdQd0IxODti6m v9DAk7fSBMYPdT6uxmwzLs3dxQin6BuQ1+iABbfdvJC9PBzoDYUaKA+pmi6NfvR15XZJ K5gUI70vPlILsiGy78cL/XAwkaZPcMJigUwrsoJ4nDkMH9S3yc5T/x+83n9WrV4khxbo eztQ== X-Gm-Message-State: AOAM533dABILCh0J8NOfQQfPhYljwfNfnSg1uAljpOMKYYpbdQDe/t3c v5mbnRDyw0Q4jlQKIIgQqR78sisgXOs= X-Google-Smtp-Source: ABdhPJzlW0WaPMyp/bOXIZS1s6MQhTk2krjEOUAtgwDxh+MZT2t41lytl+fXSJDNza7yoec7QJVyrwx/9iA= X-Received: from seanjc.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e5]) (user=seanjc job=sendgmr) by 2002:a17:902:9f97:b0:15d:1b87:6164 with SMTP id g23-20020a1709029f9700b0015d1b876164mr1107742plq.71.1651266032987; Fri, 29 Apr 2022 14:00:32 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 29 Apr 2022 21:00:19 +0000 In-Reply-To: <20220429210025.3293691-1-seanjc@google.com> Message-Id: <20220429210025.3293691-3-seanjc@google.com> Mime-Version: 1.0 References: <20220429210025.3293691-1-seanjc@google.com> X-Mailer: git-send-email 2.36.0.464.gb9c8b46e94-goog Subject: [PATCH v3 2/8] Revert "KVM: Fix race between mmu_notifier invalidation and pfncache refresh" From: Sean Christopherson To: Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Sean Christopherson , Lai Jiangshan , David Woodhouse , Mingwei Zhang Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This reverts commit c496097d2c0bdc229f82d72b4b1e55d64974c316. Signed-off-by: Sean Christopherson --- virt/kvm/kvm_main.c | 9 ------ virt/kvm/pfncache.c | 70 ++++++++++++++------------------------------- 2 files changed, 21 insertions(+), 58 deletions(-) diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c index 0848430f36c6..dfb7dabdbc63 100644 --- a/virt/kvm/kvm_main.c +++ b/virt/kvm/kvm_main.c @@ -705,15 +705,6 @@ static int kvm_mmu_notifier_invalidate_range_start(str= uct mmu_notifier *mn, kvm->mn_active_invalidate_count++; spin_unlock(&kvm->mn_invalidate_lock); =20 - /* - * Invalidate pfn caches _before_ invalidating the secondary MMUs, i.e. - * before acquiring mmu_lock, to avoid holding mmu_lock while acquiring - * each cache's lock. There are relatively few caches in existence at - * any given time, and the caches themselves can check for hva overlap, - * i.e. don't need to rely on memslot overlap checks for performance. - * Because this runs without holding mmu_lock, the pfn caches must use - * mn_active_invalidate_count (see above) instead of mmu_notifier_count. - */ gfn_to_pfn_cache_invalidate_start(kvm, range->start, range->end, hva_range.may_block); =20 diff --git a/virt/kvm/pfncache.c b/virt/kvm/pfncache.c index 71c84a43024c..dd84676615f1 100644 --- a/virt/kvm/pfncache.c +++ b/virt/kvm/pfncache.c @@ -112,63 +112,29 @@ static void __release_gpc(struct kvm *kvm, kvm_pfn_t = pfn, void *khva, gpa_t gpa) } } =20 -static kvm_pfn_t hva_to_pfn_retry(struct kvm *kvm, struct gfn_to_pfn_cache= *gpc) +static kvm_pfn_t hva_to_pfn_retry(struct kvm *kvm, unsigned long uhva) { - bool first_attempt =3D true; unsigned long mmu_seq; kvm_pfn_t new_pfn; + int retry; =20 - lockdep_assert_held_write(&gpc->lock); - - for (;;) { + do { mmu_seq =3D kvm->mmu_notifier_seq; smp_rmb(); =20 - write_unlock_irq(&gpc->lock); - - /* Opportunistically check for resched while the lock isn't held. */ - if (!first_attempt) - cond_resched(); - /* We always request a writeable mapping */ - new_pfn =3D hva_to_pfn(gpc->uhva, false, NULL, true, NULL); - - write_lock_irq(&gpc->lock); - + new_pfn =3D hva_to_pfn(uhva, false, NULL, true, NULL); if (is_error_noslot_pfn(new_pfn)) break; =20 - first_attempt =3D false; - - /* - * Wait for mn_active_invalidate_count, not mmu_notifier_count, - * to go away, as the invalidation in the mmu_notifier event - * occurs _before_ mmu_notifier_count is elevated. - * - * Note, mn_active_invalidate_count can change at any time as - * it's not protected by gpc->lock. But, it is guaranteed to - * be elevated before the mmu_notifier acquires gpc->lock, and - * isn't dropped until after mmu_notifier_seq is updated. So, - * this task may get a false positive of sorts, i.e. see an - * elevated count and wait even though it's technically safe to - * proceed (becase the mmu_notifier will invalidate the cache - * _after_ it's refreshed here), but the cache will never be - * refreshed with stale data, i.e. won't get false negatives. - */ - if (kvm->mn_active_invalidate_count) - continue; - - /* - * Ensure mn_active_invalidate_count is read before - * mmu_notifier_seq. This pairs with the smp_wmb() in - * mmu_notifier_invalidate_range_end() to guarantee either the - * old (non-zero) value of mn_active_invalidate_count or the - * new (incremented) value of mmu_notifier_seq is observed. - */ - smp_rmb(); - if (kvm->mmu_notifier_seq =3D=3D mmu_seq) + KVM_MMU_READ_LOCK(kvm); + retry =3D mmu_notifier_retry_hva(kvm, mmu_seq, uhva); + KVM_MMU_READ_UNLOCK(kvm); + if (!retry) break; - } + + cond_resched(); + } while (1); =20 return new_pfn; } @@ -224,6 +190,7 @@ int kvm_gfn_to_pfn_cache_refresh(struct kvm *kvm, struc= t gfn_to_pfn_cache *gpc, * drop the lock and do the HVA to PFN lookup again. */ if (!old_valid || old_uhva !=3D gpc->uhva) { + unsigned long uhva =3D gpc->uhva; void *new_khva =3D NULL; =20 /* Placeholders for "hva is valid but not yet mapped" */ @@ -231,10 +198,15 @@ int kvm_gfn_to_pfn_cache_refresh(struct kvm *kvm, str= uct gfn_to_pfn_cache *gpc, gpc->khva =3D NULL; gpc->valid =3D true; =20 - new_pfn =3D hva_to_pfn_retry(kvm, gpc); + write_unlock_irq(&gpc->lock); + + new_pfn =3D hva_to_pfn_retry(kvm, uhva); if (is_error_noslot_pfn(new_pfn)) { ret =3D -EFAULT; - } else if (gpc->usage & KVM_HOST_USES_PFN) { + goto map_done; + } + + if (gpc->usage & KVM_HOST_USES_PFN) { if (new_pfn =3D=3D old_pfn) { new_khva =3D old_khva; old_pfn =3D KVM_PFN_ERR_FAULT; @@ -250,10 +222,10 @@ int kvm_gfn_to_pfn_cache_refresh(struct kvm *kvm, str= uct gfn_to_pfn_cache *gpc, new_khva +=3D page_offset; else ret =3D -EFAULT; - } else { - /* Nothing more to do, the pfn is consumed only by the guest. */ } =20 + map_done: + write_lock_irq(&gpc->lock); if (ret) { gpc->valid =3D false; gpc->pfn =3D KVM_PFN_ERR_FAULT; --=20 2.36.0.464.gb9c8b46e94-goog From nobody Fri May 15 09:18:34 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 56204C433EF for ; Fri, 29 Apr 2022 21:00:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1381209AbiD2VEM (ORCPT ); Fri, 29 Apr 2022 17:04:12 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37020 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1381017AbiD2VDy (ORCPT ); Fri, 29 Apr 2022 17:03:54 -0400 Received: from mail-pf1-x44a.google.com (mail-pf1-x44a.google.com [IPv6:2607:f8b0:4864:20::44a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 312C8D3AD9 for ; Fri, 29 Apr 2022 14:00:35 -0700 (PDT) Received: by mail-pf1-x44a.google.com with SMTP id g5-20020a62e305000000b0050d2dba0c5dso4689435pfh.8 for ; Fri, 29 Apr 2022 14:00:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=reply-to:date:in-reply-to:message-id:mime-version:references :subject:from:to:cc; bh=ayARFRjTSHz16Sd2ddu94zxJtoFCAjCuQpe45Qe94lo=; b=S95GfX/S/Ks+zuVXI9MoanzvK+iNrqVyrr1B7wA9fsQYHWIIF7qz13FY0UTigxet+M wRPQ9E53NWyDIv1aIqeo2WS6J5ptzZA2F61t+SdgEVs0kpPGbZFH2Ww0Bi7VG/lJsWtx Cf+H/dzboMT5ekyWzpTbzid/0RSPY+t3OfMAttpBH2mzESnW+SiezSkQ/7B670VCrPBK Zd21KJFNjsz2gH8Hm6B+wJ0s0LIoQ1G4Pp/+LFUggTYBueZiu+PoIFasUxqwqnjHHX4y WrO7U/orAlGWkDlCeZkQnjsX/Ke8MWzM/P/sNK1IwjDwvZryISr/BLUoBy6nvAO5ARzS AnAA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:reply-to:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=ayARFRjTSHz16Sd2ddu94zxJtoFCAjCuQpe45Qe94lo=; b=BPG2JFdNoCcesA6F0EyIcWkQ1RoZaS3ihQEQzsw7KHJqI5BUKiid1bJAJb5by7/VBk cOK/JdNdD0b+2D+tgBSCUNrPY+vdyZ0VOd1k/z6KO+XBXEA118PqdOoZ5SoKIpbtNGbx DCDG+xl+pusv714IFen7uy/D22N7SZBVwQRQDvEYUnLLURCyMTeS9Md03bWB/nMyOYan 0k05vXpu8qOt8v6YavKThbXowNZtUMGMN7PBvZWrQ4OGm69Sox/UhO2yxLsx4Vztx9sF lKykyq5uPEkYiU0SasX7juafZOWU6eJ6Zykswha9kMIRe+7YbJ+Mh7urjU0GSk1IhhS4 tayQ== X-Gm-Message-State: AOAM530bCIWv/+VsRigCK7WvK23XcGMR3iMM9CG1XpMQfGR2opbviFSv pO+E39ZW5EQpjJM9f2CHKaE1M9VVXbM= X-Google-Smtp-Source: ABdhPJzPKEzWmZs45NDM3ffvoqgv6goI1WUDSnwE5tvuhIKuYY6ojV/10fbm9Okicd3b6vFAIa1lp9w/6GE= X-Received: from seanjc.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e5]) (user=seanjc job=sendgmr) by 2002:a17:902:bb90:b0:156:2c05:b34f with SMTP id m16-20020a170902bb9000b001562c05b34fmr1170995pls.53.1651266034679; Fri, 29 Apr 2022 14:00:34 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 29 Apr 2022 21:00:20 +0000 In-Reply-To: <20220429210025.3293691-1-seanjc@google.com> Message-Id: <20220429210025.3293691-4-seanjc@google.com> Mime-Version: 1.0 References: <20220429210025.3293691-1-seanjc@google.com> X-Mailer: git-send-email 2.36.0.464.gb9c8b46e94-goog Subject: [PATCH v3 3/8] KVM: Drop unused @gpa param from gfn=>pfn cache's __release_gpc() helper From: Sean Christopherson To: Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Sean Christopherson , Lai Jiangshan , David Woodhouse , Mingwei Zhang Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Drop the @pga param from __release_gpc() and rename the helper to make it more obvious that the cache itself is not being released. The helper will be reused by a future commit to release a pfn+khva combination that is _never_ associated with the cache, at which point the current name would go from slightly misleading to blatantly wrong. No functional change intended. Cc: stable@vger.kernel.org Signed-off-by: Sean Christopherson --- virt/kvm/pfncache.c | 10 +++------- 1 file changed, 3 insertions(+), 7 deletions(-) diff --git a/virt/kvm/pfncache.c b/virt/kvm/pfncache.c index dd84676615f1..e05a6a1b8eff 100644 --- a/virt/kvm/pfncache.c +++ b/virt/kvm/pfncache.c @@ -95,7 +95,7 @@ bool kvm_gfn_to_pfn_cache_check(struct kvm *kvm, struct g= fn_to_pfn_cache *gpc, } EXPORT_SYMBOL_GPL(kvm_gfn_to_pfn_cache_check); =20 -static void __release_gpc(struct kvm *kvm, kvm_pfn_t pfn, void *khva, gpa_= t gpa) +static void gpc_release_pfn_and_khva(struct kvm *kvm, kvm_pfn_t pfn, void = *khva) { /* Unmap the old page if it was mapped before, and release it */ if (!is_error_noslot_pfn(pfn)) { @@ -146,7 +146,6 @@ int kvm_gfn_to_pfn_cache_refresh(struct kvm *kvm, struc= t gfn_to_pfn_cache *gpc, unsigned long page_offset =3D gpa & ~PAGE_MASK; kvm_pfn_t old_pfn, new_pfn; unsigned long old_uhva; - gpa_t old_gpa; void *old_khva; bool old_valid; int ret =3D 0; @@ -160,7 +159,6 @@ int kvm_gfn_to_pfn_cache_refresh(struct kvm *kvm, struc= t gfn_to_pfn_cache *gpc, =20 write_lock_irq(&gpc->lock); =20 - old_gpa =3D gpc->gpa; old_pfn =3D gpc->pfn; old_khva =3D gpc->khva - offset_in_page(gpc->khva); old_uhva =3D gpc->uhva; @@ -244,7 +242,7 @@ int kvm_gfn_to_pfn_cache_refresh(struct kvm *kvm, struc= t gfn_to_pfn_cache *gpc, out: write_unlock_irq(&gpc->lock); =20 - __release_gpc(kvm, old_pfn, old_khva, old_gpa); + gpc_release_pfn_and_khva(kvm, old_pfn, old_khva); =20 return ret; } @@ -254,14 +252,12 @@ void kvm_gfn_to_pfn_cache_unmap(struct kvm *kvm, stru= ct gfn_to_pfn_cache *gpc) { void *old_khva; kvm_pfn_t old_pfn; - gpa_t old_gpa; =20 write_lock_irq(&gpc->lock); =20 gpc->valid =3D false; =20 old_khva =3D gpc->khva - offset_in_page(gpc->khva); - old_gpa =3D gpc->gpa; old_pfn =3D gpc->pfn; =20 /* @@ -273,7 +269,7 @@ void kvm_gfn_to_pfn_cache_unmap(struct kvm *kvm, struct= gfn_to_pfn_cache *gpc) =20 write_unlock_irq(&gpc->lock); =20 - __release_gpc(kvm, old_pfn, old_khva, old_gpa); + gpc_release_pfn_and_khva(kvm, old_pfn, old_khva); } EXPORT_SYMBOL_GPL(kvm_gfn_to_pfn_cache_unmap); =20 --=20 2.36.0.464.gb9c8b46e94-goog From nobody Fri May 15 09:18:34 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3A478C433EF for ; Fri, 29 Apr 2022 21:01:02 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1381214AbiD2VET (ORCPT ); Fri, 29 Apr 2022 17:04:19 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37130 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1381161AbiD2VD4 (ORCPT ); Fri, 29 Apr 2022 17:03:56 -0400 Received: from mail-pj1-x1049.google.com (mail-pj1-x1049.google.com [IPv6:2607:f8b0:4864:20::1049]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A5265D3ADC for ; Fri, 29 Apr 2022 14:00:36 -0700 (PDT) Received: by mail-pj1-x1049.google.com with SMTP id mm2-20020a17090b358200b001bf529127dfso4557730pjb.6 for ; Fri, 29 Apr 2022 14:00:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=reply-to:date:in-reply-to:message-id:mime-version:references :subject:from:to:cc; bh=PGUqiH/JglfCNg66NlUe0oA1IPz5o9GrdggIDdddkRE=; b=s/vbE87ZICW9wq0xlvyre3d0eVPg+hDf89OEwfE8pjHkZ5PUqoD/1vk4hhdAJKIyHQ 8rpJfCXFOB/J0SnSALJ4ick7jy62clfywkqsbYHNqegMPIM5RKcfaYdoboJ1/uyUk/QQ 5QD5H/49BtbQaJdfUPDsoY/cXP0CcVkRIwfe17PKpsiFB7eW071pViEO7cKA4yBSN6b2 1exe+gLb02QqzYQZCl1Oeljtj6nxa9ghPXX0HsMiFiuP0gCWnB5ZSC2iW+kiu94UQJle PEQhvzTFUVKe5Kp+QKMIhn4o4u87Tkobx7MOFs57gbj+LTg13JTxv4N9YxdvpaOUgAlp 73cA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:reply-to:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=PGUqiH/JglfCNg66NlUe0oA1IPz5o9GrdggIDdddkRE=; b=KF2KYhhQ4LQbeGfukq11sTfaAbggHAgd93gWUBXXNaFK33AK+oqrXKh1trioBXRfW0 xKro+5104C1aOBJLhqTFQJkSmaxsXDJSzAb2Jt5JivLutlxfB68PjY1yDKgO/j87gnJ0 jF3R/k/kqKyDOHjTG3GOzgSFuuOFXZo2cuo9c7laUHOScAPcajMwTry0clBb5s8dd0WD tf3DfSHtXQJ/juJakIattTOUWt+R5muzNWHY69ihPA5rfDlep2nipxtxvhMh8p108cpu eP5T2Dyzm1zVjvcCPVmERsaGohNSbn+efx0BECmHyXoyFPlBzH9tO/yHQJZtrbDonnM9 IbAQ== X-Gm-Message-State: AOAM533iR4zitevXG1GxLlYO2ehXNnKS/T8Q790V47BgoGUQwVSNQ6cV 8K2CHroAnN6LwtYitX3GkzjD1e8iHxA= X-Google-Smtp-Source: ABdhPJw+Ha/bmwyj6MBf0DPJjOjQzCpbQ3w89P8CYxsLPsL/ID7MWV7hYSLf9zEanA8oDKqLAG74+J5T8kc= X-Received: from seanjc.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e5]) (user=seanjc job=sendgmr) by 2002:a17:903:240f:b0:158:b871:33ac with SMTP id e15-20020a170903240f00b00158b87133acmr1131724plo.135.1651266036182; Fri, 29 Apr 2022 14:00:36 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 29 Apr 2022 21:00:21 +0000 In-Reply-To: <20220429210025.3293691-1-seanjc@google.com> Message-Id: <20220429210025.3293691-5-seanjc@google.com> Mime-Version: 1.0 References: <20220429210025.3293691-1-seanjc@google.com> X-Mailer: git-send-email 2.36.0.464.gb9c8b46e94-goog Subject: [PATCH v3 4/8] KVM: Put the extra pfn reference when reusing a pfn in the gpc cache From: Sean Christopherson To: Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Sean Christopherson , Lai Jiangshan , David Woodhouse , Mingwei Zhang Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Put the struct page reference to pfn acquired by hva_to_pfn() when the old and new pfns for a gfn=3D>pfn cache match. The cache already has a reference via the old/current pfn, and will only put one reference when the cache is done with the pfn. Fixes: 982ed0de4753 ("KVM: Reinstate gfn_to_pfn_cache with invalidation sup= port") Cc: stable@vger.kernel.org Signed-off-by: Sean Christopherson --- virt/kvm/pfncache.c | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/virt/kvm/pfncache.c b/virt/kvm/pfncache.c index e05a6a1b8eff..40cbe90d52e0 100644 --- a/virt/kvm/pfncache.c +++ b/virt/kvm/pfncache.c @@ -206,6 +206,14 @@ int kvm_gfn_to_pfn_cache_refresh(struct kvm *kvm, stru= ct gfn_to_pfn_cache *gpc, =20 if (gpc->usage & KVM_HOST_USES_PFN) { if (new_pfn =3D=3D old_pfn) { + /* + * Reuse the existing pfn and khva, but put the + * reference acquired hva_to_pfn_retry(); the + * cache still holds a reference to the pfn + * from the previous refresh. + */ + gpc_release_pfn_and_khva(kvm, new_pfn, NULL); + new_khva =3D old_khva; old_pfn =3D KVM_PFN_ERR_FAULT; old_khva =3D NULL; --=20 2.36.0.464.gb9c8b46e94-goog From nobody Fri May 15 09:18:34 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9CA40C433F5 for ; Fri, 29 Apr 2022 21:01:08 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1381225AbiD2VEZ (ORCPT ); Fri, 29 Apr 2022 17:04:25 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37304 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1381174AbiD2VD6 (ORCPT ); Fri, 29 Apr 2022 17:03:58 -0400 Received: from mail-yb1-xb4a.google.com (mail-yb1-xb4a.google.com [IPv6:2607:f8b0:4864:20::b4a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 09811D3AF1 for ; Fri, 29 Apr 2022 14:00:38 -0700 (PDT) Received: by mail-yb1-xb4a.google.com with SMTP id b11-20020a5b008b000000b00624ea481d55so8408972ybp.19 for ; Fri, 29 Apr 2022 14:00:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=reply-to:date:in-reply-to:message-id:mime-version:references :subject:from:to:cc; bh=HXnih4STfuZSP9gQw8KiDoHwzhD9DuieHszM+gFMr1Y=; b=XUor+V+kBp1C1oo8PgjLYIomhABX7qX1FK3OvrrssZKV2LGa1wVtR2fX7FmiO3Dj8q s/C6rxtHIM+f10wkgkdP1GT95Jx6oEXtwNbEPDq90hBtuX4SjnwDtDUBv5QH48ULnFZ/ tAj74mheXTeuehfK7ONkYLCNjDfKmNqLgYorwHVPUbnlnkqf3jBceecFM+AIVDnuCFvQ bRIriuLOLIpVBILX99cAWhHrv4CAxzrIqm65Mc7d6UV19o9FOHFQCJauzyU8Z3H/tu14 4zePI93u2GtS4RjN3HvQfKy2lRMr4+TiXkXtyTtCRZATNPODREUDJ7t9fZ4wuOidIQOx gLDg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:reply-to:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=HXnih4STfuZSP9gQw8KiDoHwzhD9DuieHszM+gFMr1Y=; b=nTDSdol2gW51WoCbQWaoZ4dZ/fXRRAufqWFsuI6UBP1KNPo0GczzgatIQOdkkqVc2g Ekez6yv29ZbcwyaoxJrlzx/qT7byPsocNgMtwrsq7/I92oTuzs8nxq0fcbucCdAFqT45 IcoXt0diguGyA9oaeNpq/QTqpVsGYoks58huD2H5eNmHzeIy78MYjdIB7duMzJNkBZaD +osFkjP8dKF+Xhuk80grSdMBhisyoVZOa8vfDbAHxUPm8vUEw2EwbIVKvzda0Tkb0FHT F27LnJfcL6EXRCXSxXXd1IhcLDrpUetF+KI6STjChuJ3WokhuiMRQx7p8FkIczgUvf+A 9w/A== X-Gm-Message-State: AOAM530G3Kp+kuAtQMcJ50tK+J4dAdOxi6bJme7b0frQK0x7gEhPmVDM txyrYBRAjS12gPri7XAHEgcM69toZ6M= X-Google-Smtp-Source: ABdhPJyOiW1bBv1iaSrEwJmsjNe+xWJVUni+JWC9RdV/XBpoocyR86zZqSRIhmMyOrk1oDuZ1D/TpXRitQ0= X-Received: from seanjc.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e5]) (user=seanjc job=sendgmr) by 2002:a25:31c2:0:b0:641:660f:230f with SMTP id x185-20020a2531c2000000b00641660f230fmr1364455ybx.472.1651266037988; Fri, 29 Apr 2022 14:00:37 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 29 Apr 2022 21:00:22 +0000 In-Reply-To: <20220429210025.3293691-1-seanjc@google.com> Message-Id: <20220429210025.3293691-6-seanjc@google.com> Mime-Version: 1.0 References: <20220429210025.3293691-1-seanjc@google.com> X-Mailer: git-send-email 2.36.0.464.gb9c8b46e94-goog Subject: [PATCH v3 5/8] KVM: Do not incorporate page offset into gfn=>pfn cache user address From: Sean Christopherson To: Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Sean Christopherson , Lai Jiangshan , David Woodhouse , Mingwei Zhang Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Don't adjust the userspace address in the gfn=3D>pfn cache by the page offset from the gpa. KVM should never use the user address directly, and all KVM operations that translate a user address to something else require the user address to be page aligned. Ignoring the offset will allow the cache to reuse a gfn=3D>hva translation in the unlikely event that the page offset of the gpa changes, but the gfn does not. And more importantly, not having to (un)adjust the user address will simplify a future bug fix. Cc: stable@vger.kernel.org Signed-off-by: Sean Christopherson --- virt/kvm/pfncache.c | 2 -- 1 file changed, 2 deletions(-) diff --git a/virt/kvm/pfncache.c b/virt/kvm/pfncache.c index 40cbe90d52e0..05cb0bcbf662 100644 --- a/virt/kvm/pfncache.c +++ b/virt/kvm/pfncache.c @@ -179,8 +179,6 @@ int kvm_gfn_to_pfn_cache_refresh(struct kvm *kvm, struc= t gfn_to_pfn_cache *gpc, ret =3D -EFAULT; goto out; } - - gpc->uhva +=3D page_offset; } =20 /* --=20 2.36.0.464.gb9c8b46e94-goog From nobody Fri May 15 09:18:34 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C3351C4332F for ; Fri, 29 Apr 2022 21:01:06 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1381221AbiD2VEY (ORCPT ); Fri, 29 Apr 2022 17:04:24 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37414 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1381175AbiD2VD7 (ORCPT ); Fri, 29 Apr 2022 17:03:59 -0400 Received: from mail-pj1-x1049.google.com (mail-pj1-x1049.google.com [IPv6:2607:f8b0:4864:20::1049]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1BD0AD3AD4 for ; Fri, 29 Apr 2022 14:00:40 -0700 (PDT) Received: by mail-pj1-x1049.google.com with SMTP id t15-20020a17090ae50f00b001d925488489so4564571pjy.3 for ; Fri, 29 Apr 2022 14:00:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=reply-to:date:in-reply-to:message-id:mime-version:references :subject:from:to:cc; bh=oVHyIBszbDyqiN0B8KQ/gaGeKqiBcFYqK348Whkf25k=; b=bHSvSCXEnTRlSKz9TJdTHolfFAIJ5mv5KXa90YK5sNIw1dDpT7ynOAc6ZfQuzdhZoc CPVvxgkxvFJeKWtamHyMP2S38WjV5YgRF7iUsNZX/uvXqyvOX/Mo0jsMAb6yeTiZrlxK Ar8AKwJgRX812sAM5Criwkt2G1TlF+95FD+aY86fDqDuIaS9z4nzGggEIljvikFmZKA9 +vUp2TitWs1TkNd4SgOWNpmFsTrbUTbg0XrzstiYYKHdDnlrrFemJXteYa5sW4nsxQ/C 7ZzOki7/Pm2n0FaJoEQ4WKGD8xaGDZ7ACMsRgJ68PASufILGXdyU7LVk1pHoz4Y8j13K sLIg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:reply-to:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=oVHyIBszbDyqiN0B8KQ/gaGeKqiBcFYqK348Whkf25k=; b=TTkece6sKwIal3cF6M8lWJYJU4XEBOXKOjIrUiiSZzOvOZWnAgJcOeIyOs4JrCTknG oZl2swK95T8kv5Xf5ydnZVylBYu52d9GlBRqH0CnUl5KpdTFxsZvnmXBBl7tqDoiAqn/ FXuJLHZGATXuB2CPx17VOuhxEJV5386htMwa4DCmqM0XtDzRh+gocoS2YT/wpnw3yhzV Rfugf8oPLlcBV0sJ/SI4x5bk/tsOkdWOIrrdX8ClmaVtB7wHjcii2k3nKCMzBS24oH1r qcM75WYgD2zxtA7PDkF+vl/gd2Pxcw3EXtEWwsXSYTJ/MCe5OtwIlBe2Fo79NNYu94iE 6Aqg== X-Gm-Message-State: AOAM5311PLRqEHDJSwo0lsuoST/SwhkVf7hdmPY88XumhTSW9LKvdxCV 4yUj2tFsoUlQf9H6jaGwvu8URjvJa/s= X-Google-Smtp-Source: ABdhPJznDy/XL7iYypgDUM2Vbyu+1rRgbREm4VNBI3n8aEqkpL+OB6uGl7AgT7gb/929JVQsmuabyqt3iUo= X-Received: from seanjc.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e5]) (user=seanjc job=sendgmr) by 2002:a17:90b:3442:b0:1d9:8af8:28ff with SMTP id lj2-20020a17090b344200b001d98af828ffmr979126pjb.201.1651266039638; Fri, 29 Apr 2022 14:00:39 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 29 Apr 2022 21:00:23 +0000 In-Reply-To: <20220429210025.3293691-1-seanjc@google.com> Message-Id: <20220429210025.3293691-7-seanjc@google.com> Mime-Version: 1.0 References: <20220429210025.3293691-1-seanjc@google.com> X-Mailer: git-send-email 2.36.0.464.gb9c8b46e94-goog Subject: [PATCH v3 6/8] KVM: Fully serialize gfn=>pfn cache refresh via mutex From: Sean Christopherson To: Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Sean Christopherson , Lai Jiangshan , David Woodhouse , Mingwei Zhang Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Protect gfn=3D>pfn cache refresh with a mutex to fully serialize refreshes. The refresh logic doesn't protect against concurrent refreshes with different GPAs (which may or may not be a desired use case, but it's allowed in the code), nor does it protect against a false negative on the memslot generation. If the first refresh sees a stale memslot generation, it will refresh the hva and generation before moving on to the hva=3D>pfn translation. If it then drops gpc->lock, a different user of the cache can come along, acquire gpc->lock, see that the memslot generation is fresh, and skip the hva=3D>pfn update due to the userspace address also matching (because it too was updated). The refresh path can already sleep during hva=3D>pfn resolution, so wrap the refresh with a mutex to ensure that any given refresh runs to completion before other callers can start their refresh. Cc: stable@vger.kernel.org Cc: Lai Jiangshan Signed-off-by: Sean Christopherson --- include/linux/kvm_types.h | 2 ++ virt/kvm/pfncache.c | 10 ++++++++++ 2 files changed, 12 insertions(+) diff --git a/include/linux/kvm_types.h b/include/linux/kvm_types.h index ac1ebb37a0ff..f328a01db4fe 100644 --- a/include/linux/kvm_types.h +++ b/include/linux/kvm_types.h @@ -19,6 +19,7 @@ struct kvm_memslots; enum kvm_mr_change; =20 #include +#include #include #include =20 @@ -69,6 +70,7 @@ struct gfn_to_pfn_cache { struct kvm_vcpu *vcpu; struct list_head list; rwlock_t lock; + struct mutex refresh_lock; void *khva; kvm_pfn_t pfn; enum pfn_cache_usage usage; diff --git a/virt/kvm/pfncache.c b/virt/kvm/pfncache.c index 05cb0bcbf662..eaef31462bbe 100644 --- a/virt/kvm/pfncache.c +++ b/virt/kvm/pfncache.c @@ -157,6 +157,13 @@ int kvm_gfn_to_pfn_cache_refresh(struct kvm *kvm, stru= ct gfn_to_pfn_cache *gpc, if (page_offset + len > PAGE_SIZE) return -EINVAL; =20 + /* + * If another task is refreshing the cache, wait for it to complete. + * There is no guarantee that concurrent refreshes will see the same + * gpa, memslots generation, etc..., so they must be fully serialized. + */ + mutex_lock(&gpc->refresh_lock); + write_lock_irq(&gpc->lock); =20 old_pfn =3D gpc->pfn; @@ -248,6 +255,8 @@ int kvm_gfn_to_pfn_cache_refresh(struct kvm *kvm, struc= t gfn_to_pfn_cache *gpc, out: write_unlock_irq(&gpc->lock); =20 + mutex_unlock(&gpc->refresh_lock); + gpc_release_pfn_and_khva(kvm, old_pfn, old_khva); =20 return ret; @@ -288,6 +297,7 @@ int kvm_gfn_to_pfn_cache_init(struct kvm *kvm, struct g= fn_to_pfn_cache *gpc, =20 if (!gpc->active) { rwlock_init(&gpc->lock); + mutex_init(&gpc->refresh_lock); =20 gpc->khva =3D NULL; gpc->pfn =3D KVM_PFN_ERR_FAULT; --=20 2.36.0.464.gb9c8b46e94-goog From nobody Fri May 15 09:18:34 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6D6BFC433F5 for ; Fri, 29 Apr 2022 21:01:13 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1381240AbiD2VEa (ORCPT ); Fri, 29 Apr 2022 17:04:30 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37584 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1381180AbiD2VEB (ORCPT ); Fri, 29 Apr 2022 17:04:01 -0400 Received: from mail-pf1-x44a.google.com (mail-pf1-x44a.google.com [IPv6:2607:f8b0:4864:20::44a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C5EF2D3AD4 for ; Fri, 29 Apr 2022 14:00:41 -0700 (PDT) Received: by mail-pf1-x44a.google.com with SMTP id b20-20020a62a114000000b0050a6280e374so4689071pff.13 for ; Fri, 29 Apr 2022 14:00:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=reply-to:date:in-reply-to:message-id:mime-version:references :subject:from:to:cc:content-transfer-encoding; bh=1jXnViPM/GAX2YPeMIk6CkyTEpsReANrOuuDgwYMTqs=; b=JdWl4x4Lwl57hshR6k9VgtJ8UR/iGgpODhmNu57jPxg9GGzwuyHqfYrq9Q78pDbHaF OAYfDzMC1t+PpXOO9ziLUDfKJiOeEKGvRbPmZwz7q5kJukXbbCCDUbjB3eOPcff+eiGC C7YgPJBbQ1BU+6UPbeT2FWBZa1ygGsO0DbZIrtUc/y4Y1ww2k8jYDKEBMQLCL70zeT1k o1zg/vmAEUxO9nGZlfk8uNC3M+nxyqiMy7oC8+hg0V/HqT6TjtB2XcDqugyUDvxOi1U1 B21msdvgaXDQyxUeYau4F/4DIMZrZGhb0TxFel1pPPRINNeDXw6lFX9/RdvxM8Klix6l ZCCg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:reply-to:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc :content-transfer-encoding; bh=1jXnViPM/GAX2YPeMIk6CkyTEpsReANrOuuDgwYMTqs=; b=4rlANrhUhQToSzsKmZ2tlmixleGvSWAYbKoX3+8YGB0INJcNQ6eoVe97NEnceKLrtP 8+/Txw+DGfdLXhzQbzJKqAQ2QyPB2udvLlh6faVDpEjchblZ5FV8ZJA3GwF2Nd9Epkk4 pSy023Bhz9cCtJC0+oCcv3dFiT7+2D02r2vvZrXjd2IAvAa+EpQostWwoHMp7tYUsuJS JEve1pv7YQqt8YR/RDFiq1PK56rQQ3fh0dkp1aJho28Rl4h9ybi7bDDG99lrbcC0XMT3 OFxx8JUtfzCi9dinYURLSBPz+zw7XryfTw56Veld+T9l0vh7gIsHSnR8sL7T6RKKovij +otw== X-Gm-Message-State: AOAM530Vcb+3/3iM9EhFOZyFzffPW/b0gf/cX/8KUgQTzEhTh9dibx0Y H3FkI+/v6hnYlhyw5oFm2T/7Qayc+eo= X-Google-Smtp-Source: ABdhPJyRLCi09J85/dEsY7xZ4AgF5tTPIASJu6vO288KUFu2eJXRWXEcMmeu79JvNLLkJQsgOcd6Zw1Pxxk= X-Received: from seanjc.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e5]) (user=seanjc job=sendgmr) by 2002:a17:902:b7c1:b0:15b:3c09:3ed3 with SMTP id v1-20020a170902b7c100b0015b3c093ed3mr1146005plz.73.1651266041292; Fri, 29 Apr 2022 14:00:41 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 29 Apr 2022 21:00:24 +0000 In-Reply-To: <20220429210025.3293691-1-seanjc@google.com> Message-Id: <20220429210025.3293691-8-seanjc@google.com> Mime-Version: 1.0 References: <20220429210025.3293691-1-seanjc@google.com> X-Mailer: git-send-email 2.36.0.464.gb9c8b46e94-goog Subject: [PATCH v3 7/8] KVM: Fix multiple races in gfn=>pfn cache refresh From: Sean Christopherson To: Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Sean Christopherson , Lai Jiangshan , David Woodhouse , Mingwei Zhang Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Rework the gfn=3D>pfn cache (gpc) refresh logic to address multiple races between the cache itself, and between the cache and mmu_notifier events. The existing refresh code attempts to guard against races with the mmu_notifier by speculatively marking the cache valid, and then marking it invalid if a mmu_notifier invalidation occurs. That handles the case where an invalidation occurs between dropping and re-acquiring gpc->lock, but it doesn't handle the scenario where the cache is refreshed after the cache was invalidated by the notifier, but before the notifier elevates mmu_notifier_count. The gpc refresh can't use the "retry" helper as its invalidation occurs _before_ mmu_notifier_count is elevated and before mmu_notifier_range_start is set/updated. CPU0 CPU1 ---- ---- gfn_to_pfn_cache_invalidate_start() | -> gpc->valid =3D false; kvm_gfn_to_pfn_cache_refresh() | |-> gpc->valid =3D true; hva_to_pfn_retry() | -> acquire kvm->mmu_lock kvm->mmu_notifier_count =3D=3D= 0 mmu_seq =3D=3D kvm->mmu_notifi= er_seq drop kvm->mmu_lock return pfn 'X' acquire kvm->mmu_lock kvm_inc_notifier_count() drop kvm->mmu_lock() kernel frees pfn 'X' kvm_gfn_to_pfn_cache_check() | |-> gpc->valid =3D=3D true caller accesses freed pfn 'X' Key off of mn_active_invalidate_count to detect that a pfncache refresh needs to wait for an in-progress mmu_notifier invalidation. While mn_active_invalidate_count is not guaranteed to be stable, it is guaranteed to be elevated prior to an invalidation acquiring gpc->lock, so either the refresh will see an active invalidation and wait, or the invalidation will run after the refresh completes. Speculatively marking the cache valid is itself flawed, as a concurrent kvm_gfn_to_pfn_cache_check() would see a valid cache with stale pfn/khva values. The KVM Xen use case explicitly allows/wants multiple users; even though the caches are allocated per vCPU, __kvm_xen_has_interrupt() can read a different vCPU (or vCPUs). Address this race by invalidating the cache prior to dropping gpc->lock (this is made possible by fixing the above mmu_notifier race). Complicating all of this is the fact that both the hva=3D>pfn resolution and mapping of the kernel address can sleep, i.e. must be done outside of gpc->lock. Fix the above races in one fell swoop, trying to fix each individual race is largely pointless and essentially impossible to test, e.g. closing one hole just shifts the focus to the other hole. Fixes: 982ed0de4753 ("KVM: Reinstate gfn_to_pfn_cache with invalidation sup= port") Cc: stable@vger.kernel.org Cc: David Woodhouse Cc: Mingwei Zhang Signed-off-by: Sean Christopherson --- virt/kvm/kvm_main.c | 9 +++ virt/kvm/pfncache.c | 193 ++++++++++++++++++++++++++++---------------- 2 files changed, 131 insertions(+), 71 deletions(-) diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c index dfb7dabdbc63..0848430f36c6 100644 --- a/virt/kvm/kvm_main.c +++ b/virt/kvm/kvm_main.c @@ -705,6 +705,15 @@ static int kvm_mmu_notifier_invalidate_range_start(str= uct mmu_notifier *mn, kvm->mn_active_invalidate_count++; spin_unlock(&kvm->mn_invalidate_lock); =20 + /* + * Invalidate pfn caches _before_ invalidating the secondary MMUs, i.e. + * before acquiring mmu_lock, to avoid holding mmu_lock while acquiring + * each cache's lock. There are relatively few caches in existence at + * any given time, and the caches themselves can check for hva overlap, + * i.e. don't need to rely on memslot overlap checks for performance. + * Because this runs without holding mmu_lock, the pfn caches must use + * mn_active_invalidate_count (see above) instead of mmu_notifier_count. + */ gfn_to_pfn_cache_invalidate_start(kvm, range->start, range->end, hva_range.may_block); =20 diff --git a/virt/kvm/pfncache.c b/virt/kvm/pfncache.c index eaef31462bbe..0535669ea2a1 100644 --- a/virt/kvm/pfncache.c +++ b/virt/kvm/pfncache.c @@ -112,31 +112,122 @@ static void gpc_release_pfn_and_khva(struct kvm *kvm= , kvm_pfn_t pfn, void *khva) } } =20 -static kvm_pfn_t hva_to_pfn_retry(struct kvm *kvm, unsigned long uhva) +static kvm_pfn_t hva_to_pfn_retry(struct kvm *kvm, struct gfn_to_pfn_cache= *gpc) { + /* Note, the new page offset may be different than the old! */ + void *old_khva =3D gpc->khva - offset_in_page(gpc->khva); + kvm_pfn_t new_pfn =3D KVM_PFN_ERR_FAULT; + void *new_khva =3D NULL; unsigned long mmu_seq; - kvm_pfn_t new_pfn; - int retry; =20 - do { + lockdep_assert_held_write(&gpc->lock); + + /* + * Invalidate the cache prior to dropping gpc->lock, the gpa=3D>uhva + * assets have already been updated and so a concurrent check() from a + * different task may not fail the gpa/uhva/generation checks. + */ + gpc->valid =3D false; + + for (;;) { mmu_seq =3D kvm->mmu_notifier_seq; smp_rmb(); =20 + write_unlock_irq(&gpc->lock); + + /* + * If the previous iteration "failed" due to an mmu_notifier + * event, release the pfn and unmap the kernel virtual address + * from the previous attempt. Unmapping might sleep, so this + * needs to be done after dropping the lock. Opportunistically + * check for resched while the lock isn't held. + */ + if (new_pfn !=3D KVM_PFN_ERR_FAULT) { + /* + * Keep the mapping if the previous iteration reused + * the existing mapping and didn't create a new one. + */ + if (new_khva =3D=3D old_khva) + new_khva =3D NULL; + + gpc_release_pfn_and_khva(kvm, new_pfn, new_khva); + + cond_resched(); + } + /* We always request a writeable mapping */ - new_pfn =3D hva_to_pfn(uhva, false, NULL, true, NULL); + new_pfn =3D hva_to_pfn(gpc->uhva, false, NULL, true, NULL); if (is_error_noslot_pfn(new_pfn)) - break; + goto out_error; + + /* + * Obtain a new kernel mapping if KVM itself will access the + * pfn. Note, kmap() and memremap() can both sleep, so this + * too must be done outside of gpc->lock! + */ + if (gpc->usage & KVM_HOST_USES_PFN) { + if (new_pfn =3D=3D gpc->pfn) { + new_khva =3D old_khva; + } else if (pfn_valid(new_pfn)) { + new_khva =3D kmap(pfn_to_page(new_pfn)); +#ifdef CONFIG_HAS_IOMEM + } else { + new_khva =3D memremap(pfn_to_hpa(new_pfn), PAGE_SIZE, MEMREMAP_WB); +#endif + } + if (!new_khva) { + kvm_release_pfn_clean(new_pfn); + goto out_error; + } + } + + write_lock_irq(&gpc->lock); =20 - KVM_MMU_READ_LOCK(kvm); - retry =3D mmu_notifier_retry_hva(kvm, mmu_seq, uhva); - KVM_MMU_READ_UNLOCK(kvm); - if (!retry) + /* + * Other tasks must wait for _this_ refresh to complete before + * attempting to refresh. + */ + WARN_ON_ONCE(gpc->valid); + + /* + * Wait for mn_active_invalidate_count, not mmu_notifier_count, + * to go away, as the invalidation in the mmu_notifier event + * occurs _before_ mmu_notifier_count is elevated. + * + * Note, mn_active_invalidate_count can change at any time as + * it's not protected by gpc->lock. But, it is guaranteed to + * be elevated before the mmu_notifier acquires gpc->lock, and + * isn't dropped until after mmu_notifier_seq is updated. So, + * this task may get a false positive of sorts, i.e. see an + * elevated count and wait even though it's technically safe to + * proceed (becase the mmu_notifier will invalidate the cache + * _after_ it's refreshed here), but the cache will never be + * refreshed with stale data, i.e. won't get false negatives. + */ + if (kvm->mn_active_invalidate_count) + continue; + + /* + * Ensure mn_active_invalidate_count is read before + * mmu_notifier_seq. This pairs with the smp_wmb() in + * mmu_notifier_invalidate_range_end() to guarantee either the + * old (non-zero) value of mn_active_invalidate_count or the + * new (incremented) value of mmu_notifier_seq is observed. + */ + smp_rmb(); + if (kvm->mmu_notifier_seq =3D=3D mmu_seq) break; + } + + gpc->valid =3D true; + gpc->pfn =3D new_pfn; + gpc->khva =3D new_khva + (gpc->gpa & ~PAGE_MASK); + return 0; =20 - cond_resched(); - } while (1); +out_error: + write_lock_irq(&gpc->lock); =20 - return new_pfn; + return -EFAULT; } =20 int kvm_gfn_to_pfn_cache_refresh(struct kvm *kvm, struct gfn_to_pfn_cache = *gpc, @@ -147,7 +238,6 @@ int kvm_gfn_to_pfn_cache_refresh(struct kvm *kvm, struc= t gfn_to_pfn_cache *gpc, kvm_pfn_t old_pfn, new_pfn; unsigned long old_uhva; void *old_khva; - bool old_valid; int ret =3D 0; =20 /* @@ -169,7 +259,6 @@ int kvm_gfn_to_pfn_cache_refresh(struct kvm *kvm, struc= t gfn_to_pfn_cache *gpc, old_pfn =3D gpc->pfn; old_khva =3D gpc->khva - offset_in_page(gpc->khva); old_uhva =3D gpc->uhva; - old_valid =3D gpc->valid; =20 /* If the userspace HVA is invalid, refresh that first */ if (gpc->gpa !=3D gpa || gpc->generation !=3D slots->generation || @@ -182,7 +271,6 @@ int kvm_gfn_to_pfn_cache_refresh(struct kvm *kvm, struc= t gfn_to_pfn_cache *gpc, gpc->uhva =3D gfn_to_hva_memslot(gpc->memslot, gfn); =20 if (kvm_is_error_hva(gpc->uhva)) { - gpc->pfn =3D KVM_PFN_ERR_FAULT; ret =3D -EFAULT; goto out; } @@ -192,60 +280,8 @@ int kvm_gfn_to_pfn_cache_refresh(struct kvm *kvm, stru= ct gfn_to_pfn_cache *gpc, * If the userspace HVA changed or the PFN was already invalid, * drop the lock and do the HVA to PFN lookup again. */ - if (!old_valid || old_uhva !=3D gpc->uhva) { - unsigned long uhva =3D gpc->uhva; - void *new_khva =3D NULL; - - /* Placeholders for "hva is valid but not yet mapped" */ - gpc->pfn =3D KVM_PFN_ERR_FAULT; - gpc->khva =3D NULL; - gpc->valid =3D true; - - write_unlock_irq(&gpc->lock); - - new_pfn =3D hva_to_pfn_retry(kvm, uhva); - if (is_error_noslot_pfn(new_pfn)) { - ret =3D -EFAULT; - goto map_done; - } - - if (gpc->usage & KVM_HOST_USES_PFN) { - if (new_pfn =3D=3D old_pfn) { - /* - * Reuse the existing pfn and khva, but put the - * reference acquired hva_to_pfn_retry(); the - * cache still holds a reference to the pfn - * from the previous refresh. - */ - gpc_release_pfn_and_khva(kvm, new_pfn, NULL); - - new_khva =3D old_khva; - old_pfn =3D KVM_PFN_ERR_FAULT; - old_khva =3D NULL; - } else if (pfn_valid(new_pfn)) { - new_khva =3D kmap(pfn_to_page(new_pfn)); -#ifdef CONFIG_HAS_IOMEM - } else { - new_khva =3D memremap(pfn_to_hpa(new_pfn), PAGE_SIZE, MEMREMAP_WB); -#endif - } - if (new_khva) - new_khva +=3D page_offset; - else - ret =3D -EFAULT; - } - - map_done: - write_lock_irq(&gpc->lock); - if (ret) { - gpc->valid =3D false; - gpc->pfn =3D KVM_PFN_ERR_FAULT; - gpc->khva =3D NULL; - } else { - /* At this point, gpc->valid may already have been cleared */ - gpc->pfn =3D new_pfn; - gpc->khva =3D new_khva; - } + if (!gpc->valid || old_uhva !=3D gpc->uhva) { + ret =3D hva_to_pfn_retry(kvm, gpc); } else { /* If the HVA=E2=86=92PFN mapping was already valid, don't unmap it. */ old_pfn =3D KVM_PFN_ERR_FAULT; @@ -253,11 +289,26 @@ int kvm_gfn_to_pfn_cache_refresh(struct kvm *kvm, str= uct gfn_to_pfn_cache *gpc, } =20 out: + /* + * Invalidate the cache and purge the pfn/khva if the refresh failed. + * Some/all of the uhva, gpa, and memslot generation info may still be + * valid, leave it as is. + */ + if (ret) { + gpc->valid =3D false; + gpc->pfn =3D KVM_PFN_ERR_FAULT; + gpc->khva =3D NULL; + } + + /* Snapshot the new pfn before dropping the lock! */ + new_pfn =3D gpc->pfn; + write_unlock_irq(&gpc->lock); =20 mutex_unlock(&gpc->refresh_lock); =20 - gpc_release_pfn_and_khva(kvm, old_pfn, old_khva); + if (old_pfn !=3D new_pfn) + gpc_release_pfn_and_khva(kvm, old_pfn, old_khva); =20 return ret; } --=20 2.36.0.464.gb9c8b46e94-goog From nobody Fri May 15 09:18:34 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id DFEA2C433EF for ; Fri, 29 Apr 2022 21:01:16 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1381248AbiD2VEc (ORCPT ); Fri, 29 Apr 2022 17:04:32 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37674 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1381182AbiD2VEC (ORCPT ); Fri, 29 Apr 2022 17:04:02 -0400 Received: from mail-pg1-x549.google.com (mail-pg1-x549.google.com [IPv6:2607:f8b0:4864:20::549]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 68AECD3ACA for ; Fri, 29 Apr 2022 14:00:43 -0700 (PDT) Received: by mail-pg1-x549.google.com with SMTP id c194-20020a6335cb000000b0039d9a489d44so4227227pga.6 for ; Fri, 29 Apr 2022 14:00:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=reply-to:date:in-reply-to:message-id:mime-version:references :subject:from:to:cc; bh=zP2FwEa8Nl8vyFhUbOzmbrhYzn9cxhFMqsrmIQIuWwA=; b=QK+xPeiwXa6e33Pjngc5P2E6KIFwQs9puOhH3flfkWXWJjvTKXQ4n91xk7xnBs0MNz CaTaxYpqw7PaqazqF3yUJGpzzeZ8ns3tbtrUYeByfLzUy3PLjdebI4YjLp7A2sYGkYzT EJp9fMGmqRLDPZUA4odmFAcwCLnNP35Um8+Ftl6Mr0VdEz0WeJ89UzdHtM8aL34pgIgy tO32M5MDejwBUSal0KDQviidOrx3DfvdQKvqZNAUCp0SKJi4yDWbB8NKxbUMzOvfSKvn J40p4UHDnCJHgrukR+KdzHhjTuPl+GWEBtkh6ZGhIpNnM/nV03UA8bWZWwuJuPOYiD5i CdtA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:reply-to:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=zP2FwEa8Nl8vyFhUbOzmbrhYzn9cxhFMqsrmIQIuWwA=; b=X/GL5ly0lx4mK4xUF8FnfX5k/rIOrY/SWBy3UhGazYwh7gT7kMa3fwwnmXVTKVJ9ot y/ks7imGG8ffIKCECs9KotoGhoieK8rxHEn2ewXHarT6XrSknBj9lwPtOe/aS3Ei3unY b6o6+xnpHuUPCSsBxxmIUNof9oLq3bPEi7LakDN0vYoaYHfwFD/6qedOiRe3amaQS3Wz /zcHMWibe3/35uQpH1pL8ZodopW5ow30l/qhz+Mag3dbds0NSOfc5ktINTuQinTXp0D9 NqNSanfmvQQJin7E6BwhJltDZnvPXgcP+7XwK/OSjoNTexvJWVD0ctRorR7Ra5+5BEX/ xVzw== X-Gm-Message-State: AOAM533yFL1H11xiCciNXwDU/mXxkLlfSX6OcMo9vSgLvyfi4PrhZzJi gsovYUnOfxtxAe5l5UxEtxhKJcgm2qg= X-Google-Smtp-Source: ABdhPJz4W33FH39XL8rGj5+zrdiDe0gJH35R1ZBKg0xwvpX8Ebhc6cVEBCEMyApi7dDt9qXfs+Vz2pnMXrs= X-Received: from seanjc.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e5]) (user=seanjc job=sendgmr) by 2002:a05:6a00:851:b0:50d:62f6:5494 with SMTP id q17-20020a056a00085100b0050d62f65494mr799201pfk.75.1651266042949; Fri, 29 Apr 2022 14:00:42 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 29 Apr 2022 21:00:25 +0000 In-Reply-To: <20220429210025.3293691-1-seanjc@google.com> Message-Id: <20220429210025.3293691-9-seanjc@google.com> Mime-Version: 1.0 References: <20220429210025.3293691-1-seanjc@google.com> X-Mailer: git-send-email 2.36.0.464.gb9c8b46e94-goog Subject: [PATCH v3 8/8] KVM: Do not pin pages tracked by gfn=>pfn caches From: Sean Christopherson To: Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Sean Christopherson , Lai Jiangshan , David Woodhouse , Mingwei Zhang Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Put the reference to any struct page mapped/tracked by a gfn=3D>pfn cache upon inserting the pfn into its associated cache, as opposed to putting the reference only when the cache is done using the pfn. In other words, don't pin pages while they're in the cache. One of the major roles of the gfn=3D>pfn cache is to play nicely with invalidation events, i.e. it exists in large part so that KVM doesn't rely on pinning pages. Signed-off-by: Sean Christopherson --- virt/kvm/pfncache.c | 36 ++++++++++++++++++++---------------- 1 file changed, 20 insertions(+), 16 deletions(-) diff --git a/virt/kvm/pfncache.c b/virt/kvm/pfncache.c index 0535669ea2a1..c4fb81929663 100644 --- a/virt/kvm/pfncache.c +++ b/virt/kvm/pfncache.c @@ -95,20 +95,16 @@ bool kvm_gfn_to_pfn_cache_check(struct kvm *kvm, struct= gfn_to_pfn_cache *gpc, } EXPORT_SYMBOL_GPL(kvm_gfn_to_pfn_cache_check); =20 -static void gpc_release_pfn_and_khva(struct kvm *kvm, kvm_pfn_t pfn, void = *khva) +static void gpc_unmap_khva(struct kvm *kvm, kvm_pfn_t pfn, void *khva) { - /* Unmap the old page if it was mapped before, and release it */ - if (!is_error_noslot_pfn(pfn)) { - if (khva) { - if (pfn_valid(pfn)) - kunmap(pfn_to_page(pfn)); + /* Unmap the old pfn/page if it was mapped before. */ + if (!is_error_noslot_pfn(pfn) && khva) { + if (pfn_valid(pfn)) + kunmap(pfn_to_page(pfn)); #ifdef CONFIG_HAS_IOMEM - else - memunmap(khva); + else + memunmap(khva); #endif - } - - kvm_release_pfn(pfn, false); } } =20 @@ -147,10 +143,10 @@ static kvm_pfn_t hva_to_pfn_retry(struct kvm *kvm, st= ruct gfn_to_pfn_cache *gpc) * Keep the mapping if the previous iteration reused * the existing mapping and didn't create a new one. */ - if (new_khva =3D=3D old_khva) - new_khva =3D NULL; + if (new_khva !=3D old_khva) + gpc_unmap_khva(kvm, new_pfn, new_khva); =20 - gpc_release_pfn_and_khva(kvm, new_pfn, new_khva); + kvm_release_pfn_clean(new_pfn); =20 cond_resched(); } @@ -222,6 +218,14 @@ static kvm_pfn_t hva_to_pfn_retry(struct kvm *kvm, str= uct gfn_to_pfn_cache *gpc) gpc->valid =3D true; gpc->pfn =3D new_pfn; gpc->khva =3D new_khva + (gpc->gpa & ~PAGE_MASK); + + /* + * Put the reference to the _new_ pfn. The pfn is now tracked by the + * cache and can be safely migrated, swapped, etc... as the cache will + * invalidate any mappings in response to relevant mmu_notifier events. + */ + kvm_release_pfn_clean(new_pfn); + return 0; =20 out_error: @@ -308,7 +312,7 @@ int kvm_gfn_to_pfn_cache_refresh(struct kvm *kvm, struc= t gfn_to_pfn_cache *gpc, mutex_unlock(&gpc->refresh_lock); =20 if (old_pfn !=3D new_pfn) - gpc_release_pfn_and_khva(kvm, old_pfn, old_khva); + gpc_unmap_khva(kvm, old_pfn, old_khva); =20 return ret; } @@ -335,7 +339,7 @@ void kvm_gfn_to_pfn_cache_unmap(struct kvm *kvm, struct= gfn_to_pfn_cache *gpc) =20 write_unlock_irq(&gpc->lock); =20 - gpc_release_pfn_and_khva(kvm, old_pfn, old_khva); + gpc_unmap_khva(kvm, old_pfn, old_khva); } EXPORT_SYMBOL_GPL(kvm_gfn_to_pfn_cache_unmap); =20 --=20 2.36.0.464.gb9c8b46e94-goog