From nobody Tue Nov 26 08:38:40 2024 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4EA7E1E8851 for ; Fri, 11 Oct 2024 02:10:55 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728612656; cv=none; b=hslTV1EatN7DWQFUVmN/JPNVUVAo0d3nVbpTres5d3YS3XAiSAWSUdWDzt8fh+J1MrNBiaL0vHi2J/fYtDXzr9hA5sTtCua/n/PVTKP1g1xAIXYVW6VfM+8gaj+YqiN7KAN/C8Iu6YSTzUaUdplrFfMFoc0h8VXvhiU5iw0Jcyo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728612656; c=relaxed/simple; bh=fzW+ChUdusmIA3bhbKK2k3ivftGQJ9q+cRZWuOg3AfQ=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=O0LLoKeHem6NFRfhcTYVdaO3jEieSIisA/0D4KPu0JI0ol9Tmd1Y8S4pNLeHyTemiqlc8T6ZUbKJC0CDB2VyXxJK0Jsvokp2BPBRUH070/rz3VuUakP9fJvt/PlP72kWWOBLQDOL8Ho6dyQxyZupj+vXN0cgWfbkmhFcrHuHf2o= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=W50wc7Wd; arc=none smtp.client-ip=209.85.128.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="W50wc7Wd" Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-6886cd07673so35268587b3.3 for ; Thu, 10 Oct 2024 19:10:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1728612654; x=1729217454; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=OygTTCCcvLlJR5HYkc6p+scOWI8DJUkTUyIz5b+Yzqs=; b=W50wc7WdMCZH6equDluATzYxsEiHw5XfgROMz26kkPkpDgUX4bTjzGENrwgsKkiYUM aDOUpI163O3U4r6NkJZrbrQSX1gHDZaZYsCPBxJynFc9zxwwZUPlg7fICUTctp9VdrgV zHmJYCqbOpxveLtns16MOKvmcemENHCjbg+UWNotKBXdnLlSUlesHx/meKOha7oTeH5D sZ11RY3Y+NNKWFxbhJfB4Td/6L4U7azcYNqBLFIKr4PF8DF5dGjn7kRzUgRU5ROQ4QeG mE9aXamCaq9aXH8hLbuC+29607Lx+Rg8XT9m7TKs1Xc6V8FEG03LYjn+k0HBeFZh6LFt pAyw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1728612654; x=1729217454; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=OygTTCCcvLlJR5HYkc6p+scOWI8DJUkTUyIz5b+Yzqs=; b=q5j2YqT9KuMk0ubszss3PgpTMcHpBCOSjxk2YKu5s3DwFGo9l9HlS60mn+6OGSHi6Q GZivDIrLk+4Ry3EoUTMm6VQ46DkjQs2h/MPrG5dTzEdazJkcKEzWIbDHsu484Bi9m/Mn GH4A70YbOSBlfBUsB3VAc0K8pERBLCiaiCOmObArughk6zqEd7M/dL0U2ahZrJSDDdFP //cvqgs0KI5dQZ0uiSRzlfh6GUJPxU+7fjGrMr7VRnU50fkurMNZzgeKb+pWIGrEHpQa 3Cp2s3BYJYHjNvBSR0gkTqQj1kdxmcTAL31RCKxEHsb6sy+8tQvft1bPXDxQ03Ncz1Kn vcmA== X-Forwarded-Encrypted: i=1; AJvYcCWb9MdG42YTGnmb+8lu7X1XSOSEG1ZfwGEg72syfDnhHA2273rSLmQHcAwgIYBlvmrybis+3SK2h22UF/I=@vger.kernel.org X-Gm-Message-State: AOJu0YyFI5odro1kG+K6kFiarDJAu+JgoAkmHPWcQMnJ9GQACO5O1Ho1 0xjPm55Z/WsZd54rZpkB7cPqJLTmdtb9dKxrup19giUqi2uwJFbowPrt/oegIQX/6EVlvPOCKgO VJQ== X-Google-Smtp-Source: AGHT+IHdRSYZ+bkxWUZczD2ZC6uRWUXF5TWEoclipxa3r0H2IwoZrfNvJ+VwB67O1mgdW36wABwHwe6bwqs= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:9d:3983:ac13:c240]) (user=seanjc job=sendgmr) by 2002:a05:690c:2e13:b0:6b2:3ecc:817 with SMTP id 00721157ae682-6e347c6547dmr20677b3.8.1728612654326; Thu, 10 Oct 2024 19:10:54 -0700 (PDT) Reply-To: Sean Christopherson Date: Thu, 10 Oct 2024 19:10:33 -0700 In-Reply-To: <20241011021051.1557902-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20241011021051.1557902-1-seanjc@google.com> X-Mailer: git-send-email 2.47.0.rc1.288.g06298d1525-goog Message-ID: <20241011021051.1557902-2-seanjc@google.com> Subject: [PATCH 01/18] KVM: x86/mmu: Flush remote TLBs iff MMU-writable flag is cleared from RO SPTE From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Yan Zhao , Sagi Shahar , "=?UTF-8?q?Alex=20Benn=C3=A9e?=" , David Matlack , James Houghton Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Don't force a remote TLB flush if KVM happens to effectively "refresh" a read-only SPTE that is still MMU-Writable, as KVM allows MMU-Writable SPTEs to have Writable TLB entries, even if the SPTE is !Writable. Remote TLBs need to be flushed only when creating a read-only SPTE for write-tracking, i.e. when installing a !MMU-Writable SPTE. In practice, especially now that KVM doesn't overwrite existing SPTEs when prefetching, KVM will rarely "refresh" a read-only, MMU-Writable SPTE, i.e. this is unlikely to eliminate many, if any, TLB flushes. But, more precisely flushing makes it easier to understand exactly when KVM does and doesn't need to flush. Note, x86 architecturally requires relevant TLB entries to be invalidated on a page fault, i.e. there is no risk of putting a vCPU into an infinite loop of read-only page faults. Cc: Yan Zhao Signed-off-by: Sean Christopherson --- arch/x86/kvm/mmu/mmu.c | 12 +++++++----- arch/x86/kvm/mmu/spte.c | 6 ------ 2 files changed, 7 insertions(+), 11 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index 55eeca931e23..176fc37540df 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -514,9 +514,12 @@ static u64 mmu_spte_update_no_track(u64 *sptep, u64 ne= w_spte) /* Rules for using mmu_spte_update: * Update the state bits, it means the mapped pfn is not changed. * - * Whenever an MMU-writable SPTE is overwritten with a read-only SPTE, rem= ote - * TLBs must be flushed. Otherwise rmap_write_protect will find a read-only - * spte, even though the writable spte might be cached on a CPU's TLB. + * If the MMU-writable flag is cleared, i.e. the SPTE is write-protected f= or + * write-tracking, remote TLBs must be flushed, even if the SPTE was read-= only, + * as KVM allows stale Writable TLB entries to exist. When dirty logging,= KVM + * flushes TLBs based on whether or not dirty bitmap/ring entries were rea= ped, + * not whether or not SPTEs were modified, i.e. only the write-tracking ca= se + * needs to flush at the time the SPTEs is modified, before dropping mmu_l= ock. * * Returns true if the TLB needs to be flushed */ @@ -533,8 +536,7 @@ static bool mmu_spte_update(u64 *sptep, u64 new_spte) * we always atomically update it, see the comments in * spte_has_volatile_bits(). */ - if (is_mmu_writable_spte(old_spte) && - !is_writable_pte(new_spte)) + if (is_mmu_writable_spte(old_spte) && !is_mmu_writable_spte(new_spte)) flush =3D true; =20 /* diff --git a/arch/x86/kvm/mmu/spte.c b/arch/x86/kvm/mmu/spte.c index f1a50a78badb..e5af69a8f101 100644 --- a/arch/x86/kvm/mmu/spte.c +++ b/arch/x86/kvm/mmu/spte.c @@ -133,12 +133,6 @@ static bool kvm_is_mmio_pfn(kvm_pfn_t pfn) */ bool spte_has_volatile_bits(u64 spte) { - /* - * Always atomically update spte if it can be updated - * out of mmu-lock, it can ensure dirty bit is not lost, - * also, it can help us to get a stable is_writable_pte() - * to ensure tlb flush is not missed. - */ if (!is_writable_pte(spte) && is_mmu_writable_spte(spte)) return true; =20 --=20 2.47.0.rc1.288.g06298d1525-goog From nobody Tue Nov 26 08:38:40 2024 Received: from mail-pf1-f202.google.com (mail-pf1-f202.google.com [209.85.210.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E91C91EABCB for ; Fri, 11 Oct 2024 02:10:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728612658; cv=none; b=YIYNcS3CIeKVWONUa9RK4J6uNKYH/uOiabJATeP5rgyD7w7Z20IHCG18mrd+jWwaFBd9YF8bAiesHcpsP/2E7UZNC9KDWKbBE6rwf18fhPF3X3YuQOjf0Iw+kop06ROGfadanhtdaGBAiAFWoP+2aeZ18O005G4C/ArLocBf70Y= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728612658; c=relaxed/simple; bh=ZfG0AxRcRvfbvEdSICe+Rwv0lp2Ksqh/F9qZHu/Bn+4=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=NMZIlOxgA/PrHZdVTaDpb6U6ApQfVHCskhiIBFggWxnY9tYk1XH+eLU2AfC02dTVDbsjYQBQyw6zJVbiURCT97DSjx7m+PIAyO7xZH6q5FBoEMSb3geoRwr7bLl3iBRXnmbCrOK4uWqUm1hLKj+M53SJnEoPHPnmx5smpVLhLDU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=Mfu/9XEN; arc=none smtp.client-ip=209.85.210.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="Mfu/9XEN" Received: by mail-pf1-f202.google.com with SMTP id d2e1a72fcca58-71df2163e82so2112593b3a.3 for ; Thu, 10 Oct 2024 19:10:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1728612656; x=1729217456; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=ggNrOR87h1WmliSveAkfM74VH4GfRIzogWsDRg6oVL4=; b=Mfu/9XENfvETfwSYCSzzSYiAJOrjTBvoKVhA2YgsQMp3zeXR2zrapgg75O/asbhoMw yk41BfS5/OJLohYmR9HXkAMPITZrfHPexvjms9P4IYdUS+KmGjK4SqMhuA9Xtnyk7W1L 75o8tTvsqLjtUSqZrO92cF2zZRqhWaECOd+WMhmtnC0DCClmormjnRRcKbci0Vd8MmDE 8evvv0cUQuW8OR8JQTxjXpUFfrrpmtQdY8D40qn4EZ7k/kLgloKlRNZchx/V91549EPO dnEDezf6R9mekA/YNc6SkOJNINxulU72Gux4VJRB6QvdzeqFgRCMh1jOWUDsGlDazOba 7RBw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1728612656; x=1729217456; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=ggNrOR87h1WmliSveAkfM74VH4GfRIzogWsDRg6oVL4=; b=i9CQGxSYGRndRkMAvmdRCAY3id29XJ34ahpkjw9FDh8H4bQPiLolmsn6B7OJXX51oh uPiyS9lz94nNz5KQkUSTEv493VzBzo0rgalV5/u79uQvVOx+LK9UlMaENb5je79Aqcfw dbqAi9Q9ENuctC05KlhhANygzkKCFBMszlwat/FGRDJPVmoSA6Yx9ltGsbCxbBRFuRAh scfN1Ehgg1cR89GiM08/5I2nGtfSi48Uwls4OpppsnSoEC2e5q9zH/fIXrzEW8SIUGgV 4YWlhxqUS8KVEA8+98SjA7LQsx/bWse/6YOKXAoJSRXAA6SFYXchtJJtlcyLpJwzDukj amaw== X-Forwarded-Encrypted: i=1; AJvYcCXthqvhbU+uTha1IqmL6DxnbrfjUsZS/3Gfwlhz1VENBWSNqFV4QBIoEGwIoWRb1SZIbRpTNm+5v4uHjNg=@vger.kernel.org X-Gm-Message-State: AOJu0Yx1YbSarMhzKd7TadZAq5NxQEFJj4dcGOgDoTzBeqFGdjOhHPiO cwm9oIT9FzDdJmPmfZ2GpviwtIJuvCfo6bi4kDC1z0rA5Z8UB3P1fmQsdGoXebl6IT+f3DR8PoQ 5ew== X-Google-Smtp-Source: AGHT+IHiJfmTGlfIJvaSgpYOtjfY/pLkr5LWn0PLu48c12kbgFTvRCTH+q9OJPqtDUvnnwwLgAkH7OMuic4= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:9d:3983:ac13:c240]) (user=seanjc job=sendgmr) by 2002:a05:6a00:91c7:b0:71e:38c7:eb9 with SMTP id d2e1a72fcca58-71e38c71335mr9078b3a.1.1728612656058; Thu, 10 Oct 2024 19:10:56 -0700 (PDT) Reply-To: Sean Christopherson Date: Thu, 10 Oct 2024 19:10:34 -0700 In-Reply-To: <20241011021051.1557902-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20241011021051.1557902-1-seanjc@google.com> X-Mailer: git-send-email 2.47.0.rc1.288.g06298d1525-goog Message-ID: <20241011021051.1557902-3-seanjc@google.com> Subject: [PATCH 02/18] KVM: x86/mmu: Always set SPTE's dirty bit if it's created as writable From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Yan Zhao , Sagi Shahar , "=?UTF-8?q?Alex=20Benn=C3=A9e?=" , David Matlack , James Houghton Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" When creating a SPTE, always set the Dirty bit if the Writable bit is set, i.e. if KVM is creating a writable mapping. If two (or more) vCPUs are racing to install a writable SPTE on a !PRESENT fault, only the "winning" vCPU will create a SPTE with W=3D1 and D=3D1, all "losers" will generate a SPTE with W=3D1 && D=3D0. As a result, tdp_mmu_map_handle_target_level() will fail to detect that the losing faults are effectively spurious, and will overwrite the D=3D1 SPTE with a D=3D0 SPTE. For normal VMs, overwriting a present SPTE is a small performance blip; KVM blasts a remote TLB flush, but otherwise life goes on. For upcoming TDX VMs, overwriting a present SPTE is much more costly, and can even lead to the VM being terminated if KVM isn't careful, e.g. if KVM attempts TDH.MEM.PAGE.AUG because the TDX code doesn't detect that the new SPTE is actually the same as the old SPTE (which would be a bug in its own right). Suggested-by: Sagi Shahar Cc: Yan Zhao Signed-off-by: Sean Christopherson --- arch/x86/kvm/mmu/spte.c | 28 +++++++++------------------- 1 file changed, 9 insertions(+), 19 deletions(-) diff --git a/arch/x86/kvm/mmu/spte.c b/arch/x86/kvm/mmu/spte.c index e5af69a8f101..09ce93c4916a 100644 --- a/arch/x86/kvm/mmu/spte.c +++ b/arch/x86/kvm/mmu/spte.c @@ -219,30 +219,21 @@ bool make_spte(struct kvm_vcpu *vcpu, struct kvm_mmu_= page *sp, if (pte_access & ACC_WRITE_MASK) { spte |=3D PT_WRITABLE_MASK | shadow_mmu_writable_mask; =20 - /* - * When overwriting an existing leaf SPTE, and the old SPTE was - * writable, skip trying to unsync shadow pages as any relevant - * shadow pages must already be unsync, i.e. the hash lookup is - * unnecessary (and expensive). - * - * The same reasoning applies to dirty page/folio accounting; - * KVM marked the folio dirty when the old SPTE was created, - * thus there's no need to mark the folio dirty again. - * - * Note, both cases rely on KVM not changing PFNs without first - * zapping the old SPTE, which is guaranteed by both the shadow - * MMU and the TDP MMU. - */ - if (is_last_spte(old_spte, level) && is_writable_pte(old_spte)) - goto out; - /* * Unsync shadow pages that are reachable by the new, writable * SPTE. Write-protect the SPTE if the page can't be unsync'd, * e.g. it's write-tracked (upper-level SPs) or has one or more * shadow pages and unsync'ing pages is not allowed. + * + * When overwriting an existing leaf SPTE, and the old SPTE was + * writable, skip trying to unsync shadow pages as any relevant + * shadow pages must already be unsync, i.e. the hash lookup is + * unnecessary (and expensive). Note, this relies on KVM not + * changing PFNs without first zapping the old SPTE, which is + * guaranteed by both the shadow MMU and the TDP MMU. */ - if (mmu_try_to_unsync_pages(vcpu->kvm, slot, gfn, synchronizing, prefetc= h)) { + if ((!is_last_spte(old_spte, level) || !is_writable_pte(old_spte)) && + mmu_try_to_unsync_pages(vcpu->kvm, slot, gfn, synchronizing, prefetc= h)) { wrprot =3D true; pte_access &=3D ~ACC_WRITE_MASK; spte &=3D ~(PT_WRITABLE_MASK | shadow_mmu_writable_mask); @@ -252,7 +243,6 @@ bool make_spte(struct kvm_vcpu *vcpu, struct kvm_mmu_pa= ge *sp, if (pte_access & ACC_WRITE_MASK) spte |=3D spte_shadow_dirty_mask(spte); =20 -out: if (prefetch && !synchronizing) spte =3D mark_spte_for_access_track(spte); =20 --=20 2.47.0.rc1.288.g06298d1525-goog From nobody Tue Nov 26 08:38:40 2024 Received: from mail-pg1-f202.google.com (mail-pg1-f202.google.com [209.85.215.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E4C2B1F4716 for ; Fri, 11 Oct 2024 02:10:59 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728612661; cv=none; b=Ua1RtzA5yBOwZTNqu1uvgBacSRn8S0h4V+N3WEo2GzN5AdSyLyPmHEwdiJjFyCi6r9QzXCRb+Ng+oEhF6PZZW6pBGbgV0D5hG+0ntFv65W8PRGDu5YbtDfeMBl91uXpPY9YfWVVO8m68wFv1y5u7I199zDk2xpL14g3QGsnHDoM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728612661; c=relaxed/simple; bh=RqYw1AtTmNIpKfrRVZilkRfxBV1RzGHNLXu7fiax26k=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=keTKeShUKAGPq1JYH5n4kxpjFtIaHqA+hv0Sok8nsFlwK2C1JMTV/52uyXigLo9xA+vqv1P0QoU0/UxE9QjynfjKs9TEXxZELpE/oJsbvNL7DuGYDqXAYbUmEn9lmWeQB41Kupt+Q1ZYhixcJ1LBpNp45oWKVXp7FopC5nmt3sA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=FAPaQCoM; arc=none smtp.client-ip=209.85.215.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="FAPaQCoM" Received: by mail-pg1-f202.google.com with SMTP id 41be03b00d2f7-7b696999c65so1125149a12.3 for ; Thu, 10 Oct 2024 19:10:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1728612659; x=1729217459; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=1cFPbD2dKRB4Ynjmw+1iKOno60r0l4Z0l+jOG9EoBK0=; b=FAPaQCoMvl+bXRlr7tk1gXpgMK1NvZajqyA2R6CyWU1/qdgk5wQVUX+c+eq7cHY4TV GYff4dFBwyiBBEGjHir/495APPWYcd104lvlWJ2FpnkKAhHaVXNz7YhsgRDTOW/eHSef IwdVc2wXrXZyDDuNUBxU8ek0+vj91by3sOA0qzhg1Wp5YcTl3kNs7hRYhqWfzrZcFPnd 40MJ3FQbAMqknYGXCRvSo4IZPDv474MnKAL0wQStrZ8GUgAqGT2ZvAy5fnHfQh0rji+T 0Oa73vYK2PcoqjcU6TzTVAPZfmMxBmLwDPauckIZCtd+xsVLT2oUcX/zjyzCPAFxTzvu e7/Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1728612659; x=1729217459; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=1cFPbD2dKRB4Ynjmw+1iKOno60r0l4Z0l+jOG9EoBK0=; b=TpmtjqtlA62OX1vR2fuCW0xsrkurrAHr40bDt6401htoO6WNHo4RV1WoJDS2ejv1GR vKkFA4OGR9qTPT4S8uxxeQ59g/GUF6g8JY8LHzDbnXFQsWewe/3UCLoAwlRDifHRq/jI SWPdsH+vYHcp3SuYnU9xtNnpm+kh4vvBDM2tNKAxMgVAtljKT5UflihevpFwPZIslVpC cmwkFTmwVkub8U7k0qeLgnKLAFWXbQmNY1ZtVQdCgLiIvtS8SIMuvSftQbv08AMpU5MW WwzBXZafml93c/o0JnmYZJyHeUm5jU3ymS/MuN53zhLIdNgklV0XiMsc0a2XGKvwb72D kC0w== X-Forwarded-Encrypted: i=1; AJvYcCUsSnH5MjDSKw19T6OvKUXoNdiE3rTabJVJ5uFlzIgBj+jwQVyAFxLXeFQQ4SrcdWxAVwmTqIu+iok+AOQ=@vger.kernel.org X-Gm-Message-State: AOJu0Yxnb+KWuiF1Y8fSGqngTVjP00h9eyY/aREkgpwO2LiA75KQo+30 E28znPyy0rPaL3oj4RzH/FCFiYH6SPER67954ycMX8mFKth1IUXB3XjlQ7Cas/U2ji5/X5+jd/E OeQ== X-Google-Smtp-Source: AGHT+IGX2CXRtmLd2XIO3ano1VLdg5QbiDrG6U3QGMBhMcsoaLOgnVSvhPE6pecTJ5VW0JuXmb1bfxqe1OQ= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:9d:3983:ac13:c240]) (user=seanjc job=sendgmr) by 2002:a65:6896:0:b0:7db:18fc:3068 with SMTP id 41be03b00d2f7-7ea535903c0mr782a12.5.1728612657867; Thu, 10 Oct 2024 19:10:57 -0700 (PDT) Reply-To: Sean Christopherson Date: Thu, 10 Oct 2024 19:10:35 -0700 In-Reply-To: <20241011021051.1557902-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20241011021051.1557902-1-seanjc@google.com> X-Mailer: git-send-email 2.47.0.rc1.288.g06298d1525-goog Message-ID: <20241011021051.1557902-4-seanjc@google.com> Subject: [PATCH 03/18] KVM: x86/mmu: Fold all of make_spte()'s writable handling into one if-else From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Yan Zhao , Sagi Shahar , "=?UTF-8?q?Alex=20Benn=C3=A9e?=" , David Matlack , James Houghton Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Now that make_spte() no longer uses a funky goto to bail out for a special case of its unsync handling, combine all of the unsync vs. writable logic into a single if-else statement. No functional change intended. Signed-off-by: Sean Christopherson --- arch/x86/kvm/mmu/spte.c | 13 ++++--------- 1 file changed, 4 insertions(+), 9 deletions(-) diff --git a/arch/x86/kvm/mmu/spte.c b/arch/x86/kvm/mmu/spte.c index 09ce93c4916a..030813781a63 100644 --- a/arch/x86/kvm/mmu/spte.c +++ b/arch/x86/kvm/mmu/spte.c @@ -217,8 +217,6 @@ bool make_spte(struct kvm_vcpu *vcpu, struct kvm_mmu_pa= ge *sp, spte |=3D (u64)pfn << PAGE_SHIFT; =20 if (pte_access & ACC_WRITE_MASK) { - spte |=3D PT_WRITABLE_MASK | shadow_mmu_writable_mask; - /* * Unsync shadow pages that are reachable by the new, writable * SPTE. Write-protect the SPTE if the page can't be unsync'd, @@ -233,16 +231,13 @@ bool make_spte(struct kvm_vcpu *vcpu, struct kvm_mmu_= page *sp, * guaranteed by both the shadow MMU and the TDP MMU. */ if ((!is_last_spte(old_spte, level) || !is_writable_pte(old_spte)) && - mmu_try_to_unsync_pages(vcpu->kvm, slot, gfn, synchronizing, prefetc= h)) { + mmu_try_to_unsync_pages(vcpu->kvm, slot, gfn, synchronizing, prefetc= h)) wrprot =3D true; - pte_access &=3D ~ACC_WRITE_MASK; - spte &=3D ~(PT_WRITABLE_MASK | shadow_mmu_writable_mask); - } + else + spte |=3D PT_WRITABLE_MASK | shadow_mmu_writable_mask | + spte_shadow_dirty_mask(spte); } =20 - if (pte_access & ACC_WRITE_MASK) - spte |=3D spte_shadow_dirty_mask(spte); - if (prefetch && !synchronizing) spte =3D mark_spte_for_access_track(spte); =20 --=20 2.47.0.rc1.288.g06298d1525-goog From nobody Tue Nov 26 08:38:40 2024 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DA27D1F4FA3 for ; Fri, 11 Oct 2024 02:11:01 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728612664; cv=none; b=AxdQazySapfHLfDR4MIxc8UjHeAjmH4fBpzSzebAYzm6DZgQ17R4O6CFiBLdnSqDsvk67TtzvCw+kY14S2hDVUP7SG7cFrrK1c6P6WKJAi4F11fjexZ7f0ybxeYr79smJ0Y7eh/pP3qF1Ie+9WtnvcmLMMB8ywZZRSzwX/9LzMk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728612664; c=relaxed/simple; bh=E3IXDKgcyUCE+TmbZTTwZzg1L4bUxpva3WFh9qzTdFk=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=GcZuNz5DUIIWvcYlY38F7a4SN43NjOXsaZjcEYXKeqsCXx15WNyZKybn1sAvjMU4yISdlHDymuxTpG9zqR3Pe5W/TGAm+o8o9jl6hmv7aR+Kxm4V9BGsvtzAPCwBOGGAOASSRfLrs8A+Kr2ZMSVbuDrmkA5CElaMyN1coP6oRJw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=p5mJoFqi; arc=none smtp.client-ip=209.85.128.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="p5mJoFqi" Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-6e2acec0109so28042307b3.3 for ; Thu, 10 Oct 2024 19:11:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1728612661; x=1729217461; darn=vger.kernel.org; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:reply-to:from:to:cc:subject:date :message-id:reply-to; bh=vfRd4YGDidyx6Q8cdQBlESrYYtQMzsNuRjsxO0/o42k=; b=p5mJoFqiWrKPNXn3tzposfZ/LwMyFOC4Qi/F6zOxZZ+WzeSw961jBZGR0nNjeoUeAd wIutWcEuNHESvCc90p17R8Jmlizw37Vhb+OCnYJ7eDweqT0DMbvZ938z5xWzgTYHaXx4 UNlwodEOof06CzkhBv2lWbN+KJ/SwjvTP6B9/NOElMthBbV8TgA4G5Z8SLEQP93/BJLs AFkkICKF+uGb7ILyteK9uzucetMrRF6M+G0B6ka9RJAJLLvSazSq3cWuWAqw0iWnj7bk 9SgeLLPICHGUtXrSSfy109fBTNPZtVrXwjKSwxlbLXY7PiPhg//1ONlaP4uuXP7n/iiL SqEQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1728612661; x=1729217461; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:reply-to:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to; bh=vfRd4YGDidyx6Q8cdQBlESrYYtQMzsNuRjsxO0/o42k=; b=hNvz3aKwLOODwOpY5ZDzxcJjlxyRMZQRxwPPMJKQwvG7kT+bUKEozGbAMF3QTZ3QRp YSi+xPijLtPteRZrKlaCxyMqff5bfIc+0RPD4G8fymQqB4okytX+MaokJFn3JHCGm8tS eXc1kxNahJm8cNdkCSf+YqZgk02pBJYxsqkLFGyqV19LfApMKarnH7/gZxSOQiYb4Zow AQxMhCcPzVTcjNhHxhYuKJWEKvhM8ZGXn+/MTtOVTfE5T6YOVm8BRzgfezF7hBkihBbI qYtLlJdF3mgseEIcIq454644DxEW3CcppbF/+q6gQO7j2QxBBwIG05LA7VBrU9PJsGij cW3A== X-Forwarded-Encrypted: i=1; AJvYcCXYFrFxhH7ldus3yQ6zWmT+A8TY0ilBudOCDVwfzE/TCU1nouPvZ+zXChXzevmrj3Nd0X/tuppHkdxuF3s=@vger.kernel.org X-Gm-Message-State: AOJu0YyTonXFDIP6yUgF5vod+Sqx1Wd0W7s9ax+ipw0hdCJzBWq2xcC0 Polye0Q2rQedfKy44KP4RxlLBPJZ2yYbOoeF/a914BMfuFKiQkIUPF5maJP4PpXE1ykwlMxHEcK WBw== X-Google-Smtp-Source: AGHT+IGmrtxjHtrBMHGZEUy3JQ2tvJIUU8sN8vV3zTrSpyXnDiKaHwy98K8xzQupsJi3j4s85TEUOMvXyAM= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:9d:3983:ac13:c240]) (user=seanjc job=sendgmr) by 2002:a05:690c:4384:b0:6e3:15a8:b26b with SMTP id 00721157ae682-6e347c70e18mr57427b3.8.1728612660902; Thu, 10 Oct 2024 19:11:00 -0700 (PDT) Reply-To: Sean Christopherson Date: Thu, 10 Oct 2024 19:10:36 -0700 In-Reply-To: <20241011021051.1557902-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20241011021051.1557902-1-seanjc@google.com> X-Mailer: git-send-email 2.47.0.rc1.288.g06298d1525-goog Message-ID: <20241011021051.1557902-5-seanjc@google.com> Subject: [PATCH 04/18] KVM: x86/mmu: Don't force flush if SPTE update clears Accessed bit From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Yan Zhao , Sagi Shahar , "=?UTF-8?q?Alex=20Benn=C3=A9e?=" , David Matlack , James Houghton Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Don't force a TLB flush if mmu_spte_update() clears the Accessed bit, as access tracking tolerates false negatives, as evidenced by the mmu_notifier hooks that explicitly test and age SPTEs without doing a TLB flush. In practice, this is very nearly a nop. spte_write_protect() and spte_clear_dirty() never clear the Accessed bit. make_spte() always sets the Accessed bit for !prefetch scenarios. FNAME(sync_spte) only sets SPTE if the protection bits are changing, i.e. if a flush will be needed regardless of the Accessed bits. And FNAME(pte_prefetch) sets SPTE if and only if the old SPTE is !PRESENT. That leaves kvm_arch_async_page_ready() as the one path that will generate a !ACCESSED SPTE *and* overwrite a PRESENT SPTE. And that's very arguably a bug, as clobbering a valid SPTE in that case is nonsensical. Tested-by: Alex Benn=C3=A9e Signed-off-by: Sean Christopherson --- arch/x86/kvm/mmu/mmu.c | 30 +++++++++--------------------- 1 file changed, 9 insertions(+), 21 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index 176fc37540df..9ccfe7eba9b4 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -521,36 +521,24 @@ static u64 mmu_spte_update_no_track(u64 *sptep, u64 n= ew_spte) * not whether or not SPTEs were modified, i.e. only the write-tracking ca= se * needs to flush at the time the SPTEs is modified, before dropping mmu_l= ock. * + * Remote TLBs also need to be flushed if the Dirty bit is cleared, as fal= se + * negatives are not acceptable, e.g. if KVM is using D-bit based PML on V= MX. + * + * Don't flush if the Accessed bit is cleared, as access tracking tolerates + * false negatives, and the one path that does care about TLB flushes, + * kvm_mmu_notifier_clear_flush_young(), uses mmu_spte_update_no_track(). + * * Returns true if the TLB needs to be flushed */ static bool mmu_spte_update(u64 *sptep, u64 new_spte) { - bool flush =3D false; u64 old_spte =3D mmu_spte_update_no_track(sptep, new_spte); =20 if (!is_shadow_present_pte(old_spte)) return false; =20 - /* - * For the spte updated out of mmu-lock is safe, since - * we always atomically update it, see the comments in - * spte_has_volatile_bits(). - */ - if (is_mmu_writable_spte(old_spte) && !is_mmu_writable_spte(new_spte)) - flush =3D true; - - /* - * Flush TLB when accessed/dirty states are changed in the page tables, - * to guarantee consistency between TLB and page tables. - */ - - if (is_accessed_spte(old_spte) && !is_accessed_spte(new_spte)) - flush =3D true; - - if (is_dirty_spte(old_spte) && !is_dirty_spte(new_spte)) - flush =3D true; - - return flush; + return (is_mmu_writable_spte(old_spte) && !is_mmu_writable_spte(new_spte)= ) || + (is_dirty_spte(old_spte) && !is_dirty_spte(new_spte)); } =20 /* --=20 2.47.0.rc1.288.g06298d1525-goog From nobody Tue Nov 26 08:38:40 2024 Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0828B1F4FCF for ; Fri, 11 Oct 2024 02:11:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728612665; cv=none; b=u6oW6aQSPDxsMI2056asxG0KIQjz5uVj5nDrO4bi79W9mQWhw67y92QpeV8eePVLim5/HK6lCs3yW5zhYAShJdz5YDYkH4mGPXcazvOt7nhlAtgfxqBCCiv3JOaf3BFXyddlBq1FYZ8H8xjkg/Ojc0IwB1uf/EiJGWqTOhqaU30= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728612665; c=relaxed/simple; bh=gog1CF3yrC7PbB7iQsGgdqYnwMXPBD/mMhmHGxpuVZg=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=nU1OIm82N0ht2MC0LCRQ73a7epFdPW+/oyrz/M6Q8kIf6m/f2QBiRhgjX8Eln9CVFedBaBxPYBRgHe0BVnHGTjPCTcVn/jkINlqyVhpNZxDQjM1SOuSV3Jb4+X4t8d9HB39FDBQKmxZLB02zz5pN+R4pFMH2+yiQFw6pUBI5J7c= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=ONC6Hy57; arc=none smtp.client-ip=209.85.219.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="ONC6Hy57" Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-e2605ce4276so2789354276.3 for ; Thu, 10 Oct 2024 19:11:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1728612663; x=1729217463; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=v8x9KibxTSkwYhpZCRmKtWyUdw0jfGm7zyAVE4Sh+6c=; b=ONC6Hy57QbsV11qpRLx0KpAX8bqfdWn1XfRhLKP0PqnYUZH7ZkyUGcN3xUn4tTAGOq /SR1wFLUBRT7ouJzRBdygu5JrWLhnY6aWLqAYpF6eIWFAoLurUYj0brEH5GrqKQw3pFu xVmGOFarBrF1jTkgriM9izIhkTiFDvCbRSbgVQzkh/aV2CIpf3bg/fdgmDwG7BbwDeEd mkVK0GhhTH37AfQEzqwKMYqB3dQl0dMqzrFv8pOBNSefMBC9aBR29a06kkeA9tijeZaN TL6UmTBi6GMbZunzBKj8sM8ypp+3sqd0yVpfdvB/yw3SULQQXzgTF2XP+G2RF6uPf+Zb xYDA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1728612663; x=1729217463; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=v8x9KibxTSkwYhpZCRmKtWyUdw0jfGm7zyAVE4Sh+6c=; b=YmLM5/RAKwlnJeuJcKbkH1LbDfMMHZgG4DBD4Xcc9d4EBjTFfti6q07EjRwZxJlhdA pgWExGl6XVy1dGyJAgPP4ixMDEsgjGAXva5zOjCbbXr+LQiGbjfDSdKUqq6tW7uHhbfH FhAv4Wk4bYb3T1dCLsdC/H+vEQKA4xmDyDyVtn0HCSXv8QcGrUfw7xyFv1jkAmS3eWp2 6yqg1dGIMvrB5mWqHNmRRVtU0GDvb1AWNXBbRGw567rQ5+7h84kPRALargL+zni/vDE7 Pqn+/c7EfIMNE8FxE7rsEttbEkavD76kU+RKQOTAlH7h6ChmPKtBWe6b1+DQ8cQy//tX Rx2g== X-Forwarded-Encrypted: i=1; AJvYcCUDP4VbGKbbJ5AYtoDknOW26fj82nZ1wbVEAtlxVR9bZBZQ14trS0WctlO120U7gQGg8m2t/lq9yRSdEk0=@vger.kernel.org X-Gm-Message-State: AOJu0Yx7eBbamShLne3oMk+JC5DbMzLlkddNvQPurBxYDspGKYqWetdj SQVY+/L5Cacq77Swet7IBBoKPPvwq83tGFyxufuIk3xH667MwmS/Xt7JVnVB0Y4riHYVrVAirgc MEw== X-Google-Smtp-Source: AGHT+IHodHjTZ5CDMtglbtHtbXCkaFSofoERPYUDk+m+jM/azzdFkEPj9rcESeONRYLUbkRsBbP0KylUz8o= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:9d:3983:ac13:c240]) (user=seanjc job=sendgmr) by 2002:a25:abcf:0:b0:e28:8f00:896a with SMTP id 3f1490d57ef6-e291a3133e4mr739276.8.1728612663019; Thu, 10 Oct 2024 19:11:03 -0700 (PDT) Reply-To: Sean Christopherson Date: Thu, 10 Oct 2024 19:10:37 -0700 In-Reply-To: <20241011021051.1557902-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20241011021051.1557902-1-seanjc@google.com> X-Mailer: git-send-email 2.47.0.rc1.288.g06298d1525-goog Message-ID: <20241011021051.1557902-6-seanjc@google.com> Subject: [PATCH 05/18] KVM: x86/mmu: Don't flush TLBs when clearing Dirty bit in shadow MMU From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Yan Zhao , Sagi Shahar , "=?UTF-8?q?Alex=20Benn=C3=A9e?=" , David Matlack , James Houghton Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Don't force a TLB flush when an SPTE update in the shadow MMU happens to clear the Dirty bit, as KVM unconditionally flushes TLBs when enabling dirty logging, and when clearing dirty logs, KVM flushes based on its software structures, not the SPTEs. I.e. the flows that care about accurate Dirty bit information already ensure there are no stale TLB entries. Opportunistically drop is_dirty_spte() as mmu_spte_update() was the sole caller. Signed-off-by: Sean Christopherson --- arch/x86/kvm/mmu/mmu.c | 14 ++++++++------ arch/x86/kvm/mmu/spte.h | 7 ------- 2 files changed, 8 insertions(+), 13 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index 9ccfe7eba9b4..faa524d5a0e8 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -521,12 +521,15 @@ static u64 mmu_spte_update_no_track(u64 *sptep, u64 n= ew_spte) * not whether or not SPTEs were modified, i.e. only the write-tracking ca= se * needs to flush at the time the SPTEs is modified, before dropping mmu_l= ock. * - * Remote TLBs also need to be flushed if the Dirty bit is cleared, as fal= se - * negatives are not acceptable, e.g. if KVM is using D-bit based PML on V= MX. - * * Don't flush if the Accessed bit is cleared, as access tracking tolerates * false negatives, and the one path that does care about TLB flushes, - * kvm_mmu_notifier_clear_flush_young(), uses mmu_spte_update_no_track(). + * kvm_mmu_notifier_clear_flush_young(), flushes if a young SPTE is found,= i.e. + * doesn't rely on lower helpers to detect the need to flush. + * + * Lastly, don't flush if the Dirty bit is cleared, as KVM unconditionally + * flushes when enabling dirty logging (see kvm_mmu_slot_apply_flags()), a= nd + * when clearing dirty logs, KVM flushes based on whether or not dirty ent= ries + * were reaped from the bitmap/ring, not whether or not dirty SPTEs were f= ound. * * Returns true if the TLB needs to be flushed */ @@ -537,8 +540,7 @@ static bool mmu_spte_update(u64 *sptep, u64 new_spte) if (!is_shadow_present_pte(old_spte)) return false; =20 - return (is_mmu_writable_spte(old_spte) && !is_mmu_writable_spte(new_spte)= ) || - (is_dirty_spte(old_spte) && !is_dirty_spte(new_spte)); + return is_mmu_writable_spte(old_spte) && !is_mmu_writable_spte(new_spte); } =20 /* diff --git a/arch/x86/kvm/mmu/spte.h b/arch/x86/kvm/mmu/spte.h index c81cac9358e0..574ca9a1fcab 100644 --- a/arch/x86/kvm/mmu/spte.h +++ b/arch/x86/kvm/mmu/spte.h @@ -363,13 +363,6 @@ static inline bool is_accessed_spte(u64 spte) : !is_access_track_spte(spte); } =20 -static inline bool is_dirty_spte(u64 spte) -{ - u64 dirty_mask =3D spte_shadow_dirty_mask(spte); - - return dirty_mask ? spte & dirty_mask : spte & PT_WRITABLE_MASK; -} - static inline u64 get_rsvd_bits(struct rsvd_bits_validate *rsvd_check, u64= pte, int level) { --=20 2.47.0.rc1.288.g06298d1525-goog From nobody Tue Nov 26 08:38:40 2024 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D553E1F706A for ; Fri, 11 Oct 2024 02:11:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728612667; cv=none; b=kf03/m701J21kkGokUF8eRyiXTcGzMLkjATPkBPSbZUKEV53NTAQ+XCB9iuU1uNhD2im4DpHlgaovITwCcsjq/hNTKlrKjVT2/d4ig9yoC4DGsTEtwby7Xc1mJNuke71XmJVofUqdGv3+wQ1oXFmJGlWis/nfbmE0+UM4G7QOQA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728612667; c=relaxed/simple; bh=6ySxcwhwrHJlZ4I/ATeASq5WDemLJ6Yi/XYtpmqlx9I=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=rKOv6ig0weVbpaULn3gLc7pXMyN3/GXpgxZzNsVNBithTRaK/yMn9yu74Rbh0bUww+rc/rtdtdwhDuoRGmZ6a6WgOHyuEAPsNeEtx/ant+I4T4MJ6X9hbTMulX1cCsNfpRVEL3cPzYcp2Ydj6ggwduz2W9psM8BNIUWZPnBL364= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=Qdz8Pfor; arc=none smtp.client-ip=209.85.128.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="Qdz8Pfor" Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-6e3497c8eb0so3111907b3.0 for ; Thu, 10 Oct 2024 19:11:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1728612665; x=1729217465; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=J6fJWudgoyhjNeokLMIpfJOGPiEm4NW5MKaOgNe3m0c=; b=Qdz8PforVMb9xifm6HVypigyoGtQkLdh8cmmpSJq/4lfTLbHDzO/EM0BzRV6d/uXaP 7fOuj0sXvznIu7E36BFljQ5hZmve8tGVcRyZ1ot+YpGrc+vYIWJOfiQ3RX256/oklpF4 t4QrvRqvOGr2/gFVefXFvHDFGtMw7gDrcloEU3rLlHp6W+Wsygnb4zZct9IcQU8hi+Pz H2zTMj6XX7HqHfZWR1W1Rn/Tf1Rw5qy3cJFKh2R0I9Wy87ynC16YuXp9GeYkMsIsMxgo cF+AiNK17BnBcWGufdj6dI6r7ksgK72CPH5OkX7cQSZPN7cZ0ylAH79WALIpfbYvAHw9 DGRg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1728612665; x=1729217465; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=J6fJWudgoyhjNeokLMIpfJOGPiEm4NW5MKaOgNe3m0c=; b=bR+PVt+P0nnYxbjMhLcDhxi6gbKmPOmT3qKPghUydMqYD3pvLRez5kpyIVRnq/fVqY 6W5jS9DU48RBolS/hKY+X+MPdbVNeL9rm0OCmmPs7W2ilc5VxWPrUN/FUZ+jT2gBxfle Uu/85h9z1JeLgQhnBLUH61HukhLxBXold7tPD8NTOm9NVMVfqaeS0+VZrNJgEGUhsB4J A9cVAEJlqqaSxvHroh2aWx6uP3Lb5D+Rt8Y5JJk6A0Xxz2zDoCouIbHNj41P6aqsi/tY lHOsQViHn615ObYryw5evMA1l/zbEZlI/q4W6wc0a46AUH41FismceoARMQd6FrHnsdX YGpg== X-Forwarded-Encrypted: i=1; AJvYcCUyB1Kvd0FaWWgIONs0rYQpIodph8ghmEFcLtKm767zxMG3DQT5dem6lntut8GMGrnrflpAXO7V3BcjeUk=@vger.kernel.org X-Gm-Message-State: AOJu0YwjjuCdcWTILdQ47NZyqoEo8utxb51o0y/Iam87FfwNm50WL9gG t2vefnkZ8Qq0sllJqp1VOALZ25c4iDuTHgbA7e3o8VF4MLgkyFBh2gK/VGJRWeLw91SKfomEKC0 yFg== X-Google-Smtp-Source: AGHT+IFbXi1dCiXSidWjoezcs3LfAr39NDQ+HaYJkXB3Sn0xMqcrzqVDZ2BFbhCf6XdaWRZ36muZ7O1FhMs= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:9d:3983:ac13:c240]) (user=seanjc job=sendgmr) by 2002:a05:690c:4706:b0:6e2:a355:7b5c with SMTP id 00721157ae682-6e32f33bf7emr548257b3.5.1728612664905; Thu, 10 Oct 2024 19:11:04 -0700 (PDT) Reply-To: Sean Christopherson Date: Thu, 10 Oct 2024 19:10:38 -0700 In-Reply-To: <20241011021051.1557902-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20241011021051.1557902-1-seanjc@google.com> X-Mailer: git-send-email 2.47.0.rc1.288.g06298d1525-goog Message-ID: <20241011021051.1557902-7-seanjc@google.com> Subject: [PATCH 06/18] KVM: x86/mmu: Drop ignored return value from kvm_tdp_mmu_clear_dirty_slot() From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Yan Zhao , Sagi Shahar , "=?UTF-8?q?Alex=20Benn=C3=A9e?=" , David Matlack , James Houghton Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Drop the return value from kvm_tdp_mmu_clear_dirty_slot() as its sole caller ignores the result (KVM flushes after clearing dirty logs based on the logs themselves, not based on SPTEs). Cc: David Matlack Signed-off-by: Sean Christopherson --- arch/x86/kvm/mmu/tdp_mmu.c | 20 ++++++-------------- arch/x86/kvm/mmu/tdp_mmu.h | 2 +- 2 files changed, 7 insertions(+), 15 deletions(-) diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c index 91caa73a905b..9c66be7fb002 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.c +++ b/arch/x86/kvm/mmu/tdp_mmu.c @@ -1492,13 +1492,12 @@ static bool tdp_mmu_need_write_protect(struct kvm_m= mu_page *sp) return kvm_mmu_page_ad_need_write_protect(sp) || !kvm_ad_enabled(); } =20 -static bool clear_dirty_gfn_range(struct kvm *kvm, struct kvm_mmu_page *ro= ot, - gfn_t start, gfn_t end) +static void clear_dirty_gfn_range(struct kvm *kvm, struct kvm_mmu_page *ro= ot, + gfn_t start, gfn_t end) { const u64 dbit =3D tdp_mmu_need_write_protect(root) ? PT_WRITABLE_MASK : shadow_dirty_mask; struct tdp_iter iter; - bool spte_set =3D false; =20 rcu_read_lock(); =20 @@ -1519,31 +1518,24 @@ static bool clear_dirty_gfn_range(struct kvm *kvm, = struct kvm_mmu_page *root, =20 if (tdp_mmu_set_spte_atomic(kvm, &iter, iter.old_spte & ~dbit)) goto retry; - - spte_set =3D true; } =20 rcu_read_unlock(); - return spte_set; } =20 /* * Clear the dirty status (D-bit or W-bit) of all the SPTEs mapping GFNs i= n the - * memslot. Returns true if an SPTE has been changed and the TLBs need to = be - * flushed. + * memslot. */ -bool kvm_tdp_mmu_clear_dirty_slot(struct kvm *kvm, +void kvm_tdp_mmu_clear_dirty_slot(struct kvm *kvm, const struct kvm_memory_slot *slot) { struct kvm_mmu_page *root; - bool spte_set =3D false; =20 lockdep_assert_held_read(&kvm->mmu_lock); for_each_valid_tdp_mmu_root_yield_safe(kvm, root, slot->as_id) - spte_set |=3D clear_dirty_gfn_range(kvm, root, slot->base_gfn, - slot->base_gfn + slot->npages); - - return spte_set; + clear_dirty_gfn_range(kvm, root, slot->base_gfn, + slot->base_gfn + slot->npages); } =20 static void clear_dirty_pt_masked(struct kvm *kvm, struct kvm_mmu_page *ro= ot, diff --git a/arch/x86/kvm/mmu/tdp_mmu.h b/arch/x86/kvm/mmu/tdp_mmu.h index 1b74e058a81c..d842bfe103ab 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.h +++ b/arch/x86/kvm/mmu/tdp_mmu.h @@ -34,7 +34,7 @@ bool kvm_tdp_mmu_test_age_gfn(struct kvm *kvm, struct kvm= _gfn_range *range); =20 bool kvm_tdp_mmu_wrprot_slot(struct kvm *kvm, const struct kvm_memory_slot *slot, int min_level); -bool kvm_tdp_mmu_clear_dirty_slot(struct kvm *kvm, +void kvm_tdp_mmu_clear_dirty_slot(struct kvm *kvm, const struct kvm_memory_slot *slot); void kvm_tdp_mmu_clear_dirty_pt_masked(struct kvm *kvm, struct kvm_memory_slot *slot, --=20 2.47.0.rc1.288.g06298d1525-goog From nobody Tue Nov 26 08:38:40 2024 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E2D121F709D for ; Fri, 11 Oct 2024 02:11:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728612669; cv=none; b=lrI6PEICkAEnscv7U/l8epbrQoVX2W5vLTwH/6PGl9F5bewx0d2UW57e85vcmAvOCO77iIBk95p3wCgHWFjUhlajwsL1Y/iaIUATPE19G+GUy/vw9rrrgLeBtjTbaZU1oAUt9i138S1MLcCHx1kQBfocWZpHCO16a0TCECT6xT8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728612669; c=relaxed/simple; bh=7T0Y7z9DkiR+xsRM8ZPFp3xmx4pcPJcLvaBoxa7kVSw=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=Kw6BTG45CQDZQXw8EhjWSQhewVw5HtcbtOppYMTG5cNZQ6LyXjDWtZwk6ZCNG836rXujWMfahFOpsL76FocXPySb5I9FO2Ym2tkasEzxzM+iiUSDQ7IKLXhr1yYStmbTJ3S0jS3ulHlBKBUHau51PTJNlWM0I6sZlCbEwtZQy8k= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=r+DkhD1N; arc=none smtp.client-ip=209.85.219.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="r+DkhD1N" Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-e28fea2adb6so1905719276.3 for ; Thu, 10 Oct 2024 19:11:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1728612667; x=1729217467; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=3KbU21e26mgdFp4CthOOUh3jNyHnk8sJBVf29vXTz6Y=; b=r+DkhD1NPgjOLAefP8T58ISsLXSVejyB9xxByT5DmBSi3HmGISaYOt3XQd2m/FUDeb FLTS07c4HXbcIEbnocyjxtaHwvbpYrmy/gwQI++Si9pvcBkqeottU1XfvLUHoaxZp1nt 7LUsnftC7NSHS8WtxS+GeNF4qeAFaoISnECX5EW/UwcsIfEnTsMifMPPAatfHdL8yoyI YvYD1BULBgj6cGkpXlWtnr4XmYYZLhFji9i/31Cx4Ogy437n/4YbgZtUEvFHOYJExfdu bBa/R3NinaHi6DC2ohRVbPk/bktfMKj+o0JJtcDXAvi9PBuIlr/r5WDS1BohuFleTv0x 7u+A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1728612667; x=1729217467; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=3KbU21e26mgdFp4CthOOUh3jNyHnk8sJBVf29vXTz6Y=; b=mzjbFuZpHo4wmIBmb29QxVgkBMNtUmWM2qpLysMeeOO1F8EL6GOexeZWeTmKcADzAF oKrGT+tVx6jxmuYzwVuYDKbisHDcE10BGpbvbhY/vIk2o3o7ceKAR2g8e9HdIrJDb7y8 lOQAZtNE21qNcn97KyqILpHK/JKCV33MT1ebl5iopv9YyidRL4Gn9a/Xvq7cRchAboFl f/NBt3/RF5ONtrwb71dQyN2fbaWFI87KwCQC7pIMGYguMyc2k/lzIP56P1Pb1TKOcrq+ X8WGsx+VZxAOMPyhOEwEuMVvjBl8xnZGx/BjoDxFPG7m9zBohbPNt+aKf9uSGuDE4cCB 24gg== X-Forwarded-Encrypted: i=1; AJvYcCUsWn1iC47FTUVHlfXSgzkk1KSHHuYUizobirVNdce2qacMOqZifFpDXgyVVk378aCizxyCWCUqjudPVS0=@vger.kernel.org X-Gm-Message-State: AOJu0YzFTGrgXAQqBqiF2p4LiaFuAKRuwK/xT3toABv6uq6k9IfvMo1S ZfZhJUhW/sUGBAxC3VIzzlok69LMJmIUWgf952DP9Ph0niTf5a24dKuMMqE4Fpz3XaqNGQX5Sp5 P5g== X-Google-Smtp-Source: AGHT+IExWa1Vf1fg4dGiNvdf7Xh8WjS5MlhdfzJm0txAXEjRaSgTANl34PRIe8HLwi1sfR/J7GW5EZ4hKo0= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:9d:3983:ac13:c240]) (user=seanjc job=sendgmr) by 2002:a25:5f50:0:b0:e28:f2a5:f1d with SMTP id 3f1490d57ef6-e2919d9e584mr1371276.4.1728612666726; Thu, 10 Oct 2024 19:11:06 -0700 (PDT) Reply-To: Sean Christopherson Date: Thu, 10 Oct 2024 19:10:39 -0700 In-Reply-To: <20241011021051.1557902-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20241011021051.1557902-1-seanjc@google.com> X-Mailer: git-send-email 2.47.0.rc1.288.g06298d1525-goog Message-ID: <20241011021051.1557902-8-seanjc@google.com> Subject: [PATCH 07/18] KVM: x86/mmu: Fold mmu_spte_update_no_track() into mmu_spte_update() From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Yan Zhao , Sagi Shahar , "=?UTF-8?q?Alex=20Benn=C3=A9e?=" , David Matlack , James Houghton Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Fold the guts of mmu_spte_update_no_track() into mmu_spte_update() now that the latter doesn't flush when clearing A/D bits, i.e. now that there is no need to explicitly avoid TLB flushes when aging SPTEs. Opportunistically WARN if mmu_spte_update() requests a TLB flush when aging SPTEs, as aging should never modify a SPTE in such a way that KVM thinks a TLB flush is needed. Signed-off-by: Sean Christopherson --- arch/x86/kvm/mmu/mmu.c | 50 ++++++++++++++++++------------------------ 1 file changed, 21 insertions(+), 29 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index faa524d5a0e8..a72ecac63e07 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -485,32 +485,6 @@ static void mmu_spte_set(u64 *sptep, u64 new_spte) __set_spte(sptep, new_spte); } =20 -/* - * Update the SPTE (excluding the PFN), but do not track changes in its - * accessed/dirty status. - */ -static u64 mmu_spte_update_no_track(u64 *sptep, u64 new_spte) -{ - u64 old_spte =3D *sptep; - - WARN_ON_ONCE(!is_shadow_present_pte(new_spte)); - check_spte_writable_invariants(new_spte); - - if (!is_shadow_present_pte(old_spte)) { - mmu_spte_set(sptep, new_spte); - return old_spte; - } - - if (!spte_has_volatile_bits(old_spte)) - __update_clear_spte_fast(sptep, new_spte); - else - old_spte =3D __update_clear_spte_slow(sptep, new_spte); - - WARN_ON_ONCE(spte_to_pfn(old_spte) !=3D spte_to_pfn(new_spte)); - - return old_spte; -} - /* Rules for using mmu_spte_update: * Update the state bits, it means the mapped pfn is not changed. * @@ -535,10 +509,23 @@ static u64 mmu_spte_update_no_track(u64 *sptep, u64 n= ew_spte) */ static bool mmu_spte_update(u64 *sptep, u64 new_spte) { - u64 old_spte =3D mmu_spte_update_no_track(sptep, new_spte); + u64 old_spte =3D *sptep; =20 - if (!is_shadow_present_pte(old_spte)) + WARN_ON_ONCE(!is_shadow_present_pte(new_spte)); + check_spte_writable_invariants(new_spte); + + if (!is_shadow_present_pte(old_spte)) { + mmu_spte_set(sptep, new_spte); return false; + } + + if (!spte_has_volatile_bits(old_spte)) + __update_clear_spte_fast(sptep, new_spte); + else + old_spte =3D __update_clear_spte_slow(sptep, new_spte); + + WARN_ON_ONCE(!is_shadow_present_pte(old_spte) || + spte_to_pfn(old_spte) !=3D spte_to_pfn(new_spte)); =20 return is_mmu_writable_spte(old_spte) && !is_mmu_writable_spte(new_spte); } @@ -1587,8 +1574,13 @@ static bool kvm_rmap_age_gfn_range(struct kvm *kvm, clear_bit((ffs(shadow_accessed_mask) - 1), (unsigned long *)sptep); } else { + /* + * WARN if mmu_spte_update() signals the need + * for a TLB flush, as Access tracking a SPTE + * should never trigger an _immediate_ flush. + */ spte =3D mark_spte_for_access_track(spte); - mmu_spte_update_no_track(sptep, spte); + WARN_ON_ONCE(mmu_spte_update(sptep, spte)); } young =3D true; } --=20 2.47.0.rc1.288.g06298d1525-goog From nobody Tue Nov 26 08:38:40 2024 Received: from mail-pf1-f202.google.com (mail-pf1-f202.google.com [209.85.210.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E45F81EABAF for ; Fri, 11 Oct 2024 02:11:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728612671; cv=none; b=isNRS3QPFUt/0/Fu1ElNrB2YzOc8NqoUlVeGkUzE63p8cRpfcGmz5odylEazGhOT43uk26wlcFuTxlxOtfKpRvzdC0EaSko+XXEKz8ajtP52LLv0Safnv0fMC78P1mZwOmNswT1SEGxRYqSh34yCfJiiQSYpQ2i2ZjCSSiaMU1I= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728612671; c=relaxed/simple; bh=Oaro4dCEU4jcQGzk11Dklpup6VCv/ez4gfRevJ+nC8s=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=e9P2RtbWOcwKnRvhLAX1jfX/ccT7bK37sSiDxpjBOtQa8bxdTK3nKP3aaqjcTTroamRlip1jf9UJsMD8DU+AmBbWRvpkczhtZzoa22RAFuTBrYObx4rgXbxO2pJ9M+nPreHYMwj+4Dab8/wrajfpMrPO3qmIxK0eQKmpyAYfah4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=WB7M3S/m; arc=none smtp.client-ip=209.85.210.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="WB7M3S/m" Received: by mail-pf1-f202.google.com with SMTP id d2e1a72fcca58-71def7a1a8aso1937035b3a.3 for ; Thu, 10 Oct 2024 19:11:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1728612669; x=1729217469; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=/d560U3Z5EC8iutMB38TtGLr0WB17Rh2gPoL7ugF38Q=; b=WB7M3S/mFw5YlZ2Um1eaaGC1gfdCVmza8EOZtjW4VsDALD2c331x53GopaQkASzX5z ZKyUU9JcIDKLKQv/9YbjENe3F/9JQ/E/10cFwupxy+mKCb5VbgW4Su2fwpRXzvv91dRn xwtCnZAaAZOR/AgUtFmbobvQA7IF9yD3DtNOijOGIHDagKv2ZizEco31Pirsl7xfKdcB OQBvzxGxLOUTuKlMki5Cd5cgNjt8r5uy4qhRygl18QUxg30mwqoSF9JYTjfCYRSXE/yL 540SCxS0rgPBpMw8C58lPCJxaWTqAJEqzWGDvGy8sqDAvmrWOJeXtQzjYCvjIXK4LP4P Q6zA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1728612669; x=1729217469; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=/d560U3Z5EC8iutMB38TtGLr0WB17Rh2gPoL7ugF38Q=; b=o0bdYdWo5KpDhOtIPMT/Aa02VWbJ4rqirk6F9DrbrH8Ar079Y86NehTpxHx0FllwMU lry4Gu88scaJy5lnDPZTP0RT1VAKOUauOtEWVqq4aiagbUl3b7AepBW43dzMFXqwhMvj oRrKVDQCAqIYmYiRDRpnbxNoSP9wBei96UFP/o8nV5abb4mOVTLW8n/nj1bAD+hTVXod vDpOtaWuYXfwDG1pxbSHA4/cJxI8uoo2vezC0hDSQHy8jVPBhEYyTGy9SMJAYQQx5AIF DBj9hOPWzlNz38vM4QKwl4CIezrtgzYWtLZCV/K6NPMZwqUrFAKRLR90xnaUsEhe+kiU YjhA== X-Forwarded-Encrypted: i=1; AJvYcCXr1iwIoxw5Xk73AKtOepuuOrqgaqsti4lsflPpMEZqFDv2qjIpTSLqjD3ewMyE/l2u2Wf8p7sEUBwc2Iw=@vger.kernel.org X-Gm-Message-State: AOJu0YzJkSJBlfTmMEY7RWyUY2dLvvDGYKRZZS8O/t4+epSaAnzOCgz4 w3qZfrV+/kfajqjeVay30ECZjs2Z9zC0+rw7e25cfuOd4FXLF2XfsYm0q96UEsieYXRFHrWym03 6Uw== X-Google-Smtp-Source: AGHT+IEDwgWqwtaUpKGOtPV1SFkzcT7nOjDPYXAUgFqKAshQBSQwxTCbIPRBqsJGwoI5XT9ksv8whQ+VyDY= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:9d:3983:ac13:c240]) (user=seanjc job=sendgmr) by 2002:a05:6a00:9161:b0:71e:1efb:3239 with SMTP id d2e1a72fcca58-71e380f82a9mr11932b3a.5.1728612668662; Thu, 10 Oct 2024 19:11:08 -0700 (PDT) Reply-To: Sean Christopherson Date: Thu, 10 Oct 2024 19:10:40 -0700 In-Reply-To: <20241011021051.1557902-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20241011021051.1557902-1-seanjc@google.com> X-Mailer: git-send-email 2.47.0.rc1.288.g06298d1525-goog Message-ID: <20241011021051.1557902-9-seanjc@google.com> Subject: [PATCH 08/18] KVM: x86/mmu: WARN and flush if resolving a TDP MMU fault clears MMU-writable From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Yan Zhao , Sagi Shahar , "=?UTF-8?q?Alex=20Benn=C3=A9e?=" , David Matlack , James Houghton Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Do a remote TLB flush if installing a leaf SPTE overwrites an existing leaf SPTE (with the same target pfn, which is enforced by a BUG() in handle_changed_spte()) and clears the MMU-Writable bit. Since the TDP MMU passes ACC_ALL to make_spte(), i.e. always requests a Writable SPTE, the only scenario in which make_spte() should create a !MMU-Writable SPTE is if the gfn is write-tracked or if KVM is prefetching a SPTE. When write-protecting for write-tracking, KVM must hold mmu_lock for write, i.e. can't race with a vCPU faulting in the SPTE. And when prefetching a SPTE, the TDP MMU takes care to avoid clobbering a shadow-present SPTE, i.e. it should be impossible to replace a MMU-writable SPTE with a !MMU-writable SPTE when handling a TDP MMU fault. Cc: David Matlack Cc: Yan Zhao Signed-off-by: Sean Christopherson --- arch/x86/kvm/mmu/tdp_mmu.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c index 9c66be7fb002..bc9e2f50dc80 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.c +++ b/arch/x86/kvm/mmu/tdp_mmu.c @@ -1033,7 +1033,9 @@ static int tdp_mmu_map_handle_target_level(struct kvm= _vcpu *vcpu, else if (tdp_mmu_set_spte_atomic(vcpu->kvm, iter, new_spte)) return RET_PF_RETRY; else if (is_shadow_present_pte(iter->old_spte) && - !is_last_spte(iter->old_spte, iter->level)) + (!is_last_spte(iter->old_spte, iter->level) || + WARN_ON_ONCE(is_mmu_writable_spte(iter->old_spte) && + !is_mmu_writable_spte(new_spte)))) kvm_flush_remote_tlbs_gfn(vcpu->kvm, iter->gfn, iter->level); =20 /* --=20 2.47.0.rc1.288.g06298d1525-goog From nobody Tue Nov 26 08:38:40 2024 Received: from mail-pg1-f202.google.com (mail-pg1-f202.google.com [209.85.215.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D18281F8EF3 for ; Fri, 11 Oct 2024 02:11:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728612673; cv=none; b=YduD6BQ3YmDHhFmUxpIqH4Qq8pb3eNe7mU5zN2AzPO7UWBOzy1TljvDrwitSEcMUMf9qWhI3Awk3vFTGytryIL6Jdrk045GP2/0YY8WWvcX1+PCDTuh5sy0NVVDURm94VPDAlhAWCejT1ubBwBsNQWTEl+dW3DNIL9mvqaz9AFo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728612673; c=relaxed/simple; bh=mBOjmFsYBYRAZuezuPUiDcUngPx/Y9dILWOQs8AcaTY=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=Jv0iPpBtjtQgpcuKKIHC8HM0h+wtoTrf/lrJ+BHyV0eIQGHXdNbbTTl3Efg7604lthmM421YTaII2LFyb0NiwJIs7e7nCWtYmHeRVE/7VxnpLBPT9of2btjeBkCljvpYzbjld9TFaumOCiSCVd6w7TZwQK+Nv9GOAberWy0U7po= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=1hiLpfZW; arc=none smtp.client-ip=209.85.215.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="1hiLpfZW" Received: by mail-pg1-f202.google.com with SMTP id 41be03b00d2f7-70ac9630e3aso1543138a12.1 for ; Thu, 10 Oct 2024 19:11:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1728612671; x=1729217471; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=dqtxSv3YnUUR/sJVpJRtlACnljXlzFu2kYd05eqQTN4=; b=1hiLpfZWchrYrZv3Zqh2nO5Np+MsGhlrPAUpvQO7p+Q9Zt5F9ARI4o7KLlttTzNdYp +nLQsX3D/xI7iQH19R8VITI1WZlqA6V7hi4e0AbytoPBloNt0uHbOB4N+nWeKvRMAmbj s2hHAOKRydMyHlj5TPizDj9losOV6Wh8CQpdVRzZAZTDv5HFJwTa0dQTvZbid7yC/Ubr jkiRdURF52kKTSR13uChX1JnUJq3bXR0QjEWdRpKPvLhe+zcmvj0jYyw3Vh+hF2U7YGz NCbytslkmmo+8B6+YJSmb4OsA2dS0vJctnPjSOrEdjUbhDwzv/YWwpRwM5LDVUielvZR CS/g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1728612671; x=1729217471; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=dqtxSv3YnUUR/sJVpJRtlACnljXlzFu2kYd05eqQTN4=; b=d5C8FUXio1lcQ/SYIlrvh0urzgVeNw1bB+NDjArOXesbnFHLG/zPdJNVRamcN79pOX VSiN3sU9yzrp9YcWkQhnlUdHxHrBC3wfgyk0LxCi2/weBX7pXu3EP8FfxEmPT6woJBrc xQgM0I++Ypwjx1IUJfFS00zZJak3On/z6eFPaGHPXsH6ChHCE1jz5hVe0E5dWy20xjmW q/50wMwZEDGJp6uqlD2L7LoHjVl8utLafXelnRRTlYzVmKQOfAKY/6h7WyUB+aylCY8g d+AEVUmYr3xVMMXRFyTgxl15vp0+0yGON3THtvlOoUOqgvxz1uexmuGTalHQB96Oj3OL Kv3A== X-Forwarded-Encrypted: i=1; AJvYcCWw+19LEZZtCpReuTfi0oTsPxVMr0OwvDGE3Tv/s9nd9SZWhXYbrlMiwY5Q/7D4zdV7YSPq1NWAOK2vyak=@vger.kernel.org X-Gm-Message-State: AOJu0YyW1w5Ok4MeW26po3itllUS6U0hC34WOKw+iQ9ybXKA7kGXkeop KsI1x3/5+n91dIUHx3rq8Y7JtnGuVn6NtgQ58a5ynb/qSts5nmQbskjjJq2B8K09dfTXsVps/0r ToQ== X-Google-Smtp-Source: AGHT+IEmcasdM9fcYdzj9EuNpBTGwy71TY+RMspUAPymGIJ09fYw9nRXHwZTY0keRu/tRkUdRmkF481gS+Q= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:9d:3983:ac13:c240]) (user=seanjc job=sendgmr) by 2002:a65:6917:0:b0:7ea:e28:b57e with SMTP id 41be03b00d2f7-7ea535910c2mr902a12.8.1728612670912; Thu, 10 Oct 2024 19:11:10 -0700 (PDT) Reply-To: Sean Christopherson Date: Thu, 10 Oct 2024 19:10:41 -0700 In-Reply-To: <20241011021051.1557902-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20241011021051.1557902-1-seanjc@google.com> X-Mailer: git-send-email 2.47.0.rc1.288.g06298d1525-goog Message-ID: <20241011021051.1557902-10-seanjc@google.com> Subject: [PATCH 09/18] KVM: x86/mmu: Add a dedicated flag to track if A/D bits are globally enabled From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Yan Zhao , Sagi Shahar , "=?UTF-8?q?Alex=20Benn=C3=A9e?=" , David Matlack , James Houghton Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add a dedicated flag to track if KVM has enabled A/D bits at the module level, instead of inferring the state based on whether or not the MMU's shadow_accessed_mask is non-zero. This will allow defining and using shadow_accessed_mask even when A/D bits aren't used by hardware. Signed-off-by: Sean Christopherson --- arch/x86/kvm/mmu/mmu.c | 6 +++--- arch/x86/kvm/mmu/spte.c | 6 ++++++ arch/x86/kvm/mmu/spte.h | 20 +++++++++----------- arch/x86/kvm/mmu/tdp_mmu.c | 4 ++-- 4 files changed, 20 insertions(+), 16 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index a72ecac63e07..06fb0fd3f87d 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -3350,7 +3350,7 @@ static bool page_fault_can_be_fast(struct kvm *kvm, s= truct kvm_page_fault *fault * by setting the Writable bit, which can be done out of mmu_lock. */ if (!fault->present) - return !kvm_ad_enabled(); + return !kvm_ad_enabled; =20 /* * Note, instruction fetches and writes are mutually exclusive, ignore @@ -3485,7 +3485,7 @@ static int fast_page_fault(struct kvm_vcpu *vcpu, str= uct kvm_page_fault *fault) * uses A/D bits for non-nested MMUs. Thus, if A/D bits are * enabled, the SPTE can't be an access-tracked SPTE. */ - if (unlikely(!kvm_ad_enabled()) && is_access_track_spte(spte)) + if (unlikely(!kvm_ad_enabled) && is_access_track_spte(spte)) new_spte =3D restore_acc_track_spte(new_spte); =20 /* @@ -5462,7 +5462,7 @@ kvm_calc_tdp_mmu_root_page_role(struct kvm_vcpu *vcpu, role.efer_nx =3D true; role.smm =3D cpu_role.base.smm; role.guest_mode =3D cpu_role.base.guest_mode; - role.ad_disabled =3D !kvm_ad_enabled(); + role.ad_disabled =3D !kvm_ad_enabled; role.level =3D kvm_mmu_get_tdp_level(vcpu); role.direct =3D true; role.has_4_byte_gpte =3D false; diff --git a/arch/x86/kvm/mmu/spte.c b/arch/x86/kvm/mmu/spte.c index 030813781a63..c9543625dda2 100644 --- a/arch/x86/kvm/mmu/spte.c +++ b/arch/x86/kvm/mmu/spte.c @@ -24,6 +24,8 @@ static bool __ro_after_init allow_mmio_caching; module_param_named(mmio_caching, enable_mmio_caching, bool, 0444); EXPORT_SYMBOL_GPL(enable_mmio_caching); =20 +bool __read_mostly kvm_ad_enabled; + u64 __read_mostly shadow_host_writable_mask; u64 __read_mostly shadow_mmu_writable_mask; u64 __read_mostly shadow_nx_mask; @@ -414,6 +416,8 @@ EXPORT_SYMBOL_GPL(kvm_mmu_set_me_spte_mask); =20 void kvm_mmu_set_ept_masks(bool has_ad_bits, bool has_exec_only) { + kvm_ad_enabled =3D has_ad_bits; + shadow_user_mask =3D VMX_EPT_READABLE_MASK; shadow_accessed_mask =3D has_ad_bits ? VMX_EPT_ACCESS_BIT : 0ull; shadow_dirty_mask =3D has_ad_bits ? VMX_EPT_DIRTY_BIT : 0ull; @@ -447,6 +451,8 @@ void kvm_mmu_reset_all_pte_masks(void) u8 low_phys_bits; u64 mask; =20 + kvm_ad_enabled =3D true; + /* * If the CPU has 46 or less physical address bits, then set an * appropriate mask to guard against L1TF attacks. Otherwise, it is diff --git a/arch/x86/kvm/mmu/spte.h b/arch/x86/kvm/mmu/spte.h index 574ca9a1fcab..908cb672c4e1 100644 --- a/arch/x86/kvm/mmu/spte.h +++ b/arch/x86/kvm/mmu/spte.h @@ -167,6 +167,15 @@ static_assert(!(SHADOW_NONPRESENT_VALUE & SPTE_MMU_PRE= SENT_MASK)); #define SHADOW_NONPRESENT_VALUE 0ULL #endif =20 + +/* + * True if A/D bits are supported in hardware and are enabled by KVM. When + * enabled, KVM uses A/D bits for all non-nested MMUs. Because L1 can dis= able + * A/D bits in EPTP12, SP and SPTE variants are needed to handle the scena= rio + * where KVM is using A/D bits for L1, but not L2. + */ +extern bool __read_mostly kvm_ad_enabled; + extern u64 __read_mostly shadow_host_writable_mask; extern u64 __read_mostly shadow_mmu_writable_mask; extern u64 __read_mostly shadow_nx_mask; @@ -285,17 +294,6 @@ static inline bool is_ept_ve_possible(u64 spte) (spte & VMX_EPT_RWX_MASK) !=3D VMX_EPT_MISCONFIG_WX_VALUE; } =20 -/* - * Returns true if A/D bits are supported in hardware and are enabled by K= VM. - * When enabled, KVM uses A/D bits for all non-nested MMUs. Because L1 can - * disable A/D bits in EPTP12, SP and SPTE variants are needed to handle t= he - * scenario where KVM is using A/D bits for L1, but not L2. - */ -static inline bool kvm_ad_enabled(void) -{ - return !!shadow_accessed_mask; -} - static inline bool sp_ad_disabled(struct kvm_mmu_page *sp) { return sp->role.ad_disabled; diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c index bc9e2f50dc80..f77927b57962 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.c +++ b/arch/x86/kvm/mmu/tdp_mmu.c @@ -1075,7 +1075,7 @@ static int tdp_mmu_map_handle_target_level(struct kvm= _vcpu *vcpu, static int tdp_mmu_link_sp(struct kvm *kvm, struct tdp_iter *iter, struct kvm_mmu_page *sp, bool shared) { - u64 spte =3D make_nonleaf_spte(sp->spt, !kvm_ad_enabled()); + u64 spte =3D make_nonleaf_spte(sp->spt, !kvm_ad_enabled); int ret =3D 0; =20 if (shared) { @@ -1491,7 +1491,7 @@ static bool tdp_mmu_need_write_protect(struct kvm_mmu= _page *sp) * from level, so it is valid to key off any shadow page to determine if * write protection is needed for an entire tree. */ - return kvm_mmu_page_ad_need_write_protect(sp) || !kvm_ad_enabled(); + return kvm_mmu_page_ad_need_write_protect(sp) || !kvm_ad_enabled; } =20 static void clear_dirty_gfn_range(struct kvm *kvm, struct kvm_mmu_page *ro= ot, --=20 2.47.0.rc1.288.g06298d1525-goog From nobody Tue Nov 26 08:38:40 2024 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D8A4E1F9420 for ; Fri, 11 Oct 2024 02:11:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728612676; cv=none; b=oiBQ1Dgznxa0KicYYpoMiMhxAKavPVOjthdKKCHYcqPoJ6JhCLOJM0xDnIHysqB3vliwBBy1A1X4kKkbM10Z3eTJXX0J1RXz7ryN9MrzVaa9e6MFwfBlsbejs18axBFfRtRKKwaq4wZmwyEGK8i0723PH5f/EnJqATw1YrT9g/Q= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728612676; c=relaxed/simple; bh=aKgSp3XrLYNy8hNKG4jop6hvDJmh/ChMqz+HQoJA+wE=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=k4p0pebN++3i0qAVxOuJvFYNjUntP/FIKMG3g6MpW67waECGQh4pCQaMKM9dko+G0ZJGMhdwTH4blfVg5AFM/IIIK7/+e450DlR+Bb4A0nA1jTWhvmdDbTIp1h1mM7YAlZYd3M3BL781e4vZq8ILdXIPX8rGbyd9T33GBnJSI2A= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=BPREzY0J; arc=none smtp.client-ip=209.85.219.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="BPREzY0J" Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-e163641feb9so1759595276.0 for ; Thu, 10 Oct 2024 19:11:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1728612673; x=1729217473; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=TrIlp1Guds3zrlzSMYuxG6o1rQn3R2bekD4gqwcbBS8=; b=BPREzY0JAOiY1f8lVJCq9+YxrYjDQeMYdT+GFTMze8Spr+ZcQB9GCEuqBA1MJX06Kx ETO7NSzcDw4/w7aK68EgUr1aWHSCDjRZjJgL5eGxKQ44AWm7CF6BeOJBcwZWTNLW63BD fHjv++pmJRKO2odF5gN0NkFTYs/0cyvNUy3U84h9wFJYhsmyq093fFFVeNDswgshT/bP pGEHgA7WvZ92VnPu4GVRZTSHkK7Xj5kEPEheZJGqwgkWm9TOfZhbGpXDoQu8DeJpz51I YYd/gQEPHtJJNRodIvLm/Hx386nUYwap44oZ9kbPLSqvm6yJPJIqzdMR99Tl2sGDFZIy 8TAQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1728612673; x=1729217473; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=TrIlp1Guds3zrlzSMYuxG6o1rQn3R2bekD4gqwcbBS8=; b=NWAScQcoaVsXzrNGnXNR18EhBuBl0ZI7a6O3HCfP58Cs7mwY8LoLdgRLEgCgfMWWFf uFH58pu4BXkVMwhmypB7TMo2u9xtA/uiAVwl38Y86dPum/JUdZQqMqqN9TKw8KdvgSCr 8t1uVEC+gvveF73yN5+iIFvTYgZ9+ceQOXNSIR1h67iL8hYr2dkYYyN/NddlnjmhXpQU K8Z+UwDWG9zC3Is/wN4WqPXzrd20U3f29DIzoYeO+l6sdd5X8bDlHiYaiY/Qa/zUCG6z Z+jkHVdYf96X5or9c86zS0zOFEGFIhspqRxS/H7jCvRYu+elBqNW82u+2CfKOEwSaw4i u8TA== X-Forwarded-Encrypted: i=1; AJvYcCU7iThF1sgdSGdaEmSjHR5MlB0ExH1CxHPW/n1cHFAB+iQ/Zz8VCLTejszEaAF0L3xFns2blNrM7OeCSvY=@vger.kernel.org X-Gm-Message-State: AOJu0YyzzGdoisuc233sA8ZTKaS4J0h0ktyBw1/o3+zjgbYmPE93Je7h UOu3GwN7kYZJTE88VAFq/rHJ9bEvpJrMkVdBuOp6LHtpmfQ+L1Xyb/5Kv7eXYM16N7aTcZfdU35 aTQ== X-Google-Smtp-Source: AGHT+IElEKizd4vm3idzgibROUYOU9UMUvmYhyDSJ+LMWh+4cN4KYm1fNLHVNNLdSguOs2K4bXih+kBr8mQ= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:9d:3983:ac13:c240]) (user=seanjc job=sendgmr) by 2002:a05:6902:1349:b0:e20:2da6:ed77 with SMTP id 3f1490d57ef6-e2918421de9mr9964276.5.1728612672856; Thu, 10 Oct 2024 19:11:12 -0700 (PDT) Reply-To: Sean Christopherson Date: Thu, 10 Oct 2024 19:10:42 -0700 In-Reply-To: <20241011021051.1557902-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20241011021051.1557902-1-seanjc@google.com> X-Mailer: git-send-email 2.47.0.rc1.288.g06298d1525-goog Message-ID: <20241011021051.1557902-11-seanjc@google.com> Subject: [PATCH 10/18] KVM: x86/mmu: Set shadow_accessed_mask for EPT even if A/D bits disabled From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Yan Zhao , Sagi Shahar , "=?UTF-8?q?Alex=20Benn=C3=A9e?=" , David Matlack , James Houghton Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Now that KVM doesn't use shadow_accessed_mask to detect if hardware A/D bits are enabled, set shadow_accessed_mask for EPT even when A/D bits are disabled in hardware. This will allow using shadow_accessed_mask for software purposes, e.g. to preserve accessed status in a non-present SPTE acros NUMA balancing, if something like that is ever desirable. Signed-off-by: Sean Christopherson --- arch/x86/kvm/mmu/spte.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/arch/x86/kvm/mmu/spte.c b/arch/x86/kvm/mmu/spte.c index c9543625dda2..e352d1821319 100644 --- a/arch/x86/kvm/mmu/spte.c +++ b/arch/x86/kvm/mmu/spte.c @@ -419,7 +419,7 @@ void kvm_mmu_set_ept_masks(bool has_ad_bits, bool has_e= xec_only) kvm_ad_enabled =3D has_ad_bits; =20 shadow_user_mask =3D VMX_EPT_READABLE_MASK; - shadow_accessed_mask =3D has_ad_bits ? VMX_EPT_ACCESS_BIT : 0ull; + shadow_accessed_mask =3D VMX_EPT_ACCESS_BIT; shadow_dirty_mask =3D has_ad_bits ? VMX_EPT_DIRTY_BIT : 0ull; shadow_nx_mask =3D 0ull; shadow_x_mask =3D VMX_EPT_EXECUTABLE_MASK; --=20 2.47.0.rc1.288.g06298d1525-goog From nobody Tue Nov 26 08:38:40 2024 Received: from mail-pg1-f202.google.com (mail-pg1-f202.google.com [209.85.215.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EBF2B1F9432 for ; Fri, 11 Oct 2024 02:11:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728612677; cv=none; b=ZfdQp6CQ/2sF3E0OHcBCkO7C89J9XGFGte9nVNBZ/+5/TKVN3y/ZkpkKjjPXgZg8sv0sXvkV8FevhE6kCxnJBeI7UfpSLD8uQKFgq9s4nh9ivhi3Hcdl11FT5arVOEThy2o60lxaZq5v+/a+rPihQYDe2LZ1JjuYpigFexE3lFw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728612677; c=relaxed/simple; bh=LZMcdQvJWnJLB3/UIdM5EZpK8QPNNstC6KLm9lhBrbo=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=CFRxwfH3+NfpRWvTsSmWBgOBMjgjUViYkvLAj8OruPHajbj+GvuT8nF5SKXdydCdHez0kpXTuGZT0Oc/AuWyssuVBAAtfIcyzcoxvjRkWTa5xzwZ1zbUJLhh21bafSYEVyZLJiUH7VSUqz07Pw+YZGgQv44tzPaXaccZ0QbEcDE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=eAmLNtWs; arc=none smtp.client-ip=209.85.215.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="eAmLNtWs" Received: by mail-pg1-f202.google.com with SMTP id 41be03b00d2f7-7c6a9c1a9b8so1191701a12.0 for ; Thu, 10 Oct 2024 19:11:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1728612675; x=1729217475; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=Hvv35DHJVsIcR1KuBMr5pv5pWI9cGFmfEf+HtUf4zac=; b=eAmLNtWsx3FWhdjImXvs4TMA5TAEasf25LSl8DYN/dKIRhoqvyBoIvwZ0LSEywVbob sxuXcXD8zoScKKUYERGypKucLm1SCHtUe+vXznvzC8hxh3LYi5+VRUIn+c1ERyiA0PS4 AyUCYWd9GbygTc1S7g6PmOxfNziPRHJtLa8HLvseMW/K2DHQkNCAq3JNnMLXD3ktkqH+ hyllqUUHfTPYSMpCDNI968l0K6qi8/4xu5OuBqzcdDxNu2jI57Lo/QyeKk8YZJxsijpP iD61TYRrSTy9mqPDqWSEOCCJ0fpTUJwSlJ/nvegfJXUkX00QEaz3W4U6OeSfHE77w+Es IP4g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1728612675; x=1729217475; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=Hvv35DHJVsIcR1KuBMr5pv5pWI9cGFmfEf+HtUf4zac=; b=JeIePgWj/WFYqu3jZDzqFw1eD/EoGdWK2vm2tiX1u0BqltqFljaPqOrf0OWJ07Pg+L 2UZG0O4wVxiMxMOt3u1C49mxdL+B/T6yUSAL1VcCXCuY+O0M3icyKGRzy+PKeAVt+UPQ qbxlXk774JXqraL7zhzVA2k5yC/E7w39Dpt/x7MG/XXGxdfH4g0+YivbhVreCcf+ewXH sicGWIrKmPPaUN/oZLJ5rg8tf8kA05ByBEkx0hI5a1Hg7wA182WfWTPVytnBb2993Det WxILYZ3Ruc9dBpH7OfbnnpK4RE69/nlHPpTsrHWhFawIyDBDf6EavLpwqTIMdl+JtFz3 31oA== X-Forwarded-Encrypted: i=1; AJvYcCV36IaoOWA73sDbmpgP4dFMufRljRpdMxuH6VP/Ys6Ke3p3QNFnbP9C1db9BH/RbG1Dk4iRb75QTMUnCvA=@vger.kernel.org X-Gm-Message-State: AOJu0Yy0svPO0alAgx+TAeO6++6YAXOtsR7/o1a7xxd8wcMOyQVbFE0T +wRmJ6si0lsnzGPIdFjHmQA3UsdlZFQTLclGfVR7fAPsiWQfK2hw2MQpUZuNT/Z4FXStdCk50tt zpw== X-Google-Smtp-Source: AGHT+IHnF6owo+1pNhj2RLKasExwUgPBpKQO4fPjKU+UOrHlJ7dmNwwiU5RcRn0ke9/PSMdjRWsWWieNTx8= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:9d:3983:ac13:c240]) (user=seanjc job=sendgmr) by 2002:a63:7d57:0:b0:7d5:e48:4286 with SMTP id 41be03b00d2f7-7ea535a6447mr696a12.7.1728612674757; Thu, 10 Oct 2024 19:11:14 -0700 (PDT) Reply-To: Sean Christopherson Date: Thu, 10 Oct 2024 19:10:43 -0700 In-Reply-To: <20241011021051.1557902-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20241011021051.1557902-1-seanjc@google.com> X-Mailer: git-send-email 2.47.0.rc1.288.g06298d1525-goog Message-ID: <20241011021051.1557902-12-seanjc@google.com> Subject: [PATCH 11/18] KVM: x86/mmu: Set shadow_dirty_mask for EPT even if A/D bits disabled From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Yan Zhao , Sagi Shahar , "=?UTF-8?q?Alex=20Benn=C3=A9e?=" , David Matlack , James Houghton Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Set shadow_dirty_mask to the architectural EPT Dirty bit value even if A/D bits are disabled at the module level, i.e. even if KVM will never enable A/D bits in hardware. Doing so provides consistent behavior for Accessed and Dirty bits, i.e. doesn't leave KVM in a state where it sets shadow_accessed_mask but not shadow_dirty_mask. Functionally, this should be one big nop, as consumption of shadow_dirty_mask is always guarded by a check that hardware A/D bits are enabled. Signed-off-by: Sean Christopherson --- arch/x86/kvm/mmu/spte.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/arch/x86/kvm/mmu/spte.c b/arch/x86/kvm/mmu/spte.c index e352d1821319..54d8c9b76050 100644 --- a/arch/x86/kvm/mmu/spte.c +++ b/arch/x86/kvm/mmu/spte.c @@ -420,7 +420,7 @@ void kvm_mmu_set_ept_masks(bool has_ad_bits, bool has_e= xec_only) =20 shadow_user_mask =3D VMX_EPT_READABLE_MASK; shadow_accessed_mask =3D VMX_EPT_ACCESS_BIT; - shadow_dirty_mask =3D has_ad_bits ? VMX_EPT_DIRTY_BIT : 0ull; + shadow_dirty_mask =3D VMX_EPT_DIRTY_BIT; shadow_nx_mask =3D 0ull; shadow_x_mask =3D VMX_EPT_EXECUTABLE_MASK; /* VMX_EPT_SUPPRESS_VE_BIT is needed for W or X violation. */ --=20 2.47.0.rc1.288.g06298d1525-goog From nobody Tue Nov 26 08:38:40 2024 Received: from mail-pl1-f201.google.com (mail-pl1-f201.google.com [209.85.214.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C21451FA25B for ; Fri, 11 Oct 2024 02:11:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728612681; cv=none; b=c9ZjTS5pOHqhKHf8/pc1vUMKbt/52k3zeAW+eN4lmznnOutFShK5bkLfen+vvahgYhoFd9ynmc5ziUfhf10WVHEK0RITugtogADXPefoKHr2h5+Rt9dpBpwKBjB6vPt6rt6Msq6HDUhU5NPO38XLbNNMWRwrlSm0tun/80I6dyE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728612681; c=relaxed/simple; bh=2l/4Exb4i+z5I5WlQFCs9q8cTS9KUprPX2RvYEEUZlk=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=X1xaOpdNxIVXdrCeq9G7WQqQTTcQUQt508eUyJNXsuIHHj1CcOYyK6NDPSYPUJljFZLpDid3uZ7e9BCwD92hS3txH7/G4zQKeS0jplcZzU78wj15KR0oTZoC9+ZgTmWBIKLCJMQdFFqYrdJVS7aiHhTW7AfDvxFEUofJYcNfHUA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=dNaVsba2; arc=none smtp.client-ip=209.85.214.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="dNaVsba2" Received: by mail-pl1-f201.google.com with SMTP id d9443c01a7336-207510f3242so20306235ad.0 for ; Thu, 10 Oct 2024 19:11:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1728612677; x=1729217477; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=swbcLNUhlO+Z3x5j0iArPCyFfm9nB4ZdSOpt0u+BRiM=; b=dNaVsba2yMFVWf0wZyQTBbH4NTDwleWChlrKE53UQKNjNA5Yw6xXhT1oMavj92mvrK hROvdPEWLNajwQcFxBhu9Sid8+byTTn2eQVxMTDMn9bNP8rydbWAV7tIO2GCez3k2mUl M/1p/Vhiz6ghe2Yh9XXbFivonf7dp6qGqH7HNO7HheXhAMjkJKXODvf34AtRHx4yoD0m /huSs46IuuP3py8sBelqjaOy5olKInvBa85dMPmMwyUCQjDKV4/d0lV0MyyQq7WGUNze wJVKojKsQsRYWHdb82IJxLtFJ0rS0WqDHiWeEarRTIJ7A2EXUzsr2G7N+gttgmmsgOM1 mqYA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1728612677; x=1729217477; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=swbcLNUhlO+Z3x5j0iArPCyFfm9nB4ZdSOpt0u+BRiM=; b=tCTmvC/0tIIkRV4syMYrrLUkFlmKXx9iUCTgUaL7EQvyRLEeO/Zrm30mURMyipv5Wi 6Q2GvTLAqc2/tktXyUINZZwRVbNaJVvo0OPwK4hXFf9svYlxikcT1JjOqJFYRbWrJZ8Z M1ITQ1Pqt50jYiR7Rnecjbacb28UTJCQJ9gV6woqUACLSEsrsq5YpagmF6pOWJEDuurC 9mxKChoNgMfVR0WjsXRVZ4R4LNeJi0b1H/30Orp1aFUMWhA7+90TSXoMowFbhUe/2b6u WUraSdkUp1y3qtUWnMy9PNuJW3rd2Y46OLak+/T0YhWV2IzdtLy73bdVFshpEevGOYeL zfdQ== X-Forwarded-Encrypted: i=1; AJvYcCUa8WfD5dEVE0z3wUrRO1rWLHRUbrOdHKb7lysIGX89dtkgrq/xSnwFOVCROYS8mrwDC1hL9nTxnTDsYtU=@vger.kernel.org X-Gm-Message-State: AOJu0YyxazLo9f+Xzc6zkYNTwcNcLJ6MlgYuKzst+ZsXDxitL7aMI/ts Vw8dsvnwtDQqlEHsw+PQaT/uWSoHtzsXRBrndqJi9QbDuefLOeQosftd6byWkmvSXeZZI4DL6PT uCg== X-Google-Smtp-Source: AGHT+IGBcUZAIBFhERfMhmqeQCLgWZgdChjmWr3g6PLndUtixM7vUgJQznL7pe9JB9jYv/yodxK5GriG5ZU= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:9d:3983:ac13:c240]) (user=seanjc job=sendgmr) by 2002:a17:902:d2c8:b0:20b:7ece:3225 with SMTP id d9443c01a7336-20ca130a1b1mr320995ad.0.1728612676949; Thu, 10 Oct 2024 19:11:16 -0700 (PDT) Reply-To: Sean Christopherson Date: Thu, 10 Oct 2024 19:10:44 -0700 In-Reply-To: <20241011021051.1557902-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20241011021051.1557902-1-seanjc@google.com> X-Mailer: git-send-email 2.47.0.rc1.288.g06298d1525-goog Message-ID: <20241011021051.1557902-13-seanjc@google.com> Subject: [PATCH 12/18] KVM: x86/mmu: Use Accessed bit even when _hardware_ A/D bits are disabled From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Yan Zhao , Sagi Shahar , "=?UTF-8?q?Alex=20Benn=C3=A9e?=" , David Matlack , James Houghton Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Use the Accessed bit in SPTEs even when A/D bits are disabled in hardware, i.e. propagate accessed information to SPTE.Accessed even when KVM is doing manual tracking by making SPTEs not-present. In addition to eliminating a small amount of code in is_accessed_spte(), this also paves the way for preserving Accessed information when a SPTE is zapped in response to a mmu_notifier PROTECTION event, e.g. if a SPTE is zapped because NUMA balancing kicks in. Note, EPT is the only flavor of paging in which A/D bits are conditionally enabled, and the Accessed (and Dirty) bit is software-available when A/D bits are disabled. Note #2, there are currently no concrete plans to preserve Accessed information. Explorations on that front were the initial catalyst, but the cleanup is the motivation for the actual commit. Signed-off-by: Sean Christopherson --- arch/x86/kvm/mmu/mmu.c | 3 ++- arch/x86/kvm/mmu/spte.c | 4 ++-- arch/x86/kvm/mmu/spte.h | 11 +---------- 3 files changed, 5 insertions(+), 13 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index 06fb0fd3f87d..5be3b5f054f1 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -3486,7 +3486,8 @@ static int fast_page_fault(struct kvm_vcpu *vcpu, str= uct kvm_page_fault *fault) * enabled, the SPTE can't be an access-tracked SPTE. */ if (unlikely(!kvm_ad_enabled) && is_access_track_spte(spte)) - new_spte =3D restore_acc_track_spte(new_spte); + new_spte =3D restore_acc_track_spte(new_spte) | + shadow_accessed_mask; =20 /* * To keep things simple, only SPTEs that are MMU-writable can diff --git a/arch/x86/kvm/mmu/spte.c b/arch/x86/kvm/mmu/spte.c index 54d8c9b76050..617479efd127 100644 --- a/arch/x86/kvm/mmu/spte.c +++ b/arch/x86/kvm/mmu/spte.c @@ -175,7 +175,7 @@ bool make_spte(struct kvm_vcpu *vcpu, struct kvm_mmu_pa= ge *sp, =20 spte |=3D shadow_present_mask; if (!prefetch || synchronizing) - spte |=3D spte_shadow_accessed_mask(spte); + spte |=3D shadow_accessed_mask; =20 /* * For simplicity, enforce the NX huge page mitigation even if not @@ -346,7 +346,7 @@ u64 mark_spte_for_access_track(u64 spte) =20 spte |=3D (spte & SHADOW_ACC_TRACK_SAVED_BITS_MASK) << SHADOW_ACC_TRACK_SAVED_BITS_SHIFT; - spte &=3D ~shadow_acc_track_mask; + spte &=3D ~(shadow_acc_track_mask | shadow_accessed_mask); =20 return spte; } diff --git a/arch/x86/kvm/mmu/spte.h b/arch/x86/kvm/mmu/spte.h index 908cb672c4e1..c8dc75337c8b 100644 --- a/arch/x86/kvm/mmu/spte.h +++ b/arch/x86/kvm/mmu/spte.h @@ -316,12 +316,6 @@ static inline bool spte_ad_need_write_protect(u64 spte) return (spte & SPTE_TDP_AD_MASK) !=3D SPTE_TDP_AD_ENABLED; } =20 -static inline u64 spte_shadow_accessed_mask(u64 spte) -{ - KVM_MMU_WARN_ON(!is_shadow_present_pte(spte)); - return spte_ad_enabled(spte) ? shadow_accessed_mask : 0; -} - static inline u64 spte_shadow_dirty_mask(u64 spte) { KVM_MMU_WARN_ON(!is_shadow_present_pte(spte)); @@ -355,10 +349,7 @@ static inline kvm_pfn_t spte_to_pfn(u64 pte) =20 static inline bool is_accessed_spte(u64 spte) { - u64 accessed_mask =3D spte_shadow_accessed_mask(spte); - - return accessed_mask ? spte & accessed_mask - : !is_access_track_spte(spte); + return spte & shadow_accessed_mask; } =20 static inline u64 get_rsvd_bits(struct rsvd_bits_validate *rsvd_check, u64= pte, --=20 2.47.0.rc1.288.g06298d1525-goog From nobody Tue Nov 26 08:38:40 2024 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 859BB1FA26C for ; Fri, 11 Oct 2024 02:11:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728612682; cv=none; b=NrmEmSjWgW7Ii29oZ+KtnJ5v1R66KNlQ46nZeeVrxxmI5doZPxCsAksPlBHJLFy8yuK/bZjtMepUNLihNV8DbF0vLRTBXXLHVh+1B5DpHLc4XckCdKP3znsg19Z7JKlonlawVkH5b8Fic7Ba1ccHxbTeZl9ZRrQc7KVscCrfHQM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728612682; c=relaxed/simple; bh=MQky5LU9doflUP6MUDx+1P9jZCBxYMqk3zoIPK03/xk=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=ERNz3X1M1RdJyy8ebsjnzJEEhpidKsNsX8gnJbSrAar9xjT+NwobMXB9lKRTthDNIwzCjg6rd3fNfvUbQidfmWNYCJXdoYuQTnulbQ1ZnIgir5AJXNoBqpsYYgnDbtzdM2J6DRtwRgDo5ZGCaLz8ZW2tQMl9tgeYxLvx2lyX1D0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=Hn+crdcV; arc=none smtp.client-ip=209.85.128.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="Hn+crdcV" Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-6e26ba37314so31088307b3.0 for ; Thu, 10 Oct 2024 19:11:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1728612678; x=1729217478; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=pofu/xUm1x5cnfvwcy3zr8J76zUU6HIdLt58yu7YvsI=; b=Hn+crdcVa3vq8a3tSKyqiXyFFqeZmnbRplEnpc5SPx4b8JD2gjo48Hr9i5KoyVE29v 09KvSlvlhDNY5ny/C9RZJXx9O9YT/7JAvjQ3sxKsJAbsFteCJ0brVhRnvsu6vQzGLhKW Eozyc6rkAqpvY6N2yP+z9ip5OwI0KzRrrSbi4DaX90O8E6ZJoWLQxus68504n9cD3iHl cxQP+YuWjg0+jt7UaKwWacbvdY5OFuCZOvmKzWa4cmNYWIq+e4LibQ/izSkyHywS/fQj X9A6PwXbjK+j65u+MFAtLlD5Epcn53OhHA0M2aeA9OAMca2RVFBsuZ0ebdOWL370W+aN ZU+g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1728612678; x=1729217478; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=pofu/xUm1x5cnfvwcy3zr8J76zUU6HIdLt58yu7YvsI=; b=iCBfr2pCDJ/60ExXjYXZCzWtk2iO293K4dddNg9wuDGTYPgw9y3uN7sKkgkbG7pmgL fdFDblR5AEKX+MsScNxt5br/M3czZYE66fZmA4Aw+Gu3PQBGCZzL27303HScd1J8DnO0 SUf1sggGG1t1Se6ibiftvYBGQ4iTAmTMhD3tRF9wHyHbBWENEUrM3lXKdL063S4/C1Cg hk2xxgeHSEvp90paY+e7+xUrCFPa17eHPmdu41WUd8WEUvWvlzMwUA1S3JpPux2A4p1Z 7S6ixlh+QFoJ23dCiZnvnITBlIiBjzyzMSLD+U6HPKnmlq4leUZIY0ZEA/PX01lyGyPn AIIg== X-Forwarded-Encrypted: i=1; AJvYcCVtNkXzeWRyiRH3kfjeyrZyT98OY5IElH1fLPX8axY3S6yWAWxRKsnFoIJoT6F9YtYLtb+P4EAb9TgPNQ4=@vger.kernel.org X-Gm-Message-State: AOJu0YyjFlWffAqIGQ+iTuv9jSfa3K4AYHLNqmyznZYOnC0A6LBhH1tW vkmdXzYFYJXHT3yHUDo7qTuBk3VH003FSnQst1EVHtMQjl0y9M7LQo45NiEVTzH51khFGkojw8D xXw== X-Google-Smtp-Source: AGHT+IERrauzYd7CMcGSLwn903JJYQaXvPNepCw4lxr2jUmibgpbopAs+w2tf/sgFyOhULKSVVUqL8Le1K0= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:9d:3983:ac13:c240]) (user=seanjc job=sendgmr) by 2002:a05:6902:2687:b0:e28:eae7:f838 with SMTP id 3f1490d57ef6-e2918e5b589mr1037276.0.1728612678637; Thu, 10 Oct 2024 19:11:18 -0700 (PDT) Reply-To: Sean Christopherson Date: Thu, 10 Oct 2024 19:10:45 -0700 In-Reply-To: <20241011021051.1557902-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20241011021051.1557902-1-seanjc@google.com> X-Mailer: git-send-email 2.47.0.rc1.288.g06298d1525-goog Message-ID: <20241011021051.1557902-14-seanjc@google.com> Subject: [PATCH 13/18] KVM: x86/mmu: Process only valid TDP MMU roots when aging a gfn range From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Yan Zhao , Sagi Shahar , "=?UTF-8?q?Alex=20Benn=C3=A9e?=" , David Matlack , James Houghton Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Skip invalid TDP MMU roots when aging a gfn range. There is zero reason to process invalid roots, as they by definition hold stale information. E.g. if a root is invalid because its from a previous memslot generation, in the unlikely event the root has a SPTE for the gfn, then odds are good that the gfn=3D>hva mapping is different, i.e. doesn't map to the hva that is being aged by the primary MMU. Signed-off-by: Sean Christopherson --- arch/x86/kvm/mmu/tdp_mmu.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c index f77927b57962..e8c061bf94ec 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.c +++ b/arch/x86/kvm/mmu/tdp_mmu.c @@ -1205,9 +1205,11 @@ static __always_inline bool kvm_tdp_mmu_handle_gfn(s= truct kvm *kvm, =20 /* * Don't support rescheduling, none of the MMU notifiers that funnel - * into this helper allow blocking; it'd be dead, wasteful code. + * into this helper allow blocking; it'd be dead, wasteful code. Note, + * this helper must NOT be used to unmap GFNs, as it processes only + * valid roots! */ - for_each_tdp_mmu_root(kvm, root, range->slot->as_id) { + for_each_valid_tdp_mmu_root(kvm, root, range->slot->as_id) { rcu_read_lock(); =20 tdp_root_for_each_leaf_pte(iter, root, range->start, range->end) --=20 2.47.0.rc1.288.g06298d1525-goog From nobody Tue Nov 26 08:38:40 2024 Received: from mail-pl1-f201.google.com (mail-pl1-f201.google.com [209.85.214.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 617A51FAC29 for ; Fri, 11 Oct 2024 02:11:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728612683; cv=none; b=Vd25CiF45Gfos1nF2lbCiuna2YyIZa7wVqDHKCFP0R88IHmUMC2zHKLSVXuiZ/GbCmHywPuRBBVFhHwG6kFwNYNvtwlb80UNJH92DWBxYALFc01hdAOUf5uLRCHUcbyFt799o1+ewbJmcMENPr9jfNoYvWfTRmd/ELTwx65+bjE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728612683; c=relaxed/simple; bh=m/5CauBdL1nt8Fd10Sy1g4VhGuxt93poof2ffX7YAoM=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=sJOeGc0GCtw0RNQ+G31WLbNpWI+mIUkfcYQFiD6oDG3WYZyQLTZ6Ta4oOmv91fxBD1JEOKbXvrwwJ56eQPSP+fjd6IcK8hPhyAIoLfxVCN5eIv3yYj3ZLnsYeBToa0eYBXaQGQZmsTgENXR+dMt8mWmWhjCYqd7LvFLbB/iKOU8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=CU3FIr26; arc=none smtp.client-ip=209.85.214.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="CU3FIr26" Received: by mail-pl1-f201.google.com with SMTP id d9443c01a7336-205529c3402so16074445ad.0 for ; Thu, 10 Oct 2024 19:11:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1728612682; x=1729217482; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=F6r5K1Obxnm5Vr48TnHxk1cPnehpFcrMpqKQdTEhjUg=; b=CU3FIr26OFZC3tciRj903lQGZIDMfFot9AXP+/hE9/Lm9k0hdT4VFQFWjMEjorD5W5 xf1Dj0fK5h57EFWI8E+p42VFraY4AnuxujbET9qI0ScB6KjZXQmnJhw+r3m4A+du/2/+ URv1FCIqDCqIvKpNZByY/KLcyBBC/utG5C6MkPZInS2ztuhSx2z0OgzhAZOrCHvQJ6kS ba5x45lPbR9urZCEOCCInOOMnOZsgRIAvEgatGhMBQUc+V/cb3gROaGYQG06u1/I68tv JuudrXaqfJLh3NOVoKmQW0BOOa04A33dZeIS59gEqn0euPaUE0GT6/a5OiYk0sNrmBHn rP1Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1728612682; x=1729217482; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=F6r5K1Obxnm5Vr48TnHxk1cPnehpFcrMpqKQdTEhjUg=; b=fuAsGEET+thWJ0mBFgIpw5R0jL+A/eqsKU2fDTaBEXZMBDtDBRw2qGHhqo+bbDVg/3 1anBOdfQmuwu+xkruiqchemee+375GeZHmkeXXc2885zuyMYGcCS0WqYDktImfEvAEgO Yp0iNsutIZWZnEZn+Acp9aUYl7bkrk27ppS5x5n9GHrgN0lLg62MFIk0tLjt6dk53M68 vG4L25IhJFcYyTWZJC639yh5walmhxvG6hhPQwJibL16vPWzJYM03AwP2UjiVtwA6cpE iQQLtC/QL+4Puas973tH+bQQS+B36OzflW3/26HoyqhlC9bvnJCATOF3GlhLZ72JtcWs XvuQ== X-Forwarded-Encrypted: i=1; AJvYcCW/uy+A8uHk196bsqQWIZzxLehRF6I6GkJjDS+XUoNjdt3kHqf1/eZJPr4BTECBBLYP63ilXbsrnE+tQR0=@vger.kernel.org X-Gm-Message-State: AOJu0YzCv/lnV33J7TljS6vD3wb7QKogMKdZKnmRBDa5p/8yUCQUT2gc /A8XchNRb6uh6/5Qkksjy/eCEm5lffS/FeuMj3b9b+E5nM7X3S1TJKFMDiKgT7xGf8alDZ1dxwL rsA== X-Google-Smtp-Source: AGHT+IErTn43l0LgEWENkKDIgkfXKjKQt8ITh9JwTsuSnXZvVQSIOElg4tIAx1YgDCRzg3lb2AMhAmBmMyg= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:9d:3983:ac13:c240]) (user=seanjc job=sendgmr) by 2002:a17:902:ecc7:b0:20b:9522:6564 with SMTP id d9443c01a7336-20ca16ddcefmr7355ad.9.1728612681515; Thu, 10 Oct 2024 19:11:21 -0700 (PDT) Reply-To: Sean Christopherson Date: Thu, 10 Oct 2024 19:10:46 -0700 In-Reply-To: <20241011021051.1557902-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20241011021051.1557902-1-seanjc@google.com> X-Mailer: git-send-email 2.47.0.rc1.288.g06298d1525-goog Message-ID: <20241011021051.1557902-15-seanjc@google.com> Subject: [PATCH 14/18] KVM: x86/mmu: Stop processing TDP MMU roots for test_age if young SPTE found From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Yan Zhao , Sagi Shahar , "=?UTF-8?q?Alex=20Benn=C3=A9e?=" , David Matlack , James Houghton Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Return immediately if a young SPTE is found when testing, but not updating, SPTEs. The return value is a boolean, i.e. whether there is one young SPTE or fifty is irrelevant (ignoring the fact that it's impossible for there to be fifty SPTEs, as KVM has a hard limit on the number of valid TDP MMU roots). Signed-off-by: Sean Christopherson --- arch/x86/kvm/mmu/tdp_mmu.c | 84 ++++++++++++++++++-------------------- 1 file changed, 40 insertions(+), 44 deletions(-) diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c index e8c061bf94ec..f412bca206c5 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.c +++ b/arch/x86/kvm/mmu/tdp_mmu.c @@ -1192,35 +1192,6 @@ bool kvm_tdp_mmu_unmap_gfn_range(struct kvm *kvm, st= ruct kvm_gfn_range *range, return flush; } =20 -typedef bool (*tdp_handler_t)(struct kvm *kvm, struct tdp_iter *iter, - struct kvm_gfn_range *range); - -static __always_inline bool kvm_tdp_mmu_handle_gfn(struct kvm *kvm, - struct kvm_gfn_range *range, - tdp_handler_t handler) -{ - struct kvm_mmu_page *root; - struct tdp_iter iter; - bool ret =3D false; - - /* - * Don't support rescheduling, none of the MMU notifiers that funnel - * into this helper allow blocking; it'd be dead, wasteful code. Note, - * this helper must NOT be used to unmap GFNs, as it processes only - * valid roots! - */ - for_each_valid_tdp_mmu_root(kvm, root, range->slot->as_id) { - rcu_read_lock(); - - tdp_root_for_each_leaf_pte(iter, root, range->start, range->end) - ret |=3D handler(kvm, &iter, range); - - rcu_read_unlock(); - } - - return ret; -} - /* * Mark the SPTEs range of GFNs [start, end) unaccessed and return non-zero * if any of the GFNs in the range have been accessed. @@ -1229,15 +1200,10 @@ static __always_inline bool kvm_tdp_mmu_handle_gfn(= struct kvm *kvm, * from the clear_young() or clear_flush_young() notifier, which uses the * return value to determine if the page has been accessed. */ -static bool age_gfn_range(struct kvm *kvm, struct tdp_iter *iter, - struct kvm_gfn_range *range) +static void kvm_tdp_mmu_age_spte(struct tdp_iter *iter) { u64 new_spte; =20 - /* If we have a non-accessed entry we don't need to change the pte. */ - if (!is_accessed_spte(iter->old_spte)) - return false; - if (spte_ad_enabled(iter->old_spte)) { iter->old_spte =3D tdp_mmu_clear_spte_bits(iter->sptep, iter->old_spte, @@ -1253,23 +1219,53 @@ static bool age_gfn_range(struct kvm *kvm, struct t= dp_iter *iter, =20 trace_kvm_tdp_mmu_spte_changed(iter->as_id, iter->gfn, iter->level, iter->old_spte, new_spte); - return true; +} + +static bool __kvm_tdp_mmu_age_gfn_range(struct kvm *kvm, + struct kvm_gfn_range *range, + bool test_only) +{ + struct kvm_mmu_page *root; + struct tdp_iter iter; + bool ret =3D false; + + /* + * Don't support rescheduling, none of the MMU notifiers that funnel + * into this helper allow blocking; it'd be dead, wasteful code. Note, + * this helper must NOT be used to unmap GFNs, as it processes only + * valid roots! + */ + for_each_valid_tdp_mmu_root(kvm, root, range->slot->as_id) { + rcu_read_lock(); + + tdp_root_for_each_leaf_pte(iter, root, range->start, range->end) { + if (!is_accessed_spte(iter.old_spte)) + continue; + + ret =3D true; + if (test_only) + break; + + kvm_tdp_mmu_age_spte(&iter); + } + + rcu_read_unlock(); + + if (ret && test_only) + break; + } + + return ret; } =20 bool kvm_tdp_mmu_age_gfn_range(struct kvm *kvm, struct kvm_gfn_range *rang= e) { - return kvm_tdp_mmu_handle_gfn(kvm, range, age_gfn_range); -} - -static bool test_age_gfn(struct kvm *kvm, struct tdp_iter *iter, - struct kvm_gfn_range *range) -{ - return is_accessed_spte(iter->old_spte); + return __kvm_tdp_mmu_age_gfn_range(kvm, range, false); } =20 bool kvm_tdp_mmu_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) { - return kvm_tdp_mmu_handle_gfn(kvm, range, test_age_gfn); + return __kvm_tdp_mmu_age_gfn_range(kvm, range, true); } =20 /* --=20 2.47.0.rc1.288.g06298d1525-goog From nobody Tue Nov 26 08:38:40 2024 Received: from mail-pg1-f202.google.com (mail-pg1-f202.google.com [209.85.215.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8029D1FB3D1 for ; Fri, 11 Oct 2024 02:11:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728612687; cv=none; b=tROObznqdRYa0Tcd1MHn6o47jC9jtX0VSlpPaJwFA+lF1KH3IrxRhivPHTGxn5ayaV/ZyS5tYyENSZ+vTHbKpZziBL8b8LlKl45LsgEPYNgY3J0DVrP91gUTkJBEzior0hVhyWWfn3RCQs7ORWUlJ3J3aIKruA0mmdmdC6xcKRo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728612687; c=relaxed/simple; bh=LNXuTGR2S+iPgGGEX03rlcOFXKsH1faw0ZPmJ58DiOE=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=L9qmva7jq7pJf7RABVn2MbEZMl00AnfI2jU/47zSvw7kx7GJWFFGd0sA8Fcbp4b4HLDlLscj7vWPVTMlsBe6lF3r7M88ccaLf/WSSNJZLGHA8DlTjXLQR4QHnflz2OCTNu8twcmPnMVsWW3u6TmCiTeYN7JaHM7QP+v3lEnZJsU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=FdgOyntl; arc=none smtp.client-ip=209.85.215.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="FdgOyntl" Received: by mail-pg1-f202.google.com with SMTP id 41be03b00d2f7-7ea38f581c9so1614823a12.0 for ; Thu, 10 Oct 2024 19:11:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1728612685; x=1729217485; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=CEg0IFfdUu4UELrQOXo0lUcfHQKNBNnbMq1djiDkWGQ=; b=FdgOyntliR+R53ku1l+8JOs7WqGHgzY5J/dK5vP4+B/B6lw9kgD+qUduihpl8medjW ms2yX97jx7+MevWVzcwv0I/VWrHae4vpGT5Umxt+PwocoL3karL2rybxy7ihNByAsvT7 cTNCE6S6CdhsYwEIcq5LbRSPEAm3smllOxDeTPuvw405/MbFTfPdsKRgcMS7KFMkkL4J sdD4CEmksfwBvI1oo/NZq+KhmArReJnxzUjI4LEFmtgJid28LVEhAeHD9fC7gqK8Ko4h pQLCcYdRzS6bcYh3GRxkYFg9XyWm4WD+6SInx85SwUnlAP8G4+fqi1tNIsJNleQv2owJ pWuA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1728612685; x=1729217485; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=CEg0IFfdUu4UELrQOXo0lUcfHQKNBNnbMq1djiDkWGQ=; b=vHQ7pL0cE65g5DodCDmmDdx4tq6NFh8gMi86tupZiMPQ97vIzwSHxxFFW6AZbYQSpG msEXTmCTNP1G18sgBvI8zPYBom6I1HpcWK6mrOTNTz7Y2KvdS5OfWoQOGqqgn2teRRti lKeqXvIEy4T5+OWuTPzpSkdT/hR0k0+2h7N+aukGp4P6BGJmhfX/oMUisfXvlDXygWo5 ajvOaKcQ+0aNl/A/YE9vHiAgor1TiXC6BzIrygddVCS2X4SR02HBnmHo4PCigx2Z53iv 19z2yq37nEA/Hvt2QZsJcVX8SWJllKkAQhuJYmf1eSIEvm1Ry0iGaAclVYntBP9KRiB7 hflw== X-Forwarded-Encrypted: i=1; AJvYcCV1y/zQJb2N1HWCdnhNIgbXFyrMTQNmA+8UPEya92Ow+SFrMS2u9Mbp2RFk0zUQ+NsowVTrUi+KqSjtmgo=@vger.kernel.org X-Gm-Message-State: AOJu0YzjKatzSZjYJD96KrOpjDhP/U9MBVbGFejRN16rpW3fcByziQZm Rf1Ek5Y2XgnrE2YQ48/gLpD5GVhWY59Mt2WEwh3JxXSkK2KR4N+Fp9fBLvQq8RNTVT1XDYwEbj8 syw== X-Google-Smtp-Source: AGHT+IHOAZfcUyAOkGK8+VjEG0Gb80ABIViSbVLqAW+P+s1wcIQgbwkQ7kkxk7jRjasA/YCkTBO3YYYn+wk= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:9d:3983:ac13:c240]) (user=seanjc job=sendgmr) by 2002:a05:6a02:f8a:b0:7cd:8b5f:2567 with SMTP id 41be03b00d2f7-7ea535255a7mr911a12.4.1728612683541; Thu, 10 Oct 2024 19:11:23 -0700 (PDT) Reply-To: Sean Christopherson Date: Thu, 10 Oct 2024 19:10:47 -0700 In-Reply-To: <20241011021051.1557902-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20241011021051.1557902-1-seanjc@google.com> X-Mailer: git-send-email 2.47.0.rc1.288.g06298d1525-goog Message-ID: <20241011021051.1557902-16-seanjc@google.com> Subject: [PATCH 15/18] KVM: x86/mmu: Dedup logic for detecting TLB flushes on leaf SPTE changes From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Yan Zhao , Sagi Shahar , "=?UTF-8?q?Alex=20Benn=C3=A9e?=" , David Matlack , James Houghton Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Now that the shadow MMU and TDP MMU have identical logic for detecting required TLB flushes when updating SPTEs, move said logic to a helper so that the TDP MMU code can benefit from the comments that are currently exclusive to the shadow MMU. No functional change intended. Signed-off-by: Sean Christopherson --- arch/x86/kvm/mmu/mmu.c | 19 +------------------ arch/x86/kvm/mmu/spte.h | 29 +++++++++++++++++++++++++++++ arch/x86/kvm/mmu/tdp_mmu.c | 3 +-- 3 files changed, 31 insertions(+), 20 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index 5be3b5f054f1..f75915ff33be 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -488,23 +488,6 @@ static void mmu_spte_set(u64 *sptep, u64 new_spte) /* Rules for using mmu_spte_update: * Update the state bits, it means the mapped pfn is not changed. * - * If the MMU-writable flag is cleared, i.e. the SPTE is write-protected f= or - * write-tracking, remote TLBs must be flushed, even if the SPTE was read-= only, - * as KVM allows stale Writable TLB entries to exist. When dirty logging,= KVM - * flushes TLBs based on whether or not dirty bitmap/ring entries were rea= ped, - * not whether or not SPTEs were modified, i.e. only the write-tracking ca= se - * needs to flush at the time the SPTEs is modified, before dropping mmu_l= ock. - * - * Don't flush if the Accessed bit is cleared, as access tracking tolerates - * false negatives, and the one path that does care about TLB flushes, - * kvm_mmu_notifier_clear_flush_young(), flushes if a young SPTE is found,= i.e. - * doesn't rely on lower helpers to detect the need to flush. - * - * Lastly, don't flush if the Dirty bit is cleared, as KVM unconditionally - * flushes when enabling dirty logging (see kvm_mmu_slot_apply_flags()), a= nd - * when clearing dirty logs, KVM flushes based on whether or not dirty ent= ries - * were reaped from the bitmap/ring, not whether or not dirty SPTEs were f= ound. - * * Returns true if the TLB needs to be flushed */ static bool mmu_spte_update(u64 *sptep, u64 new_spte) @@ -527,7 +510,7 @@ static bool mmu_spte_update(u64 *sptep, u64 new_spte) WARN_ON_ONCE(!is_shadow_present_pte(old_spte) || spte_to_pfn(old_spte) !=3D spte_to_pfn(new_spte)); =20 - return is_mmu_writable_spte(old_spte) && !is_mmu_writable_spte(new_spte); + return is_tlb_flush_required_for_leaf_spte(old_spte, new_spte); } =20 /* diff --git a/arch/x86/kvm/mmu/spte.h b/arch/x86/kvm/mmu/spte.h index c8dc75337c8b..a404279ba731 100644 --- a/arch/x86/kvm/mmu/spte.h +++ b/arch/x86/kvm/mmu/spte.h @@ -467,6 +467,35 @@ static inline bool is_mmu_writable_spte(u64 spte) return spte & shadow_mmu_writable_mask; } =20 +/* + * If the MMU-writable flag is cleared, i.e. the SPTE is write-protected f= or + * write-tracking, remote TLBs must be flushed, even if the SPTE was read-= only, + * as KVM allows stale Writable TLB entries to exist. When dirty logging,= KVM + * flushes TLBs based on whether or not dirty bitmap/ring entries were rea= ped, + * not whether or not SPTEs were modified, i.e. only the write-tracking ca= se + * needs to flush at the time the SPTEs is modified, before dropping mmu_l= ock. + * + * Don't flush if the Accessed bit is cleared, as access tracking tolerates + * false negatives, and the one path that does care about TLB flushes, + * kvm_mmu_notifier_clear_flush_young(), flushes if a young SPTE is found,= i.e. + * doesn't rely on lower helpers to detect the need to flush. + * + * Lastly, don't flush if the Dirty bit is cleared, as KVM unconditionally + * flushes when enabling dirty logging (see kvm_mmu_slot_apply_flags()), a= nd + * when clearing dirty logs, KVM flushes based on whether or not dirty ent= ries + * were reaped from the bitmap/ring, not whether or not dirty SPTEs were f= ound. + * + * Note, this logic only applies to shadow-present leaf SPTEs. The caller= is + * responsible for checking that the old SPTE is shadow-present, and is al= so + * responsible for determining whether or not a TLB flush is required when + * modifying a shadow-present non-leaf SPTE. + */ +static inline bool is_tlb_flush_required_for_leaf_spte(u64 old_spte, + u64 new_spte) +{ + return is_mmu_writable_spte(old_spte) && !is_mmu_writable_spte(new_spte); +} + static inline u64 get_mmio_spte_generation(u64 spte) { u64 gen; diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c index f412bca206c5..615c6a84fd60 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.c +++ b/arch/x86/kvm/mmu/tdp_mmu.c @@ -1034,8 +1034,7 @@ static int tdp_mmu_map_handle_target_level(struct kvm= _vcpu *vcpu, return RET_PF_RETRY; else if (is_shadow_present_pte(iter->old_spte) && (!is_last_spte(iter->old_spte, iter->level) || - WARN_ON_ONCE(is_mmu_writable_spte(iter->old_spte) && - !is_mmu_writable_spte(new_spte)))) + WARN_ON_ONCE(is_tlb_flush_required_for_leaf_spte(iter->old_spte, new_s= pte)))) kvm_flush_remote_tlbs_gfn(vcpu->kvm, iter->gfn, iter->level); =20 /* --=20 2.47.0.rc1.288.g06298d1525-goog From nobody Tue Nov 26 08:38:40 2024 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CED321FB3E9 for ; Fri, 11 Oct 2024 02:11:27 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728612690; cv=none; b=UN2xexLJHQqD49wxTpBzW91bKJh46dGGpeomUWsq25+wYRDTRcr8MhKsG2fpIDruejxaDzwuHfzRw3ULZ8QnmMB9KkE4qiGImv/PtwfUajBTF9DaWuJ2qv2dkZCedQWqiJgEwlCTZKhNCj9ieRGkSmcv8Zgd7eIXhJi2bwcTSpo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728612690; c=relaxed/simple; bh=Tp2aWz/KUkLgHNMUQeB0T53YK5NkYJtBqImftywh1pg=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=CE5eSORFRWGikF+B1iwIAdzOGEA0mMGmR5D0aPzGiSf/jxB6EE3uFmXzA5+JHHvgGOzgPCpByuYebg/cMV6GZANrEbbeza0DWMuZFfZwqEOEsdUAVzm4dNLHtR7Th3KdMDnwQALgNDceURdIWRTcheKuTrwOf4+HW30Vf1r507M= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=Va1ihGKY; arc=none smtp.client-ip=209.85.128.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="Va1ihGKY" Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-6dbbeee08f0so39660707b3.0 for ; Thu, 10 Oct 2024 19:11:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1728612687; x=1729217487; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=VgP7ZnqG8N9jdCaiPxtHytdx2vM8g+9dqJnt13kkQQE=; b=Va1ihGKY/nVmOINfhOXd8M7aT59KbsX96wpgdbJhNOvasLR7rfi4cXKK897pDwFb9U 06n2x/k5uhzaF1eHKcISXMkmp3ANm7qFdyOqtzNdf8oosPh+ZvIyi7wp5lOCpnuDYqrx EJtcBckCfISE54OIGSKBr5PGTshjngSM9e8ziO4yH5yVBaD8Z8stDD5QiWaOR+jTVvUJ T8xpaVusrO/uSap8jQ+Lv6oPyhLNNOCl3kw3zb4lpGBjSxwFRzd+2IJdaa3/Jat6O2Ve ATqFKi9KPGR5JmYDOl7qlmyMXf7k0YxYPxWXIgKVRFFkwb0XeeoGduKPTdLlm7gkjp8c Ck1g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1728612687; x=1729217487; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=VgP7ZnqG8N9jdCaiPxtHytdx2vM8g+9dqJnt13kkQQE=; b=Kb8GgAbrzLFE/qIdqMtv3zoDku8xHBXJRVrPY6s35p5YcqzxFTdMY20/QU+/NlZcH5 dXwYRndVtkjGJ53b1+hcEW/4yz+Sa2O+ei0YgX0lZfxO7Q9NoHiNiohTYYn4/qoC/kqU guceYmKUrm8Vck2/n1Yyf1IGZRzv30Rm7BTSmQaS+bs0R+vL0CuhzwJYD3PbMEhj4VAv 1FFi/01Hi4HA5CPCHZEUgbYdzmrInXUQKCUIkiXfn3kKxpd648lT+2akppYlFbuvuQRM 4FyQsszPNEcoM9sjW84gDGn8dCXLr9wXddNVNwTZFO/raq6aPc37SvQK+eWLHeTeGeX2 75lw== X-Forwarded-Encrypted: i=1; AJvYcCUKNLtpKuG1LZ2H94b5PBaMQ857/shElNlS22PwvTOE2c+UyNDaaO5LiWl/lsuVwjayV5fcHTPZdoP2HmY=@vger.kernel.org X-Gm-Message-State: AOJu0YwHiFyvRKOOcwFC96yQqf5EojwmrnV1RwEh+73YB61m3gM2mVWO H6iptLP7UUynwKjZf6+07jhv9o7phnPhS2vTDnu97E1bcIIi2/Nr4lWybinq2ctnXSYMY0Sx2vY MTA== X-Google-Smtp-Source: AGHT+IHfuBbi88SzKZ8fyOpKZIpd/P6uL6w934+Qz4f2IixKQEh8/8BEdcKQO7aaLExUXEvNpT4HwLrwVCw= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:9d:3983:ac13:c240]) (user=seanjc job=sendgmr) by 2002:a25:b104:0:b0:e29:a86:fd0d with SMTP id 3f1490d57ef6-e290bb40627mr81586276.5.1728612686619; Thu, 10 Oct 2024 19:11:26 -0700 (PDT) Reply-To: Sean Christopherson Date: Thu, 10 Oct 2024 19:10:48 -0700 In-Reply-To: <20241011021051.1557902-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20241011021051.1557902-1-seanjc@google.com> X-Mailer: git-send-email 2.47.0.rc1.288.g06298d1525-goog Message-ID: <20241011021051.1557902-17-seanjc@google.com> Subject: [PATCH 16/18] KVM: x86/mmu: Set Dirty bit for new SPTEs, even if _hardware_ A/D bits are disabled From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Yan Zhao , Sagi Shahar , "=?UTF-8?q?Alex=20Benn=C3=A9e?=" , David Matlack , James Houghton Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" When making a SPTE, set the Dirty bit in the SPTE as appropriate, even if hardware A/D bits are disabled. Only EPT allows A/D bits to be disabled, and for EPT, the bits are software-available (ignored by hardware) when A/D bits are disabled, i.e. it is perfectly legal for KVM to use the Dirty to track dirty pages in software. Signed-off-by: Sean Christopherson --- arch/x86/kvm/mmu/spte.c | 2 +- arch/x86/kvm/mmu/spte.h | 6 ------ 2 files changed, 1 insertion(+), 7 deletions(-) diff --git a/arch/x86/kvm/mmu/spte.c b/arch/x86/kvm/mmu/spte.c index 617479efd127..fd8c3c92ade0 100644 --- a/arch/x86/kvm/mmu/spte.c +++ b/arch/x86/kvm/mmu/spte.c @@ -237,7 +237,7 @@ bool make_spte(struct kvm_vcpu *vcpu, struct kvm_mmu_pa= ge *sp, wrprot =3D true; else spte |=3D PT_WRITABLE_MASK | shadow_mmu_writable_mask | - spte_shadow_dirty_mask(spte); + shadow_dirty_mask; } =20 if (prefetch && !synchronizing) diff --git a/arch/x86/kvm/mmu/spte.h b/arch/x86/kvm/mmu/spte.h index a404279ba731..e90cc401c168 100644 --- a/arch/x86/kvm/mmu/spte.h +++ b/arch/x86/kvm/mmu/spte.h @@ -316,12 +316,6 @@ static inline bool spte_ad_need_write_protect(u64 spte) return (spte & SPTE_TDP_AD_MASK) !=3D SPTE_TDP_AD_ENABLED; } =20 -static inline u64 spte_shadow_dirty_mask(u64 spte) -{ - KVM_MMU_WARN_ON(!is_shadow_present_pte(spte)); - return spte_ad_enabled(spte) ? shadow_dirty_mask : 0; -} - static inline bool is_access_track_spte(u64 spte) { return !spte_ad_enabled(spte) && (spte & shadow_acc_track_mask) =3D=3D 0; --=20 2.47.0.rc1.288.g06298d1525-goog From nobody Tue Nov 26 08:38:40 2024 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 330011F12E9 for ; Fri, 11 Oct 2024 02:11:30 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728612691; cv=none; b=BMCOPzF7k+C72of7bT2laf9sZZl0CTLV5BwsT5mrPq4kmcL7mUPljCa3tzletSjbTyAX1Orwjp8TttJuZNeSNn1CrxHqPQTXztTydmQA3QXE1YgCXvgU6JYs5mcdUFrGUGtdD5ILdSBcIHpqEns2bPUgwAlUUXyFPj9GcNUs7uE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728612691; c=relaxed/simple; bh=iyJm/h+JOO59f+egBKANfLu+0QMr2pI0zJeagTYRCD4=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=tQKpgBJTPS/Op8yz5LvFEx0Q9p5JtW5NZQftg6lE+NtrwE7JXC/ScfiFaC1ihQx3I00CuH2X2iLyx4UwbmDDDqdJrt0R1XwTo2k6nIlBZ6r+cp9E2AWRElr0BD1apfmTbZyCmQR0nUKmsJBOfTMllkkCS28ZVsDYRpCfJENMQJs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=PYPMO+dI; arc=none smtp.client-ip=209.85.128.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="PYPMO+dI" Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-6e31e5d1739so30735237b3.1 for ; Thu, 10 Oct 2024 19:11:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1728612689; x=1729217489; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=HPDDwUN824pfD2nj2HWQd9wsiBjvr8+g8+t2fSbMNlA=; b=PYPMO+dIz2MEJ9uWGnpzMDLoKh+UuyJIyZmNXuqbSiVOdTd214fzLaTbR1r/huZV6G gUCNMV6B52TgkvmtpjDNuduzkb92NtWqgasAaY+wAZYjvZ0H5I9C16O55KtJTEiAAvpc 9QHy/RtL6SGjpF98SSBX5CaAtlaI34+Y9n5jvCtnuLGPW3j/w5bmER0tDHQPup2InfJh 8NLMGe2+b0hFnbp0glpF33KZbnMpiOn7DpgyppnLX8VdkE3Eb2RDJQoxANgT2bQBpeTX hqnp3IwrCFkL1iBeNy+pL7ILktxKp96OaGtltMGYnfnPKp6V53PPQF4zzD2T5a5G9g3r F32w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1728612689; x=1729217489; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=HPDDwUN824pfD2nj2HWQd9wsiBjvr8+g8+t2fSbMNlA=; b=Ycyv2cUA0xGa/nYm8kw1lFcq3AnnxL6jmEK8qzKNBH32GETMSLWQnmARlFwRGu4vf2 sRlwn8z5o7wLJptIYTtm2c0zH3oiFZbclHDEEryCzKQQCb3g+wzmIFcffXbq+ioLw5ha VBv2OL7u0770NcmY/kDfTsrDgNd1bhtOnKMDqtoG8vGKn5beMmt0uZSGk72GGxqp8YrI 3Z6hNKo5f28DCoDkgvzQT+ePjbjX1euT4E0uS4nKPgn1ef09LnB9xBwsCXOh6132euES SsV0sLZcs+NquglqzSDKxvMrvWvnT7hUbIfURILz/4bgcWBHgf6/MA29mZERmKVxcdaq KmEw== X-Forwarded-Encrypted: i=1; AJvYcCVrZRZf7N+VXE6EEfmuuOw5YykBP8wyhoOFtdhKJsD7cwoGfcfZeDvuGVjI7aVh2Co91pkXUWB4ybf8ckQ=@vger.kernel.org X-Gm-Message-State: AOJu0YwEWp4xtz7PLqk70MQtCRg/CRgEeTLcsNyEN3tzaQnOzZRWZK+B O9tzue8J/vrX+vSPU4gcT1VfApirbm09xCxQZWTKuyPzoAC0tydD0GsjAzZhUnpScF4CfgRf5/G KPA== X-Google-Smtp-Source: AGHT+IGTSYki/MPVOSXtSoZ4BErjm1NrfbNeO31MKvV+62YjjXjQGEoBGgc7gkTIPIKCnszTY2Gpwqbhv0k= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:9d:3983:ac13:c240]) (user=seanjc job=sendgmr) by 2002:a25:d346:0:b0:e17:8e4f:981a with SMTP id 3f1490d57ef6-e291a32c730mr5851276.11.1728612688656; Thu, 10 Oct 2024 19:11:28 -0700 (PDT) Reply-To: Sean Christopherson Date: Thu, 10 Oct 2024 19:10:49 -0700 In-Reply-To: <20241011021051.1557902-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20241011021051.1557902-1-seanjc@google.com> X-Mailer: git-send-email 2.47.0.rc1.288.g06298d1525-goog Message-ID: <20241011021051.1557902-18-seanjc@google.com> Subject: [PATCH 17/18] KVM: Allow arch code to elide TLB flushes when aging a young page From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Yan Zhao , Sagi Shahar , "=?UTF-8?q?Alex=20Benn=C3=A9e?=" , David Matlack , James Houghton Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add a Kconfig to allow architectures to opt-out of a TLB flush when a young page is aged, as invalidating TLB entries is not functionally required on most KVM-supported architectures. Stale TLB entries can result in false negatives and theoretically lead to suboptimal reclaim, but in practice all observations have been that the performance gained by skipping TLB flushes outweighs any performance lost by reclaiming hot pages. E.g. the primary MMUs for x86 RISC-V, s390, and PPC Book3S elide the TLB flush for ptep_clear_flush_young(), and arm64's MMU skips the trailing DSB that's required for ordering (presumably because there are optimizations related to eliding other TLB flushes when doing make-before-break). Signed-off-by: Sean Christopherson --- virt/kvm/Kconfig | 4 ++++ virt/kvm/kvm_main.c | 20 ++++++-------------- 2 files changed, 10 insertions(+), 14 deletions(-) diff --git a/virt/kvm/Kconfig b/virt/kvm/Kconfig index fd6a3010afa8..54e959e7d68f 100644 --- a/virt/kvm/Kconfig +++ b/virt/kvm/Kconfig @@ -100,6 +100,10 @@ config KVM_GENERIC_MMU_NOTIFIER select MMU_NOTIFIER bool =20 +config KVM_ELIDE_TLB_FLUSH_IF_YOUNG + depends on KVM_GENERIC_MMU_NOTIFIER + bool + config KVM_GENERIC_MEMORY_ATTRIBUTES depends on KVM_GENERIC_MMU_NOTIFIER bool diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c index b1b10dc408a0..83b525d16b61 100644 --- a/virt/kvm/kvm_main.c +++ b/virt/kvm/kvm_main.c @@ -630,7 +630,8 @@ static __always_inline kvm_mn_ret_t __kvm_handle_hva_ra= nge(struct kvm *kvm, static __always_inline int kvm_handle_hva_range(struct mmu_notifier *mn, unsigned long start, unsigned long end, - gfn_handler_t handler) + gfn_handler_t handler, + bool flush_on_ret) { struct kvm *kvm =3D mmu_notifier_to_kvm(mn); const struct kvm_mmu_notifier_range range =3D { @@ -638,7 +639,7 @@ static __always_inline int kvm_handle_hva_range(struct = mmu_notifier *mn, .end =3D end, .handler =3D handler, .on_lock =3D (void *)kvm_null_fn, - .flush_on_ret =3D true, + .flush_on_ret =3D flush_on_ret, .may_block =3D false, }; =20 @@ -650,17 +651,7 @@ static __always_inline int kvm_handle_hva_range_no_flu= sh(struct mmu_notifier *mn unsigned long end, gfn_handler_t handler) { - struct kvm *kvm =3D mmu_notifier_to_kvm(mn); - const struct kvm_mmu_notifier_range range =3D { - .start =3D start, - .end =3D end, - .handler =3D handler, - .on_lock =3D (void *)kvm_null_fn, - .flush_on_ret =3D false, - .may_block =3D false, - }; - - return __kvm_handle_hva_range(kvm, &range).ret; + return kvm_handle_hva_range(mn, start, end, handler, false); } =20 void kvm_mmu_invalidate_begin(struct kvm *kvm) @@ -825,7 +816,8 @@ static int kvm_mmu_notifier_clear_flush_young(struct mm= u_notifier *mn, { trace_kvm_age_hva(start, end); =20 - return kvm_handle_hva_range(mn, start, end, kvm_age_gfn); + return kvm_handle_hva_range(mn, start, end, kvm_age_gfn, + !IS_ENABLED(CONFIG_KVM_ELIDE_TLB_FLUSH_IF_YOUNG)); } =20 static int kvm_mmu_notifier_clear_young(struct mmu_notifier *mn, --=20 2.47.0.rc1.288.g06298d1525-goog From nobody Tue Nov 26 08:38:40 2024 Received: from mail-pf1-f201.google.com (mail-pf1-f201.google.com [209.85.210.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 686DA1FBC8E for ; Fri, 11 Oct 2024 02:11:33 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728612694; cv=none; b=ozabo+SVDGMdrMWSiIyDzggpbsa8dgiHL6vUUlInav2dorSOMB/0iP8eJeKCD23+cJFZfkM5c0Aa1MqYLJpo2iyC1dgT5hLYj1sK5z2WS7kDi2p1jzhnQ0QYOMtWIq/xA31+Z+MTE/bXN5XH7NOry9bvKjjSSVk2Mp78FXwWXdE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728612694; c=relaxed/simple; bh=G5j/STg2XewWx6BKeygLMbbBozt/6/sTXHCfTmo3CC4=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=ZMomW0ByR6xSSrUC5M69C0JR0kZxS1Lrv34apZj95NyC0v4yjccd6P+sKkZX/hK0yzbsZzJPq65MefvosECXquqzh08XFVZ4NdhWarVWVScuXw9snjsqHtZdwuX5TMMFO7deZni5zuBfUu1Rts2FopzZ2sv2ZWcjIKLd9XH7yJs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=qnyU0Yxy; arc=none smtp.client-ip=209.85.210.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="qnyU0Yxy" Received: by mail-pf1-f201.google.com with SMTP id d2e1a72fcca58-71e04c42fecso1602484b3a.0 for ; Thu, 10 Oct 2024 19:11:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1728612693; x=1729217493; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=+DbRh+OQxnx8VQK332KGl4utXP+C7MDXmmXDxMg/Z/g=; b=qnyU0Yxy4PE13cSwLOBwthB7pVANlNz4yDsOvPN5bdOjr0dCA/rk/d0F1erhWPjw3p 5SI0bvERbLkPYRJ5H5PGh+3/2s0EeerZjJflaRGA0wEEc+C+Iz3Hvfp+Vjv1fKaDBPBr 4eNT+CnrFpa66Dl8pCcoCfcKNYix7A4dmi3T5tfh5W5jdV+ypniYWYI8oMI7tYlUeisz ZRkRKVZMNmx2O352eSDRys63l+LbZCslREIqmnU+WYQhEC1h6xnTZw12moE8OHYz7cOx MknSHaId7d4Sue6OQFMI2cvlln+7cKvJcQcjc/KPpvPA/UxbbfHc1NO7reU5MvhbDz2M yh8A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1728612693; x=1729217493; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=+DbRh+OQxnx8VQK332KGl4utXP+C7MDXmmXDxMg/Z/g=; b=sNtPmXzaHCEA5IRTnULyYaPGxtXmtwds9cVZpCvodq2dJWswGc5LnAaaORBF0pWlar MzjtOtdqOQ9eG4fCVXbwrwKUHQZYt/+SshR4qcuTBUlKKiTdnUBYoBFNYWI/n43dQgp+ vDAdnXmAMQTLywGKYH95/yYd90o4aCxsoy6uUwXLbH58j1C9jxJ4/InD+tZiwuB5cP4e d1AqFAC446V4QFtoOmHuCHp8R6GHfQ2hJFYm/auIPZ5Ki8/gQUmq4Prj4fkVBmBBjvXz UUuzBmgntr0R7tuPw8+BxNi+74Rts8wLOEkorbyKdy+ZrwabyhbLk0zoULO/nvnrn4ZX AJNA== X-Forwarded-Encrypted: i=1; AJvYcCXFBehnpaGgJ/F4YwWtj6CW/QYoApOCDuaMM9LBkdH7+YNHMtwBTD8gH0KaZAQcDVuBUiXmpNtPBPPucXg=@vger.kernel.org X-Gm-Message-State: AOJu0YynpgKbN6tBxwPKjTi70bkcqKbTJ5Z48A1qCd9yWrFKQmI4pLlT +lXBYRCTUz5E9wwt7IHTTx37/xhps6lsz3GSOU5Y1FMS2jxsGcADcbp798fsi0vhdXLu9zlUalp bSQ== X-Google-Smtp-Source: AGHT+IF5NMp0NI1CakamRfDZJw0W0RABI+lWlAVC7+uDMKwFg+XySu7vH0dBZjdw628m27BTqRJ51jdefc8= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:9d:3983:ac13:c240]) (user=seanjc job=sendgmr) by 2002:a05:6a00:6f44:b0:71e:268b:845e with SMTP id d2e1a72fcca58-71e26e53c16mr13451b3a.1.1728612692090; Thu, 10 Oct 2024 19:11:32 -0700 (PDT) Reply-To: Sean Christopherson Date: Thu, 10 Oct 2024 19:10:50 -0700 In-Reply-To: <20241011021051.1557902-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20241011021051.1557902-1-seanjc@google.com> X-Mailer: git-send-email 2.47.0.rc1.288.g06298d1525-goog Message-ID: <20241011021051.1557902-19-seanjc@google.com> Subject: [PATCH 18/18] KVM: x86: Don't emit TLB flushes when aging SPTEs for mmu_notifiers From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Yan Zhao , Sagi Shahar , "=?UTF-8?q?Alex=20Benn=C3=A9e?=" , David Matlack , James Houghton Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Follow x86's primary MMU, which hasn't flushed TLBs when clearing Accessed bits for 10+ years, and skip all TLB flushes when aging SPTEs in response to a clear_flush_young() mmu_notifier event. As documented in x86's ptep_clear_flush_young(), the probability and impact of "bad" reclaim due to stale A-bit information is relatively low, whereas the performance cost of TLB flushes is relatively high. I.e. the cost of flushing TLBs outweighs the benefits. On KVM x86, the cost of TLB flushes is even higher, as KVM doesn't batch TLB flushes for mmu_notifier events (KVM's mmu_notifier contract with MM makes it all but impossible), and sending IPIs forces all running vCPUs to go through a VM-Exit =3D> VM-Enter roundtrip. Furthermore, MGLRU aging of secondary MMUs is expected to use flush-less mmu_notifiers, i.e. flushing for the !MGLRU will make even less sense, and will be actively confusing as it wouldn't be clear why KVM "needs" to flush TLBs for legacy LRU aging, but not for MGLRU aging. Cc: James Houghton Cc: Yan Zhao Link: https://lore.kernel.org/all/20240926013506.860253-18-jthoughton@googl= e.com Signed-off-by: Sean Christopherson --- arch/x86/kvm/Kconfig | 1 + arch/x86/kvm/mmu/spte.h | 5 ++--- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/arch/x86/kvm/Kconfig b/arch/x86/kvm/Kconfig index f09f13c01c6b..1ed1e4f5d51c 100644 --- a/arch/x86/kvm/Kconfig +++ b/arch/x86/kvm/Kconfig @@ -22,6 +22,7 @@ config KVM_X86 depends on X86_LOCAL_APIC select KVM_COMMON select KVM_GENERIC_MMU_NOTIFIER + select KVM_ELIDE_TLB_FLUSH_IF_YOUNG select HAVE_KVM_IRQCHIP select HAVE_KVM_PFNCACHE select HAVE_KVM_DIRTY_RING_TSO diff --git a/arch/x86/kvm/mmu/spte.h b/arch/x86/kvm/mmu/spte.h index e90cc401c168..8b09a0d60ea6 100644 --- a/arch/x86/kvm/mmu/spte.h +++ b/arch/x86/kvm/mmu/spte.h @@ -470,9 +470,8 @@ static inline bool is_mmu_writable_spte(u64 spte) * needs to flush at the time the SPTEs is modified, before dropping mmu_l= ock. * * Don't flush if the Accessed bit is cleared, as access tracking tolerates - * false negatives, and the one path that does care about TLB flushes, - * kvm_mmu_notifier_clear_flush_young(), flushes if a young SPTE is found,= i.e. - * doesn't rely on lower helpers to detect the need to flush. + * false negatives, e.g. KVM x86 omits TLB flushes even when aging SPTEs f= or a + * mmu_notifier.clear_flush_young() event. * * Lastly, don't flush if the Dirty bit is cleared, as KVM unconditionally * flushes when enabling dirty logging (see kvm_mmu_slot_apply_flags()), a= nd --=20 2.47.0.rc1.288.g06298d1525-goog