From nobody Tue Nov 26 08:39:04 2024 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 330011F12E9 for ; Fri, 11 Oct 2024 02:11:30 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728612691; cv=none; b=BMCOPzF7k+C72of7bT2laf9sZZl0CTLV5BwsT5mrPq4kmcL7mUPljCa3tzletSjbTyAX1Orwjp8TttJuZNeSNn1CrxHqPQTXztTydmQA3QXE1YgCXvgU6JYs5mcdUFrGUGtdD5ILdSBcIHpqEns2bPUgwAlUUXyFPj9GcNUs7uE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728612691; c=relaxed/simple; bh=iyJm/h+JOO59f+egBKANfLu+0QMr2pI0zJeagTYRCD4=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=tQKpgBJTPS/Op8yz5LvFEx0Q9p5JtW5NZQftg6lE+NtrwE7JXC/ScfiFaC1ihQx3I00CuH2X2iLyx4UwbmDDDqdJrt0R1XwTo2k6nIlBZ6r+cp9E2AWRElr0BD1apfmTbZyCmQR0nUKmsJBOfTMllkkCS28ZVsDYRpCfJENMQJs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=PYPMO+dI; arc=none smtp.client-ip=209.85.128.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="PYPMO+dI" Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-6e31e5d1739so30735237b3.1 for ; Thu, 10 Oct 2024 19:11:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1728612689; x=1729217489; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=HPDDwUN824pfD2nj2HWQd9wsiBjvr8+g8+t2fSbMNlA=; b=PYPMO+dIz2MEJ9uWGnpzMDLoKh+UuyJIyZmNXuqbSiVOdTd214fzLaTbR1r/huZV6G gUCNMV6B52TgkvmtpjDNuduzkb92NtWqgasAaY+wAZYjvZ0H5I9C16O55KtJTEiAAvpc 9QHy/RtL6SGjpF98SSBX5CaAtlaI34+Y9n5jvCtnuLGPW3j/w5bmER0tDHQPup2InfJh 8NLMGe2+b0hFnbp0glpF33KZbnMpiOn7DpgyppnLX8VdkE3Eb2RDJQoxANgT2bQBpeTX hqnp3IwrCFkL1iBeNy+pL7ILktxKp96OaGtltMGYnfnPKp6V53PPQF4zzD2T5a5G9g3r F32w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1728612689; x=1729217489; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=HPDDwUN824pfD2nj2HWQd9wsiBjvr8+g8+t2fSbMNlA=; b=Ycyv2cUA0xGa/nYm8kw1lFcq3AnnxL6jmEK8qzKNBH32GETMSLWQnmARlFwRGu4vf2 sRlwn8z5o7wLJptIYTtm2c0zH3oiFZbclHDEEryCzKQQCb3g+wzmIFcffXbq+ioLw5ha VBv2OL7u0770NcmY/kDfTsrDgNd1bhtOnKMDqtoG8vGKn5beMmt0uZSGk72GGxqp8YrI 3Z6hNKo5f28DCoDkgvzQT+ePjbjX1euT4E0uS4nKPgn1ef09LnB9xBwsCXOh6132euES SsV0sLZcs+NquglqzSDKxvMrvWvnT7hUbIfURILz/4bgcWBHgf6/MA29mZERmKVxcdaq KmEw== X-Forwarded-Encrypted: i=1; AJvYcCVrZRZf7N+VXE6EEfmuuOw5YykBP8wyhoOFtdhKJsD7cwoGfcfZeDvuGVjI7aVh2Co91pkXUWB4ybf8ckQ=@vger.kernel.org X-Gm-Message-State: AOJu0YwEWp4xtz7PLqk70MQtCRg/CRgEeTLcsNyEN3tzaQnOzZRWZK+B O9tzue8J/vrX+vSPU4gcT1VfApirbm09xCxQZWTKuyPzoAC0tydD0GsjAzZhUnpScF4CfgRf5/G KPA== X-Google-Smtp-Source: AGHT+IGTSYki/MPVOSXtSoZ4BErjm1NrfbNeO31MKvV+62YjjXjQGEoBGgc7gkTIPIKCnszTY2Gpwqbhv0k= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:9d:3983:ac13:c240]) (user=seanjc job=sendgmr) by 2002:a25:d346:0:b0:e17:8e4f:981a with SMTP id 3f1490d57ef6-e291a32c730mr5851276.11.1728612688656; Thu, 10 Oct 2024 19:11:28 -0700 (PDT) Reply-To: Sean Christopherson Date: Thu, 10 Oct 2024 19:10:49 -0700 In-Reply-To: <20241011021051.1557902-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20241011021051.1557902-1-seanjc@google.com> X-Mailer: git-send-email 2.47.0.rc1.288.g06298d1525-goog Message-ID: <20241011021051.1557902-18-seanjc@google.com> Subject: [PATCH 17/18] KVM: Allow arch code to elide TLB flushes when aging a young page From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Yan Zhao , Sagi Shahar , "=?UTF-8?q?Alex=20Benn=C3=A9e?=" , David Matlack , James Houghton Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add a Kconfig to allow architectures to opt-out of a TLB flush when a young page is aged, as invalidating TLB entries is not functionally required on most KVM-supported architectures. Stale TLB entries can result in false negatives and theoretically lead to suboptimal reclaim, but in practice all observations have been that the performance gained by skipping TLB flushes outweighs any performance lost by reclaiming hot pages. E.g. the primary MMUs for x86 RISC-V, s390, and PPC Book3S elide the TLB flush for ptep_clear_flush_young(), and arm64's MMU skips the trailing DSB that's required for ordering (presumably because there are optimizations related to eliding other TLB flushes when doing make-before-break). Signed-off-by: Sean Christopherson --- virt/kvm/Kconfig | 4 ++++ virt/kvm/kvm_main.c | 20 ++++++-------------- 2 files changed, 10 insertions(+), 14 deletions(-) diff --git a/virt/kvm/Kconfig b/virt/kvm/Kconfig index fd6a3010afa8..54e959e7d68f 100644 --- a/virt/kvm/Kconfig +++ b/virt/kvm/Kconfig @@ -100,6 +100,10 @@ config KVM_GENERIC_MMU_NOTIFIER select MMU_NOTIFIER bool =20 +config KVM_ELIDE_TLB_FLUSH_IF_YOUNG + depends on KVM_GENERIC_MMU_NOTIFIER + bool + config KVM_GENERIC_MEMORY_ATTRIBUTES depends on KVM_GENERIC_MMU_NOTIFIER bool diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c index b1b10dc408a0..83b525d16b61 100644 --- a/virt/kvm/kvm_main.c +++ b/virt/kvm/kvm_main.c @@ -630,7 +630,8 @@ static __always_inline kvm_mn_ret_t __kvm_handle_hva_ra= nge(struct kvm *kvm, static __always_inline int kvm_handle_hva_range(struct mmu_notifier *mn, unsigned long start, unsigned long end, - gfn_handler_t handler) + gfn_handler_t handler, + bool flush_on_ret) { struct kvm *kvm =3D mmu_notifier_to_kvm(mn); const struct kvm_mmu_notifier_range range =3D { @@ -638,7 +639,7 @@ static __always_inline int kvm_handle_hva_range(struct = mmu_notifier *mn, .end =3D end, .handler =3D handler, .on_lock =3D (void *)kvm_null_fn, - .flush_on_ret =3D true, + .flush_on_ret =3D flush_on_ret, .may_block =3D false, }; =20 @@ -650,17 +651,7 @@ static __always_inline int kvm_handle_hva_range_no_flu= sh(struct mmu_notifier *mn unsigned long end, gfn_handler_t handler) { - struct kvm *kvm =3D mmu_notifier_to_kvm(mn); - const struct kvm_mmu_notifier_range range =3D { - .start =3D start, - .end =3D end, - .handler =3D handler, - .on_lock =3D (void *)kvm_null_fn, - .flush_on_ret =3D false, - .may_block =3D false, - }; - - return __kvm_handle_hva_range(kvm, &range).ret; + return kvm_handle_hva_range(mn, start, end, handler, false); } =20 void kvm_mmu_invalidate_begin(struct kvm *kvm) @@ -825,7 +816,8 @@ static int kvm_mmu_notifier_clear_flush_young(struct mm= u_notifier *mn, { trace_kvm_age_hva(start, end); =20 - return kvm_handle_hva_range(mn, start, end, kvm_age_gfn); + return kvm_handle_hva_range(mn, start, end, kvm_age_gfn, + !IS_ENABLED(CONFIG_KVM_ELIDE_TLB_FLUSH_IF_YOUNG)); } =20 static int kvm_mmu_notifier_clear_young(struct mmu_notifier *mn, --=20 2.47.0.rc1.288.g06298d1525-goog