From nobody Sun Apr 12 02:48:12 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3F726C19F2B for ; Mon, 1 Aug 2022 15:19:45 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233384AbiHAPTn (ORCPT ); Mon, 1 Aug 2022 11:19:43 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53536 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232357AbiHAPTl (ORCPT ); Mon, 1 Aug 2022 11:19:41 -0400 Received: from mail-pl1-x64a.google.com (mail-pl1-x64a.google.com [IPv6:2607:f8b0:4864:20::64a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C10712AC4F for ; Mon, 1 Aug 2022 08:19:40 -0700 (PDT) Received: by mail-pl1-x64a.google.com with SMTP id c15-20020a170902d48f00b0016c01db365cso7550508plg.20 for ; Mon, 01 Aug 2022 08:19:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:mime-version:message-id:date:from:to:cc; bh=S/2jI0VtfXi7m6ZMf+qwyQLgOYuMiCIT7wBl4PXRX+g=; b=OqEy/n2odJcpLuQeQrEW3uvv4W6U5wbHVoh+YVePDwAMenajqwhMu4ssIBWUMxydvo PKYVsYiXs6raDPcmybsCA24N7Xr/PKhc3cv0Vwr6BMp4HyuldkXfgDcjH9L3QbzEG3q5 bnNJS2cnVbxfJyehbXL9D+5LzG6I9WBICeXtTSON6X+SPopmA/aqmPqDBG7bmgqEp6PG 5KtSDvU7j4UB39QtuZW1XmQ721oGR4wdvlfPYnTwC+l7niSzawHFaz2uGJslDdVkhgFL 9D7Sf0+Iz1Jj8kgeI9C//InH/tv88mSYgGkCu1saaRUz09yZ27m4tFnY+LnZQP+AJSP8 CdjA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:mime-version:message-id:date:x-gm-message-state :from:to:cc; bh=S/2jI0VtfXi7m6ZMf+qwyQLgOYuMiCIT7wBl4PXRX+g=; b=zDkSnsKYOBh4pNWnFgp7Qo2AReof2+Oc8bl6V+fOmOJQl5/FIHUOSt7MrOMMgx+Lz7 1VkM+wrr/pkwUmtR58vzpJDPXf459JQ9qjdiFDx6x5JlcTXfDgCbMkhG/Bj0zF3hYv/m kaNITHT4i4cOnCC7tgM9KZZbfYfAx2jR7YZzL6JW0O3A8F1+8mLY2bPexpiI2eRJCIlJ cUWQm9LhnVgfDe78Q8LgJv+cbRAe+4PEEnNDhFKK3sUpe/4vkeby4hsukfLj/q2CnV/6 94VzeroFi6hj3OU0h832aIAEQx+5bSF8i9XPddlm3Ci5Je/42NevIGEnEdMxmWBmgQ7M qmfg== X-Gm-Message-State: AJIora9m+ZtVt8CPyGah64WAEqxalESmAD8i0GdGeWDwi0g9tuM6aURn M621IOO2esqgwkjWrHYXIVB6BBaDeYzE X-Google-Smtp-Source: AGRyM1tmitgRItVDG9wYYQJP7lV6j2QhkGGFMfWO4zg8l7GgEPHyefkScB+izDXwBytqYaqte5+sZLs4g3s8 X-Received: from vipin.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:479f]) (user=vipinsh job=sendgmr) by 2002:a63:6c45:0:b0:419:b668:6d with SMTP id h66-20020a636c45000000b00419b668006dmr13395927pgc.548.1659367180173; Mon, 01 Aug 2022 08:19:40 -0700 (PDT) Date: Mon, 1 Aug 2022 08:19:28 -0700 Message-Id: <20220801151928.270380-1-vipinsh@google.com> Mime-Version: 1.0 X-Mailer: git-send-email 2.37.1.455.g008518b4e5-goog Subject: [PATCH] KVM: x86/mmu: Make page tables for eager page splitting NUMA aware From: Vipin Sharma To: seanjc@google.com, dmatlack@google.com, pbonzini@redhat.com Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Vipin Sharma Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" tdp_mmu_alloc_sp_for_split() allocates page tables for Eager Page Splitting. Currently it does not specify a NUMA node preference, so it will try to allocate from the local node. The thread doing eager page splitting is supplied by the userspace and may not be running on the same node where it would be best for page tables to be allocated. We can improve TDP MMU eager page splitting by making tdp_mmu_alloc_sp_for_split() NUMA-aware. Specifically, when splitting a huge page, allocate the new lower level page tables on the same node as the huge page. __get_free_page() is replaced by alloc_page_nodes(). This introduces two functional changes. 1. __get_free_page() removes gfp flag __GFP_HIGHMEM via call to __get_free_pages(). This should not be an issue as __GFP_HIGHMEM flag is not passed in tdp_mmu_alloc_sp_for_split() anyway. 2. __get_free_page() calls alloc_pages() and use thread's mempolicy for the NUMA node allocation. From this commit, thread's mempolicy will not be used and first preference will be to allocate on the node where huge page was present. dirty_log_perf_test for 416 vcpu and 1GB/vcpu configuration on a 8 NUMA node machine showed dirty memory time improvements between 2% - 35% in multiple runs. Suggested-by: David Matlack Signed-off-by: Vipin Sharma --- arch/x86/kvm/mmu/tdp_mmu.c | 24 +++++++++++++++++++----- 1 file changed, 19 insertions(+), 5 deletions(-) diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c index bf2ccf9debcaa..1e30e18fc6a03 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.c +++ b/arch/x86/kvm/mmu/tdp_mmu.c @@ -1402,9 +1402,19 @@ bool kvm_tdp_mmu_wrprot_slot(struct kvm *kvm, return spte_set; } =20 -static struct kvm_mmu_page *__tdp_mmu_alloc_sp_for_split(gfp_t gfp) +/* + * Caller's responsibility to pass a valid spte which has the shadow page + * present. + */ +static int tdp_mmu_spte_to_nid(u64 spte) +{ + return page_to_nid(pfn_to_page(spte_to_pfn(spte))); +} + +static struct kvm_mmu_page *__tdp_mmu_alloc_sp_for_split(int nid, gfp_t gf= p) { struct kvm_mmu_page *sp; + struct page *spt_page; =20 gfp |=3D __GFP_ZERO; =20 @@ -1412,11 +1422,12 @@ static struct kvm_mmu_page *__tdp_mmu_alloc_sp_for_= split(gfp_t gfp) if (!sp) return NULL; =20 - sp->spt =3D (void *)__get_free_page(gfp); - if (!sp->spt) { + spt_page =3D alloc_pages_node(nid, gfp, 0); + if (!spt_page) { kmem_cache_free(mmu_page_header_cache, sp); return NULL; } + sp->spt =3D page_address(spt_page); =20 return sp; } @@ -1426,6 +1437,9 @@ static struct kvm_mmu_page *tdp_mmu_alloc_sp_for_spli= t(struct kvm *kvm, bool shared) { struct kvm_mmu_page *sp; + int nid; + + nid =3D tdp_mmu_spte_to_nid(iter->old_spte); =20 /* * Since we are allocating while under the MMU lock we have to be @@ -1436,7 +1450,7 @@ static struct kvm_mmu_page *tdp_mmu_alloc_sp_for_spli= t(struct kvm *kvm, * If this allocation fails we drop the lock and retry with reclaim * allowed. */ - sp =3D __tdp_mmu_alloc_sp_for_split(GFP_NOWAIT | __GFP_ACCOUNT); + sp =3D __tdp_mmu_alloc_sp_for_split(nid, GFP_NOWAIT | __GFP_ACCOUNT); if (sp) return sp; =20 @@ -1448,7 +1462,7 @@ static struct kvm_mmu_page *tdp_mmu_alloc_sp_for_spli= t(struct kvm *kvm, write_unlock(&kvm->mmu_lock); =20 iter->yielded =3D true; - sp =3D __tdp_mmu_alloc_sp_for_split(GFP_KERNEL_ACCOUNT); + sp =3D __tdp_mmu_alloc_sp_for_split(nid, GFP_KERNEL_ACCOUNT); =20 if (shared) read_lock(&kvm->mmu_lock); --=20 2.37.1.455.g008518b4e5-goog