From nobody Sun Feb 8 06:22:22 2026 Received: from mail-pj1-f44.google.com (mail-pj1-f44.google.com [209.85.216.44]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A04F8DF49 for ; Mon, 20 Jan 2025 01:22:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.44 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1737336178; cv=none; b=eHF6Lgfd86P5gVgPpk8/xsdr8EKxtbQ0edY0bPYgv8r4hZjN4y3uswq3vKHb8CW0H3z6Y9llWv0LB0qgXZADdDzJO2cAynZDkrLrZOjP2rZrzUNC691azmjf8c77jezpW2pvHeAquZzhy91NuVFrvfB+MlbBLIUi/a7CsNMe8ME= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1737336178; c=relaxed/simple; bh=vEdEYu8AP0CYrYbCgQmsRfBLY9BqVj+aEehVjHa63DM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=KNSh/GPujBAAnNmZpeS6beYskhUmywFaBvEVWIoUkEbvYnovKrXHFCyHOOSnqdYagkBG4nTshFg9syKy2bvttAAB76SpMYOJnLVY8wjAsYnXazYj6DY6Oh8nRrhV4NDbtMeji0H/Hh40cioRwO1/DpqwBM++P3j3lrjFr+/gwRM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=TBG1KWPR; arc=none smtp.client-ip=209.85.216.44 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="TBG1KWPR" Received: by mail-pj1-f44.google.com with SMTP id 98e67ed59e1d1-2ee9a780de4so4995954a91.3 for ; Sun, 19 Jan 2025 17:22:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1737336176; x=1737940976; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=zWHuTAWWoazPvHXkzMU7dW3hW1AXAy1GX7BhKkGbTX0=; b=TBG1KWPR38zflZHuguNm5pNpnda5PB9Ye4mWJdOK/MMy2TmfFSqc51TZuRUPW4IMk9 /Dyvaq/wqf6pp0KyWvAwZtJPZqwlHoL1zy4+Ypms0B7glkG/77v8M+6hsQ7NrQjn5Jb4 alRfLLB3Fywu1Gr/8I6JBAtfYRg5ZYEfcSTUOdPk6rh36QcDTR0C1W/thgVI/FbuDp+t Fw1wj11tel3gX4G/fmEAXfbpl3e16dApnmJSmrMaub/ryVbMqQT8PIqLYJnpRNtbCHV2 3TIANwpTDppqIH1zI1i5lli9NoYU9tUv/S9Cprl134Ciqg00IGMCYBxOroUaFQ9BGFV2 sp1A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1737336176; x=1737940976; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=zWHuTAWWoazPvHXkzMU7dW3hW1AXAy1GX7BhKkGbTX0=; b=GKu/2hs8ZVn9W2XBQDqIdELX2kkVzopIKKhRgG8+QnZcw/lkKyuXCNzKKtd3o1u3Bs 7iQ0VfuY7XmH3hdbjLwZlqs+XRNGbi+k8FZck5uuamyXfFN29QQuRLQE5bAAjQN1dSvM hS/2rwmswNgN4/fXuaqKBqVUHT124YUR1Lqnhb8/YuUwY4kSsNRKROhq4D4brfQPuszV q+j6YwTYK3W+NdSCh6a/387PeEGPd6Bsfjt5wbu+UDeHHHc3Oo+QCF/HssuH4K/hJpM2 uVTgZFFlZ3Bz3s9aO24XccwSq7jV7jr8pRiDQEuRxV5a8NrgZ6UZx2HYhMMzX5XjZ28G +kaA== X-Forwarded-Encrypted: i=1; AJvYcCVeYvRi3hfiFVcZZAYftVQUahn3+z07/JCGXYL+JrdVQQdsmT8/3ouHwGRjrMt6M88ky3p0NtZvlIFrKZk=@vger.kernel.org X-Gm-Message-State: AOJu0Yz0ETrAHNveFIkUxaxDo4B72FQ4/JtAZy+TNTHOvrEoX8x932+P ErXkx6AXOFjC5rbL/aZbX1FcrjrVCdWB0BRFh+yOuZZyoxMdXKfe X-Gm-Gg: ASbGncs4dx9hNOwGme02EU3xYHA8Fda047HinfG5N3vAfahfLDrBZIE95dYC3yojPS4 PCVybC2vIVbCQbTm1cYQA65rDuyQVy0lF0kIvy7gOk2wPZuqeG9D1xDE01t6ghLpFaSlLPJAoSr 9xLbQAScXSKNiU08iHswDvGZCxgCyTmnhrdTn8QQhOhz8/4iKPVhie+0EcLwuw+N2T37e1JlIsx whf2rmwgrRmFFyYB+XZ1Obz7DFpijCkZJsBmMAqf1qhUFnQojj2yO9TLg0vB1kmlstDnvUcebmR jg== X-Google-Smtp-Source: AGHT+IFNXuIjOpZfo/SccX3mXibk/Kq9SaUdfRtj6M+BjmKaE13u0uSI6XF6CkslRL76AcNd2Ep1XQ== X-Received: by 2002:a17:90b:1f8b:b0:2f6:539:3cd8 with SMTP id 98e67ed59e1d1-2f782ca2291mr17790924a91.18.1737336175844; Sun, 19 Jan 2025 17:22:55 -0800 (PST) Received: from EBJ9932692.tcent.cn ([43.134.20.116]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2f77629acd5sm6945514a91.37.2025.01.19.17.22.49 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Sun, 19 Jan 2025 17:22:55 -0800 (PST) From: Lance Yang To: akpm@linux-foundation.org Cc: 21cnbao@gmail.com, ryan.roberts@arm.com, dev.jain@arm.com, david@redhat.com, shy828301@gmail.com, ziy@nvidia.com, libang.li@antgroup.com, baolin.wang@linux.alibaba.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Lance Yang , Mingzhe Yang Subject: [RFC 1/2] mm/mthp: add pte_range_none_or_zeropfn() helper Date: Mon, 20 Jan 2025 09:22:29 +0800 Message-ID: <20250120012230.3343-2-ioworker0@gmail.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20250120012230.3343-1-ioworker0@gmail.com> References: <20250120012230.3343-1-ioworker0@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" In preparation for relaxing the mTHP PTE mapping restriction on the demand zero page. We intend to provide more opportunities to take advantage of mTHP. Signed-off-by: Mingzhe Yang Signed-off-by: Lance Yang --- mm/memory.c | 43 ++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 40 insertions(+), 3 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 2a20e3810534..4e148309b3e0 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -4724,18 +4724,55 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) return ret; } =20 -static bool pte_range_none(pte_t *pte, int nr_pages) +/* Flags for __pte_range_check(). */ +typedef int __bitwise prc_t; + +/* Check if PTEs are not present. */ +#define PRC_CHECK_NONE ((__force prc_t)BIT(0)) + +/* Check if PTEs are mapped to the zero page. */ +#define PRC_CHECK_ZEROPFN ((__force prc_t)BIT(1)) + +static bool __pte_range_check(pte_t *pte, int nr_pages, prc_t flags, + bool *any_zeropfn) { int i; + pte_t ptent; + + if (any_zeropfn) + *any_zeropfn =3D false; =20 for (i =3D 0; i < nr_pages; i++) { - if (!pte_none(ptep_get_lockless(pte + i))) - return false; + ptent =3D ptep_get_lockless(pte + i); + + if ((flags & PRC_CHECK_NONE) && pte_none(ptent)) + continue; + + if ((flags & PRC_CHECK_ZEROPFN) && + is_zero_pfn(pte_pfn(ptent))) { + if (any_zeropfn) + *any_zeropfn =3D true; + continue; + } + + return false; } =20 return true; } =20 +static inline bool pte_range_none(pte_t *pte, int nr_pages) +{ + return __pte_range_check(pte, nr_pages, PRC_CHECK_NONE, NULL); +} + +static inline bool pte_range_none_or_zeropfn(pte_t *pte, int nr_pages, + bool *any_zeropfn) +{ + return __pte_range_check( + pte, nr_pages, PRC_CHECK_NONE | PRC_CHECK_ZEROPFN, any_zeropfn); +} + static struct folio *alloc_anon_folio(struct vm_fault *vmf) { struct vm_area_struct *vma =3D vmf->vma; --=20 2.45.2 From nobody Sun Feb 8 06:22:22 2026 Received: from mail-pj1-f54.google.com (mail-pj1-f54.google.com [209.85.216.54]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 989B51CD2C for ; Mon, 20 Jan 2025 01:23:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.54 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1737336185; cv=none; b=Wem4tWZnqA+QFoGi968lm3RsU0hpM7dF/ccs2IWZUWy0mfIKRk46ebjLXxq2R4siK8+JS+vbwJs2JLZJBYlVXuAttxYXxJk62KrzGoPEzF9h5ErgLaZrwf3nii2AinHz2o7V9vLcpxShMw6Be34xHY+FHiPHpZb0DO3jbfUb3gc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1737336185; c=relaxed/simple; bh=tIim1MK7bI8I8SQoOmLD9o+/+C0UCC9cR/+uXy7nMtY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=TitbY7ow9vI/Xsc8D6a1dpNoqtSdKKIYDKxqV8k8WROXV4PLlNUJaQJ2qo7VoAHhR4wmF3eCXlqgLn3fuo3evPIuruBWaYfCpetNWPKZgrk/b+LWmRwL8zSnAopvdAmcOf3rQ1zpfDvQh2RBdpTAfHxiQIWseub+CPSfzEAE0eA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=ldaGLHEp; arc=none smtp.client-ip=209.85.216.54 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="ldaGLHEp" Received: by mail-pj1-f54.google.com with SMTP id 98e67ed59e1d1-2ef8c012913so5030810a91.3 for ; Sun, 19 Jan 2025 17:23:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1737336183; x=1737940983; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Hgy3I40fUOIkhs3A7OczOazm565K5yUyg2x9qlHIL+c=; b=ldaGLHEpTWz6mzwYoS9gLNE1dFhfPLBXtgfN45f75Y4JdC5qa6Dr5EdqS1GSBog4aD sPg0ho1X3wJbc6KV9OlGepSB6l8P3dc0YqYndYtJh73h4O8DMwNewDTJ957o+DSvs0Ub oo7/VdhUXNICw1dE80ugi2mqS23Iyj6Bj4g0zf8PsimfoMlMFFzWLL957G/LEiKE3I2T MzTzTtAYMGYZ03sdskvUFg9PP1Hc4jJ+OQ1gz8F7Uyi7E3eO3RtcYk4K/FiqXHc8hJ75 JMvhbyxxQ1BZIah8TdP65BaUbzdSH8XOzPaiK9Bw4uOO9N15VXM3VcYk0nvtNLuvOh1j bBjg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1737336183; x=1737940983; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Hgy3I40fUOIkhs3A7OczOazm565K5yUyg2x9qlHIL+c=; b=wmmTSlZe3E4SAI11h+jzcIaNLYgW6zA87JAKDQk4FPwveiX+wNIU45mIWGSV9gM1jR Q+vIF6leP1+8vr77TX94mZCrYKbWef/gFr/K1RN7z0tBgbwPVaFSOS+PjwbzXgeZNwZa Yy/2B7YpmGV81+Ess3bKhiQVW8uJpFjof+4otcSjiS0TZFxJPBf1D5alpxmc+1bpOshn BHGuiz93Q0is2XXt2ZZgNQ8VWaTVYySVrXUK96ufRlgH0H3FG3/xXW6h0Oq7OBHmk4pu 62nrjQr6t4rclpEoua5rT7CkI6DAN5tKmVryxlKwiJUSLNbO/RlFeas/hI7VWcx6CcM3 DDDQ== X-Forwarded-Encrypted: i=1; AJvYcCWoaJqTdeB9mbDecbhXIOTi8yUsiFkNgZQ2vyOlvuyy1MxRzUMz+WNJA3tWuOO39X+75fBbyYjwXN0+1a4=@vger.kernel.org X-Gm-Message-State: AOJu0YzdABCyMOwBlBvWPMGVjaJZ2km/+VPeItrqjM8spkUUEqjVpSNi znrUnnYBWH8kTfpotvwtfxC5QNvrZhQSBrxDr6mhar7Mm/8vqzpBvacQg2wH+fM= X-Gm-Gg: ASbGnctg5wXkISnZ3lf2xfhvLjHg4OY3s5/Cn+N9MRBR3cz/Z2+UDaS4PP0By2GDw3+ fT7ZWDgYJTKY8PpaXOSs2VYYolUDpWqqi1QF6RUveBlWDyk5QV9w5ElilmQkqVgyeIx1hmnjme8 q6akY3CMm72LH8PXSfNsjG3g0Hc+Y8WPQUGWdlzBvatzWgmDMXiQZVtUQSucmkcU06eJNzb2sPW A8ao7rJINz7itDPOiBz7zsrberXQxfvZkg25aQEktRwYmtO2TIF+vXZYjXb+x9P2PdxexXsEfES 4A== X-Google-Smtp-Source: AGHT+IGilBEXOdf30oMeYJSnhHL+cLRO5bpvVan9EzDDik+E4juIxcZXxIWdMZEkaiPwt2xug5cbNg== X-Received: by 2002:a17:90b:4ed0:b0:2ee:c1d2:bc67 with SMTP id 98e67ed59e1d1-2f782c99704mr17459100a91.16.1737336182714; Sun, 19 Jan 2025 17:23:02 -0800 (PST) Received: from EBJ9932692.tcent.cn ([43.134.20.116]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2f77629acd5sm6945514a91.37.2025.01.19.17.22.56 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Sun, 19 Jan 2025 17:23:02 -0800 (PST) From: Lance Yang To: akpm@linux-foundation.org Cc: 21cnbao@gmail.com, ryan.roberts@arm.com, dev.jain@arm.com, david@redhat.com, shy828301@gmail.com, ziy@nvidia.com, libang.li@antgroup.com, baolin.wang@linux.alibaba.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Lance Yang , Mingzhe Yang Subject: [RFC 2/2] mm/mthp: relax anon mTHP PTE Mapping restrictions Date: Mon, 20 Jan 2025 09:22:30 +0800 Message-ID: <20250120012230.3343-3-ioworker0@gmail.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20250120012230.3343-1-ioworker0@gmail.com> References: <20250120012230.3343-1-ioworker0@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Previously, mTHP could only be mapped to PTEs where all entries were none. With this change, PTEs within the range mapping the demand-zero page can now be treated as `pte_none` and remapped to a new mTHP, providing more opportunities to take advantage of mTHP. Signed-off-by: Mingzhe Yang Signed-off-by: Lance Yang --- mm/memory.c | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 4e148309b3e0..99ec75c6f0fe 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -4815,7 +4815,8 @@ static struct folio *alloc_anon_folio(struct vm_fault= *vmf) order =3D highest_order(orders); while (orders) { addr =3D ALIGN_DOWN(vmf->address, PAGE_SIZE << order); - if (pte_range_none(pte + pte_index(addr), 1 << order)) + if (pte_range_none_or_zeropfn(pte + pte_index(addr), 1 << order, + NULL)) break; order =3D next_order(&orders, order); } @@ -4867,6 +4868,7 @@ static vm_fault_t do_anonymous_page(struct vm_fault *= vmf) { struct vm_area_struct *vma =3D vmf->vma; unsigned long addr =3D vmf->address; + bool any_zeropfn =3D false; struct folio *folio; vm_fault_t ret =3D 0; int nr_pages =3D 1; @@ -4939,7 +4941,8 @@ static vm_fault_t do_anonymous_page(struct vm_fault *= vmf) if (nr_pages =3D=3D 1 && vmf_pte_changed(vmf)) { update_mmu_tlb(vma, addr, vmf->pte); goto release; - } else if (nr_pages > 1 && !pte_range_none(vmf->pte, nr_pages)) { + } else if (nr_pages > 1 && !pte_range_none_or_zeropfn( + vmf->pte, nr_pages, &any_zeropfn)) { update_mmu_tlb_range(vma, addr, vmf->pte, nr_pages); goto release; } @@ -4965,6 +4968,10 @@ static vm_fault_t do_anonymous_page(struct vm_fault = *vmf) entry =3D pte_mkuffd_wp(entry); set_ptes(vma->vm_mm, addr, vmf->pte, entry, nr_pages); =20 + /* At least one PTE was mapped to the zero page */ + if (nr_pages > 1 && any_zeropfn) + flush_tlb_range(vma, addr, addr + (nr_pages * PAGE_SIZE)); + /* No need to invalidate - it was non-present before */ update_mmu_cache_range(vmf, vma, addr, vmf->pte, nr_pages); unlock: --=20 2.45.2