From nobody Sun Oct 5 01:46:23 2025 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EEF572D320E for ; Mon, 11 Aug 2025 11:26:40 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754911602; cv=none; b=Oi4yQu0lj+WGqF4bZV8IpJHFpcQWOS03TI6iEHCLQ2XsyI5xSj6AxuYoafxvr8O87NdtpVeOn6+Vz9ASzeUOuLMk51LKFlw+r2FehX//XfrOj8x9xgMW/nwdsWGWh/o2UT1S2uB6QpmdYvd5PkGw2dYIsZQKiOxuEbr9UwALl04= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754911602; c=relaxed/simple; bh=ZGAnHpfBp1WZBWcLrOrxfVSmjre0MJHRGXddITS2PNc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=O789J7plNYybXZ0vfPYQ4nB2alsiWUkSb4LQiW4dTBNvmQfAXl9WKKBtASLp5MocwRrL9GhdQvyKsgLPKvLy9kC3DbKKfb4NkrYIGHg+6FSwh/IlOdX/m0hAItKTBwjRzmh3IP8WyECJOo4M/vxdcUUzCs957FpOnZDsDhlkMqo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=WPLysArl; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="WPLysArl" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1754911600; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Htya8lk0ohNZGoVKNkWH4Tb4G28S+mHG9GYTekXNGbw=; b=WPLysArl9RbOnsdiMOliEdoWrS/M+ImZ5BlnmQTHqUg9IGLzCZI1pt8Ra/2ZHOFTDdt5sW xgkFjt6v3dzy265ejq0/wAgqs++3JraZLPP4LQyscr/y8AZv4yZXqwa7fgyFyLBeTVivru bZZsbHetEnmF4cWu+OTGU4qx7YSyNDs= Received: from mail-wm1-f70.google.com (mail-wm1-f70.google.com [209.85.128.70]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-183-SF9kBDfoMiiMpSOC-Mu6FQ-1; Mon, 11 Aug 2025 07:26:38 -0400 X-MC-Unique: SF9kBDfoMiiMpSOC-Mu6FQ-1 X-Mimecast-MFC-AGG-ID: SF9kBDfoMiiMpSOC-Mu6FQ_1754911598 Received: by mail-wm1-f70.google.com with SMTP id 5b1f17b1804b1-459d6c83e2eso23989455e9.1 for ; Mon, 11 Aug 2025 04:26:38 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1754911597; x=1755516397; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Htya8lk0ohNZGoVKNkWH4Tb4G28S+mHG9GYTekXNGbw=; b=BOvPrRAkOJYCSzjTTYwJX5MvBFLWq1pRQuUtQWFSuu8WG2ERWNJHCGv6ZiCbCZjGXY sYk+VoMhSPb4OKXdLkC0TEIdyH5/vL+Y/iZkZ2X/7DNW1WjULV2U6UnygLatbbEn1LXr itzRILq5fTFVvLN4Vj7tPJWbIPAsfXok3J91Zo+36/TWafZIGtCVZRGgFwcMPfIMsXqL yy8B0ItxFbsFETiRyLX3ilfmeoGIGpvrCkq29mtBEYjr9uj7PSkWRSy4TaS+qxaLKgzk DvqH8SHoTQm6BVkGpTKxyp8Eugw5ZK0JzF83G5X176TwQpvc8BgEBUfmWazd6cUdCW/N 4hcw== X-Gm-Message-State: AOJu0YwdAqtLTblYyllGAQNYGFPar+JyCqSVEW4qr906600dYfcxxe2s GyeWMaaTALIWmcjtDFNQxp9a15JY62v4GfeIdrUwdu/V4jfoKQ/AlDyLWlysDPztZdawiaUqDaX wTaQC89zzXbDkQ2L+82yNdnv+R/rJHxsynmK1fHeaaZFwEhyRqjkkUlcs2822VCwfK7K5RGP5mS rahfjo0u38BhrQhZCEHjud7QMacdJVE6z+MrF9MNwmd5v/Cg== X-Gm-Gg: ASbGncuMivYKpz4Kju72mrE11kOmETvKYygDfRR0KADyfShOHNM4Vmk9NP8hiVY3zNt iFoEcyONAjc8EjVi9zQdUb0BQEs7kMZzB+nuIFiLcRcNwlar8wLP0Rj/1MxMjFM/U6bLNG8A7ss 4UkvSsrWA6LYcFrmckcEUB96OvZbhoQgqbjHWCj2lYq3jKZV2ILQ50v0W6MhBH/Anui8ltXIPuI BTU8SSK9diGvflK/I0G9jvOxRoicFDRzt151xCm3yGRwINXCOwE0vJMi56SYJ3gdHo1Z1IKYa1u A3nfUULNdWQ9ImSHJqiKklD5oDnYLjR4fLNaWPJQlmxRuzwBvI4SEIjTIA3zRG8m0YNAvacJX2L GANcrARa0MlIWFCfnaz8VtWws X-Received: by 2002:a05:600c:458b:b0:459:e3f8:92ec with SMTP id 5b1f17b1804b1-459f4eb4176mr127958165e9.10.1754911597573; Mon, 11 Aug 2025 04:26:37 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGdNxb5gc4DwQKatpkXot61uX7cGVvKpORGwTw8/tHuwPs0r2lWbt+TytrRicLCoC+zeQmPIg== X-Received: by 2002:a05:600c:458b:b0:459:e3f8:92ec with SMTP id 5b1f17b1804b1-459f4eb4176mr127957415e9.10.1754911596965; Mon, 11 Aug 2025 04:26:36 -0700 (PDT) Received: from localhost (p200300d82f06a600a397de1d2f8bb66f.dip0.t-ipconnect.de. [2003:d8:2f06:a600:a397:de1d:2f8b:b66f]) by smtp.gmail.com with UTF8SMTPSA id ffacd0b85a97d-3b79c3abf33sm40550137f8f.7.2025.08.11.04.26.35 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 11 Aug 2025 04:26:36 -0700 (PDT) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, xen-devel@lists.xenproject.org, linux-fsdevel@vger.kernel.org, nvdimm@lists.linux.dev, linuxppc-dev@lists.ozlabs.org, David Hildenbrand , Andrew Morton , Madhavan Srinivasan , Michael Ellerman , Nicholas Piggin , Christophe Leroy , Juergen Gross , Stefano Stabellini , Oleksandr Tyshchenko , Dan Williams , Matthew Wilcox , Jan Kara , Alexander Viro , Christian Brauner , Lorenzo Stoakes , "Liam R. Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Zi Yan , Baolin Wang , Nico Pache , Ryan Roberts , Dev Jain , Barry Song , Jann Horn , Pedro Falcato , Hugh Dickins , Oscar Salvador , Lance Yang , Alistair Popple , Wei Yang Subject: [PATCH v3 01/11] mm/huge_memory: move more common code into insert_pmd() Date: Mon, 11 Aug 2025 13:26:21 +0200 Message-ID: <20250811112631.759341-2-david@redhat.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20250811112631.759341-1-david@redhat.com> References: <20250811112631.759341-1-david@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Let's clean it all further up. No functional change intended. Reviewed-by: Oscar Salvador Reviewed-by: Alistair Popple Reviewed-by: Lorenzo Stoakes Reviewed-by: Wei Yang Signed-off-by: David Hildenbrand Reviewed-by: Lance Yang --- mm/huge_memory.c | 72 ++++++++++++++++-------------------------------- 1 file changed, 24 insertions(+), 48 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 2b4ea5a2ce7d2..5314a89d676f1 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1379,15 +1379,25 @@ struct folio_or_pfn { bool is_folio; }; =20 -static int insert_pmd(struct vm_area_struct *vma, unsigned long addr, +static vm_fault_t insert_pmd(struct vm_area_struct *vma, unsigned long add= r, pmd_t *pmd, struct folio_or_pfn fop, pgprot_t prot, - bool write, pgtable_t pgtable) + bool write) { struct mm_struct *mm =3D vma->vm_mm; + pgtable_t pgtable =3D NULL; + spinlock_t *ptl; pmd_t entry; =20 - lockdep_assert_held(pmd_lockptr(mm, pmd)); + if (addr < vma->vm_start || addr >=3D vma->vm_end) + return VM_FAULT_SIGBUS; =20 + if (arch_needs_pgtable_deposit()) { + pgtable =3D pte_alloc_one(vma->vm_mm); + if (!pgtable) + return VM_FAULT_OOM; + } + + ptl =3D pmd_lock(mm, pmd); if (!pmd_none(*pmd)) { const unsigned long pfn =3D fop.is_folio ? folio_pfn(fop.folio) : fop.pfn; @@ -1395,15 +1405,14 @@ static int insert_pmd(struct vm_area_struct *vma, u= nsigned long addr, if (write) { if (pmd_pfn(*pmd) !=3D pfn) { WARN_ON_ONCE(!is_huge_zero_pmd(*pmd)); - return -EEXIST; + goto out_unlock; } entry =3D pmd_mkyoung(*pmd); entry =3D maybe_pmd_mkwrite(pmd_mkdirty(entry), vma); if (pmdp_set_access_flags(vma, addr, pmd, entry, 1)) update_mmu_cache_pmd(vma, addr, pmd); } - - return -EEXIST; + goto out_unlock; } =20 if (fop.is_folio) { @@ -1424,11 +1433,17 @@ static int insert_pmd(struct vm_area_struct *vma, u= nsigned long addr, if (pgtable) { pgtable_trans_huge_deposit(mm, pmd, pgtable); mm_inc_nr_ptes(mm); + pgtable =3D NULL; } =20 set_pmd_at(mm, addr, pmd, entry); update_mmu_cache_pmd(vma, addr, pmd); - return 0; + +out_unlock: + spin_unlock(ptl); + if (pgtable) + pte_free(mm, pgtable); + return VM_FAULT_NOPAGE; } =20 /** @@ -1450,9 +1465,6 @@ vm_fault_t vmf_insert_pfn_pmd(struct vm_fault *vmf, u= nsigned long pfn, struct folio_or_pfn fop =3D { .pfn =3D pfn, }; - pgtable_t pgtable =3D NULL; - spinlock_t *ptl; - int error; =20 /* * If we had pmd_special, we could avoid all these restrictions, @@ -1464,25 +1476,9 @@ vm_fault_t vmf_insert_pfn_pmd(struct vm_fault *vmf, = unsigned long pfn, (VM_PFNMAP|VM_MIXEDMAP)); BUG_ON((vma->vm_flags & VM_PFNMAP) && is_cow_mapping(vma->vm_flags)); =20 - if (addr < vma->vm_start || addr >=3D vma->vm_end) - return VM_FAULT_SIGBUS; - - if (arch_needs_pgtable_deposit()) { - pgtable =3D pte_alloc_one(vma->vm_mm); - if (!pgtable) - return VM_FAULT_OOM; - } - pfnmap_setup_cachemode_pfn(pfn, &pgprot); =20 - ptl =3D pmd_lock(vma->vm_mm, vmf->pmd); - error =3D insert_pmd(vma, addr, vmf->pmd, fop, pgprot, write, - pgtable); - spin_unlock(ptl); - if (error && pgtable) - pte_free(vma->vm_mm, pgtable); - - return VM_FAULT_NOPAGE; + return insert_pmd(vma, addr, vmf->pmd, fop, pgprot, write); } EXPORT_SYMBOL_GPL(vmf_insert_pfn_pmd); =20 @@ -1491,35 +1487,15 @@ vm_fault_t vmf_insert_folio_pmd(struct vm_fault *vm= f, struct folio *folio, { struct vm_area_struct *vma =3D vmf->vma; unsigned long addr =3D vmf->address & PMD_MASK; - struct mm_struct *mm =3D vma->vm_mm; struct folio_or_pfn fop =3D { .folio =3D folio, .is_folio =3D true, }; - spinlock_t *ptl; - pgtable_t pgtable =3D NULL; - int error; - - if (addr < vma->vm_start || addr >=3D vma->vm_end) - return VM_FAULT_SIGBUS; =20 if (WARN_ON_ONCE(folio_order(folio) !=3D PMD_ORDER)) return VM_FAULT_SIGBUS; =20 - if (arch_needs_pgtable_deposit()) { - pgtable =3D pte_alloc_one(vma->vm_mm); - if (!pgtable) - return VM_FAULT_OOM; - } - - ptl =3D pmd_lock(mm, vmf->pmd); - error =3D insert_pmd(vma, addr, vmf->pmd, fop, vma->vm_page_prot, - write, pgtable); - spin_unlock(ptl); - if (error && pgtable) - pte_free(mm, pgtable); - - return VM_FAULT_NOPAGE; + return insert_pmd(vma, addr, vmf->pmd, fop, vma->vm_page_prot, write); } EXPORT_SYMBOL_GPL(vmf_insert_folio_pmd); =20 --=20 2.50.1 From nobody Sun Oct 5 01:46:23 2025 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4E1782D29C6 for ; Mon, 11 Aug 2025 11:26:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754911604; cv=none; b=VPx+WvD7OnfqXyQ92DcRiXDee0lz768VhhcBDwOp8Jw3x+w7qJq6SP84e9WUWyoaE+0NYeuhlxgFhuzSYevanZn9/xNOp0TbAunAcxmAbLoq3qdBwqHAHoZGjI2+gY2HrV7WeQDYb76sMen7SiIITV5rohhyzJny6SLo8d50fUI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754911604; c=relaxed/simple; bh=bM37QlmnrcS12YEaYV2r5fiBbtnDtVLS/N1j1vpTu04=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=hf0hOIXKVUFe8ne2If/dWwyD96XUSRp+1aN7YK5y8Ecd//wHPIixkoG1dCHZ02URsbs3tUPE+ynkbCbqs152pSASE9lYjIvF4AffbNEO46v6xgRGJ3aEdyM0a9xtJYsuNsiQoIVFsmsfHh/x0R3uwNzZMgsfNbUL9itDo9tJFto= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=SaKfx++Z; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="SaKfx++Z" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1754911602; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=1yyyzSa9KFCInkfs6rUjYFvQXjYygLyaHB4l27+lC7g=; b=SaKfx++ZYkVLtl26/XyEOPgyWYWRMfsltIJPjxt2BPn0oAFSxsHjCIqMfT/InpSC8wipWg fRt+aDvgAUweSFf1EP1E34zTkkTJgui0Reh0OyYCV4LLzIrsts6VZ20WNIFhpJQSIQ8TkA aT9yZyGn27Uf+/4TZlWFDc8y4SuA3FQ= Received: from mail-wr1-f72.google.com (mail-wr1-f72.google.com [209.85.221.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-329-VPKi83k2Nd-WhscmczlcAA-1; Mon, 11 Aug 2025 07:26:41 -0400 X-MC-Unique: VPKi83k2Nd-WhscmczlcAA-1 X-Mimecast-MFC-AGG-ID: VPKi83k2Nd-WhscmczlcAA_1754911600 Received: by mail-wr1-f72.google.com with SMTP id ffacd0b85a97d-3b604541741so2205049f8f.3 for ; Mon, 11 Aug 2025 04:26:41 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1754911600; x=1755516400; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=1yyyzSa9KFCInkfs6rUjYFvQXjYygLyaHB4l27+lC7g=; b=V2K2H+XX7s2w1qEJOpHsB+BG+eFj37bzQk30Og8ir705yrg9nsHyOa5Disg/Ownw9U QaRoNB/VWEdZvHNL7tqk+LFenfaiRNmqqdq5wjKGT5R4gRIwmnXl/45zGfYZMAT0GyIG EMCEqtHAAGnVmZxgq3IHutjCR3KPRuCBE2zob9rYrppLl2I/TJqJA+fjsoes0bfeU43G VzWgjUVjH5kpKK/zzASaZ4MN43uvOoNBWYzGAHpMBcfiR24hbqo3bkNnH4HaDWamcU0J N0CZRA3sd7wSGN0cxeHqKIKTW1K/WS/85xU3svjEt1IJXtTkbVHV6LgUQR0kePmUoj70 sQZw== X-Gm-Message-State: AOJu0Yz4B53tC7/GMz6QIWUIXub05vHhCaNRXjL9SvgQoSBcrmjPc9ix Y8Va5f4MXAtHCrD8HWUTGAuZTUn9XsqabSv1OebKeKtSs1Kd4mDoVExQZhQYB2XpStXgatmcP8v mM6lH0RpufcffWDb+QA1xXRLPQI+cVuGsu/kJAUt4mQRFDRujKZloEm+vvrO6sBU+uqeIvDLcAl ZmBkwLrFzT+MI8ks4O8g3cN3Q0zN/P+SVGNIAFp9diS53e2g== X-Gm-Gg: ASbGncu3i/CTRXBawNKttbTIjeqDmuv5taU1568MEPCQzDZ2D/8IKyAQcCUs1ns3sVN SIA/3E75xrkArBKlofOaqnqW1iNn3A76rpKkZ/wtgKuw8JWUTpFt8AAyzk75T+9gu1BvC2iPEt5 y7sSURAGxKa4QPKPX7mViQ2rog658ZHcH7mOeEKc0zefTGrhJL2vuJrezKMyM9mBDRrxQzB4Jvt siYpTDMPklEgsRfFztnzmEDzzJTItfQn3m2vlK+zOMNLdgfM6X4ViPufpXr2I8CHg9p6RkSF3GV yvgkIG3DSfjy402A7ukSNTxzJyfmbTha6PfEwUspLy1Gb0ynC7Zt629VrNG5Fh6DiDHsfnb68GM oNHWhUT22lbvZOo3mnzzTYl4K X-Received: by 2002:a05:6000:420f:b0:3b7:8c98:2f4c with SMTP id ffacd0b85a97d-3b900b750bfmr10920313f8f.33.1754911599965; Mon, 11 Aug 2025 04:26:39 -0700 (PDT) X-Google-Smtp-Source: AGHT+IG6Mu1XA5qq3G00OwHlXO/yqxgcBzsacx06Jz+W5A1O+vqwNvOeLKy9GaCA2AZU3zaKT01H6g== X-Received: by 2002:a05:6000:420f:b0:3b7:8c98:2f4c with SMTP id ffacd0b85a97d-3b900b750bfmr10920251f8f.33.1754911599405; Mon, 11 Aug 2025 04:26:39 -0700 (PDT) Received: from localhost (p200300d82f06a600a397de1d2f8bb66f.dip0.t-ipconnect.de. [2003:d8:2f06:a600:a397:de1d:2f8b:b66f]) by smtp.gmail.com with UTF8SMTPSA id ffacd0b85a97d-3b79c3ac158sm41049329f8f.4.2025.08.11.04.26.37 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 11 Aug 2025 04:26:38 -0700 (PDT) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, xen-devel@lists.xenproject.org, linux-fsdevel@vger.kernel.org, nvdimm@lists.linux.dev, linuxppc-dev@lists.ozlabs.org, David Hildenbrand , Andrew Morton , Madhavan Srinivasan , Michael Ellerman , Nicholas Piggin , Christophe Leroy , Juergen Gross , Stefano Stabellini , Oleksandr Tyshchenko , Dan Williams , Matthew Wilcox , Jan Kara , Alexander Viro , Christian Brauner , Lorenzo Stoakes , "Liam R. Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Zi Yan , Baolin Wang , Nico Pache , Ryan Roberts , Dev Jain , Barry Song , Jann Horn , Pedro Falcato , Hugh Dickins , Oscar Salvador , Lance Yang , Alistair Popple , Wei Yang Subject: [PATCH v3 02/11] mm/huge_memory: move more common code into insert_pud() Date: Mon, 11 Aug 2025 13:26:22 +0200 Message-ID: <20250811112631.759341-3-david@redhat.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20250811112631.759341-1-david@redhat.com> References: <20250811112631.759341-1-david@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Let's clean it all further up. No functional change intended. Reviewed-by: Oscar Salvador Reviewed-by: Alistair Popple Reviewed-by: Lorenzo Stoakes Reviewed-by: Wei Yang Signed-off-by: David Hildenbrand --- mm/huge_memory.c | 36 +++++++++++++----------------------- 1 file changed, 13 insertions(+), 23 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 5314a89d676f1..7933791b75f4d 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1507,25 +1507,30 @@ static pud_t maybe_pud_mkwrite(pud_t pud, struct vm= _area_struct *vma) return pud; } =20 -static void insert_pud(struct vm_area_struct *vma, unsigned long addr, +static vm_fault_t insert_pud(struct vm_area_struct *vma, unsigned long add= r, pud_t *pud, struct folio_or_pfn fop, pgprot_t prot, bool write) { struct mm_struct *mm =3D vma->vm_mm; + spinlock_t *ptl; pud_t entry; =20 + if (addr < vma->vm_start || addr >=3D vma->vm_end) + return VM_FAULT_SIGBUS; + + ptl =3D pud_lock(mm, pud); if (!pud_none(*pud)) { const unsigned long pfn =3D fop.is_folio ? folio_pfn(fop.folio) : fop.pfn; =20 if (write) { if (WARN_ON_ONCE(pud_pfn(*pud) !=3D pfn)) - return; + goto out_unlock; entry =3D pud_mkyoung(*pud); entry =3D maybe_pud_mkwrite(pud_mkdirty(entry), vma); if (pudp_set_access_flags(vma, addr, pud, entry, 1)) update_mmu_cache_pud(vma, addr, pud); } - return; + goto out_unlock; } =20 if (fop.is_folio) { @@ -1544,6 +1549,9 @@ static void insert_pud(struct vm_area_struct *vma, un= signed long addr, } set_pud_at(mm, addr, pud, entry); update_mmu_cache_pud(vma, addr, pud); +out_unlock: + spin_unlock(ptl); + return VM_FAULT_NOPAGE; } =20 /** @@ -1565,7 +1573,6 @@ vm_fault_t vmf_insert_pfn_pud(struct vm_fault *vmf, u= nsigned long pfn, struct folio_or_pfn fop =3D { .pfn =3D pfn, }; - spinlock_t *ptl; =20 /* * If we had pud_special, we could avoid all these restrictions, @@ -1577,16 +1584,9 @@ vm_fault_t vmf_insert_pfn_pud(struct vm_fault *vmf, = unsigned long pfn, (VM_PFNMAP|VM_MIXEDMAP)); BUG_ON((vma->vm_flags & VM_PFNMAP) && is_cow_mapping(vma->vm_flags)); =20 - if (addr < vma->vm_start || addr >=3D vma->vm_end) - return VM_FAULT_SIGBUS; - pfnmap_setup_cachemode_pfn(pfn, &pgprot); =20 - ptl =3D pud_lock(vma->vm_mm, vmf->pud); - insert_pud(vma, addr, vmf->pud, fop, pgprot, write); - spin_unlock(ptl); - - return VM_FAULT_NOPAGE; + return insert_pud(vma, addr, vmf->pud, fop, pgprot, write); } EXPORT_SYMBOL_GPL(vmf_insert_pfn_pud); =20 @@ -1603,25 +1603,15 @@ vm_fault_t vmf_insert_folio_pud(struct vm_fault *vm= f, struct folio *folio, { struct vm_area_struct *vma =3D vmf->vma; unsigned long addr =3D vmf->address & PUD_MASK; - pud_t *pud =3D vmf->pud; - struct mm_struct *mm =3D vma->vm_mm; struct folio_or_pfn fop =3D { .folio =3D folio, .is_folio =3D true, }; - spinlock_t *ptl; - - if (addr < vma->vm_start || addr >=3D vma->vm_end) - return VM_FAULT_SIGBUS; =20 if (WARN_ON_ONCE(folio_order(folio) !=3D PUD_ORDER)) return VM_FAULT_SIGBUS; =20 - ptl =3D pud_lock(mm, pud); - insert_pud(vma, addr, vmf->pud, fop, vma->vm_page_prot, write); - spin_unlock(ptl); - - return VM_FAULT_NOPAGE; + return insert_pud(vma, addr, vmf->pud, fop, vma->vm_page_prot, write); } EXPORT_SYMBOL_GPL(vmf_insert_folio_pud); #endif /* CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD */ --=20 2.50.1 From nobody Sun Oct 5 01:46:23 2025 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 153292D3758 for ; Mon, 11 Aug 2025 11:26:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754911607; cv=none; b=LmW/10Eqe+Ecpp0m0ONbt8JudlRkF+NxxZOtbbioKmCoyC7ajreTr0Suc5VYhhwwi7q2uhXyKqpPaVxS76gDGO6bCENPmtXMPR+ev1eR/mVOABUPadDLG+RiWqX+A+6trtHPRYmaCBhygJVmqYNU2C4sbVgTT7Q8zvtbEeSyaFE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754911607; c=relaxed/simple; bh=Two6z2c+eu7eJRnlv4eH9xLpdr8pvhEq/q43/f0xNIY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=okUG2qA1joHgHCHN25+UtD557dMCgDWMC/Eov4c791Vx/hjzx7uCt0pPPFqv+SGlGcPVVewY++im4g5GSgtZ2320TZlp49E8w6DTP36bCpeqIcVVnyRwL/SRNtKmJu5Q63MrDfIvUnLiY+hMe+AKj1Yj59ErqAy4spu7xaldeEE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=BDdGv38F; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="BDdGv38F" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1754911605; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=OPzS7A0H2JYY/4bVoDqq5vs7hjaSrgBmC9jJDJzHOMU=; b=BDdGv38FTCsR9k3HgS7qRI7ZDmn886QC5waZeV/bXVibiuPFFqJ4hvI5rF+s7iXQFml/dO 7ZJXWh+4SO3TgrF78QQYfuV9JZJiSWCepaD8rBFgIcf+gpKUKAL346jmjupQNqRE94oYX2 pQXC+ms/MxiAsh+EuHX5DcsCWAje4Yg= Received: from mail-wm1-f72.google.com (mail-wm1-f72.google.com [209.85.128.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-435-i9FkOdCNM7-ZDYs2HqKAcQ-1; Mon, 11 Aug 2025 07:26:43 -0400 X-MC-Unique: i9FkOdCNM7-ZDYs2HqKAcQ-1 X-Mimecast-MFC-AGG-ID: i9FkOdCNM7-ZDYs2HqKAcQ_1754911603 Received: by mail-wm1-f72.google.com with SMTP id 5b1f17b1804b1-458f710f364so28484925e9.0 for ; Mon, 11 Aug 2025 04:26:43 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1754911602; x=1755516402; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=OPzS7A0H2JYY/4bVoDqq5vs7hjaSrgBmC9jJDJzHOMU=; b=lcNwUpFGwJwQwpW1BoOgg9zouMr3k7ESZtkAJ9Wb8wLk6SFGybtuTdkAQyVisIrCkD ooiFa+7qRLkW3FTCB9L80AgHmQy6Au6dkHA0FuEEazRVTGB1J4bQqTejSxnQcSi/H2cZ Iyq9QBX+I219qKi4uc9UobzHL2D8U0EHZEOB58poVn56ovrCK7NfffIDJf/Lo0mFxa+4 xtCcGR3n8DLdsFwGTwyi66fjjBGCoy9D/UMn/79aR32TFllWyh3hC594U/8ZQG+Odmgh Dioj23tGYFneSiPuLPTvN2yxjMD0i99/I0S0T1a8LAB2VunwbMx7dzWnRuJPY8I3/Mh0 UkHQ== X-Gm-Message-State: AOJu0YxR+PWJR0nE9tV423g9UYPEDHufSKcriccXi2JwXlsizfGmeBy0 EGVU4MnvDzHxhtgoofz5SvZlb7dkrOCuBS0BgbypPawrso3OVbmdbKSfU5MiC4HbGo/Cogih9Gu uTLg8UW+eUZCTOMxW05/ifV7t/C+7fzZD0hU3H/T5McDuAJwCCtW5ydOdhO/dNIGUIxCuMXVs6P sCyC5MLc+AeGe5ZUVU3oqE/eSSrQWOvAVkyXGAKO6Qxn0Z9A== X-Gm-Gg: ASbGncvgSeRKcCoDJ5gIvlAQ79wOGZxn85v5fiRPg+in7oBpvkY7N3FpMexjJh6jhF/ 0cp21tE5vLYcZ3fqj1l4xLIvKlP8ABFtKDDcRMpNiiI9kEwW9I8pei/9kiOjo9eAjRwTsKj6x19 0EsYJUGdBYDiRVMLRbGbC7DUMlJdyk/H1ntRNPPvtSGIoGrmnbr0lhKDZdf1L4qngaXsx5IVm42 yu7Jd5Ts1kuA4pPLd5aUxWW0xnWI246fwdPrQqZPknNJ8cLc+lAHS92Wmi9YfPCEoAJ9f2N+mn4 e7DTPRo1FWNHCEU9EiEbq578DH/jgrwxypKHBtABz9jZ3kEyspAcB5K0UuPGEeIFP5OR8w/K7f/ 3MqD+Wc/4ouV+cYjI822QMBld X-Received: by 2002:a05:600c:4ec7:b0:459:e398:ed89 with SMTP id 5b1f17b1804b1-459f4ea2167mr89525865e9.1.1754911602569; Mon, 11 Aug 2025 04:26:42 -0700 (PDT) X-Google-Smtp-Source: AGHT+IF2VXT8eja0G/6oYjAzMox3Wj/YsLa0WXg9I30LI+rqq9mj5h9Avu9edvonRgK7PI+OIDYPVA== X-Received: by 2002:a05:600c:4ec7:b0:459:e398:ed89 with SMTP id 5b1f17b1804b1-459f4ea2167mr89525255e9.1.1754911602063; Mon, 11 Aug 2025 04:26:42 -0700 (PDT) Received: from localhost (p200300d82f06a600a397de1d2f8bb66f.dip0.t-ipconnect.de. [2003:d8:2f06:a600:a397:de1d:2f8b:b66f]) by smtp.gmail.com with UTF8SMTPSA id 5b1f17b1804b1-459e6dcdbbbsm122068765e9.7.2025.08.11.04.26.40 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 11 Aug 2025 04:26:41 -0700 (PDT) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, xen-devel@lists.xenproject.org, linux-fsdevel@vger.kernel.org, nvdimm@lists.linux.dev, linuxppc-dev@lists.ozlabs.org, David Hildenbrand , Andrew Morton , Madhavan Srinivasan , Michael Ellerman , Nicholas Piggin , Christophe Leroy , Juergen Gross , Stefano Stabellini , Oleksandr Tyshchenko , Dan Williams , Matthew Wilcox , Jan Kara , Alexander Viro , Christian Brauner , Lorenzo Stoakes , "Liam R. Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Zi Yan , Baolin Wang , Nico Pache , Ryan Roberts , Dev Jain , Barry Song , Jann Horn , Pedro Falcato , Hugh Dickins , Oscar Salvador , Lance Yang , Wei Yang Subject: [PATCH v3 03/11] mm/huge_memory: support huge zero folio in vmf_insert_folio_pmd() Date: Mon, 11 Aug 2025 13:26:23 +0200 Message-ID: <20250811112631.759341-4-david@redhat.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20250811112631.759341-1-david@redhat.com> References: <20250811112631.759341-1-david@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Just like we do for vmf_insert_page_mkwrite() -> ... -> insert_page_into_pte_locked() with the shared zeropage, support the huge zero folio in vmf_insert_folio_pmd(). When (un)mapping the huge zero folio in page tables, we neither adjust the refcount nor the mapcount, just like for the shared zeropage. For now, the huge zero folio is not marked as special yet, although vm_normal_page_pmd() really wants to treat it as special. We'll change that next. Reviewed-by: Oscar Salvador Reviewed-by: Lorenzo Stoakes Reviewed-by: Wei Yang Signed-off-by: David Hildenbrand --- mm/huge_memory.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 7933791b75f4d..ec89e0607424e 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1418,9 +1418,11 @@ static vm_fault_t insert_pmd(struct vm_area_struct *= vma, unsigned long addr, if (fop.is_folio) { entry =3D folio_mk_pmd(fop.folio, vma->vm_page_prot); =20 - folio_get(fop.folio); - folio_add_file_rmap_pmd(fop.folio, &fop.folio->page, vma); - add_mm_counter(mm, mm_counter_file(fop.folio), HPAGE_PMD_NR); + if (!is_huge_zero_folio(fop.folio)) { + folio_get(fop.folio); + folio_add_file_rmap_pmd(fop.folio, &fop.folio->page, vma); + add_mm_counter(mm, mm_counter_file(fop.folio), HPAGE_PMD_NR); + } } else { entry =3D pmd_mkhuge(pfn_pmd(fop.pfn, prot)); entry =3D pmd_mkspecial(entry); --=20 2.50.1 From nobody Sun Oct 5 01:46:23 2025 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 401EA2D3A8A for ; Mon, 11 Aug 2025 11:26:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754911611; cv=none; b=Hx1nD7DfKal4HhSuPlnld9XCjwG2hr3yaO3chEJiZzOEc9NW4HSZAaoTriPprMzGLJygUUp2FphTZcWje1aacxpjKGuTMnALjfzTfk+8jtwiKTcgt2aFshTlaDKkqgATnSevK+dcxxRsrU85CKSXkNa21T6KLN+PlZcQK8EkgEk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754911611; c=relaxed/simple; bh=eXpOz3QD8NbTu+ic8ivqaLXg+1OSU3HgP2CuXfIC8j4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=d7sd3w3V0XpzkEQ/HEvEUM4ingHZPh49HfMMLqxAiBZ5LgCHQErEY00fBChB3p0WRsHtIo7cjQauS5VWnfzTdw5AxLlTg31zZRGx+jlfFfmVCOYiH1JQzwDRwS0zRQBj8HYteoL6lblAaNiInVlVK6XIGeK8GlnNYsyCFqvLaSM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=iPxo9qTA; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="iPxo9qTA" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1754911608; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=2CQ37fr3U8Nz67n3Rs/zLbnQBIPHeKcdveRJNY/DHTc=; b=iPxo9qTATYiT6KlCrgNNr467d7GI8CeHtwHJB9cvKz45Jrmin7rtKqkuvjMsycEqh8BbfK k3/50cMnfqqKQJ8vFOJzOoivB5lGOOVYSW75/FAdFcOtBgtPpTh1vUCfZ1BQKDsIBupII7 3DFedeAjqIsTepFhPIB6u7T9pky1RZk= Received: from mail-wm1-f70.google.com (mail-wm1-f70.google.com [209.85.128.70]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-649-eI89kdXZNJ2Kxu3zmVtbrg-1; Mon, 11 Aug 2025 07:26:46 -0400 X-MC-Unique: eI89kdXZNJ2Kxu3zmVtbrg-1 X-Mimecast-MFC-AGG-ID: eI89kdXZNJ2Kxu3zmVtbrg_1754911605 Received: by mail-wm1-f70.google.com with SMTP id 5b1f17b1804b1-458c0845b34so31384095e9.2 for ; Mon, 11 Aug 2025 04:26:46 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1754911605; x=1755516405; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=2CQ37fr3U8Nz67n3Rs/zLbnQBIPHeKcdveRJNY/DHTc=; b=IV2xX8A3w+3gJHZEj0RIK6z58UCvsci4I/pgYTLa9SVlL5M1SueRfK2+vl+GpHZdjd FTWYTXg7YD/jicn/DtXr4Z75nUVun7XuLdNjtEOCa1OgcbcmbpY3uBCofQA4PaZCwTum dbJ2FXKw1JVcH9p2YTaa5hCq2Y0jSNIvyjHruJG1LZmdgNlp73S2mgYFWW7Yxf41MYJR RXrlrKOdjEYtNsC7qLcIGUXD6do0IA3iaILhl6dV4vtORwDOuid8hg7jYV5MxFFMnh2e vHi+u7ecnw+cqpKqWH0Y6/rLvuAIv1ffrU0PQQODiHi9bhTM8DOS9dbCmxyiwiLRQYk3 /zyg== X-Gm-Message-State: AOJu0Yx1zLY8ewODZhqwRTZOwORdlx9PHOU7iep9S/3Rdp5TzRjlfjrC +okkpIKHfZP9CcbB+fIgwX45ACrbSc0GHAWoiv/3dkv+s+AHyzn5Rp78OcriIr3dYLp2XImX/6e VstC2t32WPsXJwQf6snOriYE1WStNvxg//AhNgVwU73NBYLOpN9+wlGUDlkL2GPGM+hNe5URITg nf96I8hIG8gXR76hMoveuBarH8o7oHXdYunvmpE7to4z3isQ== X-Gm-Gg: ASbGncsmj1jlxeG8hLtmTDVuinj0WJIQYRjd525JxWaGfTEBRu1LltdHBqgX22xFlYk Y9ykr+KAM1YqBUQOUbcnonPbDceo2OHEM8Fxl0pNyM77mN5kItFeyrECnUPacmC+PuVqieJ8U1j AFl8hThbhUPQtULeyuXVPsTDGSNlEsGruQgRvBs4tQ8AwUkWWX5ACEG5OBJL4lLedp+/k5woqDv 2IjOkPleoddPS1xUSMQvDkFIX/dAhS+UF4Uv1RXlX83vWOJFGzWWefccUyrKtEgLHTQOUtP4hof C6h4U4qcXLuUhiOfdCL60VS1wehNtrSBHWwEiODPhGJV7VNCnCRgUVsgcv5DjWclkjRFLwucjBA ZUnoKP4xJXK1LKnkxc4Xqt4Ws X-Received: by 2002:a05:6000:220c:b0:3b3:9c75:acc6 with SMTP id ffacd0b85a97d-3b900b579abmr8417883f8f.59.1754911605052; Mon, 11 Aug 2025 04:26:45 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHhtoj1JC1TeyGzrT2XifHRNOizQnW6DZ2myFR7vrwjVvSFSoQpoWzRIG0dK3lnMunpNHs17g== X-Received: by 2002:a05:6000:220c:b0:3b3:9c75:acc6 with SMTP id ffacd0b85a97d-3b900b579abmr8417828f8f.59.1754911604480; Mon, 11 Aug 2025 04:26:44 -0700 (PDT) Received: from localhost (p200300d82f06a600a397de1d2f8bb66f.dip0.t-ipconnect.de. [2003:d8:2f06:a600:a397:de1d:2f8b:b66f]) by smtp.gmail.com with UTF8SMTPSA id ffacd0b85a97d-3b79c4696c8sm39325003f8f.55.2025.08.11.04.26.42 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 11 Aug 2025 04:26:43 -0700 (PDT) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, xen-devel@lists.xenproject.org, linux-fsdevel@vger.kernel.org, nvdimm@lists.linux.dev, linuxppc-dev@lists.ozlabs.org, David Hildenbrand , Andrew Morton , Madhavan Srinivasan , Michael Ellerman , Nicholas Piggin , Christophe Leroy , Juergen Gross , Stefano Stabellini , Oleksandr Tyshchenko , Dan Williams , Matthew Wilcox , Jan Kara , Alexander Viro , Christian Brauner , Lorenzo Stoakes , "Liam R. Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Zi Yan , Baolin Wang , Nico Pache , Ryan Roberts , Dev Jain , Barry Song , Jann Horn , Pedro Falcato , Hugh Dickins , Oscar Salvador , Lance Yang , Alistair Popple Subject: [PATCH v3 04/11] fs/dax: use vmf_insert_folio_pmd() to insert the huge zero folio Date: Mon, 11 Aug 2025 13:26:24 +0200 Message-ID: <20250811112631.759341-5-david@redhat.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20250811112631.759341-1-david@redhat.com> References: <20250811112631.759341-1-david@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Let's convert to vmf_insert_folio_pmd(). There is a theoretical change in behavior: in the unlikely case there is already something mapped, we'll now still call trace_dax_pmd_load_hole() and return VM_FAULT_NOPAGE. Previously, we would have returned VM_FAULT_FALLBACK, and the caller would have zapped the PMD to try a PTE fault. However, that behavior was different to other PTE+PMD faults, when there would already be something mapped, and it's not even clear if it could be triggered. Assuming the huge zero folio is already mapped, all good, no need to fallback to PTEs. Assuming there is already a leaf page table ... the behavior would be just like when trying to insert a PMD mapping a folio through dax_fault_iter()->vmf_insert_folio_pmd(). Assuming there is already something else mapped as PMD? It sounds like a BUG, and the behavior would be just like when trying to insert a PMD mapping a folio through dax_fault_iter()->vmf_insert_folio_pmd(). So, it sounds reasonable to not handle huge zero folios differently to inserting PMDs mapping folios when there already is something mapped. Reviewed-by: Alistair Popple Reviewed-by: Lorenzo Stoakes Signed-off-by: David Hildenbrand --- fs/dax.c | 47 ++++++++++------------------------------------- 1 file changed, 10 insertions(+), 37 deletions(-) diff --git a/fs/dax.c b/fs/dax.c index 4229513806bea..ae90706674a3f 100644 --- a/fs/dax.c +++ b/fs/dax.c @@ -1375,51 +1375,24 @@ static vm_fault_t dax_pmd_load_hole(struct xa_state= *xas, struct vm_fault *vmf, const struct iomap_iter *iter, void **entry) { struct address_space *mapping =3D vmf->vma->vm_file->f_mapping; - unsigned long pmd_addr =3D vmf->address & PMD_MASK; - struct vm_area_struct *vma =3D vmf->vma; struct inode *inode =3D mapping->host; - pgtable_t pgtable =3D NULL; struct folio *zero_folio; - spinlock_t *ptl; - pmd_t pmd_entry; - unsigned long pfn; + vm_fault_t ret; =20 zero_folio =3D mm_get_huge_zero_folio(vmf->vma->vm_mm); =20 - if (unlikely(!zero_folio)) - goto fallback; - - pfn =3D page_to_pfn(&zero_folio->page); - *entry =3D dax_insert_entry(xas, vmf, iter, *entry, pfn, - DAX_PMD | DAX_ZERO_PAGE); - - if (arch_needs_pgtable_deposit()) { - pgtable =3D pte_alloc_one(vma->vm_mm); - if (!pgtable) - return VM_FAULT_OOM; - } - - ptl =3D pmd_lock(vmf->vma->vm_mm, vmf->pmd); - if (!pmd_none(*(vmf->pmd))) { - spin_unlock(ptl); - goto fallback; + if (unlikely(!zero_folio)) { + trace_dax_pmd_load_hole_fallback(inode, vmf, zero_folio, *entry); + return VM_FAULT_FALLBACK; } =20 - if (pgtable) { - pgtable_trans_huge_deposit(vma->vm_mm, vmf->pmd, pgtable); - mm_inc_nr_ptes(vma->vm_mm); - } - pmd_entry =3D folio_mk_pmd(zero_folio, vmf->vma->vm_page_prot); - set_pmd_at(vmf->vma->vm_mm, pmd_addr, vmf->pmd, pmd_entry); - spin_unlock(ptl); - trace_dax_pmd_load_hole(inode, vmf, zero_folio, *entry); - return VM_FAULT_NOPAGE; + *entry =3D dax_insert_entry(xas, vmf, iter, *entry, folio_pfn(zero_folio), + DAX_PMD | DAX_ZERO_PAGE); =20 -fallback: - if (pgtable) - pte_free(vma->vm_mm, pgtable); - trace_dax_pmd_load_hole_fallback(inode, vmf, zero_folio, *entry); - return VM_FAULT_FALLBACK; + ret =3D vmf_insert_folio_pmd(vmf, zero_folio, false); + if (ret =3D=3D VM_FAULT_NOPAGE) + trace_dax_pmd_load_hole(inode, vmf, zero_folio, *entry); + return ret; } #else static vm_fault_t dax_pmd_load_hole(struct xa_state *xas, struct vm_fault = *vmf, --=20 2.50.1 From nobody Sun Oct 5 01:46:23 2025 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EBD082D1914 for ; Mon, 11 Aug 2025 11:26:50 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754911612; cv=none; b=h2cQkTHp0rU3jU6hH5Yu9gjs3lnxBJwloXeCNW2Z76WJYgJu9JqMRTQiYJBDe2HyInIkEKPh2k71qlMqVdk9/bCskw92E8GEa4C0W2WZvvRtl5r9jVpYw6sSXYdpymuqP5T/jZvuC6Sd2YW8hMYXYD8fx9DTIbqxW/X67+zBp+I= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754911612; c=relaxed/simple; bh=a+hiBZtBByVYC1UASrKiik5/n19aV+sxWoIAsC0g5EY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=QgepD/Isc0z+ABsLYBCIA7LFA4+RcUu1Lopwiw4lxCQ7olWz8LyLsAUyoF8ceuV7YnxFr2ifr6AjiuJhYeS85/N/T9zG59MrQ3kUWoWEKgd1n9amAR/itGMtRWV2FBXv4iPPISUEiegIS2ZbW3qF32Nj5LlbI58OmO1LlJKWg/M= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=BgipPYxB; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="BgipPYxB" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1754911610; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=DNeO4mnK34t1j+RunD3+GtheXBl5F9uGzS1IAgGiaS8=; b=BgipPYxBJNEbhRjG8OHzf5Rs83JZv7WfjgFz2ekgSV1ZOmAzLVUH9RBnGicgtNy79jX2AW gj6YsIGfcA8cVfHhcc9xP6W+ZI2LB1Yv4lxwAtnVFFjNN15I2FhtCsfmjAckXEHCDDm+UV E9IXzBl5ptTZ174tijVBLnFTuleO4z8= Received: from mail-wm1-f70.google.com (mail-wm1-f70.google.com [209.85.128.70]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-394-hxaF-zcWNFOmEBxQmyhAoA-1; Mon, 11 Aug 2025 07:26:48 -0400 X-MC-Unique: hxaF-zcWNFOmEBxQmyhAoA-1 X-Mimecast-MFC-AGG-ID: hxaF-zcWNFOmEBxQmyhAoA_1754911608 Received: by mail-wm1-f70.google.com with SMTP id 5b1f17b1804b1-451d3f03b74so24369585e9.3 for ; Mon, 11 Aug 2025 04:26:48 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1754911607; x=1755516407; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=DNeO4mnK34t1j+RunD3+GtheXBl5F9uGzS1IAgGiaS8=; b=iuLZmPn0GEupLlZimbxWA1I6G6bePGxoTvoRg0xJSTUK4XByMr652tV5xat1YyZR5u ABe7NBgF+xTx36sbnqGMYs/ZcR3tTQqKWeQ6DtV3V8mGnTO308QUmA+7maAgGnMvPZwq /Rgl0wz4O9o0U+wfxT6C4XUywgRSww+XzNRU0gMwjih2ZHeM1dBRks1WUXl/hwzREzUL 3qrfiGv/F7avz4prS8pJSke+XvjpQvbBAVhZRmUy4nVje3dIAOiJrs0aufu6Adbuh12u 7IqjU2ndWlyoMeUTxQxlOFNP5+KLToNGkJHV7nD2eN/NXsbF2p6L88sYFYQZ+3XsiNWD DRaQ== X-Gm-Message-State: AOJu0Yw65L72iBDSkN4UjiVCtYB4aJKqvx7a/e8I84fcVqu5JnV1ZNPH 0CS16rets4CxPtZNNkFc9JTxZLQTMcGSU3GGtO0c8dB9Z6E/uxOJk9yYS8fhFQiOBUhwF7t8DMY R/eoRTECIRKjkSN4v4bt8Gjc4qPF88vTXC7yve2fjiMDgQIeURarIlgcrE4DPtKHJkT/4L7ii5n hucqaYrMGEsw0ylMR/Vpepe4oExiAIQRnPYgGdZXVihi11BA== X-Gm-Gg: ASbGncvMYI+Y2BqQ8cJpKnkgEciZUWEpW2byHR/zEy9CUO2OfbBODuVH5YOcrchxiZf R63B+aZ3TPZgQkaeaaFAEVbrMWNkh05U0JoarDfQs5hoWRpogJgaIvLbYnijxsp1nLHnV2REVfK NWJVxijkduvuNcH1U4yEXxivGV2ag+t9gw8Qa3ZD/K790luSKCVKJZGoox25yIRgUOHU2JanhEJ z3wVt1Z5eU53GQOC8Uznyno8eK0d//SFNWDG2j19GYBIkfE20wuTQZkd92kVo7CSft0r4v0KQvy SZsGAE5RHiCNMqAWzVfwM/xANgD3mSJR9vS/Z9cP78zxHjK401E0Vq9+qSR7JsMooyN6n8X5Pk4 Eb8nwWabU19iIW2ozXqus9sxZ X-Received: by 2002:a05:600c:4746:b0:453:23fe:ca86 with SMTP id 5b1f17b1804b1-459f4ea0f2dmr115559955e9.4.1754911607499; Mon, 11 Aug 2025 04:26:47 -0700 (PDT) X-Google-Smtp-Source: AGHT+IF+SYBbVCDa86Al3Rw//iuzbWoSgvjLT8uLZDTY5p6isO9HqDz+nOTt8CKLJXWJJ2UJD3ft4A== X-Received: by 2002:a05:600c:4746:b0:453:23fe:ca86 with SMTP id 5b1f17b1804b1-459f4ea0f2dmr115559215e9.4.1754911607000; Mon, 11 Aug 2025 04:26:47 -0700 (PDT) Received: from localhost (p200300d82f06a600a397de1d2f8bb66f.dip0.t-ipconnect.de. [2003:d8:2f06:a600:a397:de1d:2f8b:b66f]) by smtp.gmail.com with UTF8SMTPSA id 5b1f17b1804b1-459e6214640sm259832575e9.1.2025.08.11.04.26.45 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 11 Aug 2025 04:26:46 -0700 (PDT) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, xen-devel@lists.xenproject.org, linux-fsdevel@vger.kernel.org, nvdimm@lists.linux.dev, linuxppc-dev@lists.ozlabs.org, David Hildenbrand , Andrew Morton , Madhavan Srinivasan , Michael Ellerman , Nicholas Piggin , Christophe Leroy , Juergen Gross , Stefano Stabellini , Oleksandr Tyshchenko , Dan Williams , Matthew Wilcox , Jan Kara , Alexander Viro , Christian Brauner , Lorenzo Stoakes , "Liam R. Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Zi Yan , Baolin Wang , Nico Pache , Ryan Roberts , Dev Jain , Barry Song , Jann Horn , Pedro Falcato , Hugh Dickins , Oscar Salvador , Lance Yang Subject: [PATCH v3 05/11] mm/huge_memory: mark PMD mappings of the huge zero folio special Date: Mon, 11 Aug 2025 13:26:25 +0200 Message-ID: <20250811112631.759341-6-david@redhat.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20250811112631.759341-1-david@redhat.com> References: <20250811112631.759341-1-david@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The huge zero folio is refcounted (+mapcounted -- is that a word?) differently than "normal" folios, similarly (but different) to the ordinary shared zeropage. For this reason, we special-case these pages in vm_normal_page*/vm_normal_folio*, and only allow selected callers to still use them (e.g., GUP can still take a reference on them). vm_normal_page_pmd() already filters out the huge zero folio, to indicate it a special (return NULL). However, so far we are not making use of pmd_special() on architectures that support it (CONFIG_ARCH_HAS_PTE_SPECIAL), like we would with the ordinary shared zeropage. Let's mark PMD mappings of the huge zero folio similarly as special, so we can avoid the manual check for the huge zero folio with CONFIG_ARCH_HAS_PTE_SPECIAL next, and only perform the check on !CONFIG_ARCH_HAS_PTE_SPECIAL. In copy_huge_pmd(), where we have a manual pmd_special() check to handle PFNMAP, we have to manually rule out the huge zero folio. That code needs a serious cleanup, but that's something for another day. While at it, update the doc regarding the shared zero folios. No functional change intended: vm_normal_page_pmd() still returns NULL when it encounters the huge zero folio. Reviewed-by: Oscar Salvador Signed-off-by: David Hildenbrand Reviewed-by: Lorenzo Stoakes --- mm/huge_memory.c | 8 ++++++-- mm/memory.c | 15 ++++++++++----- 2 files changed, 16 insertions(+), 7 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index ec89e0607424e..58bac83e7fa31 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1309,6 +1309,7 @@ static void set_huge_zero_folio(pgtable_t pgtable, st= ruct mm_struct *mm, { pmd_t entry; entry =3D folio_mk_pmd(zero_folio, vma->vm_page_prot); + entry =3D pmd_mkspecial(entry); pgtable_trans_huge_deposit(mm, pmd, pgtable); set_pmd_at(mm, haddr, pmd, entry); mm_inc_nr_ptes(mm); @@ -1418,7 +1419,9 @@ static vm_fault_t insert_pmd(struct vm_area_struct *v= ma, unsigned long addr, if (fop.is_folio) { entry =3D folio_mk_pmd(fop.folio, vma->vm_page_prot); =20 - if (!is_huge_zero_folio(fop.folio)) { + if (is_huge_zero_folio(fop.folio)) { + entry =3D pmd_mkspecial(entry); + } else { folio_get(fop.folio); folio_add_file_rmap_pmd(fop.folio, &fop.folio->page, vma); add_mm_counter(mm, mm_counter_file(fop.folio), HPAGE_PMD_NR); @@ -1643,7 +1646,8 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm= _struct *src_mm, int ret =3D -ENOMEM; =20 pmd =3D pmdp_get_lockless(src_pmd); - if (unlikely(pmd_present(pmd) && pmd_special(pmd))) { + if (unlikely(pmd_present(pmd) && pmd_special(pmd) && + !is_huge_zero_pmd(pmd))) { dst_ptl =3D pmd_lock(dst_mm, dst_pmd); src_ptl =3D pmd_lockptr(src_mm, src_pmd); spin_lock_nested(src_ptl, SINGLE_DEPTH_NESTING); diff --git a/mm/memory.c b/mm/memory.c index 0ba4f6b718471..626caedce35e0 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -555,7 +555,14 @@ static void print_bad_pte(struct vm_area_struct *vma, = unsigned long addr, * * "Special" mappings do not wish to be associated with a "struct page" (e= ither * it doesn't exist, or it exists but they don't want to touch it). In this - * case, NULL is returned here. "Normal" mappings do have a struct page. + * case, NULL is returned here. "Normal" mappings do have a struct page and + * are ordinarily refcounted. + * + * Page mappings of the shared zero folios are always considered "special"= , as + * they are not ordinarily refcounted: neither the refcount nor the mapcou= nt + * of these folios is adjusted when mapping them into user page tables. + * Selected page table walkers (such as GUP) can still identify mappings o= f the + * shared zero folios and work with the underlying "struct page". * * There are 2 broad cases. Firstly, an architecture may define a pte_spec= ial() * pte bit, in which case this function is trivial. Secondly, an architect= ure @@ -585,9 +592,8 @@ static void print_bad_pte(struct vm_area_struct *vma, u= nsigned long addr, * * VM_MIXEDMAP mappings can likewise contain memory with or without "struct * page" backing, however the difference is that _all_ pages with a struct - * page (that is, those where pfn_valid is true) are refcounted and consid= ered - * normal pages by the VM. The only exception are zeropages, which are - * *never* refcounted. + * page (that is, those where pfn_valid is true, except the shared zero + * folios) are refcounted and considered normal pages by the VM. * * The disadvantage is that pages are refcounted (which can be slower and * simply not an option for some PFNMAP users). The advantage is that we @@ -667,7 +673,6 @@ struct page *vm_normal_page_pmd(struct vm_area_struct *= vma, unsigned long addr, { unsigned long pfn =3D pmd_pfn(pmd); =20 - /* Currently it's only used for huge pfnmaps */ if (unlikely(pmd_special(pmd))) return NULL; =20 --=20 2.50.1 From nobody Sun Oct 5 01:46:23 2025 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 812142D321F for ; Mon, 11 Aug 2025 11:26:53 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754911615; cv=none; b=qqSspqG0vXskI10cebh2KIPTxbdJ7H8X1dXWT9lmT/JUiJlgL51hEyMkC4YKOB1mylWW00ujNKYmlwVPOJVfkpmDJRUCuPlUm0vjvBDXgWvAkMhVQNjYZ6cQhaNdIa+rq5dZPJZ+xWaGqDoBv2HKBiaAv9qo2Rh8yk2De3FNNKw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754911615; c=relaxed/simple; bh=5bTAXo4Wnbsgm2C4LriAvbtI/44w1uQqyTmK51I8pv4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=MEuQcmES1xmw6zNoH+iFxk8+BkSj+rw1H29CEc5EJx2zFaYye6chNJpEmToCgot4rag2xVCHzafr6PsaHVFbgJB+g0WmqNIVQA6IfXvqzHP1IuQozprnTTs87/P3mEuXvrD6LRTdbBBqsfjcazWzR1zHH8MITPEWT8ZHwY0tEOQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=XUjMLlNS; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="XUjMLlNS" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1754911612; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=UJDg4sW92zbLWKqBjb1tVZssT9gBBixv5gdvUlii8Zs=; b=XUjMLlNSncT0cLxCT9SU/uEGpJOcBdJB0j8Mv5cQTzi9e9vbJTPA3q1oCPvxqwW2QC2SZY K9Jsbad+gPXa3wsz+YLFyNaal53KDL6k4E8YaZ99UodAi6VX7BxDe/U/IQLDGyDuqizb25 GjTV9SQ07rGN6rxPnJ6CDXCkMG2BCkw= Received: from mail-wr1-f70.google.com (mail-wr1-f70.google.com [209.85.221.70]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-319-f-XtAeLXMB2awysJbAYQ7g-1; Mon, 11 Aug 2025 07:26:51 -0400 X-MC-Unique: f-XtAeLXMB2awysJbAYQ7g-1 X-Mimecast-MFC-AGG-ID: f-XtAeLXMB2awysJbAYQ7g_1754911610 Received: by mail-wr1-f70.google.com with SMTP id ffacd0b85a97d-3b604541741so2205126f8f.3 for ; Mon, 11 Aug 2025 04:26:51 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1754911610; x=1755516410; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=UJDg4sW92zbLWKqBjb1tVZssT9gBBixv5gdvUlii8Zs=; b=hQ6F54AXSIXkcSr1f44807XONxnGKQL1jCzj09sFnupBK1Z3pufP+90qWDeFq8N2uk cQ4QyqXu+YVHfwEGpmF/V6QWaHNpeXCGmbVugCKpyl1GxBAQaa187kDdge9iZJ0ELYJH oTUgLDFggYjy1h5A2RH8kHHIMhbKCGwLLB1RJzLjIeKx84F1K74GjVuqCYn96YKOVBkd f7wRXwTOpjoZD01O+BxEYbCZ+SLa/VXi2B1kbAg/U7WMcBr1Vd2owM1Mb9DzLH8FTA0d 589tPQZWMHXHuFectS+Zm2P5LzYbX9o/+6jh1oqIy4vB61/9gM/yJg5/nyMqpnTlsb+o Isjw== X-Gm-Message-State: AOJu0YzKlVtMwU+y7JSDailgfBVIhXGuWUHBpbgnQclUDq33mIRftaby pWv+MhzgZlD4MnJlElBHPTkhNZVVfZhXD0/3sE2KIVxMd55woDhAmXnMe3XiapZjwVhtoZ1BCBQ MfikJgeypbt9xQq/V6GQ6H7feLIGSniQ5cn3TfQL0edR620L0h7LcHDkTB1AeraAM1WXrUqkwbp 2Ss6qMD0fibv5RCVpBiYjA7qnuBngH/efI7+aGcbJ6m/3FKw== X-Gm-Gg: ASbGncuTQmP8mQ78LGu2utPmEtdTuVveTZ04qugEdZu5eO5+LGZV36UNZTA7o/mcHmz 9FSDQpEIvp3YtBMO32mt5YbUZapvWnxHEC53Q8vmRGHhgaf/NU0n2MzeGGdZam03LAPr8cJ0R40 iLqHKBv9rw1xHQfgEg2uL988P5DvH2ckQAmhMJZawqMUW79lPyjD8Q7tbQNbEcAOvWIZ+uASl+0 sjQFgSwVI/JLYwlBA8SJ3Joiuz3i90hw5hwPeHwH6EIWQt7G0dWq1i1UUMB7IVgT6iyX/xSTtbw 7zeCwuA65L3A8XE64naE0uqIRp4zlQRhpieITnAG88uCSwLuOQTaWPoy80qfsz8OHVxBBeVpWI+ MgaCG8JI+Wo3/w7FenoS7e3pB X-Received: by 2002:a05:6000:2010:b0:3b7:9d83:5104 with SMTP id ffacd0b85a97d-3b900b83ce4mr10356654f8f.51.1754911610221; Mon, 11 Aug 2025 04:26:50 -0700 (PDT) X-Google-Smtp-Source: AGHT+IH28XXBuV6P4eMmACbygagN9E2w90dcNlvtCQwFebmNkdYsSnJof6QrfGHsOh8VfCIdvok2gg== X-Received: by 2002:a05:6000:2010:b0:3b7:9d83:5104 with SMTP id ffacd0b85a97d-3b900b83ce4mr10356583f8f.51.1754911609552; Mon, 11 Aug 2025 04:26:49 -0700 (PDT) Received: from localhost (p200300d82f06a600a397de1d2f8bb66f.dip0.t-ipconnect.de. [2003:d8:2f06:a600:a397:de1d:2f8b:b66f]) by smtp.gmail.com with UTF8SMTPSA id ffacd0b85a97d-3b79c3bf93dsm40408983f8f.27.2025.08.11.04.26.47 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 11 Aug 2025 04:26:48 -0700 (PDT) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, xen-devel@lists.xenproject.org, linux-fsdevel@vger.kernel.org, nvdimm@lists.linux.dev, linuxppc-dev@lists.ozlabs.org, David Hildenbrand , Andrew Morton , Madhavan Srinivasan , Michael Ellerman , Nicholas Piggin , Christophe Leroy , Juergen Gross , Stefano Stabellini , Oleksandr Tyshchenko , Dan Williams , Matthew Wilcox , Jan Kara , Alexander Viro , Christian Brauner , Lorenzo Stoakes , "Liam R. Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Zi Yan , Baolin Wang , Nico Pache , Ryan Roberts , Dev Jain , Barry Song , Jann Horn , Pedro Falcato , Hugh Dickins , Oscar Salvador , Lance Yang Subject: [PATCH v3 06/11] powerpc/ptdump: rename "struct pgtable_level" to "struct ptdump_pglevel" Date: Mon, 11 Aug 2025 13:26:26 +0200 Message-ID: <20250811112631.759341-7-david@redhat.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20250811112631.759341-1-david@redhat.com> References: <20250811112631.759341-1-david@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" We want to make use of "pgtable_level" for an enum in core-mm. Other architectures seem to call "struct pgtable_level" either: * "struct pg_level" when not exposed in a header (riscv, arm) * "struct ptdump_pg_level" when expose in a header (arm64) So let's follow what arm64 does. Signed-off-by: David Hildenbrand Reviewed-by: Lorenzo Stoakes Reviewed-by: Ritesh Harjani (IBM) --- arch/powerpc/mm/ptdump/8xx.c | 2 +- arch/powerpc/mm/ptdump/book3s64.c | 2 +- arch/powerpc/mm/ptdump/ptdump.h | 4 ++-- arch/powerpc/mm/ptdump/shared.c | 2 +- 4 files changed, 5 insertions(+), 5 deletions(-) diff --git a/arch/powerpc/mm/ptdump/8xx.c b/arch/powerpc/mm/ptdump/8xx.c index b5c79b11ea3c2..4ca9cf7a90c9e 100644 --- a/arch/powerpc/mm/ptdump/8xx.c +++ b/arch/powerpc/mm/ptdump/8xx.c @@ -69,7 +69,7 @@ static const struct flag_info flag_array[] =3D { } }; =20 -struct pgtable_level pg_level[5] =3D { +struct ptdump_pg_level pg_level[5] =3D { { /* pgd */ .flag =3D flag_array, .num =3D ARRAY_SIZE(flag_array), diff --git a/arch/powerpc/mm/ptdump/book3s64.c b/arch/powerpc/mm/ptdump/boo= k3s64.c index 5ad92d9dc5d10..6b2da9241d4c4 100644 --- a/arch/powerpc/mm/ptdump/book3s64.c +++ b/arch/powerpc/mm/ptdump/book3s64.c @@ -102,7 +102,7 @@ static const struct flag_info flag_array[] =3D { } }; =20 -struct pgtable_level pg_level[5] =3D { +struct ptdump_pg_level pg_level[5] =3D { { /* pgd */ .flag =3D flag_array, .num =3D ARRAY_SIZE(flag_array), diff --git a/arch/powerpc/mm/ptdump/ptdump.h b/arch/powerpc/mm/ptdump/ptdum= p.h index 154efae96ae09..4232aa4b57eae 100644 --- a/arch/powerpc/mm/ptdump/ptdump.h +++ b/arch/powerpc/mm/ptdump/ptdump.h @@ -11,12 +11,12 @@ struct flag_info { int shift; }; =20 -struct pgtable_level { +struct ptdump_pg_level { const struct flag_info *flag; size_t num; u64 mask; }; =20 -extern struct pgtable_level pg_level[5]; +extern struct ptdump_pg_level pg_level[5]; =20 void pt_dump_size(struct seq_file *m, unsigned long delta); diff --git a/arch/powerpc/mm/ptdump/shared.c b/arch/powerpc/mm/ptdump/share= d.c index 39c30c62b7ea7..58998960eb9a4 100644 --- a/arch/powerpc/mm/ptdump/shared.c +++ b/arch/powerpc/mm/ptdump/shared.c @@ -67,7 +67,7 @@ static const struct flag_info flag_array[] =3D { } }; =20 -struct pgtable_level pg_level[5] =3D { +struct ptdump_pg_level pg_level[5] =3D { { /* pgd */ .flag =3D flag_array, .num =3D ARRAY_SIZE(flag_array), --=20 2.50.1 From nobody Sun Oct 5 01:46:23 2025 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C86252D4B4B for ; Mon, 11 Aug 2025 11:26:55 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754911617; cv=none; b=BV6iqzdTpH0bMPId9CYmcWvsUH9C7ZEuEncjdpYwT/Q12eWGXekP+N7/VUOaLyO2nd3zD50gQd4JAHj6Uju7VD62X2D7m7gPeuiBnFovEp8orG+HOMSZD8zat0lbq3aiZtWlQE6MlcHdsWYpGWwrA1NjBo4R1GJiG3BFt8GBPow= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754911617; c=relaxed/simple; bh=uqoGJxXVtswfOJ5P3HlH7BvryKB+HR0L+OmQ/N6rNM4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=mnsBe5txHHrE/3WwJwjN6T70ue20QkiQJc/XWFZMQ4ktkKzy6TzV4iXflZDQAcPwgh5rdMadmPGgjzVKb8VFSgPGV5CPdK0QbFqNBG19T3UwrhYMLUmSQkl9tz1Ke4Ct2N2Zi1RyI3/UzEF2YkJ/9DI/KtuK39MrUUf+Y4fZKzc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=gsolrKID; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="gsolrKID" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1754911615; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=P87rLpAYrvMbmMGGuIRrtBN562miOXHo6Y54j1JPV6I=; b=gsolrKIDBpHJ/0CdgUI00iwyRIt4rLXSPLrmMBHnJOyxzmLjrEVUpYC//8FlMbQGENyKu/ rqnCtXREdhMx46PpolIc47O1T+LKkQLPPE9NY/h7Y3nmZC9mjtxAJexYLI4WZiXl41nDR6 1iNzkm1zhiJqmUV6trX2D1rh1E7tf04= Received: from mail-wm1-f69.google.com (mail-wm1-f69.google.com [209.85.128.69]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-623-ccVfFmEPMlKtHmqQKo6HyQ-1; Mon, 11 Aug 2025 07:26:54 -0400 X-MC-Unique: ccVfFmEPMlKtHmqQKo6HyQ-1 X-Mimecast-MFC-AGG-ID: ccVfFmEPMlKtHmqQKo6HyQ_1754911613 Received: by mail-wm1-f69.google.com with SMTP id 5b1f17b1804b1-4538f375e86so32406045e9.3 for ; Mon, 11 Aug 2025 04:26:53 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1754911613; x=1755516413; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=P87rLpAYrvMbmMGGuIRrtBN562miOXHo6Y54j1JPV6I=; b=CZroKJVbtth+V5QOD8LO+1OBsax2vgrB7HfRw3IcHhed+VbyLf9ZgXiwFSHQjJeQta tMygB7rwyxw4ZZMTJ3MaXxzvAyNzhY0VBFoHL/AmrcWfiwqHmmySLbgJoCX8DRlitGb2 v3N7hF5AR/pUuztg/lG3c5nwIcDORMeyWi+EkGjEwl880EP60AdtIayQlzXRbCqfrONb Q6wlGkOGwo2iBPFgUo5RLzh4nkZ2IlDL4K7cJt0dkSM2FpM5XJgU3V8S++TGUkeZytxi V8PZOm2VWJX6YcTn8U8FVxvqlaXwBe56rEbHqem2Gsg2k8BSYudSyeGVX08OsiqB3hhl 9BgQ== X-Gm-Message-State: AOJu0Yxo3j6WVw1z7oFP8KitVznb1ouOkZVavYFh7oFtcZXYMsk514fc 5IJ78c8BUTQnbvMyxbaNCgqg1HU4ANtbVAM57kpMiyVbgfoKkTBJyUIaQ/6PiTZDcN9v34y30bg ORsrvfDPcFk2/3yrGC1TY0Lp+mf51148JrFfNlbRCCXeFVgkfI8+1HbHz0KpESxamN/FqTvTiVp no/whCD4e1z+7Xa6w82t/I0168s6siL3SDueP9OKnJR0XQaw== X-Gm-Gg: ASbGncvL2BK0b5+AjN+zn8SEnqiQXd60J4x2JTEqdzzvGQT4e9F1/k2PdSGJtUVQTDB LxcHCiyattR65QASNgccO71JLK4qtkUVrnj2pFUwLx9VXpxcVNIj7RnCr+Px1eK7nSZEhFtWdC4 iu1F2MwnXcaQq9+CmwxNYq/B2a0GoXABmSLJBZGtsxBYceQ7cgthEX8dayUblmjcbWM2Uyppejy 2rcl03uH17pMZ7YmwGC0Bj9ROhDavzWA8VWxvVc8Ld8OPTrABUXqahgqzdO5XhEUGBD8RfHGN0h WRaGGFry6pHTax4WM9/ikWWyBhw3dsayOo3QAhKcaWiEmQoH+6cKi/aYR5VZ9KX/faOWFpgmiX9 3sKT2U+XJ/tHrzJuQ9kQtyeqe X-Received: by 2002:a05:600c:3589:b0:459:dfa8:b881 with SMTP id 5b1f17b1804b1-459f4f3cfd9mr103523735e9.7.1754911612694; Mon, 11 Aug 2025 04:26:52 -0700 (PDT) X-Google-Smtp-Source: AGHT+IEiy2Q6M1gIlNBDGe4u7ez53xLkuOFeUzXf29NvQrYJzqDFFp1B3pMcW/PtvRUdMabZHM2Urw== X-Received: by 2002:a05:600c:3589:b0:459:dfa8:b881 with SMTP id 5b1f17b1804b1-459f4f3cfd9mr103523005e9.7.1754911612074; Mon, 11 Aug 2025 04:26:52 -0700 (PDT) Received: from localhost (p200300d82f06a600a397de1d2f8bb66f.dip0.t-ipconnect.de. [2003:d8:2f06:a600:a397:de1d:2f8b:b66f]) by smtp.gmail.com with UTF8SMTPSA id ffacd0b85a97d-3b79c3bf93dsm40409120f8f.27.2025.08.11.04.26.50 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 11 Aug 2025 04:26:51 -0700 (PDT) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, xen-devel@lists.xenproject.org, linux-fsdevel@vger.kernel.org, nvdimm@lists.linux.dev, linuxppc-dev@lists.ozlabs.org, David Hildenbrand , Andrew Morton , Madhavan Srinivasan , Michael Ellerman , Nicholas Piggin , Christophe Leroy , Juergen Gross , Stefano Stabellini , Oleksandr Tyshchenko , Dan Williams , Matthew Wilcox , Jan Kara , Alexander Viro , Christian Brauner , Lorenzo Stoakes , "Liam R. Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Zi Yan , Baolin Wang , Nico Pache , Ryan Roberts , Dev Jain , Barry Song , Jann Horn , Pedro Falcato , Hugh Dickins , Oscar Salvador , Lance Yang Subject: [PATCH v3 07/11] mm/rmap: convert "enum rmap_level" to "enum pgtable_level" Date: Mon, 11 Aug 2025 13:26:27 +0200 Message-ID: <20250811112631.759341-8-david@redhat.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20250811112631.759341-1-david@redhat.com> References: <20250811112631.759341-1-david@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Let's factor it out, and convert all checks for unsupported levels to BUILD_BUG(). The code is written in a way such that force-inlining will optimize out the levels. Signed-off-by: David Hildenbrand Reviewed-by: Lorenzo Stoakes --- include/linux/pgtable.h | 8 ++++++ include/linux/rmap.h | 60 +++++++++++++++++++---------------------- mm/rmap.c | 56 +++++++++++++++++++++----------------- 3 files changed, 66 insertions(+), 58 deletions(-) diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index 4c035637eeb77..bff5c4241bf2e 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -1958,6 +1958,14 @@ static inline bool arch_has_pfn_modify_check(void) /* Page-Table Modification Mask */ typedef unsigned int pgtbl_mod_mask; =20 +enum pgtable_level { + PGTABLE_LEVEL_PTE =3D 0, + PGTABLE_LEVEL_PMD, + PGTABLE_LEVEL_PUD, + PGTABLE_LEVEL_P4D, + PGTABLE_LEVEL_PGD, +}; + #endif /* !__ASSEMBLY__ */ =20 #if !defined(MAX_POSSIBLE_PHYSMEM_BITS) && !defined(CONFIG_64BIT) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index 6cd020eea37a2..9d40d127bdb78 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -394,18 +394,8 @@ typedef int __bitwise rmap_t; /* The anonymous (sub)page is exclusive to a single process. */ #define RMAP_EXCLUSIVE ((__force rmap_t)BIT(0)) =20 -/* - * Internally, we're using an enum to specify the granularity. We make the - * compiler emit specialized code for each granularity. - */ -enum rmap_level { - RMAP_LEVEL_PTE =3D 0, - RMAP_LEVEL_PMD, - RMAP_LEVEL_PUD, -}; - static inline void __folio_rmap_sanity_checks(const struct folio *folio, - const struct page *page, int nr_pages, enum rmap_level level) + const struct page *page, int nr_pages, enum pgtable_level level) { /* hugetlb folios are handled separately. */ VM_WARN_ON_FOLIO(folio_test_hugetlb(folio), folio); @@ -427,18 +417,18 @@ static inline void __folio_rmap_sanity_checks(const s= truct folio *folio, VM_WARN_ON_FOLIO(page_folio(page + nr_pages - 1) !=3D folio, folio); =20 switch (level) { - case RMAP_LEVEL_PTE: + case PGTABLE_LEVEL_PTE: break; - case RMAP_LEVEL_PMD: + case PGTABLE_LEVEL_PMD: /* * We don't support folios larger than a single PMD yet. So - * when RMAP_LEVEL_PMD is set, we assume that we are creating + * when PGTABLE_LEVEL_PMD is set, we assume that we are creating * a single "entire" mapping of the folio. */ VM_WARN_ON_FOLIO(folio_nr_pages(folio) !=3D HPAGE_PMD_NR, folio); VM_WARN_ON_FOLIO(nr_pages !=3D HPAGE_PMD_NR, folio); break; - case RMAP_LEVEL_PUD: + case PGTABLE_LEVEL_PUD: /* * Assume that we are creating a single "entire" mapping of the * folio. @@ -447,7 +437,7 @@ static inline void __folio_rmap_sanity_checks(const str= uct folio *folio, VM_WARN_ON_FOLIO(nr_pages !=3D HPAGE_PUD_NR, folio); break; default: - VM_WARN_ON_ONCE(true); + BUILD_BUG(); } =20 /* @@ -567,14 +557,14 @@ static inline void hugetlb_remove_rmap(struct folio *= folio) =20 static __always_inline void __folio_dup_file_rmap(struct folio *folio, struct page *page, int nr_pages, struct vm_area_struct *dst_vma, - enum rmap_level level) + enum pgtable_level level) { const int orig_nr_pages =3D nr_pages; =20 __folio_rmap_sanity_checks(folio, page, nr_pages, level); =20 switch (level) { - case RMAP_LEVEL_PTE: + case PGTABLE_LEVEL_PTE: if (!folio_test_large(folio)) { atomic_inc(&folio->_mapcount); break; @@ -587,11 +577,13 @@ static __always_inline void __folio_dup_file_rmap(str= uct folio *folio, } folio_add_large_mapcount(folio, orig_nr_pages, dst_vma); break; - case RMAP_LEVEL_PMD: - case RMAP_LEVEL_PUD: + case PGTABLE_LEVEL_PMD: + case PGTABLE_LEVEL_PUD: atomic_inc(&folio->_entire_mapcount); folio_inc_large_mapcount(folio, dst_vma); break; + default: + BUILD_BUG(); } } =20 @@ -609,13 +601,13 @@ static __always_inline void __folio_dup_file_rmap(str= uct folio *folio, static inline void folio_dup_file_rmap_ptes(struct folio *folio, struct page *page, int nr_pages, struct vm_area_struct *dst_vma) { - __folio_dup_file_rmap(folio, page, nr_pages, dst_vma, RMAP_LEVEL_PTE); + __folio_dup_file_rmap(folio, page, nr_pages, dst_vma, PGTABLE_LEVEL_PTE); } =20 static __always_inline void folio_dup_file_rmap_pte(struct folio *folio, struct page *page, struct vm_area_struct *dst_vma) { - __folio_dup_file_rmap(folio, page, 1, dst_vma, RMAP_LEVEL_PTE); + __folio_dup_file_rmap(folio, page, 1, dst_vma, PGTABLE_LEVEL_PTE); } =20 /** @@ -632,7 +624,7 @@ static inline void folio_dup_file_rmap_pmd(struct folio= *folio, struct page *page, struct vm_area_struct *dst_vma) { #ifdef CONFIG_TRANSPARENT_HUGEPAGE - __folio_dup_file_rmap(folio, page, HPAGE_PMD_NR, dst_vma, RMAP_LEVEL_PTE); + __folio_dup_file_rmap(folio, page, HPAGE_PMD_NR, dst_vma, PGTABLE_LEVEL_P= TE); #else WARN_ON_ONCE(true); #endif @@ -640,7 +632,7 @@ static inline void folio_dup_file_rmap_pmd(struct folio= *folio, =20 static __always_inline int __folio_try_dup_anon_rmap(struct folio *folio, struct page *page, int nr_pages, struct vm_area_struct *dst_vma, - struct vm_area_struct *src_vma, enum rmap_level level) + struct vm_area_struct *src_vma, enum pgtable_level level) { const int orig_nr_pages =3D nr_pages; bool maybe_pinned; @@ -665,7 +657,7 @@ static __always_inline int __folio_try_dup_anon_rmap(st= ruct folio *folio, * copying if the folio maybe pinned. */ switch (level) { - case RMAP_LEVEL_PTE: + case PGTABLE_LEVEL_PTE: if (unlikely(maybe_pinned)) { for (i =3D 0; i < nr_pages; i++) if (PageAnonExclusive(page + i)) @@ -687,8 +679,8 @@ static __always_inline int __folio_try_dup_anon_rmap(st= ruct folio *folio, } while (page++, --nr_pages > 0); folio_add_large_mapcount(folio, orig_nr_pages, dst_vma); break; - case RMAP_LEVEL_PMD: - case RMAP_LEVEL_PUD: + case PGTABLE_LEVEL_PMD: + case PGTABLE_LEVEL_PUD: if (PageAnonExclusive(page)) { if (unlikely(maybe_pinned)) return -EBUSY; @@ -697,6 +689,8 @@ static __always_inline int __folio_try_dup_anon_rmap(st= ruct folio *folio, atomic_inc(&folio->_entire_mapcount); folio_inc_large_mapcount(folio, dst_vma); break; + default: + BUILD_BUG(); } return 0; } @@ -730,7 +724,7 @@ static inline int folio_try_dup_anon_rmap_ptes(struct f= olio *folio, struct vm_area_struct *src_vma) { return __folio_try_dup_anon_rmap(folio, page, nr_pages, dst_vma, - src_vma, RMAP_LEVEL_PTE); + src_vma, PGTABLE_LEVEL_PTE); } =20 static __always_inline int folio_try_dup_anon_rmap_pte(struct folio *folio, @@ -738,7 +732,7 @@ static __always_inline int folio_try_dup_anon_rmap_pte(= struct folio *folio, struct vm_area_struct *src_vma) { return __folio_try_dup_anon_rmap(folio, page, 1, dst_vma, src_vma, - RMAP_LEVEL_PTE); + PGTABLE_LEVEL_PTE); } =20 /** @@ -770,7 +764,7 @@ static inline int folio_try_dup_anon_rmap_pmd(struct fo= lio *folio, { #ifdef CONFIG_TRANSPARENT_HUGEPAGE return __folio_try_dup_anon_rmap(folio, page, HPAGE_PMD_NR, dst_vma, - src_vma, RMAP_LEVEL_PMD); + src_vma, PGTABLE_LEVEL_PMD); #else WARN_ON_ONCE(true); return -EBUSY; @@ -778,7 +772,7 @@ static inline int folio_try_dup_anon_rmap_pmd(struct fo= lio *folio, } =20 static __always_inline int __folio_try_share_anon_rmap(struct folio *folio, - struct page *page, int nr_pages, enum rmap_level level) + struct page *page, int nr_pages, enum pgtable_level level) { VM_WARN_ON_FOLIO(!folio_test_anon(folio), folio); VM_WARN_ON_FOLIO(!PageAnonExclusive(page), folio); @@ -873,7 +867,7 @@ static __always_inline int __folio_try_share_anon_rmap(= struct folio *folio, static inline int folio_try_share_anon_rmap_pte(struct folio *folio, struct page *page) { - return __folio_try_share_anon_rmap(folio, page, 1, RMAP_LEVEL_PTE); + return __folio_try_share_anon_rmap(folio, page, 1, PGTABLE_LEVEL_PTE); } =20 /** @@ -904,7 +898,7 @@ static inline int folio_try_share_anon_rmap_pmd(struct = folio *folio, { #ifdef CONFIG_TRANSPARENT_HUGEPAGE return __folio_try_share_anon_rmap(folio, page, HPAGE_PMD_NR, - RMAP_LEVEL_PMD); + PGTABLE_LEVEL_PMD); #else WARN_ON_ONCE(true); return -EBUSY; diff --git a/mm/rmap.c b/mm/rmap.c index 84a8d8b02ef77..0e9c4041f8687 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1265,7 +1265,7 @@ static void __folio_mod_stat(struct folio *folio, int= nr, int nr_pmdmapped) =20 static __always_inline void __folio_add_rmap(struct folio *folio, struct page *page, int nr_pages, struct vm_area_struct *vma, - enum rmap_level level) + enum pgtable_level level) { atomic_t *mapped =3D &folio->_nr_pages_mapped; const int orig_nr_pages =3D nr_pages; @@ -1274,7 +1274,7 @@ static __always_inline void __folio_add_rmap(struct f= olio *folio, __folio_rmap_sanity_checks(folio, page, nr_pages, level); =20 switch (level) { - case RMAP_LEVEL_PTE: + case PGTABLE_LEVEL_PTE: if (!folio_test_large(folio)) { nr =3D atomic_inc_and_test(&folio->_mapcount); break; @@ -1300,11 +1300,11 @@ static __always_inline void __folio_add_rmap(struct= folio *folio, =20 folio_add_large_mapcount(folio, orig_nr_pages, vma); break; - case RMAP_LEVEL_PMD: - case RMAP_LEVEL_PUD: + case PGTABLE_LEVEL_PMD: + case PGTABLE_LEVEL_PUD: first =3D atomic_inc_and_test(&folio->_entire_mapcount); if (IS_ENABLED(CONFIG_NO_PAGE_MAPCOUNT)) { - if (level =3D=3D RMAP_LEVEL_PMD && first) + if (level =3D=3D PGTABLE_LEVEL_PMD && first) nr_pmdmapped =3D folio_large_nr_pages(folio); nr =3D folio_inc_return_large_mapcount(folio, vma); if (nr =3D=3D 1) @@ -1323,7 +1323,7 @@ static __always_inline void __folio_add_rmap(struct f= olio *folio, * We only track PMD mappings of PMD-sized * folios separately. */ - if (level =3D=3D RMAP_LEVEL_PMD) + if (level =3D=3D PGTABLE_LEVEL_PMD) nr_pmdmapped =3D nr_pages; nr =3D nr_pages - (nr & FOLIO_PAGES_MAPPED); /* Raced ahead of a remove and another add? */ @@ -1336,6 +1336,8 @@ static __always_inline void __folio_add_rmap(struct f= olio *folio, } folio_inc_large_mapcount(folio, vma); break; + default: + BUILD_BUG(); } __folio_mod_stat(folio, nr, nr_pmdmapped); } @@ -1427,7 +1429,7 @@ static void __page_check_anon_rmap(const struct folio= *folio, =20 static __always_inline void __folio_add_anon_rmap(struct folio *folio, struct page *page, int nr_pages, struct vm_area_struct *vma, - unsigned long address, rmap_t flags, enum rmap_level level) + unsigned long address, rmap_t flags, enum pgtable_level level) { int i; =20 @@ -1440,20 +1442,22 @@ static __always_inline void __folio_add_anon_rmap(s= truct folio *folio, =20 if (flags & RMAP_EXCLUSIVE) { switch (level) { - case RMAP_LEVEL_PTE: + case PGTABLE_LEVEL_PTE: for (i =3D 0; i < nr_pages; i++) SetPageAnonExclusive(page + i); break; - case RMAP_LEVEL_PMD: + case PGTABLE_LEVEL_PMD: SetPageAnonExclusive(page); break; - case RMAP_LEVEL_PUD: + case PGTABLE_LEVEL_PUD: /* * Keep the compiler happy, we don't support anonymous * PUD mappings. */ WARN_ON_ONCE(1); break; + default: + BUILD_BUG(); } } =20 @@ -1507,7 +1511,7 @@ void folio_add_anon_rmap_ptes(struct folio *folio, st= ruct page *page, rmap_t flags) { __folio_add_anon_rmap(folio, page, nr_pages, vma, address, flags, - RMAP_LEVEL_PTE); + PGTABLE_LEVEL_PTE); } =20 /** @@ -1528,7 +1532,7 @@ void folio_add_anon_rmap_pmd(struct folio *folio, str= uct page *page, { #ifdef CONFIG_TRANSPARENT_HUGEPAGE __folio_add_anon_rmap(folio, page, HPAGE_PMD_NR, vma, address, flags, - RMAP_LEVEL_PMD); + PGTABLE_LEVEL_PMD); #else WARN_ON_ONCE(true); #endif @@ -1609,7 +1613,7 @@ void folio_add_new_anon_rmap(struct folio *folio, str= uct vm_area_struct *vma, =20 static __always_inline void __folio_add_file_rmap(struct folio *folio, struct page *page, int nr_pages, struct vm_area_struct *vma, - enum rmap_level level) + enum pgtable_level level) { VM_WARN_ON_FOLIO(folio_test_anon(folio), folio); =20 @@ -1634,7 +1638,7 @@ static __always_inline void __folio_add_file_rmap(str= uct folio *folio, void folio_add_file_rmap_ptes(struct folio *folio, struct page *page, int nr_pages, struct vm_area_struct *vma) { - __folio_add_file_rmap(folio, page, nr_pages, vma, RMAP_LEVEL_PTE); + __folio_add_file_rmap(folio, page, nr_pages, vma, PGTABLE_LEVEL_PTE); } =20 /** @@ -1651,7 +1655,7 @@ void folio_add_file_rmap_pmd(struct folio *folio, str= uct page *page, struct vm_area_struct *vma) { #ifdef CONFIG_TRANSPARENT_HUGEPAGE - __folio_add_file_rmap(folio, page, HPAGE_PMD_NR, vma, RMAP_LEVEL_PMD); + __folio_add_file_rmap(folio, page, HPAGE_PMD_NR, vma, PGTABLE_LEVEL_PMD); #else WARN_ON_ONCE(true); #endif @@ -1672,7 +1676,7 @@ void folio_add_file_rmap_pud(struct folio *folio, str= uct page *page, { #if defined(CONFIG_TRANSPARENT_HUGEPAGE) && \ defined(CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD) - __folio_add_file_rmap(folio, page, HPAGE_PUD_NR, vma, RMAP_LEVEL_PUD); + __folio_add_file_rmap(folio, page, HPAGE_PUD_NR, vma, PGTABLE_LEVEL_PUD); #else WARN_ON_ONCE(true); #endif @@ -1680,7 +1684,7 @@ void folio_add_file_rmap_pud(struct folio *folio, str= uct page *page, =20 static __always_inline void __folio_remove_rmap(struct folio *folio, struct page *page, int nr_pages, struct vm_area_struct *vma, - enum rmap_level level) + enum pgtable_level level) { atomic_t *mapped =3D &folio->_nr_pages_mapped; int last =3D 0, nr =3D 0, nr_pmdmapped =3D 0; @@ -1689,7 +1693,7 @@ static __always_inline void __folio_remove_rmap(struc= t folio *folio, __folio_rmap_sanity_checks(folio, page, nr_pages, level); =20 switch (level) { - case RMAP_LEVEL_PTE: + case PGTABLE_LEVEL_PTE: if (!folio_test_large(folio)) { nr =3D atomic_add_negative(-1, &folio->_mapcount); break; @@ -1719,11 +1723,11 @@ static __always_inline void __folio_remove_rmap(str= uct folio *folio, =20 partially_mapped =3D nr && atomic_read(mapped); break; - case RMAP_LEVEL_PMD: - case RMAP_LEVEL_PUD: + case PGTABLE_LEVEL_PMD: + case PGTABLE_LEVEL_PUD: if (IS_ENABLED(CONFIG_NO_PAGE_MAPCOUNT)) { last =3D atomic_add_negative(-1, &folio->_entire_mapcount); - if (level =3D=3D RMAP_LEVEL_PMD && last) + if (level =3D=3D PGTABLE_LEVEL_PMD && last) nr_pmdmapped =3D folio_large_nr_pages(folio); nr =3D folio_dec_return_large_mapcount(folio, vma); if (!nr) { @@ -1743,7 +1747,7 @@ static __always_inline void __folio_remove_rmap(struc= t folio *folio, nr =3D atomic_sub_return_relaxed(ENTIRELY_MAPPED, mapped); if (likely(nr < ENTIRELY_MAPPED)) { nr_pages =3D folio_large_nr_pages(folio); - if (level =3D=3D RMAP_LEVEL_PMD) + if (level =3D=3D PGTABLE_LEVEL_PMD) nr_pmdmapped =3D nr_pages; nr =3D nr_pages - (nr & FOLIO_PAGES_MAPPED); /* Raced ahead of another remove and an add? */ @@ -1757,6 +1761,8 @@ static __always_inline void __folio_remove_rmap(struc= t folio *folio, =20 partially_mapped =3D nr && nr < nr_pmdmapped; break; + default: + BUILD_BUG(); } =20 /* @@ -1796,7 +1802,7 @@ static __always_inline void __folio_remove_rmap(struc= t folio *folio, void folio_remove_rmap_ptes(struct folio *folio, struct page *page, int nr_pages, struct vm_area_struct *vma) { - __folio_remove_rmap(folio, page, nr_pages, vma, RMAP_LEVEL_PTE); + __folio_remove_rmap(folio, page, nr_pages, vma, PGTABLE_LEVEL_PTE); } =20 /** @@ -1813,7 +1819,7 @@ void folio_remove_rmap_pmd(struct folio *folio, struc= t page *page, struct vm_area_struct *vma) { #ifdef CONFIG_TRANSPARENT_HUGEPAGE - __folio_remove_rmap(folio, page, HPAGE_PMD_NR, vma, RMAP_LEVEL_PMD); + __folio_remove_rmap(folio, page, HPAGE_PMD_NR, vma, PGTABLE_LEVEL_PMD); #else WARN_ON_ONCE(true); #endif @@ -1834,7 +1840,7 @@ void folio_remove_rmap_pud(struct folio *folio, struc= t page *page, { #if defined(CONFIG_TRANSPARENT_HUGEPAGE) && \ defined(CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD) - __folio_remove_rmap(folio, page, HPAGE_PUD_NR, vma, RMAP_LEVEL_PUD); + __folio_remove_rmap(folio, page, HPAGE_PUD_NR, vma, PGTABLE_LEVEL_PUD); #else WARN_ON_ONCE(true); #endif --=20 2.50.1 From nobody Sun Oct 5 01:46:23 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=pass(p=quarantine dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1754911640; cv=none; d=zohomail.com; s=zohoarc; b=H8JdViXpdzyvSPraT6ugZYEXlzNRGyduKDQjSAcMUbCDb+rFIKSsu2jcSO4A0No5om+mZGiu39xG8Mviq+w0PY0J5yp/sfoFZR/8DkBX2cPMB9k4Vuqycw21SdMOj4KaoOKRuOrLEk5oKZQvDFcMZ82eI5bxxY/RoyJQJv88Eww= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1754911640; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=cAxz8wtEYwnzSbLFT0UZ22FzefKNNzcleMeX5Ma3lWM=; b=KeJ75cFWPlh0vhQbCTSE9sg+4HEkdGYUPiS7MZF2KHvZYuuqq2CA1Hzo46W4pNIi7kV1gBB3Z+UxjQx2ODwqzaV48neXJ6jgD/gO5o0cdjxehCkJe1cC1Cc6+Ce7Tper0iwHPE8ALJGzlyg5zZ6iZAz2mWHZ1kdXXsctwjoGmyc= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=pass header.from= (p=quarantine dis=none) Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1754911640732988.5614177595929; Mon, 11 Aug 2025 04:27:20 -0700 (PDT) Received: from list by lists.xenproject.org with outflank-mailman.1077389.1438512 (Exim 4.92) (envelope-from ) id 1ulQg9-000883-D9; Mon, 11 Aug 2025 11:27:01 +0000 Received: by outflank-mailman (output) from mailman id 1077389.1438512; Mon, 11 Aug 2025 11:27:01 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1ulQg9-00087F-8d; Mon, 11 Aug 2025 11:27:01 +0000 Received: by outflank-mailman (input) for mailman id 1077389; Mon, 11 Aug 2025 11:26:59 +0000 Received: from se1-gles-sth1-in.inumbo.com ([159.253.27.254] helo=se1-gles-sth1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1ulQg7-0005jI-PO for xen-devel@lists.xenproject.org; Mon, 11 Aug 2025 11:26:59 +0000 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by se1-gles-sth1.inumbo.com (Halon) with ESMTPS id 17885278-76a6-11f0-a325-13f23c93f187; Mon, 11 Aug 2025 13:26:59 +0200 (CEST) Received: from mail-wm1-f72.google.com (mail-wm1-f72.google.com [209.85.128.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-617-NsEbjSgrMgSL9TmYbqp4vQ-1; Mon, 11 Aug 2025 07:26:56 -0400 Received: by mail-wm1-f72.google.com with SMTP id 5b1f17b1804b1-451d30992bcso40949665e9.2 for ; Mon, 11 Aug 2025 04:26:56 -0700 (PDT) Received: from localhost (p200300d82f06a600a397de1d2f8bb66f.dip0.t-ipconnect.de. [2003:d8:2f06:a600:a397:de1d:2f8b:b66f]) by smtp.gmail.com with UTF8SMTPSA id ffacd0b85a97d-3b79c4530b3sm41080102f8f.34.2025.08.11.04.26.52 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 11 Aug 2025 04:26:54 -0700 (PDT) X-Outflank-Mailman: Message body and most headers restored to incoming version X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 17885278-76a6-11f0-a325-13f23c93f187 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1754911617; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=cAxz8wtEYwnzSbLFT0UZ22FzefKNNzcleMeX5Ma3lWM=; b=iR1WrxDNbGUaAIjQeyLPXO0TEnwD5+xxqMDcuc7O+LhPclkuMwNCMJUr5DLu23HcfKk6+I T3N36B3f73fwPbaQWASTnvAhDwAMOdzgJu/TgvtBLhKjmGILGXnjlfj/WWT2oDt8Y/xj0H iwyCyXPSyo+hJpjJx/NaB3J2xXRESoE= X-MC-Unique: NsEbjSgrMgSL9TmYbqp4vQ-1 X-Mimecast-MFC-AGG-ID: NsEbjSgrMgSL9TmYbqp4vQ_1754911615 X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1754911615; x=1755516415; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=cAxz8wtEYwnzSbLFT0UZ22FzefKNNzcleMeX5Ma3lWM=; b=Uz+9jiyJ4Qi1/qP3l8fRQdY7aOZ7AmcLDCIJU8SjYwJowFGrBV6B0r9Nn2s1uTDu7j 4WcQgi7nXbVLECgbm1MyAGKik806ufuWbbSbXvu7vDYTXAfRklkm+cccMeXi7RzyPtfj odx1rZqcfDLPDHIwEFSjdmLrr+/PLCLino2qJukcEr9UbANbYt0mghy/2Cw8qeE/Ht2I obqgmBh1Z2/89IkG7+vTCdlb0hDTXLqZ3JSi8u97KEd7h+kIaTJJhaH2+nRFQicCqaVN yh2OpmeRdHoo5Xla3CJQKOrjDUyb0HlyCOSJfiGWOUfSBKbg1voZs1+OWfiAFM8FwHjw gP7g== X-Forwarded-Encrypted: i=1; AJvYcCWAevMYpY4Sda8k1BzIZj6XAZexcnAHIc7zwYak9O3K+3DogjUhwDZEVXW3uoW9V+5nvqlB9vkm940=@lists.xenproject.org X-Gm-Message-State: AOJu0YwH5wg25iFZ7rT40qoC7ppFsU8UmqroZtZLDvcvJRfc6g4T7eRq UXzauvH/3IpLfm+7oQFBhoNsfgmG1rJKxt5Nkv7RmbANKVfz5n2SfZqjKWmwUdttnjlrlSzsyOa 8y5mQdlrwf2saugDCL0ry5W/tBhDks1AhhAnbCYFq/f3DDyIds0/m7AUfqeoiP1ZNLpZ+ X-Gm-Gg: ASbGncuU4OugYHl6+x9pr04Xw4dFaCpJ2yjoevIk7oEn1e3QUa0BFC714X1yPknegal 0mLtOR96GW4cpIiczmUWfeAwI2HiquceN6dguhcDwh+BWjfSy0DOQc+5HNpLB6++MiuS+E7Kk/C G4ubxc6YlnURlnAqBc/mNXcW4vF6mZwq/0FUEzET4gYu14+MEV2NFJJ2bWpS8tgXZKV7dVnK74g PoKmgSQSvodCpV2lREFizxMNNLj2YHAW4gwpn+ICYjWmjcI5pQT9sK7IAk6fvgHUwg3nS/s8tU+ 4jxYb2NG1BF8MWtAZibSYK033obeZtlHq6PoddpYG56b3wdS0vH+YCiVImlwB7ekc6oirD6PU+Q Sp2pmVcMbDT1VRcWK2+0852g1 X-Received: by 2002:a05:600c:350f:b0:456:1560:7c63 with SMTP id 5b1f17b1804b1-459f4f3dde5mr124071065e9.3.1754911615200; Mon, 11 Aug 2025 04:26:55 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHKs8kYesKmGWPRfo05vS75I0Yn/yE7Pq2L/J4XRbC28zXbi7Rocgkv1+OCyQd3aXR+1J2h1Q== X-Received: by 2002:a05:600c:350f:b0:456:1560:7c63 with SMTP id 5b1f17b1804b1-459f4f3dde5mr124070425e9.3.1754911614678; Mon, 11 Aug 2025 04:26:54 -0700 (PDT) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, xen-devel@lists.xenproject.org, linux-fsdevel@vger.kernel.org, nvdimm@lists.linux.dev, linuxppc-dev@lists.ozlabs.org, David Hildenbrand , Andrew Morton , Madhavan Srinivasan , Michael Ellerman , Nicholas Piggin , Christophe Leroy , Juergen Gross , Stefano Stabellini , Oleksandr Tyshchenko , Dan Williams , Matthew Wilcox , Jan Kara , Alexander Viro , Christian Brauner , Lorenzo Stoakes , "Liam R. Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Zi Yan , Baolin Wang , Nico Pache , Ryan Roberts , Dev Jain , Barry Song , Jann Horn , Pedro Falcato , Hugh Dickins , Oscar Salvador , Lance Yang Subject: [PATCH v3 08/11] mm/memory: convert print_bad_pte() to print_bad_page_map() Date: Mon, 11 Aug 2025 13:26:28 +0200 Message-ID: <20250811112631.759341-9-david@redhat.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20250811112631.759341-1-david@redhat.com> References: <20250811112631.759341-1-david@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: 1v7BmQgpxPNvKRzGkXypQ5jUwCR6YCDHTDRpB1rcqzI_1754911615 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1754911643988116600 Content-Type: text/plain; charset="utf-8"; x-default="true" print_bad_pte() looks like something that should actually be a WARN or similar, but historically it apparently has proven to be useful to detect corruption of page tables even on production systems -- report the issue and keep the system running to make it easier to actually detect what is going wrong (e.g., multiple such messages might shed a light). As we want to unify vm_normal_page_*() handling for PTE/PMD/PUD, we'll have to take care of print_bad_pte() as well. Let's prepare for using print_bad_pte() also for non-PTEs by adjusting the implementation and renaming the function to print_bad_page_map(). Provide print_bad_pte() as a simple wrapper. Document the implicit locking requirements for the page table re-walk. To make the function a bit more readable, factor out the ratelimit check into is_bad_page_map_ratelimited() and place the printing of page table content into __print_bad_page_map_pgtable(). We'll now dump information from each level in a single line, and just stop the table walk once we hit something that is not a present page table. The report will now look something like (dumping pgd to pmd values): [ 77.943408] BUG: Bad page map in process XXX pte:80000001233f5867 [ 77.944077] addr:00007fd84bb1c000 vm_flags:08100071 anon_vma: ... [ 77.945186] pgd:10a89f067 p4d:10a89f067 pud:10e5a2067 pmd:105327067 Not using pgdp_get(), because that does not work properly on some arm configs where pgd_t is an array. Note that we are dumping all levels even when levels are folded for simplicity. Signed-off-by: David Hildenbrand Reviewed-by: Lorenzo Stoakes --- include/linux/pgtable.h | 19 ++++++++ mm/memory.c | 104 ++++++++++++++++++++++++++++++++-------- 2 files changed, 103 insertions(+), 20 deletions(-) diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index bff5c4241bf2e..33c84b38b7ec6 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -1966,6 +1966,25 @@ enum pgtable_level { PGTABLE_LEVEL_PGD, }; =20 +static inline const char *pgtable_level_to_str(enum pgtable_level level) +{ + switch (level) { + case PGTABLE_LEVEL_PTE: + return "pte"; + case PGTABLE_LEVEL_PMD: + return "pmd"; + case PGTABLE_LEVEL_PUD: + return "pud"; + case PGTABLE_LEVEL_P4D: + return "p4d"; + case PGTABLE_LEVEL_PGD: + return "pgd"; + default: + VM_WARN_ON_ONCE(1); + return "unknown"; + } +} + #endif /* !__ASSEMBLY__ */ =20 #if !defined(MAX_POSSIBLE_PHYSMEM_BITS) && !defined(CONFIG_64BIT) diff --git a/mm/memory.c b/mm/memory.c index 626caedce35e0..dc0107354d37b 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -491,22 +491,8 @@ static inline void add_mm_rss_vec(struct mm_struct *mm= , int *rss) add_mm_counter(mm, i, rss[i]); } =20 -/* - * This function is called to print an error when a bad pte - * is found. For example, we might have a PFN-mapped pte in - * a region that doesn't allow it. - * - * The calling function must still handle the error. - */ -static void print_bad_pte(struct vm_area_struct *vma, unsigned long addr, - pte_t pte, struct page *page) +static bool is_bad_page_map_ratelimited(void) { - pgd_t *pgd =3D pgd_offset(vma->vm_mm, addr); - p4d_t *p4d =3D p4d_offset(pgd, addr); - pud_t *pud =3D pud_offset(p4d, addr); - pmd_t *pmd =3D pmd_offset(pud, addr); - struct address_space *mapping; - pgoff_t index; static unsigned long resume; static unsigned long nr_shown; static unsigned long nr_unshown; @@ -518,7 +504,7 @@ static void print_bad_pte(struct vm_area_struct *vma, u= nsigned long addr, if (nr_shown =3D=3D 60) { if (time_before(jiffies, resume)) { nr_unshown++; - return; + return true; } if (nr_unshown) { pr_alert("BUG: Bad page map: %lu messages suppressed\n", @@ -529,15 +515,91 @@ static void print_bad_pte(struct vm_area_struct *vma,= unsigned long addr, } if (nr_shown++ =3D=3D 0) resume =3D jiffies + 60 * HZ; + return false; +} + +static void __print_bad_page_map_pgtable(struct mm_struct *mm, unsigned lo= ng addr) +{ + unsigned long long pgdv, p4dv, pudv, pmdv; + p4d_t p4d, *p4dp; + pud_t pud, *pudp; + pmd_t pmd, *pmdp; + pgd_t *pgdp; + + /* + * Although this looks like a fully lockless pgtable walk, it is not: + * see locking requirements for print_bad_page_map(). + */ + pgdp =3D pgd_offset(mm, addr); + pgdv =3D pgd_val(*pgdp); + + if (!pgd_present(*pgdp) || pgd_leaf(*pgdp)) { + pr_alert("pgd:%08llx\n", pgdv); + return; + } + + p4dp =3D p4d_offset(pgdp, addr); + p4d =3D p4dp_get(p4dp); + p4dv =3D p4d_val(p4d); + + if (!p4d_present(p4d) || p4d_leaf(p4d)) { + pr_alert("pgd:%08llx p4d:%08llx\n", pgdv, p4dv); + return; + } + + pudp =3D pud_offset(p4dp, addr); + pud =3D pudp_get(pudp); + pudv =3D pud_val(pud); + + if (!pud_present(pud) || pud_leaf(pud)) { + pr_alert("pgd:%08llx p4d:%08llx pud:%08llx\n", pgdv, p4dv, pudv); + return; + } + + pmdp =3D pmd_offset(pudp, addr); + pmd =3D pmdp_get(pmdp); + pmdv =3D pmd_val(pmd); + + /* + * Dumping the PTE would be nice, but it's tricky with CONFIG_HIGHPTE, + * because the table should already be mapped by the caller and + * doing another map would be bad. print_bad_page_map() should + * already take care of printing the PTE. + */ + pr_alert("pgd:%08llx p4d:%08llx pud:%08llx pmd:%08llx\n", pgdv, + p4dv, pudv, pmdv); +} + +/* + * This function is called to print an error when a bad page table entry (= e.g., + * corrupted page table entry) is found. For example, we might have a + * PFN-mapped pte in a region that doesn't allow it. + * + * The calling function must still handle the error. + * + * This function must be called during a proper page table walk, as it will + * re-walk the page table to dump information: the caller MUST prevent page + * table teardown (by holding mmap, vma or rmap lock) and MUST hold the le= af + * page table lock. + */ +static void print_bad_page_map(struct vm_area_struct *vma, + unsigned long addr, unsigned long long entry, struct page *page, + enum pgtable_level level) +{ + struct address_space *mapping; + pgoff_t index; + + if (is_bad_page_map_ratelimited()) + return; =20 mapping =3D vma->vm_file ? vma->vm_file->f_mapping : NULL; index =3D linear_page_index(vma, addr); =20 - pr_alert("BUG: Bad page map in process %s pte:%08llx pmd:%08llx\n", - current->comm, - (long long)pte_val(pte), (long long)pmd_val(*pmd)); + pr_alert("BUG: Bad page map in process %s %s:%08llx", current->comm, + pgtable_level_to_str(level), entry); + __print_bad_page_map_pgtable(vma->vm_mm, addr); if (page) - dump_page(page, "bad pte"); + dump_page(page, "bad page map"); pr_alert("addr:%px vm_flags:%08lx anon_vma:%px mapping:%px index:%lx\n", (void *)addr, vma->vm_flags, vma->anon_vma, mapping, index); pr_alert("file:%pD fault:%ps mmap:%ps mmap_prepare: %ps read_folio:%ps\n", @@ -549,6 +611,8 @@ static void print_bad_pte(struct vm_area_struct *vma, u= nsigned long addr, dump_stack(); add_taint(TAINT_BAD_PAGE, LOCKDEP_NOW_UNRELIABLE); } +#define print_bad_pte(vma, addr, pte, page) \ + print_bad_page_map(vma, addr, pte_val(pte), page, PGTABLE_LEVEL_PTE) =20 /* * vm_normal_page -- This function gets the "struct page" associated with = a pte. --=20 2.50.1 From nobody Sun Oct 5 01:46:24 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=pass(p=quarantine dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1754911645; cv=none; d=zohomail.com; s=zohoarc; b=cQ8d9QQ1T88FHNYe2Hnez+zNiwk/afqVI1dQ6HtPYFbc8p3yrqwSj3xKeyRa2Pniq5vf2LBriK9/82tZ40KrdUQZSEMCFVONYT7hv0/S2Qojl5BzDic5/ZDmSNMQElKkiPMbbUV7+xdVY+7UsvZ2NrWe63BIQncRUMCiPPguZrg= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1754911645; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=+EOzLE+RTai94Qoe+vHWeyxEes26RRi+ewlRPXeNOJQ=; b=L1UZFEGVkNcfdQguC2RNKdAk/5cgKKCGsm5My0TZPEGa9PBcXm8AsM2Z9IRJRJhuXZP92Rz8Gy/Ai8OD4iVelpHQ1hzynZrnETr+7bJAj//mWEL90I9n1/AUcl5dTTI6A1Sn/b+NtaDeEab4i9C098BOw9/R1u+fdDCntHp0ckY= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=pass header.from= (p=quarantine dis=none) Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1754911645889887.3115345264916; Mon, 11 Aug 2025 04:27:25 -0700 (PDT) Received: from list by lists.xenproject.org with outflank-mailman.1077397.1438522 (Exim 4.92) (envelope-from ) id 1ulQgC-0000Gv-OH; Mon, 11 Aug 2025 11:27:04 +0000 Received: by outflank-mailman (output) from mailman id 1077397.1438522; Mon, 11 Aug 2025 11:27:04 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1ulQgC-0000GU-Jt; Mon, 11 Aug 2025 11:27:04 +0000 Received: by outflank-mailman (input) for mailman id 1077397; Mon, 11 Aug 2025 11:27:03 +0000 Received: from se1-gles-flk1-in.inumbo.com ([94.247.172.50] helo=se1-gles-flk1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1ulQgB-000725-Ku for xen-devel@lists.xenproject.org; Mon, 11 Aug 2025 11:27:03 +0000 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by se1-gles-flk1.inumbo.com (Halon) with ESMTPS id 1904dc33-76a6-11f0-b898-0df219b8e170; Mon, 11 Aug 2025 13:27:01 +0200 (CEST) Received: from mail-wr1-f72.google.com (mail-wr1-f72.google.com [209.85.221.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-296-zm1sSPvPMrqCoENuOLkvtw-1; Mon, 11 Aug 2025 07:26:59 -0400 Received: by mail-wr1-f72.google.com with SMTP id ffacd0b85a97d-3b90ab438b9so469403f8f.2 for ; Mon, 11 Aug 2025 04:26:58 -0700 (PDT) Received: from localhost (p200300d82f06a600a397de1d2f8bb66f.dip0.t-ipconnect.de. [2003:d8:2f06:a600:a397:de1d:2f8b:b66f]) by smtp.gmail.com with UTF8SMTPSA id ffacd0b85a97d-3b79c453d6esm40173200f8f.37.2025.08.11.04.26.55 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 11 Aug 2025 04:26:56 -0700 (PDT) X-Outflank-Mailman: Message body and most headers restored to incoming version X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 1904dc33-76a6-11f0-b898-0df219b8e170 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1754911620; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=+EOzLE+RTai94Qoe+vHWeyxEes26RRi+ewlRPXeNOJQ=; b=hVVP4aueCoOS7lXCo5xbS+v7oP0xgL8ED/uuvs96r2r+XPEQFC+n58IffKorap5T7fHJrL yT7LMfceSSk69jDWhWbdXbLbhvjy37KEvsHbOpt9Rf07tU/+ZjPVpgjxc4Jzbn3oJs0EP/ IeZOU8sHJGj5IluQtLXwWiamM+NW5UA= X-MC-Unique: zm1sSPvPMrqCoENuOLkvtw-1 X-Mimecast-MFC-AGG-ID: zm1sSPvPMrqCoENuOLkvtw_1754911618 X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1754911618; x=1755516418; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=+EOzLE+RTai94Qoe+vHWeyxEes26RRi+ewlRPXeNOJQ=; b=lI9dhfdr7V54PIhojAMvMKBWU1ZVpSw9ZNbQeQfcWuCPxB5qTLfUr75BG6u9bBVTbT 3penzFpfh8SmipKnGbpT8HAQiprhhu1XRu6pwLRtcx1XKU5XQdfw29H8kuPWy+7urYX4 R1PEQ/N15RLl+5A0m5X29GJcx79BhaD5SK1rv6hKQN20i88YGUHtJyp4jaKrQ50aLiOk fQRUDacH5vbv74h3ZTgmkvdEcogoDgau9LcI7XaFzjE7Qkwu/gWpngyTXzPyaunHpgro lAL+gg0GxiKmLQc2BAmTyGCbo7bR6dj7v6gvux+qR5b1NpSmbVOPP+QzRDmJ1C/2RHWr ZxxA== X-Forwarded-Encrypted: i=1; AJvYcCVS1mJc1kMLDwpwlZmAZr9HS77uHO/rqHZYV9+I9es3L1UuSSO1GIBYBBy24saAqcxSPND9Xxlh1w0=@lists.xenproject.org X-Gm-Message-State: AOJu0YzxwYMFIZmGqIhwZ45J9bw90LtyCb0uvJ5LrgkRAfjIzdZ8Irs/ 4mIWUXoLRedL1Cm7rqWc5uyar8LjP/dC4AnEsCNM1sd9aQmBKafu613B1wpgBgULbYN3ttiDkgt vZHwjzNpxynIWAMYc7BBWZa3JvhDWeh4gMZ4le0O4aj+qNBX6kW+5MUnags36T8QEIEmS X-Gm-Gg: ASbGncsj7o/60MsuQI2+fw76dVgIcYjIDm7uA27GigFLw91Whkjiq3bu7fj2Y3+r45X wwbPAeXyo/neeHv2o2QrokpwahEP83tS1q32kee9fXKe6/6J1CVEgXHOoqeb0QepVXvVnnPPEhC dZjJUIS/dzgcDl3HR7oa/1s8t5+JARR4zCpsvZ7hjCz6pMyTVV8IEKWfriQ2mPYfjU9BJfJF9IC l3JGWzleYA3BeE6lYFlL5XyK1HafUtO2cXqazgvKVE4G4j2XgufmqxtRQpJl+80txBTlyWofrxr aBpqJRSSkv5QMINaLwPvJn48HFEPFbgJGGPox23/aL1Hb3NlE7FD0EfomzWfFtx7qKpNViGccT7 44oWh/rUQa66F5I3ygzvNVVLC X-Received: by 2002:a05:6000:40da:b0:3b7:9d87:97c6 with SMTP id ffacd0b85a97d-3b900b552ccmr11182475f8f.44.1754911617651; Mon, 11 Aug 2025 04:26:57 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHgzaZVe/bWq5ilsSm3I+nGmnBoU4tOWHlsDn35ubDMQPCamCfdsJ8Ebz2VnZDz+OwT7Da1fQ== X-Received: by 2002:a05:6000:40da:b0:3b7:9d87:97c6 with SMTP id ffacd0b85a97d-3b900b552ccmr11182439f8f.44.1754911617130; Mon, 11 Aug 2025 04:26:57 -0700 (PDT) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, xen-devel@lists.xenproject.org, linux-fsdevel@vger.kernel.org, nvdimm@lists.linux.dev, linuxppc-dev@lists.ozlabs.org, David Hildenbrand , Andrew Morton , Madhavan Srinivasan , Michael Ellerman , Nicholas Piggin , Christophe Leroy , Juergen Gross , Stefano Stabellini , Oleksandr Tyshchenko , Dan Williams , Matthew Wilcox , Jan Kara , Alexander Viro , Christian Brauner , Lorenzo Stoakes , "Liam R. Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Zi Yan , Baolin Wang , Nico Pache , Ryan Roberts , Dev Jain , Barry Song , Jann Horn , Pedro Falcato , Hugh Dickins , Oscar Salvador , Lance Yang Subject: [PATCH v3 09/11] mm/memory: factor out common code from vm_normal_page_*() Date: Mon, 11 Aug 2025 13:26:29 +0200 Message-ID: <20250811112631.759341-10-david@redhat.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20250811112631.759341-1-david@redhat.com> References: <20250811112631.759341-1-david@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: 8tLx9QVwPHJ0I7om2SMxci1oTAWjFG9Yv1KXmy5zbxg_1754911618 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1754911646774116600 Content-Type: text/plain; charset="utf-8"; x-default="true" Let's reduce the code duplication and factor out the non-pte/pmd related magic into __vm_normal_page(). To keep it simpler, check the pfn against both zero folios, which shouldn't really make a difference. It's a good question if we can even hit the !CONFIG_ARCH_HAS_PTE_SPECIAL scenario in the PMD case in practice: but doesn't really matter, as it's now all unified in vm_normal_page_pfn(). Add kerneldoc for all involved functions. Note that, as a side product, we now: * Support the find_special_page special thingy also for PMD * Don't check for is_huge_zero_pfn() anymore if we have CONFIG_ARCH_HAS_PTE_SPECIAL and the PMD is not special. The VM_WARN_ON_ONCE would catch any abuse No functional change intended. Reviewed-by: Oscar Salvador Signed-off-by: David Hildenbrand Reviewed-by: Lorenzo Stoakes --- mm/memory.c | 186 ++++++++++++++++++++++++++++++---------------------- 1 file changed, 109 insertions(+), 77 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index dc0107354d37b..78af3f243cee7 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -614,8 +614,14 @@ static void print_bad_page_map(struct vm_area_struct *= vma, #define print_bad_pte(vma, addr, pte, page) \ print_bad_page_map(vma, addr, pte_val(pte), page, PGTABLE_LEVEL_PTE) =20 -/* - * vm_normal_page -- This function gets the "struct page" associated with = a pte. +/** + * __vm_normal_page() - Get the "struct page" associated with a page table= entry. + * @vma: The VMA mapping the page table entry. + * @addr: The address where the page table entry is mapped. + * @pfn: The PFN stored in the page table entry. + * @special: Whether the page table entry is marked "special". + * @level: The page table level for error reporting purposes only. + * @entry: The page table entry value for error reporting purposes only. * * "Special" mappings do not wish to be associated with a "struct page" (e= ither * it doesn't exist, or it exists but they don't want to touch it). In this @@ -628,10 +634,10 @@ static void print_bad_page_map(struct vm_area_struct = *vma, * Selected page table walkers (such as GUP) can still identify mappings o= f the * shared zero folios and work with the underlying "struct page". * - * There are 2 broad cases. Firstly, an architecture may define a pte_spec= ial() - * pte bit, in which case this function is trivial. Secondly, an architect= ure - * may not have a spare pte bit, which requires a more complicated scheme, - * described below. + * There are 2 broad cases. Firstly, an architecture may define a "special" + * page table entry bit, such as pte_special(), in which case this functio= n is + * trivial. Secondly, an architecture may not have a spare page table + * entry bit, which requires a more complicated scheme, described below. * * A raw VM_PFNMAP mapping (ie. one that is not COWed) is always considere= d a * special mapping (even if there are underlying and valid "struct pages"). @@ -664,63 +670,94 @@ static void print_bad_page_map(struct vm_area_struct = *vma, * don't have to follow the strict linearity rule of PFNMAP mappings in * order to support COWable mappings. * + * Return: Returns the "struct page" if this is a "normal" mapping. Returns + * NULL if this is a "special" mapping. */ -struct page *vm_normal_page(struct vm_area_struct *vma, unsigned long addr, - pte_t pte) +static inline struct page *__vm_normal_page(struct vm_area_struct *vma, + unsigned long addr, unsigned long pfn, bool special, + unsigned long long entry, enum pgtable_level level) { - unsigned long pfn =3D pte_pfn(pte); - if (IS_ENABLED(CONFIG_ARCH_HAS_PTE_SPECIAL)) { - if (likely(!pte_special(pte))) - goto check_pfn; - if (vma->vm_ops && vma->vm_ops->find_special_page) - return vma->vm_ops->find_special_page(vma, addr); - if (vma->vm_flags & (VM_PFNMAP | VM_MIXEDMAP)) - return NULL; - if (is_zero_pfn(pfn)) - return NULL; - - print_bad_pte(vma, addr, pte, NULL); - return NULL; - } - - /* !CONFIG_ARCH_HAS_PTE_SPECIAL case follows: */ - - if (unlikely(vma->vm_flags & (VM_PFNMAP|VM_MIXEDMAP))) { - if (vma->vm_flags & VM_MIXEDMAP) { - if (!pfn_valid(pfn)) + if (unlikely(special)) { + if (vma->vm_ops && vma->vm_ops->find_special_page) + return vma->vm_ops->find_special_page(vma, addr); + if (vma->vm_flags & (VM_PFNMAP | VM_MIXEDMAP)) return NULL; - if (is_zero_pfn(pfn)) - return NULL; - goto out; - } else { - unsigned long off; - off =3D (addr - vma->vm_start) >> PAGE_SHIFT; - if (pfn =3D=3D vma->vm_pgoff + off) - return NULL; - if (!is_cow_mapping(vma->vm_flags)) + if (is_zero_pfn(pfn) || is_huge_zero_pfn(pfn)) return NULL; + + print_bad_page_map(vma, addr, entry, NULL, level); + return NULL; } - } + /* + * With CONFIG_ARCH_HAS_PTE_SPECIAL, any special page table + * mappings (incl. shared zero folios) are marked accordingly. + */ + } else { + if (unlikely(vma->vm_flags & (VM_PFNMAP | VM_MIXEDMAP))) { + if (vma->vm_flags & VM_MIXEDMAP) { + /* If it has a "struct page", it's "normal". */ + if (!pfn_valid(pfn)) + return NULL; + } else { + unsigned long off =3D (addr - vma->vm_start) >> PAGE_SHIFT; =20 - if (is_zero_pfn(pfn)) - return NULL; + /* Only CoW'ed anon folios are "normal". */ + if (pfn =3D=3D vma->vm_pgoff + off) + return NULL; + if (!is_cow_mapping(vma->vm_flags)) + return NULL; + } + } + + if (is_zero_pfn(pfn) || is_huge_zero_pfn(pfn)) + return NULL; + } =20 -check_pfn: if (unlikely(pfn > highest_memmap_pfn)) { - print_bad_pte(vma, addr, pte, NULL); + /* Corrupted page table entry. */ + print_bad_page_map(vma, addr, entry, NULL, level); return NULL; } - /* * NOTE! We still have PageReserved() pages in the page tables. - * eg. VDSO mappings can cause them to exist. + * For example, VDSO mappings can cause them to exist. */ -out: - VM_WARN_ON_ONCE(is_zero_pfn(pfn)); + VM_WARN_ON_ONCE(is_zero_pfn(pfn) || is_huge_zero_pfn(pfn)); return pfn_to_page(pfn); } =20 +/** + * vm_normal_page() - Get the "struct page" associated with a PTE + * @vma: The VMA mapping the @pte. + * @addr: The address where the @pte is mapped. + * @pte: The PTE. + * + * Get the "struct page" associated with a PTE. See __vm_normal_page() + * for details on "normal" and "special" mappings. + * + * Return: Returns the "struct page" if this is a "normal" mapping. Returns + * NULL if this is a "special" mapping. + */ +struct page *vm_normal_page(struct vm_area_struct *vma, unsigned long addr, + pte_t pte) +{ + return __vm_normal_page(vma, addr, pte_pfn(pte), pte_special(pte), + pte_val(pte), PGTABLE_LEVEL_PTE); +} + +/** + * vm_normal_folio() - Get the "struct folio" associated with a PTE + * @vma: The VMA mapping the @pte. + * @addr: The address where the @pte is mapped. + * @pte: The PTE. + * + * Get the "struct folio" associated with a PTE. See __vm_normal_page() + * for details on "normal" and "special" mappings. + * + * Return: Returns the "struct folio" if this is a "normal" mapping. Retur= ns + * NULL if this is a "special" mapping. + */ struct folio *vm_normal_folio(struct vm_area_struct *vma, unsigned long ad= dr, pte_t pte) { @@ -732,42 +769,37 @@ struct folio *vm_normal_folio(struct vm_area_struct *= vma, unsigned long addr, } =20 #ifdef CONFIG_PGTABLE_HAS_HUGE_LEAVES +/** + * vm_normal_page_pmd() - Get the "struct page" associated with a PMD + * @vma: The VMA mapping the @pmd. + * @addr: The address where the @pmd is mapped. + * @pmd: The PMD. + * + * Get the "struct page" associated with a PTE. See __vm_normal_page() + * for details on "normal" and "special" mappings. + * + * Return: Returns the "struct page" if this is a "normal" mapping. Returns + * NULL if this is a "special" mapping. + */ struct page *vm_normal_page_pmd(struct vm_area_struct *vma, unsigned long = addr, pmd_t pmd) { - unsigned long pfn =3D pmd_pfn(pmd); - - if (unlikely(pmd_special(pmd))) - return NULL; - - if (unlikely(vma->vm_flags & (VM_PFNMAP|VM_MIXEDMAP))) { - if (vma->vm_flags & VM_MIXEDMAP) { - if (!pfn_valid(pfn)) - return NULL; - goto out; - } else { - unsigned long off; - off =3D (addr - vma->vm_start) >> PAGE_SHIFT; - if (pfn =3D=3D vma->vm_pgoff + off) - return NULL; - if (!is_cow_mapping(vma->vm_flags)) - return NULL; - } - } - - if (is_huge_zero_pfn(pfn)) - return NULL; - if (unlikely(pfn > highest_memmap_pfn)) - return NULL; - - /* - * NOTE! We still have PageReserved() pages in the page tables. - * eg. VDSO mappings can cause them to exist. - */ -out: - return pfn_to_page(pfn); + return __vm_normal_page(vma, addr, pmd_pfn(pmd), pmd_special(pmd), + pmd_val(pmd), PGTABLE_LEVEL_PMD); } =20 +/** + * vm_normal_folio_pmd() - Get the "struct folio" associated with a PMD + * @vma: The VMA mapping the @pmd. + * @addr: The address where the @pmd is mapped. + * @pmd: The PMD. + * + * Get the "struct folio" associated with a PTE. See __vm_normal_page() + * for details on "normal" and "special" mappings. + * + * Return: Returns the "struct folio" if this is a "normal" mapping. Retur= ns + * NULL if this is a "special" mapping. + */ struct folio *vm_normal_folio_pmd(struct vm_area_struct *vma, unsigned long addr, pmd_t pmd) { --=20 2.50.1 From nobody Sun Oct 5 01:46:24 2025 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 74D252D59E8 for ; Mon, 11 Aug 2025 11:27:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754911625; cv=none; b=i66XPxZVNhio6UpCanvpfsJtQKShKHzowNnqZGegXlJaR01nn9bZoBjYTek1IVtYjaUiA7HQNudLlqJoDEV3+HsNknbOeh0p1XMJdQGqo2QSZgGXJFokpBwUU5R6h3QBcjBj1esVMyha7iu+Sz3/LwqZFQZzfxaQSUgBKCe1NS0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754911625; c=relaxed/simple; bh=NTySCngo3UWUnMxAnrpDZ1TAAIskqhgdVbp26lbNG38=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=hxnyhTsu5Myiuql9DsaHIaTPIhGwT90oUuLGtGhkE7L12NxdK6NKS1CCWQug3lv3/TiUoBbE+nVVQotQtvynvZB8cYQCQExuOTyLaJB6hsJwMY4J61N0fQ4S5qGCSDsd34ZKfoJaWZzQF66VOj5tCZaLjXbUPuR4rwihSVyEhcY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=PgOYPiwC; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="PgOYPiwC" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1754911622; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=DfX1fQMnvUejv+2+JWwebv5CirI4aCq8/b8BncrLfY4=; b=PgOYPiwCTEVhMtU+wGKsxtLBOYUCNIaeZdo/dfMtbvNurwODhz0nvt2azgs7BJGKhvvS7f XTOapxi8rOMk8agdnWGFmFiohjo/+b1uIJ3TkFEX5+q0PaC2Fa/+EzmE+vn0vfU1uWwtHX Il7DophvlLnAsbOGfXIl5RDIdd/G1P8= Received: from mail-wm1-f71.google.com (mail-wm1-f71.google.com [209.85.128.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-15-rz7Nn2EPNBSP19--lwamvQ-1; Mon, 11 Aug 2025 07:27:01 -0400 X-MC-Unique: rz7Nn2EPNBSP19--lwamvQ-1 X-Mimecast-MFC-AGG-ID: rz7Nn2EPNBSP19--lwamvQ_1754911620 Received: by mail-wm1-f71.google.com with SMTP id 5b1f17b1804b1-459e30eeea3so28456535e9.0 for ; Mon, 11 Aug 2025 04:27:01 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1754911620; x=1755516420; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=DfX1fQMnvUejv+2+JWwebv5CirI4aCq8/b8BncrLfY4=; b=D4X1gGXKgAUEvNTFZ8qhvIeiXA4Ph2WF/+krEAjVe89iRK7YTJPOZfsGl2zfmadBDM +foPO++uUHTbawX7JecSDaUqj9OH7sHnRaiaBq3K29tFyKdSwnGs06AYQLur6mBNY7T4 we1qfbVIbtNiZib2F4fjYRmJXlhEPtf+Ve3Er2v4sjCwx/TXafkzZ1KQpRVjt7yyoGg7 4MpmiwWIinkz5J/PZQgncHqQln8RzV8IYSl6znkCln2/0OA9c7BBjVFzIm8YzDsMoNk6 gM3VXQ/phN4nOTMQE0G6ICjx4JeR4yLMqctXU4S2lJS8IpMfITPOCgpA9eLrQBe/6mEc XBiA== X-Gm-Message-State: AOJu0YzW/f/He2SFlawUm76UulGHlQvNUag1a+//7ShRshhKHKfhw2tG KrVydnOP2owASfSHmbTVMnqXu/quOYFXLroPZAr/e8GujyOyGDUBJZZssFl99BMb0C7niDbahAr RScxU/E/rzE3C8FUsw1D/VDrWEWWXodqLFH/XfWPT99GjV3bRS68WYRlWSXc9HdZhTkKtikgj0q wg7qIs2I67IVyJks9pe9Mp/WPuQ6QwFOrN8tgNRtG+yHS9Qg== X-Gm-Gg: ASbGncsxz7O54D+QyzJYB5XNBGtZqlm7QWJJ05xLhQlH8BNGg1lXRP1w1KbRsq0rgFo 8ZeymU8dsKrCxRe8fMJeH4zkKF+HGF2nbD9tc9dVwhgk8ZrARC0fVPwOgtsI/Uo7EzWi7V7rpI3 avVQ9eFtGZP7WCa283cRXyAr5stzp9jGHrFQTUhkD0fU+BVNcnghUKEhQy1EP4AJhvqqzcyWSk+ Dp8uqm2lX+v1Aq2+HQYme4ET1oDfPJ2PAa2xVN5pOYZ6V2zowNOpEPFTvmJEFVN3tJpmK6Wkerp wZFYCAJaRlNDWdD5c7S5QJUBG9otonL38Ue3Lrw6uQ/D0DoAb4AUHx1lWFTj9NK9AtNKIZFyIEP /cnNIyjVp6ESNPSDnuHchitC8 X-Received: by 2002:a05:600c:c493:b0:453:66f:b96e with SMTP id 5b1f17b1804b1-459faf4758bmr80581595e9.11.1754911620408; Mon, 11 Aug 2025 04:27:00 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHBm+6FX9Xdf6wc1nTSJlLqt7RXq7PC8peG73SnR2aY3/2U0Fhn5nWzADDyt5hVIbhoPtrtZw== X-Received: by 2002:a05:600c:c493:b0:453:66f:b96e with SMTP id 5b1f17b1804b1-459faf4758bmr80581015e9.11.1754911619781; Mon, 11 Aug 2025 04:26:59 -0700 (PDT) Received: from localhost (p200300d82f06a600a397de1d2f8bb66f.dip0.t-ipconnect.de. [2003:d8:2f06:a600:a397:de1d:2f8b:b66f]) by smtp.gmail.com with UTF8SMTPSA id 5b1f17b1804b1-458be70c5f7sm376335155e9.26.2025.08.11.04.26.57 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 11 Aug 2025 04:26:59 -0700 (PDT) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, xen-devel@lists.xenproject.org, linux-fsdevel@vger.kernel.org, nvdimm@lists.linux.dev, linuxppc-dev@lists.ozlabs.org, David Hildenbrand , Andrew Morton , Madhavan Srinivasan , Michael Ellerman , Nicholas Piggin , Christophe Leroy , Juergen Gross , Stefano Stabellini , Oleksandr Tyshchenko , Dan Williams , Matthew Wilcox , Jan Kara , Alexander Viro , Christian Brauner , Lorenzo Stoakes , "Liam R. Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Zi Yan , Baolin Wang , Nico Pache , Ryan Roberts , Dev Jain , Barry Song , Jann Horn , Pedro Falcato , Hugh Dickins , Oscar Salvador , Lance Yang , Wei Yang Subject: [PATCH v3 10/11] mm: introduce and use vm_normal_page_pud() Date: Mon, 11 Aug 2025 13:26:30 +0200 Message-ID: <20250811112631.759341-11-david@redhat.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20250811112631.759341-1-david@redhat.com> References: <20250811112631.759341-1-david@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Let's introduce vm_normal_page_pud(), which ends up being fairly simple because of our new common helpers and there not being a PUD-sized zero folio. Use vm_normal_page_pud() in folio_walk_start() to resolve a TODO, structuring the code like the other (pmd/pte) cases. Defer introducing vm_normal_folio_pud() until really used. Note that we can so far get PUDs with hugetlb, daxfs and PFNMAP entries. Reviewed-by: Wei Yang Reviewed-by: Oscar Salvador Signed-off-by: David Hildenbrand Reviewed-by: Lorenzo Stoakes --- include/linux/mm.h | 2 ++ mm/memory.c | 19 +++++++++++++++++++ mm/pagewalk.c | 20 ++++++++++---------- 3 files changed, 31 insertions(+), 10 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index b626d1bacef52..8ca7d2fa71343 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2360,6 +2360,8 @@ struct folio *vm_normal_folio_pmd(struct vm_area_stru= ct *vma, unsigned long addr, pmd_t pmd); struct page *vm_normal_page_pmd(struct vm_area_struct *vma, unsigned long = addr, pmd_t pmd); +struct page *vm_normal_page_pud(struct vm_area_struct *vma, unsigned long = addr, + pud_t pud); =20 void zap_vma_ptes(struct vm_area_struct *vma, unsigned long address, unsigned long size); diff --git a/mm/memory.c b/mm/memory.c index 78af3f243cee7..6f806bf3cc994 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -809,6 +809,25 @@ struct folio *vm_normal_folio_pmd(struct vm_area_struc= t *vma, return page_folio(page); return NULL; } + +/** + * vm_normal_page_pud() - Get the "struct page" associated with a PUD + * @vma: The VMA mapping the @pud. + * @addr: The address where the @pud is mapped. + * @pud: The PUD. + * + * Get the "struct page" associated with a PUD. See __vm_normal_page() + * for details on "normal" and "special" mappings. + * + * Return: Returns the "struct page" if this is a "normal" mapping. Returns + * NULL if this is a "special" mapping. + */ +struct page *vm_normal_page_pud(struct vm_area_struct *vma, + unsigned long addr, pud_t pud) +{ + return __vm_normal_page(vma, addr, pud_pfn(pud), pud_special(pud), + pud_val(pud), PGTABLE_LEVEL_PUD); +} #endif =20 /** diff --git a/mm/pagewalk.c b/mm/pagewalk.c index 648038247a8d2..c6753d370ff4e 100644 --- a/mm/pagewalk.c +++ b/mm/pagewalk.c @@ -902,23 +902,23 @@ struct folio *folio_walk_start(struct folio_walk *fw, fw->pudp =3D pudp; fw->pud =3D pud; =20 - /* - * TODO: FW_MIGRATION support for PUD migration entries - * once there are relevant users. - */ - if (!pud_present(pud) || pud_special(pud)) { + if (pud_none(pud)) { spin_unlock(ptl); goto not_found; - } else if (!pud_leaf(pud)) { + } else if (pud_present(pud) && !pud_leaf(pud)) { spin_unlock(ptl); goto pmd_table; + } else if (pud_present(pud)) { + page =3D vm_normal_page_pud(vma, addr, pud); + if (page) + goto found; } /* - * TODO: vm_normal_page_pud() will be handy once we want to - * support PUD mappings in VM_PFNMAP|VM_MIXEDMAP VMAs. + * TODO: FW_MIGRATION support for PUD migration entries + * once there are relevant users. */ - page =3D pud_page(pud); - goto found; + spin_unlock(ptl); + goto not_found; } =20 pmd_table: --=20 2.50.1 From nobody Sun Oct 5 01:46:24 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=pass(p=quarantine dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1754911644; cv=none; d=zohomail.com; s=zohoarc; b=T1L8u80uq7mxw8HhzxDekXPdop8aEVKZ3CuIG9CcR2ZnMeUqESho1AJPj/pZRMx9UWAlHnlJ7xHSZBZ0wungKTRWRUNxVKDskSl+IO1KDZgS2qF3YeeW4RvUrzxZUCBI9T1kdrlMYL4rnXDhRHHhuhKwVVPKS3ifQptpOwiNc+0= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1754911644; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=dH19/uKoiHEMTv0r8rJb8O7lP73Me8TJFFXM58SZ7zM=; b=Bi8Sw6EhXYEIGYImUrYB6U3/pRcaFpgfJ/quoBIswTwwwYicp4+/8RK8fTty7W2JCfYd6o65pLT8re6CTLtafj9Z94nmFToWq4lm6S3blHly0OYJXI4klRBUsbqTaQ6oxIz1dC1WGNN/4G6nnLzB/ATdjP8bD04Swfa8GtINVzE= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=pass header.from= (p=quarantine dis=none) Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1754911644415114.10153363341931; Mon, 11 Aug 2025 04:27:24 -0700 (PDT) Received: from list by lists.xenproject.org with outflank-mailman.1077402.1438533 (Exim 4.92) (envelope-from ) id 1ulQgH-0000pu-3m; Mon, 11 Aug 2025 11:27:09 +0000 Received: by outflank-mailman (output) from mailman id 1077402.1438533; Mon, 11 Aug 2025 11:27:09 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1ulQgG-0000pb-Ut; Mon, 11 Aug 2025 11:27:08 +0000 Received: by outflank-mailman (input) for mailman id 1077402; Mon, 11 Aug 2025 11:27:08 +0000 Received: from se1-gles-flk1-in.inumbo.com ([94.247.172.50] helo=se1-gles-flk1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1ulQgG-000725-AI for xen-devel@lists.xenproject.org; Mon, 11 Aug 2025 11:27:08 +0000 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by se1-gles-flk1.inumbo.com (Halon) with ESMTPS id 1c05911f-76a6-11f0-b898-0df219b8e170; Mon, 11 Aug 2025 13:27:06 +0200 (CEST) Received: from mail-wm1-f72.google.com (mail-wm1-f72.google.com [209.85.128.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-22-WVUObBroM6WGvBAFp4ZeRQ-1; Mon, 11 Aug 2025 07:27:04 -0400 Received: by mail-wm1-f72.google.com with SMTP id 5b1f17b1804b1-451d3f03b74so24370385e9.3 for ; Mon, 11 Aug 2025 04:27:03 -0700 (PDT) Received: from localhost (p200300d82f06a600a397de1d2f8bb66f.dip0.t-ipconnect.de. [2003:d8:2f06:a600:a397:de1d:2f8b:b66f]) by smtp.gmail.com with UTF8SMTPSA id 5b1f17b1804b1-459e5862be7sm264659195e9.15.2025.08.11.04.27.00 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 11 Aug 2025 04:27:01 -0700 (PDT) X-Outflank-Mailman: Message body and most headers restored to incoming version X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 1c05911f-76a6-11f0-b898-0df219b8e170 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1754911625; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=dH19/uKoiHEMTv0r8rJb8O7lP73Me8TJFFXM58SZ7zM=; b=BYnqfPTN51kj9BxlD5BWsa/T+EPe2W8VlztA4ZGzQrIC1t0265HzoVMdcbWNMHKmQGuTLO cZUMDYwLQvveti3peNZnAQSpeDxDkdZGvmXQWT02YmkPkQf/DjztInidBdqfkdpwPVZiPv NxNvf6ZirjRFaUdEWfQyg9jNVnBGx6Q= X-MC-Unique: WVUObBroM6WGvBAFp4ZeRQ-1 X-Mimecast-MFC-AGG-ID: WVUObBroM6WGvBAFp4ZeRQ_1754911623 X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1754911623; x=1755516423; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=dH19/uKoiHEMTv0r8rJb8O7lP73Me8TJFFXM58SZ7zM=; b=btUPBuxqZlXuj1Fd0GgTnOwoa4EzXLi2HtuqX/MrHeV+dzpk6NDt8qp+2wfeLo+GOL SKyRN9nNfcc+sKiM9HG/XJlu/qGmStvKQVIZckCLWvXZa5Lb/unAdFoCN3grELdiJQoL eVSVjVUCLWTQwdWT7eRQaDkz9XpGW9OGcW/1pyvLK2rnxO5NvW1J5IYtidRDUJuheqiG jUjTvoojejitOiN2Wq2QktyS6YHyLqwrA/RIUcnlODt2VtjqDvo9xvYyh8QVHmL5nQtd 0USpRLx84pJCe8y7hBA4TkcSevtMHPfVMBldYIfCE/RJiDb14ilDNJ8sUwUivociB6mo S6qg== X-Forwarded-Encrypted: i=1; AJvYcCVWtB+P2xR2DD2oskFMziPrexAGTUk6VxQFB4LXubqmmpXsU4nssOtvJeOYTNv4cqHnBa92+8ioThs=@lists.xenproject.org X-Gm-Message-State: AOJu0YxPLepL3jtRLf3mFUMWhyr5rZak26mAlxxc2PteKKAZsfmcZdPU OyKWqYn+4QPsD3E+fTpiTmPpG3r8V8Dm6xJp1wf/f6Hlee0tAnPYDRBz52n28ii204U+o1AHKEi RaJNHnDbuPKOhrQeQ4zn6xVD/1ryr1PICicz7boQZgHNbhWwzEV8ZC0qsxW10uFsfN+p5 X-Gm-Gg: ASbGnctBmvuXngLy2Fu1+1/cBoMWfvWTieJA3/RlS7oEhckC2VTnlD+T2+RWoOpt2fW uTQiCPCrdRBqX8ZeSx6WrJK2tWTjJvMZDgjbakJG9o1uH+TnCXRUJKj+So0ptA7EeojSXCdPB/7 h4oyCBCXImJ7DVpgFpO3KIqhdoSFpcfDBhpZD3uj6ul5LVI+jCz2JYAY9q4joEwgWWXLLJIHiht bfVAw93PA5vVa9peazpSr3VvZOdMElLqw9vikQnlcNo7zvFICnDmZAmQgNmRUc8SMKEXpl0zL1G 8psu4MM2lwaeYO5ORaihHSwUZQ3vJ5kfY3dirTPKhCrPexzqt5zTaJ/JUhFFTqd3CZqDk1GhbIX F4gvJkrrzfrXcI1cmlAHPujpk X-Received: by 2002:a5d:5d0a:0:b0:3a5:2653:7308 with SMTP id ffacd0b85a97d-3b900b5742bmr9602283f8f.57.1754911622728; Mon, 11 Aug 2025 04:27:02 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGGXEgHnsOD9ZoHC3rozcCl2T1/SKhNFUvb83wEfWy/U4w8SmZE0De8ROcIUkzkJT6XleOZhA== X-Received: by 2002:a5d:5d0a:0:b0:3a5:2653:7308 with SMTP id ffacd0b85a97d-3b900b5742bmr9602254f8f.57.1754911622182; Mon, 11 Aug 2025 04:27:02 -0700 (PDT) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, xen-devel@lists.xenproject.org, linux-fsdevel@vger.kernel.org, nvdimm@lists.linux.dev, linuxppc-dev@lists.ozlabs.org, David Hildenbrand , Andrew Morton , Madhavan Srinivasan , Michael Ellerman , Nicholas Piggin , Christophe Leroy , Juergen Gross , Stefano Stabellini , Oleksandr Tyshchenko , Dan Williams , Matthew Wilcox , Jan Kara , Alexander Viro , Christian Brauner , Lorenzo Stoakes , "Liam R. Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Zi Yan , Baolin Wang , Nico Pache , Ryan Roberts , Dev Jain , Barry Song , Jann Horn , Pedro Falcato , Hugh Dickins , Oscar Salvador , Lance Yang , David Vrabel , Wei Yang Subject: [PATCH v3 11/11] mm: rename vm_ops->find_special_page() to vm_ops->find_normal_page() Date: Mon, 11 Aug 2025 13:26:31 +0200 Message-ID: <20250811112631.759341-12-david@redhat.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20250811112631.759341-1-david@redhat.com> References: <20250811112631.759341-1-david@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: 366KhDGigQpE7Fn0odaRIeW7EKBxbwAR70ugJmENHGQ_1754911623 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1754911645771124100 Content-Type: text/plain; charset="utf-8"; x-default="true" ... and hide it behind a kconfig option. There is really no need for any !xen code to perform this check. The naming is a bit off: we want to find the "normal" page when a PTE was marked "special". So it's really not "finding a special" page. Improve the documentation, and add a comment in the code where XEN ends up performing the pte_mkspecial() through a hypercall. More details can be found in commit 923b2919e2c3 ("xen/gntdev: mark userspace PTEs as special on x86 PV guests"). Cc: David Vrabel Reviewed-by: Oscar Salvador Reviewed-by: Lorenzo Stoakes Reviewed-by: Wei Yang Signed-off-by: David Hildenbrand --- drivers/xen/Kconfig | 1 + drivers/xen/gntdev.c | 5 +++-- include/linux/mm.h | 18 +++++++++++++----- mm/Kconfig | 2 ++ mm/memory.c | 12 ++++++++++-- tools/testing/vma/vma_internal.h | 18 +++++++++++++----- 6 files changed, 42 insertions(+), 14 deletions(-) diff --git a/drivers/xen/Kconfig b/drivers/xen/Kconfig index 24f485827e039..f9a35ed266ecf 100644 --- a/drivers/xen/Kconfig +++ b/drivers/xen/Kconfig @@ -138,6 +138,7 @@ config XEN_GNTDEV depends on XEN default m select MMU_NOTIFIER + select FIND_NORMAL_PAGE help Allows userspace processes to use grants. =20 diff --git a/drivers/xen/gntdev.c b/drivers/xen/gntdev.c index 1f21607656182..26f13b37c78e6 100644 --- a/drivers/xen/gntdev.c +++ b/drivers/xen/gntdev.c @@ -321,6 +321,7 @@ static int find_grant_ptes(pte_t *pte, unsigned long ad= dr, void *data) BUG_ON(pgnr >=3D map->count); pte_maddr =3D arbitrary_virt_to_machine(pte).maddr; =20 + /* Note: this will perform a pte_mkspecial() through the hypercall. */ gnttab_set_map_op(&map->map_ops[pgnr], pte_maddr, flags, map->grants[pgnr].ref, map->grants[pgnr].domid); @@ -528,7 +529,7 @@ static void gntdev_vma_close(struct vm_area_struct *vma) gntdev_put_map(priv, map); } =20 -static struct page *gntdev_vma_find_special_page(struct vm_area_struct *vm= a, +static struct page *gntdev_vma_find_normal_page(struct vm_area_struct *vma, unsigned long addr) { struct gntdev_grant_map *map =3D vma->vm_private_data; @@ -539,7 +540,7 @@ static struct page *gntdev_vma_find_special_page(struct= vm_area_struct *vma, static const struct vm_operations_struct gntdev_vmops =3D { .open =3D gntdev_vma_open, .close =3D gntdev_vma_close, - .find_special_page =3D gntdev_vma_find_special_page, + .find_normal_page =3D gntdev_vma_find_normal_page, }; =20 /* ------------------------------------------------------------------ */ diff --git a/include/linux/mm.h b/include/linux/mm.h index 8ca7d2fa71343..3868ca1a25f9c 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -657,13 +657,21 @@ struct vm_operations_struct { struct mempolicy *(*get_policy)(struct vm_area_struct *vma, unsigned long addr, pgoff_t *ilx); #endif +#ifdef CONFIG_FIND_NORMAL_PAGE /* - * Called by vm_normal_page() for special PTEs to find the - * page for @addr. This is useful if the default behavior - * (using pte_page()) would not find the correct page. + * Called by vm_normal_page() for special PTEs in @vma at @addr. This + * allows for returning a "normal" page from vm_normal_page() even + * though the PTE indicates that the "struct page" either does not exist + * or should not be touched: "special". + * + * Do not add new users: this really only works when a "normal" page + * was mapped, but then the PTE got changed to something weird (+ + * marked special) that would not make pte_pfn() identify the originally + * inserted page. */ - struct page *(*find_special_page)(struct vm_area_struct *vma, - unsigned long addr); + struct page *(*find_normal_page)(struct vm_area_struct *vma, + unsigned long addr); +#endif /* CONFIG_FIND_NORMAL_PAGE */ }; =20 #ifdef CONFIG_NUMA_BALANCING diff --git a/mm/Kconfig b/mm/Kconfig index e443fe8cd6cf2..59a04d0b2e272 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -1381,6 +1381,8 @@ config PT_RECLAIM =20 Note: now only empty user PTE page table pages will be reclaimed. =20 +config FIND_NORMAL_PAGE + def_bool n =20 source "mm/damon/Kconfig" =20 diff --git a/mm/memory.c b/mm/memory.c index 6f806bf3cc994..002c28795d8b7 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -639,6 +639,12 @@ static void print_bad_page_map(struct vm_area_struct *= vma, * trivial. Secondly, an architecture may not have a spare page table * entry bit, which requires a more complicated scheme, described below. * + * With CONFIG_FIND_NORMAL_PAGE, we might have the "special" bit set on + * page table entries that actually map "normal" pages: however, that page + * cannot be looked up through the PFN stored in the page table entry, but + * instead will be looked up through vm_ops->find_normal_page(). So far, t= his + * only applies to PTEs. + * * A raw VM_PFNMAP mapping (ie. one that is not COWed) is always considere= d a * special mapping (even if there are underlying and valid "struct pages"). * COWed pages of a VM_PFNMAP are always normal. @@ -679,8 +685,10 @@ static inline struct page *__vm_normal_page(struct vm_= area_struct *vma, { if (IS_ENABLED(CONFIG_ARCH_HAS_PTE_SPECIAL)) { if (unlikely(special)) { - if (vma->vm_ops && vma->vm_ops->find_special_page) - return vma->vm_ops->find_special_page(vma, addr); +#ifdef CONFIG_FIND_NORMAL_PAGE + if (vma->vm_ops && vma->vm_ops->find_normal_page) + return vma->vm_ops->find_normal_page(vma, addr); +#endif /* CONFIG_FIND_NORMAL_PAGE */ if (vma->vm_flags & (VM_PFNMAP | VM_MIXEDMAP)) return NULL; if (is_zero_pfn(pfn) || is_huge_zero_pfn(pfn)) diff --git a/tools/testing/vma/vma_internal.h b/tools/testing/vma/vma_inter= nal.h index 3639aa8dd2b06..cb1c2a8afe265 100644 --- a/tools/testing/vma/vma_internal.h +++ b/tools/testing/vma/vma_internal.h @@ -467,13 +467,21 @@ struct vm_operations_struct { struct mempolicy *(*get_policy)(struct vm_area_struct *vma, unsigned long addr, pgoff_t *ilx); #endif +#ifdef CONFIG_FIND_NORMAL_PAGE /* - * Called by vm_normal_page() for special PTEs to find the - * page for @addr. This is useful if the default behavior - * (using pte_page()) would not find the correct page. + * Called by vm_normal_page() for special PTEs in @vma at @addr. This + * allows for returning a "normal" page from vm_normal_page() even + * though the PTE indicates that the "struct page" either does not exist + * or should not be touched: "special". + * + * Do not add new users: this really only works when a "normal" page + * was mapped, but then the PTE got changed to something weird (+ + * marked special) that would not make pte_pfn() identify the originally + * inserted page. */ - struct page *(*find_special_page)(struct vm_area_struct *vma, - unsigned long addr); + struct page *(*find_normal_page)(struct vm_area_struct *vma, + unsigned long addr); +#endif /* CONFIG_FIND_NORMAL_PAGE */ }; =20 struct vm_unmapped_area_info { --=20 2.50.1