From nobody Sat Jul 25 15:51:08 2026 Received: from mail-pl1-f182.google.com (mail-pl1-f182.google.com [209.85.214.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5F61D414A0F for ; Thu, 16 Jul 2026 12:42:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.182 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784205773; cv=none; b=G+OAJI4VnqHA2kaJuIUTlFC98JuseXlMyXeakVitdWIFgL4PW3lGd5Amm9f/03iR2KkRlAkxWOtB+8m+smUA7N1w/3svzj1kRvWTvLBlLNtpBaFkdRKujEc+gSePvt7CVcXLSGBBtyzG/NwQX5nuFdXg8iylz1GscxcMTCYUDv0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784205773; c=relaxed/simple; bh=jmB8ZxaP9KZ9H5hjyIDQqs433spDH1VvQeRVLO+7GUs=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=kT23ztah2w5zCAx/dmSlbuF6T23P8hsoEfwlV1E7VKasqGRa/dmBsm9BN6FtUfdx5oiZfMtoz87FyHlV5uLuvRosq9T+0j/ajztiNbO7Zy08T9cnzZdYMXBB89+Rk9bx4l+QvxNby5BUVVeRuUopo52hy+O1IDBRr2NsccJpQV8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=P9+z8fHh; arc=none smtp.client-ip=209.85.214.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="P9+z8fHh" Received: by mail-pl1-f182.google.com with SMTP id d9443c01a7336-2cacf197759so54057995ad.2 for ; Thu, 16 Jul 2026 05:42:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1784205771; x=1784810571; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=ldlfwiuLgMW2soE43Rq8OVhlEemN/KhXMrXMzIbx114=; b=P9+z8fHh1N/OKq9OL7RTrKa97dgUNt51cIK+ukj8LHEsIGohEQgvmrSj+qckvzk5Lm /lMgeCUdDFDFHHT0wxp2mjdO6NgcQC3ctH4AmfUKoxB+UuQo0r3Mx2A+pIRjaqrllyKj q0v7KRT/weXliMYhn/PiOjPesRRpJw0+Z6lkr+fQqPf0Fmm3442uQadFfJndNAo6VESD ojx2zg749LzadpGbNP7NoOQAMoKM0OD8xetnorJ92lZmfIIoEgEg+8+BtPhsgg+TIqWd pJFBEF5MdisH0Gx8J7PKq9F/UINF0nXRXgWAZRX64m8hD2A8UDB6zcQC6hKyoiw5EF8a zV6g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1784205771; x=1784810571; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=ldlfwiuLgMW2soE43Rq8OVhlEemN/KhXMrXMzIbx114=; b=gCHOQw6IJ0zTdykH5Ef7PbmA1Sh3bhLIidPgkT9uB4O9aJW6tOhAz8sHkyU89MVBNc L5O0uuyhgtZgpTTFHGnbeRyzJCH19QZ206AUEJUaU1qZyweNF6+zlavQ27cXrotxiR/f oiZB0E07UNbfF8d9uKkG+YG467hoPfM2uLybMgTvfGSlRs9HsOmpRddtnPd3dbOyK4dG 5LfBRwkJ14LtH+k4WBYg779zPesKiMwiMvabEySTXztQt4NEw32rg1wXOjyqjxVDQIUH qnTMCyuqI1ZFpCyODls2ym7PI7mK8wHRUKLNfYhVsEwH0dX2vyHaNwA13wzqdGCFo2CT t15Q== X-Forwarded-Encrypted: i=1; AHgh+RoivIpNKgkf+UjaUelFiOrwZYG9yqZrwq9ma4T9I0e0MgDcnk7kDUhu56R9Ovdxp9RVRyUDKhdXY2zeO6Y=@vger.kernel.org X-Gm-Message-State: AOJu0YxD5dQdvJ+4Zv2fm5LblVIXZleFSOhEi7saO258LwX1T+hO5VSS 84LrDPwu3LO3UewUga3qAOG9UHYlj/dLN7J/GXLWjGXZK4UL+syNiyuSgd5thh1BcLU= X-Gm-Gg: AfdE7cn9RmS4N7RwZuQ4YPziGA3riWI3fPJj9xidDStYU2dQt8ofNSpvG/P4hnz4XVZ OLkxU4d1Q+FKXWrAc3Mj4ql3dcihGUmNKCfFsGnyKtAD6w8wwOnyzNITDUa87U2aFaeo1+wcRzS pfHMCaxAd12XPqBBfD2Jo8d4j+6vUIUjdyyZrI6h/m7Z0FVtLj7DvPREPZyAS1+7FpqS4z9K03O IcBJ7RqDqYiJsUT4PCrELk+mEQqE0z+o2hWz3lfwMLTbdZdoUKt9tsv8Mw4ZqH0LsZ2Rjvp+y0U IaXY/lSQ6XaaYXgGFxJNLKAfUBJgxMNgHaVDNWH7D9Gx+g3VB5L8byMDzai+mNxBZz16lOv6Nh6 IgaC/mNiSUp14IoZ+XA9CCNYUJXSurhAKC1z8I33rNO2hNd1NycuqCWLbTufdMwCVCj+dlH/W9H 3Q+iWhT6wQKyPf+eEJ98jdW1s7+QWweRXaTA== X-Received: by 2002:a05:6a21:10b:b0:3bf:b1d3:393e with SMTP id adf61e73a8af0-3c110009b76mr25226010637.2.1784205770620; Thu, 16 Jul 2026 05:42:50 -0700 (PDT) Received: from L6YN4KR4K9.bytedance.net ([2408:8606:1800:501::2:f]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-cb258eeaba2sm1730793a12.12.2026.07.16.05.42.42 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 16 Jul 2026 05:42:50 -0700 (PDT) From: Yunhui Cui To: linux-riscv@lists.infradead.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: linux-arch@vger.kernel.org, kvm-riscv@lists.infradead.org, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-efi@vger.kernel.org, linux-perf-users@vger.kernel.org, kasan-dev@googlegroups.com, Yunhui Cui Subject: [PATCH v2 01/15] riscv: introduce raw PTE helpers Date: Thu, 16 Jul 2026 20:41:36 +0800 Message-Id: <2bde82cde4c262d35663d386e8a16455c76ec3e6.1784201104.git.cuiyunhui@bytedance.com> X-Mailer: git-send-email 2.39.2 (Apple Git-143) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Svnapot needs RISC-V to distinguish between two PTE views: the raw encoded entries used by architecture code, and the logical per-page PTE view that will later be exposed to generic MM paths. Split the low-level RISC-V PTE operations from the public helpers so the raw helpers can keep operating on the hardware encoding, while follow-up changes build the Svnapot-aware public view on top. Signed-off-by: Yunhui Cui --- arch/riscv/include/asm/pgtable.h | 129 +++++++++++++++++++++++++++---- 1 file changed, 114 insertions(+), 15 deletions(-) diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgta= ble.h index a810655ce8f9b..8ca5da9534967 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -9,6 +9,7 @@ #include #include =20 +#include #include =20 #ifndef CONFIG_MMU @@ -628,11 +629,13 @@ static inline int pte_same(pte_t pte_a, pte_t pte_b) * a page table are directly modified. Thus, the following hook is * made available. */ -static inline void set_pte(pte_t *ptep, pte_t pteval) +static inline void __set_pte(pte_t *ptep, pte_t pteval) { WRITE_ONCE(*ptep, pteval); } =20 +#define __set_pte __set_pte + void flush_icache_pte(struct mm_struct *mm, pte_t pte); =20 static inline void __set_pte_at(struct mm_struct *mm, pte_t *ptep, pte_t p= teval) @@ -640,13 +643,13 @@ static inline void __set_pte_at(struct mm_struct *mm,= pte_t *ptep, pte_t pteval) if (pte_present(pteval) && pte_exec(pteval)) flush_icache_pte(mm, pteval); =20 - set_pte(ptep, pteval); + __set_pte(ptep, pteval); } =20 #define PFN_PTE_SHIFT _PAGE_PFN_SHIFT =20 -static inline void set_ptes(struct mm_struct *mm, unsigned long addr, - pte_t *ptep, pte_t pteval, unsigned int nr) +static inline void __set_ptes(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, pte_t pteval, unsigned int nr) { page_table_check_ptes_set(mm, addr, ptep, pteval, nr); =20 @@ -658,24 +661,85 @@ static inline void set_ptes(struct mm_struct *mm, uns= igned long addr, pte_val(pteval) +=3D 1 << _PAGE_PFN_SHIFT; } } -#define set_ptes set_ptes =20 -static inline void pte_clear(struct mm_struct *mm, - unsigned long addr, pte_t *ptep) +#define __set_ptes __set_ptes + +static inline void __pte_clear(struct mm_struct *mm, + unsigned long addr, pte_t *ptep) { __set_pte_at(mm, ptep, __pte(0)); } =20 +#define __pte_clear __pte_clear + +#define __ptep_get __ptep_get +static inline pte_t __ptep_get(pte_t *ptep) +{ + return READ_ONCE(*ptep); +} + +#define __ptep_get_lockless __ptep_get_lockless +static inline pte_t __ptep_get_lockless(pte_t *ptep) +{ + return __ptep_get(ptep); +} + +static inline void __clear_young_dirty_pte(struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep, + pte_t pte, cydp_t flags) +{ + pte_t old_pte; + + do { + old_pte =3D pte; + + if (flags & CYDP_CLEAR_YOUNG) + pte =3D pte_mkold(pte); + if (flags & CYDP_CLEAR_DIRTY) + pte =3D pte_mkclean(pte); + + pte_val(pte) =3D cmpxchg_relaxed(&pte_val(*ptep), + pte_val(old_pte), + pte_val(pte)); + } while (pte_val(pte) !=3D pte_val(old_pte)); +} + +static inline void __clear_young_dirty_ptes(struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep, + unsigned int nr, cydp_t flags) +{ + pte_t pte; + + for (;;) { + pte =3D __ptep_get(ptep); + + if (flags =3D=3D (CYDP_CLEAR_YOUNG | CYDP_CLEAR_DIRTY)) + __set_pte(ptep, pte_mkclean(pte_mkold(pte))); + else + __clear_young_dirty_pte(vma, addr, ptep, pte, flags); + + if (--nr =3D=3D 0) + break; + ptep++; + addr +=3D PAGE_SIZE; + } +} + #define __HAVE_ARCH_PTEP_SET_ACCESS_FLAGS /* defined in mm/pgtable.c */ extern int ptep_set_access_flags(struct vm_area_struct *vma, unsigned long= address, pte_t *ptep, pte_t entry, int dirty); +int __ptep_set_access_flags(struct vm_area_struct *vma, + unsigned long address, pte_t *ptep, + pte_t entry, int dirty); #define __HAVE_ARCH_PTEP_TEST_AND_CLEAR_YOUNG /* defined in mm/pgtable.c */ bool ptep_test_and_clear_young(struct vm_area_struct *vma, - unsigned long address, pte_t *ptep); + unsigned long address, pte_t *ptep); +bool __ptep_test_and_clear_young(struct vm_area_struct *vma, + unsigned long address, pte_t *ptep); =20 #define __HAVE_ARCH_PTEP_GET_AND_CLEAR -static inline pte_t ptep_get_and_clear(struct mm_struct *mm, - unsigned long address, pte_t *ptep) +static inline pte_t __ptep_get_and_clear(struct mm_struct *mm, + unsigned long address, pte_t *ptep) { #ifdef CONFIG_SMP pte_t pte =3D __pte(xchg(&ptep->pte, 0)); @@ -690,9 +754,24 @@ static inline pte_t ptep_get_and_clear(struct mm_struc= t *mm, return pte; } =20 +#define __ptep_get_and_clear __ptep_get_and_clear + +static inline pte_t __ptep_clear_flush(struct vm_area_struct *vma, + unsigned long address, pte_t *ptep) +{ + pte_t pte =3D __ptep_get_and_clear(vma->vm_mm, address, ptep); + + if (pte_accessible(vma->vm_mm, pte)) + flush_tlb_page(vma, address); + + return pte; +} + +#define __ptep_clear_flush __ptep_clear_flush + #define __HAVE_ARCH_PTEP_SET_WRPROTECT -static inline void ptep_set_wrprotect(struct mm_struct *mm, - unsigned long address, pte_t *ptep) +static inline void __ptep_set_wrprotect(struct mm_struct *mm, + unsigned long address, pte_t *ptep) { pte_t read_pte =3D READ_ONCE(*ptep); /* @@ -705,9 +784,11 @@ static inline void ptep_set_wrprotect(struct mm_struct= *mm, ((pte_val(read_pte) & ~(unsigned long)_PAGE_WRITE) | _PAGE_READ)); } =20 +#define __ptep_set_wrprotect __ptep_set_wrprotect + #define __HAVE_ARCH_PTEP_CLEAR_YOUNG_FLUSH -static inline bool ptep_clear_flush_young(struct vm_area_struct *vma, - unsigned long address, pte_t *ptep) +static inline bool __ptep_clear_flush_young(struct vm_area_struct *vma, + unsigned long address, pte_t *ptep) { /* * This comment is borrowed from x86, but applies equally to RISC-V: @@ -724,9 +805,27 @@ static inline bool ptep_clear_flush_young(struct vm_ar= ea_struct *vma, * shouldn't really matter because there's no real memory * pressure for swapout to react to. ] */ - return ptep_test_and_clear_young(vma, address, ptep); + return __ptep_test_and_clear_young(vma, address, ptep); } =20 +#define __ptep_clear_flush_young __ptep_clear_flush_young + +#define ptep_get __ptep_get +#define ptep_get_lockless __ptep_get_lockless +#define set_pte __set_pte +#define set_ptes __set_ptes + +static inline void pte_clear(struct mm_struct *mm, + unsigned long addr, pte_t *ptep) +{ + __pte_clear(mm, addr, ptep); +} + +#define ptep_get_and_clear __ptep_get_and_clear +#define clear_young_dirty_ptes __clear_young_dirty_ptes +#define ptep_set_wrprotect __ptep_set_wrprotect +#define ptep_clear_flush_young __ptep_clear_flush_young + #define pgprot_nx pgprot_nx static inline pgprot_t pgprot_nx(pgprot_t _prot) { --=20 2.39.5 From nobody Sat Jul 25 15:51:08 2026 Received: from mail-pl1-f178.google.com (mail-pl1-f178.google.com [209.85.214.178]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5E49C41E6CC for ; Thu, 16 Jul 2026 12:43:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.178 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784205782; cv=none; b=gsVIyDJv6iJEu6Uhf6S8P7s8u+fqyW29UI0I0iiFmfaY+PRwrHlZAuzMxZgTL+t3+cyR76zgL2wWcyaFUUxfdwRZWAhb3zMv9KxNClNjrr6i4qKpso6z62/uGZZUk/U4mi4Bpt74/COx2hLOrC5RUgl/Ykh5+y+n/qQxjXKTCSE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784205782; c=relaxed/simple; bh=Nx5Yl6cR2reFM/yshhQpyFD0LbkKqFxW/sLhFH1f5LA=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=t110NoSJpE/ilvD0AgqpM91+eX86TgVi+5tg5gW5bYDbRPk2M7N9xBm2gLia7lRldecvnzXiJI5Jl4p1tOhul7lCrCnzJiR93pSRxQQGvQKm0N9BUUJ67ELUadbz8mD/Q/49GLUUKMPjZeNMIJkmpIKQhTUGlWdxyKfgqiNRalk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=EtgFCpD2; arc=none smtp.client-ip=209.85.214.178 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="EtgFCpD2" Received: by mail-pl1-f178.google.com with SMTP id d9443c01a7336-2caed617615so74417185ad.3 for ; Thu, 16 Jul 2026 05:43:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1784205780; x=1784810580; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=rFSd9gJ52z3Dc4l0Wp7sqaI2N1wGspZkVOhizZNhqfU=; b=EtgFCpD25fizNROUuJswoYa2JWJH1h0tXeDyFL52FqKuZmI/LyPrWDjoUytXc+PrER 8OtNfdQyYXq19G9qIEhUNVs9ZIBS/SgP1iI+t43ZZg9QA9zsIuiNscE5gduprrf4nS36 obfCBtbUg9woy/R6IIbU1kbgI3BAMaZM2jTWtkGhiVQXrXeHydb8YS8yNc761ziWRSqf SfIMJ2Oj3hoS35shg1PcFwZcYJM3Q7mOTHaQj9254ohEq7HOFw+cbeLUZcLp3r0kA+xe nEz+wRjOR2XHrBMt6bMOWn8xA8RZDHFrz8SdP7R3LGJTpSQJTXbKsLhD0IFBVmmbxZT2 ddZw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1784205780; x=1784810580; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=rFSd9gJ52z3Dc4l0Wp7sqaI2N1wGspZkVOhizZNhqfU=; b=J0qnzN9+ti70AwjebjufQNuT2JmKnPX1diRt/RiQf+IzV56ilIYJZShrB0ZXS2fCFm O90/yfezsZLvYSlaOghOVmFALcz/1X8zZsGRK6S+kAy6vkRQ+t5oeBwGZSUV3uFh3xBH I8+X8RLmD2nYFsHl17vLLJLHU0rs1JWEGSl0SFfItJDQAYKkVTl6UC741Nf3RLuzhErQ uRJc+IECcLuuk3Ahf2V0wODxlX1iJ/FZM525XAnvE0IM2sB2luqMpgiBkcdWrvab5VKA DexXSjXDz82zNGRpW6XS0ixZ7vZj5yyD/4MaJX/8XINjpiSQxlrrsIlpitMkRGjmbuVn /E5A== X-Forwarded-Encrypted: i=1; AHgh+RqwrBjB+7CNlxENpMMeK1AceSCPMj+3/rPHgGPCy9r7BR92vjZX49MdYkoutVQisdpakLrA80LAAhoW9dU=@vger.kernel.org X-Gm-Message-State: AOJu0YzSU6Pqc0z5h5/zrffvGB5fHbb+CNh++63CsCfZykGBc1Wwu5u0 di19IvjioBPoVyzsYnFDwoMOQnNIqJlvCAU+UTfFx6PgzJMv9OZwR+QrsJPqK4HY2uQ= X-Gm-Gg: AfdE7cnjGCWj0Ak6/jpN/+PHQT8pV61XmN+ykWn2SCuG61T58qIw1OGj4jzZbfA3vm9 Llb6BDKodXcESMLqsJtk2iWwJ/oJOHXFaFdEYpzhqs3Z6HWWg3P86I403A8DyT140cuZEus23PW HP/LltnMrmJrZgukq0guSrX54t8YOUulw0vDiLB0BMcwiV4LSGxH6YQlR6q4KXOZ9ZR3okIrVdp 1gjY9by9J5DjyglS8KMpe5OF9w43KoynzW4Ts27VFRYU0UgWNiRB6MMoCfhxweZ3BFvdikgF48I sqtuilnsgs3WvWmifWKNnEJedl22rhVQwC3Y822gnUt24fLW8lT7q5HQ6c7ebOeUyEc0+Ae4iCC egIywpe2vnr7amJCHDaU3TANSurNOdVp9panOHaSv5036wgD5wNgCaCPl1idxoxx0wVw77Q5p2c /iNavmImEcSmKJv/dBVSGDXSk= X-Received: by 2002:a05:6a21:d82:b0:3c0:9c19:65a2 with SMTP id adf61e73a8af0-3c36c45b9b5mr7903332637.74.1784205779488; Thu, 16 Jul 2026 05:42:59 -0700 (PDT) Received: from L6YN4KR4K9.bytedance.net ([2408:8606:1800:501::2:f]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-cb258eeaba2sm1730793a12.12.2026.07.16.05.42.51 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 16 Jul 2026 05:42:58 -0700 (PDT) From: Yunhui Cui To: linux-riscv@lists.infradead.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: linux-arch@vger.kernel.org, kvm-riscv@lists.infradead.org, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-efi@vger.kernel.org, linux-perf-users@vger.kernel.org, kasan-dev@googlegroups.com, Yunhui Cui Subject: [PATCH v2 02/15] riscv: switch arch page-table users to raw PTE helpers Date: Thu, 16 Jul 2026 20:41:37 +0800 Message-Id: <304b13bffd2f5299da4ae80c37803e527fb1719d.1784201104.git.cuiyunhui@bytedance.com> X-Mailer: git-send-email 2.39.2 (Apple Git-143) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Convert RISC-V internal page-table walkers and mutators over to the raw PTE helper layer so they keep operating on the encoded hardware entries. This keeps arch-private users independent from the Svnapot-aware public PTE view that will be added on top for generic mm paths. Add a raw get-and-clear helper without page-table-check side effects so NAPOT-aware arch callers can report the effective non-NAPOT leaf view explicitly when tearing down contiguous mappings. Also update the hugetlb Svnapot clear/rewrite paths so they keep operating on the contiguous block with page-table-check behavior consistent with the raw-helper model. Signed-off-by: Yunhui Cui --- arch/riscv/include/asm/kfence.h | 4 +- arch/riscv/include/asm/pgtable.h | 14 +++- arch/riscv/kernel/efi.c | 4 +- arch/riscv/kernel/hibernate.c | 3 +- arch/riscv/kvm/gstage.c | 25 +++--- arch/riscv/kvm/mmu.c | 4 +- arch/riscv/mm/fault.c | 4 +- arch/riscv/mm/hugetlbpage.c | 135 ++++++++++++++++++++----------- arch/riscv/mm/init.c | 8 +- arch/riscv/mm/kasan_init.c | 16 ++-- arch/riscv/mm/pageattr.c | 12 +-- arch/riscv/mm/pgtable.c | 34 +++++--- 12 files changed, 165 insertions(+), 98 deletions(-) diff --git a/arch/riscv/include/asm/kfence.h b/arch/riscv/include/asm/kfenc= e.h index 29cb3a6ee113d..8325962a9d7e8 100644 --- a/arch/riscv/include/asm/kfence.h +++ b/arch/riscv/include/asm/kfence.h @@ -19,9 +19,9 @@ static inline bool kfence_protect_page(unsigned long addr= , bool protect) pte_t *pte =3D virt_to_kpte(addr); =20 if (protect) { - set_pte(pte, __pte(pte_val(ptep_get(pte)) & ~_PAGE_PRESENT)); + __set_pte(pte, __pte(pte_val(__ptep_get(pte)) & ~_PAGE_PRESENT)); } else { - set_pte(pte, __pte(pte_val(ptep_get(pte)) | _PAGE_PRESENT)); + __set_pte(pte, __pte(pte_val(__ptep_get(pte)) | _PAGE_PRESENT)); mark_new_valid_map(); } =20 diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgta= ble.h index 8ca5da9534967..a0f64e980e981 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -9,7 +9,6 @@ #include #include =20 -#include #include =20 #ifndef CONFIG_MMU @@ -124,6 +123,7 @@ =20 #ifndef __ASSEMBLER__ =20 +#include #include #include #include @@ -468,6 +468,11 @@ static inline pte_t pte_mkdirty(pte_t pte) return __pte(pte_val(pte) | _PAGE_DIRTY | _PAGE_SOFT_DIRTY); } =20 +static inline pte_t riscv_pte_mkhwdirty(pte_t pte) +{ + return __pte(pte_val(pte) | _PAGE_DIRTY); +} + static inline pte_t pte_mkclean(pte_t pte) { return __pte(pte_val(pte) & ~(_PAGE_DIRTY)); @@ -756,6 +761,13 @@ static inline pte_t __ptep_get_and_clear(struct mm_str= uct *mm, =20 #define __ptep_get_and_clear __ptep_get_and_clear =20 +static inline pte_t __ptep_get_and_clear_noptc(pte_t *ptep) +{ + return __pte(atomic_long_xchg((atomic_long_t *)ptep, 0)); +} + +#define __ptep_get_and_clear_noptc __ptep_get_and_clear_noptc + static inline pte_t __ptep_clear_flush(struct vm_area_struct *vma, unsigned long address, pte_t *ptep) { diff --git a/arch/riscv/kernel/efi.c b/arch/riscv/kernel/efi.c index 2d3cc57b45352..81ea2546d2922 100644 --- a/arch/riscv/kernel/efi.c +++ b/arch/riscv/kernel/efi.c @@ -60,7 +60,7 @@ int __init efi_create_mapping(struct mm_struct *mm, efi_m= emory_desc_t *md) static int __init set_permissions(pte_t *ptep, unsigned long addr, void *d= ata) { efi_memory_desc_t *md =3D data; - pte_t pte =3D ptep_get(ptep); + pte_t pte =3D __ptep_get(ptep); unsigned long val; =20 if (md->attribute & EFI_MEMORY_RO) { @@ -72,7 +72,7 @@ static int __init set_permissions(pte_t *ptep, unsigned l= ong addr, void *data) val =3D pte_val(pte) & ~_PAGE_EXEC; pte =3D __pte(val); } - set_pte(ptep, pte); + __set_pte(ptep, pte); =20 return 0; } diff --git a/arch/riscv/kernel/hibernate.c b/arch/riscv/kernel/hibernate.c index 982843828adb7..c7d5c7d4bbcde 100644 --- a/arch/riscv/kernel/hibernate.c +++ b/arch/riscv/kernel/hibernate.c @@ -186,7 +186,8 @@ static int temp_pgtable_map_pte(pmd_t *dst_pmdp, pmd_t = *src_pmdp, unsigned long pte_t pte =3D READ_ONCE(*src_ptep); =20 if (pte_present(pte)) - set_pte(dst_ptep, __pte(pte_val(pte) | pgprot_val(prot))); + __set_pte(dst_ptep, + __pte(pte_val(pte) | pgprot_val(prot))); } while (dst_ptep++, src_ptep++, start +=3D PAGE_SIZE, start < end); =20 return 0; diff --git a/arch/riscv/kvm/gstage.c b/arch/riscv/kvm/gstage.c index c4c3b79567f10..67b2c75f5cec1 100644 --- a/arch/riscv/kvm/gstage.c +++ b/arch/riscv/kvm/gstage.c @@ -88,7 +88,7 @@ bool kvm_riscv_gstage_get_leaf(struct kvm_gstage *gstage,= gpa_t addr, *ptep_level =3D current_level; ptep =3D (pte_t *)gstage->pgd; ptep =3D &ptep[gstage_pte_index(gstage, addr, current_level)]; - while (ptep && pte_val(ptep_get(ptep))) { + while (ptep && pte_val(__ptep_get(ptep))) { if (gstage_pte_leaf(ptep)) { *ptep_level =3D current_level; *ptepp =3D ptep; @@ -98,7 +98,7 @@ bool kvm_riscv_gstage_get_leaf(struct kvm_gstage *gstage,= gpa_t addr, if (current_level) { current_level--; *ptep_level =3D current_level; - ptep =3D (pte_t *)gstage_pte_page_vaddr(ptep_get(ptep)); + ptep =3D (pte_t *)gstage_pte_page_vaddr(__ptep_get(ptep)); ptep =3D &ptep[gstage_pte_index(gstage, addr, current_level)]; } else { ptep =3D NULL; @@ -152,18 +152,18 @@ int kvm_riscv_gstage_set_pte(struct kvm_gstage *gstag= e, if (gstage_pte_leaf(ptep)) return -EEXIST; =20 - if (!pte_val(ptep_get(ptep))) { + if (!pte_val(__ptep_get(ptep))) { if (!pcache) return -ENOMEM; next_ptep =3D kvm_mmu_memory_cache_alloc(pcache); if (!next_ptep) return -ENOMEM; - set_pte(ptep, pfn_pte(PFN_DOWN(__pa(next_ptep)), - __pgprot(_PAGE_TABLE))); + __set_pte(ptep, pfn_pte(PFN_DOWN(__pa(next_ptep)), + __pgprot(_PAGE_TABLE))); } else { if (gstage_pte_leaf(ptep)) return -EEXIST; - next_ptep =3D (pte_t *)gstage_pte_page_vaddr(ptep_get(ptep)); + next_ptep =3D (pte_t *)gstage_pte_page_vaddr(__ptep_get(ptep)); } =20 current_level--; @@ -171,7 +171,7 @@ int kvm_riscv_gstage_set_pte(struct kvm_gstage *gstage, } =20 if (pte_val(*ptep) !=3D pte_val(map->pte)) { - set_pte(ptep, map->pte); + __set_pte(ptep, map->pte); if (gstage_pte_leaf(ptep)) gstage_tlb_flush(gstage, current_level, map->addr); } @@ -371,18 +371,18 @@ bool kvm_riscv_gstage_op_pte(struct kvm_gstage *gstag= e, gpa_t addr, =20 WARN_ON(addr & (page_size - 1)); =20 - if (!pte_val(ptep_get(ptep))) + if (!pte_val(__ptep_get(ptep))) return false; =20 if (ptep_level && !gstage_pte_leaf(ptep)) { - next_ptep =3D (pte_t *)gstage_pte_page_vaddr(ptep_get(ptep)); + next_ptep =3D (pte_t *)gstage_pte_page_vaddr(__ptep_get(ptep)); next_ptep_level =3D ptep_level - 1; ret =3D gstage_level_to_page_size(gstage, next_ptep_level, &next_page_si= ze); if (ret) return false; =20 if (op =3D=3D GSTAGE_OP_CLEAR) - set_pte(ptep, __pte(0)); + __set_pte(ptep, __pte(0)); for (i =3D 0; i < PTRS_PER_PTE; i++) flush |=3D kvm_riscv_gstage_op_pte(gstage, addr + i * next_page_size, &next_ptep[i], next_ptep_level, op); @@ -391,9 +391,10 @@ bool kvm_riscv_gstage_op_pte(struct kvm_gstage *gstage= , gpa_t addr, } else { old_pte =3D *ptep; if (op =3D=3D GSTAGE_OP_CLEAR) - set_pte(ptep, __pte(0)); + __set_pte(ptep, __pte(0)); else if (op =3D=3D GSTAGE_OP_WP) - set_pte(ptep, __pte(pte_val(ptep_get(ptep)) & ~_PAGE_WRITE)); + __set_pte(ptep, + __pte(pte_val(__ptep_get(ptep)) & ~_PAGE_WRITE)); if (pte_val(*ptep) !=3D pte_val(old_pte)) flush =3D true; } diff --git a/arch/riscv/kvm/mmu.c b/arch/riscv/kvm/mmu.c index 082f9b2617338..48dfc88d8cea6 100644 --- a/arch/riscv/kvm/mmu.c +++ b/arch/riscv/kvm/mmu.c @@ -278,7 +278,7 @@ bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range = *range) &ptep, &ptep_level)) return false; =20 - return ptep_test_and_clear_young(NULL, 0, ptep); + return __ptep_test_and_clear_young(NULL, 0, ptep); } =20 bool kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) @@ -298,7 +298,7 @@ bool kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_r= ange *range) &ptep, &ptep_level)) return false; =20 - return pte_young(ptep_get(ptep)); + return pte_young(__ptep_get(ptep)); } =20 static bool fault_supports_gstage_huge_mapping(struct kvm_memory_slot *mem= slot, diff --git a/arch/riscv/mm/fault.c b/arch/riscv/mm/fault.c index 04ed6f8acae4f..0b725ed723023 100644 --- a/arch/riscv/mm/fault.c +++ b/arch/riscv/mm/fault.c @@ -69,7 +69,7 @@ static void show_pte(unsigned long addr) if (!ptep) goto out; =20 - pte =3D ptep_get(ptep); + pte =3D __ptep_get(ptep); pr_cont(", pte=3D%016lx", pte_val(pte)); pte_unmap(ptep); out: @@ -231,7 +231,7 @@ static inline void vmalloc_fault(struct pt_regs *regs, = int code, unsigned long a * silently loop forever. */ pte_k =3D pte_offset_kernel(pmd_k, addr); - if (!pte_present(ptep_get(pte_k))) { + if (!pte_present(__ptep_get(pte_k))) { no_context(regs, addr); return; } diff --git a/arch/riscv/mm/hugetlbpage.c b/arch/riscv/mm/hugetlbpage.c index a6d217112cf46..5ededac02161d 100644 --- a/arch/riscv/mm/hugetlbpage.c +++ b/arch/riscv/mm/hugetlbpage.c @@ -7,7 +7,7 @@ pte_t huge_ptep_get(struct mm_struct *mm, unsigned long add= r, pte_t *ptep) { unsigned long pte_num; int i; - pte_t orig_pte =3D ptep_get(ptep); + pte_t orig_pte =3D __ptep_get(ptep); =20 if (!pte_present(orig_pte) || !pte_napot(orig_pte)) return orig_pte; @@ -15,10 +15,10 @@ pte_t huge_ptep_get(struct mm_struct *mm, unsigned long= addr, pte_t *ptep) pte_num =3D napot_pte_num(napot_cont_order(orig_pte)); =20 for (i =3D 0; i < pte_num; i++, ptep++) { - pte_t pte =3D ptep_get(ptep); + pte_t pte =3D __ptep_get(ptep); =20 if (pte_dirty(pte)) - orig_pte =3D pte_mkdirty(orig_pte); + orig_pte =3D riscv_pte_mkhwdirty(orig_pte); =20 if (pte_young(pte)) orig_pte =3D pte_mkyoung(orig_pte); @@ -74,7 +74,7 @@ pte_t *huge_pte_alloc(struct mm_struct *mm, =20 out: if (pte) { - pte_t pteval =3D ptep_get_lockless(pte); + pte_t pteval =3D __ptep_get_lockless(pte); =20 WARN_ON_ONCE(pte_present(pteval) && !pte_huge(pteval)); } @@ -145,28 +145,52 @@ unsigned long hugetlb_mask_last_page(struct hstate *h) return 0UL; } =20 +static unsigned long napot_hugetlb_block_addr(pte_t pte, unsigned long add= r) +{ + unsigned long order; + + if (!pte_napot(pte)) + return addr; + + order =3D napot_cont_order(pte); + + return addr & napot_cont_mask(order); +} + +static pte_t napot_hugetlb_get_and_clear(struct mm_struct *mm, + unsigned long addr, pte_t *ptep) +{ + pte_t pte; + + pte =3D __ptep_get_and_clear_noptc(ptep); + page_table_check_pte_clear(mm, addr, pte_mknonnapot(pte, addr)); + + return pte; +} + static pte_t get_clear_contig(struct mm_struct *mm, unsigned long addr, pte_t *ptep, unsigned long ncontig) { - pte_t pte, tmp_pte; - bool present; - - pte =3D ptep_get_and_clear(mm, addr, ptep); - present =3D pte_present(pte); - while (--ncontig) { - ptep++; - addr +=3D PAGE_SIZE; - tmp_pte =3D ptep_get_and_clear(mm, addr, ptep); - if (present) { - if (pte_dirty(tmp_pte)) - pte =3D pte_mkdirty(pte); - if (pte_young(tmp_pte)) - pte =3D pte_mkyoung(pte); - } + pte_t orig_pte =3D __ptep_get(ptep); + unsigned long i; + + addr =3D napot_hugetlb_block_addr(orig_pte, addr); + if (pte_napot(orig_pte)) + ptep =3D huge_pte_offset(mm, addr, + napot_cont_size(napot_cont_order(orig_pte))); + + for (i =3D 0; i < ncontig; i++, addr +=3D PAGE_SIZE, ptep++) { + pte_t pte =3D napot_hugetlb_get_and_clear(mm, addr, ptep); + + if (pte_dirty(pte)) + orig_pte =3D riscv_pte_mkhwdirty(orig_pte); + if (pte_young(pte)) + orig_pte =3D pte_mkyoung(orig_pte); } - return pte; + + return orig_pte; } =20 static pte_t get_clear_contig_flush(struct mm_struct *mm, @@ -174,10 +198,13 @@ static pte_t get_clear_contig_flush(struct mm_struct = *mm, pte_t *ptep, unsigned long pte_num) { + pte_t pte =3D __ptep_get(ptep); pte_t orig_pte =3D get_clear_contig(mm, addr, ptep, pte_num); struct vm_area_struct vma =3D TLB_FLUSH_VMA(mm, 0); bool valid =3D !pte_none(orig_pte); =20 + addr =3D napot_hugetlb_block_addr(pte, addr); + if (valid) flush_tlb_range(&vma, addr, addr + (PAGE_SIZE * pte_num)); =20 @@ -207,14 +234,31 @@ static void clear_flush(struct mm_struct *mm, unsigned long ncontig) { struct vm_area_struct vma =3D TLB_FLUSH_VMA(mm, 0); + pte_t pte =3D __ptep_get(ptep); unsigned long i, saddr =3D addr; =20 + addr =3D napot_hugetlb_block_addr(pte, addr); + if (pte_napot(pte)) + ptep =3D huge_pte_offset(mm, addr, + napot_cont_size(napot_cont_order(pte))); + saddr =3D addr; + for (i =3D 0; i < ncontig; i++, addr +=3D pgsize, ptep++) - ptep_get_and_clear(mm, addr, ptep); + napot_hugetlb_get_and_clear(mm, addr, ptep); =20 flush_tlb_range(&vma, saddr, addr); } =20 +static void set_huge_napot_ptes(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, pte_t pte, unsigned long pte_num) +{ + unsigned long i; + + page_table_check_ptes_set(mm, addr, ptep, pte, pte_num); + for (i =3D 0; i < pte_num; i++) + __set_pte_at(mm, ptep + i, pte); +} + static int num_contig_ptes_from_size(unsigned long sz, size_t *pgsize) { unsigned long hugepage_shift; @@ -256,19 +300,18 @@ void set_huge_pte_at(struct mm_struct *mm, =20 if (!pte_present(pte)) { for (i =3D 0; i < pte_num; i++, ptep++, addr +=3D pgsize) - set_ptes(mm, addr, ptep, pte, 1); + __set_ptes(mm, addr, ptep, pte, 1); return; } =20 if (!pte_napot(pte)) { - set_ptes(mm, addr, ptep, pte, 1); + __set_ptes(mm, addr, ptep, pte, 1); return; } =20 clear_flush(mm, addr, ptep, pgsize, pte_num); =20 - for (i =3D 0; i < pte_num; i++, ptep++, addr +=3D pgsize) - set_pte_at(mm, addr, ptep, pte); + set_huge_napot_ptes(mm, addr, ptep, pte, pte_num); } =20 int huge_ptep_set_access_flags(struct vm_area_struct *vma, @@ -280,10 +323,10 @@ int huge_ptep_set_access_flags(struct vm_area_struct = *vma, struct mm_struct *mm =3D vma->vm_mm; unsigned long order; pte_t orig_pte; - int i, pte_num; + int pte_num; =20 if (!pte_napot(pte)) - return ptep_set_access_flags(vma, addr, ptep, pte, dirty); + return __ptep_set_access_flags(vma, addr, ptep, pte, dirty); =20 order =3D napot_cont_order(pte); pte_num =3D napot_pte_num(order); @@ -291,13 +334,12 @@ int huge_ptep_set_access_flags(struct vm_area_struct = *vma, orig_pte =3D get_clear_contig_flush(mm, addr, ptep, pte_num); =20 if (pte_dirty(orig_pte)) - pte =3D pte_mkdirty(pte); + pte =3D riscv_pte_mkhwdirty(pte); =20 if (pte_young(orig_pte)) pte =3D pte_mkyoung(pte); =20 - for (i =3D 0; i < pte_num; i++, addr +=3D PAGE_SIZE, ptep++) - set_pte_at(mm, addr, ptep, pte); + set_huge_napot_ptes(mm, addr, ptep, pte, pte_num); =20 return true; } @@ -306,14 +348,13 @@ pte_t huge_ptep_get_and_clear(struct mm_struct *mm, unsigned long addr, pte_t *ptep, unsigned long sz) { - size_t pgsize; - pte_t orig_pte =3D ptep_get(ptep); + pte_t orig_pte =3D __ptep_get(ptep); int pte_num; =20 if (!pte_napot(orig_pte)) - return ptep_get_and_clear(mm, addr, ptep); + return __ptep_get_and_clear(mm, addr, ptep); =20 - pte_num =3D num_contig_ptes_from_size(sz, &pgsize); + pte_num =3D napot_pte_num(napot_cont_order(orig_pte)); =20 return get_clear_contig(mm, addr, ptep, pte_num); } @@ -322,13 +363,13 @@ void huge_ptep_set_wrprotect(struct mm_struct *mm, unsigned long addr, pte_t *ptep) { - pte_t pte =3D ptep_get(ptep); + pte_t pte =3D __ptep_get(ptep); unsigned long order; pte_t orig_pte; - int i, pte_num; + int pte_num; =20 if (!pte_napot(pte)) { - ptep_set_wrprotect(mm, addr, ptep); + __ptep_set_wrprotect(mm, addr, ptep); return; } =20 @@ -339,19 +380,18 @@ void huge_ptep_set_wrprotect(struct mm_struct *mm, =20 orig_pte =3D pte_wrprotect(orig_pte); =20 - for (i =3D 0; i < pte_num; i++, addr +=3D PAGE_SIZE, ptep++) - set_pte_at(mm, addr, ptep, orig_pte); + set_huge_napot_ptes(mm, addr, ptep, orig_pte, pte_num); } =20 pte_t huge_ptep_clear_flush(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep) { - pte_t pte =3D ptep_get(ptep); + pte_t pte =3D __ptep_get(ptep); int pte_num; =20 if (!pte_napot(pte)) - return ptep_clear_flush(vma, addr, ptep); + return __ptep_clear_flush(vma, addr, ptep); =20 pte_num =3D napot_pte_num(napot_cont_order(pte)); =20 @@ -363,19 +403,16 @@ void huge_pte_clear(struct mm_struct *mm, pte_t *ptep, unsigned long sz) { - size_t pgsize; - pte_t pte =3D ptep_get(ptep); - int i, pte_num; + pte_t pte =3D __ptep_get(ptep); + int pte_num; =20 if (!pte_napot(pte)) { - pte_clear(mm, addr, ptep); + __pte_clear(mm, addr, ptep); return; } =20 - pte_num =3D num_contig_ptes_from_size(sz, &pgsize); - - for (i =3D 0; i < pte_num; i++, addr +=3D pgsize, ptep++) - pte_clear(mm, addr, ptep); + pte_num =3D napot_pte_num(napot_cont_order(pte)); + get_clear_contig(mm, addr, ptep, pte_num); } =20 static bool is_napot_size(unsigned long size) diff --git a/arch/riscv/mm/init.c b/arch/riscv/mm/init.c index e2be64bacf160..d44bb7db22b43 100644 --- a/arch/riscv/mm/init.c +++ b/arch/riscv/mm/init.c @@ -378,9 +378,9 @@ void __set_fixmap(enum fixed_addresses idx, phys_addr_t= phys, pgprot_t prot) ptep =3D &fixmap_pte[pte_index(addr)]; =20 if (pgprot_val(prot)) - set_pte(ptep, pfn_pte(phys >> PAGE_SHIFT, prot)); + __set_pte(ptep, pfn_pte(phys >> PAGE_SHIFT, prot)); else - pte_clear(&init_mm, addr, ptep); + __pte_clear(&init_mm, addr, ptep); local_flush_tlb_page(addr); } =20 @@ -1571,11 +1571,11 @@ static void __meminit remove_pte_mapping(pte_t *pte= _base, unsigned long addr, un next =3D end; =20 ptep =3D pte_base + pte_index(addr); - pte =3D ptep_get(ptep); + pte =3D __ptep_get(ptep); if (!pte_present(*ptep)) continue; =20 - pte_clear(&init_mm, addr, ptep); + __pte_clear(&init_mm, addr, ptep); if (is_vmemmap) free_vmemmap_storage(pte_page(pte), PAGE_SIZE, altmap); } diff --git a/arch/riscv/mm/kasan_init.c b/arch/riscv/mm/kasan_init.c index 1f3aa9611187f..c5d5513b206ca 100644 --- a/arch/riscv/mm/kasan_init.c +++ b/arch/riscv/mm/kasan_init.c @@ -39,9 +39,9 @@ static void __init kasan_populate_pte(pmd_t *pmd, unsigne= d long vaddr, unsigned ptep =3D pte_offset_kernel(pmd, vaddr); =20 do { - if (pte_none(ptep_get(ptep))) { + if (pte_none(__ptep_get(ptep))) { phys_addr =3D memblock_phys_alloc(PAGE_SIZE, PAGE_SIZE); - set_pte(ptep, pfn_pte(PFN_DOWN(phys_addr), PAGE_KERNEL)); + __set_pte(ptep, pfn_pte(PFN_DOWN(phys_addr), PAGE_KERNEL)); memset(__va(phys_addr), KASAN_SHADOW_INIT, PAGE_SIZE); } } while (ptep++, vaddr +=3D PAGE_SIZE, vaddr !=3D end); @@ -327,8 +327,8 @@ asmlinkage void __init kasan_early_init(void) KASAN_SHADOW_END - (1UL << (64 - KASAN_SHADOW_SCALE_SHIFT))); =20 for (i =3D 0; i < PTRS_PER_PTE; ++i) - set_pte(kasan_early_shadow_pte + i, - pfn_pte(virt_to_pfn(kasan_early_shadow_page), PAGE_KERNEL)); + __set_pte(kasan_early_shadow_pte + i, + pfn_pte(virt_to_pfn(kasan_early_shadow_page), PAGE_KERNEL)); =20 for (i =3D 0; i < PTRS_PER_PMD; ++i) set_pmd(kasan_early_shadow_pmd + i, @@ -520,10 +520,10 @@ void __init kasan_init(void) kasan_mem_to_shadow((const void *)MODULES_VADDR + SZ_2G)); =20 for (i =3D 0; i < PTRS_PER_PTE; i++) - set_pte(&kasan_early_shadow_pte[i], - mk_pte(virt_to_page(kasan_early_shadow_page), - __pgprot(_PAGE_PRESENT | _PAGE_READ | - _PAGE_ACCESSED))); + __set_pte(&kasan_early_shadow_pte[i], + mk_pte(virt_to_page(kasan_early_shadow_page), + __pgprot(_PAGE_PRESENT | _PAGE_READ | + _PAGE_ACCESSED))); =20 memset(kasan_early_shadow_page, KASAN_SHADOW_INIT, PAGE_SIZE); init_task.kasan_depth =3D 0; diff --git a/arch/riscv/mm/pageattr.c b/arch/riscv/mm/pageattr.c index 3f76db3d27699..e0271e2a0b295 100644 --- a/arch/riscv/mm/pageattr.c +++ b/arch/riscv/mm/pageattr.c @@ -68,10 +68,10 @@ static int pageattr_pmd_entry(pmd_t *pmd, unsigned long= addr, static int pageattr_pte_entry(pte_t *pte, unsigned long addr, unsigned long next, struct mm_walk *walk) { - pte_t val =3D ptep_get(pte); + pte_t val =3D __ptep_get(pte); =20 val =3D __pte(set_pageattr_masks(pte_val(val), walk)); - set_pte(pte, val); + __set_pte(pte, val); =20 return 0; } @@ -121,7 +121,7 @@ static int __split_linear_mapping_pmd(pud_t *pudp, =20 ptep_new =3D (pte_t *)page_address(pte_page); for (i =3D 0; i < PTRS_PER_PTE; ++i, ++ptep_new) - set_pte(ptep_new, pfn_pte(pfn + i, prot)); + __set_pte(ptep_new, pfn_pte(pfn + i, prot)); =20 smp_wmb(); =20 @@ -406,14 +406,14 @@ static int debug_pagealloc_set_page(pte_t *pte, unsig= ned long addr, void *data) { int enable =3D *(int *)data; =20 - unsigned long val =3D pte_val(ptep_get(pte)); + unsigned long val =3D pte_val(__ptep_get(pte)); =20 if (enable) val |=3D _PAGE_PRESENT; else val &=3D ~_PAGE_PRESENT; =20 - set_pte(pte, __pte(val)); + __set_pte(pte, __pte(val)); =20 return 0; } @@ -466,5 +466,5 @@ bool kernel_page_present(struct page *page) return true; =20 pte =3D pte_offset_kernel(pmd, addr); - return pte_present(ptep_get(pte)); + return pte_present(__ptep_get(pte)); } diff --git a/arch/riscv/mm/pgtable.c b/arch/riscv/mm/pgtable.c index 9c4427d0b1874..43bd542c6ff02 100644 --- a/arch/riscv/mm/pgtable.c +++ b/arch/riscv/mm/pgtable.c @@ -5,14 +5,13 @@ #include #include =20 -int ptep_set_access_flags(struct vm_area_struct *vma, - unsigned long address, pte_t *ptep, - pte_t entry, int dirty) +int __ptep_set_access_flags(struct vm_area_struct *vma, + unsigned long address, pte_t *ptep, + pte_t entry, int dirty) { if (riscv_has_extension_unlikely(RISCV_ISA_EXT_SVVPTC)) { - if (!pte_same(ptep_get(ptep), entry)) { + if (!pte_same(__ptep_get(ptep), entry)) { __set_pte_at(vma->vm_mm, ptep, entry); - /* Here only not svadu is impacted */ flush_tlb_page(vma, address); return true; } @@ -20,7 +19,7 @@ int ptep_set_access_flags(struct vm_area_struct *vma, return false; } =20 - if (!pte_same(ptep_get(ptep), entry)) + if (!pte_same(__ptep_get(ptep), entry)) __set_pte_at(vma->vm_mm, ptep, entry); /* * update_mmu_cache will unconditionally execute, handling both @@ -29,13 +28,30 @@ int ptep_set_access_flags(struct vm_area_struct *vma, return true; } =20 -bool ptep_test_and_clear_young(struct vm_area_struct *vma, - unsigned long address, pte_t *ptep) +int ptep_set_access_flags(struct vm_area_struct *vma, + unsigned long address, pte_t *ptep, + pte_t entry, int dirty) { - if (!pte_young(ptep_get(ptep))) + return __ptep_set_access_flags(vma, address, ptep, entry, dirty); +} + +bool __ptep_test_and_clear_young(struct vm_area_struct *vma, + unsigned long address, + pte_t *ptep) +{ + if (!pte_young(__ptep_get(ptep))) return false; + return test_and_clear_bit(_PAGE_ACCESSED_OFFSET, &pte_val(*ptep)); } +EXPORT_SYMBOL_GPL(__ptep_test_and_clear_young); + +bool ptep_test_and_clear_young(struct vm_area_struct *vma, + unsigned long address, + pte_t *ptep) +{ + return __ptep_test_and_clear_young(vma, address, ptep); +} EXPORT_SYMBOL_GPL(ptep_test_and_clear_young); =20 #ifdef CONFIG_64BIT --=20 2.39.5 From nobody Sat Jul 25 15:51:08 2026 Received: from mail-pg1-f181.google.com (mail-pg1-f181.google.com [209.85.215.181]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B310A41F5D6 for ; Thu, 16 Jul 2026 12:43:10 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.181 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784205793; cv=none; b=o/IuwGsFIolJ1gRooNC95OMEdxjYcF3cQVoZ6pB8emulzxYnxHF4P0h+uj2HqoJR3CEFtJ3UX5Vc3/C9zUraYW70xlHyUY13flf+PZDA8PEPBDp9LDi4BWZkAlGlObTM0eW9DgOBHqKZ4G42O4T2anC/lQyqllkgC71D+sooG6o= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784205793; c=relaxed/simple; bh=K4BILLDY2841H2ZYcXQzOoDFaiqflkui3P4ZaLrKaBQ=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=aNpNKXlOeQ/konBGXtEDcif4fqrC/I7O0QU1sdb8KCYVE2itGbf7pTdsjP/6ggm8vaZROTxuR/xgSWaGbgSkItffVD3hDcE3YY+hRagF5AXUtcdDFdvlIB6QtY028oO6JcvZ3lxh/dQ9HaKpBZmPReNLdiSl6CvAk42wU6oiFlI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=lMGlT2lj; arc=none smtp.client-ip=209.85.215.181 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="lMGlT2lj" Received: by mail-pg1-f181.google.com with SMTP id 41be03b00d2f7-ca97d139d8dso2109738a12.2 for ; Thu, 16 Jul 2026 05:43:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1784205790; x=1784810590; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=6QhVe14fP0y9rm/shff3jRao3xNlRAMh2yWf3Ip6xG8=; b=lMGlT2ljRCwC6NxeA0OUNZVyF2zzzdWsZ0aONLB8seSytpDUSpku11PAz86Z6EXghJ qE6f67bbwTM5EgO2s3LkiGRN2rL4qE9zkvZA6e/o/hswfuvUPgYEC2FqMT6ttQSg39jg Z6itqQGVtbcpVWoTO9AxaLKbLVqIO+D2CdBaKgRUm4yDcDyqV80+iiffMDKizX8b5Zg4 qMNya+YxyRWxs6vdiA3S87lH2qEaa7PmTFbtz4tJpAgoyLuEEJu2ffHMY4ViauotaJcr RGFCNxfgphMaLfNWmC4U8hP+kGXZW7jkdGlWBdAXzdGoSRb7nXezU6eEkZaPrsthXSQo nHVA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1784205790; x=1784810590; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=6QhVe14fP0y9rm/shff3jRao3xNlRAMh2yWf3Ip6xG8=; b=FjtUZb3jwxgvIgkFoZJ90Gq6KEDnBHV2OsFBE6HutKNX6AYBQnMDAXlPIpJUYjGtEC y2WTQdPCeNdMeE8nTrifu0LMvaT0+WLZYzm6rn6l0gCU+SPERCP20W5wl1qVLFSGXGD/ FPQReQLj2zr9ikgP17GHgkDK+KH3Bp61a+FyQOtcmrYLJ4nOyKGrfyLBaS7ZatnYS2iD loq1mguOpD4qr3xpuzd8Osl5hV0TCKdE2F355atBoz6rrt//SsMy0pLjicvzt0wqI056 7JrESi7YGy4SSiDILTqadxOHFgtS1+kNyBXyQOr71ZDTer/13MFNct5j8Trc4ZCidpRi H4rA== X-Forwarded-Encrypted: i=1; AHgh+Rp2WIU7tjULlsOLg9Q881KC4qutCU7necpLEaeoMwFPSa5gvC80MnzAi65D8nL4uUHK78BYU4nyjR/dm5Y=@vger.kernel.org X-Gm-Message-State: AOJu0YzklRZYhWkuqUpFgQzgBoIwAlk/9abrs/Zo5bffC2d54GJRJj0B PIO9KmVLLXXD5N3RDVFlp3XE57TGlvOdqOMuZDFevo1+WYfEs7GDagNdRfaMlgbszK0= X-Gm-Gg: AfdE7ckFZDNC9gcJ0aJXrv2Iz485jUjr7mPeqXZ0q/IWnyVmLGMDxtFBWyYbSsYY2uY h7qM5U12FsJbp6qj0axUd9plx9EzU3h58LUBJvsrP1f68tf3c1wuI313K/AH2JresBxGo6W9ml6 S58+2OKasMVQAvrvjJ5JoPwz8P1qWfcYtBrj/8MSgI+82wrkYxQ0hwT2W5fiyfWYBwmOx95sqKh X4FSQt8QQ6YW3yVGB/pevGpO8HKOHHkyjKPb4wOA7uHgOOZ3xTpTbYzgpHt6aj7/srAMxpBR9y5 V/VCCQjFbGJ7Iio40K1MEEfh5gTZOiTakWfoE9FyHLB7OaA1BEPgroGtQrOz2DKkL1O0wlDCCua uw9wIs5Tpw5nUSqb5LZHqOcCyNyM/UzbJRWgcJUaoN6olwBDI6cizMe22GH0drUPM+UT+Zdys10 MKlxWOCLkxkWcfOBWXLZDtbe4= X-Received: by 2002:a05:6a20:6a26:b0:3c3:990b:2292 with SMTP id adf61e73a8af0-3c3990b4230mr1076478637.57.1784205789867; Thu, 16 Jul 2026 05:43:09 -0700 (PDT) Received: from L6YN4KR4K9.bytedance.net ([2408:8606:1800:501::2:f]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-cb258eeaba2sm1730793a12.12.2026.07.16.05.43.00 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 16 Jul 2026 05:43:07 -0700 (PDT) From: Yunhui Cui To: linux-riscv@lists.infradead.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: linux-arch@vger.kernel.org, kvm-riscv@lists.infradead.org, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-efi@vger.kernel.org, linux-perf-users@vger.kernel.org, kasan-dev@googlegroups.com, Yunhui Cui Subject: [PATCH v2 03/15] riscv/mm: implement Svnapot contpte read-side helpers Date: Thu, 16 Jul 2026 20:41:38 +0800 Message-Id: <0c93b57b5315147c6ae6586371d795fb629dcc7a.1784201104.git.cuiyunhui@bytedance.com> X-Mailer: git-send-email 2.39.2 (Apple Git-143) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Svnapot keeps a contiguous mapping encoded in a block of page-table entries, while generic MM code expects the public PTE getters to return a normal per-page view for the address being queried. Without a dedicated read path, exposing the raw NAPOT encoding through the public helpers would leak arch-private representation details into generic MM users and make folded mappings harder to integrate with existing PTE walkers. Add Svnapot-aware contpte read-side helpers and route the public PTE getters through them, so generic MM sees the logical sub-PTE for the current page while RISC-V raw helpers continue to operate on the encoded entries. Signed-off-by: Yunhui Cui --- arch/riscv/include/asm/pgtable.h | 50 +++++- arch/riscv/mm/Makefile | 1 + arch/riscv/mm/contpte.c | 273 +++++++++++++++++++++++++++++++ 3 files changed, 322 insertions(+), 2 deletions(-) create mode 100644 arch/riscv/mm/contpte.c diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgta= ble.h index a0f64e980e981..fce6d04ea65db 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -312,6 +312,11 @@ static inline unsigned long pte_napot(pte_t pte) return 0; } =20 +static inline pte_t pte_mknapot(pte_t pte, unsigned int order) +{ + return pte; +} + #endif /* CONFIG_RISCV_ISA_SVNAPOT */ =20 /* Yields the page frame number (PFN) of a page table entry */ @@ -350,6 +355,11 @@ static inline int pte_present(pte_t pte) return (pte_val(pte) & (_PAGE_PRESENT | _PAGE_PROT_NONE)); } =20 +static inline bool pte_present_napot(pte_t pte) +{ + return pte_present(pte) && pte_napot(pte); +} + #define pte_accessible pte_accessible static inline unsigned long pte_accessible(struct mm_struct *mm, pte_t a) { @@ -538,6 +548,13 @@ static inline pte_t pte_swp_clear_soft_dirty(pte_t pte) #define pte_leaf_size(pte) (pte_napot(pte) ? \ napot_cont_size(napot_cont_order(pte)) :\ PAGE_SIZE) + +void __napotpte_try_fold(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, pte_t pte); +void __napotpte_try_unfold(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, pte_t pte); +pte_t napotpte_ptep_get(pte_t *ptep, pte_t orig_pte); +pte_t napotpte_ptep_get_lockless(pte_t *ptep); #endif =20 #ifdef CONFIG_ARCH_HAS_PTE_PROTNONE @@ -689,6 +706,37 @@ static inline pte_t __ptep_get_lockless(pte_t *ptep) return __ptep_get(ptep); } =20 +#ifdef CONFIG_RISCV_ISA_SVNAPOT + +#define ptep_get ptep_get +static inline pte_t ptep_get(pte_t *ptep) +{ + pte_t pte =3D __ptep_get(ptep); + + if (likely(!pte_present_napot(pte))) + return pte; + + return napotpte_ptep_get(ptep, pte); +} + +#define ptep_get_lockless ptep_get_lockless +static inline pte_t ptep_get_lockless(pte_t *ptep) +{ + pte_t pte =3D __ptep_get_lockless(ptep); + + if (likely(!pte_present_napot(pte))) + return pte; + + return napotpte_ptep_get_lockless(ptep); +} + +#else + +#define ptep_get __ptep_get +#define ptep_get_lockless __ptep_get_lockless + +#endif + static inline void __clear_young_dirty_pte(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep, pte_t pte, cydp_t flags) @@ -822,8 +870,6 @@ static inline bool __ptep_clear_flush_young(struct vm_a= rea_struct *vma, =20 #define __ptep_clear_flush_young __ptep_clear_flush_young =20 -#define ptep_get __ptep_get -#define ptep_get_lockless __ptep_get_lockless #define set_pte __set_pte #define set_ptes __set_ptes =20 diff --git a/arch/riscv/mm/Makefile b/arch/riscv/mm/Makefile index b916a68d324ad..fd260903cd340 100644 --- a/arch/riscv/mm/Makefile +++ b/arch/riscv/mm/Makefile @@ -19,6 +19,7 @@ obj-y +=3D context.o obj-y +=3D pmem.o =20 obj-$(CONFIG_HUGETLB_PAGE) +=3D hugetlbpage.o +obj-$(CONFIG_RISCV_ISA_SVNAPOT) +=3D contpte.o obj-$(CONFIG_PTDUMP) +=3D ptdump.o obj-$(CONFIG_KASAN) +=3D kasan_init.o =20 diff --git a/arch/riscv/mm/contpte.c b/arch/riscv/mm/contpte.c new file mode 100644 index 0000000000000..43ea76e424492 --- /dev/null +++ b/arch/riscv/mm/contpte.c @@ -0,0 +1,273 @@ +// SPDX-License-Identifier: GPL-2.0-only + +#include +#include +#include +#include +#include +#include +#include +#include + +#include + +static inline bool napot_hw_supported(void) +{ + return riscv_has_extension_unlikely(RISCV_ISA_EXT_SVNAPOT); +} + +static inline bool mm_is_user(struct mm_struct *mm) +{ + if (unlikely(mm_is_efi(mm))) + return false; + + return mm !=3D &init_mm; +} + +static inline unsigned int napotpte_order(void) +{ + return NAPOT_CONT64KB_ORDER; +} + +static inline unsigned long napotpte_size(void) +{ + return napot_cont_size(napotpte_order()); +} + +static inline unsigned int napotpte_pte_num(void) +{ + return napot_pte_num(napotpte_order()); +} + +static inline unsigned long napot_align_addr(unsigned long addr) +{ + return ALIGN_DOWN(addr, napotpte_size()); +} + +static inline pte_t *napot_align_ptep(pte_t *ptep) +{ + return PTR_ALIGN_DOWN(ptep, napotpte_pte_num() * sizeof(*ptep)); +} + +static inline pte_t pte_mask_ad(pte_t pte) +{ + return pte_mkold(pte_mkclean(pte)); +} + +void __napotpte_try_unfold(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, pte_t pte); + +static inline unsigned long pte_protval_no_pfn_no_napot(pte_t pte) +{ + return (pte_val(pte) & ~_PAGE_PFN_MASK) & ~_PAGE_NAPOT; +} + +static inline pte_t napotpte_subpte(pte_t *ptep, pte_t pte) +{ + unsigned long pfn; + pgprot_t prot; + + if (!pte_present_napot(pte)) + return pte; + + pfn =3D pte_pfn(pte) + (ptep - napot_align_ptep(ptep)); + prot =3D __pgprot(pte_protval_no_pfn_no_napot(pte)); + + return pfn_pte(pfn, prot); +} + +static inline pte_t __napot_ptep_get_and_clear(struct mm_struct *mm, + unsigned long addr, pte_t *ptep) +{ + pte_t pte; + + pte =3D __pte(atomic_long_xchg((atomic_long_t *)ptep, 0)); + page_table_check_pte_clear(mm, addr, pte_mknonnapot(pte, addr)); + + return pte; +} + +static void napotpte_convert(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, pte_t target) +{ + unsigned long start_addr, end, ptent_addr; + pte_t *start_ptep; + pte_t ptent, pte; + unsigned int i, nr; + + start_addr =3D napot_align_addr(addr); + start_ptep =3D napot_align_ptep(ptep); + nr =3D napotpte_pte_num(); + end =3D start_addr + napotpte_size(); + + for (i =3D 0; i < nr; i++) { + ptent_addr =3D start_addr + i * PAGE_SIZE; + ptent =3D __napot_ptep_get_and_clear(mm, ptent_addr, + start_ptep + i); + if (pte_dirty(ptent)) + target =3D riscv_pte_mkhwdirty(target); + if (pte_young(ptent)) + target =3D pte_mkyoung(target); + } + + flush_tlb_mm_range(mm, start_addr, end, PAGE_SIZE); + + page_table_check_ptes_set(mm, start_addr, start_ptep, target, nr); + if (pte_napot(target)) { + for (i =3D 0; i < nr; i++) + __set_pte_at(mm, start_ptep + i, target); + return; + } + + for (i =3D 0; i < nr; i++) { + pte =3D pfn_pte(pte_pfn(target) + i, + __pgprot(pte_protval_no_pfn_no_napot(target))); + if (pte_dirty(target)) + pte =3D riscv_pte_mkhwdirty(pte); + if (pte_young(target)) + pte =3D pte_mkyoung(pte); + __set_pte_at(mm, start_ptep + i, pte); + } +} + +static inline bool napotpte_is_consistent(pte_t pte, pte_t orig_pte) +{ + return pte_present_napot(pte) && + pte_val(pte_mask_ad(pte)) =3D=3D pte_val(pte_mask_ad(orig_pte)); +} + +void __napotpte_try_fold(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, pte_t pte) +{ + struct page *page; + struct folio *folio; + unsigned long folio_start, folio_end; + unsigned long cont_start, cont_end; + unsigned long pfn; + pgprot_t prot; + pte_t expected, cur; + pte_t *start; + unsigned int i, nr; + + if (!napot_hw_supported() || !mm_is_user(mm)) + return; + + if (!pte_present(pte) || pte_napot(pte) || pte_special(pte)) + return; + + /* + * Driver __GFP_COMP pages inserted by vm_insert_page() have valid + * compound metadata. Fold them only after verifying the whole supported + * Svnapot range is identical and contiguous. + */ + page =3D pte_page(pte); + folio =3D page_folio(page); + folio_start =3D addr - (page - &folio->page) * PAGE_SIZE; + folio_end =3D folio_start + folio_nr_pages(folio) * PAGE_SIZE; + cont_start =3D napot_align_addr(addr); + cont_end =3D cont_start + napotpte_size(); + if (folio_start > cont_start || folio_end < cont_end) + return; + + nr =3D napotpte_pte_num(); + start =3D napot_align_ptep(ptep); + + pfn =3D ALIGN_DOWN(pte_pfn(pte), nr); + prot =3D pte_pgprot(pte_mask_ad(pte)); + expected =3D pfn_pte(pfn, prot); + + /* + * The caller must hold the PTL across validation and conversion. + * Software PTE updates are serialized by the PTL; hardware A/D updates + * may race and are re-read and folded into target by napotpte_convert(). + */ + for (i =3D 0; i < nr; i++) { + cur =3D READ_ONCE(start[i]); + if (pte_val(pte_mask_ad(cur)) !=3D pte_val(expected)) + return; + pte_val(expected) +=3D 1UL << _PAGE_PFN_SHIFT; + } + + expected =3D pte_mknapot(pfn_pte(pfn, prot), napotpte_order()); + napotpte_convert(mm, addr, ptep, expected); +} +EXPORT_SYMBOL(__napotpte_try_fold); + +void __napotpte_try_unfold(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, pte_t pte) +{ + pte_t target; + pgprot_t prot; + + if (!napot_hw_supported() || !mm_is_user(mm) || + !pte_present_napot(pte)) + return; + + prot =3D __pgprot(pte_protval_no_pfn_no_napot(pte)); + target =3D pfn_pte(pte_pfn(pte), prot); + + napotpte_convert(mm, addr, ptep, target); +} +EXPORT_SYMBOL(__napotpte_try_unfold); + +pte_t napotpte_ptep_get(pte_t *ptep, pte_t orig_pte) +{ + pte_t pte, cur; + pte_t *start; + unsigned int i, nr; + + if (!napot_hw_supported() || !pte_present_napot(orig_pte)) + return orig_pte; + + pte =3D orig_pte; + start =3D napot_align_ptep(ptep); + nr =3D napotpte_pte_num(); + + for (i =3D 0; i < nr; i++) { + cur =3D READ_ONCE(start[i]); + if (!napotpte_is_consistent(cur, orig_pte)) + return napotpte_subpte(ptep, orig_pte); + if (pte_dirty(cur)) + pte =3D riscv_pte_mkhwdirty(pte); + if (pte_young(cur)) + pte =3D pte_mkyoung(pte); + } + + return napotpte_subpte(ptep, pte); +} +EXPORT_SYMBOL(napotpte_ptep_get); + +pte_t napotpte_ptep_get_lockless(pte_t *orig_ptep) +{ + pte_t orig_pte, pte; + pte_t *ptep; + unsigned int i, nr; + + if (!napot_hw_supported()) + return READ_ONCE(*orig_ptep); + + nr =3D napotpte_pte_num(); + +retry: + orig_pte =3D READ_ONCE(*orig_ptep); + if (!pte_present_napot(orig_pte)) + return orig_pte; + + ptep =3D napot_align_ptep(orig_ptep); + + for (i =3D 0; i < nr; i++, ptep++) { + pte =3D READ_ONCE(*ptep); + + if (!napotpte_is_consistent(pte, orig_pte)) + goto retry; + + if (pte_dirty(pte)) + orig_pte =3D riscv_pte_mkhwdirty(orig_pte); + + if (pte_young(pte)) + orig_pte =3D pte_mkyoung(orig_pte); + } + + return napotpte_subpte(orig_ptep, orig_pte); +} +EXPORT_SYMBOL(napotpte_ptep_get_lockless); --=20 2.39.5 From nobody Sat Jul 25 15:51:08 2026 Received: from mail-pg1-f173.google.com (mail-pg1-f173.google.com [209.85.215.173]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 767D441F5D6 for ; Thu, 16 Jul 2026 12:43:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.173 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784205801; cv=none; b=ZUpvXi8Fv1y63DX0Qhap8Iyvd7+XeHt9mxhWfpvDzszB3Ahda+EPx5BTr2grGidVeNb9f2EZlNoZ3sUZ33J/SikRTZZc5F+eHmkpB6p2Uzo9f/c5prXeXU1v3/Z7Y6x2pYo8Sw9d3SsW5r9AeqIAwRhYqixcqJsh7+wOsF1YBAI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784205801; c=relaxed/simple; bh=kCa2xNVMj+V75V+49VB/MuMM2EguhYMLPZCSxZBA3PY=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=Kf5SCQuOO4mpcRGuAscZr/jKeZII6wI+M1bvD46tDQtKB+rG/n6THuhY76S1fO4BnuLVnM6o09pgBGTNRQVOnqqm9njJhq5Gla2XMzRL0N/WbeJev8GFmbRUwwms3TLzcD485D51usfcuErPMKyW3QYcr+B+kOpRSoVgSq8AeAA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=lql1FO7q; arc=none smtp.client-ip=209.85.215.173 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="lql1FO7q" Received: by mail-pg1-f173.google.com with SMTP id 41be03b00d2f7-ca97d139d5fso4724070a12.0 for ; Thu, 16 Jul 2026 05:43:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1784205799; x=1784810599; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=fNlHrO3Snp57unD4Dpln5m5m4ypo+fcvDFEAF+5w4Yg=; b=lql1FO7q4dAp67bq1/QIwfUShcZSQ0cUjD/mv/6RSw57AzzgpCBlQKRD3iNXN3QlUS PwfQmOe7aioQ7LveoIoVQ8+b9y3NToJ+03Qd44nTAfTukElpNerxNSpJhfpTtXbC3OSO eM8cH+CtFziL8s8LF7q4H8WpsskRCOfZ3y+kYZVDTjcF9i2McaHqJbDlGk4ls1+J5Xoj eqZc1sJO8XmAY+88IXkKXc+TnA/UgjGhvIWobGB5KFTVDUP3CoXnbLmSwp4RYolZWiNV nakVZCaeiqaMYbOGc4Q21lSlSDtWJTFn6efwHKCF+OM0SBM1+jpvDiVDFu5SV/d2MW93 wgYA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1784205799; x=1784810599; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=fNlHrO3Snp57unD4Dpln5m5m4ypo+fcvDFEAF+5w4Yg=; b=lfu4UPeDwS5GQXAaBmGEypM60DCJgVtVlm1R51ortnG1/GjwpV8sqfPkV9m6v9N7DP SpZVwG/jgfyDZyfgUXn91GTRsIqHFPk/OupTfX0bFjQepZuF5Xfw6ZrrnYP3CsbmHEfC 8DslwbSDun21C5kOOtYixeag9+Eu9S2Jv5Fq4w9H3uAIHcJC2NSFR9Z5CUjhfPn9Ov7t k7AG/xIN/sP+S2Xz5RmkkjTXpLh8q+8cHZZxL0Mqz7Z0DMoMzPlI6MpqEv9KTWZLqXG2 qxJJs6vByqRHzrj6WSLIMHpOjS/Ba7LTI8VtcuyzWrGPpgSII1yOl6H87cghPsWe8YLv TJgw== X-Forwarded-Encrypted: i=1; AHgh+RoaCtYF23EM9vsCCGix2YYpw+brW6JfQwnxWDoTaXN8Qw/qop2RtTMHlE2haYvZPAvE767VgWCWhiXpyfQ=@vger.kernel.org X-Gm-Message-State: AOJu0YwWSYMal8leUvOuIHoJED1OVFGrdLY6Rrzjx8fxFG4jO6VthHBo KMzm6JOPyJfpFglmuVZlymLZlbsdraT+H5glCE5IDwFHBDn/A8aeMzyrohMGB6sNgJw= X-Gm-Gg: AfdE7cnSUtF+CaaA2AYCx2hmrPPuYKxL0L7hsb5knvgSQynNJBx25QQ6of8XyU/hNJ+ oqYbaaJnZU0AZIsThD2Lfr6lOHf/2ip4Cs9IdhemlBFq3M19J7C8QaJHBlPl36AbA31HVNIz19W TOhtQIoUyRjgb4BzIgWJzxFKePMGhZAksMcq1yFCwRef5vcfPSEqggFEqcxxipdOiH8NuNWSuYD gAksi16JLyJ2WzrakrFLyTbbYSCbVJ9xxQRk64ier5Z6L8z9bYcWIbLNbMPyRHxELm6pUnZwOgO Pb8+063ilVNQ01QAslFJNMlUzUrNR76rg54u8CEZKh19U2Z1EY+67loW2C0JOy2FdScmgpYDjYw PW7cApFAdHoaAcnviI7ICIm626qnSoP/xfYiaJPjy1SISlaPEESg0lbYFTTm4UDAFPW9NgsheqV Wk5NDXKhEvbf8CgEEP9sr6v9c= X-Received: by 2002:a05:6a21:730e:b0:3c1:279c:526c with SMTP id adf61e73a8af0-3c3576426d3mr10915500637.63.1784205798567; Thu, 16 Jul 2026 05:43:18 -0700 (PDT) Received: from L6YN4KR4K9.bytedance.net ([2408:8606:1800:501::2:f]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-cb258eeaba2sm1730793a12.12.2026.07.16.05.43.10 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 16 Jul 2026 05:43:17 -0700 (PDT) From: Yunhui Cui To: linux-riscv@lists.infradead.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: linux-arch@vger.kernel.org, kvm-riscv@lists.infradead.org, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-efi@vger.kernel.org, linux-perf-users@vger.kernel.org, kasan-dev@googlegroups.com, Yunhui Cui Subject: [PATCH v2 04/15] riscv/mm: implement Svnapot contpte update helpers Date: Thu, 16 Jul 2026 20:41:39 +0800 Message-Id: <08e6e04deb4c415f6819d677858e3eca1270b0f1.1784201104.git.cuiyunhui@bytedance.com> X-Mailer: git-send-email 2.39.2 (Apple Git-143) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Svnapot support needs the public RISC-V PTE update helpers to preserve the same logical per-page view that generic MM users see on the read side. Without a dedicated update path, operations such as set_ptes(), ptep_set_access_flags(), and related clear or write-protect flows would end up touching the raw NAPOT encoding directly, which leaks arch-private representation details into generic MM code and makes folded mappings harder to update safely. Add Svnapot-aware contpte update helpers and route the public update-side PTE operations through them, while keeping the raw helpers available for RISC-V code that still needs direct access to the encoded entries. This also wires the update path into the related mechanics needed when a Svnapot block is unfolded and rewritten, so page-table-check, access-flag updates, and non-Svnapot builds continue to observe the expected public PTE semantics. Signed-off-by: Yunhui Cui --- arch/riscv/include/asm/pgtable.h | 281 ++++++++++++++++++++++---- arch/riscv/mm/contpte.c | 329 ++++++++++++++++++++++++++++++- arch/riscv/mm/pgtable.c | 18 ++ 3 files changed, 588 insertions(+), 40 deletions(-) diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgta= ble.h index fce6d04ea65db..771faabb2b00b 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -413,6 +413,30 @@ static inline int pte_special(pte_t pte) return pte_val(pte) & _PAGE_SPECIAL; } =20 +#ifdef CONFIG_RISCV_ISA_SVNAPOT +static inline pte_t pte_mknonnapot(pte_t pte, unsigned long addr) +{ + unsigned long pfn; + unsigned long offset; + pgprot_t prot; + + if (!pte_present_napot(pte)) + return pte; + + offset =3D (addr & (napot_cont_size(napot_cont_order(pte)) - 1)) >> + PAGE_SHIFT; + pfn =3D pte_pfn(pte) + offset; + prot =3D __pgprot((pte_val(pte) & ~_PAGE_PFN_MASK) & ~_PAGE_NAPOT); + + return pfn_pte(pfn, prot); +} +#else +static inline pte_t pte_mknonnapot(pte_t pte, unsigned long addr) +{ + return pte; +} +#endif + /* static inline pte_t pte_rdprotect(pte_t pte) */ =20 static inline pte_t pte_wrprotect(pte_t pte) @@ -555,6 +579,25 @@ void __napotpte_try_unfold(struct mm_struct *mm, unsig= ned long addr, pte_t *ptep, pte_t pte); pte_t napotpte_ptep_get(pte_t *ptep, pte_t orig_pte); pte_t napotpte_ptep_get_lockless(pte_t *ptep); +void napotpte_set_ptes(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, pte_t pte, unsigned int nr); +void napotpte_clear_full_ptes(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, unsigned int nr, int full); +pte_t napotpte_get_and_clear_full_ptes(struct mm_struct *mm, + unsigned long addr, pte_t *ptep, + unsigned int nr, int full); +void napotpte_clear_young_dirty_ptes(struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep, + unsigned int nr, cydp_t flags); +void napotpte_wrprotect_ptes(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, unsigned int nr); +int napotpte_ptep_set_access_flags(struct vm_area_struct *vma, + unsigned long address, pte_t *ptep, + pte_t entry, int dirty); +int napotpte_ptep_test_and_clear_young(struct vm_area_struct *vma, + unsigned long address, pte_t *ptep); +int napotpte_ptep_clear_flush_young(struct vm_area_struct *vma, + unsigned long address, pte_t *ptep); #endif =20 #ifdef CONFIG_ARCH_HAS_PTE_PROTNONE @@ -706,37 +749,6 @@ static inline pte_t __ptep_get_lockless(pte_t *ptep) return __ptep_get(ptep); } =20 -#ifdef CONFIG_RISCV_ISA_SVNAPOT - -#define ptep_get ptep_get -static inline pte_t ptep_get(pte_t *ptep) -{ - pte_t pte =3D __ptep_get(ptep); - - if (likely(!pte_present_napot(pte))) - return pte; - - return napotpte_ptep_get(ptep, pte); -} - -#define ptep_get_lockless ptep_get_lockless -static inline pte_t ptep_get_lockless(pte_t *ptep) -{ - pte_t pte =3D __ptep_get_lockless(ptep); - - if (likely(!pte_present_napot(pte))) - return pte; - - return napotpte_ptep_get_lockless(ptep); -} - -#else - -#define ptep_get __ptep_get -#define ptep_get_lockless __ptep_get_lockless - -#endif - static inline void __clear_young_dirty_pte(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep, pte_t pte, cydp_t flags) @@ -829,7 +841,6 @@ static inline pte_t __ptep_clear_flush(struct vm_area_s= truct *vma, =20 #define __ptep_clear_flush __ptep_clear_flush =20 -#define __HAVE_ARCH_PTEP_SET_WRPROTECT static inline void __ptep_set_wrprotect(struct mm_struct *mm, unsigned long address, pte_t *ptep) { @@ -870,7 +881,169 @@ static inline bool __ptep_clear_flush_young(struct vm= _area_struct *vma, =20 #define __ptep_clear_flush_young __ptep_clear_flush_young =20 -#define set_pte __set_pte +#ifdef CONFIG_RISCV_ISA_SVNAPOT + +static __always_inline void napotpte_try_fold(struct mm_struct *mm, + unsigned long addr, pte_t *ptep, + pte_t pte) +{ + const unsigned long contmask =3D napot_pte_num(NAPOT_CONT64KB_ORDER) - 1; + bool valign =3D ((addr >> PAGE_SHIFT) & contmask) =3D=3D contmask; + + if (unlikely(valign)) { + bool palign =3D (pte_pfn(pte) & contmask) =3D=3D contmask; + + if (unlikely(palign && pte_present(pte) && !pte_napot(pte) && + !pte_special(pte))) + __napotpte_try_fold(mm, addr, ptep, pte); + } +} + +static __always_inline void napotpte_try_unfold(struct mm_struct *mm, + unsigned long addr, pte_t *ptep, + pte_t pte) +{ + if (unlikely(pte_present_napot(pte))) + __napotpte_try_unfold(mm, addr, ptep, pte); +} + +/* + * Public PTE helpers may transparently handle Svnapot-encoded mappings. + * NAPOT-aware arch users should stick to the private/raw __pte* helpers. + */ +#define ptep_get ptep_get +static inline pte_t ptep_get(pte_t *ptep) +{ + pte_t pte =3D __ptep_get(ptep); + + if (likely(!pte_present_napot(pte))) + return pte; + + return napotpte_ptep_get(ptep, pte); +} + +#define ptep_get_lockless ptep_get_lockless +static inline pte_t ptep_get_lockless(pte_t *ptep) +{ + pte_t pte =3D __ptep_get_lockless(ptep); + + if (likely(!pte_present_napot(pte))) + return pte; + + return napotpte_ptep_get_lockless(ptep); +} + +#define set_ptes set_ptes +static inline void set_ptes(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, pte_t pteval, unsigned int nr) +{ + pteval =3D pte_mknonnapot(pteval, addr); + + if (likely(nr =3D=3D 1)) { + napotpte_try_unfold(mm, addr, ptep, __ptep_get(ptep)); + __set_ptes(mm, addr, ptep, pteval, 1); + napotpte_try_fold(mm, addr, ptep, pteval); + return; + } + + napotpte_set_ptes(mm, addr, ptep, pteval, nr); +} + +static inline void pte_clear(struct mm_struct *mm, + unsigned long addr, pte_t *ptep) +{ + napotpte_try_unfold(mm, addr, ptep, __ptep_get(ptep)); + __pte_clear(mm, addr, ptep); +} + +static inline pte_t ptep_get_and_clear(struct mm_struct *mm, + unsigned long addr, pte_t *ptep) +{ + napotpte_try_unfold(mm, addr, ptep, __ptep_get(ptep)); + return __ptep_get_and_clear(mm, addr, ptep); +} + +#define clear_young_dirty_ptes clear_young_dirty_ptes +static inline void clear_young_dirty_ptes(struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep, + unsigned int nr, cydp_t flags) +{ + if (likely(nr =3D=3D 1 && !pte_present_napot(__ptep_get(ptep)))) + __clear_young_dirty_ptes(vma, addr, ptep, nr, flags); + else + napotpte_clear_young_dirty_ptes(vma, addr, ptep, nr, flags); +} + +#define clear_full_ptes clear_full_ptes +static inline void clear_full_ptes(struct mm_struct *mm, unsigned long add= r, + pte_t *ptep, unsigned int nr, int full) +{ + if (likely(nr =3D=3D 1)) { + napotpte_try_unfold(mm, addr, ptep, __ptep_get(ptep)); + __ptep_get_and_clear(mm, addr, ptep); + return; + } + + napotpte_clear_full_ptes(mm, addr, ptep, nr, full); +} + +#define get_and_clear_full_ptes get_and_clear_full_ptes +static inline pte_t get_and_clear_full_ptes(struct mm_struct *mm, + unsigned long addr, pte_t *ptep, + unsigned int nr, int full) +{ + if (likely(nr =3D=3D 1)) { + napotpte_try_unfold(mm, addr, ptep, __ptep_get(ptep)); + return __ptep_get_and_clear(mm, addr, ptep); + } + + return napotpte_get_and_clear_full_ptes(mm, addr, ptep, nr, full); +} + +#define wrprotect_ptes wrprotect_ptes +static inline void wrprotect_ptes(struct mm_struct *mm, + unsigned long address, pte_t *ptep, + unsigned int nr) +{ + if (likely(nr =3D=3D 1)) { + napotpte_try_unfold(mm, address, ptep, __ptep_get(ptep)); + __ptep_set_wrprotect(mm, address, ptep); + return; + } + + napotpte_wrprotect_ptes(mm, address, ptep, nr); +} + +#define __HAVE_ARCH_PTEP_SET_WRPROTECT +static inline void ptep_set_wrprotect(struct mm_struct *mm, + unsigned long address, pte_t *ptep) +{ + wrprotect_ptes(mm, address, ptep, 1); +} + +#define __HAVE_ARCH_PTEP_CLEAR_YOUNG_FLUSH +static inline int ptep_clear_flush_young(struct vm_area_struct *vma, + unsigned long address, pte_t *ptep) +{ + pte_t orig_pte =3D __ptep_get(ptep); + + if (likely(!pte_present_napot(orig_pte))) + return __ptep_clear_flush_young(vma, address, ptep); + + return napotpte_ptep_clear_flush_young(vma, address, ptep); +} + +#else /* CONFIG_RISCV_ISA_SVNAPOT */ + +#define napotpte_ptep_set_access_flags(vma, address, ptep, entry, dirty) \ + ({ (void)(vma); (void)(address); (void)(ptep); (void)(entry); \ + (void)(dirty); 0; }) + +#define napotpte_ptep_test_and_clear_young(vma, address, ptep) \ + ({ (void)(vma); (void)(address); (void)(ptep); 0; }) + +#define ptep_get __ptep_get +#define ptep_get_lockless __ptep_get_lockless #define set_ptes __set_ptes =20 static inline void pte_clear(struct mm_struct *mm, @@ -881,9 +1054,51 @@ static inline void pte_clear(struct mm_struct *mm, =20 #define ptep_get_and_clear __ptep_get_and_clear #define clear_young_dirty_ptes __clear_young_dirty_ptes + +static inline void __clear_full_ptes(struct mm_struct *mm, + unsigned long addr, pte_t *ptep, + unsigned int nr, int full) +{ + for (;;) { + __ptep_get_and_clear(mm, addr, ptep); + if (--nr =3D=3D 0) + break; + ptep++; + addr +=3D PAGE_SIZE; + } +} + +static inline pte_t __get_and_clear_full_ptes(struct mm_struct *mm, + unsigned long addr, pte_t *ptep, + unsigned int nr, int full) +{ + pte_t pte, tmp_pte; + + pte =3D __ptep_get_and_clear(mm, addr, ptep); + while (--nr) { + ptep++; + addr +=3D PAGE_SIZE; + tmp_pte =3D __ptep_get_and_clear(mm, addr, ptep); + if (pte_dirty(tmp_pte)) + pte =3D riscv_pte_mkhwdirty(pte); + if (pte_young(tmp_pte)) + pte =3D pte_mkyoung(pte); + } + + return pte; +} + +#define clear_full_ptes __clear_full_ptes +#define get_and_clear_full_ptes __get_and_clear_full_ptes +#define __HAVE_ARCH_PTEP_SET_WRPROTECT #define ptep_set_wrprotect __ptep_set_wrprotect +#define __HAVE_ARCH_PTEP_CLEAR_YOUNG_FLUSH #define ptep_clear_flush_young __ptep_clear_flush_young =20 +#endif /* CONFIG_RISCV_ISA_SVNAPOT */ + +#define set_pte __set_pte + #define pgprot_nx pgprot_nx static inline pgprot_t pgprot_nx(pgprot_t _prot) { diff --git a/arch/riscv/mm/contpte.c b/arch/riscv/mm/contpte.c index 43ea76e424492..3f8a0a2970afb 100644 --- a/arch/riscv/mm/contpte.c +++ b/arch/riscv/mm/contpte.c @@ -62,6 +62,24 @@ static inline unsigned long pte_protval_no_pfn_no_napot(= pte_t pte) return (pte_val(pte) & ~_PAGE_PFN_MASK) & ~_PAGE_NAPOT; } =20 +static inline void napotpte_clear_young_dirty_pte(pte_t *ptep, cydp_t flag= s) +{ + pte_t old_pte, new_pte; + unsigned long old_val, new_val; + + do { + old_pte =3D READ_ONCE(*ptep); + new_pte =3D old_pte; + if (flags & CYDP_CLEAR_YOUNG) + new_pte =3D pte_mkold(new_pte); + if (flags & CYDP_CLEAR_DIRTY) + new_pte =3D pte_mkclean(new_pte); + + old_val =3D pte_val(old_pte); + new_val =3D pte_val(new_pte); + } while (cmpxchg_relaxed(&pte_val(*ptep), old_val, new_val) !=3D old_val); +} + static inline pte_t napotpte_subpte(pte_t *ptep, pte_t pte) { unsigned long pfn; @@ -76,13 +94,34 @@ static inline pte_t napotpte_subpte(pte_t *ptep, pte_t = pte) return pfn_pte(pfn, prot); } =20 -static inline pte_t __napot_ptep_get_and_clear(struct mm_struct *mm, - unsigned long addr, pte_t *ptep) +static void __clear_full_ptes(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, unsigned int nr, int full) { - pte_t pte; + for (;;) { + __ptep_get_and_clear(mm, addr, ptep); + if (--nr =3D=3D 0) + break; + ptep++; + addr +=3D PAGE_SIZE; + } +} =20 - pte =3D __pte(atomic_long_xchg((atomic_long_t *)ptep, 0)); - page_table_check_pte_clear(mm, addr, pte_mknonnapot(pte, addr)); +static pte_t __get_and_clear_full_ptes(struct mm_struct *mm, + unsigned long addr, pte_t *ptep, + unsigned int nr, int full) +{ + pte_t pte, tmp_pte; + + pte =3D __ptep_get_and_clear(mm, addr, ptep); + while (--nr) { + ptep++; + addr +=3D PAGE_SIZE; + tmp_pte =3D __ptep_get_and_clear(mm, addr, ptep); + if (pte_dirty(tmp_pte)) + pte =3D riscv_pte_mkhwdirty(pte); + if (pte_young(tmp_pte)) + pte =3D pte_mkyoung(pte); + } =20 return pte; } @@ -102,8 +141,9 @@ static void napotpte_convert(struct mm_struct *mm, unsi= gned long addr, =20 for (i =3D 0; i < nr; i++) { ptent_addr =3D start_addr + i * PAGE_SIZE; - ptent =3D __napot_ptep_get_and_clear(mm, ptent_addr, - start_ptep + i); + ptent =3D __ptep_get_and_clear_noptc(start_ptep + i); + page_table_check_pte_clear(mm, ptent_addr, + pte_mknonnapot(ptent, ptent_addr)); if (pte_dirty(ptent)) target =3D riscv_pte_mkhwdirty(target); if (pte_young(ptent)) @@ -136,6 +176,22 @@ static inline bool napotpte_is_consistent(pte_t pte, p= te_t orig_pte) pte_val(pte_mask_ad(pte)) =3D=3D pte_val(pte_mask_ad(orig_pte)); } =20 +static bool napotpte_all_subptes_same(pte_t *ptep, pte_t expected_pte) +{ + pte_t *start; + unsigned int i, nr; + + start =3D napot_align_ptep(ptep); + nr =3D napotpte_pte_num(); + + for (i =3D 0; i < nr; i++) { + if (!pte_same(READ_ONCE(start[i]), expected_pte)) + return false; + } + + return true; +} + void __napotpte_try_fold(struct mm_struct *mm, unsigned long addr, pte_t *ptep, pte_t pte) { @@ -271,3 +327,262 @@ pte_t napotpte_ptep_get_lockless(pte_t *orig_ptep) return napotpte_subpte(orig_ptep, orig_pte); } EXPORT_SYMBOL(napotpte_ptep_get_lockless); + +static void napotpte_try_unfold_range(struct mm_struct *mm, + unsigned long addr, pte_t *ptep, + unsigned int nr) +{ + unsigned long next; + pte_t pte; + unsigned int chunk; + + while (nr) { + pte =3D READ_ONCE(*ptep); + if (pte_present_napot(pte)) { + __napotpte_try_unfold(mm, addr, ptep, pte); + next =3D napot_align_addr(addr) + napotpte_size(); + chunk =3D (next - addr) >> PAGE_SHIFT; + } else { + chunk =3D 1; + } + + if (chunk > nr) + chunk =3D nr; + + ptep +=3D chunk; + addr +=3D chunk * PAGE_SIZE; + nr -=3D chunk; + } +} + +static void napotpte_try_unfold_partial(struct mm_struct *mm, + unsigned long addr, pte_t *ptep, + unsigned int nr) +{ + pte_t pte; + + if (ptep !=3D napot_align_ptep(ptep) || nr < napotpte_pte_num()) { + pte =3D READ_ONCE(*ptep); + if (pte_present_napot(pte)) + __napotpte_try_unfold(mm, addr, ptep, pte); + } + + if (ptep + nr !=3D napot_align_ptep(ptep + nr)) { + unsigned long last_addr; + pte_t *last_ptep; + + last_addr =3D addr + PAGE_SIZE * (nr - 1); + last_ptep =3D ptep + nr - 1; + pte =3D READ_ONCE(*last_ptep); + if (pte_present_napot(pte)) + __napotpte_try_unfold(mm, last_addr, last_ptep, pte); + } +} + +void napotpte_set_ptes(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, pte_t pte, unsigned int nr) +{ + unsigned long next, end; + unsigned long pfn, size, boundary; + pgprot_t prot; + unsigned int chunk, i; + pte_t cur; + + if (!napot_hw_supported() || !mm_is_user(mm)) { + __set_ptes(mm, addr, ptep, pte, nr); + return; + } + + size =3D napotpte_size(); + end =3D addr + ((unsigned long)nr << PAGE_SHIFT); + pfn =3D pte_pfn(pte); + prot =3D __pgprot(pte_protval_no_pfn_no_napot(pte)); + + do { + boundary =3D (addr + size) & ~(size - 1); + next =3D (boundary - 1 < end - 1) ? boundary : end; + chunk =3D (next - addr) >> PAGE_SHIFT; + + cur =3D pfn_pte(pfn, prot); + if (((addr | next | (pfn << PAGE_SHIFT)) & (size - 1)) =3D=3D 0) { + cur =3D pte_mknapot(cur, napotpte_order()); + page_table_check_ptes_set(mm, addr, ptep, cur, chunk); + for (i =3D 0; i < chunk; i++) + __set_pte_at(mm, ptep + i, cur); + } else { + __set_ptes(mm, addr, ptep, cur, chunk); + } + + addr =3D next; + ptep +=3D chunk; + pfn +=3D chunk; + } while (addr !=3D end); +} +EXPORT_SYMBOL(napotpte_set_ptes); + +void napotpte_clear_full_ptes(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, unsigned int nr, int full) +{ + if (!napot_hw_supported() || !mm_is_user(mm)) { + __clear_full_ptes(mm, addr, ptep, nr, full); + return; + } + + /* + * Svnapot stores identical napot-encoded entries across the whole block + * rather than per-page PFNs, so batch zap paths must unfold the covered + * range before the generic MM consumes ordinary per-page PTEs. + */ + napotpte_try_unfold_range(mm, addr, ptep, nr); + __clear_full_ptes(mm, addr, ptep, nr, full); +} +EXPORT_SYMBOL(napotpte_clear_full_ptes); + +pte_t napotpte_get_and_clear_full_ptes(struct mm_struct *mm, + unsigned long addr, pte_t *ptep, + unsigned int nr, int full) +{ + if (!napot_hw_supported() || !mm_is_user(mm)) + return __get_and_clear_full_ptes(mm, addr, ptep, nr, full); + + napotpte_try_unfold_range(mm, addr, ptep, nr); + + return __get_and_clear_full_ptes(mm, addr, ptep, nr, full); +} +EXPORT_SYMBOL(napotpte_get_and_clear_full_ptes); + +void napotpte_clear_young_dirty_ptes(struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep, + unsigned int nr, cydp_t flags) +{ + struct mm_struct *mm; + unsigned long start, end; + unsigned int total; + + mm =3D vma->vm_mm; + if (!napot_hw_supported() || !mm_is_user(mm)) { + __clear_young_dirty_ptes(vma, addr, ptep, nr, flags); + return; + } + + start =3D addr; + end =3D start + nr * PAGE_SIZE; + + if (pte_present_napot(READ_ONCE(*(ptep + nr - 1)))) + end =3D ALIGN(end, napotpte_size()); + + if (pte_present_napot(READ_ONCE(*ptep))) { + start =3D napot_align_addr(start); + ptep =3D napot_align_ptep(ptep); + } + + total =3D (end - start) >> PAGE_SHIFT; + for (; total; total--, ptep++, start +=3D PAGE_SIZE) + napotpte_clear_young_dirty_pte(ptep, flags); +} +EXPORT_SYMBOL(napotpte_clear_young_dirty_ptes); + +void napotpte_wrprotect_ptes(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, unsigned int nr) +{ + unsigned int i; + + if (!napot_hw_supported() || !mm_is_user(mm)) { + for (i =3D 0; i < nr; i++, ptep++, addr +=3D PAGE_SIZE) + __ptep_set_wrprotect(mm, addr, ptep); + return; + } + + napotpte_try_unfold_partial(mm, addr, ptep, nr); + + for (i =3D 0; i < nr; i++, ptep++, addr +=3D PAGE_SIZE) + __ptep_set_wrprotect(mm, addr, ptep); +} +EXPORT_SYMBOL(napotpte_wrprotect_ptes); + +int napotpte_ptep_set_access_flags(struct vm_area_struct *vma, + unsigned long address, pte_t *ptep, + pte_t entry, int dirty) +{ + pte_t raw_pte, napot_pte; + pte_t *start; + pgprot_t prot; + unsigned long start_addr; + unsigned int i, nr; + bool changed; + + raw_pte =3D READ_ONCE(*ptep); + if (!napot_hw_supported() || !pte_present_napot(raw_pte)) + return 0; + + prot =3D pte_pgprot(entry); + napot_pte =3D pfn_pte(pte_pfn(raw_pte), prot); + napot_pte =3D pte_mknapot(napot_pte, napotpte_order()); + + if (napotpte_all_subptes_same(ptep, napot_pte)) + return !riscv_has_extension_unlikely(RISCV_ISA_EXT_SVVPTC); + + if (pte_write(raw_pte) !=3D pte_write(napot_pte)) { + __napotpte_try_unfold(vma->vm_mm, address, ptep, raw_pte); + entry =3D pte_mknonnapot(entry, address); + + return __ptep_set_access_flags(vma, address, ptep, entry, + dirty); + } + + start =3D napot_align_ptep(ptep); + address =3D napot_align_addr(address); + start_addr =3D address; + nr =3D napotpte_pte_num(); + changed =3D false; + + for (i =3D 0; i < nr; i++, start++, address +=3D PAGE_SIZE) { + if (__ptep_set_access_flags(vma, address, start, napot_pte, 0)) + changed =3D true; + } + + if (changed) + flush_tlb_range(vma, start_addr, start_addr + napotpte_size()); + + return changed; +} +EXPORT_SYMBOL(napotpte_ptep_set_access_flags); + +int napotpte_ptep_test_and_clear_young(struct vm_area_struct *vma, + unsigned long address, pte_t *ptep) +{ + pte_t *start; + unsigned int i, nr; + int young; + + if (!napot_hw_supported() || !pte_present_napot(READ_ONCE(*ptep))) + return 0; + + start =3D napot_align_ptep(ptep); + nr =3D napotpte_pte_num(); + young =3D 0; + + for (i =3D 0; i < nr; i++) + young |=3D test_and_clear_bit(_PAGE_ACCESSED_OFFSET, + &pte_val(start[i])); + + return young; +} +EXPORT_SYMBOL(napotpte_ptep_test_and_clear_young); + +int napotpte_ptep_clear_flush_young(struct vm_area_struct *vma, + unsigned long address, pte_t *ptep) +{ + unsigned long start_addr; + int young; + + young =3D napotpte_ptep_test_and_clear_young(vma, address, ptep); + if (!young) + return 0; + + start_addr =3D napot_align_addr(address); + flush_tlb_range(vma, start_addr, start_addr + napotpte_size()); + + return young; +} +EXPORT_SYMBOL(napotpte_ptep_clear_flush_young); diff --git a/arch/riscv/mm/pgtable.c b/arch/riscv/mm/pgtable.c index 43bd542c6ff02..50e4c4cc10691 100644 --- a/arch/riscv/mm/pgtable.c +++ b/arch/riscv/mm/pgtable.c @@ -1,6 +1,7 @@ // SPDX-License-Identifier: GPL-2.0 =20 #include +#include #include #include #include @@ -32,6 +33,16 @@ int ptep_set_access_flags(struct vm_area_struct *vma, unsigned long address, pte_t *ptep, pte_t entry, int dirty) { + pte_t raw_pte; + + entry =3D pte_mknonnapot(entry, address); + + raw_pte =3D READ_ONCE(*ptep); + if (riscv_has_extension_unlikely(RISCV_ISA_EXT_SVNAPOT) && + pte_present_napot(raw_pte)) + return napotpte_ptep_set_access_flags(vma, address, ptep, entry, + dirty); + return __ptep_set_access_flags(vma, address, ptep, entry, dirty); } =20 @@ -50,6 +61,13 @@ bool ptep_test_and_clear_young(struct vm_area_struct *vm= a, unsigned long address, pte_t *ptep) { + pte_t raw_pte; + + raw_pte =3D READ_ONCE(*ptep); + if (riscv_has_extension_unlikely(RISCV_ISA_EXT_SVNAPOT) && + pte_present_napot(raw_pte)) + return napotpte_ptep_test_and_clear_young(vma, address, ptep); + return __ptep_test_and_clear_young(vma, address, ptep); } EXPORT_SYMBOL_GPL(ptep_test_and_clear_young); --=20 2.39.5 From nobody Sat Jul 25 15:51:08 2026 Received: from mail-pf1-f182.google.com (mail-pf1-f182.google.com [209.85.210.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id ED6C341F5F5 for ; Thu, 16 Jul 2026 12:43:29 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.182 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784205812; cv=none; b=N9estcxtQbGJxD0zRhCXzqOizYVDVhcDm0Adzl8FnlPmGNx/YNTkvMZh4k6/a5y3npHt1w4ltPVJOuwGVuCjGW7QVU/XIVFc2nipAQqTLUInxlLD04U5aRTWKUku0KUahCrUJ2qoDqICmN0XxGkA/PH9X8icDtvZzdDUxfoBUaE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784205812; c=relaxed/simple; bh=j3DRDn0DA4zW7+RaJieI7hhedH7/qlD6z/+kzboZM5c=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=o23/IBS20D1F1rEb1e2QEbpkSbPz0oYkOclcfmg4LtTwpEdYFVIaeNMDkC3M8+pGmGsCtOWmsyOt5HNq6LgU49JEh0/+/erZyU9Ox9y+el0kRrbnK5KFeXAXSpIZEQ3NSMgp98EqAGQH8dw8zbygwkC0WT+6wv5zJSOAulsDh/A= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=ZKCGVPRH; arc=none smtp.client-ip=209.85.210.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="ZKCGVPRH" Received: by mail-pf1-f182.google.com with SMTP id d2e1a72fcca58-84864086bfeso6950309b3a.1 for ; Thu, 16 Jul 2026 05:43:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1784205809; x=1784810609; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=qNKb6kLr/7ftZXllrSyALai0ySExbjYkU3LL0V9fZEA=; b=ZKCGVPRHVU3pwvQP7Kc1KsHl6B1v9zZCKHN2OZwXvHk9oCTG8nnfk24vjNT3Ck5I9X 5BytkSM6SJ4FoWl3nxCQL6uxKRByiEFgCdK2Vur+7PX1TiJDsfQB9HfOtlkm3Pgijpf5 0iL3nSqFw7cFqn74Omm1Dd5N5PDVvOyIkMvoMEii9K6+mE82zmlA8Sf0IqCrglKqSLNA jSHCUx0KrMJycAXeoQShxu71bo/RGjYDPplQGeEz2ndWRRohFyyUpa2roCHdhYcbzk4M VR8+vjt9xsqBakj8laRport5azY733lQIzUZuu1r3zYzN/wPOH/nE8tuKfqL2p/U4IB/ Ob4A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1784205809; x=1784810609; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=qNKb6kLr/7ftZXllrSyALai0ySExbjYkU3LL0V9fZEA=; b=WgUnd5HbrQx0nDOXNWulu6P4NOKdjV79sUtoGwAbiwxUBL09342fXVx1JXE0wgFfwH lbWFWk7MGSxdpZmeO9k0MKA78bhsh5EogDaywAmHu2rHE4xXIzhgqfCDGPWBBv3uW4KV QtwqpIFf6JdxJXIkvFkVJ03ei23ZdPFQleXGzcW6L38qOxI4k0IoV5i4Cq/o4Ju9R7N1 fyRvAVPrnd4r9vIb/OlysTYfBeZ2TfTtzD5zYdl7FgMNMpMOdrgsSTMzeB+ecPOf1eHS QKci9bewSIrrsg9a4lWKbOHZ8fVUUkXw+ooloePgLLvSuV/lTdvjiJsy/WkiWr8fRVd2 hZyg== X-Forwarded-Encrypted: i=1; AHgh+RpDNfyu4Hqr3vxkytHiPrD2Z0RiAHxmtDoF2NKJZW5dMrYypQody/ECgLq+7mi+A53mSu345Q2SCuaARD0=@vger.kernel.org X-Gm-Message-State: AOJu0YyaConTosw5nDJAexL8Rjbk9oONWncUcTrY3IgRhFsRBCbe0TLV fXZ0ld1S4VNgUEKSSwiVclE6+Cw33Ay4sINqZvdXVdJNmHL2kDxJvBpYh5d1GR3CS7C8Jmh1c4T uIEoYw74= X-Gm-Gg: AfdE7clMUe+5j220TldP7GezR6nPoJ04t2xyLtKLtzICeXjVeUwXIZ/M9Z/eMh8/nbG 7VrTNvRIVoc/6J2r5jm9FY0QGozPfAsyOscV3frajH0geUT1v5NRLvtVBcziq97SjQJ6IIWlB7K N5KT+d0t0Kl8/wE2/VlzwBTAwRIkGUJ87v6CznPwokaZvWsqOrCnei6HGMLsHYPuRHh8IQD6UR3 3cuNnTUifbYWdJI1NEnbvpZo5a6rMY/iZbFYtOR+s00Ii2UEH7veu6U81NFIzvS16FxMxzYc09m 3u8gcfPy9wQtP3CKQlECeFogY+5daJC1mGoQqNgvrSGqmI2KTOPADkZf1z51+vMjrAQuMUVQaRp nNinui44mp5kAxQpYbqVvwXLhKQKZbH4EeKM27t/iYD3OHqSkXovt4Vd+wGHPIGNnpImT8Fjy2f EEHGtJS17AJi5iJ2Y0DjOghDY= X-Received: by 2002:a05:6a20:e615:b0:3c0:9c19:6590 with SMTP id adf61e73a8af0-3c3576c37f6mr12231332637.74.1784205808961; Thu, 16 Jul 2026 05:43:28 -0700 (PDT) Received: from L6YN4KR4K9.bytedance.net ([2408:8606:1800:501::2:f]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-cb258eeaba2sm1730793a12.12.2026.07.16.05.43.19 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 16 Jul 2026 05:43:26 -0700 (PDT) From: Yunhui Cui To: linux-riscv@lists.infradead.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: linux-arch@vger.kernel.org, kvm-riscv@lists.infradead.org, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-efi@vger.kernel.org, linux-perf-users@vger.kernel.org, kasan-dev@googlegroups.com, Yunhui Cui Subject: [PATCH v2 05/15] mm: extend pte batch and leaf-size helpers Date: Thu, 16 Jul 2026 20:41:40 +0800 Message-Id: X-Mailer: git-send-email 2.39.2 (Apple Git-143) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Folded mappings such as RISC-V Svnapot need two extra pieces of generic MM support. First, architecture pte_batch_hint() implementations need access to the same batch flags and normalization rules used by folio_pte_batch(), so folded ranges can participate in batching with semantics consistent with generic folio comparisons. Second, leaf-size users may need the original PTE pointer instead of the public ptep_get() view. For folded mappings, the public helper can return a logical per-page sub-PTE, while the effective leaf size still depends on the raw encoded entry in the page table. Move the batch flag helpers into pgtable.h, extend pte_batch_hint() to take the batch flags, add __ptep_leaf_size(), and update the generic callers to use the new interfaces. Signed-off-by: Yunhui Cui --- arch/arm64/include/asm/pgtable.h | 9 +++++- include/linux/pgtable.h | 53 +++++++++++++++++++++++++++++++- kernel/events/core.c | 2 +- mm/internal.h | 39 ++--------------------- mm/mincore.c | 2 +- mm/mremap.c | 2 +- 6 files changed, 65 insertions(+), 42 deletions(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgta= ble.h index a2681d7553584..018781a1fcbee 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -5,6 +5,13 @@ #ifndef __ASM_PGTABLE_H #define __ASM_PGTABLE_H =20 +#ifndef __ASSEMBLY__ +#ifndef __LINUX_FPB_T +#define __LINUX_FPB_T +typedef int __bitwise fpb_t; +#endif +#endif + #include #include =20 @@ -1705,7 +1712,7 @@ static __always_inline void contpte_try_unfold(struct= mm_struct *mm, } =20 #define pte_batch_hint pte_batch_hint -static inline unsigned int pte_batch_hint(pte_t *ptep, pte_t pte) +static inline unsigned int pte_batch_hint(pte_t *ptep, pte_t pte, fpb_t fl= ags) { if (!pte_valid_cont(pte)) return 1; diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index 8c093c119e5a8..da14328093a86 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -3,6 +3,35 @@ #define _LINUX_PGTABLE_H =20 #include + +#ifndef __ASSEMBLY__ +#ifndef __LINUX_FPB_T +#define __LINUX_FPB_T +typedef int __bitwise fpb_t; +#endif + +/* Compare PTEs respecting the dirty bit. */ +#define FPB_RESPECT_DIRTY ((__force fpb_t)BIT(0)) + +/* Compare PTEs respecting the soft-dirty bit. */ +#define FPB_RESPECT_SOFT_DIRTY ((__force fpb_t)BIT(1)) + +/* Compare PTEs respecting the writable bit. */ +#define FPB_RESPECT_WRITE ((__force fpb_t)BIT(2)) + +/* + * Merge PTE write bits: if any PTE in the batch is writable, modify the + * PTE at @ptentp to be writable. + */ +#define FPB_MERGE_WRITE ((__force fpb_t)BIT(3)) + +/* + * Merge PTE young and dirty bits: if any PTE in the batch is young or dir= ty, + * modify the PTE at @ptentp to be young or dirty, respectively. + */ +#define FPB_MERGE_YOUNG_DIRTY ((__force fpb_t)BIT(4)) +#endif /* !__ASSEMBLY__ */ + #include =20 #define PMD_ORDER (PMD_SHIFT - PAGE_SHIFT) @@ -397,6 +426,7 @@ static inline void lazy_mmu_mode_resume(void) {} * pte_batch_hint - Number of pages that can be added to batch without sca= nning. * @ptep: Page table pointer for the entry. * @pte: Page table entry. + * @flags: Flags that control PTE normalization for batch comparisons. * * Some architectures know that a set of contiguous ptes all map the same * contiguous memory with the same permissions. In this case, it can provi= de a @@ -407,7 +437,7 @@ static inline void lazy_mmu_mode_resume(void) {} * * May be overridden by the architecture, else pte_batch_hint is always 1. */ -static inline unsigned int pte_batch_hint(pte_t *ptep, pte_t pte) +static inline unsigned int pte_batch_hint(pte_t *ptep, pte_t pte, fpb_t fl= ags) { return 1; } @@ -1855,6 +1885,7 @@ static inline pmd_t pmd_swp_clear_soft_dirty(pmd_t pm= d) return pmd; } #endif + #else /* !CONFIG_HAVE_ARCH_SOFT_DIRTY */ static inline int pte_soft_dirty(pte_t pte) { @@ -1917,6 +1948,17 @@ static inline pmd_t pmd_swp_clear_soft_dirty(pmd_t p= md) } #endif =20 +static inline pte_t __pte_batch_clear_ignored(pte_t pte, fpb_t flags) +{ + if (!(flags & FPB_RESPECT_DIRTY)) + pte =3D pte_mkclean(pte); + if (likely(!(flags & FPB_RESPECT_SOFT_DIRTY))) + pte =3D pte_clear_soft_dirty(pte); + if (likely(!(flags & FPB_RESPECT_WRITE))) + pte =3D pte_wrprotect(pte); + return pte_mkold(pte); +} + #ifndef __HAVE_PFNMAP_TRACKING /* * Interfaces that can be used by architecture code to keep track of @@ -2414,6 +2456,15 @@ static inline const char *pgtable_level_to_str(enum = pgtable_level level) #define __pte_leaf_size(x,y) pte_leaf_size(y) #endif =20 +#ifndef __ptep_leaf_size +#ifndef __ASSEMBLY__ +static inline unsigned long __ptep_leaf_size(pmd_t pmd, pte_t *ptep, pte_t= pte) +{ + return __pte_leaf_size(pmd, pte); +} +#endif /* !__ASSEMBLY__ */ +#endif + /* * We always define pmd_pfn for all archs as it's used in lots of generic * code. Now it happens too for pud_pfn (and can happen for larger diff --git a/kernel/events/core.c b/kernel/events/core.c index 51c1200ea3fdd..37b860abe9e61 100644 --- a/kernel/events/core.c +++ b/kernel/events/core.c @@ -8551,7 +8551,7 @@ static u64 perf_get_pgtable_size(struct mm_struct *mm= , unsigned long addr) =20 pte =3D ptep_get_lockless(ptep); if (pte_present(pte)) - size =3D __pte_leaf_size(pmd, pte); + size =3D __ptep_leaf_size(pmd, ptep, pte); pte_unmap(ptep); #endif /* CONFIG_HAVE_GUP_FAST */ =20 diff --git a/mm/internal.h b/mm/internal.h index 874be94cf2570..aff00a3fdff50 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -343,41 +343,6 @@ static inline int anon_vma_prepare(struct vm_area_stru= ct *vma) return __anon_vma_prepare(vma); } =20 -/* Flags for folio_pte_batch(). */ -typedef int __bitwise fpb_t; - -/* Compare PTEs respecting the dirty bit. */ -#define FPB_RESPECT_DIRTY ((__force fpb_t)BIT(0)) - -/* Compare PTEs respecting the soft-dirty bit. */ -#define FPB_RESPECT_SOFT_DIRTY ((__force fpb_t)BIT(1)) - -/* Compare PTEs respecting the writable bit. */ -#define FPB_RESPECT_WRITE ((__force fpb_t)BIT(2)) - -/* - * Merge PTE write bits: if any PTE in the batch is writable, modify the - * PTE at @ptentp to be writable. - */ -#define FPB_MERGE_WRITE ((__force fpb_t)BIT(3)) - -/* - * Merge PTE young and dirty bits: if any PTE in the batch is young or dir= ty, - * modify the PTE at @ptentp to be young or dirty, respectively. - */ -#define FPB_MERGE_YOUNG_DIRTY ((__force fpb_t)BIT(4)) - -static inline pte_t __pte_batch_clear_ignored(pte_t pte, fpb_t flags) -{ - if (!(flags & FPB_RESPECT_DIRTY)) - pte =3D pte_mkclean(pte); - if (likely(!(flags & FPB_RESPECT_SOFT_DIRTY))) - pte =3D pte_clear_soft_dirty(pte); - if (likely(!(flags & FPB_RESPECT_WRITE))) - pte =3D pte_wrprotect(pte); - return pte_mkold(pte); -} - /** * folio_pte_batch_flags - detect a PTE batch for a large folio * @folio: The large folio to detect a PTE batch for. @@ -430,7 +395,7 @@ static inline unsigned int folio_pte_batch_flags(struct= folio *folio, max_nr =3D min_t(unsigned long, max_nr, folio_pfn(folio) + folio_nr_pages(folio) - pte_pfn(pte)); =20 - nr =3D pte_batch_hint(ptep, pte); + nr =3D pte_batch_hint(ptep, pte, flags); expected_pte =3D __pte_batch_clear_ignored(pte_advance_pfn(pte, nr), flag= s); ptep =3D ptep + nr; =20 @@ -447,7 +412,7 @@ static inline unsigned int folio_pte_batch_flags(struct= folio *folio, any_dirty |=3D pte_dirty(pte); } =20 - cur_nr =3D pte_batch_hint(ptep, pte); + cur_nr =3D pte_batch_hint(ptep, pte, flags); expected_pte =3D pte_advance_pfn(expected_pte, cur_nr); ptep +=3D cur_nr; nr +=3D cur_nr; diff --git a/mm/mincore.c b/mm/mincore.c index 53b9828037713..e23ef3f270db6 100644 --- a/mm/mincore.c +++ b/mm/mincore.c @@ -191,7 +191,7 @@ static int mincore_pte_range(pmd_t *pmd, unsigned long = addr, unsigned long end, __mincore_unmapped_range(addr, addr + PAGE_SIZE, vma, vec); else if (pte_present(pte)) { - unsigned int batch =3D pte_batch_hint(ptep, pte); + unsigned int batch =3D pte_batch_hint(ptep, pte, 0); =20 if (batch > 1) { unsigned int max_nr =3D (end - addr) >> PAGE_SHIFT; diff --git a/mm/mremap.c b/mm/mremap.c index 384ef4cc2195b..f8b585e59771d 100644 --- a/mm/mremap.c +++ b/mm/mremap.c @@ -184,7 +184,7 @@ static int mremap_folio_pte_batch(struct vm_area_struct= *vma, unsigned long addr return 1; =20 /* Avoid expensive folio lookup if we stand no chance of benefit. */ - if (pte_batch_hint(ptep, pte) =3D=3D 1) + if (pte_batch_hint(ptep, pte, 0) =3D=3D 1) return 1; =20 folio =3D vm_normal_folio(vma, addr, pte); --=20 2.39.5 From nobody Sat Jul 25 15:51:08 2026 Received: from mail-pg1-f182.google.com (mail-pg1-f182.google.com [209.85.215.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7FEAB41E6A5 for ; Thu, 16 Jul 2026 12:43:38 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.182 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784205820; cv=none; b=tb1CbgbJPLbCddJRvwrndpckFOn+vptJIFJpTqEBjeSrzbKOWMRHlFbDkUBkpcMlzjOSiJzo6yz/JrYuCoWOTzJW1oENjOjrpfwEMsgfRmq/x53Ls5i+/NxS12riZNaIN3V5NpcimUnDXuCgYKdyAy30b2NE0up3NdpAiIn12H4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784205820; c=relaxed/simple; bh=0k8mNrfRtb2i2Q5iPvc0HG7PxgNCQ+rZMgvG4YMKhTs=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=ixkKIqSYPsPD5TQnpzqdgNMNzV4YyLDLmd2nfpIWxU+fuIPQ1Fh8AooiI4VE6YCwUQHNr4So8VFVveZF0EPFJ7Yu8bNd/suIldLExUQEwlqpsdJ0cyNTM1duhHk0kIq8h1snUzM0qKBMBLXoCiyPWnxRG74qJVlOjhpdmfvfY4c= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=i8x6AC9e; arc=none smtp.client-ip=209.85.215.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="i8x6AC9e" Received: by mail-pg1-f182.google.com with SMTP id 41be03b00d2f7-ca7bea5e5b3so4804192a12.1 for ; Thu, 16 Jul 2026 05:43:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1784205818; x=1784810618; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=Z21FwdUtYh7DDT6q0cWRgWwFAJ8AfleqYSoyR88c720=; b=i8x6AC9elZaZQMgyoJBO/FBURc8HJZZDeQFZnTN+h9igcesukhcYcoPVC1U/PNqn9D WVFLxqxfoJaUSObtSZZcZDfrDfW4hRPWSn2r4fPmLLVzpz2ivBWGcxx44iZgsYykmUOM XhOm5Twe/Dg7viY1Yo9KDO3jaFIl2sjrkj8IYqEwaHgmD/3934toJ7KNTJJ6SV41HJcX NsoWgEclGiObpOj5MDXI9wiOkLlWnQQx3XxdSxD2KDMrNN2CEEKRdq9P/FKVdsfW0m33 5uHZ9fhdfMyof84L6mPCzWXmlHo39UrRsxFvK6HpYt8cXNP9F29FnvK7fLbgmT2uIf0G O/cQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1784205818; x=1784810618; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=Z21FwdUtYh7DDT6q0cWRgWwFAJ8AfleqYSoyR88c720=; b=nRnFeQH9hjDI1mkempNBSCHFkWiqHsVaaJuqKq8GR1NJmjJZYeq7ZLFZ9/MpWHDmB7 aeihqCnwq1uba+baApYMNRvF/ejLLTD7YelxfkBJP4Jr5+Z2IIvsnDha+Mr45LniY5e4 yeSlactEKt4RiWuMLptRQihbhGqoRbVjE2r0klPKjWQhDSSuQlqyERfioGU4Fwv0BMu/ X/qMP/raIRcfJ0TQDt//BPEIXY/4rul37cXlRrUT3g1i94mc58zsfhu2MjCkYeNb78eL VZSW1R5iC1XR3YZshaaonErQWu7WCoU8lffY8tRGQZrMPwmlW30jVx8IWeLPZ3RLOtZx oebg== X-Forwarded-Encrypted: i=1; AHgh+Rp7U2DyIHbeeVc9eLkVHWXMvzTPmV07U5TZH3U2MBqjshGI1Hqg5W0WYYzl/8thOCD4dbXjw+4BZLmfCjs=@vger.kernel.org X-Gm-Message-State: AOJu0YwqzRbVOairl6/t5a5i2m+yGbiIk1fBqb8kPL9+Ix0NRmpGPxru yBlq8cfSLAuSLBBZW36CktPPlI+qzNVBpzhQiJSVyHMRogst/Sa0HHcjVzpAXBuPukwCgdy9+X7 OKIMXY0w= X-Gm-Gg: AfdE7cncH+5xlfB8lg6UFRR4gbN5Cl2t8dFCkBaX0zx6Is9gi+gdz+T/NO42+M5XUW8 qW4eMVAxtuF/RDW4HKb0qic1U6Xj0WoHXEZ0/76w8znj0l7eZ5Xz+D/ffAv0/wHezGHKGbeQ9ri I/9CrbStLT5/uHcFiW5Bo4VkSOKdcRPuK5j4dw3FOmds83TS4ncV3TqalgdXGrshKpJfyakRXF7 qAyUEtKk1s44ZrenGNayJyHRqLZ+OGcAeZilLJEs98NEI+3ELUhQGGqqEi0F4CgGmEAwkj5Y5Pl 61qLoSO8ZJsIY3MVtD4n98yuY0YrkZItGSM8MqQsa0mimUITqAFAWEiFOyiEFKaa6Ptn6foFiDm GodUz4eDUw2zSMCyl20XnPQ3RqXrNGV9nAUCc1mx6gLv6cRELK3LrJDArQs5S7+5GeLS3lqM6NL AVtUFohdhdxCG5RYpNQSbNSLg= X-Received: by 2002:a05:6a21:a98:b0:3bf:6c08:fb88 with SMTP id adf61e73a8af0-3c36c3e7638mr7532060637.56.1784205817786; Thu, 16 Jul 2026 05:43:37 -0700 (PDT) Received: from L6YN4KR4K9.bytedance.net ([2408:8606:1800:501::2:f]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-cb258eeaba2sm1730793a12.12.2026.07.16.05.43.29 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 16 Jul 2026 05:43:37 -0700 (PDT) From: Yunhui Cui To: linux-riscv@lists.infradead.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: linux-arch@vger.kernel.org, kvm-riscv@lists.infradead.org, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-efi@vger.kernel.org, linux-perf-users@vger.kernel.org, kasan-dev@googlegroups.com, Yunhui Cui Subject: [PATCH v2 06/15] riscv: make pte_batch_hint() honor folio batch flags Date: Thu, 16 Jul 2026 20:41:41 +0800 Message-Id: <40bc1db57d1fdad482a32ca6aa95031f7d5676e1.1784201104.git.cuiyunhui@bytedance.com> X-Mailer: git-send-email 2.39.2 (Apple Git-143) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Svnapot folded mappings may be exposed through a logical per-page PTE view instead of the raw encoded block entry. Generic folio batching code already defines which PTE bits are ignored when comparing entries, and folded mappings need to use the same comparison rules. Otherwise RISC-V can reject a batch that generic MM would consider equivalent simply because the hint path compared a different view of the same mapping. Install a Svnapot-aware pte_batch_hint() implementation. If the caller passes a logical sub-PTE instead of a raw NAPOT entry, rebuild the corresponding block PTE first, apply the same folio batch flags as the generic batching code, and only return a multi-entry hint if the remaining raw PTEs in the folded block still match. Signed-off-by: Yunhui Cui --- arch/riscv/include/asm/pgtable.h | 26 +++++++++++++ arch/riscv/mm/contpte.c | 66 ++++++++++++++++++++++++++++++++ 2 files changed, 92 insertions(+) diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgta= ble.h index 771faabb2b00b..da7ffea0dcd98 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -6,6 +6,13 @@ #ifndef _ASM_RISCV_PGTABLE_H #define _ASM_RISCV_PGTABLE_H =20 +#ifndef __ASSEMBLY__ +#ifndef __LINUX_FPB_T +#define __LINUX_FPB_T +typedef int __bitwise fpb_t; +#endif +#endif + #include #include =20 @@ -591,6 +598,8 @@ void napotpte_clear_young_dirty_ptes(struct vm_area_str= uct *vma, unsigned int nr, cydp_t flags); void napotpte_wrprotect_ptes(struct mm_struct *mm, unsigned long addr, pte_t *ptep, unsigned int nr); +unsigned int napotpte_pte_batch_hint_from_pte(pte_t *ptep, pte_t orig_pte, + fpb_t flags); int napotpte_ptep_set_access_flags(struct vm_area_struct *vma, unsigned long address, pte_t *ptep, pte_t entry, int dirty); @@ -1033,6 +1042,16 @@ static inline int ptep_clear_flush_young(struct vm_a= rea_struct *vma, return napotpte_ptep_clear_flush_young(vma, address, ptep); } =20 +#define pte_batch_hint pte_batch_hint +static inline unsigned int +pte_batch_hint(pte_t *ptep, pte_t pte, fpb_t flags) +{ + if (!pte_present(pte)) + return 1; + + return napotpte_pte_batch_hint_from_pte(ptep, pte, flags); +} + #else /* CONFIG_RISCV_ISA_SVNAPOT */ =20 #define napotpte_ptep_set_access_flags(vma, address, ptep, entry, dirty) \ @@ -1095,6 +1114,13 @@ static inline pte_t __get_and_clear_full_ptes(struct= mm_struct *mm, #define __HAVE_ARCH_PTEP_CLEAR_YOUNG_FLUSH #define ptep_clear_flush_young __ptep_clear_flush_young =20 +#define pte_batch_hint pte_batch_hint +static inline unsigned int +pte_batch_hint(pte_t *ptep, pte_t pte, fpb_t flags) +{ + return 1; +} + #endif /* CONFIG_RISCV_ISA_SVNAPOT */ =20 #define set_pte __set_pte diff --git a/arch/riscv/mm/contpte.c b/arch/riscv/mm/contpte.c index 3f8a0a2970afb..9ffdffdc2b0c0 100644 --- a/arch/riscv/mm/contpte.c +++ b/arch/riscv/mm/contpte.c @@ -176,6 +176,32 @@ static inline bool napotpte_is_consistent(pte_t pte, p= te_t orig_pte) pte_val(pte_mask_ad(pte)) =3D=3D pte_val(pte_mask_ad(orig_pte)); } =20 +static inline bool +napotpte_is_batch_consistent(pte_t pte, pte_t batch_pte, fpb_t flags) +{ + return pte_present_napot(pte) && + pte_val(__pte_batch_clear_ignored(pte, flags)) =3D=3D + pte_val(batch_pte); +} + +static inline pte_t +napotpte_normalize_batch_pte(pte_t *ptep, pte_t orig_pte, fpb_t flags) +{ + unsigned long pfn; + pgprot_t prot; + unsigned int off; + + if (pte_present_napot(orig_pte)) + return __pte_batch_clear_ignored(orig_pte, flags); + + off =3D ptep - napot_align_ptep(ptep); + pfn =3D pte_pfn(orig_pte) - off; + prot =3D __pgprot(pte_protval_no_pfn_no_napot(orig_pte)); + + return __pte_batch_clear_ignored(pte_mknapot(pfn_pte(pfn, prot), + napotpte_order()), flags); +} + static bool napotpte_all_subptes_same(pte_t *ptep, pte_t expected_pte) { pte_t *start; @@ -328,6 +354,46 @@ pte_t napotpte_ptep_get_lockless(pte_t *orig_ptep) } EXPORT_SYMBOL(napotpte_ptep_get_lockless); =20 +unsigned int napotpte_pte_batch_hint_from_pte(pte_t *ptep, pte_t orig_pte, + fpb_t flags) +{ + pte_t batch_pte, pte; + pte_t *start; + unsigned int i, nr, off; + + if (!napot_hw_supported()) + return 1; + + if (!pte_present_napot(orig_pte) && !pte_present_napot(READ_ONCE(*ptep))) + return 1; + + /* + * @orig_pte may be either the raw NAPOT entry read from the page + * table or the logical sub-PTE returned by ptep_get(). In the latter + * case, the public view has the NAPOT bit stripped and the PFN adjusted + * by the slot offset within the folded block. + * + * If the caller passes a logical sub-PTE, rebuild the corresponding + * block PTE first. Then apply the same folio batch flags as the generic + * batching code, and only return a multi-entry hint if every remaining + * raw PTE in the folded block still matches. + */ + batch_pte =3D napotpte_normalize_batch_pte(ptep, orig_pte, flags); + + start =3D napot_align_ptep(ptep); + nr =3D napotpte_pte_num(); + off =3D ptep - start; + + for (i =3D off; i < nr; i++) { + pte =3D READ_ONCE(start[i]); + if (!napotpte_is_batch_consistent(pte, batch_pte, flags)) + return 1; + } + + return nr - off; +} +EXPORT_SYMBOL(napotpte_pte_batch_hint_from_pte); + static void napotpte_try_unfold_range(struct mm_struct *mm, unsigned long addr, pte_t *ptep, unsigned int nr) --=20 2.39.5 From nobody Sat Jul 25 15:51:08 2026 Received: from mail-pf1-f175.google.com (mail-pf1-f175.google.com [209.85.210.175]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2BFCC41DE0B for ; Thu, 16 Jul 2026 12:43:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.175 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784205828; cv=none; b=LwYsXtWVr6Ih7cXRazXlfyXWL+1EK2+pSuDxOAROSCnLWSFL81QQ2TrANheFwspFs1FGhoyjcCzPzDzuFnjTFtlU9Az9FgAVLOQ9DtS+RIsLfCebPw+fE66/nVyItacMOXQm6WkxkeERE8QjZ+OrlZLHuG+xEhnTnxXJxDCGF+w= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784205828; c=relaxed/simple; bh=GTL7lfLqQg2oKlzbVmjbLCLKjuhCvVW9dLIhh8F61C0=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=LXbJiCbBYevnFUljwzlu3fv/m61khK4w+enRGkcev+HPtNKiuY2d5vT0U7QOkuqjrXlaocwEsUhqrm7KKRFaUYQC6KocFvcEHouWux8eoyysVuKLvft2YFmb7SYoqcFppTXb8jhHoe104J6WZEY2+BFl4LQxsKXpI5gSUvk0hls= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=F6q0PAQX; arc=none smtp.client-ip=209.85.210.175 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="F6q0PAQX" Received: by mail-pf1-f175.google.com with SMTP id d2e1a72fcca58-848479c9bd5so2590967b3a.3 for ; Thu, 16 Jul 2026 05:43:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1784205826; x=1784810626; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=qpVNfPM2loki1/2WSjY19UJOAg4x0MrZ4b63t+Pbr9Q=; b=F6q0PAQXwf5d0w1jOqLx2K0qoVg2fk8qgi+lsTSlnjUvCqj7fSXus6MPZqkQPM9Kaw SaDC+UrciFz7zxDe3qWf6RvsX2I/ss5sSWunCKtTFaxpkSCq9jRZR7Tat7rrixvVsYnA Xz00ht/xgMvKkXnJFZUYgkMr+xUnXsCLhihGDkT2BaPTEctRxiaoJiUdvNubcWyKZO5O bKpz0cq8h71x+3JBALKBTDw7Ku4SZxlza2f8fZLSua4FnbBaffiDTbvwKd7Z0FNJeJw7 KqP8uF+gmXcRK+K26obDJFO9yCszjcUWWGH7QBAm7XhGhAsKMPieDsv8HM1aoj+pKARU pRGg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1784205826; x=1784810626; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=qpVNfPM2loki1/2WSjY19UJOAg4x0MrZ4b63t+Pbr9Q=; b=nVdkb2cpX927xntwNRJgnSAyi+GG4ambQKDARHFy5FO2RvoqNXulK+zFFxpesEmvzv ClPTH/eLZRVOZDaOZPPqqpGeirjTKVLQE10pjK8mLXPB5XxUuyaWvvors/WUqDZ8ASnY WUA0fGwMUqAE8ZRmpD+/ca7SaH3VETaKbRoPuyFFWJvpfnIKOKy1B4N9d6yViDbV/HYI NhDidn6gwXZDc4BlbgEQlAKIPcq7pLa9Jm6T2jhfjc6Kun6UvSx360aHcZyp9l6xxjvt BciQPJjSSWEj6baOfO4BHNoRKgx1bY4FtKb/Tjs0KI7RFtVJCBRTUck2fbFntJRYX580 05OQ== X-Forwarded-Encrypted: i=1; AHgh+RqxjXEnwuQmTjv8Xw+44FWB2FpRzwFyAs5bF+fvTIB1yQVNvMhfob0jNQliJG1zTSuOT4iiedY+J9x0KW8=@vger.kernel.org X-Gm-Message-State: AOJu0YylIEp/SW5XeK+22NqJFXRokyTLKuwLqI3/hHB6v7Lo5zfnzl3F mTItCJMiDw4ZurJzmunLm45QIImT0Ingc1ghJl9dMD1kTJ7FxqHFsj79tnX4LTAygQs= X-Gm-Gg: AfdE7cm0nViic0zFWZcjDPqKF0xoCpinuV0t0qVVYHW/BvOrqwkWMiolmDOzx0MjNo/ wdOwaHJvr0qC2dEHpZTgCs2wybM/iIylQfGDieHA8Gs6wwsz8u/VuANFZgS3zjcst0nePPtWvdN WQHIgdYRjch07vVvXtM8bwpPAwPkCfmWsL3GRGA4ArOvYUFB6AOzeeSFr1xxPXilmt+imvvHptR mQQplPwqQk0uHKsbd713XmUv1neRJbN7xMEvswIIdWZNrn9w090L2YS2lfCd7fnXRj8NKNPRCtr fV8EXB43AB3Qcs0fBR0LWHZKjwRnk3CW58wECMVprf1nkwLvlrsWEmveDgkzvUBXNd9lHNdGARw r4oX6dj6vK5cg0cQhvTjpY30dpgJw+AEcGTwooBXflonelOD+roH81NsLahG11ipE1hfkG/g00u qKYfQNF2pwPsqrAj+W9fNMw3o= X-Received: by 2002:a05:6a00:451c:b0:848:2f74:d8d8 with SMTP id d2e1a72fcca58-8488986dd0bmr19929855b3a.73.1784205826475; Thu, 16 Jul 2026 05:43:46 -0700 (PDT) Received: from L6YN4KR4K9.bytedance.net ([2408:8606:1800:501::2:f]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-cb258eeaba2sm1730793a12.12.2026.07.16.05.43.38 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 16 Jul 2026 05:43:45 -0700 (PDT) From: Yunhui Cui To: linux-riscv@lists.infradead.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: linux-arch@vger.kernel.org, kvm-riscv@lists.infradead.org, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-efi@vger.kernel.org, linux-perf-users@vger.kernel.org, kasan-dev@googlegroups.com, Yunhui Cui Subject: [PATCH v2 07/15] riscv/mm: preserve Svnapot leaf-size semantics for page-table consumers Date: Thu, 16 Jul 2026 20:41:42 +0800 Message-Id: <180084b20de76f7ad61641e9b7abe6d103caf5f3.1784201104.git.cuiyunhui@bytedance.com> X-Mailer: git-send-email 2.39.2 (Apple Git-143) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Svnapot keeps a folded 64KB mapping encoded in the page table, while the public PTE helpers may expose a logical per-page sub-PTE view to generic MM users. That logical view is suitable for leaf-style PTE consumers, but it must not change the effective leaf size reported for the mapping. Use the original page-table entry for leaf-size queries, so page-table consumers continue to observe the Svnapot leaf size instead of incorrectly treating a folded mapping as PAGE_SIZE entries. Signed-off-by: Yunhui Cui --- arch/riscv/include/asm/pgtable.h | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgta= ble.h index da7ffea0dcd98..e4b9c05014c00 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -580,6 +580,17 @@ static inline pte_t pte_swp_clear_soft_dirty(pte_t pte) napot_cont_size(napot_cont_order(pte)) :\ PAGE_SIZE) =20 +#define __ptep_leaf_size __ptep_leaf_size +static inline unsigned long __ptep_leaf_size(pmd_t pmd, pte_t *ptep, pte_t= pte) +{ + pte_t raw_pte =3D READ_ONCE(*ptep); + + if (pte_present_napot(raw_pte)) + return pte_leaf_size(raw_pte); + + return PAGE_SIZE; +} + void __napotpte_try_fold(struct mm_struct *mm, unsigned long addr, pte_t *ptep, pte_t pte); void __napotpte_try_unfold(struct mm_struct *mm, unsigned long addr, --=20 2.39.5 From nobody Sat Jul 25 15:51:08 2026 Received: from mail-pg1-f170.google.com (mail-pg1-f170.google.com [209.85.215.170]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 15E8B41D633 for ; Thu, 16 Jul 2026 12:43:55 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.170 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784205837; cv=none; b=NvoqF9KeYb67fU23kaVIxzBFM1sXNdwy7Nr3sxGd4FXN7qaK7FkCECUEbvG6RMIqazteOsgNFCS1hld7zMv8i2KgWFY6MkQNIiqnhZRKsDB1sy4WGve44IsV/iIPuK87BHMJ1VCvz8EVyeyJ/c5tJpWUvL/hV4OIU3QHznO1N/M= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784205837; c=relaxed/simple; bh=90nS3q4DtnsYAyUyFabfkm3foM3+3QOdYnVlr+MPWhA=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=H1OcVgJSWWpPbPvGborcTYvMCFfqABDNqv9G9WB+ijuPfh8J3THKqtXXgwj2LJEEa8uMiY2+q0bgddipKZLpdto2eLZI6Lc3NPoSNpexdvWPtymNpp5z9T5UzNrinjXnSRbeUc9JXianGHT3KRnEeuf+ltzKyPNe5o1yv4OKWzI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=LHQ5UwQN; arc=none smtp.client-ip=209.85.215.170 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="LHQ5UwQN" Received: by mail-pg1-f170.google.com with SMTP id 41be03b00d2f7-c9d1fff21edso4285925a12.1 for ; Thu, 16 Jul 2026 05:43:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1784205835; x=1784810635; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=wO7rgHUMjvOOREbomqim3hRFXtwbnBvZKsCnDiin3lw=; b=LHQ5UwQNIf/GJkpohzKV5XQoQrxPWGPirBss/d4VcG/+5OpbyR/WeSHPbyBY1SsZv0 1m61k69AYNpVglRebg2jjshqg9IEbKrpzH+8Fx0orFDIFtxNtoYOOfx36dwWCR2LpjdK Vu699vFjZMbQrCCUsnnTVnBhemixvPbZam/wKUME2NUlHkPK1ejPKZli4XxYQe1mqPql 3l6WsxnHEzkGsTLDxdldqgEfC1F/xL/4LWg8w5Lx+/Px0F18aPbfQ9U2jsM08yPSqnBG SWnWmCQ3vaC1dUdF/51QRQMp0QJwZkPqFPMqpY2FLwCVNW0UNtiye/uj6kEVUdavI+cQ Y/ww== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1784205835; x=1784810635; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=wO7rgHUMjvOOREbomqim3hRFXtwbnBvZKsCnDiin3lw=; b=kg44QQhl+/4lOH7SXoFTOcaMgSSwYHH9hfNO57DYD5YTTHBWJrNBBRJCUQpOXYF+H2 QBIIy/VvSTl/11lRbulCDv+Jou7EsbSsDoyxRAE8H1nkX7x2sOHjIo57Fq1rs41tPpjt Tuw/bjKGiViWfY7z8dKMlnn3m2cjHPHz/qyeJAMJy/3bj9iUexSnUa+oGWEAGr7Z2But v8qBLHYrI1ufsZF74OiGWIkEnBexipiXVSZ3JjptDxxBO0I/JByZ0aPsU5UFKrPs3G+D no+htvTFGW+g+zApXU2xISyuIfXvXKm/1JXvAptOQZhF0GrWjCGYEvSamaK8aIeJCe6l V7Vg== X-Forwarded-Encrypted: i=1; AHgh+RrX9rCUWMXL3Vp/R4w+meIOhGaJl0NtoU87QDojqtGZOLE8bCosBg04T7ugSIlPTs6URqucxxxBvSFnoIk=@vger.kernel.org X-Gm-Message-State: AOJu0YxUVv+Ohb1qXlW+MblNzFjyagt+HdW+SEsuK9pxKRvEckfdcGEw y0jvIk0BTav79rjSGLAsk9ZptDGV5Y4KTTBrpCoK5kPl5syc7oaytLF0AQLBSujAK/Y= X-Gm-Gg: AfdE7cmZnQijCzUzDlJ26+sTZjkop+wtDvwmP8y3Fnvu5EPbE21wY4iOW82b+oBKxVb iwDSYupL1d4lnqVIljR20moyi/SeGUgjmXqp/+7rqA/Q5wJHhtQhz0SptlIlDHYKbJGycgJS8Zv GSGUk3TwEELPvVMd7mXsRGiR9lYgjJlUBooYohG/el2mA/2bkmUE7Azo9D7WIFKj75byv4zXgxP sRwTH5+zf9b2NEksUNYD9IamGU5h4CuEo40pyPNS/G6sUICBIU/CWiLfj7JYxJ1aocrQWn/Nbj0 OJsAuS7RbMM5UI7AhDZm2uPo6wtuoXpg70SAlpEPvfryx+5BTw563qMFel4wBUoAUJb4Kz3NT0V Ralt2kJk0kU8W+HlxcGeGtlBvO72YsJeqc6AEAEiiHIuQS+tAunyq1qfQ8MwRqX5822smpUiQ7p yLrKjdOWK9I2QJvBTI1mV4l0A3007V2R+eWw== X-Received: by 2002:a05:6a20:20c3:b0:3c3:859b:c20c with SMTP id adf61e73a8af0-3c3859be277mr3653741637.15.1784205835267; Thu, 16 Jul 2026 05:43:55 -0700 (PDT) Received: from L6YN4KR4K9.bytedance.net ([2408:8606:1800:501::2:f]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-cb258eeaba2sm1730793a12.12.2026.07.16.05.43.47 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 16 Jul 2026 05:43:54 -0700 (PDT) From: Yunhui Cui To: linux-riscv@lists.infradead.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: linux-arch@vger.kernel.org, kvm-riscv@lists.infradead.org, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-efi@vger.kernel.org, linux-perf-users@vger.kernel.org, kasan-dev@googlegroups.com, Yunhui Cui Subject: [PATCH v2 08/15] riscv/mm: avoid redundant Svnapot A/D aggregation Date: Thu, 16 Jul 2026 20:41:43 +0800 Message-Id: <66e43057442a0f37e0c7348016ccef382387898d.1784201104.git.cuiyunhui@bytedance.com> X-Mailer: git-send-email 2.39.2 (Apple Git-143) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Svnapot mappings encode a 64KB range as a NAPOT PTE block, while the public getter returns a logical base-page PTE. To preserve that PTE view, the getter folds accessed and dirty information from the whole NAPOT block into the returned PTE. This aggregation is an OR-style operation. Once the returned PTE has both accessed and dirty state set, later entries in the block cannot change the result. Avoid reading the rest of the block in that case. The lockless getter still validates each entry before consuming its A/D state, so it preserves the self-consistent range requirement while also avoiding reads that cannot affect the returned PTE. Signed-off-by: Yunhui Cui --- arch/riscv/mm/contpte.c | 60 ++++++++++++++++++++++++++++++++++++++--- 1 file changed, 56 insertions(+), 4 deletions(-) diff --git a/arch/riscv/mm/contpte.c b/arch/riscv/mm/contpte.c index 9ffdffdc2b0c0..70f9a0668b50c 100644 --- a/arch/riscv/mm/contpte.c +++ b/arch/riscv/mm/contpte.c @@ -309,10 +309,34 @@ pte_t napotpte_ptep_get(pte_t *ptep, pte_t orig_pte) cur =3D READ_ONCE(start[i]); if (!napotpte_is_consistent(cur, orig_pte)) return napotpte_subpte(ptep, orig_pte); - if (pte_dirty(cur)) + if (pte_dirty(cur)) { pte =3D riscv_pte_mkhwdirty(pte); - if (pte_young(cur)) + for (; i < nr; i++) { + cur =3D READ_ONCE(start[i]); + if (!napotpte_is_consistent(cur, orig_pte)) + return napotpte_subpte(ptep, orig_pte); + if (pte_young(cur)) { + pte =3D pte_mkyoung(pte); + break; + } + } + break; + } + + if (pte_young(cur)) { pte =3D pte_mkyoung(pte); + i++; + for (; i < nr; i++) { + cur =3D READ_ONCE(start[i]); + if (!napotpte_is_consistent(cur, orig_pte)) + return napotpte_subpte(ptep, orig_pte); + if (pte_dirty(cur)) { + pte =3D riscv_pte_mkhwdirty(pte); + break; + } + } + break; + } } =20 return napotpte_subpte(ptep, pte); @@ -343,11 +367,39 @@ pte_t napotpte_ptep_get_lockless(pte_t *orig_ptep) if (!napotpte_is_consistent(pte, orig_pte)) goto retry; =20 - if (pte_dirty(pte)) + if (pte_dirty(pte)) { orig_pte =3D riscv_pte_mkhwdirty(orig_pte); + for (; i < nr; i++, ptep++) { + pte =3D READ_ONCE(*ptep); + + if (!napotpte_is_consistent(pte, orig_pte)) + goto retry; =20 - if (pte_young(pte)) + if (pte_young(pte)) { + orig_pte =3D pte_mkyoung(orig_pte); + break; + } + } + break; + } + + if (pte_young(pte)) { orig_pte =3D pte_mkyoung(orig_pte); + i++; + ptep++; + for (; i < nr; i++, ptep++) { + pte =3D READ_ONCE(*ptep); + + if (!napotpte_is_consistent(pte, orig_pte)) + goto retry; + + if (pte_dirty(pte)) { + orig_pte =3D riscv_pte_mkhwdirty(orig_pte); + break; + } + } + break; + } } =20 return napotpte_subpte(orig_ptep, orig_pte); --=20 2.39.5 From nobody Sat Jul 25 15:51:08 2026 Received: from mail-pg1-f171.google.com (mail-pg1-f171.google.com [209.85.215.171]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E9CA9420486 for ; Thu, 16 Jul 2026 12:44:04 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.171 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784205846; cv=none; b=bxIvD8d/MxYoTxb9Bvn55GzDW4hoQVKwOOZ6/88TQzqa/OyGBUQMzgX4gV7bOlFlRp0B4W09csC3SEYsflGzKeVdYF7q3scG11tfUjvDpkLP//27qzlwrTtz42dgovdDAALr/qiIP74ZG0JXU/Eja+sYKrj541ZiWYBQ8IrA5ig= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784205846; c=relaxed/simple; bh=1jB7gIqPJlL2X3fTP78amNRlL24ELpsJZv6OaAlT5aE=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=BBXSYcmjPCs47jtBrX74BO8wQfrmHtAgwp8UDDrK9eHTM1Gy4CT0JulUVw0u3M+5f/R3us2mWC0f++dr6g/5LmLgflUHvgbdXmzA6qjRxrlAz9ipNfQmkN9TIkhruDKxXW57a6bVtbT+wKx2tSFFy8ZWcdSaP6x8pMDrCj5qlJk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=eo10wiwM; arc=none smtp.client-ip=209.85.215.171 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="eo10wiwM" Received: by mail-pg1-f171.google.com with SMTP id 41be03b00d2f7-c998fd549a8so4401602a12.2 for ; Thu, 16 Jul 2026 05:44:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1784205844; x=1784810644; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=RVEhlrHEVqIN0Ar3D+s2eFCiOScwlS1S3mz2hTaXKkY=; b=eo10wiwMJQKMiTLy6l++isI9WnaIKsrmIMX98HY/aWNRTseylUp9wXtVLd48pDhNVv w8Prp7kvIZcZAwIkecfTumAPfcs9aF5hn/PVmZIL9gXKvpO5wgwoVZG7GdshGTRAVXxv zl1KJ13PB5ybYnl9+YYLN6JmdfLRPq7XvZMSBFUvgHiIs2Uub+7PhboAACzDXZ78iz3k 59NCleD1Im9lQyKT1IpSC9aFZE/8vf090gL1weV2QdydCYdAp51EDhNb/HRXDUpC5Ukc 052I1oev5G6ieXJ/r+oosCFIZ8lQfQgo2KfkidFSMZ9Ene/usGtZIv4E4m6a5aZARB4K F/aA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1784205844; x=1784810644; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=RVEhlrHEVqIN0Ar3D+s2eFCiOScwlS1S3mz2hTaXKkY=; b=JTwFEwMxPldtmUlwqX8+wBepqGAlz92sufC/eogfaa5CFWMx6dibU8WnjBoco9gRvV z93qSlNdPu+zD7jSnl3zVnFu1sORsUWuuwox8yTdhH3pYZGxyEL+Hv0q3enOAwfSUwni J18u9nWSlBoobV+PbsG4DnmVq0kYHL5Ph7hvabHpkyE0l67jjygLDYcp7f7VThABarws nUQ3caAjjFHbUuDbDgQx6wCuAVcLFXrRGvjg9bEg/gSuwRYFMlmnNsfB1cO85OmYJIoR /CqhLkWvLvHbkwj99oudxXMbtAqqaExm6LBfs9Rv6rThucpg3P91lHotLOV16xwrv+WD LIjg== X-Forwarded-Encrypted: i=1; AHgh+RoftX79x2qwqRfxqdn2Qb3dAeYCKnmdxwCHdRC+GDMv0MBE438jqo06qUZybjs3MainkQjy7gdkF6zcbT4=@vger.kernel.org X-Gm-Message-State: AOJu0YzwpRCiwjKJ5F2MOXyh+x2KRncvh3tZxnnTy4VgLLAkUhtp/o+S 9oRSKUnv1K+Fwo2DpH8s85LkANO0hlwpslXLW4KIbXmeZndaEBxMxUE+tgHaPLyJiCg= X-Gm-Gg: AfdE7cl6DvhEupe+usHuOD/zQ4KAXL2wNrBinWe7lBv/puIfpQQ3+dZ+duDhYmE93Th m/LEv8OdEgWvbTscxC/9x83JwqMbzZSGoohuwnNlWaJw3mO12HDNKtGkM0tlroJPHZ62/ryosgA g5TtTR5YlmP5DM/hsG7MJVASZw/IWKJqDOXIxHpCZeliUERR1J0XryZZcOjNpwKW/leA2TnWW5J c0MK6+W8lX5B1+vvMsIUB4vmjF7zJinkX4aArB6m9vV6Ml+nQzzS1oBuzL1RBsVpHi3rM2s5OKb 8l0zEEzpTzaJOsGjSAoWVLleK/7LXbuWvrAMKuFjctLhlVHRplW83LQqkOg3czZ2P5JbsfV9aSd 3Dwu19BemUT6OIZHcP5j8SjpJ3bsgmrBiaKFmAgPu2hlGIMBhevSE5ONgPepXxWRN0dUI5WVLp0 aHta17IR6o9EPkkK9Ppi1Mobg= X-Received: by 2002:a05:6a20:4308:b0:3ba:cd5b:3dc8 with SMTP id adf61e73a8af0-3c36c1f1e9dmr7530059637.31.1784205844222; Thu, 16 Jul 2026 05:44:04 -0700 (PDT) Received: from L6YN4KR4K9.bytedance.net ([2408:8606:1800:501::2:f]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-cb258eeaba2sm1730793a12.12.2026.07.16.05.43.55 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 16 Jul 2026 05:44:03 -0700 (PDT) From: Yunhui Cui To: linux-riscv@lists.infradead.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: linux-arch@vger.kernel.org, kvm-riscv@lists.infradead.org, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-efi@vger.kernel.org, linux-perf-users@vger.kernel.org, kasan-dev@googlegroups.com, Yunhui Cui Subject: [PATCH v2 09/15] riscv/mm: avoid Svnapot consistency checks in ptep_get() Date: Thu, 16 Jul 2026 20:41:44 +0800 Message-Id: <687dadf430c49a840e24976e37ba09f991bc3222.1784201104.git.cuiyunhui@bytedance.com> X-Mailer: git-send-email 2.39.2 (Apple Git-143) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" napotpte_ptep_get() builds the returned logical sub-PTE from the initial PTE observed at the target address. Neighbouring entries in the NAPOT block are only scanned to gather accessed and dirty state, so they cannot contribute PFN, permission, present or writable state to the returned PTE. Drop the range consistency checks from this ptep_get() path. Callers that rely on the block-wide A/D view are expected to hold the PTL. There are generic lockless ptep_get() users, such as userfaultfd_must_wait() and HMM walks after pte_offset_map(), but those paths use the present, marker, write and PFN state and do not consume the gathered A/D bits. Keep the consistency checks in napotpte_ptep_get_lockless(). That API must return a self-consistent PTE snapshot to lockless walkers and fault retry paths. napotpte_is_consistent() verifies that every scanned entry still belongs to the same NAPOT mapping, ignoring A/D bits, and retries if a concurrent unfold, rewrite or refold tears the range while A/D state is being gathered. Signed-off-by: Yunhui Cui --- arch/riscv/include/asm/pgtable.h | 3 +++ arch/riscv/mm/contpte.c | 19 +++++++++++++------ 2 files changed, 16 insertions(+), 6 deletions(-) diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgta= ble.h index e4b9c05014c00..dd56c20a817bb 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -930,6 +930,9 @@ static __always_inline void napotpte_try_unfold(struct = mm_struct *mm, /* * Public PTE helpers may transparently handle Svnapot-encoded mappings. * NAPOT-aware arch users should stick to the private/raw __pte* helpers. + * Callers relying on the block-wide A/D view from ptep_get() are + * expected to hold the PTL. Lockless callers that require a + * self-consistent Svnapot range must use ptep_get_lockless(). */ #define ptep_get ptep_get static inline pte_t ptep_get(pte_t *ptep) diff --git a/arch/riscv/mm/contpte.c b/arch/riscv/mm/contpte.c index 70f9a0668b50c..0f66426b0ecfd 100644 --- a/arch/riscv/mm/contpte.c +++ b/arch/riscv/mm/contpte.c @@ -294,6 +294,13 @@ EXPORT_SYMBOL(__napotpte_try_unfold); =20 pte_t napotpte_ptep_get(pte_t *ptep, pte_t orig_pte) { + /* + * Gather access/dirty bits from the whole NAPOT range for the + * ptep_get() view. The returned sub-PTE is built from orig_pte; + * neighbouring entries only contribute A/D state. Lockless callers + * requiring a self-consistent range must use ptep_get_lockless(). + */ + pte_t pte, cur; pte_t *start; unsigned int i, nr; @@ -307,14 +314,10 @@ pte_t napotpte_ptep_get(pte_t *ptep, pte_t orig_pte) =20 for (i =3D 0; i < nr; i++) { cur =3D READ_ONCE(start[i]); - if (!napotpte_is_consistent(cur, orig_pte)) - return napotpte_subpte(ptep, orig_pte); if (pte_dirty(cur)) { pte =3D riscv_pte_mkhwdirty(pte); for (; i < nr; i++) { cur =3D READ_ONCE(start[i]); - if (!napotpte_is_consistent(cur, orig_pte)) - return napotpte_subpte(ptep, orig_pte); if (pte_young(cur)) { pte =3D pte_mkyoung(pte); break; @@ -328,8 +331,6 @@ pte_t napotpte_ptep_get(pte_t *ptep, pte_t orig_pte) i++; for (; i < nr; i++) { cur =3D READ_ONCE(start[i]); - if (!napotpte_is_consistent(cur, orig_pte)) - return napotpte_subpte(ptep, orig_pte); if (pte_dirty(cur)) { pte =3D riscv_pte_mkhwdirty(pte); break; @@ -345,6 +346,12 @@ EXPORT_SYMBOL(napotpte_ptep_get); =20 pte_t napotpte_ptep_get_lockless(pte_t *orig_ptep) { + /* + * ptep_get_lockless() must return a self-consistent PTE without the + * PTL. Recheck that the whole NAPOT range still describes the same + * mapping, ignoring A/D bits, and retry if a concurrent update tears + * the range while A/D state is being gathered. + */ pte_t orig_pte, pte; pte_t *ptep; unsigned int i, nr; --=20 2.39.5 From nobody Sat Jul 25 15:51:08 2026 Received: from mail-pf1-f175.google.com (mail-pf1-f175.google.com [209.85.210.175]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7FF3141F5D3 for ; Thu, 16 Jul 2026 12:44:14 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.175 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784205856; cv=none; b=RM/EGdzkLKkdiuME5bDqvKZ8NrxlsKEqY7j3QwPWm+7oGGxFDBbB5B7f4pwNoS0PT1sbH33LyL/ph3HVdWQS2HXbIs2zuP5MrRO0KchrLZwVmJayqCClWxEucDz0RS+TFIgYRrFBClM1tpeAIlq0kHiBIYxlaYXeYjDSyMaTCnM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784205856; c=relaxed/simple; bh=4deY2p8H6GZ6zm5fDTusItQoJGUZoIhd5ZPqjCZKV8M=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=OcVvOazkgRxD8hGfgAraMyzFhws0Pdc+HwDwOXji4Lxqw8jjAe98W9c41F6nbcrDOqSQ9EOIRhuz4hIsPwypE1jpgdjiV80JEHR/dDd1tsfXMzCK7pA23L5f5mBDxfG2SKZ1zBeW32YBzKkT9LxcO+OcUNCIWMzD3HARt1zWbh4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=ZAj+7QOj; arc=none smtp.client-ip=209.85.210.175 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="ZAj+7QOj" Received: by mail-pf1-f175.google.com with SMTP id d2e1a72fcca58-84862b0d5f8so2852272b3a.3 for ; Thu, 16 Jul 2026 05:44:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1784205854; x=1784810654; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=ZqzuyyB7zI40Qj8rdd4/j/Hz77YcBH2L+r/0EHWbI6M=; b=ZAj+7QOjASlsckuQXpHLU0FAunWhMybQoKD5W1AGa50FQayUMJnImiyviiMSbReByf va4QGfRhrqAk25/wcSzdNnk7Uh+wRnkJmRmorSkk3ZueuR5PhG4jW5y5CLMzyErWfUCe PV3JOuaWu6DLe1gZvFlvl2NcWkbb2atm7j7MfDmn56Ndaz6yPyc4P3Buux7GIlpsjUz1 O9OcZzo5ZEGSFis8y/e4dj8Edgugo5P3zZOn62SIZPiC5lk0LQU85KcX58BXc361Qf7H n5qvwflHbCKVpNF1yk7nWMc5X5VbqNH7sSb1x+IXZ4m8L7mNy8+vS2S33zqMuWOk0PaE 6fYg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1784205854; x=1784810654; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=ZqzuyyB7zI40Qj8rdd4/j/Hz77YcBH2L+r/0EHWbI6M=; b=XjwBqoa1ut7L1c9JTP/c8iNSjFaNMUWun7/+98AtRiiiFLW+TT599cVkhtbxz7sWUH zYyAoXZy2LFFGJWAAj41feWPqtQvmyObVAz1MUeHmmMweSplD9z54BC+dJT7OKJetXde Hcp0oXCTDfnWEz2DeX843a8plCDJ1zZ3kz6sQA6JyjutLYqSOoMsuA15YECdxcXHtDRK xfV+6v7QkPROG3YRujG7lZMqHCEPitaUmEFdT847CDhL0uo7oO9i8Yru0B4AjdvMfC/I A93KuaN1V4MY+Vs8yZQtl6UBf+lAyzsnkmIRr9oMcFdB6xUCysWgyHbZeIhOyJ5/N8Ez 2aDQ== X-Forwarded-Encrypted: i=1; AHgh+RqwCHKtPWBLtTRXmt5a/D9ZNDEbQ+5D/UYWu4GbfyKisba3rygter75OHN/XnTK7hehJeBPKJREGC10gto=@vger.kernel.org X-Gm-Message-State: AOJu0YyG1V562x/MVFfDhT+iSgiEI0/8/sZS9FrknTZtaqOyahLMoq6/ hheTi9e7KH+ZHLM+Tiagv4Ko1FEI6DZgeXjtwvtostw704XFh/EDmLUjXoTgSJjJwnc= X-Gm-Gg: AfdE7ckQPQDLcK02C72UYw3e004JnYktkV1m9fxNMegoqmbTaeeYk/ZhlT1F5HbR38k HUaqv0L8lHcHQcjAnoZMbC25WbkaOeT13vH0znOTgiJcwT+FKUDwTnlLOI86yUWSdvFqCbM8/8k fYuxgrW4KmQmHO1mi+C/AZpsdWPzhtt2hINwxfvXnzPOaHlyWY1eJMKDdJL8i7rzlTH2emhH0DP KFbOPa0Eoy0Jyu5qrD5LNjjHYiIPS0YqfCYWcs+dptp5cwi1YSRF86vn2wDktZ3Y2q5XmKk2PSb I2Xbk8XAaXE73Q9Gj0FvChtiblu5FOrJBK/AQdsZXfXsyBjzhfJerRwTnbVsu5WgSCc/tUmDEjI w2DytqaLgCaot5X7+YG7I6rG9hP74XnGqodZRj/7EJF08FcHGruHX4sIeAaz/NB6SZ5TYmtp4+U rbcQXJLCAhRKdTLDPt3ocAQ/U= X-Received: by 2002:a05:6a00:21d2:b0:845:c6fc:2821 with SMTP id d2e1a72fcca58-84889604f51mr19555102b3a.22.1784205853705; Thu, 16 Jul 2026 05:44:13 -0700 (PDT) Received: from L6YN4KR4K9.bytedance.net ([2408:8606:1800:501::2:f]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-cb258eeaba2sm1730793a12.12.2026.07.16.05.44.04 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 16 Jul 2026 05:44:12 -0700 (PDT) From: Yunhui Cui To: linux-riscv@lists.infradead.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: linux-arch@vger.kernel.org, kvm-riscv@lists.infradead.org, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-efi@vger.kernel.org, linux-perf-users@vger.kernel.org, kasan-dev@googlegroups.com, Yunhui Cui Subject: [PATCH v2 10/15] mm/gup: add fast-GUP specific lockless PTE helpers Date: Thu, 16 Jul 2026 20:41:45 +0800 Message-Id: <138d74aa9863ae8ff445e16a2798d062895b75bc.1784201104.git.cuiyunhui@bytedance.com> X-Mailer: git-send-email 2.39.2 (Apple Git-143) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" fast-GUP snapshots a PTE without holding the PTL, pins the page, and then revalidates that the PTE did not change. Keep both reads under the same lockless PTE semantics by defaulting the fast-GUP revalidation to ptep_get_lockless(). Introduce fast-GUP specific PTE snapshot and revalidation helpers. The default implementation preserves the existing ptep_get_lockless() semantics, while allowing architectures to override the helpers when the public lockless getter provides extra semantics that fast-GUP does not consume. Signed-off-by: Yunhui Cui --- include/linux/pgtable.h | 18 ++++++++++++++++++ mm/gup.c | 6 ++++-- 2 files changed, 22 insertions(+), 2 deletions(-) diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index da14328093a86..62943fcbf7046 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -813,6 +813,24 @@ static inline pte_t ptep_get_lockless(pte_t *ptep) } #endif =20 +#ifndef gup_ptep_get_lockless +static inline pte_t gup_ptep_get_lockless(pte_t *ptep, pte_t *rawp) +{ + pte_t pte =3D ptep_get_lockless(ptep); + + *rawp =3D pte; + + return pte; +} +#endif + +#ifndef gup_ptep_revalidate +static inline bool gup_ptep_revalidate(pte_t *ptep, pte_t raw_pte) +{ + return pte_val(raw_pte) =3D=3D pte_val(ptep_get_lockless(ptep)); +} +#endif + #ifndef pmdp_get_lockless static inline pmd_t pmdp_get_lockless(pmd_t *pmdp) { diff --git a/mm/gup.c b/mm/gup.c index 99902c15703b0..72fb147193e55 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -2842,10 +2842,12 @@ static int gup_fast_pte_range(pmd_t pmd, pmd_t *pmd= p, unsigned long addr, if (!ptep) return 0; do { - pte_t pte =3D ptep_get_lockless(ptep); + pte_t raw_pte, pte; struct page *page; struct folio *folio; =20 + pte =3D gup_ptep_get_lockless(ptep, &raw_pte); + /* * Always fallback to ordinary GUP on PROT_NONE-mapped pages: * pte_access_permitted() better should reject these pages @@ -2871,7 +2873,7 @@ static int gup_fast_pte_range(pmd_t pmd, pmd_t *pmdp,= unsigned long addr, goto pte_unmap; =20 if (unlikely(pmd_val(pmd) !=3D pmd_val(pmdp_get_lockless(pmdp))) || - unlikely(pte_val(pte) !=3D pte_val(ptep_get_lockless(ptep)))) { + unlikely(!gup_ptep_revalidate(ptep, raw_pte))) { gup_put_folio(folio, 1, flags); goto pte_unmap; } --=20 2.39.5 From nobody Sat Jul 25 15:51:08 2026 Received: from mail-pf1-f169.google.com (mail-pf1-f169.google.com [209.85.210.169]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A863341F371 for ; Thu, 16 Jul 2026 12:44:23 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.169 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784205865; cv=none; b=eDVDW8EVuc98lP8nVEoQsu4Frl6hZYgnLubLDoD4thbEIa8PafDlb/kywOwXetJ4MbspATClEcMBMm3bjQQw6oFeUqWnC1ioH2bEXIDMxDV/Joe3DCWNCyp8/SL1a+dSXbopyyoYGPqA8CvvsjBrloleexqirWKBjHQ7YYYHz6A= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784205865; c=relaxed/simple; bh=rftdHtLNV98O/408NnTvtHdmuZ/zkcn7vT2O02oj0qc=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=NVst4jS7lcszRCbOg9SNESJf79Cn1C9S+saAZjgpFgXL0M8dt/3pkrVpzvL4aZTJLxXbUp/tGejjtxjhoR3eYAHcvmoYVCu4zmSwHpjG0eY5F3ceNhoRX8tha1K4xM7xDO5GNeknEeblXpw09KCs4LQwyAm+t3bYwnjMuYuk7bI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=St7eDzRl; arc=none smtp.client-ip=209.85.210.169 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="St7eDzRl" Received: by mail-pf1-f169.google.com with SMTP id d2e1a72fcca58-848593533cbso3358421b3a.3 for ; Thu, 16 Jul 2026 05:44:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1784205863; x=1784810663; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=MW3VkNC58mnWKcn8BtrFop2mUDTlw3bfVwg/aYM+yrM=; b=St7eDzRlxPTWCxHLHWoBJomIpw2Aw3SH5gTQBQlCcxnQd4tuAszGXCMIZDPuSq4VbB LkL5Y7Wzu2Tmvl54F6R9RdKK3SHWkuxSxiuEaZoTouaqTni1jlHo9cDKTwRnvaDkVgSA qnwyncSsWLdJsm1InJCJ9cBGs5XNR9HjzIqUNReUUweHUYEb995UrPL8kZCJxom3ocSW u9LOmPUwngXgZADEAMCOD7PT+wtybezb37fKnHAR5zDLVV3eBpRsFDmS5jM5BLeIAY2v kFLeZx7WZEubSA8oog8n1GUMtv0gjDuQNqKrVXuAQmI+krN7QhPNrQk/PJz0P299kqpI laJg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1784205863; x=1784810663; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=MW3VkNC58mnWKcn8BtrFop2mUDTlw3bfVwg/aYM+yrM=; b=MrzQpWF3ue+e1TheM/KJpeVCfVBC40AsSJnx3ihlklLKQJ2gRO6cRif87yLc76sBjV L4Qy96z2bbpnhbe4QvS4irYhJPGXj1e0tc5+xn7dalcdY55LwhJgFygevtPZyN3PT/Hw +k9HCVm3uREHiMrRWRXWOqM4sIvluSyctt2Rql4KwzYYozOvAZHdsdvEeQompZiGC6gt ISDHXuPHFEr+Q7OOq464oZ4cRnlqhXkzqck8hivb05eyodzMQg8te0gB3xFhb8JjSyU5 Y3bcxuUp5l5RnCliRRs3dVWx1wZlj6aWzLx70GXwYGJHHy/DRszWCnhSy5DL8q9A20Uk X9kA== X-Forwarded-Encrypted: i=1; AHgh+RrP6Akcr3Wz6McS3HdIkNh3xNhpZ5Vi3whUgdfAS3F77UEb+G/F7BF6R/jWh6OewD96YRgpy6HwRnOJFII=@vger.kernel.org X-Gm-Message-State: AOJu0YxY0kS8dWxUs8oKEZwIw3h3ksQX0COdpm/DUMtvo49ZoS9aX5JZ qCt2TJZo+5JippFBYfV7ET1RoGU65CBAYFn/BpqQncot2TdkNQVG7KKtTqH6egtOQoQ= X-Gm-Gg: AfdE7clGGLRwuv6zxk7RIeQXqwxFeZJ/irq/mWDO9B3153vx98N9ImTOE2iHUeMCn+w Th4sYbnK9VOy4goshEzk1omI72K4xYf5KCEGdbq+2bqDNvp6uoNN1qFsxRmUgmDgTP7sAM/EBCR SjGOWvAQQiCGRZtdJ0IXCNHE0n2EH71pI7vTp1sQY7kbgeIlzh1lo/2/lkxlRn6Zryy4w8UTBoH ybSLhPXeuITP5m0o1HArcJeKJabvCrffzzHGDRK8vpdGarIBOROg/OcH2NSOrHdG4iyDajqhEVC mjrRqIiIx3DLp0T05P/w60/5j2p3m1Q3hUf0vOniWth53bMu9qCQPQjFFpUD56wrmKFTC3SAx2S 3a17hsyftyFQ1MVeekXYHE3R0ICtswtT03K9ZzY1yRodm5cc8xcw1AvIOOszYuND0n/MwtQq/f2 d/HNvKiNBPxh4x9KPZnePsx6M= X-Received: by 2002:a05:6a00:4c98:b0:848:4b27:36ce with SMTP id d2e1a72fcca58-84a673c79a1mr6078389b3a.73.1784205863094; Thu, 16 Jul 2026 05:44:23 -0700 (PDT) Received: from L6YN4KR4K9.bytedance.net ([2408:8606:1800:501::2:f]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-cb258eeaba2sm1730793a12.12.2026.07.16.05.44.14 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 16 Jul 2026 05:44:21 -0700 (PDT) From: Yunhui Cui To: linux-riscv@lists.infradead.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: linux-arch@vger.kernel.org, kvm-riscv@lists.infradead.org, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-efi@vger.kernel.org, linux-perf-users@vger.kernel.org, kasan-dev@googlegroups.com, Yunhui Cui Subject: [PATCH v2 11/15] riscv: mm: avoid Svnapot A/D aggregation in fast-GUP Date: Thu, 16 Jul 2026 20:41:46 +0800 Message-Id: <95699db9c9b90fd1f95310e9d95b000eb8cfc616.1784201104.git.cuiyunhui@bytedance.com> X-Mailer: git-send-email 2.39.2 (Apple Git-143) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" fast-GUP only needs the PTE state for the current virtual address and revalidates the raw PTE after pinning the page. It does not consume the block-wide accessed/dirty aggregation provided by the public Svnapot lockless getter. Override the fast-GUP PTE helpers to translate a NAPOT PTE into the current sub-PTE without scanning the whole NAPOT block. Keep the public ptep_get_lockless() semantics unchanged for other lockless users. With 64K THP enabled in always mode, fio 4K random direct read throughput improves by about 10% over the previous implementation on the same system, from 2.615M to 2.874M IOPS and from 9.97GiB/s to 11.0GiB/s, while average clat drops from 95.54 usec to 86.97 usec. Signed-off-by: Yunhui Cui --- arch/riscv/include/asm/pgtable.h | 20 ++++++++++++++++++++ arch/riscv/mm/contpte.c | 9 +++++++++ 2 files changed, 29 insertions(+) diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgta= ble.h index dd56c20a817bb..3727c148fa0fc 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -597,6 +597,7 @@ void __napotpte_try_unfold(struct mm_struct *mm, unsign= ed long addr, pte_t *ptep, pte_t pte); pte_t napotpte_ptep_get(pte_t *ptep, pte_t orig_pte); pte_t napotpte_ptep_get_lockless(pte_t *ptep); +pte_t napotpte_ptep_get_gup_fast(pte_t *ptep, pte_t orig_pte); void napotpte_set_ptes(struct mm_struct *mm, unsigned long addr, pte_t *ptep, pte_t pte, unsigned int nr); void napotpte_clear_full_ptes(struct mm_struct *mm, unsigned long addr, @@ -956,6 +957,25 @@ static inline pte_t ptep_get_lockless(pte_t *ptep) return napotpte_ptep_get_lockless(ptep); } =20 +#define gup_ptep_get_lockless gup_ptep_get_lockless +static inline pte_t gup_ptep_get_lockless(pte_t *ptep, pte_t *rawp) +{ + pte_t pte =3D __ptep_get_lockless(ptep); + + *rawp =3D pte; + + if (likely(!pte_present_napot(pte))) + return pte; + + return napotpte_ptep_get_gup_fast(ptep, pte); +} + +#define gup_ptep_revalidate gup_ptep_revalidate +static inline bool gup_ptep_revalidate(pte_t *ptep, pte_t raw_pte) +{ + return pte_val(raw_pte) =3D=3D pte_val(__ptep_get_lockless(ptep)); +} + #define set_ptes set_ptes static inline void set_ptes(struct mm_struct *mm, unsigned long addr, pte_t *ptep, pte_t pteval, unsigned int nr) diff --git a/arch/riscv/mm/contpte.c b/arch/riscv/mm/contpte.c index 0f66426b0ecfd..b0b8ff0aade3a 100644 --- a/arch/riscv/mm/contpte.c +++ b/arch/riscv/mm/contpte.c @@ -413,6 +413,15 @@ pte_t napotpte_ptep_get_lockless(pte_t *orig_ptep) } EXPORT_SYMBOL(napotpte_ptep_get_lockless); =20 +pte_t napotpte_ptep_get_gup_fast(pte_t *orig_ptep, pte_t orig_pte) +{ + if (!napot_hw_supported() || !pte_present_napot(orig_pte)) + return orig_pte; + + return napotpte_subpte(orig_ptep, orig_pte); +} +EXPORT_SYMBOL(napotpte_ptep_get_gup_fast); + unsigned int napotpte_pte_batch_hint_from_pte(pte_t *ptep, pte_t orig_pte, fpb_t flags) { --=20 2.39.5 From nobody Sat Jul 25 15:51:08 2026 Received: from mail-pg1-f182.google.com (mail-pg1-f182.google.com [209.85.215.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A3B4441F5F2 for ; Thu, 16 Jul 2026 12:44:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.182 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784205873; cv=none; b=KDg14mZUJUvp0veCSYWnzzT4ctwBhHbB5HAHDcOel7pWrNIkTAhMeS0arhgz0Mk8iSOAOimVrFiw32mOpW2up+Gn5vcc/MqsiI2TohKqYT790jVSSh3YuU+ciujzDfUwFwkj7DyYirFHT09rJbj6ukqqZDlWXpCXDO7+G17aExk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784205873; c=relaxed/simple; bh=z8jC4aCjR+Oesc+bLFA5i5JVDTkwm+Fh6FK3DV0FuyQ=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=BAOHP1Fs3DDv1XLhBWtj3+TcGMk8qgI2nLvY1VL4mMq+ilsNiKqiAfzJY6UVfDSl/r4s4kRmcgqKHh5RuS6mJRPQT9fcHxFvXddjlWov36q9pLOfRCgaiJzcbDaGWPZL/xQ5ERFKsIU3L8MdwF5gvjJw7wfblWZK2qdGNPTz08Y= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=C45IRpyv; arc=none smtp.client-ip=209.85.215.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="C45IRpyv" Received: by mail-pg1-f182.google.com with SMTP id 41be03b00d2f7-ca913a601fbso4545374a12.3 for ; Thu, 16 Jul 2026 05:44:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1784205872; x=1784810672; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=s0haKjxFEA+XqfWLiPeMGRklJhvkZG2bsdABvuKjaV0=; b=C45IRpyvX6HmHdRZR3ZZ+1wy3BA5jkkEbDBp8+0GIb3o9EyyKPwbNvHw2gGop+WH2c pJiaNzVp6Tniz9si9bxp1FJoCGkPE8k84euMkqm9kHAOFakSHiXZCDq0Q+XNGenmtk0X AcNSdJFQzBIV3POW6J76B1U56B9a2HCGgTukkQN1LtUpyVMDf+QoJHwDHIP6cYbYhScP 4Dl8WlrRhjr4kPy3MarWxpCJp9GcpM6nLT8uYIJ1cgPvf+qJwvLg+0bPsQ6qcGg1d3A1 k+9C/SiPoGlnyNUZlQMiavpIldAGPonCUQBrMKCOqletZW6U9r8Ivbrkc7cDS/RvKyUG xz0Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1784205872; x=1784810672; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=s0haKjxFEA+XqfWLiPeMGRklJhvkZG2bsdABvuKjaV0=; b=Fm8jamd1UROZj2Qop8Q9WNADocwSX71oGoeHRYb2z63z3ZGIsJBrzCAJb3SMDjYD5M HujmoDGY89uTp/eCuNAilDUKihD+BMMqIv1BDimo34Z4QNWgVQUPgrjtKSZyn45HB5YF /aCzKTwMkVysRwQNANQ+pDx6uO7CZrqmyksmmvGbLNfdUFgvOaZMGN0X9GMVyZM6gKL0 UA9RKsMvauF+ISlSQx63r6U6XNHPJcXucWpz/m4yRIkSFJYBig7VfLylVOYXLUGAg9Gl 6iUtqngmsqMA5TIL+sJponX1IBj8f6D7qtGxhQk6kYlPZNCI7mK1JoD58Kha+hba3M/W 47jg== X-Forwarded-Encrypted: i=1; AHgh+RpJTX9tWEx0A+PthmQbg2sdS1Ps3wAklKKwrwm/sb7ocXYsK5cRxvGSUAn/LpkwMWl7V/rkgit/EaHX3P4=@vger.kernel.org X-Gm-Message-State: AOJu0YxSG2MRQZJ7pa22yQEZFgLtvBiQJO8ifC2vpRuyZXcw0CQngC9t dNEk2eWoV2tf70QGRFNkYLouib8TWzoUxchJ8/nz9nGWOx3QXdbtQa/9rPL4yA6Ye7Q= X-Gm-Gg: AfdE7cmclusHxJ4c2SUc5fsUiWBNBumbqr4yff3fLx+3O2WZ2smU2owtux7CjoP0A75 tO8hszgPf9Id5jLnSkqd5qWEt0vQjn3XBKwM9oaFi0xYQW8MiAaYD+tTor+vHpEBeMJ2xUfivuM PUnzgc9T3nTNU/qKskw3S08tAZ41aLPWvXe/dIc6A46oP7tNi/2XHBUAV/c6jmwx924MRXf1vSz Pveu7XesYibdsMGiU332DLM0ClrbR22I5L4X+9cnKnufeYWAgQgucJ7LR9T1NTob2g+nGP+cJnH 0yPVb9pko5GWWXbOaXY7xHnZpUJ/FFtpOy662rF0viMzDDpmIIaA386z0OiCWwBfERxnYxfTzLd kHlBg73lJqrUFo7rqouRUBQoAoNzdb09mc0ojzFIBS2VLpvfVGV4iKN+pfkct2BYnIUhpUupnQq wVtr7fvzIfXntLhkzmgfUqlAPqWSq21kEhLA== X-Received: by 2002:a05:6a20:e615:b0:3c0:9c19:6590 with SMTP id adf61e73a8af0-3c3576c37f6mr12235735637.74.1784205871769; Thu, 16 Jul 2026 05:44:31 -0700 (PDT) Received: from L6YN4KR4K9.bytedance.net ([2408:8606:1800:501::2:f]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-cb258eeaba2sm1730793a12.12.2026.07.16.05.44.23 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 16 Jul 2026 05:44:31 -0700 (PDT) From: Yunhui Cui To: linux-riscv@lists.infradead.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: linux-arch@vger.kernel.org, kvm-riscv@lists.infradead.org, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-efi@vger.kernel.org, linux-perf-users@vger.kernel.org, kasan-dev@googlegroups.com, Yunhui Cui Subject: [PATCH v2 12/15] arm64: mm: avoid contpte A/D aggregation in fast-GUP Date: Thu, 16 Jul 2026 20:41:47 +0800 Message-Id: <92fbae3247b4bf0d7989ba20a8763ecb836d0279.1784201104.git.cuiyunhui@bytedance.com> X-Mailer: git-send-email 2.39.2 (Apple Git-143) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" fast-GUP only needs the PTE state for the current virtual address and revalidates the raw PTE after pinning the page. It does not consume the contpte block-wide accessed/dirty aggregation provided by the public lockless getter. Override the fast-GUP PTE helpers to use the raw PTE for the current entry and raw revalidation. Keep the public ptep_get_lockless() semantics unchanged for other lockless users. Signed-off-by: Yunhui Cui --- arch/arm64/include/asm/pgtable.h | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgta= ble.h index 018781a1fcbee..d1e08dce9f7c5 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -1760,6 +1760,22 @@ static inline pte_t ptep_get_lockless(pte_t *ptep) return contpte_ptep_get_lockless(ptep); } =20 +#define gup_ptep_get_lockless gup_ptep_get_lockless +static inline pte_t gup_ptep_get_lockless(pte_t *ptep, pte_t *rawp) +{ + pte_t pte =3D __ptep_get(ptep); + + *rawp =3D pte; + + return pte; +} + +#define gup_ptep_revalidate gup_ptep_revalidate +static inline bool gup_ptep_revalidate(pte_t *ptep, pte_t raw_pte) +{ + return pte_val(raw_pte) =3D=3D pte_val(__ptep_get(ptep)); +} + static inline void set_pte(pte_t *ptep, pte_t pte) { /* --=20 2.39.5 From nobody Sat Jul 25 15:51:08 2026 Received: from mail-pl1-f175.google.com (mail-pl1-f175.google.com [209.85.214.175]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 572E741F5CB for ; Thu, 16 Jul 2026 12:44:41 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.175 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784205884; cv=none; b=AwbdF3a+DVr4cIToHrScCD1WWhG7SZfBdPM5I4CLNHS+GGKP7ZiNQ6XO95tEfVJHwDEjBdiOiYyGHg7BqgKL88zMjsh55grUXOeT39AjnJXz7s6UGlEDnYEzkJx8s2wVGhkwdxZsxkYSiV3hN940aAR5c0WfkgxNUNY18xyJKG4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784205884; c=relaxed/simple; bh=C9vr1GpyOKZPJayEcfEUxAQT/YmOuTmwy42kU/s05EI=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=f6jfwa/OqL0cDiaDrKRhUNsM3z7I/0txu88bjubsoGl/iVZlE43Fn5DIkEWMC5ObHI2nCI3lqwjQOf7A0H/PC2pllpAE9uDRXAoiL1VtQIJwtvLCQO2lAjIJu5bghx3MVpt2kLkZ0J4c+4WWVzK7wPiiylRwWoLo1+6KSCQFDcE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=gyogljhk; arc=none smtp.client-ip=209.85.214.175 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="gyogljhk" Received: by mail-pl1-f175.google.com with SMTP id d9443c01a7336-2cab973140bso79475765ad.3 for ; Thu, 16 Jul 2026 05:44:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1784205881; x=1784810681; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=xl9SXJgNY1wsw8DmDdohBTpB25DAPwcBNYqh5j7w6mI=; b=gyogljhkXmG1zV+1zNRnCZRoh21QCNeeGkdNQELVkKXZH4wUKLl1pSeGbO5DaqQ1xX EcgA2ESGbQqnTh6ou9FS172rAiOIeTOxWAgaUqRVLLmtlW0+foeAfUOQxcnYOJKHPE9l IeGjgzvqMPw6fgvV47OczXuexfBLjQZSOclJiYivqiPs1vMBcbwvrp8RR/2wtgXD46rD d8HZtq7Sv8P7vIn5DZmaTrVuICREhUejrDDon2KLJnUeGZwrNwLfw8x52DRXyICLNmSG 7ySDml4AWSjLzOc5TTNg4h27Ia9n9pr5EfMXRmxUSxPdwi3JNFuLFjIARTLEzy9ZqQkI f6Rg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1784205881; x=1784810681; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=xl9SXJgNY1wsw8DmDdohBTpB25DAPwcBNYqh5j7w6mI=; b=JaACkZ+dY8qAl5wEBc+D0q7cqII42svR3adOf7aCQ4wB0WbgD3pvslwo6eXCgLR64F r7YKhOnPWRJJAmqfXGwIhUlexG4E3mzPhdYT+ispDkJpdJrYAEGBQDb7Mb8cW0Jl5a3V mB9ticL0FbNZKCeH+kC8Jip0x1o2+UT4U99SB5AdaiUiA8aWE62VZF3fC7/76fGE3e2q S9/bnpVxqJxhTiGM8i1VbbGoYgReuMjIQJ8Q1nvZmHst6B1jIxhr4MAh7RM/NLXll839 Lm1YIhpjWs7cuwIa96zr+MTv1d4e43jD8TdMVjlP088SLe6lOR8cOxUhovmT4dGBek4M V5bQ== X-Forwarded-Encrypted: i=1; AHgh+RqmcSQiWXVc/o37q0LYsImI6d3+jjDPPwYNz96vnX94zXvx+P6D3sZLpsv0he4kfLq6hx2vP0ceB72L3A4=@vger.kernel.org X-Gm-Message-State: AOJu0YyjKICJUW+UzB+/tUBkiGa4JHZjISXtN+HCmRrQ37TlxZzJQakT Lvsd5Go7ugVGNsbjWO6bXSCMzW5zpKLNeMZF+PTwYQWT0aZZigF3Wqu74DTA4r86jls= X-Gm-Gg: AfdE7cnH5o4kOf/ekc3I1N/y/FUf0FCTeLow4lYbVfbt8WkM8xhmv/OrYonHr02ICAY netttK90sWs0ADu077JA0TSp0Dw5fvr464J8M3Frnsmlo7nhVB5RqjsO9CZrLNvpTcUOOVOK+ZY 7Um9CMNeCMT4R6gLyi5uMjvfQ0I31frySoWG+jIqBiFlD8dOoWEyG3UhkVmAF7OCqNU7+Y9PgTA 0jXzSgYkG8T+ggpwNkjY2lZz517O10W+oz6cdL2de+VyZP/L7BCgex56BV4UqtS+nCbJkcMJ1W6 bksNlTrkDYv1FFo9uyxBSzc1MktMT7nIcfSlTz7bov/mgwJm4M112DSuI/2PQAwZKDOrbQZGHNz LK7m/OdPMk06Tm4mrt/LSM7r2s0beH8XS+Xo8tM366+NldL81snoOMWwgmK0vtitVxoH8CQoAgn 5mrIFOrLDaKhD4oA6K/bOz86o= X-Received: by 2002:a05:6a21:114c:b0:3b4:b275:c5b1 with SMTP id adf61e73a8af0-3c36c1cf118mr8945730637.26.1784205880687; Thu, 16 Jul 2026 05:44:40 -0700 (PDT) Received: from L6YN4KR4K9.bytedance.net ([2408:8606:1800:501::2:f]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-cb258eeaba2sm1730793a12.12.2026.07.16.05.44.32 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 16 Jul 2026 05:44:39 -0700 (PDT) From: Yunhui Cui To: linux-riscv@lists.infradead.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: linux-arch@vger.kernel.org, kvm-riscv@lists.infradead.org, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-efi@vger.kernel.org, linux-perf-users@vger.kernel.org, kasan-dev@googlegroups.com, Yunhui Cui Subject: [PATCH v2 13/15] riscv/mm: remove redundant TLB flush in napotpte_convert Date: Thu, 16 Jul 2026 20:41:48 +0800 Message-Id: X-Mailer: git-send-email 2.39.2 (Apple Git-143) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" napotpte_convert() switches a NAPOT block between the Svnapot encoding and the equivalent set of base PTEs. The conversion itself preserves the effective translations: PFNs and permissions are kept the same while the page-table representation changes. Do not issue an SFENCE.VMA in the middle of this representation change. For partial updates, such as unfolding a writable NAPOT block before write-protecting one subrange, a flush at conversion time would not be the operation that enforces the final permission downgrade. Writable base-PTE translations could still be refilled after the conversion and before the subrange is write-protected. The required TLB maintenance belongs to the operation that actually changes the effective mapping. The RISC-V privileged architecture discussion clarifies that Svnapot is an encoding of a naturally aligned range. When napotpte_convert() only switches between the NAPOT encoding and the corresponding set of base PTEs for the same effective translations, the conversion itself does not require an extra TLB invalidation beyond the usual page-table update rules. Link: https://lists.riscv.org/g/tech-privileged/message/2950 Signed-off-by: Yunhui Cui --- arch/riscv/mm/contpte.c | 14 ++++++++++---- 1 file changed, 10 insertions(+), 4 deletions(-) diff --git a/arch/riscv/mm/contpte.c b/arch/riscv/mm/contpte.c index b0b8ff0aade3a..8289f39d6e188 100644 --- a/arch/riscv/mm/contpte.c +++ b/arch/riscv/mm/contpte.c @@ -129,7 +129,7 @@ static pte_t __get_and_clear_full_ptes(struct mm_struct= *mm, static void napotpte_convert(struct mm_struct *mm, unsigned long addr, pte_t *ptep, pte_t target) { - unsigned long start_addr, end, ptent_addr; + unsigned long start_addr, ptent_addr; pte_t *start_ptep; pte_t ptent, pte; unsigned int i, nr; @@ -137,7 +137,6 @@ static void napotpte_convert(struct mm_struct *mm, unsi= gned long addr, start_addr =3D napot_align_addr(addr); start_ptep =3D napot_align_ptep(ptep); nr =3D napotpte_pte_num(); - end =3D start_addr + napotpte_size(); =20 for (i =3D 0; i < nr; i++) { ptent_addr =3D start_addr + i * PAGE_SIZE; @@ -150,8 +149,15 @@ static void napotpte_convert(struct mm_struct *mm, uns= igned long addr, target =3D pte_mkyoung(target); } =20 - flush_tlb_mm_range(mm, start_addr, end, PAGE_SIZE); - + /* + * This conversion only changes the representation of the same + * effective translations: NAPOT encoding <-> equivalent base PTEs. + * A flush here would not replace the TLB maintenance required by a + * later permission downgrade, because writable base-PTE translations + * could be refilled after this conversion and before that downgrade. + * Leave TLB invalidation to the operation that changes the effective + * mapping. + */ page_table_check_ptes_set(mm, start_addr, start_ptep, target, nr); if (pte_napot(target)) { for (i =3D 0; i < nr; i++) --=20 2.39.5 From nobody Sat Jul 25 15:51:08 2026 Received: from mail-pg1-f172.google.com (mail-pg1-f172.google.com [209.85.215.172]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D8B7C42048B for ; Thu, 16 Jul 2026 12:44:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784205891; cv=none; b=mzGlNPW8bDqVrYrxN/canAKmeBk3KpX1w0NHXM7o7QolqSxvDYqnE9js6+MM/0BHMDfhQVW7I4NdhiKaPvR3HdgTToUPOm2SwLoYNbc9VQlN/Xf+LAe7prw5SMHN0rAdUZ/TCJKz5WsxkSIac5htXciVI1YwdMuWpjEY9n8qRx8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784205891; c=relaxed/simple; bh=8QB7Nm89twoLMmPuv/JvcX/X4fT+yA9XR/MJbmS0oGI=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=NcQTmR2EErpIGyME4dGpEJrB2FgVYwo6Xy/J2HUwz4L4Zo6demH3aGZroy9091N8XXrMw0GOnWxBg/YvMZslU5UO4ahcaS+ZEy8iBfXB3o1tUI4N4hjyn8s9EaDKXQpVwKZmh+nQwy3/HvyZ2kFUtMXIh+dLfVHxsRrIggUH+EU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=EN4Ycm9g; arc=none smtp.client-ip=209.85.215.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="EN4Ycm9g" Received: by mail-pg1-f172.google.com with SMTP id 41be03b00d2f7-c9d1fc053e0so2849515a12.1 for ; Thu, 16 Jul 2026 05:44:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1784205889; x=1784810689; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=P62EVeZeD6Lr+EK73ByExkJlzskm2POoZpW1kwtVtAQ=; b=EN4Ycm9g0slYOsJopCov6pgza00nzUbxRgBoSu5NYgP8H4wRZnzQbKMJBIslkss026 CVeC+X3pJWuKv/yminjErCaOhH78K6kiunf6OnjQjVAsKOCbKm6IEsQ/kylcQ8hrq1hF TQgVMzZ9ISmKRUR4YX10lCclrIlonYJJSTHXl/1nD7xQhg+X5X3/CyDSzMwTT9YnK70o e3EVd2d8ftse+yNHg1yP1cDp5BJt8wO2yzuFgaqtqKqWSblQF1hiMfwB5kfAX3cv/Vil x7u0r2f0Az9SkEk9j0VZ44ODMu04mq4YzOH69cX6U+QpvOGEIgRGkU+ZGBxSRtZi3Jmx gdNw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1784205889; x=1784810689; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=P62EVeZeD6Lr+EK73ByExkJlzskm2POoZpW1kwtVtAQ=; b=NIdK+EVzqdmkmURxbB+PTowZJUrdo804mtgUrVcG8SqXJW8BHQLSuppDBFHUAaHBN0 RWF3emyHBS7uOyhm0vP6ogcUP+MP2iGic3DHsJbUp0Vt0//5vip52rillfsxKzCPdmBX qiIehsEwdJ3aXmFJ5fKZV8F1nYHBLxtdIMjZui7QtIUP9AWeabj/hsh3NwxXse/6YMoU jPJHSDNDyAY2r35MLHRj4Gl3bnX7exFOSoZ/cF3JP/g5UkKZIH7rt/7373MnNNAcq/P4 yPyeC0AtLafaIzAOS8bwgAQjS3i6cebHodnx0NCDRE/iZM/McOFg56+KyECnCs+UVUJj xh1g== X-Forwarded-Encrypted: i=1; AHgh+Rr2VL1bZPzWwpjvOIV5tP8YHBiMJhUTFaG7+C+3xZzUs9LziciXkOgjRW70WSf5s7ljKnhO7QskKx9PmU4=@vger.kernel.org X-Gm-Message-State: AOJu0Yz6uNUlCuIqJ/1HNK7D1PsvFOfT146LQ6fuWDHsZRxQFLMIaCmz lUycMh3Dmn2JCDkJja6MoUtpq1N4sGlFgYCD7Kgx7PTCMs4QlAbeUUlc4lDN1QLRiyQ= X-Gm-Gg: AfdE7cnTSLfKJlnRAqHDMCIeSXz1pLD0UdfdDcJxsUOkTno7LLUdyLeNCY2BCT86Ykt wNL51aSkemox8D8fI0C3XVcmUG7w45shziR3Flph/e7e8Amo2ZOnhy6RiX3eURhX8XqFEHTRqUp tusYFVrkZSrP8pmvDws9W28Y1U48x92mLTX11cfLgOsKxAGi+Yrwl+zhyyaNcrqX4dDcmtntcc4 W9KXx5ONmEj3EtxGafNJb9CWrBUqY4r7uTgr/lrkv0iWSvIwcQBtBKoMGhmctpN4Ku1mvBRibA1 lUkV4VJtSSRmStZhJFPxpJ/RnKaDSTNoBWZjHcJywA9ctnQaGqmx3jfTL17D+mO2u4+yiGaR1BT HHVbK7aM6HPVyDXI94Bu1PGVeTguUTS/Thks1D/b7y6Ndo29s46yznUZc3YRkgRRAYtnZdfOSfI jKRLw9YYMRiH/6A9RqRrmtdbs= X-Received: by 2002:a05:6a21:7117:b0:3c3:89ce:b5bc with SMTP id adf61e73a8af0-3c389ced5b6mr3647677637.15.1784205889298; Thu, 16 Jul 2026 05:44:49 -0700 (PDT) Received: from L6YN4KR4K9.bytedance.net ([2408:8606:1800:501::2:f]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-cb258eeaba2sm1730793a12.12.2026.07.16.05.44.41 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 16 Jul 2026 05:44:48 -0700 (PDT) From: Yunhui Cui To: linux-riscv@lists.infradead.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: linux-arch@vger.kernel.org, kvm-riscv@lists.infradead.org, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-efi@vger.kernel.org, linux-perf-users@vger.kernel.org, kasan-dev@googlegroups.com, Yunhui Cui Subject: [PATCH v2 14/15] riscv/mm: optimize mprotect for Svnapot mappings Date: Thu, 16 Jul 2026 20:41:49 +0800 Message-Id: <8b79ddc3311e1dc32b150b7efbaed5a187ee5ccd.1784201104.git.cuiyunhui@bytedance.com> X-Mailer: git-send-email 2.39.2 (Apple Git-143) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Svnapot represents a naturally aligned range as a block of base PTEs, but generic mprotect() still processes a full block one base PTE at a time. For a full-block permission change, that repeatedly clears and reinstalls the same logical mapping and then has to reconstruct the folded Svnapot block afterwards, making the operation unnecessarily expensive. Use the batched protection transaction hooks to handle a full Svnapot block as one operation. The fast path is restricted to user mappings where hardware Svnapot is supported, the range is exactly one aligned Svnapot block, and every sub-PTE is consistent with the block PTE. The generic per-PTE fallback remains in place for partial, unaligned, special, non-user, or otherwise inconsistent mappings. libMicro mprotect median latency improves on the full-block cases: mprot_tw4m (-l 4m -w -t -f /dev/zero): 40.4 us -> 4.3 us mprot_tw128k (-l 128k -w -t -f /dev/zero): about 15-16% faster Signed-off-by: Yunhui Cui --- arch/riscv/include/asm/pgtable.h | 8 ++++ arch/riscv/mm/contpte.c | 76 ++++++++++++++++++++++++++++++++ 2 files changed, 84 insertions(+) diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgta= ble.h index 3727c148fa0fc..b546e6614c558 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -619,6 +619,14 @@ int napotpte_ptep_test_and_clear_young(struct vm_area_= struct *vma, unsigned long address, pte_t *ptep); int napotpte_ptep_clear_flush_young(struct vm_area_struct *vma, unsigned long address, pte_t *ptep); +#define modify_prot_start_ptes modify_prot_start_ptes +pte_t modify_prot_start_ptes(struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep, + unsigned int nr); +#define modify_prot_commit_ptes modify_prot_commit_ptes +void modify_prot_commit_ptes(struct vm_area_struct *vma, unsigned long add= r, + pte_t *ptep, pte_t old_pte, pte_t pte, + unsigned int nr); #endif =20 #ifdef CONFIG_ARCH_HAS_PTE_PROTNONE diff --git a/arch/riscv/mm/contpte.c b/arch/riscv/mm/contpte.c index 8289f39d6e188..6a1156629d115 100644 --- a/arch/riscv/mm/contpte.c +++ b/arch/riscv/mm/contpte.c @@ -190,6 +190,82 @@ napotpte_is_batch_consistent(pte_t pte, pte_t batch_pt= e, fpb_t flags) pte_val(batch_pte); } =20 +static bool napotpte_can_modify_prot_block(struct mm_struct *mm, + unsigned long addr, pte_t *ptep, + unsigned int nr, pte_t *raw_pte) +{ + pte_t raw, pte; + unsigned int i; + + if (!napot_hw_supported() || !mm_is_user(mm)) + return false; + + if (nr !=3D napotpte_pte_num()) + return false; + + if (addr !=3D napot_align_addr(addr) || ptep !=3D napot_align_ptep(ptep)) + return false; + + raw =3D READ_ONCE(*ptep); + if (!pte_present_napot(raw) || pte_special(raw)) + return false; + + for (i =3D 0; i < nr; i++) { + pte =3D READ_ONCE(ptep[i]); + if (!napotpte_is_consistent(pte, raw)) + return false; + } + + *raw_pte =3D raw; + return true; +} + +pte_t modify_prot_start_ptes(struct vm_area_struct *vma, unsigned long add= r, + pte_t *ptep, unsigned int nr) +{ + struct mm_struct *mm =3D vma->vm_mm; + unsigned long ptent_addr; + pte_t raw, pte, ptent; + unsigned int i; + + if (napotpte_can_modify_prot_block(mm, addr, ptep, nr, &raw)) { + pte =3D pte_mknonnapot(raw, addr); + + for (i =3D 0; i < nr; i++) { + ptent_addr =3D addr + i * PAGE_SIZE; + ptent =3D __ptep_get_and_clear_noptc(ptep + i); + page_table_check_pte_clear(mm, ptent_addr, + pte_mknonnapot(ptent, ptent_addr)); + if (pte_dirty(ptent)) + pte =3D riscv_pte_mkhwdirty(pte); + if (pte_young(ptent)) + pte =3D pte_mkyoung(pte); + } + + return pte; + } + + pte =3D __ptep_modify_prot_start(vma, addr, ptep); + while (--nr) { + ptep++; + addr +=3D PAGE_SIZE; + ptent =3D __ptep_modify_prot_start(vma, addr, ptep); + if (pte_dirty(ptent)) + pte =3D pte_mkdirty(pte); + if (pte_young(ptent)) + pte =3D pte_mkyoung(pte); + } + + return pte; +} + +void modify_prot_commit_ptes(struct vm_area_struct *vma, unsigned long add= r, + pte_t *ptep, pte_t old_pte, pte_t pte, + unsigned int nr) +{ + set_ptes(vma->vm_mm, addr, ptep, pte, nr); +} + static inline pte_t napotpte_normalize_batch_pte(pte_t *ptep, pte_t orig_pte, fpb_t flags) { --=20 2.39.5 From nobody Sat Jul 25 15:51:08 2026 Received: from mail-pl1-f174.google.com (mail-pl1-f174.google.com [209.85.214.174]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CE14141F5F5 for ; Thu, 16 Jul 2026 12:44:58 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.174 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784205900; cv=none; b=obrbAWc4OOBjoASqO/IHmJZp+bj5/M5a1jSPx8cekaVPjTV/XuYYJIiu8KXVhKLUmbcOEyoPTGvqrENigv0LozUBQc3+hjzSLYbPScfqS6l5cpthnR+TpXIF/k9v756thEOjlvhkFsevCylNGSxgGXKzq67bKV+cwtnG7kSbGaY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784205900; c=relaxed/simple; bh=ybAp/Hmnc3TAueG66xnodqF9+Bp3yZdy6u6JbQGTD4E=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=i967Y7jEv4q4ZJBLhV5I3PeQd4u2hX04WOpvZgaY3pe1CsVc0ZhWN41x+z0coK00m2q208Oy9Ids7Tolmh4RTapYWIxHd075bWP7e8sGF1euap2MR5JARl/+365x8F4G2xy2ydF24O161AMsVK7npBYktcbPkAULa1VcsJ50iPk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=MUbSTRUR; arc=none smtp.client-ip=209.85.214.174 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="MUbSTRUR" Received: by mail-pl1-f174.google.com with SMTP id d9443c01a7336-2cc891373e0so77791435ad.2 for ; Thu, 16 Jul 2026 05:44:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1784205898; x=1784810698; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=igNuDXnwdUCT2dm/vIcI/M9kQJ6fMwWaxma7wgKaCpI=; b=MUbSTRURVANI8QAKE7ELCYR0rI7ExejRVAjeV9XMDIhl5okUOjsWO4JBdblNUhYmtH ppZCrNRxUk/330nLIPhHo+fI5J7cJCukbhaGP807E9LqxMy/bkaVJlicpb5Hjf8yXYl+ PTpc2IF71XhO4uJ5QPFicMgiSMYwV3BPjtsF6IntBi05yOsDRQJq8vh0d2fD+NxHLidh wqbdlz3QrVgO2dw/WO7ATLh3cBaJ5YUZHLaM0eGXOPLxS3VCeRW3IVMv/kiull43rv0F UqXNjc7ykwlt10+k986cuFuk1GZ6hnGni/37BVT4BjOQxTvkEWmsyFXXWca6MZ4ce/v3 Cc7w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1784205898; x=1784810698; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=igNuDXnwdUCT2dm/vIcI/M9kQJ6fMwWaxma7wgKaCpI=; b=bDXr04hRt0b4iFzu6JcjsqvVJzHMT5YBkFO3wB9B0aOmg6jKmmtl9zOEqCPK4JZlHE fShPK6fSX5GyNfzi65eVRaesWb/F5kTdEjcLHgZMoRWRY+LgRsPqR9bHMichanHqRfPc 9UDj9xd2KmB6jmicOGxd46cnrS8/Rs6qamI1vV7wmUP9XcMmi2zo75flZa8xXjh6gWXG dseFH9TQDtAAwqAH5rtvaYtcB/ImXmEmYu+9CSV3EglnB/lPnhUSDNt8RuCZk1kkILMP Uh8Ge7g+DfEWbQXcGVI2IPX9JksOwxRzE8DCQAR9n96I2x4VyWNRUaIj9hb+JGmIzJ4v bseg== X-Forwarded-Encrypted: i=1; AHgh+RqSBYd0VGx3te+5tp1+Lh/8By5zwiVKgWt9SBVUZky4VDJ3W0oICOoXvdg1vx9odyYys5yMKrH3U4gE73g=@vger.kernel.org X-Gm-Message-State: AOJu0YwPX0QfrXKQnhgc8nfk1tKzIZ1R03742l4kc2knf3pDTwTUtxHp ZftTq2bX7u1tdKngM0WwhFjINJqva3vR68SDhxng1eg0SAJmAZiqwE3yHw8C7JmU5+4= X-Gm-Gg: AfdE7ck2o8CEATIcNfIIvAo5Xm2qxYZ4QM4mVrX33agW/FkTsN9p9EvHDBVJSD/sURP bZ44Apuu4YQM61BNmg8fjsgGqPW2efiXCg1Mux8Y/wHZI9E26WTcAnMUCqNi/mMGVt6km3g8JRG /UNg0QjFae+P8cx1etCU5WV1CQU0bjvqrmVI9TcaSaId76dihJH/e/qAC4vEbXr2GNsndYQct30 /JF2/F6EPajdV+RWNQfVQuJ/C8pV4Rbd5D2JMexbRl54KiYdV99cSIJbqIZkrzCuenh+CBUlKot p4ayCZQ43cCehQE+T8wwkwX9o2QHyNXgpstFxMcZukmwDUDlCoWrow+CgqiAqCvKNvDBKCMoVnH n8k4/KDpckv+gpJUP08jhtsOlBF3SUwjR39ZjS2TdTKEwOjAbS2Toxs/cPdeeJw5vmfQf48bbhJ 3grksoLHxkDptJOnaMWg8Yzx8= X-Received: by 2002:a05:6a21:7318:b0:3bf:a881:4094 with SMTP id adf61e73a8af0-3c36c35d245mr7182183637.50.1784205898111; Thu, 16 Jul 2026 05:44:58 -0700 (PDT) Received: from L6YN4KR4K9.bytedance.net ([2408:8606:1800:501::2:f]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-cb258eeaba2sm1730793a12.12.2026.07.16.05.44.49 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 16 Jul 2026 05:44:57 -0700 (PDT) From: Yunhui Cui To: linux-riscv@lists.infradead.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: linux-arch@vger.kernel.org, kvm-riscv@lists.infradead.org, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-efi@vger.kernel.org, linux-perf-users@vger.kernel.org, kasan-dev@googlegroups.com, Yunhui Cui Subject: [PATCH v2 15/15] riscv: mm: Request large exec folios for Svnapot Date: Thu, 16 Jul 2026 20:41:50 +0800 Message-Id: <88438f09b79ceab1548aff17384814b7f699a43d.1784201104.git.cuiyunhui@bytedance.com> X-Mailer: git-send-email 2.39.2 (Apple Git-143) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The Svnapot MM support in this series allows generic MM paths to operate on folded 64KB executable mappings while preserving a per-page public PTE view. To benefit from that more often, executable file-backed mappings should be faulted in with a folio size that can be folded into a Svnapot range. Request 64KB executable folios when Svnapot is available, so executable text mappings are more likely to be installed as folded Svnapot blocks and benefit from lower iTLB pressure. Keep the default order-0 behavior when Svnapot is unavailable. Signed-off-by: Yunhui Cui --- arch/riscv/include/asm/pgtable.h | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgta= ble.h index b546e6614c558..6801ba135c483 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -296,6 +296,14 @@ static __always_inline bool has_svnapot(void) return riscv_has_extension_likely(RISCV_ISA_EXT_SVNAPOT); } =20 +/* + * Request executable file-backed mappings to be read into 64KB folios when + * Svnapot is available. A naturally aligned 64KB folio can be folded into= a + * Svnapot PTE range, reducing iTLB pressure for executable text. + */ +#define exec_folio_order() \ + (has_svnapot() ? NAPOT_CONT64KB_ORDER : 0) + static inline unsigned long pte_napot(pte_t pte) { return pte_val(pte) & _PAGE_NAPOT; --=20 2.39.5