From nobody Wed Dec 17 15:51:21 2025 Received: from mail-pl1-f178.google.com (mail-pl1-f178.google.com [209.85.214.178]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CE95B1A840F for ; Wed, 8 Jan 2025 06:59:31 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.178 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1736319573; cv=none; b=MxEEUUSzYEjhxpvE9FxbJIC2rgvDEuU2lZQ/W+YbcxY9t5HLdYsZK7F3VKORIrIAo++hduxHUneqeAsnepy3UCOlBzGIKBJXmZ5FoHjs69zaEfB1v5H3gCJpoQlBXtm1i+MUIsCyHNjo4+EFgInaMQM8cGHpqKyFYRWiX03Ua7w= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1736319573; c=relaxed/simple; bh=AE6ZwrHtnhWSEtac69meSe/nEVBLF2lFdOsyctg6W6Y=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=ab57qDgY9DHaLbAOVG/PnT2FfTllFzibHsvga2WMYuRJ7jfbRJVDu1UL6LaNHpftMN8fFVU2ZlnpSbGG4JjHRWs7+JsAqzd0SeFCKmuAvkXGCAnvL0RmcPuWvfHtu5jR0e0I9MBN0Fkm1usuzldZYsoJtVtHn25vVu44QfdvB20= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=MReWuM1F; arc=none smtp.client-ip=209.85.214.178 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="MReWuM1F" Received: by mail-pl1-f178.google.com with SMTP id d9443c01a7336-21649a7bcdcso234357595ad.1 for ; Tue, 07 Jan 2025 22:59:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1736319571; x=1736924371; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=qK7UmD8A9xmARwj2p/zsqM766rIMmkR9OD65YlbhPf0=; b=MReWuM1FcU+WcynwHHWF/bym8Ucfejx+42G3U6wV4nAM/MLkFLKZo6rdMdlyOKhuJW JnqvqvSSnOfGtHfiErot7QdSnSiQsiO1QXEO7YS13NP4OdN41iypLNx9eFqIbODuFut8 eLUn5tpPrCQKvLtp0Vab0ZSl9cAnJJOZy23OtgN8taFH/JNv4lEhCdfJN4VarCXdZhil GF6jd41/5A/Ku/cptyN+Mv3xT6k4+t78tst1wKWY7nJDrnuV7WH1Rxusvakk9EwrSq69 qMq31F5AaHIatuxhGD3MDUmvfN58mxzY0ulFnb+KNfKhWZuWqyhv2AA8JE+Sdb2U7t/D BfcQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736319571; x=1736924371; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=qK7UmD8A9xmARwj2p/zsqM766rIMmkR9OD65YlbhPf0=; b=Q3OKUD/B6zG6P/c3YLNKDgBB/FLltp/FV6FZg2kuwPCs2VydpvlmWXJXpR2cI7KreH Qp+o2tZ4tV4H8McuXJcYtATv5eKXLZ5M94u9tI+nYTeYNd9egpkWCPYqnQFM5N18thvx nYC6pKLq4vmzS2lTLNySBy7nIemVCBjsVdnI+m5rVHgbxOliaa8BT4aDpeZ9zqdVM+2c VISQa5sqYLXXf9yTt5MihZ3w+sfx70C3+HGfyR0LYszKwB1QoNOTXlgCqhoko1y03khR Ew5xpJCsk/kuUSPfZs/HEAk6gczVYfE3DBDFQSe/lC23AZCpZhoZ54K0pI7yb/VtSy0K EsYg== X-Forwarded-Encrypted: i=1; AJvYcCWqIKrNXX1rKMDFD3zQbL0tgqwPVte6UKGBTsrpbqF716vbrXnzZEfFEooZhZPxxoiZmUzEgd7iAJikTK0=@vger.kernel.org X-Gm-Message-State: AOJu0Yx/p2Zl3lhWNYuc1x5NsTonZzf91iqhzjhpu6wRnSVqjfqJ8Chr Qv316DeWympTmnRKWbxtrCSjsqZHCRBa+0B8RwOmzWzkwl21SkpbOm6cVJL9lZ0= X-Gm-Gg: ASbGncvA2qIK8bssyrFZRxNsQA7J3GqeFLKIw/ebeA8gRCS5J31RJuPYBmvhoVclHIj 775EZZJZfTWWILpHgL+YlzCzIK3G095uKA4bz8ngkYHXUQJNUQHSMYxn+fj1D8nkf30Xc8dsRZ9 NnzGjn5LIpBpmQOpz53sUS6eST/Lxn1bpsNlTP5xK2m1Vgn4JFyXAtmQ8GT+KKMxZ2wBMJ814Tx aIVMULju72MxVDlCRpQPRiOqhPCEB82vQLWpCXI7TM/FfCidrTG3G7ULQxoJmpal9e67kSWKLxv hDUxZOsVpDon8Hfo5jzH+gneytE= X-Google-Smtp-Source: AGHT+IHPv/02/1SYwL+5wvpHkGRfZSWRMZ218tgsJxwry8Gh8x8rNq2cUbRiuSy0SgfT7Mb+CNVIfg== X-Received: by 2002:a17:903:186:b0:216:34e5:6e49 with SMTP id d9443c01a7336-21a84001250mr22590365ad.57.1736319571238; Tue, 07 Jan 2025 22:59:31 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.244]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-219dca023a3sm320067275ad.250.2025.01.07.22.59.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 07 Jan 2025 22:59:30 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, agordeev@linux.ibm.com, kevin.brodsky@arm.com, alex@ghiti.fr, andreas@gaisler.com, palmer@dabbelt.com, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com, arnd@arndb.de, will@kernel.org, aneesh.kumar@kernel.org, npiggin@gmail.com, dave.hansen@linux.intel.com, rppt@kernel.org, ryan.roberts@arm.com Cc: linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, linux-arch@vger.kernel.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, loongarch@lists.linux.dev, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-sh@vger.kernel.org, linux-um@lists.infradead.org, Qi Zheng Subject: [PATCH v5 01/17] Revert "mm: pgtable: make ptlock be freed by RCU" Date: Wed, 8 Jan 2025 14:57:17 +0800 Message-Id: <366002e0af83f0d5cad3f356db036cb6447492f7.1736317725.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This reverts commit 2f3443770437e49abc39af26962d293851cbab6d. Signed-off-by: Qi Zheng Reviewed-by: Kevin Brodsky --- include/linux/mm.h | 2 +- include/linux/mm_types.h | 9 +-------- mm/memory.c | 22 ++++++---------------- 3 files changed, 8 insertions(+), 25 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index d61b9c7a3a7b0..c49bc7b764535 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2925,7 +2925,7 @@ void ptlock_free(struct ptdesc *ptdesc); =20 static inline spinlock_t *ptlock_ptr(struct ptdesc *ptdesc) { - return &(ptdesc->ptl->ptl); + return ptdesc->ptl; } #else /* ALLOC_SPLIT_PTLOCKS */ static inline void ptlock_cache_init(void) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 90ab8293d714a..6b27db7f94963 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -434,13 +434,6 @@ FOLIO_MATCH(flags, _flags_2a); FOLIO_MATCH(compound_head, _head_2a); #undef FOLIO_MATCH =20 -#if ALLOC_SPLIT_PTLOCKS -struct pt_lock { - spinlock_t ptl; - struct rcu_head rcu; -}; -#endif - /** * struct ptdesc - Memory descriptor for page tables. * @__page_flags: Same as page flags. Powerpc only. @@ -489,7 +482,7 @@ struct ptdesc { union { unsigned long _pt_pad_2; #if ALLOC_SPLIT_PTLOCKS - struct pt_lock *ptl; + spinlock_t *ptl; #else spinlock_t ptl; #endif diff --git a/mm/memory.c b/mm/memory.c index b9b05c3f93f11..9423967b24180 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -7034,34 +7034,24 @@ static struct kmem_cache *page_ptl_cachep; =20 void __init ptlock_cache_init(void) { - page_ptl_cachep =3D kmem_cache_create("page->ptl", sizeof(struct pt_lock)= , 0, + page_ptl_cachep =3D kmem_cache_create("page->ptl", sizeof(spinlock_t), 0, SLAB_PANIC, NULL); } =20 bool ptlock_alloc(struct ptdesc *ptdesc) { - struct pt_lock *pt_lock; + spinlock_t *ptl; =20 - pt_lock =3D kmem_cache_alloc(page_ptl_cachep, GFP_KERNEL); - if (!pt_lock) + ptl =3D kmem_cache_alloc(page_ptl_cachep, GFP_KERNEL); + if (!ptl) return false; - ptdesc->ptl =3D pt_lock; + ptdesc->ptl =3D ptl; return true; } =20 -static void ptlock_free_rcu(struct rcu_head *head) -{ - struct pt_lock *pt_lock; - - pt_lock =3D container_of(head, struct pt_lock, rcu); - kmem_cache_free(page_ptl_cachep, pt_lock); -} - void ptlock_free(struct ptdesc *ptdesc) { - struct pt_lock *pt_lock =3D ptdesc->ptl; - - call_rcu(&pt_lock->rcu, ptlock_free_rcu); + kmem_cache_free(page_ptl_cachep, ptdesc->ptl); } #endif =20 --=20 2.20.1 From nobody Wed Dec 17 15:51:21 2025 Received: from mail-pl1-f170.google.com (mail-pl1-f170.google.com [209.85.214.170]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0A5EC1A9B38 for ; Wed, 8 Jan 2025 06:59:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.170 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1736319588; cv=none; b=nbX42URmFPyu7cKnCFM4VJ2oiMrVmfwFDSqvCnlx7eOsvXaQiiT+LTLyoUDAcxXnCozrtWIQNhmt4se0Kv4agQSVj2ca40FraWi3GXNKWJ3TwzpnWkHUm1T8eawiAVLQeCv910KozCEOVlSl2HH3+MmqMZUR7UQHHp7eL6vzuww= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1736319588; c=relaxed/simple; bh=ma5h64nM92pGVw8WsUJDDYGPNfXZFgLftucZEsbzZpw=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=IGK6gBUL6pOMdP4AUYxJ52glh490rbiwFwA1QvYih7CK13mPh6usEEsshlUM0MN/5FSvKqBtSdpCGX4I836npmnJ+UxagBuecjROebHkWyrHAWDkvDuEIznanajN3hl/MHNY6U+Mp21zyTUjveSU5q2coXebPCVuGfnUW0oQT1Q= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=lkjsf6mt; arc=none smtp.client-ip=209.85.214.170 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="lkjsf6mt" Received: by mail-pl1-f170.google.com with SMTP id d9443c01a7336-216395e151bso7101935ad.0 for ; Tue, 07 Jan 2025 22:59:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1736319586; x=1736924386; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=WgpljFzcR37DW+k0izImY02NXJU+2N4CamZhPf6BMmU=; b=lkjsf6mtx7iieDBNQgHtDsNwfH81+Ahx5znscEAY+uMyGeqVwDbxKilg3fHzJFKf7m yyZZ7rtjWp0OBVs0551xO1oX49k2Oy9EQMjn5paprIo8V/l6Yy5iT6s9+oCibH4HIooX 31HtOM0khddPN9cF0IM7JQ96jiBOW7NVDLoFWud+rNan94G1yv//xmjhloxmIF8XRxhm AEouloJS4x6ju49vfHa+FORb2R7BT80AgfdZjVlEvUA4SMjvzUUAK93CguXuZ6a55P1A ZALeoJVB36ib9+/NtHSGuMIWrm0PG9Xjbzz1sB9Ynbww8v5X85/P8EQZcFSLqhbymKos VSog== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736319586; x=1736924386; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=WgpljFzcR37DW+k0izImY02NXJU+2N4CamZhPf6BMmU=; b=dbi/xBTfySfw9pYIHM8ENoNJVDe4407rzUBrkWnrJXCXgrGUDT2fuDAczhT8AbkrB8 JQCKjsM7ewJ4X5jM2RaNb9trw6bqpRYmKKE7TnJn6r80AGnqJOccjQzqIeR1FWu7/7TT hsxdaC855NMGT3ZdcezfJcPT6kRd/NvR0J37VUFxXZIF5SzTsl6oshlHNjmZVIynrM4k EO/CYKvHJfDoFrPqI8TLQE/lbzNm8uSFCgG4I52y9euXp5i20MLJgOVXTmjSgwpuH1Ug j95MaWomjntTYJ5JFkMMTpQ5hDZ4aCT9P1NSp3ghB0TGQpmeriCccA66dzRuzKY11bP9 rEog== X-Forwarded-Encrypted: i=1; AJvYcCX2RzsAKBYLJmHtiqzzTGqoAMXfx8kSgf8XrknLEiZfin4c1q9Gjqis5RQ/RZH1a/upl4sJJVFnzZmka3s=@vger.kernel.org X-Gm-Message-State: AOJu0YywCkjyaeHWFcZmCvZp9zWN0kBx1ORJJU6nK4itNwl1Y/zI/1LI UIvGMhdiAcRzQKCkzj+doPZSGE6RmAiSK5HnKYwW0gxuCAQvzrO8M+MtMxSFPis= X-Gm-Gg: ASbGncvfiMY8digo57NgdH+HuhXbqWAlgO/oZcHORtx+e/CgNTn8KKVVGAHwgeWOabO DKIVaW4itHL2ChYXzQchBgtH2xj4qXDfZXHBc9zj7KhiLy9S6LdX69TvI3xaQ22n+CjuSy/nvSb hFAdF0wV8T5X7G5ykprSIzOCDh8EpchRpgSwnZQCt2K6RTF9gcKObsE7xClYUlTz0/s3zSQ6QHQ Yv50B2dy2EOl5Jcu/uSfmfdHUmXehJF9wT36zG1i9wyvW/HvTeF/ZFu7V07UWHQUu/HNq5GOBeq Vnwhz0KY6LNbQOqs7vZXYS/4uXU= X-Google-Smtp-Source: AGHT+IEXmx6aGyXUatDDUSC3BIwwvJRBe2xxgGmNdG6bwRG2a4u7PgHO01+qr8GDOf9tBGHx3RRkKg== X-Received: by 2002:a17:902:fc4d:b0:216:5db1:5dc1 with SMTP id d9443c01a7336-21a83bb9991mr32206305ad.1.1736319586298; Tue, 07 Jan 2025 22:59:46 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.244]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-219dca023a3sm320067275ad.250.2025.01.07.22.59.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 07 Jan 2025 22:59:45 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, agordeev@linux.ibm.com, kevin.brodsky@arm.com, alex@ghiti.fr, andreas@gaisler.com, palmer@dabbelt.com, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com, arnd@arndb.de, will@kernel.org, aneesh.kumar@kernel.org, npiggin@gmail.com, dave.hansen@linux.intel.com, rppt@kernel.org, ryan.roberts@arm.com Cc: linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, linux-arch@vger.kernel.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, loongarch@lists.linux.dev, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-sh@vger.kernel.org, linux-um@lists.infradead.org, Qi Zheng , Alexandre Ghiti Subject: [PATCH v5 02/17] riscv: mm: Skip pgtable level check in {pud,p4d}_alloc_one Date: Wed, 8 Jan 2025 14:57:18 +0800 Message-Id: <93a1c6bddc0ded9f1a9f15658c1e4af5c93d1194.1736317725.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Kevin Brodsky {pmd,pud,p4d}_alloc_one() is never called if the corresponding page table level is folded, as {pmd,pud,p4d}_alloc() already does the required check. We can therefore remove the runtime page table level checks in {pud,p4d}_alloc_one. The PUD helper becomes equivalent to the generic version, so we remove it altogether. This is consistent with the way arm64 and x86 handle this situation (runtime check in p4d_free() only). Signed-off-by: Kevin Brodsky Signed-off-by: Qi Zheng Acked-by: Dave Hansen Reviewed-by: Alexandre Ghiti --- arch/riscv/include/asm/pgalloc.h | 22 ++++------------------ 1 file changed, 4 insertions(+), 18 deletions(-) diff --git a/arch/riscv/include/asm/pgalloc.h b/arch/riscv/include/asm/pgal= loc.h index f52264304f772..8ad0bbe838a24 100644 --- a/arch/riscv/include/asm/pgalloc.h +++ b/arch/riscv/include/asm/pgalloc.h @@ -12,7 +12,6 @@ #include =20 #ifdef CONFIG_MMU -#define __HAVE_ARCH_PUD_ALLOC_ONE #define __HAVE_ARCH_PUD_FREE #include =20 @@ -88,15 +87,6 @@ static inline void pgd_populate_safe(struct mm_struct *m= m, pgd_t *pgd, } } =20 -#define pud_alloc_one pud_alloc_one -static inline pud_t *pud_alloc_one(struct mm_struct *mm, unsigned long add= r) -{ - if (pgtable_l4_enabled) - return __pud_alloc_one(mm, addr); - - return NULL; -} - #define pud_free pud_free static inline void pud_free(struct mm_struct *mm, pud_t *pud) { @@ -118,15 +108,11 @@ static inline void __pud_free_tlb(struct mmu_gather *= tlb, pud_t *pud, #define p4d_alloc_one p4d_alloc_one static inline p4d_t *p4d_alloc_one(struct mm_struct *mm, unsigned long add= r) { - if (pgtable_l5_enabled) { - gfp_t gfp =3D GFP_PGTABLE_USER; - - if (mm =3D=3D &init_mm) - gfp =3D GFP_PGTABLE_KERNEL; - return (p4d_t *)get_zeroed_page(gfp); - } + gfp_t gfp =3D GFP_PGTABLE_USER; =20 - return NULL; + if (mm =3D=3D &init_mm) + gfp =3D GFP_PGTABLE_KERNEL; + return (p4d_t *)get_zeroed_page(gfp); } =20 static inline void __p4d_free(struct mm_struct *mm, p4d_t *p4d) --=20 2.20.1 From nobody Wed Dec 17 15:51:21 2025 Received: from mail-pj1-f43.google.com (mail-pj1-f43.google.com [209.85.216.43]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4397D1AA7A9 for ; Wed, 8 Jan 2025 07:00:02 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.43 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1736319613; cv=none; b=DFFbBNf9yNuyLt+qakmvvIN+Z9UDgmqSsl/SCtUZC/Ms1+usSfOjNwtiVH8bIfDHMXw/0M+GQp20GNveFVkoXkWKyZ4aTMuV+RJlNv102OSV2ga0/wR8MvI1jUyLmOrZwIFC1NRgf9wkqKEEeQIHbwrGu7G5yqcEYW/p+kMlASU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1736319613; c=relaxed/simple; bh=VQrwZRR3rAwbwOGD5AjoKk2Z15695i3scVkWYvbG/zY=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=NfiipTUwow6x9Tz2ItN002eB925/Uygl9QR55Y/jqs8cnj21Nl4GJt6MBX6UqKBMHVmmiO3NrV1XdwPL9rqPtYaA3fK7cI5tdBevSLnAuHuBE4h2YTYtE40g/xVYmUSiGLQZY4uYFoNXcSB/rLzgR7U3siqpb9kv+Xv8iWLoLW8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=PwL4L98R; arc=none smtp.client-ip=209.85.216.43 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="PwL4L98R" Received: by mail-pj1-f43.google.com with SMTP id 98e67ed59e1d1-2f4409fc8fdso829795a91.1 for ; Tue, 07 Jan 2025 23:00:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1736319601; x=1736924401; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Hi4+N7L/R07Fq6g8WRpMxop4cD3t01a1BHl7awbkQjI=; b=PwL4L98Rbmpe1X6rM/dgxkxd+0TIxSikkfeVs12QRoRGeMcU8es8p7YfDdHNl+b7a6 k8bWiuo+jVC/JQwWbmDXAqm4yE6c61Q3aQ/nV81QlS+rl8uDZhI6Z+4hVP2oOjuxrRuV e2rKXwb2YKeeXfZkH37z5Z8VSzi80cOwt2dFim2CZ5o9aW0mhRS9K2ZS6uahokb/qNOb Crktu7QOEHRgYg9uAyn1RgYokeIt5EfKmpl0Aq+Iy13QG8MDbdQ6NRDXoS9du09CGR2G q6Bbk+EZScLOY4ri/jl7GYCPE+XWdQXXayaZNnYGWsdgbUwjOUKZ3AYYX6oGagpOyzs+ nbMA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736319601; x=1736924401; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Hi4+N7L/R07Fq6g8WRpMxop4cD3t01a1BHl7awbkQjI=; b=pOYUfgt2gFIOacA5RA8WqlzvadbAdp0C09Rp/qze7IaArXfmN8Wgojdr3s8dg+sq67 KSR/zdquvq9kEhL0YI9gH0y7GkjiKesvAxhxrycdT4exVLR93R0PKLSLWbhe64fzk9f9 EWucTVFZ+4kFsNUFZbGL6nOuBcS2ZeRdcofJOzn3QlZ5uDShN8z0fjN/lr6rEUOICshL weiHCi+ATwIA6ly7H5eiGxIBuAAlYgZEBsrmE1nHcIi41zAF90ZY9Q9CQmQwpvHQm8j+ Bt/8zNNQSQWO6XfWl69N++UJQCpRUoUmT0ezc7lOa3FyyFFJb71/HkFbpt85yYrfZHsO dKjw== X-Forwarded-Encrypted: i=1; AJvYcCWZiVRvZXABnjQ9VSZp3dbxcoFAjT40zzM6NQjj9nrGCQhQunmEmPxmabB7wGgOG7+vxsJR7g5+3vJoMbc=@vger.kernel.org X-Gm-Message-State: AOJu0YyX9WB9AW3BDnLeVbSjQ8Lsx9hILavD+XLgSsfWA+AXWUoGEYbr cgFO4R4h5UKgo5BEt8FSVEs/dVlZA9odxRa/9fMYM/Zopx9kMOXg8diljfy+eww= X-Gm-Gg: ASbGncvi3FI+7vYcuEANUI0T12KVFUBuOn5VOl7TZIMVcsTg60OceFmrohe+eOQOnVS JVGur3uZdVJIxOUXubcx+N+j1YbCdcXM53Mbb6Hiq8fL9tTRD0yjRxBV0qeBK844YJ6d+wCbQ+j 3zY3pS0BCRMBdRGoES4phVvESILB2OFj4qc+hBhUXhqkr9yh+vLUKmNnBkEp+kpVC5Q9ZdPtVIZ SaVoXLzC5Ju4XiJ7V3qv+RmyTPfFEwX5o+gve8xZkIG2cOf18n+y5u6B5lxyIf0FLQc6BPf5mcS Q8aAKveDaSNbwKjKbT2wniUGOnI= X-Google-Smtp-Source: AGHT+IGjcc9iZhJOiA9Wc8tHUVktzF7/LrG3apZd5+oOKb6c7hgnTezIflh22cvX8oiOIwiosByWAA== X-Received: by 2002:a17:90b:4d09:b0:2ef:2980:4411 with SMTP id 98e67ed59e1d1-2f53cbd9b2fmr9342892a91.9.1736319601534; Tue, 07 Jan 2025 23:00:01 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.244]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-219dca023a3sm320067275ad.250.2025.01.07.22.59.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 07 Jan 2025 23:00:01 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, agordeev@linux.ibm.com, kevin.brodsky@arm.com, alex@ghiti.fr, andreas@gaisler.com, palmer@dabbelt.com, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com, arnd@arndb.de, will@kernel.org, aneesh.kumar@kernel.org, npiggin@gmail.com, dave.hansen@linux.intel.com, rppt@kernel.org, ryan.roberts@arm.com Cc: linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, linux-arch@vger.kernel.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, loongarch@lists.linux.dev, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-sh@vger.kernel.org, linux-um@lists.infradead.org, Qi Zheng Subject: [PATCH v5 03/17] asm-generic: pgalloc: Provide generic p4d_{alloc_one,free} Date: Wed, 8 Jan 2025 14:57:19 +0800 Message-Id: <26d69c74a29183ecc335b9b407040d8e4cd70c6a.1736317725.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Kevin Brodsky Four architectures currently implement 5-level pgtables: arm64, riscv, x86 and s390. The first three have essentially the same implementation for p4d_alloc_one() and p4d_free(), so we've got an opportunity to reduce duplication like at the lower levels. Provide a generic version of p4d_alloc_one() and p4d_free(), and make use of it on those architectures. Their implementation is the same as at PUD level, except that p4d_free() performs a runtime check by calling mm_p4d_folded(). 5-level pgtables depend on a runtime-detected hardware feature on all supported architectures, so we might as well include this check in the generic implementation. No runtime check is required in p4d_alloc_one() as the top-level p4d_alloc() already does the required check. Signed-off-by: Kevin Brodsky Acked-by: Dave Hansen Signed-off-by: Qi Zheng --- arch/arm64/include/asm/pgalloc.h | 17 ------------ arch/riscv/include/asm/pgalloc.h | 23 ---------------- arch/x86/include/asm/pgalloc.h | 18 ------------- include/asm-generic/pgalloc.h | 45 ++++++++++++++++++++++++++++++++ 4 files changed, 45 insertions(+), 58 deletions(-) diff --git a/arch/arm64/include/asm/pgalloc.h b/arch/arm64/include/asm/pgal= loc.h index e75422864d1bd..2965f5a7e39e3 100644 --- a/arch/arm64/include/asm/pgalloc.h +++ b/arch/arm64/include/asm/pgalloc.h @@ -85,23 +85,6 @@ static inline void pgd_populate(struct mm_struct *mm, pg= d_t *pgdp, p4d_t *p4dp) __pgd_populate(pgdp, __pa(p4dp), pgdval); } =20 -static inline p4d_t *p4d_alloc_one(struct mm_struct *mm, unsigned long add= r) -{ - gfp_t gfp =3D GFP_PGTABLE_USER; - - if (mm =3D=3D &init_mm) - gfp =3D GFP_PGTABLE_KERNEL; - return (p4d_t *)get_zeroed_page(gfp); -} - -static inline void p4d_free(struct mm_struct *mm, p4d_t *p4d) -{ - if (!pgtable_l5_enabled()) - return; - BUG_ON((unsigned long)p4d & (PAGE_SIZE-1)); - free_page((unsigned long)p4d); -} - #define __p4d_free_tlb(tlb, p4d, addr) p4d_free((tlb)->mm, p4d) #else static inline void __pgd_populate(pgd_t *pgdp, phys_addr_t p4dp, pgdval_t = prot) diff --git a/arch/riscv/include/asm/pgalloc.h b/arch/riscv/include/asm/pgal= loc.h index 8ad0bbe838a24..551d614d3369c 100644 --- a/arch/riscv/include/asm/pgalloc.h +++ b/arch/riscv/include/asm/pgalloc.h @@ -105,29 +105,6 @@ static inline void __pud_free_tlb(struct mmu_gather *t= lb, pud_t *pud, } } =20 -#define p4d_alloc_one p4d_alloc_one -static inline p4d_t *p4d_alloc_one(struct mm_struct *mm, unsigned long add= r) -{ - gfp_t gfp =3D GFP_PGTABLE_USER; - - if (mm =3D=3D &init_mm) - gfp =3D GFP_PGTABLE_KERNEL; - return (p4d_t *)get_zeroed_page(gfp); -} - -static inline void __p4d_free(struct mm_struct *mm, p4d_t *p4d) -{ - BUG_ON((unsigned long)p4d & (PAGE_SIZE-1)); - free_page((unsigned long)p4d); -} - -#define p4d_free p4d_free -static inline void p4d_free(struct mm_struct *mm, p4d_t *p4d) -{ - if (pgtable_l5_enabled) - __p4d_free(mm, p4d); -} - static inline void __p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d, unsigned long addr) { diff --git a/arch/x86/include/asm/pgalloc.h b/arch/x86/include/asm/pgalloc.h index dcd836b59bebd..dd4841231bb9f 100644 --- a/arch/x86/include/asm/pgalloc.h +++ b/arch/x86/include/asm/pgalloc.h @@ -147,24 +147,6 @@ static inline void pgd_populate_safe(struct mm_struct = *mm, pgd_t *pgd, p4d_t *p4 set_pgd_safe(pgd, __pgd(_PAGE_TABLE | __pa(p4d))); } =20 -static inline p4d_t *p4d_alloc_one(struct mm_struct *mm, unsigned long add= r) -{ - gfp_t gfp =3D GFP_KERNEL_ACCOUNT; - - if (mm =3D=3D &init_mm) - gfp &=3D ~__GFP_ACCOUNT; - return (p4d_t *)get_zeroed_page(gfp); -} - -static inline void p4d_free(struct mm_struct *mm, p4d_t *p4d) -{ - if (!pgtable_l5_enabled()) - return; - - BUG_ON((unsigned long)p4d & (PAGE_SIZE-1)); - free_page((unsigned long)p4d); -} - extern void ___p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d); =20 static inline void __p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d, diff --git a/include/asm-generic/pgalloc.h b/include/asm-generic/pgalloc.h index 7c48f5fbf8aa7..59131629ac9cc 100644 --- a/include/asm-generic/pgalloc.h +++ b/include/asm-generic/pgalloc.h @@ -215,6 +215,51 @@ static inline void pud_free(struct mm_struct *mm, pud_= t *pud) =20 #endif /* CONFIG_PGTABLE_LEVELS > 3 */ =20 +#if CONFIG_PGTABLE_LEVELS > 4 + +static inline p4d_t *__p4d_alloc_one_noprof(struct mm_struct *mm, unsigned= long addr) +{ + gfp_t gfp =3D GFP_PGTABLE_USER; + struct ptdesc *ptdesc; + + if (mm =3D=3D &init_mm) + gfp =3D GFP_PGTABLE_KERNEL; + gfp &=3D ~__GFP_HIGHMEM; + + ptdesc =3D pagetable_alloc_noprof(gfp, 0); + if (!ptdesc) + return NULL; + + return ptdesc_address(ptdesc); +} +#define __p4d_alloc_one(...) alloc_hooks(__p4d_alloc_one_noprof(__VA_ARGS_= _)) + +#ifndef __HAVE_ARCH_P4D_ALLOC_ONE +static inline p4d_t *p4d_alloc_one_noprof(struct mm_struct *mm, unsigned l= ong addr) +{ + return __p4d_alloc_one_noprof(mm, addr); +} +#define p4d_alloc_one(...) alloc_hooks(p4d_alloc_one_noprof(__VA_ARGS__)) +#endif + +static inline void __p4d_free(struct mm_struct *mm, p4d_t *p4d) +{ + struct ptdesc *ptdesc =3D virt_to_ptdesc(p4d); + + BUG_ON((unsigned long)p4d & (PAGE_SIZE-1)); + pagetable_free(ptdesc); +} + +#ifndef __HAVE_ARCH_P4D_FREE +static inline void p4d_free(struct mm_struct *mm, p4d_t *p4d) +{ + if (!mm_p4d_folded(mm)) + __p4d_free(mm, p4d); +} +#endif + +#endif /* CONFIG_PGTABLE_LEVELS > 4 */ + #ifndef __HAVE_ARCH_PGD_FREE static inline void pgd_free(struct mm_struct *mm, pgd_t *pgd) { --=20 2.20.1 From nobody Wed Dec 17 15:51:21 2025 Received: from mail-pl1-f175.google.com (mail-pl1-f175.google.com [209.85.214.175]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 922C6199939 for ; Wed, 8 Jan 2025 07:00:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.175 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1736319623; cv=none; b=exNNLVYcOtB69p8P2PV2aiBJHJgzw+N2GNTTrZBU733/wU9DU4IaSsuGLy6v2hWqelF3fi60DqRW3SA9sRNbpy10kHRaHNMzdW4zw/8vZ2lfcoP1fTeL9QYAGYaJkXmXHm6+k+kBq8v7ofcN3URSF9q8ERPlpYf/68X/G4siqWU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1736319623; c=relaxed/simple; bh=e3kN1Di5Fcouv11d71USulaPcPOxTFiQrC41b2k4smg=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=GcHogE97zX1YXLFqj6e2t+ib32cqpOM6+JDlw2MO6fxdj1+xhi/cgDic8lBNBaVMMYsI4n8jZpAWBYg04dg51klEtx4ESXAmtyUJx5xKvUkieuMPxBPjht9H5Arg1QiAF4aWWHu+bHyC0XW92nITeKOvQRGV6oJIC4W3ja77YrI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=OPgXEBib; arc=none smtp.client-ip=209.85.214.175 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="OPgXEBib" Received: by mail-pl1-f175.google.com with SMTP id d9443c01a7336-21a7ed0155cso22821145ad.3 for ; Tue, 07 Jan 2025 23:00:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1736319619; x=1736924419; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=/cYfjcYM2i48jLO7NuvQJ4j04U6+7fBnrQjgIuvbLyg=; b=OPgXEBibu+bf+w1PQZU9CTvwiiDqbXH7bCsUbtvMlw8zPtCkSIFAawjvdGCb3QQKsM hkvF6T2/WWT5g24HcpHc5p3yvkjLwm/I30ck06Tv4B62TE5pF4epBrGbri+tK3FcY+lh v6odZaUX7Y2QYRCvoMnlF0mRc5N9lCL1OcP6XV6s4Hs+VhuA826McnUM7itxM7t7UFOM feTXCHOMCPKYF8zu+jnNZfc5nFmqa7pmOxItgYsleNvcSTb7gcDURL8jVIf6y0stT42U lL8rOatEdBu0B1rHQM3Fv6LaiFkrV2W1q0xfpwozSY4R+FrCMte28/DCD/DkYZWjEsfX +tMg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736319619; x=1736924419; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=/cYfjcYM2i48jLO7NuvQJ4j04U6+7fBnrQjgIuvbLyg=; b=UwzAKCaVDtIhopPsEKkqxCbESSLHgZp0uCKGukvU9th60dLIEe2xvp3SKbJilaLLrr 3lhcji/IuaZYkT4IsfvO/Da7U22dbQfbjdyBjJV+4+mlnSg7JuCtoJH3yD9CaHl/V0BQ enFIhsEyYE8a5x8199qK3+IRmocwxXYEjysGVdarBBccibl31j1L7OHQxK7rec+h3O7T G/L/Ejhqln/AV34uKrlJ3ymnjtlp7OOKXh0LV+fZ4xUwwNPcleP0AU82ZjZ+PiF886pQ PTf4s09sQy1lcv1/thcH2ADkk9d0kQG7J8ZhKDK4mStq2JqbTLokrGdWhfEGlCxvvbBg gx8w== X-Forwarded-Encrypted: i=1; AJvYcCXQ6k5t/k3jNS8fFr0NwRr0elKlH8vJXzng7pZJoYbXNE2WSKLTsFb71Of19iQxu5VZ0Hhb0Pc00DaeMuM=@vger.kernel.org X-Gm-Message-State: AOJu0Yy324Kcuo+OSh1+KdMbboXHtIc/EvYm+RUxzPiSFz82q4SU2Rt7 DzL+LPi7HwSKU3ePoQDmCr/lb2Kwi/HN7nZF8leNff0agcw0AqiWaid/UUC69E0= X-Gm-Gg: ASbGncvzqLUt78rDZ11Byu94cmeeYYpFDOU/xM5sJtWvEy+3dDIZbfDEeZVsgKI95OY HdDzBcjkCMQmcf5AB2nOrYvBNq8GABgL3JRbm5UGEmrD/B3BS1mPvGWAo9RpgXoRb90D4r7wWzZ r15hNj4gkzAIzR8SzpE/t8k7LXPpbSdMh8BoJcJ2LA/0maTYuMNHctKFUTyT/huTj7dg8p6x+jw vBOfbcGjXgRWsP853ezKVLRNmWS/C+4iGIzgKNeOsbXfOgDeq/dG2wOJJ1UtXjSoT9zs3dhl5Pv aYCM/96S49zc7D1XxyVl8mIGroc= X-Google-Smtp-Source: AGHT+IExFPt1bVFS1B+HfK/rni8xCakw0mpYmNpiHCGWtvviMVtfvk5MaDovSIdy5xSOdSeWSopXEQ== X-Received: by 2002:a17:903:2311:b0:210:fce4:11ec with SMTP id d9443c01a7336-21a83f42687mr30246255ad.1.1736319619595; Tue, 07 Jan 2025 23:00:19 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.244]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-219dca023a3sm320067275ad.250.2025.01.07.23.00.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 07 Jan 2025 23:00:19 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, agordeev@linux.ibm.com, kevin.brodsky@arm.com, alex@ghiti.fr, andreas@gaisler.com, palmer@dabbelt.com, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com, arnd@arndb.de, will@kernel.org, aneesh.kumar@kernel.org, npiggin@gmail.com, dave.hansen@linux.intel.com, rppt@kernel.org, ryan.roberts@arm.com Cc: linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, linux-arch@vger.kernel.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, loongarch@lists.linux.dev, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-sh@vger.kernel.org, linux-um@lists.infradead.org, Qi Zheng Subject: [PATCH v5 04/17] mm: pgtable: add statistics for P4D level page table Date: Wed, 8 Jan 2025 14:57:20 +0800 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Like other levels of page tables, add statistics for P4D level page table. Signed-off-by: Qi Zheng Originally-by: Peter Zijlstra (Intel) Reviewed-by: Kevin Brodsky --- arch/riscv/include/asm/pgalloc.h | 6 +++++- arch/x86/mm/pgtable.c | 3 +++ include/asm-generic/pgalloc.h | 2 ++ include/linux/mm.h | 16 ++++++++++++++++ 4 files changed, 26 insertions(+), 1 deletion(-) diff --git a/arch/riscv/include/asm/pgalloc.h b/arch/riscv/include/asm/pgal= loc.h index 551d614d3369c..3466fbe2e508d 100644 --- a/arch/riscv/include/asm/pgalloc.h +++ b/arch/riscv/include/asm/pgalloc.h @@ -108,8 +108,12 @@ static inline void __pud_free_tlb(struct mmu_gather *t= lb, pud_t *pud, static inline void __p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d, unsigned long addr) { - if (pgtable_l5_enabled) + if (pgtable_l5_enabled) { + struct ptdesc *ptdesc =3D virt_to_ptdesc(p4d); + + pagetable_p4d_dtor(ptdesc); riscv_tlb_remove_ptdesc(tlb, virt_to_ptdesc(p4d)); + } } #endif /* __PAGETABLE_PMD_FOLDED */ =20 diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c index 69a357b15974a..3d6e84da45b24 100644 --- a/arch/x86/mm/pgtable.c +++ b/arch/x86/mm/pgtable.c @@ -94,6 +94,9 @@ void ___pud_free_tlb(struct mmu_gather *tlb, pud_t *pud) #if CONFIG_PGTABLE_LEVELS > 4 void ___p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d) { + struct ptdesc *ptdesc =3D virt_to_ptdesc(p4d); + + pagetable_p4d_dtor(ptdesc); paravirt_release_p4d(__pa(p4d) >> PAGE_SHIFT); paravirt_tlb_remove_table(tlb, virt_to_page(p4d)); } diff --git a/include/asm-generic/pgalloc.h b/include/asm-generic/pgalloc.h index 59131629ac9cc..bb482eeca0c3e 100644 --- a/include/asm-generic/pgalloc.h +++ b/include/asm-generic/pgalloc.h @@ -230,6 +230,7 @@ static inline p4d_t *__p4d_alloc_one_noprof(struct mm_s= truct *mm, unsigned long if (!ptdesc) return NULL; =20 + pagetable_p4d_ctor(ptdesc); return ptdesc_address(ptdesc); } #define __p4d_alloc_one(...) alloc_hooks(__p4d_alloc_one_noprof(__VA_ARGS_= _)) @@ -247,6 +248,7 @@ static inline void __p4d_free(struct mm_struct *mm, p4d= _t *p4d) struct ptdesc *ptdesc =3D virt_to_ptdesc(p4d); =20 BUG_ON((unsigned long)p4d & (PAGE_SIZE-1)); + pagetable_p4d_dtor(ptdesc); pagetable_free(ptdesc); } =20 diff --git a/include/linux/mm.h b/include/linux/mm.h index c49bc7b764535..5d82f42ddd5cc 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -3175,6 +3175,22 @@ static inline void pagetable_pud_dtor(struct ptdesc = *ptdesc) lruvec_stat_sub_folio(folio, NR_PAGETABLE); } =20 +static inline void pagetable_p4d_ctor(struct ptdesc *ptdesc) +{ + struct folio *folio =3D ptdesc_folio(ptdesc); + + __folio_set_pgtable(folio); + lruvec_stat_add_folio(folio, NR_PAGETABLE); +} + +static inline void pagetable_p4d_dtor(struct ptdesc *ptdesc) +{ + struct folio *folio =3D ptdesc_folio(ptdesc); + + __folio_clear_pgtable(folio); + lruvec_stat_sub_folio(folio, NR_PAGETABLE); +} + extern void __init pagecache_init(void); extern void free_initmem(void); =20 --=20 2.20.1 From nobody Wed Dec 17 15:51:21 2025 Received: from mail-pl1-f182.google.com (mail-pl1-f182.google.com [209.85.214.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A6B441A9B2A for ; Wed, 8 Jan 2025 07:00:35 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.182 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1736319638; cv=none; b=c5TzOfIObvaEbVO2AQy/juz/rvvjYsZ7K2MhLfEJ7PlaSdr4Dx8AqDVAAyV+Dc9BAwB3btYKkVgBYluzD0vOYLvMTowD+oo/Dbdvika3DEZQejMw2UHESuTmgL8v81AfCTnaepYSccGZnhKNWQbmOg8qZLneCGaPLhNAPTg0Q+Q= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1736319638; c=relaxed/simple; bh=8PYWh2+orWwYf6986S1qPEzSyDMn0jEmjPeP/Gs1msM=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=FfWOM1nJQSk5fGPymjdhb2AHGUK50R/hRXUyZjfADWYtwaIJD7RowH63VvkBkB44DaCGCz84rZLXQMvxCOwlRlMYnpazob8ZGmq7CHQudkHVK1qy36SWvOxwxNzvhjXSXGiTdX5zTxDKOAaFoHpLTnOAjo4mkKH2qSTcMCsrGY8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=D36doFD2; arc=none smtp.client-ip=209.85.214.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="D36doFD2" Received: by mail-pl1-f182.google.com with SMTP id d9443c01a7336-21654fdd5daso225077625ad.1 for ; Tue, 07 Jan 2025 23:00:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1736319635; x=1736924435; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=s3ajwCEJzuBWHLhfIS3uOIg9mbtSga1CIaOnP3nLDr0=; b=D36doFD2eeEmR2bn6CJX8f5wpGnEPnzm3By1w8nWkBeMfXzl3kPPwr4mbqOkXEZhov VrwoHhOaSlU+az/WyLPiDSwtFodV/d7k98qHBI47BBbBmYjGz7mLnhwnUKtd9lRrnBFi bhczabQP8GmoHfwUxnHtM0Vh0K7nIolVZ4keO6qU+QqCEJjqJ1t+1kgiuPAylWLj3lf7 OYlKkbHZQNSSaIU8ju8jUadGcCbCbdPcu1WPO2WK/TdX0ptilPW+h759vInIarzE90sV 78hM/2Eyh7svckrSCheuAzoM3IR2SdV33HEvReFeV1KYoQqUzwFE4/vjdTllVFU7wCtY 0s4g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736319635; x=1736924435; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=s3ajwCEJzuBWHLhfIS3uOIg9mbtSga1CIaOnP3nLDr0=; b=KNjgCWvm90dHB/MVUUkMGVYzhat+Q515o95P8tLfRpLhoOPLkVgRDBeVB7HNAADWFs sPRLMevv1/PClIk2FDcu7jXSi6M9+CBWzuOF5dy3MeT1g3p6AQRvTjOn00HAbhIq4KME VmApFQSgGQvnQ6WqF585EZOfpkd+XQUZmg74kPvrnuLXGM9D4D7mwDZ2nbJmS1KKHxOj 82LmrOWwq6ZvBkUinazvla2SIiZbuukaHqfk7Fh6MA/9YD25vz/npmEp/R3TAvPVRzEB hrvqHgLh3UsQm3Vx0juoKpagrwnkPgJ1ZFWp/qhy9nzMowdlSDz7XrNqGk2Zvkz+PlLT xMxQ== X-Forwarded-Encrypted: i=1; AJvYcCX1oYlRY5kaZ2qYPtQ6fCHvqTs+5cygFG/kDUNzNODCGkl5UQLSctt+qyxjmrRYm3fGFmvlXrON286nRbs=@vger.kernel.org X-Gm-Message-State: AOJu0Yzb0Jo0p07R2lt1bW3mTMJczWnvWpcqoV69IRWDHTWTw4qhecd/ rkEoqDR9Pwfc8LN0UUPI6+Fhj9q2x3DdD1RGPxsbgczzcwMJRYhODtGlMYF9nJE= X-Gm-Gg: ASbGnctKTGUwxcYaxtQHoxf1Aarpmo0pwkATANL68TuAJWOfOxWH1lXh6zOtq0vfmRy /1KOY25qcrFkcBckx9V2aLXKn7NpyEn5Fr9u8hNZ5MPPeSO9NUyJofbEGAysHbeXZ4BRUw2imFu cDbskjgQ2DCwcLBGQlsjzGN5dMoCrykqKvkCe4KGzsN2hwbbld4wg8Ae9ZRH8TQjnjCC+Mtnqpe dIoAqGPyTQewUt9QamQSz/saHewGQkt+elG3JhdSjvVzTCsoiNFZ8KGk7iCUCXQk7TJ9WESb74j mJ1Ps2DNaCPc2kCcGrp6F7Xc9QE= X-Google-Smtp-Source: AGHT+IFQatZJEh1vMaYN2M2lTtxd4TB3DgwS5fDlIHVUeP0cQjVCzCSP2sGAFE7qJT7hKS5hKn1bKA== X-Received: by 2002:a17:902:d4d2:b0:215:9642:4d6d with SMTP id d9443c01a7336-21a83f55157mr23731385ad.17.1736319635015; Tue, 07 Jan 2025 23:00:35 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.244]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-219dca023a3sm320067275ad.250.2025.01.07.23.00.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 07 Jan 2025 23:00:34 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, agordeev@linux.ibm.com, kevin.brodsky@arm.com, alex@ghiti.fr, andreas@gaisler.com, palmer@dabbelt.com, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com, arnd@arndb.de, will@kernel.org, aneesh.kumar@kernel.org, npiggin@gmail.com, dave.hansen@linux.intel.com, rppt@kernel.org, ryan.roberts@arm.com Cc: linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, linux-arch@vger.kernel.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, loongarch@lists.linux.dev, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-sh@vger.kernel.org, linux-um@lists.infradead.org, Qi Zheng Subject: [PATCH v5 05/17] arm64: pgtable: use mmu gather to free p4d level page table Date: Wed, 8 Jan 2025 14:57:21 +0800 Message-Id: <3fd48525397b34a64f7c0eb76746da30814dc941.1736317725.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Like other levels of page tables, also use mmu gather mechanism to free p4d level page table. Signed-off-by: Qi Zheng Originally-by: Peter Zijlstra (Intel) Reviewed-by: Kevin Brodsky Cc: linux-arm-kernel@lists.infradead.org Acked-by: Will Deacon --- arch/arm64/include/asm/pgalloc.h | 1 - arch/arm64/include/asm/tlb.h | 14 ++++++++++++++ 2 files changed, 14 insertions(+), 1 deletion(-) diff --git a/arch/arm64/include/asm/pgalloc.h b/arch/arm64/include/asm/pgal= loc.h index 2965f5a7e39e3..1b4509d3382c6 100644 --- a/arch/arm64/include/asm/pgalloc.h +++ b/arch/arm64/include/asm/pgalloc.h @@ -85,7 +85,6 @@ static inline void pgd_populate(struct mm_struct *mm, pgd= _t *pgdp, p4d_t *p4dp) __pgd_populate(pgdp, __pa(p4dp), pgdval); } =20 -#define __p4d_free_tlb(tlb, p4d, addr) p4d_free((tlb)->mm, p4d) #else static inline void __pgd_populate(pgd_t *pgdp, phys_addr_t p4dp, pgdval_t = prot) { diff --git a/arch/arm64/include/asm/tlb.h b/arch/arm64/include/asm/tlb.h index a947c6e784ed2..445282cde9afb 100644 --- a/arch/arm64/include/asm/tlb.h +++ b/arch/arm64/include/asm/tlb.h @@ -111,4 +111,18 @@ static inline void __pud_free_tlb(struct mmu_gather *t= lb, pud_t *pudp, } #endif =20 +#if CONFIG_PGTABLE_LEVELS > 4 +static inline void __p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4dp, + unsigned long addr) +{ + struct ptdesc *ptdesc =3D virt_to_ptdesc(p4dp); + + if (!pgtable_l5_enabled()) + return; + + pagetable_p4d_dtor(ptdesc); + tlb_remove_ptdesc(tlb, ptdesc); +} +#endif + #endif --=20 2.20.1 From nobody Wed Dec 17 15:51:21 2025 Received: from mail-pl1-f182.google.com (mail-pl1-f182.google.com [209.85.214.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 34E6A1A8403 for ; Wed, 8 Jan 2025 07:00:50 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.182 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1736319651; cv=none; b=ap8tDYXsP9IXqrnPOCYoRy/Q7MziAnRTjDYfALzs1Yl97U8WMBVVVA0V+YWYbigUKEAd/BozuUXkHkg7xINYjQnTsbNTmt4MYIGoIRze466Xs9PO/xNCyy1+rWdAPZN49A6JhK3xbu7rrd1z3rHCnHclTWdJdx+icPKi9Xowxe0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1736319651; c=relaxed/simple; bh=aGqvHL2vBr0F32b5nn3s88qKn1CLr8WeccygZVRBAEw=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=qzyMTPMaZt6LRXCLlu4d3D+9sNhi6J8BAloMGHahFW8Y1x5zqpUqjyrcT+HXYkVk8MI+yTGyknI8X/PimC5RaIxo6eOeujH9qhfo5cluhg1Q5asW5wnHhtZ5Dg9UkaFwTiBbwT8HJlKQYk0hhQADx9SRM9Ztp53xc9q0qbsnRqQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=R1wYrQBv; arc=none smtp.client-ip=209.85.214.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="R1wYrQBv" Received: by mail-pl1-f182.google.com with SMTP id d9443c01a7336-21661be2c2dso222200475ad.1 for ; Tue, 07 Jan 2025 23:00:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1736319650; x=1736924450; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=evJHXxUH8wpRsLAfhvt7S2AIimf3Ggw2vmlVHfN1TNM=; b=R1wYrQBvekglq61bdhkezbMUyzBomFK02UdYR3kFqRnxl/ukKmfQF9ATptRIOvqpeF 9Vz9wk1St+iBjzEkYfMiMTuBqylqgHpzKXP2eLtyC9QzQQDr99+l3Hxx0wNtpd1HkncN PbJ3uFm44HoZZl9h/IBko4TGSwZicRkhowjkSA4Qlc0RX/gX+7YeXTzCM7V7pfFBMlv4 J+wkM/ri2+s7Wklf1T32NKvdukAsl1n02xAkd/D+ST/Xh0uOjERNXbW5WVR5eu+8sRU9 8jYYfXAGwkfqIpAs/o8/O0p0ybRVYFIYjrVx4jhSJzzIpwbx9Vjn8A3IRUBBQTY3OPkB bVQA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736319650; x=1736924450; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=evJHXxUH8wpRsLAfhvt7S2AIimf3Ggw2vmlVHfN1TNM=; b=MbvYZLkPivmiTBMZHyWeDhcC36aGkUhgoWhuyeiuplLIMEoOiE+5CzEbNIWuL1dOZB lTwKozY1/QCh6lLpI47DxFfxE+8f0DJBqRCe1kcokGDdD4J+fO6CJSW+pEmEQ1IbIv0G iozdh2CveSvv54up/twP2M1PmfoHCJpT7qjv26YrWeoG+Rlt1MZE/tEHQ4uuQPzgHjb7 Y5O7OQZrOrpTg3GtO0BZmH+/O8DKlCqNs3v/Qoj5kEsFnFaZYWS/gvUuBrU3NTn3hXdB oqm+MrARl/GkyuSFOzwi7KRaSVz3vvB0h8lxAc8nCzpqv/qa8V0rZZqEgD05TARdigD6 b4bg== X-Forwarded-Encrypted: i=1; AJvYcCW6spB4XWn5aMbX+AtIwNLiER3sET15659fe1UUoxx/gtbE0O/NJps593c60uPhkcdqW3qVSDiWfeQKGnY=@vger.kernel.org X-Gm-Message-State: AOJu0Yxll+EmMSmAdt8b+xg+XbF7/Ai58oj3qLJ7uJggsL3oYMHugeH5 KmhN8CMnMV1YpEFtmNCKsAyXwC1Pk+KSRYe9nvBO6wj3JNUT5ZNBHlkbqZk6f6kjfiLpkJTifeE 6 X-Gm-Gg: ASbGncugh86AOu/AnxxM9YQc817vmTKnpuZ4xvb3PZX/Ix8Osfe4OxDKfhB77VuzUlQ bMkOCJqg7q+y2Qau3RTQQbrTntQz9tkFZNRmLg0zZ8WL6VYQ/V1J5ctf/qsx/6xkinAwcmwv8ff 02925yclmYttnpSe30gT1ABASoYopaK3MsFBX0jTNHCf+bD5fpTMcVKyMIpXP0QHTXaqpvu2J1q AN5qgW1AA59ydxjL0NFo0U4AuXxg7e3nlIvVhWVYBUfUxLYIo9GDahPFOtc/7PKblCGqBkHhori ulE8pKg7OM9UkcC8eB0V333AzMc= X-Google-Smtp-Source: AGHT+IF9X6lXCsguwyHO5KdCHubvdTIDd4Y888fAuDMWr6lGszjnzuDzetJYvdD8sXl2Z0Y5NvXbNg== X-Received: by 2002:a17:902:cec3:b0:216:7ee9:220b with SMTP id d9443c01a7336-21a83f4f43amr30110425ad.22.1736319649715; Tue, 07 Jan 2025 23:00:49 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.244]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-219dca023a3sm320067275ad.250.2025.01.07.23.00.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 07 Jan 2025 23:00:49 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, agordeev@linux.ibm.com, kevin.brodsky@arm.com, alex@ghiti.fr, andreas@gaisler.com, palmer@dabbelt.com, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com, arnd@arndb.de, will@kernel.org, aneesh.kumar@kernel.org, npiggin@gmail.com, dave.hansen@linux.intel.com, rppt@kernel.org, ryan.roberts@arm.com Cc: linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, linux-arch@vger.kernel.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, loongarch@lists.linux.dev, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-sh@vger.kernel.org, linux-um@lists.infradead.org, Qi Zheng Subject: [PATCH v5 06/17] s390: pgtable: add statistics for PUD and P4D level page table Date: Wed, 8 Jan 2025 14:57:22 +0800 Message-Id: <4707dffce228ccec5c6662810566dd12b5741c4b.1736317725.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Like PMD and PTE level page table, also add statistics for PUD and P4D page table. Signed-off-by: Qi Zheng Suggested-by: Peter Zijlstra (Intel) Reviewed-by: Kevin Brodsky Cc: linux-s390@vger.kernel.org Acked-by: Alexander Gordeev --- arch/s390/include/asm/pgalloc.h | 29 +++++++++++++++++++++-------- arch/s390/include/asm/tlb.h | 2 ++ 2 files changed, 23 insertions(+), 8 deletions(-) diff --git a/arch/s390/include/asm/pgalloc.h b/arch/s390/include/asm/pgallo= c.h index 7b84ef6dc4b6d..a0c1ca5d8423c 100644 --- a/arch/s390/include/asm/pgalloc.h +++ b/arch/s390/include/asm/pgalloc.h @@ -53,29 +53,42 @@ static inline p4d_t *p4d_alloc_one(struct mm_struct *mm= , unsigned long address) { unsigned long *table =3D crst_table_alloc(mm); =20 - if (table) - crst_table_init(table, _REGION2_ENTRY_EMPTY); + if (!table) + return NULL; + crst_table_init(table, _REGION2_ENTRY_EMPTY); + pagetable_p4d_ctor(virt_to_ptdesc(table)); + return (p4d_t *) table; } =20 static inline void p4d_free(struct mm_struct *mm, p4d_t *p4d) { - if (!mm_p4d_folded(mm)) - crst_table_free(mm, (unsigned long *) p4d); + if (mm_p4d_folded(mm)) + return; + + pagetable_p4d_dtor(virt_to_ptdesc(p4d)); + crst_table_free(mm, (unsigned long *) p4d); } =20 static inline pud_t *pud_alloc_one(struct mm_struct *mm, unsigned long add= ress) { unsigned long *table =3D crst_table_alloc(mm); - if (table) - crst_table_init(table, _REGION3_ENTRY_EMPTY); + + if (!table) + return NULL; + crst_table_init(table, _REGION3_ENTRY_EMPTY); + pagetable_pud_ctor(virt_to_ptdesc(table)); + return (pud_t *) table; } =20 static inline void pud_free(struct mm_struct *mm, pud_t *pud) { - if (!mm_pud_folded(mm)) - crst_table_free(mm, (unsigned long *) pud); + if (mm_pud_folded(mm)) + return; + + pagetable_pud_dtor(virt_to_ptdesc(pud)); + crst_table_free(mm, (unsigned long *) pud); } =20 static inline pmd_t *pmd_alloc_one(struct mm_struct *mm, unsigned long vma= ddr) diff --git a/arch/s390/include/asm/tlb.h b/arch/s390/include/asm/tlb.h index e95b2c8081eb8..907d57a68145c 100644 --- a/arch/s390/include/asm/tlb.h +++ b/arch/s390/include/asm/tlb.h @@ -122,6 +122,7 @@ static inline void p4d_free_tlb(struct mmu_gather *tlb,= p4d_t *p4d, { if (mm_p4d_folded(tlb->mm)) return; + pagetable_p4d_dtor(virt_to_ptdesc(p4d)); __tlb_adjust_range(tlb, address, PAGE_SIZE); tlb->mm->context.flush_mm =3D 1; tlb->freed_tables =3D 1; @@ -140,6 +141,7 @@ static inline void pud_free_tlb(struct mmu_gather *tlb,= pud_t *pud, { if (mm_pud_folded(tlb->mm)) return; + pagetable_pud_dtor(virt_to_ptdesc(pud)); tlb->mm->context.flush_mm =3D 1; tlb->freed_tables =3D 1; tlb->cleared_p4ds =3D 1; --=20 2.20.1 From nobody Wed Dec 17 15:51:21 2025 Received: from mail-pl1-f180.google.com (mail-pl1-f180.google.com [209.85.214.180]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5F9E01A705C for ; Wed, 8 Jan 2025 07:01:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.180 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1736319668; cv=none; b=JjhXZ7xbUfTcW0wImTh8XROityFSGKpOEc1mVFGzPbir0jD0S7SimGR1HaXlnN3D0QR1I/FexXS9HJqjPO63m/v5/RHAQsasOIEHb3Z45Bb/lVO21YX/lav1DkKc2hZFHuWapMmzPVeUD8bSVVfMofz7qG83MnaP2a4xAy2M2x0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1736319668; c=relaxed/simple; bh=SSgtTdsZlM0tZ0GMhABJlNQcE/o2NlRBTYOMSY7xRuk=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=jwsoYbkJGGuwYzryoFJ7xBRp/yLe9vRzT2MDFElP2HQW85BtWO4bpYRb8ZW9RJAp02g8bV4KWTRnEzpfPArrDOdhNflHS0PuZOxFEZFjUCjSi/WxwiSDsoJfJGckiGJNgIy6nyuyS+j8q1dhBxXr4E51vncrt0TdfKL474Aw3xQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=exidiqMP; arc=none smtp.client-ip=209.85.214.180 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="exidiqMP" Received: by mail-pl1-f180.google.com with SMTP id d9443c01a7336-2165448243fso46470775ad.1 for ; Tue, 07 Jan 2025 23:01:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1736319665; x=1736924465; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=3i/HRrd9DcQR4JIafWbJ+2XekX+xgLIenO/u5trArdk=; b=exidiqMPyzsHbAb4CVG8K7MZ0gInlcW2DUMSbbhcfHZZZhL3TTtVwefu9H+ozocJwo Jaqj46OJ/j0T1bgxeE2uGPdje3ZakeszSUfAc9NgqSYXjwKPn0Q5b/H/zyZv41MlGy3y UvCAJnf7bEEAKodWVKEw8trxJDlyOXZ0TP/87Pe4+JoQ2sGUx8MXVwT20/Yl6mr4CR3q 7gF28Ysta6dK0VNsUBotfNJoHCDzkzYC98ZTiNkBmSzRtYj9kD7J3z5CbfQNfaKtl+tv mfArlfu8mj4FB3WmdwCtjB7HNnmKj/Wl1DytPlcpZn+PWzTWfAc6vkx6HvYP4mmgclle ZiMA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736319665; x=1736924465; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=3i/HRrd9DcQR4JIafWbJ+2XekX+xgLIenO/u5trArdk=; b=NDdW81O9SS9Lh3dxNKmjj1TLSEEppp++YBbOeGfg+kNhnJGr3461tmdVdniB/a0QAe JCfaqSsh+ScZQoadDC4fsyzLfSNXYFtKp0/eFuyoFO8Fp6xkqp3AS+8Q7xH5yoSJc9ej 9o2PxM1KFpop39IuRaPazrt5uGjM0TITEo0UikImlNizaqYGrSzoY1JYSGYeBVFmp+ik ffG0IiIPun89LQnR3ht2Ri6/RO3kNa55lbAaZ4sYc01N2Jdx84qdT86rXlMSc/WiUxWg jwTnhlMFR+ZDlXRdilnk+cJEgq17aZ9x+8c9cva/PO+GvhTizSrVawPoavtzIxXE91kJ WggA== X-Forwarded-Encrypted: i=1; AJvYcCUqiAjTNuzUEw6cM8ASWvdwsFImjpx00A9jxQfQ5nBk27ExPvUbrJYyk0EfN2x8W4oC0k8hYPFzmNAEEX0=@vger.kernel.org X-Gm-Message-State: AOJu0Yw4y6wIK3jKizMiLUxZ7fLtz48Is+syupnWOFJxHjlmrTlZeFFZ oC5dT6K9o7YqQXK6Hs+2g4W+oOzcsqLpNXYFuMpoR2fZgwW8AM8a3/qorZCMg50= X-Gm-Gg: ASbGncsn8Z5erfY2bZW2OUxr8Df+npj1rXxZ49ybMMsZ3x1/9ZvKxrGaiNjK/M3pRoJ c3xPd099x3z9tG7cAZBW2eoPQGd+aZ0VPYXJAw+SLHDrQy3qGUuM/zLl+HqCwHsNa4aQziZARN9 RKAaDUpG4Y7eQMSRE+pdKbRyh5ps1LJzpspmkAtJX3FmqAl8cp1uAOT0ixErfR7ChRMZ2+UMZHO ek9ETofK3oOgWSdDixW3VBFgbzCc1+e2oig/3GAACW/vmZXR8cckKEhyIE/c0QH+I49HQlSeJ0i KEW60NO7/qZEV2fkxDbiHOyQkUk= X-Google-Smtp-Source: AGHT+IEn/Q+3hOSV0Xf6HR7089ZKBzaD+1qnLFodeVKvlP0f0VEvA1wx4/vhMKgRrGPfbs0rdLpQmg== X-Received: by 2002:a17:903:a4f:b0:219:d28a:ca23 with SMTP id d9443c01a7336-21a83fe4c55mr23137765ad.36.1736319664669; Tue, 07 Jan 2025 23:01:04 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.244]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-219dca023a3sm320067275ad.250.2025.01.07.23.00.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 07 Jan 2025 23:01:04 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, agordeev@linux.ibm.com, kevin.brodsky@arm.com, alex@ghiti.fr, andreas@gaisler.com, palmer@dabbelt.com, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com, arnd@arndb.de, will@kernel.org, aneesh.kumar@kernel.org, npiggin@gmail.com, dave.hansen@linux.intel.com, rppt@kernel.org, ryan.roberts@arm.com Cc: linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, linux-arch@vger.kernel.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, loongarch@lists.linux.dev, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-sh@vger.kernel.org, linux-um@lists.infradead.org, Qi Zheng Subject: [PATCH v5 07/17] mm: pgtable: introduce pagetable_dtor() Date: Wed, 8 Jan 2025 14:57:23 +0800 Message-Id: <47f44fff9dc68d9d9e9a0d6c036df275f820598a.1736317725.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The pagetable_p*_dtor() are exactly the same except for the handling of ptlock. If we make ptlock_free() handle the case where ptdesc->ptl is NULL and remove VM_BUG_ON_PAGE() from pmd_ptlock_free(), we can unify pagetable_p*_dtor() into one function. Let's introduce pagetable_dtor() to do this. Later, pagetable_dtor() will be moved to tlb_remove_ptdesc(), so that ptlock and page table pages can be freed together (regardless of whether RCU is used). This prevents the use-after-free problem where the ptlock is freed immediately but the page table pages is freed later via RCU. Signed-off-by: Qi Zheng Originally-by: Peter Zijlstra (Intel) Reviewed-by: Kevin Brodsky Acked-by: Alexander Gordeev --- Documentation/mm/split_page_table_lock.rst | 4 +- arch/arm/include/asm/tlb.h | 4 +- arch/arm64/include/asm/tlb.h | 8 ++-- arch/csky/include/asm/pgalloc.h | 2 +- arch/hexagon/include/asm/pgalloc.h | 2 +- arch/loongarch/include/asm/pgalloc.h | 2 +- arch/m68k/include/asm/mcf_pgalloc.h | 4 +- arch/m68k/include/asm/sun3_pgalloc.h | 2 +- arch/m68k/mm/motorola.c | 2 +- arch/mips/include/asm/pgalloc.h | 2 +- arch/nios2/include/asm/pgalloc.h | 2 +- arch/openrisc/include/asm/pgalloc.h | 2 +- arch/powerpc/mm/book3s64/mmu_context.c | 2 +- arch/powerpc/mm/book3s64/pgtable.c | 2 +- arch/powerpc/mm/pgtable-frag.c | 4 +- arch/riscv/include/asm/pgalloc.h | 8 ++-- arch/riscv/mm/init.c | 4 +- arch/s390/include/asm/pgalloc.h | 6 +-- arch/s390/include/asm/tlb.h | 6 +-- arch/s390/mm/pgalloc.c | 2 +- arch/sh/include/asm/pgalloc.h | 2 +- arch/sparc/mm/init_64.c | 2 +- arch/sparc/mm/srmmu.c | 2 +- arch/um/include/asm/pgalloc.h | 6 +-- arch/x86/mm/pgtable.c | 12 ++--- include/asm-generic/pgalloc.h | 8 ++-- include/linux/mm.h | 52 ++++------------------ mm/memory.c | 3 +- 28 files changed, 62 insertions(+), 95 deletions(-) diff --git a/Documentation/mm/split_page_table_lock.rst b/Documentation/mm/= split_page_table_lock.rst index 581446d4a4eba..8e1ceb0a6619a 100644 --- a/Documentation/mm/split_page_table_lock.rst +++ b/Documentation/mm/split_page_table_lock.rst @@ -62,7 +62,7 @@ Support of split page table lock by an architecture =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D =20 There's no need in special enabling of PTE split page table lock: everythi= ng -required is done by pagetable_pte_ctor() and pagetable_pte_dtor(), which +required is done by pagetable_pte_ctor() and pagetable_dtor(), which must be called on PTE table allocation / freeing. =20 Make sure the architecture doesn't use slab allocator for page table @@ -73,7 +73,7 @@ PMD split lock only makes sense if you have more than two= page table levels. =20 PMD split lock enabling requires pagetable_pmd_ctor() call on PMD table -allocation and pagetable_pmd_dtor() on freeing. +allocation and pagetable_dtor() on freeing. =20 Allocation usually happens in pmd_alloc_one(), freeing in pmd_free() and pmd_free_tlb(), but make sure you cover all PMD table allocation / freeing diff --git a/arch/arm/include/asm/tlb.h b/arch/arm/include/asm/tlb.h index f40d06ad5d2a3..ef79bf1e8563f 100644 --- a/arch/arm/include/asm/tlb.h +++ b/arch/arm/include/asm/tlb.h @@ -41,7 +41,7 @@ __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pte, uns= igned long addr) { struct ptdesc *ptdesc =3D page_ptdesc(pte); =20 - pagetable_pte_dtor(ptdesc); + pagetable_dtor(ptdesc); =20 #ifndef CONFIG_ARM_LPAE /* @@ -61,7 +61,7 @@ __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmdp, unsig= ned long addr) #ifdef CONFIG_ARM_LPAE struct ptdesc *ptdesc =3D virt_to_ptdesc(pmdp); =20 - pagetable_pmd_dtor(ptdesc); + pagetable_dtor(ptdesc); tlb_remove_ptdesc(tlb, ptdesc); #endif } diff --git a/arch/arm64/include/asm/tlb.h b/arch/arm64/include/asm/tlb.h index 445282cde9afb..408d0f36a8a8f 100644 --- a/arch/arm64/include/asm/tlb.h +++ b/arch/arm64/include/asm/tlb.h @@ -82,7 +82,7 @@ static inline void __pte_free_tlb(struct mmu_gather *tlb,= pgtable_t pte, { struct ptdesc *ptdesc =3D page_ptdesc(pte); =20 - pagetable_pte_dtor(ptdesc); + pagetable_dtor(ptdesc); tlb_remove_ptdesc(tlb, ptdesc); } =20 @@ -92,7 +92,7 @@ static inline void __pmd_free_tlb(struct mmu_gather *tlb,= pmd_t *pmdp, { struct ptdesc *ptdesc =3D virt_to_ptdesc(pmdp); =20 - pagetable_pmd_dtor(ptdesc); + pagetable_dtor(ptdesc); tlb_remove_ptdesc(tlb, ptdesc); } #endif @@ -106,7 +106,7 @@ static inline void __pud_free_tlb(struct mmu_gather *tl= b, pud_t *pudp, if (!pgtable_l4_enabled()) return; =20 - pagetable_pud_dtor(ptdesc); + pagetable_dtor(ptdesc); tlb_remove_ptdesc(tlb, ptdesc); } #endif @@ -120,7 +120,7 @@ static inline void __p4d_free_tlb(struct mmu_gather *tl= b, p4d_t *p4dp, if (!pgtable_l5_enabled()) return; =20 - pagetable_p4d_dtor(ptdesc); + pagetable_dtor(ptdesc); tlb_remove_ptdesc(tlb, ptdesc); } #endif diff --git a/arch/csky/include/asm/pgalloc.h b/arch/csky/include/asm/pgallo= c.h index 9c84c9012e534..f1ce5b7b28f22 100644 --- a/arch/csky/include/asm/pgalloc.h +++ b/arch/csky/include/asm/pgalloc.h @@ -63,7 +63,7 @@ static inline pgd_t *pgd_alloc(struct mm_struct *mm) =20 #define __pte_free_tlb(tlb, pte, address) \ do { \ - pagetable_pte_dtor(page_ptdesc(pte)); \ + pagetable_dtor(page_ptdesc(pte)); \ tlb_remove_page_ptdesc(tlb, page_ptdesc(pte)); \ } while (0) =20 diff --git a/arch/hexagon/include/asm/pgalloc.h b/arch/hexagon/include/asm/= pgalloc.h index 55988625e6fbc..40e42a0e71673 100644 --- a/arch/hexagon/include/asm/pgalloc.h +++ b/arch/hexagon/include/asm/pgalloc.h @@ -89,7 +89,7 @@ static inline void pmd_populate_kernel(struct mm_struct *= mm, pmd_t *pmd, =20 #define __pte_free_tlb(tlb, pte, addr) \ do { \ - pagetable_pte_dtor((page_ptdesc(pte))); \ + pagetable_dtor((page_ptdesc(pte))); \ tlb_remove_page_ptdesc((tlb), (page_ptdesc(pte))); \ } while (0) =20 diff --git a/arch/loongarch/include/asm/pgalloc.h b/arch/loongarch/include/= asm/pgalloc.h index a7b9c9e73593d..7211dff8c969e 100644 --- a/arch/loongarch/include/asm/pgalloc.h +++ b/arch/loongarch/include/asm/pgalloc.h @@ -57,7 +57,7 @@ static inline pte_t *pte_alloc_one_kernel(struct mm_struc= t *mm) =20 #define __pte_free_tlb(tlb, pte, address) \ do { \ - pagetable_pte_dtor(page_ptdesc(pte)); \ + pagetable_dtor(page_ptdesc(pte)); \ tlb_remove_page_ptdesc((tlb), page_ptdesc(pte)); \ } while (0) =20 diff --git a/arch/m68k/include/asm/mcf_pgalloc.h b/arch/m68k/include/asm/mc= f_pgalloc.h index 302c5bf67179e..22d6c1fcabfb4 100644 --- a/arch/m68k/include/asm/mcf_pgalloc.h +++ b/arch/m68k/include/asm/mcf_pgalloc.h @@ -37,7 +37,7 @@ static inline void __pte_free_tlb(struct mmu_gather *tlb,= pgtable_t pgtable, { struct ptdesc *ptdesc =3D virt_to_ptdesc(pgtable); =20 - pagetable_pte_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); } =20 @@ -61,7 +61,7 @@ static inline void pte_free(struct mm_struct *mm, pgtable= _t pgtable) { struct ptdesc *ptdesc =3D virt_to_ptdesc(pgtable); =20 - pagetable_pte_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); } =20 diff --git a/arch/m68k/include/asm/sun3_pgalloc.h b/arch/m68k/include/asm/s= un3_pgalloc.h index 4a137eecb6fe4..2b626cb3ad0ae 100644 --- a/arch/m68k/include/asm/sun3_pgalloc.h +++ b/arch/m68k/include/asm/sun3_pgalloc.h @@ -19,7 +19,7 @@ extern const char bad_pmd_string[]; =20 #define __pte_free_tlb(tlb, pte, addr) \ do { \ - pagetable_pte_dtor(page_ptdesc(pte)); \ + pagetable_dtor(page_ptdesc(pte)); \ tlb_remove_page_ptdesc((tlb), page_ptdesc(pte)); \ } while (0) =20 diff --git a/arch/m68k/mm/motorola.c b/arch/m68k/mm/motorola.c index c1761d309fc61..81715cece70c6 100644 --- a/arch/m68k/mm/motorola.c +++ b/arch/m68k/mm/motorola.c @@ -201,7 +201,7 @@ int free_pointer_table(void *table, int type) list_del(dp); mmu_page_dtor((void *)page); if (type =3D=3D TABLE_PTE) - pagetable_pte_dtor(virt_to_ptdesc((void *)page)); + pagetable_dtor(virt_to_ptdesc((void *)page)); free_page (page); return 1; } else if (ptable_list[type].next !=3D dp) { diff --git a/arch/mips/include/asm/pgalloc.h b/arch/mips/include/asm/pgallo= c.h index f4440edcd8fe2..36d9805033c4b 100644 --- a/arch/mips/include/asm/pgalloc.h +++ b/arch/mips/include/asm/pgalloc.h @@ -56,7 +56,7 @@ static inline void pgd_free(struct mm_struct *mm, pgd_t *= pgd) =20 #define __pte_free_tlb(tlb, pte, address) \ do { \ - pagetable_pte_dtor(page_ptdesc(pte)); \ + pagetable_dtor(page_ptdesc(pte)); \ tlb_remove_page_ptdesc((tlb), page_ptdesc(pte)); \ } while (0) =20 diff --git a/arch/nios2/include/asm/pgalloc.h b/arch/nios2/include/asm/pgal= loc.h index ce6bb8e74271f..12a536b7bfbd4 100644 --- a/arch/nios2/include/asm/pgalloc.h +++ b/arch/nios2/include/asm/pgalloc.h @@ -30,7 +30,7 @@ extern pgd_t *pgd_alloc(struct mm_struct *mm); =20 #define __pte_free_tlb(tlb, pte, addr) \ do { \ - pagetable_pte_dtor(page_ptdesc(pte)); \ + pagetable_dtor(page_ptdesc(pte)); \ tlb_remove_page_ptdesc((tlb), (page_ptdesc(pte))); \ } while (0) =20 diff --git a/arch/openrisc/include/asm/pgalloc.h b/arch/openrisc/include/as= m/pgalloc.h index c6a73772a5466..596e2355824e3 100644 --- a/arch/openrisc/include/asm/pgalloc.h +++ b/arch/openrisc/include/asm/pgalloc.h @@ -68,7 +68,7 @@ extern pte_t *pte_alloc_one_kernel(struct mm_struct *mm); =20 #define __pte_free_tlb(tlb, pte, addr) \ do { \ - pagetable_pte_dtor(page_ptdesc(pte)); \ + pagetable_dtor(page_ptdesc(pte)); \ tlb_remove_page_ptdesc((tlb), (page_ptdesc(pte))); \ } while (0) =20 diff --git a/arch/powerpc/mm/book3s64/mmu_context.c b/arch/powerpc/mm/book3= s64/mmu_context.c index 1715b07c630c9..4e1e45420bd49 100644 --- a/arch/powerpc/mm/book3s64/mmu_context.c +++ b/arch/powerpc/mm/book3s64/mmu_context.c @@ -253,7 +253,7 @@ static void pmd_frag_destroy(void *pmd_frag) count =3D ((unsigned long)pmd_frag & ~PAGE_MASK) >> PMD_FRAG_SIZE_SHIFT; /* We allow PTE_FRAG_NR fragments from a PTE page */ if (atomic_sub_and_test(PMD_FRAG_NR - count, &ptdesc->pt_frag_refcount)) { - pagetable_pmd_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); } } diff --git a/arch/powerpc/mm/book3s64/pgtable.c b/arch/powerpc/mm/book3s64/= pgtable.c index 3745425280808..3f28e4acd920b 100644 --- a/arch/powerpc/mm/book3s64/pgtable.c +++ b/arch/powerpc/mm/book3s64/pgtable.c @@ -477,7 +477,7 @@ void pmd_fragment_free(unsigned long *pmd) =20 BUG_ON(atomic_read(&ptdesc->pt_frag_refcount) <=3D 0); if (atomic_dec_and_test(&ptdesc->pt_frag_refcount)) { - pagetable_pmd_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); } } diff --git a/arch/powerpc/mm/pgtable-frag.c b/arch/powerpc/mm/pgtable-frag.c index e89f64a0f24ae..713268ccb1a0e 100644 --- a/arch/powerpc/mm/pgtable-frag.c +++ b/arch/powerpc/mm/pgtable-frag.c @@ -25,7 +25,7 @@ void pte_frag_destroy(void *pte_frag) count =3D ((unsigned long)pte_frag & ~PAGE_MASK) >> PTE_FRAG_SIZE_SHIFT; /* We allow PTE_FRAG_NR fragments from a PTE page */ if (atomic_sub_and_test(PTE_FRAG_NR - count, &ptdesc->pt_frag_refcount)) { - pagetable_pte_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); } } @@ -111,7 +111,7 @@ static void pte_free_now(struct rcu_head *head) struct ptdesc *ptdesc; =20 ptdesc =3D container_of(head, struct ptdesc, pt_rcu_head); - pagetable_pte_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); } =20 diff --git a/arch/riscv/include/asm/pgalloc.h b/arch/riscv/include/asm/pgal= loc.h index 3466fbe2e508d..b6793c5c99296 100644 --- a/arch/riscv/include/asm/pgalloc.h +++ b/arch/riscv/include/asm/pgalloc.h @@ -100,7 +100,7 @@ static inline void __pud_free_tlb(struct mmu_gather *tl= b, pud_t *pud, if (pgtable_l4_enabled) { struct ptdesc *ptdesc =3D virt_to_ptdesc(pud); =20 - pagetable_pud_dtor(ptdesc); + pagetable_dtor(ptdesc); riscv_tlb_remove_ptdesc(tlb, ptdesc); } } @@ -111,7 +111,7 @@ static inline void __p4d_free_tlb(struct mmu_gather *tl= b, p4d_t *p4d, if (pgtable_l5_enabled) { struct ptdesc *ptdesc =3D virt_to_ptdesc(p4d); =20 - pagetable_p4d_dtor(ptdesc); + pagetable_dtor(ptdesc); riscv_tlb_remove_ptdesc(tlb, virt_to_ptdesc(p4d)); } } @@ -144,7 +144,7 @@ static inline void __pmd_free_tlb(struct mmu_gather *tl= b, pmd_t *pmd, { struct ptdesc *ptdesc =3D virt_to_ptdesc(pmd); =20 - pagetable_pmd_dtor(ptdesc); + pagetable_dtor(ptdesc); riscv_tlb_remove_ptdesc(tlb, ptdesc); } =20 @@ -155,7 +155,7 @@ static inline void __pte_free_tlb(struct mmu_gather *tl= b, pgtable_t pte, { struct ptdesc *ptdesc =3D page_ptdesc(pte); =20 - pagetable_pte_dtor(ptdesc); + pagetable_dtor(ptdesc); riscv_tlb_remove_ptdesc(tlb, ptdesc); } #endif /* CONFIG_MMU */ diff --git a/arch/riscv/mm/init.c b/arch/riscv/mm/init.c index fc53ce748c804..8d703fb51b1dc 100644 --- a/arch/riscv/mm/init.c +++ b/arch/riscv/mm/init.c @@ -1558,7 +1558,7 @@ static void __meminit free_pte_table(pte_t *pte_start= , pmd_t *pmd) return; } =20 - pagetable_pte_dtor(ptdesc); + pagetable_dtor(ptdesc); if (PageReserved(page)) free_reserved_page(page); else @@ -1580,7 +1580,7 @@ static void __meminit free_pmd_table(pmd_t *pmd_start= , pud_t *pud, bool is_vmemm } =20 if (!is_vmemmap) - pagetable_pmd_dtor(ptdesc); + pagetable_dtor(ptdesc); if (PageReserved(page)) free_reserved_page(page); else diff --git a/arch/s390/include/asm/pgalloc.h b/arch/s390/include/asm/pgallo= c.h index a0c1ca5d8423c..5fced6d3c36b0 100644 --- a/arch/s390/include/asm/pgalloc.h +++ b/arch/s390/include/asm/pgalloc.h @@ -66,7 +66,7 @@ static inline void p4d_free(struct mm_struct *mm, p4d_t *= p4d) if (mm_p4d_folded(mm)) return; =20 - pagetable_p4d_dtor(virt_to_ptdesc(p4d)); + pagetable_dtor(virt_to_ptdesc(p4d)); crst_table_free(mm, (unsigned long *) p4d); } =20 @@ -87,7 +87,7 @@ static inline void pud_free(struct mm_struct *mm, pud_t *= pud) if (mm_pud_folded(mm)) return; =20 - pagetable_pud_dtor(virt_to_ptdesc(pud)); + pagetable_dtor(virt_to_ptdesc(pud)); crst_table_free(mm, (unsigned long *) pud); } =20 @@ -109,7 +109,7 @@ static inline void pmd_free(struct mm_struct *mm, pmd_t= *pmd) { if (mm_pmd_folded(mm)) return; - pagetable_pmd_dtor(virt_to_ptdesc(pmd)); + pagetable_dtor(virt_to_ptdesc(pmd)); crst_table_free(mm, (unsigned long *) pmd); } =20 diff --git a/arch/s390/include/asm/tlb.h b/arch/s390/include/asm/tlb.h index 907d57a68145c..dde847a5be545 100644 --- a/arch/s390/include/asm/tlb.h +++ b/arch/s390/include/asm/tlb.h @@ -102,7 +102,7 @@ static inline void pmd_free_tlb(struct mmu_gather *tlb,= pmd_t *pmd, { if (mm_pmd_folded(tlb->mm)) return; - pagetable_pmd_dtor(virt_to_ptdesc(pmd)); + pagetable_dtor(virt_to_ptdesc(pmd)); __tlb_adjust_range(tlb, address, PAGE_SIZE); tlb->mm->context.flush_mm =3D 1; tlb->freed_tables =3D 1; @@ -122,7 +122,7 @@ static inline void p4d_free_tlb(struct mmu_gather *tlb,= p4d_t *p4d, { if (mm_p4d_folded(tlb->mm)) return; - pagetable_p4d_dtor(virt_to_ptdesc(p4d)); + pagetable_dtor(virt_to_ptdesc(p4d)); __tlb_adjust_range(tlb, address, PAGE_SIZE); tlb->mm->context.flush_mm =3D 1; tlb->freed_tables =3D 1; @@ -141,7 +141,7 @@ static inline void pud_free_tlb(struct mmu_gather *tlb,= pud_t *pud, { if (mm_pud_folded(tlb->mm)) return; - pagetable_pud_dtor(virt_to_ptdesc(pud)); + pagetable_dtor(virt_to_ptdesc(pud)); tlb->mm->context.flush_mm =3D 1; tlb->freed_tables =3D 1; tlb->cleared_p4ds =3D 1; diff --git a/arch/s390/mm/pgalloc.c b/arch/s390/mm/pgalloc.c index 58696a0c4e4ac..569de24d33761 100644 --- a/arch/s390/mm/pgalloc.c +++ b/arch/s390/mm/pgalloc.c @@ -182,7 +182,7 @@ unsigned long *page_table_alloc(struct mm_struct *mm) =20 static void pagetable_pte_dtor_free(struct ptdesc *ptdesc) { - pagetable_pte_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); } =20 diff --git a/arch/sh/include/asm/pgalloc.h b/arch/sh/include/asm/pgalloc.h index 5d8577ab15911..96d938fdf2244 100644 --- a/arch/sh/include/asm/pgalloc.h +++ b/arch/sh/include/asm/pgalloc.h @@ -34,7 +34,7 @@ static inline void pmd_populate(struct mm_struct *mm, pmd= _t *pmd, =20 #define __pte_free_tlb(tlb, pte, addr) \ do { \ - pagetable_pte_dtor(page_ptdesc(pte)); \ + pagetable_dtor(page_ptdesc(pte)); \ tlb_remove_page_ptdesc((tlb), (page_ptdesc(pte))); \ } while (0) =20 diff --git a/arch/sparc/mm/init_64.c b/arch/sparc/mm/init_64.c index 21f8cbbd0581c..05882bca5b732 100644 --- a/arch/sparc/mm/init_64.c +++ b/arch/sparc/mm/init_64.c @@ -2915,7 +2915,7 @@ static void __pte_free(pgtable_t pte) { struct ptdesc *ptdesc =3D virt_to_ptdesc(pte); =20 - pagetable_pte_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); } =20 diff --git a/arch/sparc/mm/srmmu.c b/arch/sparc/mm/srmmu.c index 9df51a62333d6..e3a72c884b867 100644 --- a/arch/sparc/mm/srmmu.c +++ b/arch/sparc/mm/srmmu.c @@ -372,7 +372,7 @@ void pte_free(struct mm_struct *mm, pgtable_t ptep) page =3D pfn_to_page(__nocache_pa((unsigned long)ptep) >> PAGE_SHIFT); spin_lock(&mm->page_table_lock); if (page_ref_dec_return(page) =3D=3D 1) - pagetable_pte_dtor(page_ptdesc(page)); + pagetable_dtor(page_ptdesc(page)); spin_unlock(&mm->page_table_lock); =20 srmmu_free_nocache(ptep, SRMMU_PTE_TABLE_SIZE); diff --git a/arch/um/include/asm/pgalloc.h b/arch/um/include/asm/pgalloc.h index 04fb4e6969a46..f0af23c3aeb2b 100644 --- a/arch/um/include/asm/pgalloc.h +++ b/arch/um/include/asm/pgalloc.h @@ -27,7 +27,7 @@ extern pgd_t *pgd_alloc(struct mm_struct *); =20 #define __pte_free_tlb(tlb, pte, address) \ do { \ - pagetable_pte_dtor(page_ptdesc(pte)); \ + pagetable_dtor(page_ptdesc(pte)); \ tlb_remove_page_ptdesc((tlb), (page_ptdesc(pte))); \ } while (0) =20 @@ -35,7 +35,7 @@ do { \ =20 #define __pmd_free_tlb(tlb, pmd, address) \ do { \ - pagetable_pmd_dtor(virt_to_ptdesc(pmd)); \ + pagetable_dtor(virt_to_ptdesc(pmd)); \ tlb_remove_page_ptdesc((tlb), virt_to_ptdesc(pmd)); \ } while (0) =20 @@ -43,7 +43,7 @@ do { \ =20 #define __pud_free_tlb(tlb, pud, address) \ do { \ - pagetable_pud_dtor(virt_to_ptdesc(pud)); \ + pagetable_dtor(virt_to_ptdesc(pud)); \ tlb_remove_page_ptdesc((tlb), virt_to_ptdesc(pud)); \ } while (0) =20 diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c index 3d6e84da45b24..a6cd9660e29ec 100644 --- a/arch/x86/mm/pgtable.c +++ b/arch/x86/mm/pgtable.c @@ -60,7 +60,7 @@ early_param("userpte", setup_userpte); =20 void ___pte_free_tlb(struct mmu_gather *tlb, struct page *pte) { - pagetable_pte_dtor(page_ptdesc(pte)); + pagetable_dtor(page_ptdesc(pte)); paravirt_release_pte(page_to_pfn(pte)); paravirt_tlb_remove_table(tlb, pte); } @@ -77,7 +77,7 @@ void ___pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd) #ifdef CONFIG_X86_PAE tlb->need_flush_all =3D 1; #endif - pagetable_pmd_dtor(ptdesc); + pagetable_dtor(ptdesc); paravirt_tlb_remove_table(tlb, ptdesc_page(ptdesc)); } =20 @@ -86,7 +86,7 @@ void ___pud_free_tlb(struct mmu_gather *tlb, pud_t *pud) { struct ptdesc *ptdesc =3D virt_to_ptdesc(pud); =20 - pagetable_pud_dtor(ptdesc); + pagetable_dtor(ptdesc); paravirt_release_pud(__pa(pud) >> PAGE_SHIFT); paravirt_tlb_remove_table(tlb, virt_to_page(pud)); } @@ -96,7 +96,7 @@ void ___p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d) { struct ptdesc *ptdesc =3D virt_to_ptdesc(p4d); =20 - pagetable_p4d_dtor(ptdesc); + pagetable_dtor(ptdesc); paravirt_release_p4d(__pa(p4d) >> PAGE_SHIFT); paravirt_tlb_remove_table(tlb, virt_to_page(p4d)); } @@ -233,7 +233,7 @@ static void free_pmds(struct mm_struct *mm, pmd_t *pmds= [], int count) if (pmds[i]) { ptdesc =3D virt_to_ptdesc(pmds[i]); =20 - pagetable_pmd_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); mm_dec_nr_pmds(mm); } @@ -867,7 +867,7 @@ int pud_free_pmd_page(pud_t *pud, unsigned long addr) =20 free_page((unsigned long)pmd_sv); =20 - pagetable_pmd_dtor(virt_to_ptdesc(pmd)); + pagetable_dtor(virt_to_ptdesc(pmd)); free_page((unsigned long)pmd); =20 return 1; diff --git a/include/asm-generic/pgalloc.h b/include/asm-generic/pgalloc.h index bb482eeca0c3e..4afb346eae255 100644 --- a/include/asm-generic/pgalloc.h +++ b/include/asm-generic/pgalloc.h @@ -109,7 +109,7 @@ static inline void pte_free(struct mm_struct *mm, struc= t page *pte_page) { struct ptdesc *ptdesc =3D page_ptdesc(pte_page); =20 - pagetable_pte_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); } =20 @@ -153,7 +153,7 @@ static inline void pmd_free(struct mm_struct *mm, pmd_t= *pmd) struct ptdesc *ptdesc =3D virt_to_ptdesc(pmd); =20 BUG_ON((unsigned long)pmd & (PAGE_SIZE-1)); - pagetable_pmd_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); } #endif @@ -202,7 +202,7 @@ static inline void __pud_free(struct mm_struct *mm, pud= _t *pud) struct ptdesc *ptdesc =3D virt_to_ptdesc(pud); =20 BUG_ON((unsigned long)pud & (PAGE_SIZE-1)); - pagetable_pud_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); } =20 @@ -248,7 +248,7 @@ static inline void __p4d_free(struct mm_struct *mm, p4d= _t *p4d) struct ptdesc *ptdesc =3D virt_to_ptdesc(p4d); =20 BUG_ON((unsigned long)p4d & (PAGE_SIZE-1)); - pagetable_p4d_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); } =20 diff --git a/include/linux/mm.h b/include/linux/mm.h index 5d82f42ddd5cc..cad11fa10c192 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2992,6 +2992,15 @@ static inline bool ptlock_init(struct ptdesc *ptdesc= ) { return true; } static inline void ptlock_free(struct ptdesc *ptdesc) {} #endif /* defined(CONFIG_SPLIT_PTE_PTLOCKS) */ =20 +static inline void pagetable_dtor(struct ptdesc *ptdesc) +{ + struct folio *folio =3D ptdesc_folio(ptdesc); + + ptlock_free(ptdesc); + __folio_clear_pgtable(folio); + lruvec_stat_sub_folio(folio, NR_PAGETABLE); +} + static inline bool pagetable_pte_ctor(struct ptdesc *ptdesc) { struct folio *folio =3D ptdesc_folio(ptdesc); @@ -3003,15 +3012,6 @@ static inline bool pagetable_pte_ctor(struct ptdesc = *ptdesc) return true; } =20 -static inline void pagetable_pte_dtor(struct ptdesc *ptdesc) -{ - struct folio *folio =3D ptdesc_folio(ptdesc); - - ptlock_free(ptdesc); - __folio_clear_pgtable(folio); - lruvec_stat_sub_folio(folio, NR_PAGETABLE); -} - pte_t *___pte_offset_map(pmd_t *pmd, unsigned long addr, pmd_t *pmdvalp); static inline pte_t *__pte_offset_map(pmd_t *pmd, unsigned long addr, pmd_t *pmdvalp) @@ -3088,14 +3088,6 @@ static inline bool pmd_ptlock_init(struct ptdesc *pt= desc) return ptlock_init(ptdesc); } =20 -static inline void pmd_ptlock_free(struct ptdesc *ptdesc) -{ -#ifdef CONFIG_TRANSPARENT_HUGEPAGE - VM_BUG_ON_PAGE(ptdesc->pmd_huge_pte, ptdesc_page(ptdesc)); -#endif - ptlock_free(ptdesc); -} - #define pmd_huge_pte(mm, pmd) (pmd_ptdesc(pmd)->pmd_huge_pte) =20 #else @@ -3106,7 +3098,6 @@ static inline spinlock_t *pmd_lockptr(struct mm_struc= t *mm, pmd_t *pmd) } =20 static inline bool pmd_ptlock_init(struct ptdesc *ptdesc) { return true; } -static inline void pmd_ptlock_free(struct ptdesc *ptdesc) {} =20 #define pmd_huge_pte(mm, pmd) ((mm)->pmd_huge_pte) =20 @@ -3131,15 +3122,6 @@ static inline bool pagetable_pmd_ctor(struct ptdesc = *ptdesc) return true; } =20 -static inline void pagetable_pmd_dtor(struct ptdesc *ptdesc) -{ - struct folio *folio =3D ptdesc_folio(ptdesc); - - pmd_ptlock_free(ptdesc); - __folio_clear_pgtable(folio); - lruvec_stat_sub_folio(folio, NR_PAGETABLE); -} - /* * No scalability reason to split PUD locks yet, but follow the same patte= rn * as the PMD locks to make it easier if we decide to. The VM should not = be @@ -3167,14 +3149,6 @@ static inline void pagetable_pud_ctor(struct ptdesc = *ptdesc) lruvec_stat_add_folio(folio, NR_PAGETABLE); } =20 -static inline void pagetable_pud_dtor(struct ptdesc *ptdesc) -{ - struct folio *folio =3D ptdesc_folio(ptdesc); - - __folio_clear_pgtable(folio); - lruvec_stat_sub_folio(folio, NR_PAGETABLE); -} - static inline void pagetable_p4d_ctor(struct ptdesc *ptdesc) { struct folio *folio =3D ptdesc_folio(ptdesc); @@ -3183,14 +3157,6 @@ static inline void pagetable_p4d_ctor(struct ptdesc = *ptdesc) lruvec_stat_add_folio(folio, NR_PAGETABLE); } =20 -static inline void pagetable_p4d_dtor(struct ptdesc *ptdesc) -{ - struct folio *folio =3D ptdesc_folio(ptdesc); - - __folio_clear_pgtable(folio); - lruvec_stat_sub_folio(folio, NR_PAGETABLE); -} - extern void __init pagecache_init(void); extern void free_initmem(void); =20 diff --git a/mm/memory.c b/mm/memory.c index 9423967b24180..ad871e564568b 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -7051,7 +7051,8 @@ bool ptlock_alloc(struct ptdesc *ptdesc) =20 void ptlock_free(struct ptdesc *ptdesc) { - kmem_cache_free(page_ptl_cachep, ptdesc->ptl); + if (ptdesc->ptl) + kmem_cache_free(page_ptl_cachep, ptdesc->ptl); } #endif =20 --=20 2.20.1 From nobody Wed Dec 17 15:51:21 2025 Received: from mail-pl1-f174.google.com (mail-pl1-f174.google.com [209.85.214.174]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4FE271A9B5C for ; Wed, 8 Jan 2025 07:01:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.174 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1736319682; cv=none; b=uHJybGVWuMdfRRocPltsQDjb722I0bHTU4FYdxDGwxDNbg+EK7/gNuaRA1/A1hxSVP0cVC8ivDfOHeMBhCddT6hHbBR5ebpeYWpKZ0EL6GEYDFM9b8oX/wAtRFcMA7Ekh9WDP5EmSKhWAf+czDyfkcPgfakEUiW+ioXLgvEsAQM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1736319682; c=relaxed/simple; bh=wfTgGSVhXTkiP60cV7XgYAcbBAoscYS/kqOY2RPeVxA=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=JUF6ENS0LTAB74MAi9Y43fdyDr5fA57Qls56HTGn5yY8aduN8QOO2FgicZGUwoqRKtApYrxn5AHvW3qwc2MePb9ZrnX7rRqx7o2bdgzXX7px+R1AKBrxWiQWgYoISqJPosSQaqb/AFmwkB3gpA9wqfyDp6vvEE2PRASV3AemvVo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=jzhuXRQy; arc=none smtp.client-ip=209.85.214.174 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="jzhuXRQy" Received: by mail-pl1-f174.google.com with SMTP id d9443c01a7336-2163dc5155fso239942795ad.0 for ; Tue, 07 Jan 2025 23:01:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1736319680; x=1736924480; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=xRYydr57EpvJJkVepk/VS3bq1fMuYMijkZdKL5zejaA=; b=jzhuXRQygiP56bBOGwYPpbPtA5GEJz1GaN11tiBhJYnUV114ySJcMCAR4Vl+T6q7iB iO9qveGtmAH/WLYpOnkh6/dQ4fWRdvMBfQkgqPlto4qRGfb24KnxiIpSTcAjkUG5xJsW 6vcIMFe9+GK1j0KNvpwYHFkFKfOIdYXc7cJOcNkjfpRMruUxLep7SGgaYlmoaWVVHOLz pL7s9qE7xJ9WxO4pGKdd71605tq/4DHIUMhLXggZULfR7eGOU/yxQ09VIcxeQyYh3b10 KrZxsMr40DPAULhPtjk37jYc3p6WqftsSh5wCMycWAPG2NO7InOzihDWbXycUDdm0j4K TlCg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736319680; x=1736924480; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=xRYydr57EpvJJkVepk/VS3bq1fMuYMijkZdKL5zejaA=; b=njndpXIQ6lCW56YS415OzMsQl6JTZzdKG1Jd9ILAt7fqEaTPI5DMA9+m9QDbjy29n1 l+jsULTQvBADBGqioLWsaBUa0iqPjeLUiDoGkgnb4h7ggXDthFiZM0LOfJMRMpypbxmK Dq5ZGgXMrOP5pyRBp4W2CTPQW778MKqdJVczaMx7gW+Mj8rD1G9ourKA+A7s3C12FY7u fHlmq5QWrtUiMjxwdG9iVkjEYQYo5nb/xDJ2V2sHWTWyI2y69Tb+TzRQiDnngmkQ4Nje b9MHV/zbalhU1jEZtE1OHNXGc+UbtB7miRZZO5dSiTwYwHdIghgM/wv5qiWxJnU1tzn4 M+gQ== X-Forwarded-Encrypted: i=1; AJvYcCVBrUFvuuIltoIwVSNZePCJTQ4GfIHXpj2yD+VboAGZE42H5O9K08gjlpauG9/ANV8G1iAiTVvhTzCWamE=@vger.kernel.org X-Gm-Message-State: AOJu0YxwwORMh/v7AgwAp1Y6ybTfQIVa+kmYP2Cf0aAnFd5lqOkueib1 QLT4smsa7bZLJQpfja0ni839/Vfsfsk2qIGolYMGwwG21iwD1fZtevqnWK3hD+o= X-Gm-Gg: ASbGncsyklcKas2s80XvlQAyn5zFaG99cEKH10P3jemKgY7CJ1b0veCXP0o59Xr0Spy fueB1n9GbYH28MqATRaXmBpZ9yt1CEqH8Ij3PBc3uDIycmkLKwDVrEsU8pOrKG3YkV+T/C1+x1S gEggK5YtupDX6bjlk1O6767KhqpnGz3X48pCDK4ih5SHEwEsZJ065gEPOwkrWjt4C6DryzolZqZ e2suZvgJwNeSHH5XlJnDjzaA0dVkv40gcSdzmYKFqnjmJWerarXypJbN3+RfzFSwUmjB1mhAgVt fVJV/lfEE770uzAIccCGjVMetUg= X-Google-Smtp-Source: AGHT+IGA+MEO+KKVHOehVcxKdAaP4kVFc+4x/MGW5FidU7ZYZhG+ejtfRbHdQiRj7WdzfqmV0LUjgQ== X-Received: by 2002:a17:902:d352:b0:215:8847:435c with SMTP id d9443c01a7336-21a83f4b133mr25527805ad.12.1736319679689; Tue, 07 Jan 2025 23:01:19 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.244]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-219dca023a3sm320067275ad.250.2025.01.07.23.01.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 07 Jan 2025 23:01:19 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, agordeev@linux.ibm.com, kevin.brodsky@arm.com, alex@ghiti.fr, andreas@gaisler.com, palmer@dabbelt.com, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com, arnd@arndb.de, will@kernel.org, aneesh.kumar@kernel.org, npiggin@gmail.com, dave.hansen@linux.intel.com, rppt@kernel.org, ryan.roberts@arm.com Cc: linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, linux-arch@vger.kernel.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, loongarch@lists.linux.dev, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-sh@vger.kernel.org, linux-um@lists.infradead.org, Qi Zheng Subject: [PATCH v5 08/17] arm: pgtable: move pagetable_dtor() to __tlb_remove_table() Date: Wed, 8 Jan 2025 14:57:24 +0800 Message-Id: <327b4b8990729edd4ce97d9d5acbdaff2d9fa1d1.1736317725.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Move pagetable_dtor() to __tlb_remove_table(), so that ptlock and page table pages can be freed together (regardless of whether RCU is used). This prevents the use-after-free problem where the ptlock is freed immediately but the page table pages is freed later via RCU. Page tables shouldn't have swap cache, so use pagetable_free() instead of free_page_and_swap_cache() to free page table pages. Signed-off-by: Qi Zheng Suggested-by: Peter Zijlstra (Intel) Reviewed-by: Kevin Brodsky Cc: linux-arm-kernel@lists.infradead.org --- arch/arm/include/asm/tlb.h | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/arch/arm/include/asm/tlb.h b/arch/arm/include/asm/tlb.h index ef79bf1e8563f..59854c6b97bc5 100644 --- a/arch/arm/include/asm/tlb.h +++ b/arch/arm/include/asm/tlb.h @@ -26,12 +26,14 @@ =20 #else /* !CONFIG_MMU */ =20 -#include #include =20 static inline void __tlb_remove_table(void *_table) { - free_page_and_swap_cache((struct page *)_table); + struct ptdesc *ptdesc =3D (struct ptdesc *)_table; + + pagetable_dtor(ptdesc); + pagetable_free(ptdesc); } =20 #include @@ -41,7 +43,9 @@ __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pte, uns= igned long addr) { struct ptdesc *ptdesc =3D page_ptdesc(pte); =20 +#ifndef CONFIG_MMU_GATHER_TABLE_FREE pagetable_dtor(ptdesc); +#endif =20 #ifndef CONFIG_ARM_LPAE /* @@ -61,7 +65,6 @@ __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmdp, unsig= ned long addr) #ifdef CONFIG_ARM_LPAE struct ptdesc *ptdesc =3D virt_to_ptdesc(pmdp); =20 - pagetable_dtor(ptdesc); tlb_remove_ptdesc(tlb, ptdesc); #endif } --=20 2.20.1 From nobody Wed Dec 17 15:51:21 2025 Received: from mail-pl1-f172.google.com (mail-pl1-f172.google.com [209.85.214.172]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AFC311A9B58 for ; Wed, 8 Jan 2025 07:01:35 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1736319697; cv=none; b=qYpEWF+OBSN/uXyccb8C2CTNjUEKomwVLVGj0T2L5nBr9D6CttjRipT5yudiRB4Hy0bj/rtAOl01FmJqbZV39OhH+bQS8JUdJSbAOFBW5WWGU2+wi9MhAxeBOlEbWSSqR5v/l/xdCBya4Wxas9O6AJLoAAehXMW0J4MyauQF8eU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1736319697; c=relaxed/simple; bh=kKEIUjlBJNltvHAujR36Eou/m03YkuA8sY1cVhwRw1I=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=iNaOdHr0bxM1nWlOF+90uWvuW6E3M4zUDa+dvfJCp3AHACwtSuVIUdy/M8zepX6L5DbjLylSZYWGUxVRQsBUGNDdFeRHECd6DO2eIvdoHfx/eNMUw5bI6dCDHQ2nuggb4Ws9QMycdWuoPh7txRW6JhUVYvWg5JHxac5rUQEjcFQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=L8ZxJxdm; arc=none smtp.client-ip=209.85.214.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="L8ZxJxdm" Received: by mail-pl1-f172.google.com with SMTP id d9443c01a7336-21a7ed0155cso22839135ad.3 for ; Tue, 07 Jan 2025 23:01:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1736319695; x=1736924495; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=HOmYFC5rDCHPndkDeYhMazAtFRSG5eQpr+EJP/iMn+M=; b=L8ZxJxdmY6LNQI27QQGrNiHhtgxy/3FYuT66k5rt7mBuNf3HimMV/K2x3UjXSqzB1F priRkl2qgbkp0jS1msuRSsPs81zcmwZH2DNPzltbq78j5MJfhGtInl7UEZn26SVu72KG oh8dvuGze25QJXFrKmhuxnH8zLIUvHVxDl0RZy2lrl3Qlv5eM2HoQTfxXi1UmVUiAbS9 hUieP8dEztNcCY2yCtCRlyvQz+m9HOxLvv9EkZkKAEsX7VOZ5ebMF1bNf8sCPrc6xjmS 5j6wPMG5cftqmF1FC1uCahcnrkx5dLyY7TpJdJhVqUBCJ4vsQNbZsmaC5S+YDYz8XfqT RE7Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736319695; x=1736924495; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=HOmYFC5rDCHPndkDeYhMazAtFRSG5eQpr+EJP/iMn+M=; b=qtfWmrdiEKJCN2fQZQO7vUUWvCsXC5g3YntIKxSu+0v/B5/U+od3ek9gGEMiN6CtIF wY35VkDdLEjK9f2/7FvS2oR3UV9PL7ol75rZs8AxVFBMBLR/wN8OG4dYxaNqyP0xudp2 5NOxsV+718G4taoZaSTRu3TOrEMjLLw8IPWKwkgHcLUPmA/ezV2/S3RZW5NlVGsbZcax 9BlfDSOhyq70a1jMyLequcLeJmiyuhEIhL4xKyaiGJDRyCSdTYpAcPFnAskEsDfE6Nwh rDxWCfMlFqQQPXTgTITNSBv38g9MgpkkuYbapBPvg5dGXQ8cT1lXX1ZydUpTYER7DM5l 4UrQ== X-Forwarded-Encrypted: i=1; AJvYcCU1wXQ7PJagUM0EBGeC9o90wCZaMLnq8/+KAML7aHTxECo6L6nAHD+r6yNEEJTwSPh+2R0iOJsBurhAv+w=@vger.kernel.org X-Gm-Message-State: AOJu0Yy4/vsm3BqCs6BV/67GN4w3EkKW4oVPSvGjimhkyft8KNwa6ZEV 5eBlWFVhCKRfQ1Uc/2LkFrqIr3BUwr43wTNAbF/xksSkJp53phLxEsLRx7imZF8= X-Gm-Gg: ASbGncv+ggX3kqb43OtRmGwpiUAtcjY36D6ESDnsAMhNKhZhffWxSIDr9SR85kvjcRD vtgHOCTSV/KFMeaM3CRWUxQMdSE7DiqQPEYdjkd3liwcHOcgm0IoTutmpChy7EnC+W3kHW7VGPg dEY5H/FaTdd6sZNEjGiVLM0e4oUucU+1pjTkgayf/TibURWU7g0hHnrr61l91qcSWAyGgoh7Hz9 Pwz+KRc7DBvrCLoU9bdeII1iuFAfnG7E2u7F21WPxmEPJhb5knA3EB6v+qgQunP88NElF4bsZHE FeJIs16NKD5uIxsh2nQucHHumzU= X-Google-Smtp-Source: AGHT+IG2O0LK8ylHfVSG3OwO7HgAN7h4l7Hu7AhiZODZJ2StNi5+XwyqKLd4EhZPQDdK0Gl46ps1VQ== X-Received: by 2002:a17:902:d50d:b0:215:7421:27c with SMTP id d9443c01a7336-21a83f696a9mr29561025ad.29.1736319695027; Tue, 07 Jan 2025 23:01:35 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.244]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-219dca023a3sm320067275ad.250.2025.01.07.23.01.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 07 Jan 2025 23:01:34 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, agordeev@linux.ibm.com, kevin.brodsky@arm.com, alex@ghiti.fr, andreas@gaisler.com, palmer@dabbelt.com, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com, arnd@arndb.de, will@kernel.org, aneesh.kumar@kernel.org, npiggin@gmail.com, dave.hansen@linux.intel.com, rppt@kernel.org, ryan.roberts@arm.com Cc: linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, linux-arch@vger.kernel.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, loongarch@lists.linux.dev, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-sh@vger.kernel.org, linux-um@lists.infradead.org, Qi Zheng Subject: [PATCH v5 09/17] arm64: pgtable: move pagetable_dtor() to __tlb_remove_table() Date: Wed, 8 Jan 2025 14:57:25 +0800 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Move pagetable_dtor() to __tlb_remove_table(), so that ptlock and page table pages can be freed together (regardless of whether RCU is used). This prevents the use-after-free problem where the ptlock is freed immediately but the page table pages is freed later via RCU. Page tables shouldn't have swap cache, so use pagetable_free() instead of free_page_and_swap_cache() to free page table pages. Signed-off-by: Qi Zheng Suggested-by: Peter Zijlstra (Intel) Reviewed-by: Kevin Brodsky Cc: linux-arm-kernel@lists.infradead.org Acked-by: Will Deacon --- arch/arm64/include/asm/tlb.h | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-) diff --git a/arch/arm64/include/asm/tlb.h b/arch/arm64/include/asm/tlb.h index 408d0f36a8a8f..93591a80b5bfb 100644 --- a/arch/arm64/include/asm/tlb.h +++ b/arch/arm64/include/asm/tlb.h @@ -9,11 +9,13 @@ #define __ASM_TLB_H =20 #include -#include =20 static inline void __tlb_remove_table(void *_table) { - free_page_and_swap_cache((struct page *)_table); + struct ptdesc *ptdesc =3D (struct ptdesc *)_table; + + pagetable_dtor(ptdesc); + pagetable_free(ptdesc); } =20 #define tlb_flush tlb_flush @@ -82,7 +84,6 @@ static inline void __pte_free_tlb(struct mmu_gather *tlb,= pgtable_t pte, { struct ptdesc *ptdesc =3D page_ptdesc(pte); =20 - pagetable_dtor(ptdesc); tlb_remove_ptdesc(tlb, ptdesc); } =20 @@ -92,7 +93,6 @@ static inline void __pmd_free_tlb(struct mmu_gather *tlb,= pmd_t *pmdp, { struct ptdesc *ptdesc =3D virt_to_ptdesc(pmdp); =20 - pagetable_dtor(ptdesc); tlb_remove_ptdesc(tlb, ptdesc); } #endif @@ -106,7 +106,6 @@ static inline void __pud_free_tlb(struct mmu_gather *tl= b, pud_t *pudp, if (!pgtable_l4_enabled()) return; =20 - pagetable_dtor(ptdesc); tlb_remove_ptdesc(tlb, ptdesc); } #endif @@ -120,7 +119,6 @@ static inline void __p4d_free_tlb(struct mmu_gather *tl= b, p4d_t *p4dp, if (!pgtable_l5_enabled()) return; =20 - pagetable_dtor(ptdesc); tlb_remove_ptdesc(tlb, ptdesc); } #endif --=20 2.20.1 From nobody Wed Dec 17 15:51:21 2025 Received: from mail-pj1-f43.google.com (mail-pj1-f43.google.com [209.85.216.43]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B35E51ACEAD for ; Wed, 8 Jan 2025 07:01:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.43 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1736319714; cv=none; b=fMIGlNgKyvCG1AfGS8+vh+jmckXGxWzuwo0CCcVxjhyA9kyGRJJLDVv0mEnpXuhTMQnOYjHwNxr70We+JWJ8Y+fRYnvfJjqDbIC/bwLe3wxEXknYJJqGwJeNSWiyU2R27iUtpz1vX1zkTzys+rLhE3fq5AOEAWw2AgnXJjXamXY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1736319714; c=relaxed/simple; bh=3/6yQnBUbRs579cSVPuwhDvWyeaNwYaSaUo7i5vWdIA=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=nF67W2EQmjOVXTz3qTSLyvjNpmT6VjfGLZbiWhjByWOtnSqv/+u7dqzliR/vJBoK6e5e2j8ToOUugvkBCUJKDgnUDy+G7ifZhfbLXqYMZsre59hGoWeXvJF9NDNc0K8i5B+lYvKsJgG6K1nGJ0vkMyyp9fg31F0ZKtj3rnixzhE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=XaKDBFUP; arc=none smtp.client-ip=209.85.216.43 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="XaKDBFUP" Received: by mail-pj1-f43.google.com with SMTP id 98e67ed59e1d1-2ee786b3277so17927952a91.1 for ; Tue, 07 Jan 2025 23:01:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1736319711; x=1736924511; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=xJFt+T0ZGRLwl7qj4QUye9dvBL3lzNJEqvkgjrbvemg=; b=XaKDBFUPAfRhHsAMkagqKXgJOg+swTWxwXZaVBAw0tHSaZwd6dUU/3ziaxh0bwYFyU H5nQP0MGq0N9YssWPOolwKrzhRkNTYvQvl1eh4kx8WlNuBvQnQprToCAtB1Phld5W0/6 ooOjqVt14C4kwUipBm31LgV8Puf52P1zmWdBKPgbIwReyegE7ACXz9G4bSSTTnviJDjc 8SN9yRnaQHrOm/ctzBnYSwRHV+4H+E1UxzV7TfV52enIHU2CzcdQyJQVaEU4xkI9uihA jrC0xttgJ6p+1FNADqlF0LApZIfcFVkkcQSsLoqYq3rRlE7fMfdByXI5woPKcUGwAPVV aZKg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736319711; x=1736924511; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=xJFt+T0ZGRLwl7qj4QUye9dvBL3lzNJEqvkgjrbvemg=; b=vbYacO1rlyswOwkz1Sqq85A2qU0E0MoBDC0pXTdv23yTvVozq/roNe/dgsKytBh6s9 n1xYXuTTUd8UujjmmxegPwEF5tBy9x1J/FnCWTkRdyMJhf6VEhZnK8U+QJeywoClfvOW 6jpm5+aazp2PWAL7tXkZMJYnIFNIFjgTL7WVBIKNM+zFJqiXiE+64aQqgIn2R/xg4wGE isY3chNgSpvsham18Tj3PDUD9+LdRf66BXh2Fg9/1h2PVDBRtLeM2JbcCCw1q4Rvhcx4 Nqh0e1REPqawvPKutAiFwB2slzoTzVvjQhBx9C/fk/zxJf8i8NowYtU+K/DQTANn36ax lgCQ== X-Forwarded-Encrypted: i=1; AJvYcCVwS3UUQ0NAcAGWBsPlRDjrM+vRaGcfzTOmzGAV6lmtLHKL8+cg5QTPNcInZ1zryfTrF5RAKCsJ5ihJvwg=@vger.kernel.org X-Gm-Message-State: AOJu0YyQletXM8IAiBuYQsYl4JKPgPH3hfCEJaFyXjbhhuk3WPe+wbyx 0TdKOIvvYYvPZc3qFtijTqljY9qtaYywgbX0OYSUf78jXVXz3PtrN0Sgv7UcdXU= X-Gm-Gg: ASbGnctGMZoVFjDRdPxfiSb5EMpcqo83rLdgcLktpLwyXQ50X0PO3DVxywDPW3ww7yK W4cwqJm6PM3hcXXdITCQX9UXDsA6mxNSt1vPGf52kHDI3uEMZCUc6U9OfSrEd2teGt6xnpoJIBf Uyc/OsJ5YwMQU0RCNB6QGZtYRhwz1Hs54eUHOhek3l86l8UE5lh4gr9Sy6Pvan+rKBH3mkrAZRh GszwFObk1JaWuHmLCuusU7Mht/5YJ9J7GX56MzLmFNKlsXLpdBo5Zx95HBW92GWsqa5mweV2Tvu psSQ9Y0c+4yYgNunbPI8GwHhYfg= X-Google-Smtp-Source: AGHT+IHbJSdEFlMcYFI5dK7LOGyjdYP9+xvlrgbj38Eli2e60ze0npY5GY+Stk/tuNQHFcVXscZRAQ== X-Received: by 2002:a17:90b:2dc2:b0:2ee:863e:9ffc with SMTP id 98e67ed59e1d1-2f548ee5378mr2382141a91.21.1736319711135; Tue, 07 Jan 2025 23:01:51 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.244]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-219dca023a3sm320067275ad.250.2025.01.07.23.01.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 07 Jan 2025 23:01:50 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, agordeev@linux.ibm.com, kevin.brodsky@arm.com, alex@ghiti.fr, andreas@gaisler.com, palmer@dabbelt.com, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com, arnd@arndb.de, will@kernel.org, aneesh.kumar@kernel.org, npiggin@gmail.com, dave.hansen@linux.intel.com, rppt@kernel.org, ryan.roberts@arm.com Cc: linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, linux-arch@vger.kernel.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, loongarch@lists.linux.dev, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-sh@vger.kernel.org, linux-um@lists.infradead.org, Qi Zheng Subject: [PATCH v5 10/17] riscv: pgtable: move pagetable_dtor() to __tlb_remove_table() Date: Wed, 8 Jan 2025 14:57:26 +0800 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Move pagetable_dtor() to __tlb_remove_table(), so that ptlock and page table pages can be freed together (regardless of whether RCU is used). This prevents the use-after-free problem where the ptlock is freed immediately but the page table pages is freed later via RCU. Page tables shouldn't have swap cache, so use pagetable_free() instead of free_page_and_swap_cache() to free page table pages. By the way, move the comment above __tlb_remove_table() to riscv_tlb_remove_ptdesc(), it will be more appropriate. Signed-off-by: Qi Zheng Suggested-by: Peter Zijlstra (Intel) Reviewed-by: Kevin Brodsky Cc: linux-riscv@lists.infradead.org --- arch/riscv/include/asm/pgalloc.h | 38 ++++++++++++++------------------ arch/riscv/include/asm/tlb.h | 14 ++++-------- 2 files changed, 21 insertions(+), 31 deletions(-) diff --git a/arch/riscv/include/asm/pgalloc.h b/arch/riscv/include/asm/pgal= loc.h index b6793c5c99296..c8907b8317115 100644 --- a/arch/riscv/include/asm/pgalloc.h +++ b/arch/riscv/include/asm/pgalloc.h @@ -15,12 +15,22 @@ #define __HAVE_ARCH_PUD_FREE #include =20 +/* + * While riscv platforms with riscv_ipi_for_rfence as true require an IPI = to + * perform TLB shootdown, some platforms with riscv_ipi_for_rfence as fals= e use + * SBI to perform TLB shootdown. To keep software pagetable walkers safe i= n this + * case we switch to RCU based table free (MMU_GATHER_RCU_TABLE_FREE). See= the + * comment below 'ifdef CONFIG_MMU_GATHER_RCU_TABLE_FREE' in include/asm-g= eneric/tlb.h + * for more details. + */ static inline void riscv_tlb_remove_ptdesc(struct mmu_gather *tlb, void *p= t) { - if (riscv_use_sbi_for_rfence()) + if (riscv_use_sbi_for_rfence()) { tlb_remove_ptdesc(tlb, pt); - else + } else { + pagetable_dtor(pt); tlb_remove_page_ptdesc(tlb, pt); + } } =20 static inline void pmd_populate_kernel(struct mm_struct *mm, @@ -97,23 +107,15 @@ static inline void pud_free(struct mm_struct *mm, pud_= t *pud) static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pud, unsigned long addr) { - if (pgtable_l4_enabled) { - struct ptdesc *ptdesc =3D virt_to_ptdesc(pud); - - pagetable_dtor(ptdesc); - riscv_tlb_remove_ptdesc(tlb, ptdesc); - } + if (pgtable_l4_enabled) + riscv_tlb_remove_ptdesc(tlb, virt_to_ptdesc(pud)); } =20 static inline void __p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d, unsigned long addr) { - if (pgtable_l5_enabled) { - struct ptdesc *ptdesc =3D virt_to_ptdesc(p4d); - - pagetable_dtor(ptdesc); + if (pgtable_l5_enabled) riscv_tlb_remove_ptdesc(tlb, virt_to_ptdesc(p4d)); - } } #endif /* __PAGETABLE_PMD_FOLDED */ =20 @@ -142,10 +144,7 @@ static inline pgd_t *pgd_alloc(struct mm_struct *mm) static inline void __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd, unsigned long addr) { - struct ptdesc *ptdesc =3D virt_to_ptdesc(pmd); - - pagetable_dtor(ptdesc); - riscv_tlb_remove_ptdesc(tlb, ptdesc); + riscv_tlb_remove_ptdesc(tlb, virt_to_ptdesc(pmd)); } =20 #endif /* __PAGETABLE_PMD_FOLDED */ @@ -153,10 +152,7 @@ static inline void __pmd_free_tlb(struct mmu_gather *t= lb, pmd_t *pmd, static inline void __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pte, unsigned long addr) { - struct ptdesc *ptdesc =3D page_ptdesc(pte); - - pagetable_dtor(ptdesc); - riscv_tlb_remove_ptdesc(tlb, ptdesc); + riscv_tlb_remove_ptdesc(tlb, page_ptdesc(pte)); } #endif /* CONFIG_MMU */ =20 diff --git a/arch/riscv/include/asm/tlb.h b/arch/riscv/include/asm/tlb.h index 1f6c38420d8e0..ded8724b3c4f7 100644 --- a/arch/riscv/include/asm/tlb.h +++ b/arch/riscv/include/asm/tlb.h @@ -11,19 +11,13 @@ struct mmu_gather; static void tlb_flush(struct mmu_gather *tlb); =20 #ifdef CONFIG_MMU -#include =20 -/* - * While riscv platforms with riscv_ipi_for_rfence as true require an IPI = to - * perform TLB shootdown, some platforms with riscv_ipi_for_rfence as fals= e use - * SBI to perform TLB shootdown. To keep software pagetable walkers safe i= n this - * case we switch to RCU based table free (MMU_GATHER_RCU_TABLE_FREE). See= the - * comment below 'ifdef CONFIG_MMU_GATHER_RCU_TABLE_FREE' in include/asm-g= eneric/tlb.h - * for more details. - */ static inline void __tlb_remove_table(void *table) { - free_page_and_swap_cache(table); + struct ptdesc *ptdesc =3D (struct ptdesc *)table; + + pagetable_dtor(ptdesc); + pagetable_free(ptdesc); } =20 #endif /* CONFIG_MMU */ --=20 2.20.1 From nobody Wed Dec 17 15:51:21 2025 Received: from mail-pl1-f174.google.com (mail-pl1-f174.google.com [209.85.214.174]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7BE211CEAD6 for ; Wed, 8 Jan 2025 07:02:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.174 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1736319729; cv=none; b=gHS/zCk7zLMF/QGzkYMmqbtIRNDHPoBdS0edq/UuALIJeVqBAu12yUSdYDDttO3DgjCckIaa3TpJ9EXAgyFNfySyBbMk47gOWjW4wjWf/WzJNqnINyL1ApEPCCTuZdahox1TX1c2fejc4R4m3LYRt6xpJ0rnfomiRQ5IthGSBxA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1736319729; c=relaxed/simple; bh=JuAN2Ut1lswaL65ulGMlfU8k8Ha5cBPlqTo0nAX0MOM=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=RqUar893VBo47SIyOAyJIO5ubFO/m0bIXkvZ7lqNUVKSiOruBJrVTab8daVSvkwklH2DMDDHwbWPXE/9Q+lCJ48+C5rfU4TKyUyHPVJWFVionLioL9ydrRyM7iVgKi0i1owWr9ULu544ZGnGU7OmDCd5L0hYCJ5gOTJab5/e894= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=b7jrqUl4; arc=none smtp.client-ip=209.85.214.174 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="b7jrqUl4" Received: by mail-pl1-f174.google.com with SMTP id d9443c01a7336-21a1e6fd923so29774065ad.1 for ; Tue, 07 Jan 2025 23:02:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1736319727; x=1736924527; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=0duyzCo9dAwX01bCrGbAFGgpbJyQOIdvGclplTLFMvs=; b=b7jrqUl4fyO36P35+maTH6nESz1UXx4QbMZgwdyF8ruXVbA5JZzw9/RWuFhjg7Oooa KEwo4BjAGWvO7YA4Od3qk3fO3qe0i0bgaHQqfvB/QV4oZqGF9gMKQQVPnNTc9j8w10VZ vd6tuVvbG7k1/pRF6e5ltUxWYVelCBNb8e8mZiI72CVKBbalBt/3m7VvbNkBZonXOen+ WDdgHWZh1C0FwxaAH9Bhk6D2hW7sDNrcBwZJGoHLiJDZuBI6shlgyPfueWxypWJtwot6 s8CDtv0/8tVIUL2IesC60ugd7UVpBSk8yOIWBJ9YpLexT+1DY3CeYNagUADaWi/VYjgw dnAw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736319727; x=1736924527; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=0duyzCo9dAwX01bCrGbAFGgpbJyQOIdvGclplTLFMvs=; b=iHmY/FiSAFnEsLLzs/OPs06DgCsHyXd6MVIciaksDzfDQw/Tojz78lKCDpKYnNWIZe N7VdY34vVKNZDgLZOoMW5112Bo5d6SC3xtdW+LGdI4CSxjDbGhLceFTFlsJH1WyfEL/T S02HBYrfVLC6mw83dOSdnAjbaNWhXoE18ZrM4TMP5ZpemkADNYWe381Uxs1Ox2z+3Z6M 6kDvKhLJ+LccxGE0Mlsm1zxTCGteuVTiXjL9axCPr50vvmTpC4sb2BE59c46kscxFxxT W5U9QeHoxKsBouHarQJnzSWdLtx+AhTD8DCT3U95/elxJbTmiSVBb9QvXqvkGarviNTN C6LA== X-Forwarded-Encrypted: i=1; AJvYcCUO7eVD+UhW2Bwkq5gxrNfIpsrzzDPrqIzt1m4gKAHeBcTDw1PyGK4K45xkubL1w/tT2hOeu9s0HTi8L78=@vger.kernel.org X-Gm-Message-State: AOJu0Ywi5fTctwoNP2N79dG7z8hGmP0UxPdLX2rqK2BU1dFiwxjunU75 gWE2+Z2tjvrCspycI2sTLdmrPuO90WKplDo+4sWm4/sGfemqEZBdgA0zpq7V5B8= X-Gm-Gg: ASbGnct+gEHrbq9kec+2AqGxt3aeomtfMlRBMpAWQlfPbZmQ/qyRy2oUftzxBIwZF34 NmDG6cRyX9R5YqonbR45+Z9einFY6aQ9MSozvbCZBlWM4a1Ol4fuSgVktstlnNxvhJhCcuHEN4n qX5uOtyA9ZF9b9exKgWacU06vFtgpxjOEQd2Ys4T/NLsaL+hs7yZMEDVByDGgW1HdChCkfqcima NKC1PHDFJsKDCYa3TY0QQI3BUWit6s0WxtM9lFPENTMtGf8cOSRlgMgDPTvHN5Zq3qvU/Z/1wY8 ekAFk5doU2qOwKZ7UEyXpMYZCEA= X-Google-Smtp-Source: AGHT+IFHWir1ocwTjoDZn2JicpigLOS4g472SxX6v3Yl/J2b2LHmsbTHHj3CMM+1OfXcdbPj30xuWQ== X-Received: by 2002:a17:902:c406:b0:216:7ee9:21ff with SMTP id d9443c01a7336-21a83fdf307mr27262725ad.49.1736319726731; Tue, 07 Jan 2025 23:02:06 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.244]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-219dca023a3sm320067275ad.250.2025.01.07.23.01.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 07 Jan 2025 23:02:06 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, agordeev@linux.ibm.com, kevin.brodsky@arm.com, alex@ghiti.fr, andreas@gaisler.com, palmer@dabbelt.com, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com, arnd@arndb.de, will@kernel.org, aneesh.kumar@kernel.org, npiggin@gmail.com, dave.hansen@linux.intel.com, rppt@kernel.org, ryan.roberts@arm.com Cc: linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, linux-arch@vger.kernel.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, loongarch@lists.linux.dev, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-sh@vger.kernel.org, linux-um@lists.infradead.org, Qi Zheng Subject: [PATCH v5 11/17] x86: pgtable: convert __tlb_remove_table() to use struct ptdesc Date: Wed, 8 Jan 2025 14:57:27 +0800 Message-Id: <39f60f93143ff77cf5d6b3c3e75af0ffc1480adb.1736317725.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Convert __tlb_remove_table() to use struct ptdesc, which will help to move pagetable_dtor() to __tlb_remove_table(). And page tables shouldn't have swap cache, so use pagetable_free() instead of free_page_and_swap_cache() to free page table pages. Signed-off-by: Qi Zheng Reviewed-by: Kevin Brodsky --- arch/x86/include/asm/tlb.h | 16 +++++++++------- arch/x86/kernel/paravirt.c | 4 +++- arch/x86/mm/pgtable.c | 12 +++++++----- 3 files changed, 19 insertions(+), 13 deletions(-) diff --git a/arch/x86/include/asm/tlb.h b/arch/x86/include/asm/tlb.h index 73f0786181cc9..680ec3d47915a 100644 --- a/arch/x86/include/asm/tlb.h +++ b/arch/x86/include/asm/tlb.h @@ -31,24 +31,26 @@ static inline void tlb_flush(struct mmu_gather *tlb) */ static inline void __tlb_remove_table(void *table) { - free_page_and_swap_cache(table); + struct ptdesc *ptdesc =3D (struct ptdesc *)table; + + pagetable_free(ptdesc); } =20 #ifdef CONFIG_PT_RECLAIM static inline void __tlb_remove_table_one_rcu(struct rcu_head *head) { - struct page *page; + struct ptdesc *ptdesc; =20 - page =3D container_of(head, struct page, rcu_head); - put_page(page); + ptdesc =3D container_of(head, struct ptdesc, pt_rcu_head); + __tlb_remove_table(ptdesc); } =20 static inline void __tlb_remove_table_one(void *table) { - struct page *page; + struct ptdesc *ptdesc; =20 - page =3D table; - call_rcu(&page->rcu_head, __tlb_remove_table_one_rcu); + ptdesc =3D table; + call_rcu(&ptdesc->pt_rcu_head, __tlb_remove_table_one_rcu); } #define __tlb_remove_table_one __tlb_remove_table_one #endif /* CONFIG_PT_RECLAIM */ diff --git a/arch/x86/kernel/paravirt.c b/arch/x86/kernel/paravirt.c index 7bdcf152778c0..dc1ab9301d2c6 100644 --- a/arch/x86/kernel/paravirt.c +++ b/arch/x86/kernel/paravirt.c @@ -62,7 +62,9 @@ void __init native_pv_lock_init(void) #ifndef CONFIG_PT_RECLAIM static void native_tlb_remove_table(struct mmu_gather *tlb, void *table) { - tlb_remove_page(tlb, table); + struct ptdesc *ptdesc =3D (struct ptdesc *)table; + + tlb_remove_page(tlb, ptdesc_page(ptdesc)); } #else static void native_tlb_remove_table(struct mmu_gather *tlb, void *table) diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c index a6cd9660e29ec..f9516024cbe5d 100644 --- a/arch/x86/mm/pgtable.c +++ b/arch/x86/mm/pgtable.c @@ -23,7 +23,9 @@ EXPORT_SYMBOL(physical_mask); static inline void paravirt_tlb_remove_table(struct mmu_gather *tlb, void *table) { - tlb_remove_page(tlb, table); + struct ptdesc *ptdesc =3D (struct ptdesc *)table; + + tlb_remove_page(tlb, ptdesc_page(ptdesc)); } #else static inline @@ -62,7 +64,7 @@ void ___pte_free_tlb(struct mmu_gather *tlb, struct page = *pte) { pagetable_dtor(page_ptdesc(pte)); paravirt_release_pte(page_to_pfn(pte)); - paravirt_tlb_remove_table(tlb, pte); + paravirt_tlb_remove_table(tlb, page_ptdesc(pte)); } =20 #if CONFIG_PGTABLE_LEVELS > 2 @@ -78,7 +80,7 @@ void ___pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd) tlb->need_flush_all =3D 1; #endif pagetable_dtor(ptdesc); - paravirt_tlb_remove_table(tlb, ptdesc_page(ptdesc)); + paravirt_tlb_remove_table(tlb, ptdesc); } =20 #if CONFIG_PGTABLE_LEVELS > 3 @@ -88,7 +90,7 @@ void ___pud_free_tlb(struct mmu_gather *tlb, pud_t *pud) =20 pagetable_dtor(ptdesc); paravirt_release_pud(__pa(pud) >> PAGE_SHIFT); - paravirt_tlb_remove_table(tlb, virt_to_page(pud)); + paravirt_tlb_remove_table(tlb, ptdesc); } =20 #if CONFIG_PGTABLE_LEVELS > 4 @@ -98,7 +100,7 @@ void ___p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d) =20 pagetable_dtor(ptdesc); paravirt_release_p4d(__pa(p4d) >> PAGE_SHIFT); - paravirt_tlb_remove_table(tlb, virt_to_page(p4d)); + paravirt_tlb_remove_table(tlb, ptdesc); } #endif /* CONFIG_PGTABLE_LEVELS > 4 */ #endif /* CONFIG_PGTABLE_LEVELS > 3 */ --=20 2.20.1 From nobody Wed Dec 17 15:51:21 2025 Received: from mail-pj1-f45.google.com (mail-pj1-f45.google.com [209.85.216.45]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AC85D1DFE0F for ; Wed, 8 Jan 2025 07:02:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.45 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1736319745; cv=none; b=mV1nWTEssHl3HUGwxvD8WTOzTvzASgtUlGeONfL9k5q3IgdQS/oS7NF06C+EdOxwW6Xm7m9SoHAYOjKhcDBx17fqUfOFYQQM3k22Ab7/EvziA0ncijp/BsuChgBoWczDDA8KaZZm2u5VeJDZZOsd6kB4XXdoUZQdGNJw2h0l1+w= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1736319745; c=relaxed/simple; bh=mRvQQIHi+4LQZG1SlLZU2M6iRiMg420TF1JW/rjWMH4=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=c2Y2zcLDr0c2GWXA03LPLZBmJCNqDL9ABYL2jgHAMjuUwN/c2kVGKgZ/K3pDwW+t4ds6/XgKBLkPC1uwODEqugzwBO5kquXhKWleFr6coSiabc8GpKJeL1CWkaaPa/nwzxZ+LE5o5+qy372Wmi82Z37kWxD3nOlY6DMXAeRUNWk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=TbTLzel2; arc=none smtp.client-ip=209.85.216.45 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="TbTLzel2" Received: by mail-pj1-f45.google.com with SMTP id 98e67ed59e1d1-2ef714374c0so851505a91.0 for ; Tue, 07 Jan 2025 23:02:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1736319742; x=1736924542; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=DRkfj+TNyCYVuLoiDWLKznOk7pFXzbzTll42IhJYX5s=; b=TbTLzel2ez/j6B2xHXZtXIzj5+Ccj4K+825gfLGpFCgBQYFHgC1dfLx+V1JaXUz29H 5aQRDq6Bo07uJOg2g0jG4PnWMcFwqq1xe63rPDEfLzrQJVBp1qab07GwOVFJYlUbTw0O MaQsnN9R7d/6h1/Fss1ltLZjwV+ujZGU07MyOx1EkfEUV0FlG7ShL08Uurx01Bto7lo0 zfFpDsOMF4XSOrTySbUrUdudYbxreEfsyL9y+eNVMA0AvZ+9Ay50LqrcHa2zgJiNgyiw 6sUAIeSzb/fGiqcE6/9jfEGiK4S/ZPsacJ07L7oXE15M470Foecqv0JPvRDryS27WQQO ko3Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736319742; x=1736924542; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=DRkfj+TNyCYVuLoiDWLKznOk7pFXzbzTll42IhJYX5s=; b=N4KjWuUbZI8kSOCxkDZxuTl/+Jxzc79ZdHASahBGnptWIXYSK4lzneOb2UEDvt9x7K GWL+arpghRCs2pZfvq0fTkH3oWiRfIT7fFTz6ouIw70tYHN3T7EUFGNhJ3X6n9FfUQ4S AbE2al6briKuXk6ci0+ZS2WOMkac9m7kkuvTOZPpAxUR2r8lal38q1wh33DRS0xZ5J71 O9a2m4/vfYwA+FtiD1pvu3EfLn2mVy0F3Sfx9JYiMa/qUJlIU3hEzMiDIl5cj60xgrhd EGZxKexrsqE2zV4G7VptRkbCE+lS8G93gxxzdNnDx47UoWkN89yeidJBeVkIPP42OIWL mm9Q== X-Forwarded-Encrypted: i=1; AJvYcCWqAy6RTdIUgIyxVbKsjtr1DBWo3n0QHHc3VUgEQGwf8LtVFarMVWwmdQ2nSAMZVsgiT+uEpeHxnnunuz0=@vger.kernel.org X-Gm-Message-State: AOJu0Yx3mfu6TWD3KkTx/fltoF2RoWQGda3A2DaA7cC70KZAvFGCDWpk UBr1JVhyVbDbGZ5jfhJ26Jvwqy85PuAMcLSwF0gH/0NEZ33u8VSMp9S+fsacxtA= X-Gm-Gg: ASbGncvjQ4Q1gXOPKN5IQheRFIe/uACacUXOoilVYDXgKroRYyzyreT+TnSicgk8kb4 lt70roW3cfxW3dm2Xc7BDX7LDmCSCejgTjBl734Cuvbj3m6QUaHE5vgCn0FT9HJxzdtM7QvqeO6 N0v7mvW4j9yKkEyxaTn12qwXRsrLiA86cU2CSe6ojcGlg1DzLsaZbu7R1sluiUCDBiUjxHj3Bxb Pp3rwrfr9X55rqlPARPk2KIdkQjHGR3mq/B7sqnSYk4bGpdT/50dJx/UxOUJ8MI2TfsSq5m9UZN Jzi0ulWHKcRRVp9uChNRNI18B2s= X-Google-Smtp-Source: AGHT+IEyar1kSnzqQimnz8qjA8akoGYyLgYubsk1qekCNbk2Sp0If2dwICCDFtvO4jCrehuyEyMuwg== X-Received: by 2002:a17:90a:dfc4:b0:2ee:8cbb:de28 with SMTP id 98e67ed59e1d1-2f548602842mr3182102a91.8.1736319741885; Tue, 07 Jan 2025 23:02:21 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.244]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-219dca023a3sm320067275ad.250.2025.01.07.23.02.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 07 Jan 2025 23:02:21 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, agordeev@linux.ibm.com, kevin.brodsky@arm.com, alex@ghiti.fr, andreas@gaisler.com, palmer@dabbelt.com, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com, arnd@arndb.de, will@kernel.org, aneesh.kumar@kernel.org, npiggin@gmail.com, dave.hansen@linux.intel.com, rppt@kernel.org, ryan.roberts@arm.com Cc: linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, linux-arch@vger.kernel.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, loongarch@lists.linux.dev, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-sh@vger.kernel.org, linux-um@lists.infradead.org, Qi Zheng Subject: [PATCH v5 12/17] x86: pgtable: move pagetable_dtor() to __tlb_remove_table() Date: Wed, 8 Jan 2025 14:57:28 +0800 Message-Id: <27b3cdc8786bebd4f748380bf82f796482718504.1736317725.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Move pagetable_dtor() to __tlb_remove_table(), so that ptlock and page table pages can be freed together (regardless of whether RCU is used). This prevents the use-after-free problem where the ptlock is freed immediately but the page table pages is freed later via RCU. Signed-off-by: Qi Zheng Suggested-by: Peter Zijlstra (Intel) Reviewed-by: Kevin Brodsky Cc: x86@kernel.org --- arch/x86/include/asm/tlb.h | 1 + arch/x86/kernel/paravirt.c | 1 + arch/x86/mm/pgtable.c | 16 ++++------------ 3 files changed, 6 insertions(+), 12 deletions(-) diff --git a/arch/x86/include/asm/tlb.h b/arch/x86/include/asm/tlb.h index 680ec3d47915a..f64730be5ad67 100644 --- a/arch/x86/include/asm/tlb.h +++ b/arch/x86/include/asm/tlb.h @@ -33,6 +33,7 @@ static inline void __tlb_remove_table(void *table) { struct ptdesc *ptdesc =3D (struct ptdesc *)table; =20 + pagetable_dtor(ptdesc); pagetable_free(ptdesc); } =20 diff --git a/arch/x86/kernel/paravirt.c b/arch/x86/kernel/paravirt.c index dc1ab9301d2c6..1ccaa3397a670 100644 --- a/arch/x86/kernel/paravirt.c +++ b/arch/x86/kernel/paravirt.c @@ -64,6 +64,7 @@ static void native_tlb_remove_table(struct mmu_gather *tl= b, void *table) { struct ptdesc *ptdesc =3D (struct ptdesc *)table; =20 + pagetable_dtor(ptdesc); tlb_remove_page(tlb, ptdesc_page(ptdesc)); } #else diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c index f9516024cbe5d..c02aa0427a6a5 100644 --- a/arch/x86/mm/pgtable.c +++ b/arch/x86/mm/pgtable.c @@ -25,6 +25,7 @@ void paravirt_tlb_remove_table(struct mmu_gather *tlb, vo= id *table) { struct ptdesc *ptdesc =3D (struct ptdesc *)table; =20 + pagetable_dtor(ptdesc); tlb_remove_page(tlb, ptdesc_page(ptdesc)); } #else @@ -62,7 +63,6 @@ early_param("userpte", setup_userpte); =20 void ___pte_free_tlb(struct mmu_gather *tlb, struct page *pte) { - pagetable_dtor(page_ptdesc(pte)); paravirt_release_pte(page_to_pfn(pte)); paravirt_tlb_remove_table(tlb, page_ptdesc(pte)); } @@ -70,7 +70,6 @@ void ___pte_free_tlb(struct mmu_gather *tlb, struct page = *pte) #if CONFIG_PGTABLE_LEVELS > 2 void ___pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd) { - struct ptdesc *ptdesc =3D virt_to_ptdesc(pmd); paravirt_release_pmd(__pa(pmd) >> PAGE_SHIFT); /* * NOTE! For PAE, any changes to the top page-directory-pointer-table @@ -79,28 +78,21 @@ void ___pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd) #ifdef CONFIG_X86_PAE tlb->need_flush_all =3D 1; #endif - pagetable_dtor(ptdesc); - paravirt_tlb_remove_table(tlb, ptdesc); + paravirt_tlb_remove_table(tlb, virt_to_ptdesc(pmd)); } =20 #if CONFIG_PGTABLE_LEVELS > 3 void ___pud_free_tlb(struct mmu_gather *tlb, pud_t *pud) { - struct ptdesc *ptdesc =3D virt_to_ptdesc(pud); - - pagetable_dtor(ptdesc); paravirt_release_pud(__pa(pud) >> PAGE_SHIFT); - paravirt_tlb_remove_table(tlb, ptdesc); + paravirt_tlb_remove_table(tlb, virt_to_ptdesc(pud)); } =20 #if CONFIG_PGTABLE_LEVELS > 4 void ___p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d) { - struct ptdesc *ptdesc =3D virt_to_ptdesc(p4d); - - pagetable_dtor(ptdesc); paravirt_release_p4d(__pa(p4d) >> PAGE_SHIFT); - paravirt_tlb_remove_table(tlb, ptdesc); + paravirt_tlb_remove_table(tlb, virt_to_ptdesc(p4d)); } #endif /* CONFIG_PGTABLE_LEVELS > 4 */ #endif /* CONFIG_PGTABLE_LEVELS > 3 */ --=20 2.20.1 From nobody Wed Dec 17 15:51:21 2025 Received: from mail-pl1-f176.google.com (mail-pl1-f176.google.com [209.85.214.176]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id F33741ABEA5 for ; Wed, 8 Jan 2025 07:02:37 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.176 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1736319760; cv=none; b=LDbAdrzeG9Sdyc/wQ0OSgC5a070Bij6Aa73h47uAJrcWEzfLpZW8sdbpgMe+ri78gHZB16VUivqXpxny3Mgol4wZtEsHnvyZ3jOXUK3TPUAwE68ZYbcGVCjlFPKYlKxn5XQWHeeNdjUxGEHr0abkcEQSbvcrb35w8Z7ir9RNADc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1736319760; c=relaxed/simple; bh=Kx0FsgNq46QBGZYoDJIY0xToNCyy11ntqe8BvR51hdk=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=WznZm2I6EZqqDDnXDxRx+S6bC4E66VTak74E4IlXczyjuOv8PyEhnWL94g8zFuWjIp9Tb7fv3I0JfkIQIIk/bs0RgIN0ohqOnPB1nQLktErq+PMN/TYiqRV/vtqCccnPEL9ieHY3Wj4YaZhLUMqq/xbPkLfF9GdqyeVTYsqjgd8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=UwMWDmGP; arc=none smtp.client-ip=209.85.214.176 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="UwMWDmGP" Received: by mail-pl1-f176.google.com with SMTP id d9443c01a7336-2163bd70069so50543475ad.0 for ; Tue, 07 Jan 2025 23:02:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1736319757; x=1736924557; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=oFL9SGbXxYqRlw6G0JwTVWRijc5/6EOrJfB/UywRwlQ=; b=UwMWDmGPRQyV0OPMr+JXUg8p94KGwNfE/gk3eQ2xe0hPBJYYBpIZUQYd/gvaM6nBdb fz1LW/ZqdT4Z1Qj/VklBy8eUQkhnfXhQMcLvZrXkJau51xQwoG3Zgge+9xRvzXcZ1N5K D8KGmNfhI9ZbIjqdqeNe3bGMBVHZ2T/YE+vsGFQQIZzPKoBvX+/EEb+Zv8qBeupm7oHE yFVwxVK+J9c5oqlj/RIr6VmPln3pac7G/qI8Fd0MzPvYdBTboPiiRKYH3H/UP9NROOWP R0s/iBYejwTIr43XmQTP5qn5XauQbh8Ktut2LOMexVEjKLUugB4DGNwJeBmLAgutGyzQ tOvA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736319757; x=1736924557; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=oFL9SGbXxYqRlw6G0JwTVWRijc5/6EOrJfB/UywRwlQ=; b=UKofCQt5ZAqvonsXUNXM/mTijToQq76SOWvFQ/BjkzYvmg+6EQ4zEIBVCaXAT1SRCU WCwzWkTBAPIOHT+xxif6o7Xo5ksN+TlaElHIs/NNiO6jT+XXqM3R/Nut5BBoqxYCZk+u G6FHwchP6xGnRTke7m8AGOPsc8HC4lR8X+XI3OBN7BET8aV40XXbmVSXhu7L4pm/0is+ 9eR8r8EucT88Oez85vuZf1/ZCrDKrz0Wp6ClDEveQimQOTjvIE6by89pKXnNirXiDjT0 AtoSVGsNWxNq14JzLc8fM/ZDVsDcluIOXqSkM0IA2+4Hzr1tX2zIAUGuPgm5az3MWmZP 3WIQ== X-Forwarded-Encrypted: i=1; AJvYcCVSFHvUd++2+UWcJ+aJQkdOu0QceE9QxcCjf5Vq5O/T5fXt1smHvFoDQCAqEyKsJw69Sby+6h2K+f5k+ck=@vger.kernel.org X-Gm-Message-State: AOJu0YxArgHqDtMdVEODI5L7BpP7x7ywAXTfhpwDwoHaOEtv7tkh2Gil ZsSVo27YFZ5OjvEhOPUFI6NJBK31pw1J3SBqZW2CuvacfS60XI4drEBMPsP02Ro= X-Gm-Gg: ASbGncsIzaKhvVxTv4G64be7Pudu9155FbNFvyMRhBF9S+RAbQEYr3hPl6B7Ha8d6ay REHG488u+gdt2qnlgykDkiWsPM7uBmiCQy7OpbgPXNXgqi4ozwB3kffYKx0h4dMnhABSripEYOM hheQy4rgnyX+EUtfBn1bjsywEE7Rd4tXtLXAkNbewIyc5amHHm30UvM/qggz4mTPUyORaIkjy2r AS8Hzy+8YT+QEPxEbOATR9xLilu43Ng/WXrDf/iffwZhATuR9Tlur/b2uc2C1fmEwmM7ZpFOHEI Scn7hjvCrAmwXH6J4kCGhOtniUE= X-Google-Smtp-Source: AGHT+IFBkkoQabtOk36FtfqYc8ZUQnhLYoik+Qreb3ophWoDY5tLIXttH+D63mfzQSCD0xoXOJPU5A== X-Received: by 2002:a17:902:dac8:b0:215:773a:c168 with SMTP id d9443c01a7336-21a83f48cf9mr30903435ad.1.1736319757144; Tue, 07 Jan 2025 23:02:37 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.244]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-219dca023a3sm320067275ad.250.2025.01.07.23.02.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 07 Jan 2025 23:02:36 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, agordeev@linux.ibm.com, kevin.brodsky@arm.com, alex@ghiti.fr, andreas@gaisler.com, palmer@dabbelt.com, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com, arnd@arndb.de, will@kernel.org, aneesh.kumar@kernel.org, npiggin@gmail.com, dave.hansen@linux.intel.com, rppt@kernel.org, ryan.roberts@arm.com Cc: linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, linux-arch@vger.kernel.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, loongarch@lists.linux.dev, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-sh@vger.kernel.org, linux-um@lists.infradead.org, Qi Zheng Subject: [PATCH v5 13/17] s390: pgtable: consolidate PxD and PTE TLB free paths Date: Wed, 8 Jan 2025 14:57:29 +0800 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Call pagetable_dtor() for PMD|PUD|P4D tables just before ptdesc is freed - same as it is done for PTE tables. That allows consolidating TLB free paths for all table types. Signed-off-by: Qi Zheng Suggested-by: Peter Zijlstra (Intel) Reviewed-by: Kevin Brodsky Cc: linux-s390@vger.kernel.org Acked-by: Alexander Gordeev --- arch/s390/include/asm/tlb.h | 3 --- arch/s390/mm/pgalloc.c | 14 ++++---------- 2 files changed, 4 insertions(+), 13 deletions(-) diff --git a/arch/s390/include/asm/tlb.h b/arch/s390/include/asm/tlb.h index dde847a5be545..d5b27a2445c96 100644 --- a/arch/s390/include/asm/tlb.h +++ b/arch/s390/include/asm/tlb.h @@ -102,7 +102,6 @@ static inline void pmd_free_tlb(struct mmu_gather *tlb,= pmd_t *pmd, { if (mm_pmd_folded(tlb->mm)) return; - pagetable_dtor(virt_to_ptdesc(pmd)); __tlb_adjust_range(tlb, address, PAGE_SIZE); tlb->mm->context.flush_mm =3D 1; tlb->freed_tables =3D 1; @@ -122,7 +121,6 @@ static inline void p4d_free_tlb(struct mmu_gather *tlb,= p4d_t *p4d, { if (mm_p4d_folded(tlb->mm)) return; - pagetable_dtor(virt_to_ptdesc(p4d)); __tlb_adjust_range(tlb, address, PAGE_SIZE); tlb->mm->context.flush_mm =3D 1; tlb->freed_tables =3D 1; @@ -141,7 +139,6 @@ static inline void pud_free_tlb(struct mmu_gather *tlb,= pud_t *pud, { if (mm_pud_folded(tlb->mm)) return; - pagetable_dtor(virt_to_ptdesc(pud)); tlb->mm->context.flush_mm =3D 1; tlb->freed_tables =3D 1; tlb->cleared_p4ds =3D 1; diff --git a/arch/s390/mm/pgalloc.c b/arch/s390/mm/pgalloc.c index 569de24d33761..c73b89811a264 100644 --- a/arch/s390/mm/pgalloc.c +++ b/arch/s390/mm/pgalloc.c @@ -180,7 +180,7 @@ unsigned long *page_table_alloc(struct mm_struct *mm) return table; } =20 -static void pagetable_pte_dtor_free(struct ptdesc *ptdesc) +static void pagetable_dtor_free(struct ptdesc *ptdesc) { pagetable_dtor(ptdesc); pagetable_free(ptdesc); @@ -190,20 +190,14 @@ void page_table_free(struct mm_struct *mm, unsigned l= ong *table) { struct ptdesc *ptdesc =3D virt_to_ptdesc(table); =20 - pagetable_pte_dtor_free(ptdesc); + pagetable_dtor_free(ptdesc); } =20 void __tlb_remove_table(void *table) { struct ptdesc *ptdesc =3D virt_to_ptdesc(table); - struct page *page =3D ptdesc_page(ptdesc); =20 - if (compound_order(page) =3D=3D CRST_ALLOC_ORDER) { - /* pmd, pud, or p4d */ - pagetable_free(ptdesc); - return; - } - pagetable_pte_dtor_free(ptdesc); + pagetable_dtor_free(ptdesc); } =20 #ifdef CONFIG_TRANSPARENT_HUGEPAGE @@ -211,7 +205,7 @@ static void pte_free_now(struct rcu_head *head) { struct ptdesc *ptdesc =3D container_of(head, struct ptdesc, pt_rcu_head); =20 - pagetable_pte_dtor_free(ptdesc); + pagetable_dtor_free(ptdesc); } =20 void pte_free_defer(struct mm_struct *mm, pgtable_t pgtable) --=20 2.20.1 From nobody Wed Dec 17 15:51:21 2025 Received: from mail-pl1-f171.google.com (mail-pl1-f171.google.com [209.85.214.171]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id ED3421ACEBB for ; Wed, 8 Jan 2025 07:02:53 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.171 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1736319777; cv=none; b=kmtNRyR2SGStp0LOaF55+zb6DIBeWG7APWvl97OzWpbxfEEqOp0sKrTH9nGfefDx2HeIHmErSIezNxdbicyCu99ZuJR9JKpBk2wCRJjEckEbmnE7hPe0fYo9o0zUAx7ZPOWx1nVJTrpTKc0pJt9s8zx/aAftrAXGzWP+eue4y/A= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1736319777; c=relaxed/simple; bh=3fHJSkC+ZRrYKXKXXlSAUW/WqaHC6KrKlc6jXb0Ndak=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=ez2UTY13l+EZbSMLuiYigb8/4P39BwsG/2kMLTYIYWYXqh5Wp2FCZuSm/RsFlLSqR2C0j/guJWT5VIZu/EHAa82JlX7nSrlhxHCS+JcRusZp1PWJrgaP/isTwyK+4+eknnxXbjeyqXMTwvGdkRhTAtLM3C4S4Q1k9romyE+521o= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=BlMHKzg5; arc=none smtp.client-ip=209.85.214.171 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="BlMHKzg5" Received: by mail-pl1-f171.google.com with SMTP id d9443c01a7336-2163b0c09afso240882285ad.0 for ; Tue, 07 Jan 2025 23:02:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1736319773; x=1736924573; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=DKWQZ+PEOlL8JpVZj7ISn1dZhyFPgkd436O3CmTW6X0=; b=BlMHKzg5JG/eiSIOT/UAoSoU8noIALWj9CszGIHOn92KggCPQDapKxVndZxErkYgGF 2VE3zEedsamZbv7Lf13KU1VoHm4Oa+9Pc6EFBQstArbuaYJBUafcx7kzDGwx3ckcvV6L 2IeDZMpzdaz0LJve8BH7m+CLJ2oKizbjDG1SXOhIETIOwcvqeYWp4K7pJ0AZMFbN12xL 6RUkTyd4AOrMJCujayK21H1SKvIwXaA7oauzCiHwzqTVuo+DuCNM4GnwSa5OHXUMWYOz kTyVxw2SKHo8/uVf5K37pCuixxQgXwYhIqfvLkEmLnx1o7hCRugF/9SnY9i+DVF6Np/u 0LhQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736319773; x=1736924573; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=DKWQZ+PEOlL8JpVZj7ISn1dZhyFPgkd436O3CmTW6X0=; b=l9McicwO2uVwZ2e0NTJriQok00NmwAQfDBodM0A/eoKRwIC95ubFZe2uKjydbIEiv6 fNxYSdtBmKmBrG1e5tJsnwqapkCXPDX9WqkAhra7buMihdh5P0cWBLUA8i4KaWKlOwwS cAU0cfqDrqxnBO8xCJ0FTvuWIeN75hwaNMwu32F8as9ILt3BK9OBVZg+SeXOTvuQCzbO pzeIRAudBQ2yxlwIcYbMfujDq3nOc2Ob7VgYgasJ1aXGly6YXonugL68OsZ3T9l4NSIP oVXnkxO1fF6x5hM+fs3AL6g6JHNAJ1aPB+ewV/DvNjH43J+pKsZ42w/baLy0jV/ijydE EFLQ== X-Forwarded-Encrypted: i=1; AJvYcCU2w6OQzvuLLF7MGtxKk5k7CBc5rtMuRlilYf86QbzTAbWCeEUgtz2+KcffDvWGLKW5wkb2wscO1X4q4j4=@vger.kernel.org X-Gm-Message-State: AOJu0YyJwoY6IoywpPrqRlYhEG/RR/jLtaZ4tcrVJfFotGCqwFCMma76 ccEMnBcotlvtgsFfM8Hq5hXVnNglI6zCWufl3yh4KtsipXkna61oAh3Yk2EC0Gs= X-Gm-Gg: ASbGncvvl+RFOMW7Gk1M7f/BJ3sLyyddtW5oP1u3k0cMnE37bhS/zmBoyqliBSvIUlc odMPwBzTMqsA+nINeiHu5yeME7c6p0vNl6vQBsJkRGFSaprYZ1sfSPlmCj7ucPdFcTu1SHLunKw LdcpLUc7qaoMPbUPFeJOW4GaO2PtPo4iXr6aMQeUFQQ2U/bs+7QGbDJk5v9XF8ChX4PDIOZv3no vZWknfXY68XQrO7no/bOIt5KNWBJaRyM4vUD+vbaq+dO2yqFlQbRsgZSYL8u3Ijp359dtl9IltN HvXSMQ4hK18ROCaLHI+eZSuYgus= X-Google-Smtp-Source: AGHT+IHGzG078/xZKjf5ni4u4VqQSEqesUDiW4uhWhxryubYr9zfzyAKAXvb8eE6PvXFj9EqVM5aSQ== X-Received: by 2002:a17:903:244a:b0:211:fcad:d6ea with SMTP id d9443c01a7336-21a83fcf7a9mr28371525ad.45.1736319773330; Tue, 07 Jan 2025 23:02:53 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.244]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-219dca023a3sm320067275ad.250.2025.01.07.23.02.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 07 Jan 2025 23:02:52 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, agordeev@linux.ibm.com, kevin.brodsky@arm.com, alex@ghiti.fr, andreas@gaisler.com, palmer@dabbelt.com, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com, arnd@arndb.de, will@kernel.org, aneesh.kumar@kernel.org, npiggin@gmail.com, dave.hansen@linux.intel.com, rppt@kernel.org, ryan.roberts@arm.com Cc: linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, linux-arch@vger.kernel.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, loongarch@lists.linux.dev, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-sh@vger.kernel.org, linux-um@lists.infradead.org, Qi Zheng Subject: [PATCH v5 14/17] mm: pgtable: introduce generic __tlb_remove_table() Date: Wed, 8 Jan 2025 14:57:30 +0800 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Several architectures (arm, arm64, riscv and x86) define exactly the same __tlb_remove_table(), just introduce generic __tlb_remove_table() to eliminate these duplications. The s390 __tlb_remove_table() is nearly the same, so also make s390 __tlb_remove_table() version generic. Signed-off-by: Qi Zheng Reviewed-by: Kevin Brodsky Acked-by: Andreas Larsson # sparc Acked-by: Alexander Gordeev # s390 --- arch/arm/include/asm/tlb.h | 9 --------- arch/arm64/include/asm/tlb.h | 7 ------- arch/powerpc/include/asm/tlb.h | 1 + arch/riscv/include/asm/tlb.h | 12 ------------ arch/s390/include/asm/tlb.h | 9 ++++----- arch/s390/mm/pgalloc.c | 7 ------- arch/sparc/include/asm/tlb_64.h | 1 + arch/x86/include/asm/tlb.h | 17 ----------------- include/asm-generic/tlb.h | 15 +++++++++++++-- 9 files changed, 19 insertions(+), 59 deletions(-) diff --git a/arch/arm/include/asm/tlb.h b/arch/arm/include/asm/tlb.h index 59854c6b97bc5..b8eebdb598631 100644 --- a/arch/arm/include/asm/tlb.h +++ b/arch/arm/include/asm/tlb.h @@ -27,15 +27,6 @@ #else /* !CONFIG_MMU */ =20 #include - -static inline void __tlb_remove_table(void *_table) -{ - struct ptdesc *ptdesc =3D (struct ptdesc *)_table; - - pagetable_dtor(ptdesc); - pagetable_free(ptdesc); -} - #include =20 static inline void diff --git a/arch/arm64/include/asm/tlb.h b/arch/arm64/include/asm/tlb.h index 93591a80b5bfb..8d762607285cc 100644 --- a/arch/arm64/include/asm/tlb.h +++ b/arch/arm64/include/asm/tlb.h @@ -10,13 +10,6 @@ =20 #include =20 -static inline void __tlb_remove_table(void *_table) -{ - struct ptdesc *ptdesc =3D (struct ptdesc *)_table; - - pagetable_dtor(ptdesc); - pagetable_free(ptdesc); -} =20 #define tlb_flush tlb_flush static void tlb_flush(struct mmu_gather *tlb); diff --git a/arch/powerpc/include/asm/tlb.h b/arch/powerpc/include/asm/tlb.h index 1ca7d4c4b90db..2058e8d3e0138 100644 --- a/arch/powerpc/include/asm/tlb.h +++ b/arch/powerpc/include/asm/tlb.h @@ -37,6 +37,7 @@ extern void tlb_flush(struct mmu_gather *tlb); */ #define tlb_needs_table_invalidate() radix_enabled() =20 +#define __HAVE_ARCH_TLB_REMOVE_TABLE /* Get the generic bits... */ #include =20 diff --git a/arch/riscv/include/asm/tlb.h b/arch/riscv/include/asm/tlb.h index ded8724b3c4f7..50b63b5c15bd8 100644 --- a/arch/riscv/include/asm/tlb.h +++ b/arch/riscv/include/asm/tlb.h @@ -10,18 +10,6 @@ struct mmu_gather; =20 static void tlb_flush(struct mmu_gather *tlb); =20 -#ifdef CONFIG_MMU - -static inline void __tlb_remove_table(void *table) -{ - struct ptdesc *ptdesc =3D (struct ptdesc *)table; - - pagetable_dtor(ptdesc); - pagetable_free(ptdesc); -} - -#endif /* CONFIG_MMU */ - #define tlb_flush tlb_flush #include =20 diff --git a/arch/s390/include/asm/tlb.h b/arch/s390/include/asm/tlb.h index d5b27a2445c96..f39f8c4723f15 100644 --- a/arch/s390/include/asm/tlb.h +++ b/arch/s390/include/asm/tlb.h @@ -22,7 +22,6 @@ * Pages used for the page tables is a different story. FIXME: more */ =20 -void __tlb_remove_table(void *_table); static inline void tlb_flush(struct mmu_gather *tlb); static inline bool __tlb_remove_page_size(struct mmu_gather *tlb, struct page *page, bool delay_rmap, int page_size); @@ -87,7 +86,7 @@ static inline void pte_free_tlb(struct mmu_gather *tlb, p= gtable_t pte, tlb->cleared_pmds =3D 1; if (mm_alloc_pgste(tlb->mm)) gmap_unlink(tlb->mm, (unsigned long *)pte, address); - tlb_remove_ptdesc(tlb, pte); + tlb_remove_ptdesc(tlb, virt_to_ptdesc(pte)); } =20 /* @@ -106,7 +105,7 @@ static inline void pmd_free_tlb(struct mmu_gather *tlb,= pmd_t *pmd, tlb->mm->context.flush_mm =3D 1; tlb->freed_tables =3D 1; tlb->cleared_puds =3D 1; - tlb_remove_ptdesc(tlb, pmd); + tlb_remove_ptdesc(tlb, virt_to_ptdesc(pmd)); } =20 /* @@ -124,7 +123,7 @@ static inline void p4d_free_tlb(struct mmu_gather *tlb,= p4d_t *p4d, __tlb_adjust_range(tlb, address, PAGE_SIZE); tlb->mm->context.flush_mm =3D 1; tlb->freed_tables =3D 1; - tlb_remove_ptdesc(tlb, p4d); + tlb_remove_ptdesc(tlb, virt_to_ptdesc(p4d)); } =20 /* @@ -142,7 +141,7 @@ static inline void pud_free_tlb(struct mmu_gather *tlb,= pud_t *pud, tlb->mm->context.flush_mm =3D 1; tlb->freed_tables =3D 1; tlb->cleared_p4ds =3D 1; - tlb_remove_ptdesc(tlb, pud); + tlb_remove_ptdesc(tlb, virt_to_ptdesc(pud)); } =20 =20 diff --git a/arch/s390/mm/pgalloc.c b/arch/s390/mm/pgalloc.c index c73b89811a264..3e002dea6278f 100644 --- a/arch/s390/mm/pgalloc.c +++ b/arch/s390/mm/pgalloc.c @@ -193,13 +193,6 @@ void page_table_free(struct mm_struct *mm, unsigned lo= ng *table) pagetable_dtor_free(ptdesc); } =20 -void __tlb_remove_table(void *table) -{ - struct ptdesc *ptdesc =3D virt_to_ptdesc(table); - - pagetable_dtor_free(ptdesc); -} - #ifdef CONFIG_TRANSPARENT_HUGEPAGE static void pte_free_now(struct rcu_head *head) { diff --git a/arch/sparc/include/asm/tlb_64.h b/arch/sparc/include/asm/tlb_6= 4.h index 3037187482db7..1a6e694418e39 100644 --- a/arch/sparc/include/asm/tlb_64.h +++ b/arch/sparc/include/asm/tlb_64.h @@ -33,6 +33,7 @@ void flush_tlb_pending(void); #define tlb_needs_table_invalidate() (false) #endif =20 +#define __HAVE_ARCH_TLB_REMOVE_TABLE #include =20 #endif /* _SPARC64_TLB_H */ diff --git a/arch/x86/include/asm/tlb.h b/arch/x86/include/asm/tlb.h index f64730be5ad67..3858dbf75880e 100644 --- a/arch/x86/include/asm/tlb.h +++ b/arch/x86/include/asm/tlb.h @@ -20,23 +20,6 @@ static inline void tlb_flush(struct mmu_gather *tlb) flush_tlb_mm_range(tlb->mm, start, end, stride_shift, tlb->freed_tables); } =20 -/* - * While x86 architecture in general requires an IPI to perform TLB - * shootdown, enablement code for several hypervisors overrides - * .flush_tlb_others hook in pv_mmu_ops and implements it by issuing - * a hypercall. To keep software pagetable walkers safe in this case we - * switch to RCU based table free (MMU_GATHER_RCU_TABLE_FREE). See the com= ment - * below 'ifdef CONFIG_MMU_GATHER_RCU_TABLE_FREE' in include/asm-generic/t= lb.h - * for more details. - */ -static inline void __tlb_remove_table(void *table) -{ - struct ptdesc *ptdesc =3D (struct ptdesc *)table; - - pagetable_dtor(ptdesc); - pagetable_free(ptdesc); -} - #ifdef CONFIG_PT_RECLAIM static inline void __tlb_remove_table_one_rcu(struct rcu_head *head) { diff --git a/include/asm-generic/tlb.h b/include/asm-generic/tlb.h index 709830274b756..69de47c7ef3c5 100644 --- a/include/asm-generic/tlb.h +++ b/include/asm-generic/tlb.h @@ -153,8 +153,9 @@ * * Useful if your architecture has non-page page directories. * - * When used, an architecture is expected to provide __tlb_remove_table() - * which does the actual freeing of these pages. + * When used, an architecture is expected to provide __tlb_remove_table()= or + * use the generic __tlb_remove_table(), which does the actual freeing of= these + * pages. * * MMU_GATHER_RCU_TABLE_FREE * @@ -207,6 +208,16 @@ struct mmu_table_batch { #define MAX_TABLE_BATCH \ ((PAGE_SIZE - sizeof(struct mmu_table_batch)) / sizeof(void *)) =20 +#ifndef __HAVE_ARCH_TLB_REMOVE_TABLE +static inline void __tlb_remove_table(void *table) +{ + struct ptdesc *ptdesc =3D (struct ptdesc *)table; + + pagetable_dtor(ptdesc); + pagetable_free(ptdesc); +} +#endif + extern void tlb_remove_table(struct mmu_gather *tlb, void *table); =20 #else /* !CONFIG_MMU_GATHER_HAVE_TABLE_FREE */ --=20 2.20.1 From nobody Wed Dec 17 15:51:21 2025 Received: from mail-pl1-f179.google.com (mail-pl1-f179.google.com [209.85.214.179]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4A38F1ACEAD for ; Wed, 8 Jan 2025 07:03:10 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.179 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1736319792; cv=none; b=i/aKoGthEZtNksqA4qYWbrELP6yNafiXtVZsRYTfqoskDhURejlPXlJen2nU/HRpcMEN1EyUTKwxYABBqbu0VgL80kVU9ewINNv4JrwkOmKKFB1qVKtSquTrfXzCWHFqltkpnr5NT9WLI70p1wPotgI0x8tj772kspZDrB0PKhM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1736319792; c=relaxed/simple; bh=gLSOIPe2CJRryLc4wKd5NXWeTP0qwE2dyCNWoD9rNdw=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=a1kjwMrt9n608FX/2ZqLqQY6UYHFOypw/4zV2dv8XZ7Rb1L6l7EojYEGgNqZltpT7XyPZ01E1a2J2FO9AJ4vCXT3JqADthYezbvfV7TUVa4bN4xmYY92fHPTfKG1gFTbw8XosikFFwgp/699fGHlZ7BT1I+Lcf0DuqePaNZSddg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=Bc8F7GKr; arc=none smtp.client-ip=209.85.214.179 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="Bc8F7GKr" Received: by mail-pl1-f179.google.com with SMTP id d9443c01a7336-21a1e6fd923so29795355ad.1 for ; Tue, 07 Jan 2025 23:03:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1736319790; x=1736924590; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=SzpcYcAmR7wIjJMr9qjJ0ueu9TAADg8s+gPSbWL3su4=; b=Bc8F7GKrRjcHB2HIcLoYn88Woo9KON8uOwtbAdUbkiup4eT1T8q3WIrXnhHRt0eiJ7 qkXW464TRApGLYye716AioJBoAkncAu6XIq8mD6AbrhOvou/Ukr9HvylWS35usYh4jzq jX46dY6H5JjJ0+Lc0wcSjBjpZEdLbZoEBFS58NVgzJhlz2v48yD6NpFxqiNtgoRTFm/5 0a0JaCJTkonzNeBpa8VKAB52B1hA5qBdeRcW/W4zaaDV2g611bOyffVFEUYBSmWgy2S+ iXK5sQyPa4Vs0y9zhWP8cgqkGg79a9W0iAdS9g9IICI2gSq3/OeO6CYwyO30Jz9z9W/M x+hw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736319790; x=1736924590; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=SzpcYcAmR7wIjJMr9qjJ0ueu9TAADg8s+gPSbWL3su4=; b=A4JsSDVaLLTAKotpeocbJZJ3sJbw+D7AxO5KsZwIMw7v/ou29dlmhMpOtLJ+zdGyb0 fga0OfE1vyPuHgOBhn1bhrt/fDp1kz9zTnzF2P3dr/7871F91JrXOEk1FAdJ4a/BqdB6 yH621VyHQLonQYMJgUCjklu3qGXtx8Jldj6EQdtfBYovuKHCEy+QHvsbSg7NFlxemtKm OjFqVH8tdBWG5jPPGAqC2XQoBe/4a0Tidx89rlMajRPypz19/oNCH18QPQjW1vZDUbmF IiXONolv1R1OjU53EAnBXgc2F5n6cBGfdfX6Xh3SztZfqfTpPuRJMh9DHuZfCzPQTEyI mjfg== X-Forwarded-Encrypted: i=1; AJvYcCVLZAtEzlwl1f8kkbOX3t7Yz2Z2QQ2sL/BOP+RRI68FoTi/cEuZIJgLi1mgqOGuQmbHdw/rQDcmzYsvWk4=@vger.kernel.org X-Gm-Message-State: AOJu0YxOnj3UzYI5wPmT1Tqxxf+JCTRQLvn4RzR/C3A9jpKsRf0i2Cfv qoLmiGkOQSlIQ228BJBOrBlVZt05D6ElUlPgETWisyJliKG1nI6C/WvovxPtb+Q= X-Gm-Gg: ASbGnctqBVz4D5Fe9OI1nzEgrh1Dtbas9VnrC+8m+zAiTR+p0V77VjNyQ4Fd7+ymt7h bGwnnkHp08J/DiAYIpqVjJUeWn/CKQURq6htbE5d8Oyvi+2pqQ9vAJZBLsQCa6uYZ64Rnc0ZgCH jdR9jpuwku9h5BW7/Wx203ZGC4nCl9BZ9h3nOg8nZohdisihjJHTRL2CvVFmlpJUcLSQyR230qO ndct0bqN1B2znhcLVtBQxN392eWO7q1S2dwbhe7NTs/TSFavBAaK+BuB3uJDl8Eh5cvSonItLW/ dyXRrRnCp/XYRB/Jr/fWdOvv8Kk= X-Google-Smtp-Source: AGHT+IGLH2wnugoAqXpKOjRMOcNm05e1d3DjACkQOqMsLyNNJS4LCZNqibWg6/9XtyaXTuj3byMh6A== X-Received: by 2002:a17:902:e74f:b0:216:7cde:523 with SMTP id d9443c01a7336-21a83f6710dmr31869555ad.32.1736319789694; Tue, 07 Jan 2025 23:03:09 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.244]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-219dca023a3sm320067275ad.250.2025.01.07.23.02.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 07 Jan 2025 23:03:09 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, agordeev@linux.ibm.com, kevin.brodsky@arm.com, alex@ghiti.fr, andreas@gaisler.com, palmer@dabbelt.com, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com, arnd@arndb.de, will@kernel.org, aneesh.kumar@kernel.org, npiggin@gmail.com, dave.hansen@linux.intel.com, rppt@kernel.org, ryan.roberts@arm.com Cc: linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, linux-arch@vger.kernel.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, loongarch@lists.linux.dev, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-sh@vger.kernel.org, linux-um@lists.infradead.org, Qi Zheng Subject: [PATCH v5 15/17] mm: pgtable: completely move pagetable_dtor() to generic tlb_remove_table() Date: Wed, 8 Jan 2025 14:57:31 +0800 Message-Id: <0c733ac867b287ec08190676496d1decebf49da2.1736317725.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" For the generic tlb_remove_table(), it is implemented in the following two forms: 1) CONFIG_MMU_GATHER_TABLE_FREE is enabled tlb_remove_table --> generic __tlb_remove_table() 2) CONFIG_MMU_GATHER_TABLE_FREE is disabled tlb_remove_table --> tlb_remove_page For case 1), the pagetable_dtor() has already been moved to generic __tlb_remove_table(). For case 2), now only arm will call tlb_remove_table()/tlb_remove_ptdesc() when CONFIG_MMU_GATHER_TABLE_FREE is disabled. Let's move pagetable_dtor() completely to generic tlb_remove_table(), so that the architectures can follow more easily. Signed-off-by: Qi Zheng Suggested-by: Kevin Brodsky Reviewed-by: Kevin Brodsky --- arch/arm/include/asm/tlb.h | 4 ---- include/asm-generic/tlb.h | 10 ++++++++-- 2 files changed, 8 insertions(+), 6 deletions(-) diff --git a/arch/arm/include/asm/tlb.h b/arch/arm/include/asm/tlb.h index b8eebdb598631..ea4fbe7b17f6f 100644 --- a/arch/arm/include/asm/tlb.h +++ b/arch/arm/include/asm/tlb.h @@ -34,10 +34,6 @@ __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pte, un= signed long addr) { struct ptdesc *ptdesc =3D page_ptdesc(pte); =20 -#ifndef CONFIG_MMU_GATHER_TABLE_FREE - pagetable_dtor(ptdesc); -#endif - #ifndef CONFIG_ARM_LPAE /* * With the classic ARM MMU, a pte page has two corresponding pmd diff --git a/include/asm-generic/tlb.h b/include/asm-generic/tlb.h index 69de47c7ef3c5..53ae7748f555b 100644 --- a/include/asm-generic/tlb.h +++ b/include/asm-generic/tlb.h @@ -220,14 +220,20 @@ static inline void __tlb_remove_table(void *table) =20 extern void tlb_remove_table(struct mmu_gather *tlb, void *table); =20 -#else /* !CONFIG_MMU_GATHER_HAVE_TABLE_FREE */ +#else /* !CONFIG_MMU_GATHER_TABLE_FREE */ =20 +static inline void tlb_remove_page(struct mmu_gather *tlb, struct page *pa= ge); /* * Without MMU_GATHER_TABLE_FREE the architecture is assumed to have page = based * page directories and we can use the normal page batching to free them. */ -#define tlb_remove_table(tlb, page) tlb_remove_page((tlb), (page)) +static inline void tlb_remove_table(struct mmu_gather *tlb, void *table) +{ + struct page *page =3D (struct page *)table; =20 + pagetable_dtor(page_ptdesc(page)); + tlb_remove_page(tlb, page); +} #endif /* CONFIG_MMU_GATHER_TABLE_FREE */ =20 #ifdef CONFIG_MMU_GATHER_RCU_TABLE_FREE --=20 2.20.1 From nobody Wed Dec 17 15:51:21 2025 Received: from mail-pl1-f169.google.com (mail-pl1-f169.google.com [209.85.214.169]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DE9341A0739 for ; Wed, 8 Jan 2025 07:03:24 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.169 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1736319806; cv=none; b=CN53HjlZw0iNTzT76/kJj+kV1NmRGGaZ94uSn2bkvJgq9Htf1mk9ihQFGfED8xnBqCLUrtvSX2ETtMvvxRHd14xyxmeaIHi09TLKmmvRzlzRSu8xBHh+6d0vCMe8e2LE82KkYlOgRWFrql7Yoa2VVbnU/36pe/SVQgpbvVms3nU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1736319806; c=relaxed/simple; bh=/Z4BDRon0Bevgr5mvsJCh3x7OaoERaQ2cVgkjt99Nxk=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=CEE8b0bFnu2BM4Ek38Ja7mbXW9NX3GY78X+Xs3KGn+8TD9bxKMd4TqKkmbPlgFvW/88IVWkOobEf0bTRBH6otUQIQQd75e+PRauN6IxNfIrSfpy7cT53RjzkgfEGhwpP0lcvoaINlnfvEX2zoU59ef6HnJrmJXej4CHs5CyqkVk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=Bujf4aZk; arc=none smtp.client-ip=209.85.214.169 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="Bujf4aZk" Received: by mail-pl1-f169.google.com with SMTP id d9443c01a7336-2163b0c09afso240888365ad.0 for ; Tue, 07 Jan 2025 23:03:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1736319804; x=1736924604; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=x40aCfPUIT+7f+SOGlgpQ1mZ5+3GERu3xGzgSvQpPds=; b=Bujf4aZkjw72UhClVTmyLsiBn7MRga03gknCk8ch02tYmNn1EcQy2aGfTVul9Yuewg TkJR7hhvPKitum95BdkelwMwuLaaKtKkw+qL1gu+zxtOlRuReYrDeRekypnnhGx3bwSX VeghCEO1Cibcla+ajxtqSM/3LCNY0osW+PgdjK6yl9oxZdT6a05n4fvUcpwTum4C90Gu 4Dqa4X5b9EckXWyvd+9srkK09ugKfoIlUcb4B4lB8uzwsSj9/vVDgFhrs/n84BM39IcA kPoGAj4W6B6lsBxtUKebiUewcFE7zu2wfF5lNOcy0yFeaRNUdEFSyIyZU042WDt/wRTZ lbrg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736319804; x=1736924604; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=x40aCfPUIT+7f+SOGlgpQ1mZ5+3GERu3xGzgSvQpPds=; b=YK2F7D7hSXE9/BpFPF7g3ZBUjeIhbOdEH55hJJ6F6uAB6N/xMStlijcPq9GHOZPRcR WmQv4vyxeh4KKzhoOsd/3jP040I3sfkVTtEgB4gndwTEHlIfMT2pdxQIcQ1QcgqyV1Hw LJX/6TKN0tRz/E8qYC5RVIPTIFpew+ICuom9l6vXhqds+I+bjqY0RczAiop2zhxg8Bm5 +u0meF2gvxWtEPbq5gWg7LSgjmotRv5irXcTMvABQwmLbBFun2lj4Wz7i3QG/U18ttvR c528lvW1pSf0pnxLSzyG89Brwkql6cALTJhWodTtEvMkQ1TeLAyFJKmUWtAMhEhnO24x N5wA== X-Forwarded-Encrypted: i=1; AJvYcCUb/hmna3Ymtye8bG5bDfgsxusO2Fxb3XiocImvLROIWR+DfnRBZ4id7WtIHtO+PS2YzBDCD1oNXigJgPU=@vger.kernel.org X-Gm-Message-State: AOJu0YzpSy1JyYkPqYMr0zuzUegJHhmHCJQLJ/dzRWJiFgORsvfu1k0x W3uqjQLZO7VrucI6NVnmvatz1w2jg51Ta4VbwL2LKLQVNic4kC+seArTGrtaizI= X-Gm-Gg: ASbGncvAsAI88bbJK5rjTBVYCop3zpAD7I7L4dzVRA3URBlzm0qu5Hnkqix60Aa8NLx xo9mU87jA1e8QvPmeoquBr/hn9BC+HfxEEeN47sQr2goocACd0vAbk0sXbdTr028dcsPEoGIs3p D/OkNuiEBmRPSfOYS8qJE3+ImheI6l7Veh76X4CL8hGNGprsqAeuELoNq9UOzdh47EncsHSUWR7 VpPUPA/N4Hs4XNMlXch1AtP2eAzeDOrgYA8vDSvOv8bGRPWQwdtaGk/NMnDTff/shxC3fgTrOGV +lYH6ipufYDpVXouWuSQa8kYpG0= X-Google-Smtp-Source: AGHT+IH8IKObAIjKqBGW3XQDBdiOawtDgYf1Tuj1urKOF9Kp4Be/TSELj5l2Ma8a+iBaK6i0ld/PMQ== X-Received: by 2002:a17:902:da8e:b0:216:410d:4c67 with SMTP id d9443c01a7336-21a83fb1395mr28350885ad.41.1736319804422; Tue, 07 Jan 2025 23:03:24 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.244]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-219dca023a3sm320067275ad.250.2025.01.07.23.03.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 07 Jan 2025 23:03:23 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, agordeev@linux.ibm.com, kevin.brodsky@arm.com, alex@ghiti.fr, andreas@gaisler.com, palmer@dabbelt.com, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com, arnd@arndb.de, will@kernel.org, aneesh.kumar@kernel.org, npiggin@gmail.com, dave.hansen@linux.intel.com, rppt@kernel.org, ryan.roberts@arm.com Cc: linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, linux-arch@vger.kernel.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, loongarch@lists.linux.dev, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-sh@vger.kernel.org, linux-um@lists.infradead.org, Qi Zheng Subject: [PATCH v5 16/17] mm: pgtable: move __tlb_remove_table_one() in x86 to generic file Date: Wed, 8 Jan 2025 14:57:32 +0800 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The __tlb_remove_table_one() in x86 does not contain architecture-specific content, so move it to the generic file. Signed-off-by: Qi Zheng Reviewed-by: Kevin Brodsky --- arch/x86/include/asm/tlb.h | 19 ------------------- mm/mmu_gather.c | 20 ++++++++++++++++++-- 2 files changed, 18 insertions(+), 21 deletions(-) diff --git a/arch/x86/include/asm/tlb.h b/arch/x86/include/asm/tlb.h index 3858dbf75880e..77f52bc1578a7 100644 --- a/arch/x86/include/asm/tlb.h +++ b/arch/x86/include/asm/tlb.h @@ -20,25 +20,6 @@ static inline void tlb_flush(struct mmu_gather *tlb) flush_tlb_mm_range(tlb->mm, start, end, stride_shift, tlb->freed_tables); } =20 -#ifdef CONFIG_PT_RECLAIM -static inline void __tlb_remove_table_one_rcu(struct rcu_head *head) -{ - struct ptdesc *ptdesc; - - ptdesc =3D container_of(head, struct ptdesc, pt_rcu_head); - __tlb_remove_table(ptdesc); -} - -static inline void __tlb_remove_table_one(void *table) -{ - struct ptdesc *ptdesc; - - ptdesc =3D table; - call_rcu(&ptdesc->pt_rcu_head, __tlb_remove_table_one_rcu); -} -#define __tlb_remove_table_one __tlb_remove_table_one -#endif /* CONFIG_PT_RECLAIM */ - static inline void invlpg(unsigned long addr) { asm volatile("invlpg (%0)" ::"r" (addr) : "memory"); diff --git a/mm/mmu_gather.c b/mm/mmu_gather.c index 1e21022bcf339..7aa6f18c500b2 100644 --- a/mm/mmu_gather.c +++ b/mm/mmu_gather.c @@ -311,13 +311,29 @@ static inline void tlb_table_invalidate(struct mmu_ga= ther *tlb) } } =20 -#ifndef __tlb_remove_table_one +#ifdef CONFIG_PT_RECLAIM +static inline void __tlb_remove_table_one_rcu(struct rcu_head *head) +{ + struct ptdesc *ptdesc; + + ptdesc =3D container_of(head, struct ptdesc, pt_rcu_head); + __tlb_remove_table(ptdesc); +} + +static inline void __tlb_remove_table_one(void *table) +{ + struct ptdesc *ptdesc; + + ptdesc =3D table; + call_rcu(&ptdesc->pt_rcu_head, __tlb_remove_table_one_rcu); +} +#else static inline void __tlb_remove_table_one(void *table) { tlb_remove_table_sync_one(); __tlb_remove_table(table); } -#endif +#endif /* CONFIG_PT_RECLAIM */ =20 static void tlb_remove_table_one(void *table) { --=20 2.20.1 From nobody Wed Dec 17 15:51:21 2025 Received: from mail-pl1-f178.google.com (mail-pl1-f178.google.com [209.85.214.178]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C46DA1AAA29 for ; Wed, 8 Jan 2025 07:03:40 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.178 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1736319822; cv=none; b=qIE8/cvA+7YOKV2OcJ3KJNhvM5PzaJI5xZiLu3oZ3A2s7i6ogRzHgGHJo28SBMOoB9+JOZtahT0lQ4qm+9JDOXBzvhTwcflpR7V9yoLStDS10itxJm9rn8HBR1TdcudT3dyrcUtFDf21KAsuX8109gTsIOl6xSJkwSDbEQJ/n+g= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1736319822; c=relaxed/simple; bh=6Tm9tzzLwfPI5sWMJ+gGPjjpg7Ij0ovc/RrCPGoP6W0=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=szCGDzfmVfMS7VqaCo2DQB5s8UiS10AbxOtzUnuOMecOR2E36zPt7UPkHCgVRAfCLa9xZ2w6+lQQh0DK6YfQ8gbYKGt/AcWrMBoHc21gbwWsPGYAcfrvyKo0EvK4wID0e6SL8ewKwxOnfGZK/5eQlEirVBexteiRWg84hekI5aQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=KtvLg51X; arc=none smtp.client-ip=209.85.214.178 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="KtvLg51X" Received: by mail-pl1-f178.google.com with SMTP id d9443c01a7336-216395e151bso7145995ad.0 for ; Tue, 07 Jan 2025 23:03:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1736319820; x=1736924620; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=SmV2C/123ilQKoYc6h41pKDzG2aiaIfZ7nQkaBcWxp0=; b=KtvLg51Xue1YBzIOtDDnaKVLDxho5Y6O34b3QLXYGujr9xXAirsHXiFIDJ98IjU9oz Z0hAvHLsx9XPp3Agplnwmdgv1qA4pbR9wYkuCUL2dZ92URLD0cl6irNm7klKTt/Is0tm 4JLj7aE1faDKWpUwyXHPqT4sgeuViPZFzB2ipUzZMoMJ296x2IbGR20ZTsAMf8/Au32o CIOBHWY/teEwvunVztgAgLPbdBy3ANmKwDUgjV3S+uTok9chdNvvXzRT2hGQqLXZcP1s LvQFH6G/sHCtN/HFAGvMHOg0FcCEXTRuZI0StA33iKs4vlrGO5YuafH2wd1ZbnK0lL1T VwLQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736319820; x=1736924620; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=SmV2C/123ilQKoYc6h41pKDzG2aiaIfZ7nQkaBcWxp0=; b=SmHqPyzPscsMebZMqfmk1bYjF8+E4bfJu4f9rPUE1d8E0cRco++cukJOnVnsJKtItl KECQ+E+piV7Mc5GoL61yL16F0WOmrXt000MsM1JVUMH0Mlaz/+OlDsI1S3H2rATgo/u3 7zh0nRj9qMsHWxxsx+Q403YhNZ1yK4MS3z/Wk136/4/8YpSsm4Vg6tAKJXhpUVIF0dHx NeSrL6QJzjPq4Gi9vhiYSP1glCU2QsFS/ax08xDbVEXaCB4C2XqtiLC7UJp0iLs8PX+4 b1f/JyvELk9rqH3opKByyTSNKg4kZqVcQfeR2/pP/8OD//3Ia0mWGjNpzHCAXBANlh8b arZQ== X-Forwarded-Encrypted: i=1; AJvYcCVvmd5TFzFR0RJj+JsIDRLWMdqRUtx1fo8ztZmkSs1TR/lNiAlrnzXCWStn6u6PWrQqIa5jwLI9cT2/MRM=@vger.kernel.org X-Gm-Message-State: AOJu0YxRu32jDXpQUpPx9CiesUTj/l5eRnC715bdMyPTHcouBkyw3Hkj sx8Df6pJpa+bahyEfVLvhkJXEi50HdzaN7Wn0AZbaLRwWP4ZVCcPfHjLNse0sk8= X-Gm-Gg: ASbGnct5z3JrOMvGV0KIXO5lEtouEshTHVxcdT7SgFH7jDIK+sS4G5yx0j+Hs3NeTBN KB7P9KLS3NGKCpN4H6bEY54qAans72yRhDjEgtSC9ZteK1FPH4bdHB1W75Uf7wfy2sOtmd+quna TxNix8sZszUNtwqitlOwCQNoujwXPVAOtwNfCrjTwJLypYFX6cpyUUExmbN+VDO1pWdukOP06nR 4HuT4tftablKmJDlh1ZtI77jjkwg8N7G2ysrpW7iTC4B+rfeHTvNVe1o+YWoPMDQ76wqXkhC6ya snaF2YbVEqP5h5xAJVwbdA9duFU= X-Google-Smtp-Source: AGHT+IFPhscn4ro6sBJsBOR2a8pRlJGCRsiSDouRo+43OGNlwIzbjZLHQRkjQ8iB7a2FmZbbQjyk4w== X-Received: by 2002:a17:902:fc46:b0:216:4fad:35d0 with SMTP id d9443c01a7336-21a7a1d4731mr100818035ad.9.1736319820276; Tue, 07 Jan 2025 23:03:40 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.244]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-219dca023a3sm320067275ad.250.2025.01.07.23.03.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 07 Jan 2025 23:03:39 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, agordeev@linux.ibm.com, kevin.brodsky@arm.com, alex@ghiti.fr, andreas@gaisler.com, palmer@dabbelt.com, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com, arnd@arndb.de, will@kernel.org, aneesh.kumar@kernel.org, npiggin@gmail.com, dave.hansen@linux.intel.com, rppt@kernel.org, ryan.roberts@arm.com Cc: linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, linux-arch@vger.kernel.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, loongarch@lists.linux.dev, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-sh@vger.kernel.org, linux-um@lists.infradead.org, Qi Zheng Subject: [PATCH v5 17/17] mm: pgtable: introduce generic pagetable_dtor_free() Date: Wed, 8 Jan 2025 14:57:33 +0800 Message-Id: <1663a0565aca881d1338ceb7d1db4aa9c333abd6.1736317725.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The pte_free(), pmd_free(), __pud_free() and __p4d_free() in asm-generic/pgalloc.h and the generic __tlb_remove_table() are basically the same, so let's introduce pagetable_dtor_free() to deduplicate them. In addition, the pagetable_dtor_free() in s390 does the same thing, so let's s390 also calls generic pagetable_dtor_free(). Signed-off-by: Qi Zheng Suggested-by: Peter Zijlstra (Intel) Reviewed-by: Kevin Brodsky Acked-by: Alexander Gordeev # s390 --- arch/s390/mm/pgalloc.c | 6 ------ include/asm-generic/pgalloc.h | 12 ++++-------- include/asm-generic/tlb.h | 3 +-- include/linux/mm.h | 6 ++++++ 4 files changed, 11 insertions(+), 16 deletions(-) diff --git a/arch/s390/mm/pgalloc.c b/arch/s390/mm/pgalloc.c index 3e002dea6278f..a4e7619020931 100644 --- a/arch/s390/mm/pgalloc.c +++ b/arch/s390/mm/pgalloc.c @@ -180,12 +180,6 @@ unsigned long *page_table_alloc(struct mm_struct *mm) return table; } =20 -static void pagetable_dtor_free(struct ptdesc *ptdesc) -{ - pagetable_dtor(ptdesc); - pagetable_free(ptdesc); -} - void page_table_free(struct mm_struct *mm, unsigned long *table) { struct ptdesc *ptdesc =3D virt_to_ptdesc(table); diff --git a/include/asm-generic/pgalloc.h b/include/asm-generic/pgalloc.h index 4afb346eae255..e3977ddca15e4 100644 --- a/include/asm-generic/pgalloc.h +++ b/include/asm-generic/pgalloc.h @@ -109,8 +109,7 @@ static inline void pte_free(struct mm_struct *mm, struc= t page *pte_page) { struct ptdesc *ptdesc =3D page_ptdesc(pte_page); =20 - pagetable_dtor(ptdesc); - pagetable_free(ptdesc); + pagetable_dtor_free(ptdesc); } =20 =20 @@ -153,8 +152,7 @@ static inline void pmd_free(struct mm_struct *mm, pmd_t= *pmd) struct ptdesc *ptdesc =3D virt_to_ptdesc(pmd); =20 BUG_ON((unsigned long)pmd & (PAGE_SIZE-1)); - pagetable_dtor(ptdesc); - pagetable_free(ptdesc); + pagetable_dtor_free(ptdesc); } #endif =20 @@ -202,8 +200,7 @@ static inline void __pud_free(struct mm_struct *mm, pud= _t *pud) struct ptdesc *ptdesc =3D virt_to_ptdesc(pud); =20 BUG_ON((unsigned long)pud & (PAGE_SIZE-1)); - pagetable_dtor(ptdesc); - pagetable_free(ptdesc); + pagetable_dtor_free(ptdesc); } =20 #ifndef __HAVE_ARCH_PUD_FREE @@ -248,8 +245,7 @@ static inline void __p4d_free(struct mm_struct *mm, p4d= _t *p4d) struct ptdesc *ptdesc =3D virt_to_ptdesc(p4d); =20 BUG_ON((unsigned long)p4d & (PAGE_SIZE-1)); - pagetable_dtor(ptdesc); - pagetable_free(ptdesc); + pagetable_dtor_free(ptdesc); } =20 #ifndef __HAVE_ARCH_P4D_FREE diff --git a/include/asm-generic/tlb.h b/include/asm-generic/tlb.h index 53ae7748f555b..e402aef79c93e 100644 --- a/include/asm-generic/tlb.h +++ b/include/asm-generic/tlb.h @@ -213,8 +213,7 @@ static inline void __tlb_remove_table(void *table) { struct ptdesc *ptdesc =3D (struct ptdesc *)table; =20 - pagetable_dtor(ptdesc); - pagetable_free(ptdesc); + pagetable_dtor_free(ptdesc); } #endif =20 diff --git a/include/linux/mm.h b/include/linux/mm.h index cad11fa10c192..94078c488e904 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -3001,6 +3001,12 @@ static inline void pagetable_dtor(struct ptdesc *ptd= esc) lruvec_stat_sub_folio(folio, NR_PAGETABLE); } =20 +static inline void pagetable_dtor_free(struct ptdesc *ptdesc) +{ + pagetable_dtor(ptdesc); + pagetable_free(ptdesc); +} + static inline bool pagetable_pte_ctor(struct ptdesc *ptdesc) { struct folio *folio =3D ptdesc_folio(ptdesc); --=20 2.20.1