From nobody Sun May 10 06:27:04 2026 Received: from mail-pl1-f181.google.com (mail-pl1-f181.google.com [209.85.214.181]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0A99A1A23A8 for ; Mon, 30 Dec 2024 09:08:52 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.181 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1735549736; cv=none; b=MtjHdwzDksNvWjErSplk6pUu7VCXRJIc7be9BI+mZxTF3oUwnAcY7ra8re+utjhovZwRwcTpNwgJv1EHrvzKBYRnTqMvBMeVutWRgLPDN+JwceOW+fLz8YQvHaAHmjDbEvB0cUxS6OkoU3Pbj8OG6saGsudvw2taAxlD39CtAKk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1735549736; c=relaxed/simple; bh=eSQ1cqjI0k4SQFzb/dlIm3hCd3v5CZjZJpupq9IDdtI=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=tO5EKcsbwKQJFjcBbbhJ0ElLeQLJCAYjXF2plHLYc7EqZH/J8A1SAPTO7Dw9HVLoZvMzrO4BD2gLwdnt6hTNQ1gVA1vRNCkmZ0L2oMOHaJUaD+IShYcT00LmEkUvik4u+yJ5t3RzC0jRobyCKjqaRmcr3yZsi6f6vWA//CS/anc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=fkkQ/Cxs; arc=none smtp.client-ip=209.85.214.181 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="fkkQ/Cxs" Received: by mail-pl1-f181.google.com with SMTP id d9443c01a7336-2162c0f6a39so123632225ad.0 for ; Mon, 30 Dec 2024 01:08:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1735549732; x=1736154532; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=SyzxI6H4ZAISLyWZ1lhQ49SW3wSD1SDR6GIh5AHA3wQ=; b=fkkQ/Cxs0W3KUmzY4uZ7gTo6taPzqts1z1s+LhKxsSEnR3BoLO5qnre0TuWPFqjdiI oZbHhtRswoYOz51aj23l6M1fZjo/I1mBXx+a7Iai15xRuXyvAHJQHi+jOiOb1Kt1iiWj b9yugJcKfuBfMcahL5WcGTkQ2AOO159KQg7pL88kbXV2DsKPGDYO3zoODj1R8Nz1+IVM 3dgWvyPIq+WT9jjP6YepgFxz8nnK9nW1tA34qbI1rSfyWgLVXlRV8s7tdv2RlahIzFgG OZKrVMu0Gki7Vizl9XGtt8CU8qyhtBGzRgbciWyjzphP+sqVbQqRU3xSu4Lv4riW0t9C jXAQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1735549732; x=1736154532; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=SyzxI6H4ZAISLyWZ1lhQ49SW3wSD1SDR6GIh5AHA3wQ=; b=q4vbkCotQmtABbPblcosyFQCMfRiKZtOjLDd2tu94tvMiwDI2UuEoFOCSTvAaKxm0o ia8PDJZsGP11l3Evis9f47cwyIasSfSb+j8dVdhPgEyjuQfv0Rz0AbQCMgxhYFjHXv5q xCnNModY7dzCOFs0ARa0zbzpGV3CkbMXYkm5EIrCj+Z/Fg+Kyaaqn3dAjHyoPSJcQlWj mTXpjApMwXgIfgUO6b0cJEQRiHHQ5omNoYFpgCYyYEDN9N/T8Cy+jP02LOwk7i+ZEdKe TxGkUCCbSkBfxkXr+Pvh3BpaNe9/lHn7kQPUstnKZAMRaq6fa9W/9FBjyiA1P2E+hUtV DeHw== X-Forwarded-Encrypted: i=1; AJvYcCX9EW81SuqacPEYk8aigv7dcL4377mDIjQaU5TnkNk156fDYcZHkUBPlSA6tBWKrOLA3Ko/6sTb0/0bnZs=@vger.kernel.org X-Gm-Message-State: AOJu0YyHXBJNB1lHriVszO9+e+Zov5S5MXsniw2oEjljfCesB6JdQnT2 1GQv6ECDOqtA+yjm4XWuuKEitgh1yx3wNgxDRXWjLPJ5k3QqZBF7i9cX1hKl2OA= X-Gm-Gg: ASbGncs4j984nGBoprFmyftkMhUdxU0LEbtmc6gL9Nm6LKSQA8y1xk2gw5anvlFDL14 JfR7ZeRs6lQGv83UkJRRw1088PRzHdf5CHVmnv7GRdckjo1aNlF/qHnGV27Sw/rpPTfh54n8eGj 50mbzHJIH84PaPzSViRaIsMHw2jH/wWz119BeFf4WD6z4RQ0ilfFTpW8TJpCXTdJO6nEsmFq+UG zUQf1LcOmmTE7f3MnSCuZGTwU+sgr0pZgCIpd+X9b3Nwcqd7NAMtnjQvq/3nvZZBWqhyU4Ucuh0 kjGWLzTorExtQhPaYnyhUA== X-Google-Smtp-Source: AGHT+IEXAwBIqSClrkb4EYoc2OhmhpgilmCz/IsEh6Chc/FsWIzsRB3p/FpLbUSGkXAy0ZMKjVVExg== X-Received: by 2002:a05:6a20:4320:b0:1e6:50a0:f982 with SMTP id adf61e73a8af0-1e650a0fa30mr7283235637.20.1735549732420; Mon, 30 Dec 2024 01:08:52 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-842aba72f7csm17057841a12.4.2024.12.30.01.08.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 30 Dec 2024 01:08:51 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, agordeev@linux.ibm.com, kevin.brodsky@arm.com, palmer@dabbelt.com, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com, arnd@arndb.de, will@kernel.org, aneesh.kumar@kernel.org, npiggin@gmail.com, dave.hansen@linux.intel.com, rppt@kernel.org, ryan.roberts@arm.com Cc: linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, linux-arch@vger.kernel.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, loongarch@lists.linux.dev, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-sh@vger.kernel.org, linux-um@lists.infradead.org, Qi Zheng Subject: [PATCH v4 01/15] Revert "mm: pgtable: make ptlock be freed by RCU" Date: Mon, 30 Dec 2024 17:07:36 +0800 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This reverts commit 2f3443770437e49abc39af26962d293851cbab6d. Signed-off-by: Qi Zheng Reviewed-by: Kevin Brodsky --- include/linux/mm.h | 2 +- include/linux/mm_types.h | 9 +-------- mm/memory.c | 22 ++++++---------------- 3 files changed, 8 insertions(+), 25 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index d61b9c7a3a7b0..c49bc7b764535 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2925,7 +2925,7 @@ void ptlock_free(struct ptdesc *ptdesc); =20 static inline spinlock_t *ptlock_ptr(struct ptdesc *ptdesc) { - return &(ptdesc->ptl->ptl); + return ptdesc->ptl; } #else /* ALLOC_SPLIT_PTLOCKS */ static inline void ptlock_cache_init(void) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 90ab8293d714a..6b27db7f94963 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -434,13 +434,6 @@ FOLIO_MATCH(flags, _flags_2a); FOLIO_MATCH(compound_head, _head_2a); #undef FOLIO_MATCH =20 -#if ALLOC_SPLIT_PTLOCKS -struct pt_lock { - spinlock_t ptl; - struct rcu_head rcu; -}; -#endif - /** * struct ptdesc - Memory descriptor for page tables. * @__page_flags: Same as page flags. Powerpc only. @@ -489,7 +482,7 @@ struct ptdesc { union { unsigned long _pt_pad_2; #if ALLOC_SPLIT_PTLOCKS - struct pt_lock *ptl; + spinlock_t *ptl; #else spinlock_t ptl; #endif diff --git a/mm/memory.c b/mm/memory.c index b9b05c3f93f11..9423967b24180 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -7034,34 +7034,24 @@ static struct kmem_cache *page_ptl_cachep; =20 void __init ptlock_cache_init(void) { - page_ptl_cachep =3D kmem_cache_create("page->ptl", sizeof(struct pt_lock)= , 0, + page_ptl_cachep =3D kmem_cache_create("page->ptl", sizeof(spinlock_t), 0, SLAB_PANIC, NULL); } =20 bool ptlock_alloc(struct ptdesc *ptdesc) { - struct pt_lock *pt_lock; + spinlock_t *ptl; =20 - pt_lock =3D kmem_cache_alloc(page_ptl_cachep, GFP_KERNEL); - if (!pt_lock) + ptl =3D kmem_cache_alloc(page_ptl_cachep, GFP_KERNEL); + if (!ptl) return false; - ptdesc->ptl =3D pt_lock; + ptdesc->ptl =3D ptl; return true; } =20 -static void ptlock_free_rcu(struct rcu_head *head) -{ - struct pt_lock *pt_lock; - - pt_lock =3D container_of(head, struct pt_lock, rcu); - kmem_cache_free(page_ptl_cachep, pt_lock); -} - void ptlock_free(struct ptdesc *ptdesc) { - struct pt_lock *pt_lock =3D ptdesc->ptl; - - call_rcu(&pt_lock->rcu, ptlock_free_rcu); + kmem_cache_free(page_ptl_cachep, ptdesc->ptl); } #endif =20 --=20 2.20.1 From nobody Sun May 10 06:27:04 2026 Received: from mail-pl1-f182.google.com (mail-pl1-f182.google.com [209.85.214.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EEEFC1A23A5 for ; Mon, 30 Dec 2024 09:09:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.182 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1735549748; cv=none; b=V5SuOebzBBw8Jl+cFYNFnrsATZRNZsBHmIuSrS3pQTL2OA33o8/4DX3NAMcn5zbv3IyiEhreFETNlWTLqU7kG9PF22zYiYUT/Tj2q+iF8CD1nLJthcWXfpQjTkTNsoopE3x5dhgj7gSaL9tN20OrXbejwhIVofuyHKUNfTGzEA4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1735549748; c=relaxed/simple; bh=PKxGSwR+56aVOtqiYfVzo+PhXISkTKu7YVtS9mMtvvo=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=fvqMWtf6hUAk7AS5+YbRaG1kAGIvcwekZTW7/w0LjeRDKAj2z9KwG0/YM1wZlFHyeVCO3/wsoB7hg5qYrwa+g176Uw7YeDAvylGAoeb8FI0yBiIQ2fZbgYxMBe2JJS91oVlir3PJrgBPtDxv+uP9tqQqQL+efp9PIAHj1UPmnc4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=UqK3sU+2; arc=none smtp.client-ip=209.85.214.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="UqK3sU+2" Received: by mail-pl1-f182.google.com with SMTP id d9443c01a7336-21649a7bcdcso113075935ad.1 for ; Mon, 30 Dec 2024 01:09:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1735549745; x=1736154545; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=mHzGZQny5JmimEm0kycepUlCOKttN6NXbKtm7VNKgbw=; b=UqK3sU+2wcAUPiAKKXdtQ0aEJDqkRo6Yya3Wbj9oIqILyZNKbqjce1teTm/9Pe/6TJ TT8TcgAmn7YerrmNKsaGqC/WVERQJSYXRT1OqR6pmXQ/mhI169SR8N03MZJaP4hc5/U7 Y68xqwOFUJu5yye3K1J4GXggf6nRlCdi08h2F1f64ee3FILFskQeGXmz7dX8Kdt6LEXA DAFVaNIPS1JpGxoZ5OiMJSwjmI+OFqDDuwo8UB4xy1LXw3q0u1V1kadgXpWOzHB45i65 ZYWnCzkSVkqGj0YzoFzawN2g9hKsI1PUKJX/uPdhmHDeFZ+JqSgtP3rgsKecpfYbHO3w anMA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1735549745; x=1736154545; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=mHzGZQny5JmimEm0kycepUlCOKttN6NXbKtm7VNKgbw=; b=GYzwelT+yV8C7HjZEJHvl5UXCEmpLUvVc+4bewJAQ1quc8ssUMiFBO6/W+uhUdBm3p CbkRzrkNKW8eltoOdw6GYbCoxMLAWNDc30RDPOr3T4DLXMO5AkgFZ2IiIycnx/z11oc2 Z2EnqQcGeQ0BCReDeuedHWvswv+ltd5EFlvC3ArgSTQ01a+N0PF3bjsjJr55iZNb5KVa R/FP9AlJjn78eTLigB8vOHiR8r9kMDabSJCsn7QVohxaq3h8wVr3VJIC3YH3Qow5+Gmw hud0oFDn1rBO0jWZitR8NqgJ7N+E5nXvzOyLIVJC6udXyGHFdGoLYMHDE2HW8bB83Ngj Qs3A== X-Forwarded-Encrypted: i=1; AJvYcCWo2AyvtE4Rdrvv1ga66d26h0yJdr3gOvnOVJS3R1Cvslu5cTB0aI9THd7r+/4HD9HjMksNAKj5qS6cXcM=@vger.kernel.org X-Gm-Message-State: AOJu0YxNqfI4cDt9muVgGTAq/43mN7s3WijiFJ6k2KdrsfBvPqxcgunZ LSFp/0SOKwrl8GmtNl69OTdIzi5s7g+0ZHHaBOUppVStYCl4vrcNUKpwJGH//9g= X-Gm-Gg: ASbGncuZE7DC97+UVRmowwn03cKpfecPf7ZVSZdVv8FuzTT6cmHi+Bkx6Ruadb5hewo 7XC3ul8KNIiOJ2S6QTbiVbUdyvEjV6ttB4ilMgkANVg+HQd66Bt95zDZmGFP0XeoUXimT3qsw+d TlpJbZJVAbd8sqMD3nPS31ED99JVLhL5u8gXQmmpTzwvLs774kmQlRNV9xzFqxCdc46IcEr8TMp 5T/r/HAhhutExisoVykFAYlkXDUTWw49Kd2bMGLTmUHFC21TjfNj7E0vm8XVpkDVgdONpo6T/Vh zM/0IqgAPJ0UsIKzHEJt7w== X-Google-Smtp-Source: AGHT+IEdmvGR6U/qwRYH5ruA02a+sFRctKWchidIfEBAoumo12DzBepIE5K3QuQVJJ60xG/AvigJ4Q== X-Received: by 2002:a05:6a20:a127:b0:1db:ddba:8795 with SMTP id adf61e73a8af0-1e5e08028cfmr53967973637.36.1735549745320; Mon, 30 Dec 2024 01:09:05 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-842aba72f7csm17057841a12.4.2024.12.30.01.08.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 30 Dec 2024 01:09:04 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, agordeev@linux.ibm.com, kevin.brodsky@arm.com, palmer@dabbelt.com, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com, arnd@arndb.de, will@kernel.org, aneesh.kumar@kernel.org, npiggin@gmail.com, dave.hansen@linux.intel.com, rppt@kernel.org, ryan.roberts@arm.com Cc: linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, linux-arch@vger.kernel.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, loongarch@lists.linux.dev, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-sh@vger.kernel.org, linux-um@lists.infradead.org, Qi Zheng , Palmer Dabbelt Subject: [PATCH v4 02/15] riscv: mm: Skip pgtable level check in {pud,p4d}_alloc_one Date: Mon, 30 Dec 2024 17:07:37 +0800 Message-Id: <84ddf857508b98a195a790bc6ff6ab8849b44633.1735549103.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Kevin Brodsky {pmd,pud,p4d}_alloc_one() is never called if the corresponding page table level is folded, as {pmd,pud,p4d}_alloc() already does the required check. We can therefore remove the runtime page table level checks in {pud,p4d}_alloc_one. The PUD helper becomes equivalent to the generic version, so we remove it altogether. This is consistent with the way arm64 and x86 handle this situation (runtime check in p4d_free() only). Signed-off-by: Kevin Brodsky Acked-by: Dave Hansen Signed-off-by: Qi Zheng Acked-by: Palmer Dabbelt Reviewed-by: Alexandre Ghiti Reviewed-by: Kevin Brodsky --- arch/riscv/include/asm/pgalloc.h | 22 ++++------------------ 1 file changed, 4 insertions(+), 18 deletions(-) diff --git a/arch/riscv/include/asm/pgalloc.h b/arch/riscv/include/asm/pgal= loc.h index f52264304f772..8ad0bbe838a24 100644 --- a/arch/riscv/include/asm/pgalloc.h +++ b/arch/riscv/include/asm/pgalloc.h @@ -12,7 +12,6 @@ #include =20 #ifdef CONFIG_MMU -#define __HAVE_ARCH_PUD_ALLOC_ONE #define __HAVE_ARCH_PUD_FREE #include =20 @@ -88,15 +87,6 @@ static inline void pgd_populate_safe(struct mm_struct *m= m, pgd_t *pgd, } } =20 -#define pud_alloc_one pud_alloc_one -static inline pud_t *pud_alloc_one(struct mm_struct *mm, unsigned long add= r) -{ - if (pgtable_l4_enabled) - return __pud_alloc_one(mm, addr); - - return NULL; -} - #define pud_free pud_free static inline void pud_free(struct mm_struct *mm, pud_t *pud) { @@ -118,15 +108,11 @@ static inline void __pud_free_tlb(struct mmu_gather *= tlb, pud_t *pud, #define p4d_alloc_one p4d_alloc_one static inline p4d_t *p4d_alloc_one(struct mm_struct *mm, unsigned long add= r) { - if (pgtable_l5_enabled) { - gfp_t gfp =3D GFP_PGTABLE_USER; - - if (mm =3D=3D &init_mm) - gfp =3D GFP_PGTABLE_KERNEL; - return (p4d_t *)get_zeroed_page(gfp); - } + gfp_t gfp =3D GFP_PGTABLE_USER; =20 - return NULL; + if (mm =3D=3D &init_mm) + gfp =3D GFP_PGTABLE_KERNEL; + return (p4d_t *)get_zeroed_page(gfp); } =20 static inline void __p4d_free(struct mm_struct *mm, p4d_t *p4d) --=20 2.20.1 From nobody Sun May 10 06:27:04 2026 Received: from mail-pl1-f178.google.com (mail-pl1-f178.google.com [209.85.214.178]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AAD441A239C for ; Mon, 30 Dec 2024 09:09:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.178 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1735549761; cv=none; b=iVOGqngQf9OHyLXyFrBoECiExI5jNsQefdaKjyKoxsMY4K6RzQIlXa5Howb3sENpu3tXyIruEHATwCm01WQcCj6+MX8nUJ/IX0xKS3jOJw6LrqHGx4rytqE48mxbq90M3u3uSNzJBjwl/dolLAHG8C/ynDnegmOWA3vd3MMuny0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1735549761; c=relaxed/simple; bh=VQrwZRR3rAwbwOGD5AjoKk2Z15695i3scVkWYvbG/zY=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=df5PXOWirdLF5Ky1y5D3XjKIvuQJfaKIUpmgD8//nDsnC8OeWTvnFd06nGLWAADU93DRTKHiN1/xE3qSJtdomi0hJ8IHDVooBeOQFrWHVRlNfCgbX3FjBrZEQkCjamxI8ZpAyC45LRuM+qWipA34bsxprfSGkewHcbwv3ygNCEk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=W1EIz8Sw; arc=none smtp.client-ip=209.85.214.178 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="W1EIz8Sw" Received: by mail-pl1-f178.google.com with SMTP id d9443c01a7336-21644aca3a0so17730345ad.3 for ; Mon, 30 Dec 2024 01:09:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1735549758; x=1736154558; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Hi4+N7L/R07Fq6g8WRpMxop4cD3t01a1BHl7awbkQjI=; b=W1EIz8SwrFGzIPvVSTgGDInoWHqdgxJVSp7apqZTLAW2HlHtEDtnjUqJvIlQo4bzVk /syWMiQA39pCZO0/DoZFOC7QUkscXEDiaF3ebSv+siTPp6PVBqsSrpR2+wXCyVoMPkdI EkbBfs9Thjr9x3yll1cZ/P0uQDt+pE/zbhF90dwCdHI5uhVQ/nLfJIK/7sAyKOSqxiuV pAHk/GdRvVhYljIzy0buCAImw0PnA2xuaKNQZQhkkPTODHZcWLIfV4Pm52THd3LRcpoj 82dMp3Up3sE/+SaD6UMBzza195ZuehCMECQGAZP2IOmSDzlirqTw6dNJtJhuS6VJswbZ yv5A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1735549758; x=1736154558; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Hi4+N7L/R07Fq6g8WRpMxop4cD3t01a1BHl7awbkQjI=; b=belmsTxLWylZr8pCzGwXc75h4hvnahXT+87whClyAZaC6526ij7yW5wy+isUBkt0iH 2eEPsIuAjLoDGlJQzSIWlm7Gju5oAvYwEiCw+g4KVqxx5IhZqbtBPUdKeMnSTvIbnRTn +zbLVOGnw8nUaIZ18ZscX+1t4njuINhcHrUUKNq2Q/0AyVUA6tjI5bswFtC2my3cEhto 93FbaNSVRWmAAe/wBPHJifqZ+WXlmWQGyY90otktvi72xo2oq8r/UjL2yBKGonlPYg3X goTsQVTxTECZGzxoI0w0JXf/SxzryP7Ul209tMmZwJsxrb8m4JwedxKU82K4obiMz35B MLrQ== X-Forwarded-Encrypted: i=1; AJvYcCVuAPMhjqAGdNhKi/FE3gM4MA2ugaXzsoRnRzxFSyMS7pjxQMqFawNkwBKBtc2CPd95wTP+n1QsTMHZLfs=@vger.kernel.org X-Gm-Message-State: AOJu0YxaXptHQVPHIQq1i1dxuqF2W97xem4bcv6s9uduBeycHcPa6Dhd 2a5HaJElVaqld7jl3RkyhspYf7wxvb1P/cyEqzDuHm4bEXypndpG0r3oOzVvjVI= X-Gm-Gg: ASbGncvzB44CnP/sUBGI0py5QkNOECd3cVRsUFHKOaz0h60gZn5TvBgMOcgfYWRF0gY NMb7X+wq3IKzBer6+LT4GKifmsNwBPAuVIu5WncF17I7EFZYRaR4gHf+/zRdNvE9UjtihZvu8aO AcTbf/q0gefZ4mx/YLpWeqML9gcGb1FY0rDQ+aAOghJmS6xHNd7qTrwRgvZ7IvPehXq81k8yk07 BcUq0A21Dkfnz2ngDi/zJCw4OfFWqbAj0uKP1YveODJoZi/EZonqvuSs1GivcuYpY/T794YT37s TBIXyJDnVL0q5/gOPLN5Lg== X-Google-Smtp-Source: AGHT+IHmMXvIfmrfsWwSSN68U3q7TJmVN0XL9C78+ztBjPSSvkZSld2lKuaEpxPQ8tk0dZT2wkXZcA== X-Received: by 2002:a17:902:d488:b0:212:4ac2:4919 with SMTP id d9443c01a7336-219e6e9f9aemr366243385ad.17.1735549758001; Mon, 30 Dec 2024 01:09:18 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-842aba72f7csm17057841a12.4.2024.12.30.01.09.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 30 Dec 2024 01:09:17 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, agordeev@linux.ibm.com, kevin.brodsky@arm.com, palmer@dabbelt.com, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com, arnd@arndb.de, will@kernel.org, aneesh.kumar@kernel.org, npiggin@gmail.com, dave.hansen@linux.intel.com, rppt@kernel.org, ryan.roberts@arm.com Cc: linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, linux-arch@vger.kernel.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, loongarch@lists.linux.dev, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-sh@vger.kernel.org, linux-um@lists.infradead.org, Qi Zheng Subject: [PATCH v4 03/15] asm-generic: pgalloc: Provide generic p4d_{alloc_one,free} Date: Mon, 30 Dec 2024 17:07:38 +0800 Message-Id: <4c4bcc1aa565c6252183553aecd5e5cbd1a0f6ea.1735549103.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Kevin Brodsky Four architectures currently implement 5-level pgtables: arm64, riscv, x86 and s390. The first three have essentially the same implementation for p4d_alloc_one() and p4d_free(), so we've got an opportunity to reduce duplication like at the lower levels. Provide a generic version of p4d_alloc_one() and p4d_free(), and make use of it on those architectures. Their implementation is the same as at PUD level, except that p4d_free() performs a runtime check by calling mm_p4d_folded(). 5-level pgtables depend on a runtime-detected hardware feature on all supported architectures, so we might as well include this check in the generic implementation. No runtime check is required in p4d_alloc_one() as the top-level p4d_alloc() already does the required check. Signed-off-by: Kevin Brodsky Acked-by: Dave Hansen Signed-off-by: Qi Zheng Reviewed-by: Kevin Brodsky --- arch/arm64/include/asm/pgalloc.h | 17 ------------ arch/riscv/include/asm/pgalloc.h | 23 ---------------- arch/x86/include/asm/pgalloc.h | 18 ------------- include/asm-generic/pgalloc.h | 45 ++++++++++++++++++++++++++++++++ 4 files changed, 45 insertions(+), 58 deletions(-) diff --git a/arch/arm64/include/asm/pgalloc.h b/arch/arm64/include/asm/pgal= loc.h index e75422864d1bd..2965f5a7e39e3 100644 --- a/arch/arm64/include/asm/pgalloc.h +++ b/arch/arm64/include/asm/pgalloc.h @@ -85,23 +85,6 @@ static inline void pgd_populate(struct mm_struct *mm, pg= d_t *pgdp, p4d_t *p4dp) __pgd_populate(pgdp, __pa(p4dp), pgdval); } =20 -static inline p4d_t *p4d_alloc_one(struct mm_struct *mm, unsigned long add= r) -{ - gfp_t gfp =3D GFP_PGTABLE_USER; - - if (mm =3D=3D &init_mm) - gfp =3D GFP_PGTABLE_KERNEL; - return (p4d_t *)get_zeroed_page(gfp); -} - -static inline void p4d_free(struct mm_struct *mm, p4d_t *p4d) -{ - if (!pgtable_l5_enabled()) - return; - BUG_ON((unsigned long)p4d & (PAGE_SIZE-1)); - free_page((unsigned long)p4d); -} - #define __p4d_free_tlb(tlb, p4d, addr) p4d_free((tlb)->mm, p4d) #else static inline void __pgd_populate(pgd_t *pgdp, phys_addr_t p4dp, pgdval_t = prot) diff --git a/arch/riscv/include/asm/pgalloc.h b/arch/riscv/include/asm/pgal= loc.h index 8ad0bbe838a24..551d614d3369c 100644 --- a/arch/riscv/include/asm/pgalloc.h +++ b/arch/riscv/include/asm/pgalloc.h @@ -105,29 +105,6 @@ static inline void __pud_free_tlb(struct mmu_gather *t= lb, pud_t *pud, } } =20 -#define p4d_alloc_one p4d_alloc_one -static inline p4d_t *p4d_alloc_one(struct mm_struct *mm, unsigned long add= r) -{ - gfp_t gfp =3D GFP_PGTABLE_USER; - - if (mm =3D=3D &init_mm) - gfp =3D GFP_PGTABLE_KERNEL; - return (p4d_t *)get_zeroed_page(gfp); -} - -static inline void __p4d_free(struct mm_struct *mm, p4d_t *p4d) -{ - BUG_ON((unsigned long)p4d & (PAGE_SIZE-1)); - free_page((unsigned long)p4d); -} - -#define p4d_free p4d_free -static inline void p4d_free(struct mm_struct *mm, p4d_t *p4d) -{ - if (pgtable_l5_enabled) - __p4d_free(mm, p4d); -} - static inline void __p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d, unsigned long addr) { diff --git a/arch/x86/include/asm/pgalloc.h b/arch/x86/include/asm/pgalloc.h index dcd836b59bebd..dd4841231bb9f 100644 --- a/arch/x86/include/asm/pgalloc.h +++ b/arch/x86/include/asm/pgalloc.h @@ -147,24 +147,6 @@ static inline void pgd_populate_safe(struct mm_struct = *mm, pgd_t *pgd, p4d_t *p4 set_pgd_safe(pgd, __pgd(_PAGE_TABLE | __pa(p4d))); } =20 -static inline p4d_t *p4d_alloc_one(struct mm_struct *mm, unsigned long add= r) -{ - gfp_t gfp =3D GFP_KERNEL_ACCOUNT; - - if (mm =3D=3D &init_mm) - gfp &=3D ~__GFP_ACCOUNT; - return (p4d_t *)get_zeroed_page(gfp); -} - -static inline void p4d_free(struct mm_struct *mm, p4d_t *p4d) -{ - if (!pgtable_l5_enabled()) - return; - - BUG_ON((unsigned long)p4d & (PAGE_SIZE-1)); - free_page((unsigned long)p4d); -} - extern void ___p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d); =20 static inline void __p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d, diff --git a/include/asm-generic/pgalloc.h b/include/asm-generic/pgalloc.h index 7c48f5fbf8aa7..59131629ac9cc 100644 --- a/include/asm-generic/pgalloc.h +++ b/include/asm-generic/pgalloc.h @@ -215,6 +215,51 @@ static inline void pud_free(struct mm_struct *mm, pud_= t *pud) =20 #endif /* CONFIG_PGTABLE_LEVELS > 3 */ =20 +#if CONFIG_PGTABLE_LEVELS > 4 + +static inline p4d_t *__p4d_alloc_one_noprof(struct mm_struct *mm, unsigned= long addr) +{ + gfp_t gfp =3D GFP_PGTABLE_USER; + struct ptdesc *ptdesc; + + if (mm =3D=3D &init_mm) + gfp =3D GFP_PGTABLE_KERNEL; + gfp &=3D ~__GFP_HIGHMEM; + + ptdesc =3D pagetable_alloc_noprof(gfp, 0); + if (!ptdesc) + return NULL; + + return ptdesc_address(ptdesc); +} +#define __p4d_alloc_one(...) alloc_hooks(__p4d_alloc_one_noprof(__VA_ARGS_= _)) + +#ifndef __HAVE_ARCH_P4D_ALLOC_ONE +static inline p4d_t *p4d_alloc_one_noprof(struct mm_struct *mm, unsigned l= ong addr) +{ + return __p4d_alloc_one_noprof(mm, addr); +} +#define p4d_alloc_one(...) alloc_hooks(p4d_alloc_one_noprof(__VA_ARGS__)) +#endif + +static inline void __p4d_free(struct mm_struct *mm, p4d_t *p4d) +{ + struct ptdesc *ptdesc =3D virt_to_ptdesc(p4d); + + BUG_ON((unsigned long)p4d & (PAGE_SIZE-1)); + pagetable_free(ptdesc); +} + +#ifndef __HAVE_ARCH_P4D_FREE +static inline void p4d_free(struct mm_struct *mm, p4d_t *p4d) +{ + if (!mm_p4d_folded(mm)) + __p4d_free(mm, p4d); +} +#endif + +#endif /* CONFIG_PGTABLE_LEVELS > 4 */ + #ifndef __HAVE_ARCH_PGD_FREE static inline void pgd_free(struct mm_struct *mm, pgd_t *pgd) { --=20 2.20.1 From nobody Sun May 10 06:27:04 2026 Received: from mail-pl1-f172.google.com (mail-pl1-f172.google.com [209.85.214.172]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 235BD1A239F for ; Mon, 30 Dec 2024 09:09:31 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1735549773; cv=none; b=TWlyy0dBEooWwaPSWL6ydug4xD2i4EUMLc+Q+STho1QhM8tQn9oTpCn9ymMeVyDZPX7lKkK+aGwu/Suel24bOIeg5MORgOTrFKaHEKt9salltaUMu9buidCab/9bpt01plVKq5cspQ8rFQjk7O4CKCQ/Hx4eLHfJABhOkQMW8A8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1735549773; c=relaxed/simple; bh=iGGseqx9Ut5ViSPAG0PhwW+aGehMNfQFpL91q5ypQVw=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=W8V1u/NRUNlmlnJEFtk1moCbE1PRld6zQ+wQc7FcMXlWFWTh7VIZNXyxvGYiUJQuJjQSoh/sgO3u/Evg8VHAzT/jqnyPZQSpe6j8BrDFOfC4KU0RSgf9MRpHUP3KwSRtWyT+ePP64MHJMGCxE3Jh1iKnAyuuionPOu1GLKvjmDQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=LU4QW+MN; arc=none smtp.client-ip=209.85.214.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="LU4QW+MN" Received: by mail-pl1-f172.google.com with SMTP id d9443c01a7336-21644aca3a0so17733085ad.3 for ; Mon, 30 Dec 2024 01:09:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1735549770; x=1736154570; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=t6vBY23hDAeU19z0pXyQ9vuAeZavJyfGUN8npoqwlxU=; b=LU4QW+MNOhbE1wMpxZf5W8BWtU4X4oesBCIsibBGNOHSaMuRf/wJZx1ELNXh9cA7hS 89G5ezs+gPAJPOFoQsy9KLlYwcajFd4kC7pkPs0o118RQTdClzCk/GCrvuGtdYHN0TCw /T2HwLN7LR5bih6T5OKPqLyUw1eZSumtHmJZ2VHa9b6542IXnxnqYSTmdUPOzu1X3e5u 9LxelDQFkyZ/Mvfr/D+5OWJfteXzxmKSpnEe/hMSkBBmMj19bN1uIxrFYKYbg5VamT7j +CLw+iNk3Lx6FpLsOOexKakWrkKYcE7rkZPaQayuWIWfdyvCzdHtWaP26ygOogTCPmed 9EzA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1735549770; x=1736154570; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=t6vBY23hDAeU19z0pXyQ9vuAeZavJyfGUN8npoqwlxU=; b=sw5ht4dsrW9g1nc99zGbIegkph/YKLdbU4QgN7lA6dsoybjd3thQfyEPYYEXZb3+NY LZZG2dC+BX0GbOvhhaRxh6C2Tvo2iQNQVXyPPuBxOIsbfw7ij0zqZvrFW2qO9v1yqyDN BHaklgSaHDeKmF4ns5a9IBkGLHPLrB9nCEOPMUpw0S82rpnxtEl51DMx5bNZBJSSdndo w2XrtTFZT7XnfttWpHgW6iad/dvk9Jvg+NOYtdrOmKjPZfRJ6bsspfRhd6y7q7B8zr9O 4k+hpbc/oiEO9EQhFMr0vc3D0ViWCpZAZVo3+nW1d1jUciW+yPFZ9yxVPURBKp1nvTOM 5/+Q== X-Forwarded-Encrypted: i=1; AJvYcCWMT60Hy3gZOJpCqcxOnJqEHxxQmiMMk+DWWD/ocBNm6n73PhYAFbQwXADop7ZXxqhIchJ5RmBXicMzKo0=@vger.kernel.org X-Gm-Message-State: AOJu0Yyua9cGTSYmQRFH7jcVRlaamRRt4VynwnncxO3165OU7ImScmBp uFX+dg7ZlD+dfSnLE3MB6eImznkWB9i4DjpjmWOwz7s8Apm9H/fkEjZh0R1t3Zk= X-Gm-Gg: ASbGncsTv+RALWstPknduWpHu+JGTbzmQGQmZszcUCEJD3qHNHHPeJeHjL8Rvu3Zrv+ Djuh0+/tOQQj3xjM1B6IaRnQhbRYYO8CvIdSJY024GdzWQil7JXMpjq7Qq8BLq9wJp0s76MYF9u OeXHn87O40B10p44YUg9GXiM4nXrAnGpCfqk3IkkZIBzqgvYVTTbYgji/Ie8qIbVvRKJzIuwulM 5U5qInA842VT1P0FNc04CycDNWlHOUEffjn2tEQEl0KWkTCvfFn804rQQsdGqA3ype/XH58KjAF uQDlUxj41IpeWDd5hIrJQw== X-Google-Smtp-Source: AGHT+IFhgZxpyzdRj713xWVIn5lnq5dMIdipHJPkhLMIEEbSziCo236oEOGt7kn6Wpg3lXZNnJqwCQ== X-Received: by 2002:a05:6a21:1517:b0:1e1:adb9:d1a6 with SMTP id adf61e73a8af0-1e5e08439e8mr55857078637.41.1735549770532; Mon, 30 Dec 2024 01:09:30 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-842aba72f7csm17057841a12.4.2024.12.30.01.09.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 30 Dec 2024 01:09:30 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, agordeev@linux.ibm.com, kevin.brodsky@arm.com, palmer@dabbelt.com, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com, arnd@arndb.de, will@kernel.org, aneesh.kumar@kernel.org, npiggin@gmail.com, dave.hansen@linux.intel.com, rppt@kernel.org, ryan.roberts@arm.com Cc: linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, linux-arch@vger.kernel.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, loongarch@lists.linux.dev, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-sh@vger.kernel.org, linux-um@lists.infradead.org, Qi Zheng Subject: [PATCH v4 04/15] mm: pgtable: add statistics for P4D level page table Date: Mon, 30 Dec 2024 17:07:39 +0800 Message-Id: <2fa644e37ab917292f5c342e40fa805aa91afbbd.1735549103.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Like other levels of page tables, add statistics for P4D level page table. Signed-off-by: Qi Zheng Originally-by: Peter Zijlstra (Intel) Reviewed-by: Kevin Brodsky --- arch/riscv/include/asm/pgalloc.h | 6 +++++- arch/x86/mm/pgtable.c | 3 +++ include/asm-generic/pgalloc.h | 2 ++ include/linux/mm.h | 16 ++++++++++++++++ 4 files changed, 26 insertions(+), 1 deletion(-) diff --git a/arch/riscv/include/asm/pgalloc.h b/arch/riscv/include/asm/pgal= loc.h index 551d614d3369c..3466fbe2e508d 100644 --- a/arch/riscv/include/asm/pgalloc.h +++ b/arch/riscv/include/asm/pgalloc.h @@ -108,8 +108,12 @@ static inline void __pud_free_tlb(struct mmu_gather *t= lb, pud_t *pud, static inline void __p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d, unsigned long addr) { - if (pgtable_l5_enabled) + if (pgtable_l5_enabled) { + struct ptdesc *ptdesc =3D virt_to_ptdesc(p4d); + + pagetable_p4d_dtor(ptdesc); riscv_tlb_remove_ptdesc(tlb, virt_to_ptdesc(p4d)); + } } #endif /* __PAGETABLE_PMD_FOLDED */ =20 diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c index 69a357b15974a..3d6e84da45b24 100644 --- a/arch/x86/mm/pgtable.c +++ b/arch/x86/mm/pgtable.c @@ -94,6 +94,9 @@ void ___pud_free_tlb(struct mmu_gather *tlb, pud_t *pud) #if CONFIG_PGTABLE_LEVELS > 4 void ___p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d) { + struct ptdesc *ptdesc =3D virt_to_ptdesc(p4d); + + pagetable_p4d_dtor(ptdesc); paravirt_release_p4d(__pa(p4d) >> PAGE_SHIFT); paravirt_tlb_remove_table(tlb, virt_to_page(p4d)); } diff --git a/include/asm-generic/pgalloc.h b/include/asm-generic/pgalloc.h index 59131629ac9cc..bb482eeca0c3e 100644 --- a/include/asm-generic/pgalloc.h +++ b/include/asm-generic/pgalloc.h @@ -230,6 +230,7 @@ static inline p4d_t *__p4d_alloc_one_noprof(struct mm_s= truct *mm, unsigned long if (!ptdesc) return NULL; =20 + pagetable_p4d_ctor(ptdesc); return ptdesc_address(ptdesc); } #define __p4d_alloc_one(...) alloc_hooks(__p4d_alloc_one_noprof(__VA_ARGS_= _)) @@ -247,6 +248,7 @@ static inline void __p4d_free(struct mm_struct *mm, p4d= _t *p4d) struct ptdesc *ptdesc =3D virt_to_ptdesc(p4d); =20 BUG_ON((unsigned long)p4d & (PAGE_SIZE-1)); + pagetable_p4d_dtor(ptdesc); pagetable_free(ptdesc); } =20 diff --git a/include/linux/mm.h b/include/linux/mm.h index c49bc7b764535..5d82f42ddd5cc 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -3175,6 +3175,22 @@ static inline void pagetable_pud_dtor(struct ptdesc = *ptdesc) lruvec_stat_sub_folio(folio, NR_PAGETABLE); } =20 +static inline void pagetable_p4d_ctor(struct ptdesc *ptdesc) +{ + struct folio *folio =3D ptdesc_folio(ptdesc); + + __folio_set_pgtable(folio); + lruvec_stat_add_folio(folio, NR_PAGETABLE); +} + +static inline void pagetable_p4d_dtor(struct ptdesc *ptdesc) +{ + struct folio *folio =3D ptdesc_folio(ptdesc); + + __folio_clear_pgtable(folio); + lruvec_stat_sub_folio(folio, NR_PAGETABLE); +} + extern void __init pagecache_init(void); extern void free_initmem(void); =20 --=20 2.20.1 From nobody Sun May 10 06:27:04 2026 Received: from mail-pl1-f182.google.com (mail-pl1-f182.google.com [209.85.214.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7F6FA1A23BE for ; Mon, 30 Dec 2024 09:09:43 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.182 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1735549785; cv=none; b=H1NEyYB2V/TGoPtl1KQfx5zrP77dgnOpN0uqsxYWOJnheAkKHfEqaKQGwzu343y7QMdjQ1FFGuZduTeTw7gK/0EOyQjaoySPTxK41w4whgcB5g625/vMy4I2WnKc1Igczl6ye4/fM3/TdTmvFNzqBglGQXpda/0IIu4zJuZjxEw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1735549785; c=relaxed/simple; bh=jiyynG1cSpqMCUPKcTsBXQhnImP0khzMQlCaAfJaSPQ=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=peQumK/9pL4Gb0hLH53lKBSuAGvBwE1d8Ht+CrrAdtiHWCpE4B7HpbZN/dWTEePE9xLFPuaZHc5z4XFyB8Er/nT/W3AAip2v/qMEqlRQIwQDP0AFY+VefurF/DO//HEharwGbboLmPzXgm+QcnDlPqYRv8AOWS/6MNyHg935ZLg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=fdipIao0; arc=none smtp.client-ip=209.85.214.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="fdipIao0" Received: by mail-pl1-f182.google.com with SMTP id d9443c01a7336-216395e151bso83725565ad.0 for ; Mon, 30 Dec 2024 01:09:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1735549783; x=1736154583; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=eYmLHVK0AtU6ilI7ucnXt5jLNeYE7S1uMrvnVL/VK/o=; b=fdipIao0MfSzxuzWBCZMky5alX88xvN/MbnJ7aPPBrjTGMYdonXH3cQrXGOHC5BHGE 1CbLRJGaZxP1k1OqLK5kcQIBeZFLwaPeflUsR6iUfioW884T1x2GaElMYG8WcXvobHur qSyL2ak4CGHxE0+qkv3L3oZZwquzLAy7uhHSGY06u61YsJVW1GyrK7KB3k7iLma6kKHd /A7xHVXQyHnufhe1DeBAMI+ir7bl7aIUJ5Q9fy+XvlRUZnCk2K+cttwdro0r0LiMX0h1 z1uBoeakBxn5ROMm17+siFl282KcIVBDdT0DWK2hUfV+gA1XeXj30OutE3Sw5VsybpCB af1A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1735549783; x=1736154583; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=eYmLHVK0AtU6ilI7ucnXt5jLNeYE7S1uMrvnVL/VK/o=; b=pzE1kzz5pxsczs1IRH7j8g/PVgMQ8sVVV0x85kb6nNqBG5cAwX9qlgIg2RZOm3Eaeo ZcOoeoH3ku3hPeFSQyYFC76ulSVMyixiWNXTgt1rkIP55PaojF2DxyrQrTGWh8O6N747 uVoxreLrXln3f/tTREuBBybgr+2zTxMdSosu6WYytZMp4GKIT5nU7EHsmhyw2NLGbCuY V3qMAPWjL/kUMmTyY8cT1j70aOjxEXWqB0K3pPRAcld/uCJzt8WLj770iFtVBpbggRtK sme0kiDHaZhaKu95pmmBWnoY/oEgAhJBIJ80eLxOR1t3MoyYJa+io70LRA3sHhwLX75U 9ONQ== X-Forwarded-Encrypted: i=1; AJvYcCW/mniz/ZPLSk/SmXyhIgBvmku+ZQT/thwGP1+D3T52oTqW5dkbPRhHJQz7HSw490I5VkeV/CufIbfqeZE=@vger.kernel.org X-Gm-Message-State: AOJu0YwyWvlQTucujI7hbIkSJvkSoD2kpvHQESWPulF3UaxpN5ggjuo/ gWEYcMJNr50mvvElYIUo4NZMk/tj+WWRNG8V2SaZlJWP/4gWTnzytXNg2KRLvkA= X-Gm-Gg: ASbGnct3za/CSeJB71A3VjDhRs2uBQRi90kD7jwgaxyvYOIc2gLsbK9fFPhrKWwdRO7 84QhOzmySszKrXNFf0flXcCJKeiRN33m716LwZ5gKWxMF9e1pXTuX7potl4ZmKBKFBeJ/5R9x7g 5s9AWldTH5QxTfrkN3eT2ZOR0Sxc1e1HsvjPXhNFsia3qydONSkHY6x+ZDhD1RUMk6QdqwvbG4P NrC2WDQsmD9rOrjBO1ap3BLkhzsvSESbO27UIVHfJjOchCeSX4uyfaAVeaQZiYDTWZq0mfGzrGh 4OsWN7JvDlN9+Dg0MLRGLg== X-Google-Smtp-Source: AGHT+IFA+DEWtAOVikEXL1Ne96u+hKYy4bbBxlgPSiON1NV20vNo54NDlhKj2iEhoHPJgNNx6Etn8w== X-Received: by 2002:a17:903:2a87:b0:216:3f6e:fabd with SMTP id d9443c01a7336-219da5cc2d0mr588821545ad.7.1735549782882; Mon, 30 Dec 2024 01:09:42 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-842aba72f7csm17057841a12.4.2024.12.30.01.09.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 30 Dec 2024 01:09:42 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, agordeev@linux.ibm.com, kevin.brodsky@arm.com, palmer@dabbelt.com, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com, arnd@arndb.de, will@kernel.org, aneesh.kumar@kernel.org, npiggin@gmail.com, dave.hansen@linux.intel.com, rppt@kernel.org, ryan.roberts@arm.com Cc: linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, linux-arch@vger.kernel.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, loongarch@lists.linux.dev, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-sh@vger.kernel.org, linux-um@lists.infradead.org, Qi Zheng Subject: [PATCH v4 05/15] arm64: pgtable: use mmu gather to free p4d level page table Date: Mon, 30 Dec 2024 17:07:40 +0800 Message-Id: <7c12112047ac230809aacd0379259414b9b0d3a3.1735549103.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Like other levels of page tables, also use mmu gather mechanism to free p4d level page table. Signed-off-by: Qi Zheng Originally-by: Peter Zijlstra (Intel) Cc: linux-arm-kernel@lists.infradead.org Reviewed-by: Kevin Brodsky --- arch/arm64/include/asm/pgalloc.h | 1 - arch/arm64/include/asm/tlb.h | 14 ++++++++++++++ 2 files changed, 14 insertions(+), 1 deletion(-) diff --git a/arch/arm64/include/asm/pgalloc.h b/arch/arm64/include/asm/pgal= loc.h index 2965f5a7e39e3..1b4509d3382c6 100644 --- a/arch/arm64/include/asm/pgalloc.h +++ b/arch/arm64/include/asm/pgalloc.h @@ -85,7 +85,6 @@ static inline void pgd_populate(struct mm_struct *mm, pgd= _t *pgdp, p4d_t *p4dp) __pgd_populate(pgdp, __pa(p4dp), pgdval); } =20 -#define __p4d_free_tlb(tlb, p4d, addr) p4d_free((tlb)->mm, p4d) #else static inline void __pgd_populate(pgd_t *pgdp, phys_addr_t p4dp, pgdval_t = prot) { diff --git a/arch/arm64/include/asm/tlb.h b/arch/arm64/include/asm/tlb.h index a947c6e784ed2..445282cde9afb 100644 --- a/arch/arm64/include/asm/tlb.h +++ b/arch/arm64/include/asm/tlb.h @@ -111,4 +111,18 @@ static inline void __pud_free_tlb(struct mmu_gather *t= lb, pud_t *pudp, } #endif =20 +#if CONFIG_PGTABLE_LEVELS > 4 +static inline void __p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4dp, + unsigned long addr) +{ + struct ptdesc *ptdesc =3D virt_to_ptdesc(p4dp); + + if (!pgtable_l5_enabled()) + return; + + pagetable_p4d_dtor(ptdesc); + tlb_remove_ptdesc(tlb, ptdesc); +} +#endif + #endif --=20 2.20.1 From nobody Sun May 10 06:27:04 2026 Received: from mail-pl1-f179.google.com (mail-pl1-f179.google.com [209.85.214.179]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4A4671A23BB for ; Mon, 30 Dec 2024 09:09:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.179 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1735549799; cv=none; b=q/pYwn2xMGs7HdoiCQ2vqU/9KoCa99KaPerbKCCsJeoAiclZ/rejgK2ZTlhOrT662E4qQNg9Fbk1wuAmFjq5+cM0OlLZ9Y4utHOBZu0y/CljFcDVp4CuybqzK+dlgiNZklAYttan4fPnOM6qq4IXnuYYdYzge75Fm/riWG/kcAU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1735549799; c=relaxed/simple; bh=gsn2MxaDHMIkVDeNJoAZ1GXzg4UbDMQdMp9AlgDJXWI=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=jgclOSWxR+nY7UY+jbGbr3Pi/jvbAztJDfeHkEqTg8O64uDTtAhjlxSqsSx9K+3t8NXyS7MckM3UEnwSVMLIyb4xK+HzARm1+VJNvntFt32DyPw6hQhfK+4GPtKsz4osTxvjMqhm3LQSg/1JIjCj9EJWrJwCoK3WTpi769TzJqM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=ZdDBIIoe; arc=none smtp.client-ip=209.85.214.179 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="ZdDBIIoe" Received: by mail-pl1-f179.google.com with SMTP id d9443c01a7336-21654fdd5daso108563395ad.1 for ; Mon, 30 Dec 2024 01:09:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1735549795; x=1736154595; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=xY41yoAggnrpKZDOvMIy0+FhpPRF9QFn+bDK7HKPzWw=; b=ZdDBIIoelCJjmLBiAvvITNIgs9m6LIoCNsC5x8RDr8WBQ/EfIiCbfp7WwIxMKjwY13 m9Tu8tZuobXFHMie0svrZpxYb810XVGhwhsiiFS8xegb+19Z0mkzzb6zr1p06EpCB00q ASZJz5wujN5irKhpTpcdgLubw2t3o2KQodRClOS/h3Ch9DQG7pEEEnWG3WtPDFMqw8rC nNxn6lZW81uNhA+v5NPe4AAUVn5YS88g8RFWQfjtOk6+dQE5QtB7LKV33ORmBrFTGqvO 8qJ/Ug6fXbwNi5F9KypqIzGrAMB6Bip/zuS4djeXVdQoQSpnGTP6oQghkUwh/iEST0vl KI6w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1735549795; x=1736154595; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=xY41yoAggnrpKZDOvMIy0+FhpPRF9QFn+bDK7HKPzWw=; b=F/HuS9WrZO2IHp23VwfcnSQr58MVLFQ1+8Q3mrLLRt6NAabgf3nDN9G3z78ICZv/aq m4efS/AuCgpC2DYZELC6Er5NKFauXCZrAcRf76eu9KXEYrCkbDhar9u9coqg5A9yJGI6 S72Xk1Z6GQpqy7rYXtq93pyD3b4R8NKxxvVXP81HQjvZfX3r6zSzKs1oxro2bhYFcSF7 PUSRQAghxx+L95x637feTMWF+LIP0HmB4hVPqjkEQ7/p0degUvX68u2CY8RjQvZGMZOf rZBmVcYUBcwv8WHD815YiBPXMuWmy0F9P9pL88aHMnFp6nk6fdU1IBm7RZNv4XG6SL/e LAOA== X-Forwarded-Encrypted: i=1; AJvYcCWaqtApjtDe0OQ/4HZfDga1MYtdl55uDyRINsn2YuBLOBWsGr/Z1KKJ1D2l5pmMai/7ZXAS8VcOif/y/dY=@vger.kernel.org X-Gm-Message-State: AOJu0YwXVFLq6k2qcJ03C5zJecGqfFsIiOPbW4DY3+L51t92m60LApgb RiOtDtzQY/FwexEuIB2YtHvWNWKd8mRsfnGy4mRM7VQJxJ3ojnQqPiwfLFR7+vA= X-Gm-Gg: ASbGncs6BUTQEBGFB/sWpsdi2ST6kRjsx/1kI7aWnJRpSHK8rCJJwBBDv4pltgtA11z RelFd3bPWf6ml4pFYmgEUWAorxF1qG4pcNuCIh88vHzR+5C5vw2pUHFHzmuKQbjUCObnidqBrlf RCoyEK+L6fSd92wIiyHqsmgVcxf26y9ZzNs9Vw2ZgPyXxUxipmZ6B9cESAqsamF77feQLtar1Uk Tmduf3VzI2Lc8Qq0TNv4H3rxl+bJsyi2MAGfGGd5K/5K/QKnjVwJpE7q/0uP+QSfdjW/7OEOKpr 1mcNYOR676oQZRHdAOYS1w== X-Google-Smtp-Source: AGHT+IGnRshrEAoIvJb/W9ouyd0xHkDgdKkbYKOXyUcaItniQPw/Z8/M3j6tdqefIia4rt8bUMG2kg== X-Received: by 2002:a05:6a21:328d:b0:1db:f68a:d943 with SMTP id adf61e73a8af0-1e5e0461b11mr51051580637.17.1735549795427; Mon, 30 Dec 2024 01:09:55 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-842aba72f7csm17057841a12.4.2024.12.30.01.09.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 30 Dec 2024 01:09:54 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, agordeev@linux.ibm.com, kevin.brodsky@arm.com, palmer@dabbelt.com, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com, arnd@arndb.de, will@kernel.org, aneesh.kumar@kernel.org, npiggin@gmail.com, dave.hansen@linux.intel.com, rppt@kernel.org, ryan.roberts@arm.com Cc: linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, linux-arch@vger.kernel.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, loongarch@lists.linux.dev, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-sh@vger.kernel.org, linux-um@lists.infradead.org, Qi Zheng Subject: [PATCH v4 06/15] s390: pgtable: add statistics for PUD and P4D level page table Date: Mon, 30 Dec 2024 17:07:41 +0800 Message-Id: <35be22a2b1666df729a9fc108c2da5cce266e4be.1735549103.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Like PMD and PTE level page table, also add statistics for PUD and P4D page table. Signed-off-by: Qi Zheng Suggested-by: Peter Zijlstra (Intel) Cc: linux-s390@vger.kernel.org Reviewed-by: Kevin Brodsky --- arch/s390/include/asm/pgalloc.h | 29 +++++++++++++++++++------- arch/s390/include/asm/tlb.h | 37 +++++++++++++++++---------------- 2 files changed, 40 insertions(+), 26 deletions(-) diff --git a/arch/s390/include/asm/pgalloc.h b/arch/s390/include/asm/pgallo= c.h index 7b84ef6dc4b6d..a0c1ca5d8423c 100644 --- a/arch/s390/include/asm/pgalloc.h +++ b/arch/s390/include/asm/pgalloc.h @@ -53,29 +53,42 @@ static inline p4d_t *p4d_alloc_one(struct mm_struct *mm= , unsigned long address) { unsigned long *table =3D crst_table_alloc(mm); =20 - if (table) - crst_table_init(table, _REGION2_ENTRY_EMPTY); + if (!table) + return NULL; + crst_table_init(table, _REGION2_ENTRY_EMPTY); + pagetable_p4d_ctor(virt_to_ptdesc(table)); + return (p4d_t *) table; } =20 static inline void p4d_free(struct mm_struct *mm, p4d_t *p4d) { - if (!mm_p4d_folded(mm)) - crst_table_free(mm, (unsigned long *) p4d); + if (mm_p4d_folded(mm)) + return; + + pagetable_p4d_dtor(virt_to_ptdesc(p4d)); + crst_table_free(mm, (unsigned long *) p4d); } =20 static inline pud_t *pud_alloc_one(struct mm_struct *mm, unsigned long add= ress) { unsigned long *table =3D crst_table_alloc(mm); - if (table) - crst_table_init(table, _REGION3_ENTRY_EMPTY); + + if (!table) + return NULL; + crst_table_init(table, _REGION3_ENTRY_EMPTY); + pagetable_pud_ctor(virt_to_ptdesc(table)); + return (pud_t *) table; } =20 static inline void pud_free(struct mm_struct *mm, pud_t *pud) { - if (!mm_pud_folded(mm)) - crst_table_free(mm, (unsigned long *) pud); + if (mm_pud_folded(mm)) + return; + + pagetable_pud_dtor(virt_to_ptdesc(pud)); + crst_table_free(mm, (unsigned long *) pud); } =20 static inline pmd_t *pmd_alloc_one(struct mm_struct *mm, unsigned long vma= ddr) diff --git a/arch/s390/include/asm/tlb.h b/arch/s390/include/asm/tlb.h index e95b2c8081eb8..b946964afce8e 100644 --- a/arch/s390/include/asm/tlb.h +++ b/arch/s390/include/asm/tlb.h @@ -110,24 +110,6 @@ static inline void pmd_free_tlb(struct mmu_gather *tlb= , pmd_t *pmd, tlb_remove_ptdesc(tlb, pmd); } =20 -/* - * p4d_free_tlb frees a pud table and clears the CRSTE for the - * region second table entry from the tlb. - * If the mm uses a four level page table the single p4d is freed - * as the pgd. p4d_free_tlb checks the asce_limit against 8PB - * to avoid the double free of the p4d in this case. - */ -static inline void p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d, - unsigned long address) -{ - if (mm_p4d_folded(tlb->mm)) - return; - __tlb_adjust_range(tlb, address, PAGE_SIZE); - tlb->mm->context.flush_mm =3D 1; - tlb->freed_tables =3D 1; - tlb_remove_ptdesc(tlb, p4d); -} - /* * pud_free_tlb frees a pud table and clears the CRSTE for the * region third table entry from the tlb. @@ -140,11 +122,30 @@ static inline void pud_free_tlb(struct mmu_gather *tl= b, pud_t *pud, { if (mm_pud_folded(tlb->mm)) return; + pagetable_pud_dtor(virt_to_ptdesc(pud)); tlb->mm->context.flush_mm =3D 1; tlb->freed_tables =3D 1; tlb->cleared_p4ds =3D 1; tlb_remove_ptdesc(tlb, pud); } =20 +/* + * p4d_free_tlb frees a p4d table and clears the CRSTE for the + * region second table entry from the tlb. + * If the mm uses a four level page table the single p4d is freed + * as the pgd. p4d_free_tlb checks the asce_limit against 8PB + * to avoid the double free of the p4d in this case. + */ +static inline void p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d, + unsigned long address) +{ + if (mm_p4d_folded(tlb->mm)) + return; + pagetable_p4d_dtor(virt_to_ptdesc(p4d)); + __tlb_adjust_range(tlb, address, PAGE_SIZE); + tlb->mm->context.flush_mm =3D 1; + tlb->freed_tables =3D 1; + tlb_remove_ptdesc(tlb, p4d); +} =20 #endif /* _S390_TLB_H */ --=20 2.20.1 From nobody Sun May 10 06:27:04 2026 Received: from mail-pl1-f173.google.com (mail-pl1-f173.google.com [209.85.214.173]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7C2A51A23A6 for ; Mon, 30 Dec 2024 09:10:08 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.173 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1735549811; cv=none; b=QVJeH3cDZcYatm2IqbQsrouuOo8Qo68viLoO7f7Cc8IohCq5kFZXBIFj1qs9v99QYLFJ5pQPyM1lXAFVLue1q9rWOlhpTT4L3p64YmBde6EkdUFa67a6VmLwwGpXxqWyfaOGmH+U9MrrltB7isd8dj5EBhB6dYx2JwXbr14PI1A= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1735549811; c=relaxed/simple; bh=RLIPEsUjaxmI3JDqp+aULApH2pPota4RtMIKgFUGpCI=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=MUHyvfy2bXj5NE1s5AJh35W9Uk95gmokX0rkD+BsaKZpXPlfZdflr1jo9e4z6WJDudLV3h31LV/5ORdsHP+XQCtxCf/2Wn15LmEi4Mv4KoieNAmZQVQ9ridfkYJe2glfJr7Dl5lPWgYscdNF7KvofKnpKPs9vJdI18l9qXGWRBA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=USvUUB1m; arc=none smtp.client-ip=209.85.214.173 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="USvUUB1m" Received: by mail-pl1-f173.google.com with SMTP id d9443c01a7336-2163b0c09afso112722915ad.0 for ; Mon, 30 Dec 2024 01:10:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1735549808; x=1736154608; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=if0a5qgfKyLZEs2UFcbOJ2pRLoezhNdJsDQpJdHhbkw=; b=USvUUB1maQ4pqZsdJmVagFvKABro5Etm6zL+fK+fq6wPD/2lsL2XOj0upmk2CHJlLF V6OiABD20OVuQK3SdjpuZ+tBBr/cPZ5dFBP0RXwDfSVqPH7Rq6LIIl85ihIIKTSHr9Iz Zn6ACS5L0kRZvFcH56pyv7OPZRiyUQhGrsHd3f/gzIl084nV+/S9jIuSnZbRT34egq7f r1fFRjDF/CcCPtwHD2vQ5RBKrhxqBwKfiG2lfbRuco39DQzhD4bFFZbM3W8Ysf3fU08K edDIT3+tlS3cVIeIhjy1lbL3AqV9dl4pgSbm+fuR7Y/zQXz3HPby8owXcZf0taJcTkL9 oh2w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1735549808; x=1736154608; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=if0a5qgfKyLZEs2UFcbOJ2pRLoezhNdJsDQpJdHhbkw=; b=wUY0tYpNapvzb9Qy3ZD15ZLtWo2VV5vFLSndPpgZLNxeZiy98yJiLCVupTivBq2doi vF1JboXUg8WYcKnnEt28CQLZpa005EaTxtH91vr8QFsk45tsyf/uGSRcYIVRv6y82ngW sbEX7OMUndXf3NtJxSLuAXEcoBEs00AYqpjuLGPpGlHBLv+Ivow9ndiJ6oPTuxuw+ZXc 4iOhVeip0T6M5eBL9PljRFMbxZA+iteWd0dFv1TH7PXd6WEqVDhBxePksVhXtbd1QTdV EQ7ABkrTBWPVW0CAU6ezmyFqZ5FVUmqJn8JxI3NZRx4UwuhaSy2YF5ssdCMY3VrE5s9i Ja0Q== X-Forwarded-Encrypted: i=1; AJvYcCW/2B28r5NUXIhbfvnK6Mf97GAPOHwP46eFVjqPzkLoqU1D/bloQeTD2oe4mGKF164hbRNKb4pEDuESNqI=@vger.kernel.org X-Gm-Message-State: AOJu0YwS6iE2jj7e28JeiYzAE64ks60Uybu9T0DlMOrN3V5MeDYFqW62 IYnzD2mxzcDk2PcXtL9FXe8eMD8FFcZ2wklLF6ldBydYtD5dFlg5WU1uRRUUZ9w= X-Gm-Gg: ASbGncsfAkvLzJhrCh7NYHV1rYTTwNdkhI76dUy1aYnzbfN7P4IUhNh5euCVyvQFRsv nwAuqWDL3xXi1YRuQ0YkxlHo1BtYBWOpXgjy75YGRbYRA9X2S0KQUlrmFGlYP6/x2/AYyQSv7ZJ r5+cDZUnugveSYmXxiEsH52Gg+puSudkwmA3rleSzsulyvYNJuDmZ+0KWY8JvqcIW3b+WRKJpya wouS+jwmFUc1FQEbLWnZG/jaM/li6Vx+fHcyfVf2nYHxKWCvJO5wDlqhKr56TsQVaQf6b3BGtxj 48XLBYDEdqupfT1ZgBDmBQ== X-Google-Smtp-Source: AGHT+IG+hQGJZEvqJjf0GGYYEDMNOYDO3VKGK36HZYKhi78RAlEt/pRApnM1X9rb4tsPHPP4epGREg== X-Received: by 2002:a05:6a20:431d:b0:1e1:bf3d:a190 with SMTP id adf61e73a8af0-1e5e080c83fmr53344548637.30.1735549807869; Mon, 30 Dec 2024 01:10:07 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-842aba72f7csm17057841a12.4.2024.12.30.01.09.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 30 Dec 2024 01:10:07 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, agordeev@linux.ibm.com, kevin.brodsky@arm.com, palmer@dabbelt.com, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com, arnd@arndb.de, will@kernel.org, aneesh.kumar@kernel.org, npiggin@gmail.com, dave.hansen@linux.intel.com, rppt@kernel.org, ryan.roberts@arm.com Cc: linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, linux-arch@vger.kernel.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, loongarch@lists.linux.dev, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-sh@vger.kernel.org, linux-um@lists.infradead.org, Qi Zheng Subject: [PATCH v4 07/15] mm: pgtable: introduce pagetable_dtor() Date: Mon, 30 Dec 2024 17:07:42 +0800 Message-Id: <8ada95453180c71b7fca92b9a9f11fa0f92d45a6.1735549103.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The pagetable_p*_dtor() are exactly the same except for the handling of ptlock. If we make ptlock_free() handle the case where ptdesc->ptl is NULL and remove VM_BUG_ON_PAGE() from pmd_ptlock_free(), we can unify pagetable_p*_dtor() into one function. Let's introduce pagetable_dtor() to do this. Later, pagetable_dtor() will be moved to tlb_remove_ptdesc(), so that ptlock and page table pages can be freed together (regardless of whether RCU is used). This prevents the use-after-free problem where the ptlock is freed immediately but the page table pages is freed later via RCU. Signed-off-by: Qi Zheng Originally-by: Peter Zijlstra (Intel) Reviewed-by: Kevin Brodsky --- Documentation/mm/split_page_table_lock.rst | 4 +- arch/arm/include/asm/tlb.h | 4 +- arch/arm64/include/asm/tlb.h | 8 ++-- arch/csky/include/asm/pgalloc.h | 2 +- arch/hexagon/include/asm/pgalloc.h | 2 +- arch/loongarch/include/asm/pgalloc.h | 2 +- arch/m68k/include/asm/mcf_pgalloc.h | 4 +- arch/m68k/include/asm/sun3_pgalloc.h | 2 +- arch/m68k/mm/motorola.c | 2 +- arch/mips/include/asm/pgalloc.h | 2 +- arch/nios2/include/asm/pgalloc.h | 2 +- arch/openrisc/include/asm/pgalloc.h | 2 +- arch/powerpc/mm/book3s64/mmu_context.c | 2 +- arch/powerpc/mm/book3s64/pgtable.c | 2 +- arch/powerpc/mm/pgtable-frag.c | 4 +- arch/riscv/include/asm/pgalloc.h | 8 ++-- arch/riscv/mm/init.c | 4 +- arch/s390/include/asm/pgalloc.h | 6 +-- arch/s390/include/asm/tlb.h | 6 +-- arch/s390/mm/pgalloc.c | 2 +- arch/sh/include/asm/pgalloc.h | 2 +- arch/sparc/mm/init_64.c | 2 +- arch/sparc/mm/srmmu.c | 2 +- arch/um/include/asm/pgalloc.h | 6 +-- arch/x86/mm/pgtable.c | 12 ++--- include/asm-generic/pgalloc.h | 8 ++-- include/linux/mm.h | 52 ++++------------------ mm/memory.c | 3 +- 28 files changed, 62 insertions(+), 95 deletions(-) diff --git a/Documentation/mm/split_page_table_lock.rst b/Documentation/mm/= split_page_table_lock.rst index 581446d4a4eba..8e1ceb0a6619a 100644 --- a/Documentation/mm/split_page_table_lock.rst +++ b/Documentation/mm/split_page_table_lock.rst @@ -62,7 +62,7 @@ Support of split page table lock by an architecture =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D =20 There's no need in special enabling of PTE split page table lock: everythi= ng -required is done by pagetable_pte_ctor() and pagetable_pte_dtor(), which +required is done by pagetable_pte_ctor() and pagetable_dtor(), which must be called on PTE table allocation / freeing. =20 Make sure the architecture doesn't use slab allocator for page table @@ -73,7 +73,7 @@ PMD split lock only makes sense if you have more than two= page table levels. =20 PMD split lock enabling requires pagetable_pmd_ctor() call on PMD table -allocation and pagetable_pmd_dtor() on freeing. +allocation and pagetable_dtor() on freeing. =20 Allocation usually happens in pmd_alloc_one(), freeing in pmd_free() and pmd_free_tlb(), but make sure you cover all PMD table allocation / freeing diff --git a/arch/arm/include/asm/tlb.h b/arch/arm/include/asm/tlb.h index f40d06ad5d2a3..ef79bf1e8563f 100644 --- a/arch/arm/include/asm/tlb.h +++ b/arch/arm/include/asm/tlb.h @@ -41,7 +41,7 @@ __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pte, uns= igned long addr) { struct ptdesc *ptdesc =3D page_ptdesc(pte); =20 - pagetable_pte_dtor(ptdesc); + pagetable_dtor(ptdesc); =20 #ifndef CONFIG_ARM_LPAE /* @@ -61,7 +61,7 @@ __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmdp, unsig= ned long addr) #ifdef CONFIG_ARM_LPAE struct ptdesc *ptdesc =3D virt_to_ptdesc(pmdp); =20 - pagetable_pmd_dtor(ptdesc); + pagetable_dtor(ptdesc); tlb_remove_ptdesc(tlb, ptdesc); #endif } diff --git a/arch/arm64/include/asm/tlb.h b/arch/arm64/include/asm/tlb.h index 445282cde9afb..408d0f36a8a8f 100644 --- a/arch/arm64/include/asm/tlb.h +++ b/arch/arm64/include/asm/tlb.h @@ -82,7 +82,7 @@ static inline void __pte_free_tlb(struct mmu_gather *tlb,= pgtable_t pte, { struct ptdesc *ptdesc =3D page_ptdesc(pte); =20 - pagetable_pte_dtor(ptdesc); + pagetable_dtor(ptdesc); tlb_remove_ptdesc(tlb, ptdesc); } =20 @@ -92,7 +92,7 @@ static inline void __pmd_free_tlb(struct mmu_gather *tlb,= pmd_t *pmdp, { struct ptdesc *ptdesc =3D virt_to_ptdesc(pmdp); =20 - pagetable_pmd_dtor(ptdesc); + pagetable_dtor(ptdesc); tlb_remove_ptdesc(tlb, ptdesc); } #endif @@ -106,7 +106,7 @@ static inline void __pud_free_tlb(struct mmu_gather *tl= b, pud_t *pudp, if (!pgtable_l4_enabled()) return; =20 - pagetable_pud_dtor(ptdesc); + pagetable_dtor(ptdesc); tlb_remove_ptdesc(tlb, ptdesc); } #endif @@ -120,7 +120,7 @@ static inline void __p4d_free_tlb(struct mmu_gather *tl= b, p4d_t *p4dp, if (!pgtable_l5_enabled()) return; =20 - pagetable_p4d_dtor(ptdesc); + pagetable_dtor(ptdesc); tlb_remove_ptdesc(tlb, ptdesc); } #endif diff --git a/arch/csky/include/asm/pgalloc.h b/arch/csky/include/asm/pgallo= c.h index 9c84c9012e534..f1ce5b7b28f22 100644 --- a/arch/csky/include/asm/pgalloc.h +++ b/arch/csky/include/asm/pgalloc.h @@ -63,7 +63,7 @@ static inline pgd_t *pgd_alloc(struct mm_struct *mm) =20 #define __pte_free_tlb(tlb, pte, address) \ do { \ - pagetable_pte_dtor(page_ptdesc(pte)); \ + pagetable_dtor(page_ptdesc(pte)); \ tlb_remove_page_ptdesc(tlb, page_ptdesc(pte)); \ } while (0) =20 diff --git a/arch/hexagon/include/asm/pgalloc.h b/arch/hexagon/include/asm/= pgalloc.h index 55988625e6fbc..40e42a0e71673 100644 --- a/arch/hexagon/include/asm/pgalloc.h +++ b/arch/hexagon/include/asm/pgalloc.h @@ -89,7 +89,7 @@ static inline void pmd_populate_kernel(struct mm_struct *= mm, pmd_t *pmd, =20 #define __pte_free_tlb(tlb, pte, addr) \ do { \ - pagetable_pte_dtor((page_ptdesc(pte))); \ + pagetable_dtor((page_ptdesc(pte))); \ tlb_remove_page_ptdesc((tlb), (page_ptdesc(pte))); \ } while (0) =20 diff --git a/arch/loongarch/include/asm/pgalloc.h b/arch/loongarch/include/= asm/pgalloc.h index a7b9c9e73593d..7211dff8c969e 100644 --- a/arch/loongarch/include/asm/pgalloc.h +++ b/arch/loongarch/include/asm/pgalloc.h @@ -57,7 +57,7 @@ static inline pte_t *pte_alloc_one_kernel(struct mm_struc= t *mm) =20 #define __pte_free_tlb(tlb, pte, address) \ do { \ - pagetable_pte_dtor(page_ptdesc(pte)); \ + pagetable_dtor(page_ptdesc(pte)); \ tlb_remove_page_ptdesc((tlb), page_ptdesc(pte)); \ } while (0) =20 diff --git a/arch/m68k/include/asm/mcf_pgalloc.h b/arch/m68k/include/asm/mc= f_pgalloc.h index 302c5bf67179e..22d6c1fcabfb4 100644 --- a/arch/m68k/include/asm/mcf_pgalloc.h +++ b/arch/m68k/include/asm/mcf_pgalloc.h @@ -37,7 +37,7 @@ static inline void __pte_free_tlb(struct mmu_gather *tlb,= pgtable_t pgtable, { struct ptdesc *ptdesc =3D virt_to_ptdesc(pgtable); =20 - pagetable_pte_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); } =20 @@ -61,7 +61,7 @@ static inline void pte_free(struct mm_struct *mm, pgtable= _t pgtable) { struct ptdesc *ptdesc =3D virt_to_ptdesc(pgtable); =20 - pagetable_pte_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); } =20 diff --git a/arch/m68k/include/asm/sun3_pgalloc.h b/arch/m68k/include/asm/s= un3_pgalloc.h index 4a137eecb6fe4..2b626cb3ad0ae 100644 --- a/arch/m68k/include/asm/sun3_pgalloc.h +++ b/arch/m68k/include/asm/sun3_pgalloc.h @@ -19,7 +19,7 @@ extern const char bad_pmd_string[]; =20 #define __pte_free_tlb(tlb, pte, addr) \ do { \ - pagetable_pte_dtor(page_ptdesc(pte)); \ + pagetable_dtor(page_ptdesc(pte)); \ tlb_remove_page_ptdesc((tlb), page_ptdesc(pte)); \ } while (0) =20 diff --git a/arch/m68k/mm/motorola.c b/arch/m68k/mm/motorola.c index c1761d309fc61..81715cece70c6 100644 --- a/arch/m68k/mm/motorola.c +++ b/arch/m68k/mm/motorola.c @@ -201,7 +201,7 @@ int free_pointer_table(void *table, int type) list_del(dp); mmu_page_dtor((void *)page); if (type =3D=3D TABLE_PTE) - pagetable_pte_dtor(virt_to_ptdesc((void *)page)); + pagetable_dtor(virt_to_ptdesc((void *)page)); free_page (page); return 1; } else if (ptable_list[type].next !=3D dp) { diff --git a/arch/mips/include/asm/pgalloc.h b/arch/mips/include/asm/pgallo= c.h index f4440edcd8fe2..36d9805033c4b 100644 --- a/arch/mips/include/asm/pgalloc.h +++ b/arch/mips/include/asm/pgalloc.h @@ -56,7 +56,7 @@ static inline void pgd_free(struct mm_struct *mm, pgd_t *= pgd) =20 #define __pte_free_tlb(tlb, pte, address) \ do { \ - pagetable_pte_dtor(page_ptdesc(pte)); \ + pagetable_dtor(page_ptdesc(pte)); \ tlb_remove_page_ptdesc((tlb), page_ptdesc(pte)); \ } while (0) =20 diff --git a/arch/nios2/include/asm/pgalloc.h b/arch/nios2/include/asm/pgal= loc.h index ce6bb8e74271f..12a536b7bfbd4 100644 --- a/arch/nios2/include/asm/pgalloc.h +++ b/arch/nios2/include/asm/pgalloc.h @@ -30,7 +30,7 @@ extern pgd_t *pgd_alloc(struct mm_struct *mm); =20 #define __pte_free_tlb(tlb, pte, addr) \ do { \ - pagetable_pte_dtor(page_ptdesc(pte)); \ + pagetable_dtor(page_ptdesc(pte)); \ tlb_remove_page_ptdesc((tlb), (page_ptdesc(pte))); \ } while (0) =20 diff --git a/arch/openrisc/include/asm/pgalloc.h b/arch/openrisc/include/as= m/pgalloc.h index c6a73772a5466..596e2355824e3 100644 --- a/arch/openrisc/include/asm/pgalloc.h +++ b/arch/openrisc/include/asm/pgalloc.h @@ -68,7 +68,7 @@ extern pte_t *pte_alloc_one_kernel(struct mm_struct *mm); =20 #define __pte_free_tlb(tlb, pte, addr) \ do { \ - pagetable_pte_dtor(page_ptdesc(pte)); \ + pagetable_dtor(page_ptdesc(pte)); \ tlb_remove_page_ptdesc((tlb), (page_ptdesc(pte))); \ } while (0) =20 diff --git a/arch/powerpc/mm/book3s64/mmu_context.c b/arch/powerpc/mm/book3= s64/mmu_context.c index 1715b07c630c9..4e1e45420bd49 100644 --- a/arch/powerpc/mm/book3s64/mmu_context.c +++ b/arch/powerpc/mm/book3s64/mmu_context.c @@ -253,7 +253,7 @@ static void pmd_frag_destroy(void *pmd_frag) count =3D ((unsigned long)pmd_frag & ~PAGE_MASK) >> PMD_FRAG_SIZE_SHIFT; /* We allow PTE_FRAG_NR fragments from a PTE page */ if (atomic_sub_and_test(PMD_FRAG_NR - count, &ptdesc->pt_frag_refcount)) { - pagetable_pmd_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); } } diff --git a/arch/powerpc/mm/book3s64/pgtable.c b/arch/powerpc/mm/book3s64/= pgtable.c index 3745425280808..3f28e4acd920b 100644 --- a/arch/powerpc/mm/book3s64/pgtable.c +++ b/arch/powerpc/mm/book3s64/pgtable.c @@ -477,7 +477,7 @@ void pmd_fragment_free(unsigned long *pmd) =20 BUG_ON(atomic_read(&ptdesc->pt_frag_refcount) <=3D 0); if (atomic_dec_and_test(&ptdesc->pt_frag_refcount)) { - pagetable_pmd_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); } } diff --git a/arch/powerpc/mm/pgtable-frag.c b/arch/powerpc/mm/pgtable-frag.c index e89f64a0f24ae..713268ccb1a0e 100644 --- a/arch/powerpc/mm/pgtable-frag.c +++ b/arch/powerpc/mm/pgtable-frag.c @@ -25,7 +25,7 @@ void pte_frag_destroy(void *pte_frag) count =3D ((unsigned long)pte_frag & ~PAGE_MASK) >> PTE_FRAG_SIZE_SHIFT; /* We allow PTE_FRAG_NR fragments from a PTE page */ if (atomic_sub_and_test(PTE_FRAG_NR - count, &ptdesc->pt_frag_refcount)) { - pagetable_pte_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); } } @@ -111,7 +111,7 @@ static void pte_free_now(struct rcu_head *head) struct ptdesc *ptdesc; =20 ptdesc =3D container_of(head, struct ptdesc, pt_rcu_head); - pagetable_pte_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); } =20 diff --git a/arch/riscv/include/asm/pgalloc.h b/arch/riscv/include/asm/pgal= loc.h index 3466fbe2e508d..b6793c5c99296 100644 --- a/arch/riscv/include/asm/pgalloc.h +++ b/arch/riscv/include/asm/pgalloc.h @@ -100,7 +100,7 @@ static inline void __pud_free_tlb(struct mmu_gather *tl= b, pud_t *pud, if (pgtable_l4_enabled) { struct ptdesc *ptdesc =3D virt_to_ptdesc(pud); =20 - pagetable_pud_dtor(ptdesc); + pagetable_dtor(ptdesc); riscv_tlb_remove_ptdesc(tlb, ptdesc); } } @@ -111,7 +111,7 @@ static inline void __p4d_free_tlb(struct mmu_gather *tl= b, p4d_t *p4d, if (pgtable_l5_enabled) { struct ptdesc *ptdesc =3D virt_to_ptdesc(p4d); =20 - pagetable_p4d_dtor(ptdesc); + pagetable_dtor(ptdesc); riscv_tlb_remove_ptdesc(tlb, virt_to_ptdesc(p4d)); } } @@ -144,7 +144,7 @@ static inline void __pmd_free_tlb(struct mmu_gather *tl= b, pmd_t *pmd, { struct ptdesc *ptdesc =3D virt_to_ptdesc(pmd); =20 - pagetable_pmd_dtor(ptdesc); + pagetable_dtor(ptdesc); riscv_tlb_remove_ptdesc(tlb, ptdesc); } =20 @@ -155,7 +155,7 @@ static inline void __pte_free_tlb(struct mmu_gather *tl= b, pgtable_t pte, { struct ptdesc *ptdesc =3D page_ptdesc(pte); =20 - pagetable_pte_dtor(ptdesc); + pagetable_dtor(ptdesc); riscv_tlb_remove_ptdesc(tlb, ptdesc); } #endif /* CONFIG_MMU */ diff --git a/arch/riscv/mm/init.c b/arch/riscv/mm/init.c index fc53ce748c804..8d703fb51b1dc 100644 --- a/arch/riscv/mm/init.c +++ b/arch/riscv/mm/init.c @@ -1558,7 +1558,7 @@ static void __meminit free_pte_table(pte_t *pte_start= , pmd_t *pmd) return; } =20 - pagetable_pte_dtor(ptdesc); + pagetable_dtor(ptdesc); if (PageReserved(page)) free_reserved_page(page); else @@ -1580,7 +1580,7 @@ static void __meminit free_pmd_table(pmd_t *pmd_start= , pud_t *pud, bool is_vmemm } =20 if (!is_vmemmap) - pagetable_pmd_dtor(ptdesc); + pagetable_dtor(ptdesc); if (PageReserved(page)) free_reserved_page(page); else diff --git a/arch/s390/include/asm/pgalloc.h b/arch/s390/include/asm/pgallo= c.h index a0c1ca5d8423c..5fced6d3c36b0 100644 --- a/arch/s390/include/asm/pgalloc.h +++ b/arch/s390/include/asm/pgalloc.h @@ -66,7 +66,7 @@ static inline void p4d_free(struct mm_struct *mm, p4d_t *= p4d) if (mm_p4d_folded(mm)) return; =20 - pagetable_p4d_dtor(virt_to_ptdesc(p4d)); + pagetable_dtor(virt_to_ptdesc(p4d)); crst_table_free(mm, (unsigned long *) p4d); } =20 @@ -87,7 +87,7 @@ static inline void pud_free(struct mm_struct *mm, pud_t *= pud) if (mm_pud_folded(mm)) return; =20 - pagetable_pud_dtor(virt_to_ptdesc(pud)); + pagetable_dtor(virt_to_ptdesc(pud)); crst_table_free(mm, (unsigned long *) pud); } =20 @@ -109,7 +109,7 @@ static inline void pmd_free(struct mm_struct *mm, pmd_t= *pmd) { if (mm_pmd_folded(mm)) return; - pagetable_pmd_dtor(virt_to_ptdesc(pmd)); + pagetable_dtor(virt_to_ptdesc(pmd)); crst_table_free(mm, (unsigned long *) pmd); } =20 diff --git a/arch/s390/include/asm/tlb.h b/arch/s390/include/asm/tlb.h index b946964afce8e..74b6fba4c2ee3 100644 --- a/arch/s390/include/asm/tlb.h +++ b/arch/s390/include/asm/tlb.h @@ -102,7 +102,7 @@ static inline void pmd_free_tlb(struct mmu_gather *tlb,= pmd_t *pmd, { if (mm_pmd_folded(tlb->mm)) return; - pagetable_pmd_dtor(virt_to_ptdesc(pmd)); + pagetable_dtor(virt_to_ptdesc(pmd)); __tlb_adjust_range(tlb, address, PAGE_SIZE); tlb->mm->context.flush_mm =3D 1; tlb->freed_tables =3D 1; @@ -122,7 +122,7 @@ static inline void pud_free_tlb(struct mmu_gather *tlb,= pud_t *pud, { if (mm_pud_folded(tlb->mm)) return; - pagetable_pud_dtor(virt_to_ptdesc(pud)); + pagetable_dtor(virt_to_ptdesc(pud)); tlb->mm->context.flush_mm =3D 1; tlb->freed_tables =3D 1; tlb->cleared_p4ds =3D 1; @@ -141,7 +141,7 @@ static inline void p4d_free_tlb(struct mmu_gather *tlb,= p4d_t *p4d, { if (mm_p4d_folded(tlb->mm)) return; - pagetable_p4d_dtor(virt_to_ptdesc(p4d)); + pagetable_dtor(virt_to_ptdesc(p4d)); __tlb_adjust_range(tlb, address, PAGE_SIZE); tlb->mm->context.flush_mm =3D 1; tlb->freed_tables =3D 1; diff --git a/arch/s390/mm/pgalloc.c b/arch/s390/mm/pgalloc.c index 58696a0c4e4ac..569de24d33761 100644 --- a/arch/s390/mm/pgalloc.c +++ b/arch/s390/mm/pgalloc.c @@ -182,7 +182,7 @@ unsigned long *page_table_alloc(struct mm_struct *mm) =20 static void pagetable_pte_dtor_free(struct ptdesc *ptdesc) { - pagetable_pte_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); } =20 diff --git a/arch/sh/include/asm/pgalloc.h b/arch/sh/include/asm/pgalloc.h index 5d8577ab15911..96d938fdf2244 100644 --- a/arch/sh/include/asm/pgalloc.h +++ b/arch/sh/include/asm/pgalloc.h @@ -34,7 +34,7 @@ static inline void pmd_populate(struct mm_struct *mm, pmd= _t *pmd, =20 #define __pte_free_tlb(tlb, pte, addr) \ do { \ - pagetable_pte_dtor(page_ptdesc(pte)); \ + pagetable_dtor(page_ptdesc(pte)); \ tlb_remove_page_ptdesc((tlb), (page_ptdesc(pte))); \ } while (0) =20 diff --git a/arch/sparc/mm/init_64.c b/arch/sparc/mm/init_64.c index 21f8cbbd0581c..05882bca5b732 100644 --- a/arch/sparc/mm/init_64.c +++ b/arch/sparc/mm/init_64.c @@ -2915,7 +2915,7 @@ static void __pte_free(pgtable_t pte) { struct ptdesc *ptdesc =3D virt_to_ptdesc(pte); =20 - pagetable_pte_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); } =20 diff --git a/arch/sparc/mm/srmmu.c b/arch/sparc/mm/srmmu.c index 9df51a62333d6..e3a72c884b867 100644 --- a/arch/sparc/mm/srmmu.c +++ b/arch/sparc/mm/srmmu.c @@ -372,7 +372,7 @@ void pte_free(struct mm_struct *mm, pgtable_t ptep) page =3D pfn_to_page(__nocache_pa((unsigned long)ptep) >> PAGE_SHIFT); spin_lock(&mm->page_table_lock); if (page_ref_dec_return(page) =3D=3D 1) - pagetable_pte_dtor(page_ptdesc(page)); + pagetable_dtor(page_ptdesc(page)); spin_unlock(&mm->page_table_lock); =20 srmmu_free_nocache(ptep, SRMMU_PTE_TABLE_SIZE); diff --git a/arch/um/include/asm/pgalloc.h b/arch/um/include/asm/pgalloc.h index 04fb4e6969a46..f0af23c3aeb2b 100644 --- a/arch/um/include/asm/pgalloc.h +++ b/arch/um/include/asm/pgalloc.h @@ -27,7 +27,7 @@ extern pgd_t *pgd_alloc(struct mm_struct *); =20 #define __pte_free_tlb(tlb, pte, address) \ do { \ - pagetable_pte_dtor(page_ptdesc(pte)); \ + pagetable_dtor(page_ptdesc(pte)); \ tlb_remove_page_ptdesc((tlb), (page_ptdesc(pte))); \ } while (0) =20 @@ -35,7 +35,7 @@ do { \ =20 #define __pmd_free_tlb(tlb, pmd, address) \ do { \ - pagetable_pmd_dtor(virt_to_ptdesc(pmd)); \ + pagetable_dtor(virt_to_ptdesc(pmd)); \ tlb_remove_page_ptdesc((tlb), virt_to_ptdesc(pmd)); \ } while (0) =20 @@ -43,7 +43,7 @@ do { \ =20 #define __pud_free_tlb(tlb, pud, address) \ do { \ - pagetable_pud_dtor(virt_to_ptdesc(pud)); \ + pagetable_dtor(virt_to_ptdesc(pud)); \ tlb_remove_page_ptdesc((tlb), virt_to_ptdesc(pud)); \ } while (0) =20 diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c index 3d6e84da45b24..a6cd9660e29ec 100644 --- a/arch/x86/mm/pgtable.c +++ b/arch/x86/mm/pgtable.c @@ -60,7 +60,7 @@ early_param("userpte", setup_userpte); =20 void ___pte_free_tlb(struct mmu_gather *tlb, struct page *pte) { - pagetable_pte_dtor(page_ptdesc(pte)); + pagetable_dtor(page_ptdesc(pte)); paravirt_release_pte(page_to_pfn(pte)); paravirt_tlb_remove_table(tlb, pte); } @@ -77,7 +77,7 @@ void ___pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd) #ifdef CONFIG_X86_PAE tlb->need_flush_all =3D 1; #endif - pagetable_pmd_dtor(ptdesc); + pagetable_dtor(ptdesc); paravirt_tlb_remove_table(tlb, ptdesc_page(ptdesc)); } =20 @@ -86,7 +86,7 @@ void ___pud_free_tlb(struct mmu_gather *tlb, pud_t *pud) { struct ptdesc *ptdesc =3D virt_to_ptdesc(pud); =20 - pagetable_pud_dtor(ptdesc); + pagetable_dtor(ptdesc); paravirt_release_pud(__pa(pud) >> PAGE_SHIFT); paravirt_tlb_remove_table(tlb, virt_to_page(pud)); } @@ -96,7 +96,7 @@ void ___p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d) { struct ptdesc *ptdesc =3D virt_to_ptdesc(p4d); =20 - pagetable_p4d_dtor(ptdesc); + pagetable_dtor(ptdesc); paravirt_release_p4d(__pa(p4d) >> PAGE_SHIFT); paravirt_tlb_remove_table(tlb, virt_to_page(p4d)); } @@ -233,7 +233,7 @@ static void free_pmds(struct mm_struct *mm, pmd_t *pmds= [], int count) if (pmds[i]) { ptdesc =3D virt_to_ptdesc(pmds[i]); =20 - pagetable_pmd_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); mm_dec_nr_pmds(mm); } @@ -867,7 +867,7 @@ int pud_free_pmd_page(pud_t *pud, unsigned long addr) =20 free_page((unsigned long)pmd_sv); =20 - pagetable_pmd_dtor(virt_to_ptdesc(pmd)); + pagetable_dtor(virt_to_ptdesc(pmd)); free_page((unsigned long)pmd); =20 return 1; diff --git a/include/asm-generic/pgalloc.h b/include/asm-generic/pgalloc.h index bb482eeca0c3e..4afb346eae255 100644 --- a/include/asm-generic/pgalloc.h +++ b/include/asm-generic/pgalloc.h @@ -109,7 +109,7 @@ static inline void pte_free(struct mm_struct *mm, struc= t page *pte_page) { struct ptdesc *ptdesc =3D page_ptdesc(pte_page); =20 - pagetable_pte_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); } =20 @@ -153,7 +153,7 @@ static inline void pmd_free(struct mm_struct *mm, pmd_t= *pmd) struct ptdesc *ptdesc =3D virt_to_ptdesc(pmd); =20 BUG_ON((unsigned long)pmd & (PAGE_SIZE-1)); - pagetable_pmd_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); } #endif @@ -202,7 +202,7 @@ static inline void __pud_free(struct mm_struct *mm, pud= _t *pud) struct ptdesc *ptdesc =3D virt_to_ptdesc(pud); =20 BUG_ON((unsigned long)pud & (PAGE_SIZE-1)); - pagetable_pud_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); } =20 @@ -248,7 +248,7 @@ static inline void __p4d_free(struct mm_struct *mm, p4d= _t *p4d) struct ptdesc *ptdesc =3D virt_to_ptdesc(p4d); =20 BUG_ON((unsigned long)p4d & (PAGE_SIZE-1)); - pagetable_p4d_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); } =20 diff --git a/include/linux/mm.h b/include/linux/mm.h index 5d82f42ddd5cc..cad11fa10c192 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2992,6 +2992,15 @@ static inline bool ptlock_init(struct ptdesc *ptdesc= ) { return true; } static inline void ptlock_free(struct ptdesc *ptdesc) {} #endif /* defined(CONFIG_SPLIT_PTE_PTLOCKS) */ =20 +static inline void pagetable_dtor(struct ptdesc *ptdesc) +{ + struct folio *folio =3D ptdesc_folio(ptdesc); + + ptlock_free(ptdesc); + __folio_clear_pgtable(folio); + lruvec_stat_sub_folio(folio, NR_PAGETABLE); +} + static inline bool pagetable_pte_ctor(struct ptdesc *ptdesc) { struct folio *folio =3D ptdesc_folio(ptdesc); @@ -3003,15 +3012,6 @@ static inline bool pagetable_pte_ctor(struct ptdesc = *ptdesc) return true; } =20 -static inline void pagetable_pte_dtor(struct ptdesc *ptdesc) -{ - struct folio *folio =3D ptdesc_folio(ptdesc); - - ptlock_free(ptdesc); - __folio_clear_pgtable(folio); - lruvec_stat_sub_folio(folio, NR_PAGETABLE); -} - pte_t *___pte_offset_map(pmd_t *pmd, unsigned long addr, pmd_t *pmdvalp); static inline pte_t *__pte_offset_map(pmd_t *pmd, unsigned long addr, pmd_t *pmdvalp) @@ -3088,14 +3088,6 @@ static inline bool pmd_ptlock_init(struct ptdesc *pt= desc) return ptlock_init(ptdesc); } =20 -static inline void pmd_ptlock_free(struct ptdesc *ptdesc) -{ -#ifdef CONFIG_TRANSPARENT_HUGEPAGE - VM_BUG_ON_PAGE(ptdesc->pmd_huge_pte, ptdesc_page(ptdesc)); -#endif - ptlock_free(ptdesc); -} - #define pmd_huge_pte(mm, pmd) (pmd_ptdesc(pmd)->pmd_huge_pte) =20 #else @@ -3106,7 +3098,6 @@ static inline spinlock_t *pmd_lockptr(struct mm_struc= t *mm, pmd_t *pmd) } =20 static inline bool pmd_ptlock_init(struct ptdesc *ptdesc) { return true; } -static inline void pmd_ptlock_free(struct ptdesc *ptdesc) {} =20 #define pmd_huge_pte(mm, pmd) ((mm)->pmd_huge_pte) =20 @@ -3131,15 +3122,6 @@ static inline bool pagetable_pmd_ctor(struct ptdesc = *ptdesc) return true; } =20 -static inline void pagetable_pmd_dtor(struct ptdesc *ptdesc) -{ - struct folio *folio =3D ptdesc_folio(ptdesc); - - pmd_ptlock_free(ptdesc); - __folio_clear_pgtable(folio); - lruvec_stat_sub_folio(folio, NR_PAGETABLE); -} - /* * No scalability reason to split PUD locks yet, but follow the same patte= rn * as the PMD locks to make it easier if we decide to. The VM should not = be @@ -3167,14 +3149,6 @@ static inline void pagetable_pud_ctor(struct ptdesc = *ptdesc) lruvec_stat_add_folio(folio, NR_PAGETABLE); } =20 -static inline void pagetable_pud_dtor(struct ptdesc *ptdesc) -{ - struct folio *folio =3D ptdesc_folio(ptdesc); - - __folio_clear_pgtable(folio); - lruvec_stat_sub_folio(folio, NR_PAGETABLE); -} - static inline void pagetable_p4d_ctor(struct ptdesc *ptdesc) { struct folio *folio =3D ptdesc_folio(ptdesc); @@ -3183,14 +3157,6 @@ static inline void pagetable_p4d_ctor(struct ptdesc = *ptdesc) lruvec_stat_add_folio(folio, NR_PAGETABLE); } =20 -static inline void pagetable_p4d_dtor(struct ptdesc *ptdesc) -{ - struct folio *folio =3D ptdesc_folio(ptdesc); - - __folio_clear_pgtable(folio); - lruvec_stat_sub_folio(folio, NR_PAGETABLE); -} - extern void __init pagecache_init(void); extern void free_initmem(void); =20 diff --git a/mm/memory.c b/mm/memory.c index 9423967b24180..ad871e564568b 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -7051,7 +7051,8 @@ bool ptlock_alloc(struct ptdesc *ptdesc) =20 void ptlock_free(struct ptdesc *ptdesc) { - kmem_cache_free(page_ptl_cachep, ptdesc->ptl); + if (ptdesc->ptl) + kmem_cache_free(page_ptl_cachep, ptdesc->ptl); } #endif =20 --=20 2.20.1 From nobody Sun May 10 06:27:04 2026 Received: from mail-pl1-f170.google.com (mail-pl1-f170.google.com [209.85.214.170]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BF0C81A2846 for ; Mon, 30 Dec 2024 09:10:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.170 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1735549824; cv=none; b=fWDA8ScQMRYFlEdTu7/UEG1kqhA1hvEuXlz6kg6KTpB2dhz/Eaxls4u9qzThnYb2mxdXTjhxARf32QIuLP7YawX6A1pmlI0GxWKLdRnK1jxOQulLgY+RDAgie9PsabFONlSrjMkunRRpGewHvBTfMm5RnGii7xwFfgDFDIEG164= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1735549824; c=relaxed/simple; bh=H/x1u07M3HXF9oVd+o3reuerQgd2pca8MMjlY49GFUc=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=ilzMm0F0yEQJKoK5WMTT7bKdUta1oIZVpiYWGSprUkSj6QqLwwRz5LG/tBe0J9Rp5/3cR9apisNGneset1YoLjEKRjqITzLR7YiUthBz1WT1I0kPn4O552MpqXuMWCPWm7TRljkTRha3SLLDgztLcq2MGosgfFTKPWIfGTpfvwk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=QtFMIkv2; arc=none smtp.client-ip=209.85.214.170 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="QtFMIkv2" Received: by mail-pl1-f170.google.com with SMTP id d9443c01a7336-21619108a6bso107029185ad.3 for ; Mon, 30 Dec 2024 01:10:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1735549820; x=1736154620; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=qA0mhWnXo6upWemI3POPEc6VJyAkyxFJggkOB6WNq2k=; b=QtFMIkv2dn6svHZ1SXPQ4UcYxOWhSxEtecpc9ZcR84UJIkvDc0nxxdiBEQqkLzNte+ cOxd6huGH1XR7kB1cB/zsiN7Qx34VpSyDjv955svhVtX5ie0cLqDc+DpV+m3aY0sncw1 tWKBxDXeRR8moOHkZgP3jB7Zs12UvZbtQnBi8xea0wQiKkQKF1v7PoGlx7OKqV1QBzbr JMa1XWurnJ+ziDTJnN85CxxdfmjzNS2FWd1eRtGaMIIYQ9zDgH98o3haRQTCSQbHZQv/ DZ9Q2RF8X4RYJ7RuPOwnMeo6uwMeFpZeYBos5Bj51v3vzvJDZjzS0uu0C/dhUOfW02Tf fZWQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1735549820; x=1736154620; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=qA0mhWnXo6upWemI3POPEc6VJyAkyxFJggkOB6WNq2k=; b=lLR346xCzEquzPx/SZYatjvC26mH29LtfTFhnysi3JchV4DhkfDyPBY6W62ZQ5OaXJ 5DLfhKTiPm3IQouOal4H8agAGU0o5JHXcTvGRMKqL1sJEJE6iE82aUWAu5ipYzxIv73J vCJSYG821qF4NPiV3nLDpsM+cHRKJFiMVKeVYHCIHCQ6Cv3cbTpnTMUXAbMXmUv6/0mT cw+5CNNF7lSlW2YbdfHvfp21rTc3w+MPq+LFdybY9iUfCuoSO6cVvkLta5ZjV5Q5HoTX D/JusIzJHwIn3OgPqEhwgXsBmVc2dKW6Iqrk9bhuVCiI+hmyt9mFw90GYJ8gTsHK3U9t wKqg== X-Forwarded-Encrypted: i=1; AJvYcCWNZZ4v5YRfwfhwrx/fLSetI4N32fN5N26M/NLo77/+Z5lrTx9UplWv5exGhp+yIQCymIb74m37/zSjApc=@vger.kernel.org X-Gm-Message-State: AOJu0Ywjs/MCX1nYDnuVZADNDWCVG/GtNYzHirmNFrJCecjEl8YftjkS szQlQIzNQq1OAXwTXoCpSIRQwtSW2Cw/hpyRFPlPLET7R8vEulQBwZzWlhpI5EQ= X-Gm-Gg: ASbGncuvCstv2FJMHyboi2Artp9vM/0odD7F9xzaq9oHcco7FxJ3aFAuH+dPyJPRFfy no7ltLDvLjT6J/e9v80CkrtonJ/e/5F2GbZvsS4HYDELYo3PfFM7I+VFX4Az706E8e0dhfSF2v1 Xu49ZE7TA3FvZq4mDfvRdq3SEhEaanRKBk8QztPmxzioQFRJabE4e+YT2PKemOx/VdXy+2niJvi f+z+Guk2Acn3wCvKyg+i2Z+SmF9cdTuu/k/tlt5kSOkSkHJjgh3sLseRUtc8aPR9BPDocMqgezJ Q1Q1nu7RL/wfPSa8zfVJvQ== X-Google-Smtp-Source: AGHT+IFX/3Df8AQRXbgF3uoP6h0ZSX98E/foDuPWt5IdgjLk00IpLZqU8w+lhMMb/JJOCncpjT7B0g== X-Received: by 2002:a05:6a21:670b:b0:1dc:7907:6d67 with SMTP id adf61e73a8af0-1e5e081c8d0mr60502051637.40.1735549820172; Mon, 30 Dec 2024 01:10:20 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-842aba72f7csm17057841a12.4.2024.12.30.01.10.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 30 Dec 2024 01:10:19 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, agordeev@linux.ibm.com, kevin.brodsky@arm.com, palmer@dabbelt.com, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com, arnd@arndb.de, will@kernel.org, aneesh.kumar@kernel.org, npiggin@gmail.com, dave.hansen@linux.intel.com, rppt@kernel.org, ryan.roberts@arm.com Cc: linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, linux-arch@vger.kernel.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, loongarch@lists.linux.dev, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-sh@vger.kernel.org, linux-um@lists.infradead.org, Qi Zheng Subject: [PATCH v4 08/15] arm: pgtable: move pagetable_dtor() to __tlb_remove_table() Date: Mon, 30 Dec 2024 17:07:43 +0800 Message-Id: <955162bfbbcd9fbb3b074e1fe2aef4f64b61d6f9.1735549103.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Move pagetable_dtor() to __tlb_remove_table(), so that ptlock and page table pages can be freed together (regardless of whether RCU is used). This prevents the use-after-free problem where the ptlock is freed immediately but the page table pages is freed later via RCU. Page tables shouldn't have swap cache, so use pagetable_free() instead of free_page_and_swap_cache() to free page table pages. Signed-off-by: Qi Zheng Suggested-by: Peter Zijlstra (Intel) Cc: linux-arm-kernel@lists.infradead.org Reviewed-by: Kevin Brodsky --- arch/arm/include/asm/tlb.h | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/arch/arm/include/asm/tlb.h b/arch/arm/include/asm/tlb.h index ef79bf1e8563f..264ab635e807a 100644 --- a/arch/arm/include/asm/tlb.h +++ b/arch/arm/include/asm/tlb.h @@ -26,12 +26,14 @@ =20 #else /* !CONFIG_MMU */ =20 -#include #include =20 static inline void __tlb_remove_table(void *_table) { - free_page_and_swap_cache((struct page *)_table); + struct ptdesc *ptdesc =3D (struct ptdesc *)_table; + + pagetable_dtor(ptdesc); + pagetable_free(ptdesc); } =20 #include @@ -41,8 +43,6 @@ __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pte, uns= igned long addr) { struct ptdesc *ptdesc =3D page_ptdesc(pte); =20 - pagetable_dtor(ptdesc); - #ifndef CONFIG_ARM_LPAE /* * With the classic ARM MMU, a pte page has two corresponding pmd @@ -61,7 +61,6 @@ __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmdp, unsig= ned long addr) #ifdef CONFIG_ARM_LPAE struct ptdesc *ptdesc =3D virt_to_ptdesc(pmdp); =20 - pagetable_dtor(ptdesc); tlb_remove_ptdesc(tlb, ptdesc); #endif } --=20 2.20.1 From nobody Sun May 10 06:27:04 2026 Received: from mail-pl1-f176.google.com (mail-pl1-f176.google.com [209.85.214.176]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5B1A31A38F9 for ; Mon, 30 Dec 2024 09:10:33 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.176 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1735549836; cv=none; b=AsCUrAfDZOWEvdHNUJgAch4aVmKSymxpwzuhSyxf5jSGuge8TsPeM/lwsiesqoSOrjoL3bVwm2oroM0HyJy33Aqhha01eQfRGRa3lnzFhT0bNuAnD8Ia1b7ZKiZb5GFiWbRJ2P5u8rP59EPnhgU3VGER5dYB5AE/7l+FXRsPuc8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1735549836; c=relaxed/simple; bh=w9Tb6kkg2Q49UGEfUZpSUyo28N/wow0+21NvTYUEzY4=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=qDHtZ2Q2Yvy7d2nVXxcZr+kXQgwV7o/GytFPPXGhl+FH0tD2Zxde2c7jDAclOL0mLLdRLMqSAtX/87BTLpfKuFCpzrowSy2EYGOd+2kjb+G6WNHdkZtwk5zNVfDhAJNCYxFS8PxlxjP/aBq0SO947prg/6lWd5zW23612JYavfQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=SMYXNOOE; arc=none smtp.client-ip=209.85.214.176 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="SMYXNOOE" Received: by mail-pl1-f176.google.com with SMTP id d9443c01a7336-2165cb60719so116242075ad.0 for ; Mon, 30 Dec 2024 01:10:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1735549832; x=1736154632; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=0nJ0bdEaVLB0+13gaCQHdoBfFkuXXT3Pqta7GOjv8Bo=; b=SMYXNOOERJ1ARyKG8MiDTgatbG+eULYxRlgQScb+bpKUq3Zq2boEUN08L5OOeHUkaY XEEVRHousYEFM98XYCW7AIpbW3FQjJ6fD5YBJBBBmw+oa+jjQGlMz9L4pxpTjhtDHOc+ mzKJziqgTQCMgBTHzj2YA8F1aWpYo9nvXIarhYF86JwCQeoxb+lUkw+oa2X6RSW4W/Ky 6WVpzuT/Z/O5vWT2bjKx46iOw4QX9zLIN7W2nm16mDkfWzkeDn+RflIkDsdGtntb1nSB sRfvEio/iYoSPfgmK0uOUW6TRrFBXYwATq/kmJaAjmKmADifjbYyxMmfkSU2CA+OWdvc +c9w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1735549832; x=1736154632; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=0nJ0bdEaVLB0+13gaCQHdoBfFkuXXT3Pqta7GOjv8Bo=; b=bmdFV+vVdJ5fBnlWixEq3b+5sU6V+ue7ljhe/y9E4HS27KT0DMA1g30d90pnT2u4FM QXk69XFDkOx2RdbfNYC7Vw6Hoj+DWpRbph/iHLUUYdw8DKW1tzCY9N60oRpOgWTJTSOe C7ithbiXkyumJwiAdqDpM1R2dNkeeqKt6Z20tSkk5UMfwwmbxL2Wfv0QFBuj29NWzhRj zvgsKMIJ21A6JKBzz3RsXWIfn/mcKSiAwCZncx7W60GU4IKLmvpiNkwsEVqZgDmYS5eU KmOZQHVpOh1afNNmilcFR/8e0RbFmzJ8pLvL7/LGEbmJKFqAFlYRHhSH8zDa1A7uMUFc mcKQ== X-Forwarded-Encrypted: i=1; AJvYcCWcQBXjVyFOVMN9n3aoisIgb1bjFm7u6KOh9+Y9uLma8Y2Fa4cJ7JKWu/Eupoh4TomQ97N1jcoEETE/79g=@vger.kernel.org X-Gm-Message-State: AOJu0YxqJxvrVVwNmwPP7BTrQVa70a+XICYLnLYA1wFQgiPrY9bv1mo4 MNfJV7dOXbCTURkqiGijj0R7vSaPHDXCieyYWzrZoLsa6pwG1Tr8P6NMwBol0EM= X-Gm-Gg: ASbGnctTajgqZ5Ce+aB98CpKbjzdsa2zoWI+7My52v0zxLPzKGO1HWN2EBTIjbe+O6Y KFm1r9PhCMZVCi1ZZRRNiroq0ZtQMP1nqJkikAqKbk+7UoiBWSVE71wRy5GK5TU6N5pMaZ1yPa6 gCU5r/I22p2sw0ZbS8h97Cy+KhRkWgmnaE8ErKFjBxmmWHtey80yTxX47Zf89hPU1PCl2tGe531 hAJYgU1J2r5EZsegRqvEvdW2fW5XaL+GpPSmZi5JjzG3GZg/lIRd4EKfZj3uMxxiMsiwth1a+XA cBXMnVcOQPQH/2yzlrBYaA== X-Google-Smtp-Source: AGHT+IGwnjGmNRtsc6wvIZAIM8FV7PRxiURIME1KaVRQ0EplG8wu3sXxm46oGULh37KJjV0x+YOVDQ== X-Received: by 2002:a05:6a00:2181:b0:725:8c0f:6fa3 with SMTP id d2e1a72fcca58-72abdebb85dmr42631070b3a.22.1735549832564; Mon, 30 Dec 2024 01:10:32 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-842aba72f7csm17057841a12.4.2024.12.30.01.10.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 30 Dec 2024 01:10:32 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, agordeev@linux.ibm.com, kevin.brodsky@arm.com, palmer@dabbelt.com, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com, arnd@arndb.de, will@kernel.org, aneesh.kumar@kernel.org, npiggin@gmail.com, dave.hansen@linux.intel.com, rppt@kernel.org, ryan.roberts@arm.com Cc: linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, linux-arch@vger.kernel.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, loongarch@lists.linux.dev, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-sh@vger.kernel.org, linux-um@lists.infradead.org, Qi Zheng Subject: [PATCH v4 09/15] arm64: pgtable: move pagetable_dtor() to __tlb_remove_table() Date: Mon, 30 Dec 2024 17:07:44 +0800 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Move pagetable_dtor() to __tlb_remove_table(), so that ptlock and page table pages can be freed together (regardless of whether RCU is used). This prevents the use-after-free problem where the ptlock is freed immediately but the page table pages is freed later via RCU. Page tables shouldn't have swap cache, so use pagetable_free() instead of free_page_and_swap_cache() to free page table pages. Signed-off-by: Qi Zheng Suggested-by: Peter Zijlstra (Intel) Cc: linux-arm-kernel@lists.infradead.org Reviewed-by: Kevin Brodsky --- arch/arm64/include/asm/tlb.h | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-) diff --git a/arch/arm64/include/asm/tlb.h b/arch/arm64/include/asm/tlb.h index 408d0f36a8a8f..93591a80b5bfb 100644 --- a/arch/arm64/include/asm/tlb.h +++ b/arch/arm64/include/asm/tlb.h @@ -9,11 +9,13 @@ #define __ASM_TLB_H =20 #include -#include =20 static inline void __tlb_remove_table(void *_table) { - free_page_and_swap_cache((struct page *)_table); + struct ptdesc *ptdesc =3D (struct ptdesc *)_table; + + pagetable_dtor(ptdesc); + pagetable_free(ptdesc); } =20 #define tlb_flush tlb_flush @@ -82,7 +84,6 @@ static inline void __pte_free_tlb(struct mmu_gather *tlb,= pgtable_t pte, { struct ptdesc *ptdesc =3D page_ptdesc(pte); =20 - pagetable_dtor(ptdesc); tlb_remove_ptdesc(tlb, ptdesc); } =20 @@ -92,7 +93,6 @@ static inline void __pmd_free_tlb(struct mmu_gather *tlb,= pmd_t *pmdp, { struct ptdesc *ptdesc =3D virt_to_ptdesc(pmdp); =20 - pagetable_dtor(ptdesc); tlb_remove_ptdesc(tlb, ptdesc); } #endif @@ -106,7 +106,6 @@ static inline void __pud_free_tlb(struct mmu_gather *tl= b, pud_t *pudp, if (!pgtable_l4_enabled()) return; =20 - pagetable_dtor(ptdesc); tlb_remove_ptdesc(tlb, ptdesc); } #endif @@ -120,7 +119,6 @@ static inline void __p4d_free_tlb(struct mmu_gather *tl= b, p4d_t *p4dp, if (!pgtable_l5_enabled()) return; =20 - pagetable_dtor(ptdesc); tlb_remove_ptdesc(tlb, ptdesc); } #endif --=20 2.20.1 From nobody Sun May 10 06:27:04 2026 Received: from mail-pl1-f175.google.com (mail-pl1-f175.google.com [209.85.214.175]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 847521A2872 for ; Mon, 30 Dec 2024 09:10:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.175 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1735549848; cv=none; b=k0akf1g2W/s2lUkqzJZ+t9C1X+ItmPkQLJPqLPOFK0mO5A7/9g7cJbcnXB5T0AL97/2T4riQPe03m4GooyqUZlAGi5Qdq5exO7a7CQQzvFU1pnwrlKf3NNWLC1dm7MAsIV736XY1H0K3UM9Q2SpuPaxIoo6wy3I9w36+EIqDfXk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1735549848; c=relaxed/simple; bh=d8h9y7EmWepeeSdJqmfwFJOw7y0NPJCnETYa2k/Hjdk=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=TgUJgx0tpgirrqUix66dOXITn+E1DubDjV9YDL+DmlJbgtGItL7AzuaB/rwfxBU/0DYw+a+CPoyFGcW882FMHQ6TlPJ38E7Un0GRkDpKfsQIBVnjvv42aFbt0vo0bodVm4AbFqWlvTS/pDZwmu6YLQNeIJ1D5lSAuMN/TgAN700= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=E0YV3z1o; arc=none smtp.client-ip=209.85.214.175 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="E0YV3z1o" Received: by mail-pl1-f175.google.com with SMTP id d9443c01a7336-21644aca3a0so17748755ad.3 for ; Mon, 30 Dec 2024 01:10:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1735549845; x=1736154645; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=76YtbImY0jPwdXnQQVs9oYKl35yZZ81uc4A2YpilpOk=; b=E0YV3z1o6J88dE+BZfcXArcljLAUcY1glPwY/nABe6LJjkR4TYX34yIv7UCBMoHL/v 56Kx1oTLY1MOnZZTPkZU6gpbhhMZ//gdkYO0oS8E5KQTELKjm6nhkSUS7HUOYf4P615j fIe1YOqDVFnQ06oRVjT/A0AZ9vh6ZYcn9bpxcPRg/OyBQUM7glJiutzJnlXrBZRpdbji jQpGD35lWzcMr4V1rUHTuYfnIvzZaGIdwwHy9AQs+b31VM73IKNIOgjuvjuywhS2Tl+v iWdRFp/QS3MYhcfai3DREA901AzzowHG/liMw44dfKSxLGFJixByBePS3gntApnYlbEP uTBA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1735549845; x=1736154645; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=76YtbImY0jPwdXnQQVs9oYKl35yZZ81uc4A2YpilpOk=; b=JPvzKmw4csl3nSpcO7s7+FJVqud2JsxGLOF49xeTQsud5ZEpIark9QXYqqsu+WScZq FglGkI+KifsmOfHrumj+Mby9hAOTJZqTzDiCLlfr8poeWO/aJY38zMl7aUFbQemErw46 2xy5suMGdCclJaoaWKcaWBhNTpvuPd4+DA/fLb8hE6CZcTtnNIlalxbutLdBR6/t3W0S qaRUGCTJg+Ua5KPqiQA5QuIRG3dmHod7v//l4UObrZ4KQEP6lpfofkJ+WCDagQn8O27W coXj9v2yQUbviD8z/+2qqTSRqPcyrUV1SXrWcbXcq2/N3puUiXUd8y8tOv1Un+zzWYaO pF8g== X-Forwarded-Encrypted: i=1; AJvYcCWIB+d82wg+kp7fcaPwFhXFL8w9GsF30aU+soXQ1ELCplCkWRn6PlKtgYDOre6PLyRN4c15G23heToblSg=@vger.kernel.org X-Gm-Message-State: AOJu0YxoBimIZywJlDs+zsk3FX9c7xEc/MR6IoTSoqWa5jbYYXYLU+5R 44lLBKuY8/sKisiSABrIXlu2CHb4n6logycJ8fteRWgkJrAMiZ0VPtTJlGJnwVc= X-Gm-Gg: ASbGncsaHNs1tP0Pf0YaSiRZavG5qoGpS7vJcy9R7G07Xv1RV4zINO6qEjib2xT1uph UQjeOsaVATYafIHfTUHEqGNGRljbQQTwXWNDXatd7MEPFOJN4umVlKrnh4LzUKPVymBdkPaX6ZO vcYBhPo2JpJrbgRZRlsUCLKi7+GcdhO79r5lYv6uQ5oDHpPw6CnMnpWXEUP3yGX6dbV4DXtWfom KuJpy0CPfHFD3JSGLxafk00g8BOkyNHNO9UOXYXBG1mzV2aVVc9l7JbF1cpmJbgcSiopQ5v7k54 sZsVVB0rHk0Gg9ThPKEHNg== X-Google-Smtp-Source: AGHT+IE9evGbD+yEfyO2G41S4amTNhC2vfWaq9UYV9WQJrC+a4l06PyKmdWx4KtX6MVeEx+cVngAAA== X-Received: by 2002:a05:6a20:c91b:b0:1e1:d22d:cf38 with SMTP id adf61e73a8af0-1e5e05b06bbmr51533885637.21.1735549844993; Mon, 30 Dec 2024 01:10:44 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-842aba72f7csm17057841a12.4.2024.12.30.01.10.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 30 Dec 2024 01:10:44 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, agordeev@linux.ibm.com, kevin.brodsky@arm.com, palmer@dabbelt.com, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com, arnd@arndb.de, will@kernel.org, aneesh.kumar@kernel.org, npiggin@gmail.com, dave.hansen@linux.intel.com, rppt@kernel.org, ryan.roberts@arm.com Cc: linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, linux-arch@vger.kernel.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, loongarch@lists.linux.dev, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-sh@vger.kernel.org, linux-um@lists.infradead.org, Qi Zheng Subject: [PATCH v4 10/15] riscv: pgtable: move pagetable_dtor() to __tlb_remove_table() Date: Mon, 30 Dec 2024 17:07:45 +0800 Message-Id: <0e8f0b3835c15e99145e0006ac1020ae45a2b166.1735549103.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Move pagetable_dtor() to __tlb_remove_table(), so that ptlock and page table pages can be freed together (regardless of whether RCU is used). This prevents the use-after-free problem where the ptlock is freed immediately but the page table pages is freed later via RCU. Page tables shouldn't have swap cache, so use pagetable_free() instead of free_page_and_swap_cache() to free page table pages. By the way, move the comment above __tlb_remove_table() to riscv_tlb_remove_ptdesc(), it will be more appropriate. Signed-off-by: Qi Zheng Suggested-by: Peter Zijlstra (Intel) Cc: linux-riscv@lists.infradead.org Reviewed-by: Kevin Brodsky --- arch/riscv/include/asm/pgalloc.h | 38 ++++++++++++++------------------ arch/riscv/include/asm/tlb.h | 14 ++++-------- 2 files changed, 21 insertions(+), 31 deletions(-) diff --git a/arch/riscv/include/asm/pgalloc.h b/arch/riscv/include/asm/pgal= loc.h index b6793c5c99296..c8907b8317115 100644 --- a/arch/riscv/include/asm/pgalloc.h +++ b/arch/riscv/include/asm/pgalloc.h @@ -15,12 +15,22 @@ #define __HAVE_ARCH_PUD_FREE #include =20 +/* + * While riscv platforms with riscv_ipi_for_rfence as true require an IPI = to + * perform TLB shootdown, some platforms with riscv_ipi_for_rfence as fals= e use + * SBI to perform TLB shootdown. To keep software pagetable walkers safe i= n this + * case we switch to RCU based table free (MMU_GATHER_RCU_TABLE_FREE). See= the + * comment below 'ifdef CONFIG_MMU_GATHER_RCU_TABLE_FREE' in include/asm-g= eneric/tlb.h + * for more details. + */ static inline void riscv_tlb_remove_ptdesc(struct mmu_gather *tlb, void *p= t) { - if (riscv_use_sbi_for_rfence()) + if (riscv_use_sbi_for_rfence()) { tlb_remove_ptdesc(tlb, pt); - else + } else { + pagetable_dtor(pt); tlb_remove_page_ptdesc(tlb, pt); + } } =20 static inline void pmd_populate_kernel(struct mm_struct *mm, @@ -97,23 +107,15 @@ static inline void pud_free(struct mm_struct *mm, pud_= t *pud) static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pud, unsigned long addr) { - if (pgtable_l4_enabled) { - struct ptdesc *ptdesc =3D virt_to_ptdesc(pud); - - pagetable_dtor(ptdesc); - riscv_tlb_remove_ptdesc(tlb, ptdesc); - } + if (pgtable_l4_enabled) + riscv_tlb_remove_ptdesc(tlb, virt_to_ptdesc(pud)); } =20 static inline void __p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d, unsigned long addr) { - if (pgtable_l5_enabled) { - struct ptdesc *ptdesc =3D virt_to_ptdesc(p4d); - - pagetable_dtor(ptdesc); + if (pgtable_l5_enabled) riscv_tlb_remove_ptdesc(tlb, virt_to_ptdesc(p4d)); - } } #endif /* __PAGETABLE_PMD_FOLDED */ =20 @@ -142,10 +144,7 @@ static inline pgd_t *pgd_alloc(struct mm_struct *mm) static inline void __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd, unsigned long addr) { - struct ptdesc *ptdesc =3D virt_to_ptdesc(pmd); - - pagetable_dtor(ptdesc); - riscv_tlb_remove_ptdesc(tlb, ptdesc); + riscv_tlb_remove_ptdesc(tlb, virt_to_ptdesc(pmd)); } =20 #endif /* __PAGETABLE_PMD_FOLDED */ @@ -153,10 +152,7 @@ static inline void __pmd_free_tlb(struct mmu_gather *t= lb, pmd_t *pmd, static inline void __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pte, unsigned long addr) { - struct ptdesc *ptdesc =3D page_ptdesc(pte); - - pagetable_dtor(ptdesc); - riscv_tlb_remove_ptdesc(tlb, ptdesc); + riscv_tlb_remove_ptdesc(tlb, page_ptdesc(pte)); } #endif /* CONFIG_MMU */ =20 diff --git a/arch/riscv/include/asm/tlb.h b/arch/riscv/include/asm/tlb.h index 1f6c38420d8e0..ded8724b3c4f7 100644 --- a/arch/riscv/include/asm/tlb.h +++ b/arch/riscv/include/asm/tlb.h @@ -11,19 +11,13 @@ struct mmu_gather; static void tlb_flush(struct mmu_gather *tlb); =20 #ifdef CONFIG_MMU -#include =20 -/* - * While riscv platforms with riscv_ipi_for_rfence as true require an IPI = to - * perform TLB shootdown, some platforms with riscv_ipi_for_rfence as fals= e use - * SBI to perform TLB shootdown. To keep software pagetable walkers safe i= n this - * case we switch to RCU based table free (MMU_GATHER_RCU_TABLE_FREE). See= the - * comment below 'ifdef CONFIG_MMU_GATHER_RCU_TABLE_FREE' in include/asm-g= eneric/tlb.h - * for more details. - */ static inline void __tlb_remove_table(void *table) { - free_page_and_swap_cache(table); + struct ptdesc *ptdesc =3D (struct ptdesc *)table; + + pagetable_dtor(ptdesc); + pagetable_free(ptdesc); } =20 #endif /* CONFIG_MMU */ --=20 2.20.1 From nobody Sun May 10 06:27:04 2026 Received: from mail-pl1-f170.google.com (mail-pl1-f170.google.com [209.85.214.170]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 446251A4E9E for ; Mon, 30 Dec 2024 09:10:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.170 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1735549860; cv=none; b=T8Ut+GQMOUPIu9EPXIklJhnMJsAHKO8JcEbhcrJemDTMRo1w89ylGx5iK0rgg8mW/sGOfxUwYSZNGNZC49/bmv6LECa8PiPUoAD7juT7Vv8h+rOcG00SgsIhd5GVHfzzBb5RILIaOKprV0EfHiCePiA/DWai2XSwpYpMWl16kmE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1735549860; c=relaxed/simple; bh=0DrkqJj4wQeX5ghzMY0LbiVqPzUVuk7cM17qWFCGuf4=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=Ew08IYURryMsvAFCDVSej6icPw6/ub62l7PI1bZ452Y/8oBszKQ/mdAVrFOT+OdG12cOy0/FXcM7RHjAjUolOtLtmUSIBbb1P6hnlLGb/fwHhDFUxD2n92Cz+oVH7NQDNig3wzdCNlVruCcowiURnN516mM8M7+ALEzwChreBis= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=lYkoNd7F; arc=none smtp.client-ip=209.85.214.170 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="lYkoNd7F" Received: by mail-pl1-f170.google.com with SMTP id d9443c01a7336-2166f1e589cso140226635ad.3 for ; Mon, 30 Dec 2024 01:10:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1735549857; x=1736154657; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Rs6Lkouarv58ZoKb+WtHLVo9G63kikGgJGrD7ursbxc=; b=lYkoNd7FgWoHDL3Jfvlgra0o5dn+A9l/ySi9Em97ziD4wMmPDTy14BOU7dEDhA2LI1 6Ez6UWvnADqjswxV260ANzGE5+96l/uqHQ2ZCCsYvBIxorq2VsjW4e/5wzlUwtrEyMjg YMnp0VnzQES6RIPoIOJBB2GfilVAHfcptwA5gkTCHU5miK/0rZhKNeEbmapgvPizZ0gJ 9HNk75B/dcJ/Z8IScpL479Y/GSkhFUOz6eRxLAy/wBwIPAD7sLsepuh8fGdcpT/Ll0mU zu+K+EmaVnI7eYFdkAkRnzxvInR8DvApwQVDriwdoygPNx3kigYYMbj4WyAwaeRCpsF+ Q0TQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1735549857; x=1736154657; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Rs6Lkouarv58ZoKb+WtHLVo9G63kikGgJGrD7ursbxc=; b=C8yhSZ/UrrUtx1+Uv4tjQPIfhgD77Wivt0ryA3UebCHgKfmgozgBOzDugm2O+Th0ES M0doIDw5+VPhscY0rP1zi0WtZrOWkgXG2dNgBj8ZIyLZw7mpg512CgBIQu31jSBaSE3P f2YTcBqqK6FdfGiAMBFaflZmGgccrb6IksGBne6HJJ7ze7c7ybApDkWA/Q86K4qLIi8f KERwUlIq0BBNeQrnRrUEIl/SMjpP+DZdnT5JdSg0IkV8l8akNTVBVZ6mswAtyCEEGzkk wOYPwi0brl5fZOzoaa7009m7AWdb0p7vMvW8B3rFWg+dcUpPaEytagm7rhuwldrNlfFX KMtQ== X-Forwarded-Encrypted: i=1; AJvYcCX3KVNUoZjLD5TBVAtt3oblRrcRA/DZLnW/TS7vPdlP10VgFTNAey4u239iOC/S9d75VSwNuyNqbIid6V4=@vger.kernel.org X-Gm-Message-State: AOJu0YyBgFL+qsBWhFE9GY9FBReDOkiT/NnukOFs1fdhf9J24+b88l/n H1LSndZ7xwt8nIZfBvztL7gHrRspR1I8/0dSW+uGrKKLmVDzUCgAfYHsCOF/a9I= X-Gm-Gg: ASbGncuqFqxoCNDUDDKspGm/K6c5YCdfwty0yipyBiN0mxLvvoHo1NncXSmjsUAXWAW ytkpJcU+ao6+ZunqKYrPNX4gOGoEXKzRVWa3AwAhZ+mEkCGlocoar4vxcN6gNR9bIrXrNxyB5O5 3ZVPKJaGoPUYRze5afIoD9Ealv2fY0qQZwipoxzaBMEWCH0Eo9sQmryLEN/8lF6FxNbCbIf571i m2OmlEDgofrvPtUJa23lKlzg0s1AZePu/p2UnPMw/Fw/a0FK/rr5ZZL1djC2tpR9LAuDDOSSuXm JqllYJmHxcwQYHEGIKx7EA== X-Google-Smtp-Source: AGHT+IGZk471dqtvPjYzzhjuNb2uivM1pgmRrcDve4hlPbWGQce+RD+FtIOnCoiYNP05sabQ4PQAQA== X-Received: by 2002:a05:6a21:8cc2:b0:1e1:f281:8d36 with SMTP id adf61e73a8af0-1e5e0458dbdmr51544115637.10.1735549857384; Mon, 30 Dec 2024 01:10:57 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-842aba72f7csm17057841a12.4.2024.12.30.01.10.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 30 Dec 2024 01:10:56 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, agordeev@linux.ibm.com, kevin.brodsky@arm.com, palmer@dabbelt.com, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com, arnd@arndb.de, will@kernel.org, aneesh.kumar@kernel.org, npiggin@gmail.com, dave.hansen@linux.intel.com, rppt@kernel.org, ryan.roberts@arm.com Cc: linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, linux-arch@vger.kernel.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, loongarch@lists.linux.dev, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-sh@vger.kernel.org, linux-um@lists.infradead.org, Qi Zheng Subject: [PATCH v4 11/15] x86: pgtable: move pagetable_dtor() to __tlb_remove_table() Date: Mon, 30 Dec 2024 17:07:46 +0800 Message-Id: <0dc5a3bf5a692e24379c1d3b879a6d4396f0dbbd.1735549103.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Move pagetable_dtor() to __tlb_remove_table(), so that ptlock and page table pages can be freed together (regardless of whether RCU is used). This prevents the use-after-free problem where the ptlock is freed immediately but the page table pages is freed later via RCU. Page tables shouldn't have swap cache, so use pagetable_free() instead of free_page_and_swap_cache() to free page table pages. Signed-off-by: Qi Zheng Suggested-by: Peter Zijlstra (Intel) Cc: x86@kernel.org Reviewed-by: Kevin Brodsky --- arch/x86/include/asm/tlb.h | 17 ++++++++++------- arch/x86/kernel/paravirt.c | 1 + arch/x86/mm/pgtable.c | 12 ++---------- 3 files changed, 13 insertions(+), 17 deletions(-) diff --git a/arch/x86/include/asm/tlb.h b/arch/x86/include/asm/tlb.h index 73f0786181cc9..f64730be5ad67 100644 --- a/arch/x86/include/asm/tlb.h +++ b/arch/x86/include/asm/tlb.h @@ -31,24 +31,27 @@ static inline void tlb_flush(struct mmu_gather *tlb) */ static inline void __tlb_remove_table(void *table) { - free_page_and_swap_cache(table); + struct ptdesc *ptdesc =3D (struct ptdesc *)table; + + pagetable_dtor(ptdesc); + pagetable_free(ptdesc); } =20 #ifdef CONFIG_PT_RECLAIM static inline void __tlb_remove_table_one_rcu(struct rcu_head *head) { - struct page *page; + struct ptdesc *ptdesc; =20 - page =3D container_of(head, struct page, rcu_head); - put_page(page); + ptdesc =3D container_of(head, struct ptdesc, pt_rcu_head); + __tlb_remove_table(ptdesc); } =20 static inline void __tlb_remove_table_one(void *table) { - struct page *page; + struct ptdesc *ptdesc; =20 - page =3D table; - call_rcu(&page->rcu_head, __tlb_remove_table_one_rcu); + ptdesc =3D table; + call_rcu(&ptdesc->pt_rcu_head, __tlb_remove_table_one_rcu); } #define __tlb_remove_table_one __tlb_remove_table_one #endif /* CONFIG_PT_RECLAIM */ diff --git a/arch/x86/kernel/paravirt.c b/arch/x86/kernel/paravirt.c index 7bdcf152778c0..46d5d325483b0 100644 --- a/arch/x86/kernel/paravirt.c +++ b/arch/x86/kernel/paravirt.c @@ -62,6 +62,7 @@ void __init native_pv_lock_init(void) #ifndef CONFIG_PT_RECLAIM static void native_tlb_remove_table(struct mmu_gather *tlb, void *table) { + pagetable_dtor(table); tlb_remove_page(tlb, table); } #else diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c index a6cd9660e29ec..a0b0e501ba663 100644 --- a/arch/x86/mm/pgtable.c +++ b/arch/x86/mm/pgtable.c @@ -23,6 +23,7 @@ EXPORT_SYMBOL(physical_mask); static inline void paravirt_tlb_remove_table(struct mmu_gather *tlb, void *table) { + pagetable_dtor(table); tlb_remove_page(tlb, table); } #else @@ -60,7 +61,6 @@ early_param("userpte", setup_userpte); =20 void ___pte_free_tlb(struct mmu_gather *tlb, struct page *pte) { - pagetable_dtor(page_ptdesc(pte)); paravirt_release_pte(page_to_pfn(pte)); paravirt_tlb_remove_table(tlb, pte); } @@ -68,7 +68,6 @@ void ___pte_free_tlb(struct mmu_gather *tlb, struct page = *pte) #if CONFIG_PGTABLE_LEVELS > 2 void ___pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd) { - struct ptdesc *ptdesc =3D virt_to_ptdesc(pmd); paravirt_release_pmd(__pa(pmd) >> PAGE_SHIFT); /* * NOTE! For PAE, any changes to the top page-directory-pointer-table @@ -77,16 +76,12 @@ void ___pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd) #ifdef CONFIG_X86_PAE tlb->need_flush_all =3D 1; #endif - pagetable_dtor(ptdesc); - paravirt_tlb_remove_table(tlb, ptdesc_page(ptdesc)); + paravirt_tlb_remove_table(tlb, virt_to_page(pmd)); } =20 #if CONFIG_PGTABLE_LEVELS > 3 void ___pud_free_tlb(struct mmu_gather *tlb, pud_t *pud) { - struct ptdesc *ptdesc =3D virt_to_ptdesc(pud); - - pagetable_dtor(ptdesc); paravirt_release_pud(__pa(pud) >> PAGE_SHIFT); paravirt_tlb_remove_table(tlb, virt_to_page(pud)); } @@ -94,9 +89,6 @@ void ___pud_free_tlb(struct mmu_gather *tlb, pud_t *pud) #if CONFIG_PGTABLE_LEVELS > 4 void ___p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d) { - struct ptdesc *ptdesc =3D virt_to_ptdesc(p4d); - - pagetable_dtor(ptdesc); paravirt_release_p4d(__pa(p4d) >> PAGE_SHIFT); paravirt_tlb_remove_table(tlb, virt_to_page(p4d)); } --=20 2.20.1 From nobody Sun May 10 06:27:04 2026 Received: from mail-pl1-f169.google.com (mail-pl1-f169.google.com [209.85.214.169]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EB7F41A76AE for ; Mon, 30 Dec 2024 09:11:10 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.169 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1735549873; cv=none; b=J2fupmrYWFpt3vqHP3Spt/W7X3l0/Orw3k0WjjgJRuyEIWe2domx0FlUMS7NTMVrfsZp9IlJ08UPeuXlnlhYS7Bx9JGD1zmfmqck1sM+VWQQLCoeoFuRNVbvDKIDtujuUWloQmkPfU1OOMCQy5nAHlsTATbwO4mgtYFpa3iP4c4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1735549873; c=relaxed/simple; bh=ySnIl2i3Fo/t1YN/rcwyhGqxWwlx1TNUJRz33ZWxlkE=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=KUV+XNVw1IL81IhnrtiemZe7c/f5qCdF+EsPq29fHYcexPNki0XyIowFMWsv3Nfrae+8mORWCI2oA15NUW+KB0vVnWFhnQpfxTr6/D3V1hZKgMYkiQfXaMca2TvOPWHzwCQ0/r36EYEl3fTpBia0gcC/+miylwW0LMK7C07S02I= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=CbvM39v8; arc=none smtp.client-ip=209.85.214.169 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="CbvM39v8" Received: by mail-pl1-f169.google.com with SMTP id d9443c01a7336-21675fd60feso153837645ad.2 for ; Mon, 30 Dec 2024 01:11:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1735549870; x=1736154670; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=jyEzElX2/Cl2Hb0vACePBo9dDp7jwROhL3d0Mq9f7ig=; b=CbvM39v8DSxX3gTbaGPkNFXL2gTwNFbnSvW9hMdWzoZdSAtW4hbM7cGR3bLAME+DTg eLqPZxIF/CeaTOx8hjA0ITXh4+xGxzfJzxuRHVzPc5Ap2qf00l4doFQn5sAjWsiVzYuu gxk5PCTQ/K8GFKmxsi4rbVYXXE/QXdE1bbaZzuxLrh/Tgo1c3W5H0qNOca4nTF8TS4t5 LkCaPMhQtv3BdNeDF9l6NsVnG/1Gz7Bcky/2IBBDRGbanH5Hqdn2K2xkBn44LLuhyTlV Z5d+lDOSE2Oq+ggPmtp7yE6VfDvavDoBoJfe7XHWBEWbvnm0PspH8WikEnP7tCpj5j0V 6kng== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1735549870; x=1736154670; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=jyEzElX2/Cl2Hb0vACePBo9dDp7jwROhL3d0Mq9f7ig=; b=B7q8V07j8IAwC0Jh3f927fSjhm0rMpCTdUGtItRlsOdQ3vK9dVLA5Hgis9SCik4nCn 3mdS5P/3C3FNfvsqV364ka4h1ewa33XixRC7SIciYzGIQE/Nlq+cPFp4iIC8K/V5Wehq wwZsIYJutVk200UIiIbWO0Eu2wqPnMpkLYbmzH4lb6oKx3kNP9iHyWyM2VU/5+50NeTl MRFuyS2UIati4xDJQatIrf+ykEY75wBSY5AG16dNFPxcHp0vB3drelZvi2EGDUrO1Rh5 /tUeW0etnxGMPxSW61EUpcglyG/mqbpPhhNWiSb+xpfaBMSJDz0KJIPPV46DtcSAgnod rq7Q== X-Forwarded-Encrypted: i=1; AJvYcCXHj9I95LaB3tc9MBIwU/YkfXeUZlsULXOZV3T7pB+84poW2yn6jBxTqh64KmENUKQim+AZq9/16DNnrWY=@vger.kernel.org X-Gm-Message-State: AOJu0YzQM1VekIdHv3dsR9FoBgjwH8TsZNCUH8qeR9G3W3NkAuRA8LY4 7i+TkapR9D2Mnx8JfSPz93ulkG+8tulIsBiZaDx86bs7lP4IPuuFj8eYFJyqn+E= X-Gm-Gg: ASbGncupKmPeC/n02mFdfsM9ErxepZt9xnXQxGmpPbYqyptjHl/T1YtUeaXvBW/9KlM exJDddCP/S15pc3chJ8jfnjdEDJAGbmIlMBzfP/jRG2j62RhDyLvcrd/tIpCG8gfPDtgO/cgqa7 sv7HmVrAaaPUrdH4u4vTvVJibjTVJyKFOPy9FIFq5JC695t26R6fUyrKRaBhsiYIvfi9uZ3z/zO pwBrxvAVvwW9hwCON//r/aWMXySx1Q+0m7u/rMyp+1TtoBYN+hWk+Z3/LK5wQEE3a4EAVGXpxxG UAWWPfMRcQXq8o8CeS++GA== X-Google-Smtp-Source: AGHT+IHPMMlkiFtH678yJhisFXpRpeLbEmYMWQPlXwvGfvt1NUv1TR7XGzXk5FcrgJpzIcDaQ0LX6w== X-Received: by 2002:a05:6a20:7351:b0:1e0:cadd:f670 with SMTP id adf61e73a8af0-1e5e044c8ebmr53183808637.5.1735549869780; Mon, 30 Dec 2024 01:11:09 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-842aba72f7csm17057841a12.4.2024.12.30.01.10.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 30 Dec 2024 01:11:09 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, agordeev@linux.ibm.com, kevin.brodsky@arm.com, palmer@dabbelt.com, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com, arnd@arndb.de, will@kernel.org, aneesh.kumar@kernel.org, npiggin@gmail.com, dave.hansen@linux.intel.com, rppt@kernel.org, ryan.roberts@arm.com Cc: linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, linux-arch@vger.kernel.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, loongarch@lists.linux.dev, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-sh@vger.kernel.org, linux-um@lists.infradead.org, Qi Zheng Subject: [PATCH v4 12/15] s390: pgtable: also move pagetable_dtor() of PxD to __tlb_remove_table() Date: Mon, 30 Dec 2024 17:07:47 +0800 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" To unify the PxD and PTE TLB free path, also move the pagetable_dtor() of PMD|PUD|P4D to __tlb_remove_table(). Signed-off-by: Qi Zheng Suggested-by: Peter Zijlstra (Intel) Cc: linux-s390@vger.kernel.org Reviewed-by: Kevin Brodsky --- arch/s390/include/asm/tlb.h | 3 --- arch/s390/mm/pgalloc.c | 14 ++++---------- 2 files changed, 4 insertions(+), 13 deletions(-) diff --git a/arch/s390/include/asm/tlb.h b/arch/s390/include/asm/tlb.h index 74b6fba4c2ee3..79df7c0932c56 100644 --- a/arch/s390/include/asm/tlb.h +++ b/arch/s390/include/asm/tlb.h @@ -102,7 +102,6 @@ static inline void pmd_free_tlb(struct mmu_gather *tlb,= pmd_t *pmd, { if (mm_pmd_folded(tlb->mm)) return; - pagetable_dtor(virt_to_ptdesc(pmd)); __tlb_adjust_range(tlb, address, PAGE_SIZE); tlb->mm->context.flush_mm =3D 1; tlb->freed_tables =3D 1; @@ -122,7 +121,6 @@ static inline void pud_free_tlb(struct mmu_gather *tlb,= pud_t *pud, { if (mm_pud_folded(tlb->mm)) return; - pagetable_dtor(virt_to_ptdesc(pud)); tlb->mm->context.flush_mm =3D 1; tlb->freed_tables =3D 1; tlb->cleared_p4ds =3D 1; @@ -141,7 +139,6 @@ static inline void p4d_free_tlb(struct mmu_gather *tlb,= p4d_t *p4d, { if (mm_p4d_folded(tlb->mm)) return; - pagetable_dtor(virt_to_ptdesc(p4d)); __tlb_adjust_range(tlb, address, PAGE_SIZE); tlb->mm->context.flush_mm =3D 1; tlb->freed_tables =3D 1; diff --git a/arch/s390/mm/pgalloc.c b/arch/s390/mm/pgalloc.c index 569de24d33761..c73b89811a264 100644 --- a/arch/s390/mm/pgalloc.c +++ b/arch/s390/mm/pgalloc.c @@ -180,7 +180,7 @@ unsigned long *page_table_alloc(struct mm_struct *mm) return table; } =20 -static void pagetable_pte_dtor_free(struct ptdesc *ptdesc) +static void pagetable_dtor_free(struct ptdesc *ptdesc) { pagetable_dtor(ptdesc); pagetable_free(ptdesc); @@ -190,20 +190,14 @@ void page_table_free(struct mm_struct *mm, unsigned l= ong *table) { struct ptdesc *ptdesc =3D virt_to_ptdesc(table); =20 - pagetable_pte_dtor_free(ptdesc); + pagetable_dtor_free(ptdesc); } =20 void __tlb_remove_table(void *table) { struct ptdesc *ptdesc =3D virt_to_ptdesc(table); - struct page *page =3D ptdesc_page(ptdesc); =20 - if (compound_order(page) =3D=3D CRST_ALLOC_ORDER) { - /* pmd, pud, or p4d */ - pagetable_free(ptdesc); - return; - } - pagetable_pte_dtor_free(ptdesc); + pagetable_dtor_free(ptdesc); } =20 #ifdef CONFIG_TRANSPARENT_HUGEPAGE @@ -211,7 +205,7 @@ static void pte_free_now(struct rcu_head *head) { struct ptdesc *ptdesc =3D container_of(head, struct ptdesc, pt_rcu_head); =20 - pagetable_pte_dtor_free(ptdesc); + pagetable_dtor_free(ptdesc); } =20 void pte_free_defer(struct mm_struct *mm, pgtable_t pgtable) --=20 2.20.1 From nobody Sun May 10 06:27:04 2026 Received: from mail-pl1-f173.google.com (mail-pl1-f173.google.com [209.85.214.173]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EF1E51A9B2E for ; Mon, 30 Dec 2024 09:11:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.173 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1735549885; cv=none; b=urc5wlwiAlHfkwyATzW8slP90+nDqhgX98vn1bSwSw/z+LtWSgbJ41Azl5e7FwLFGy7H/cWt0Hcuzv31FtNWMB64rociQuE3qpTldUzkB2PIYhYwCbMnb9BxAeRVHBYq43yJQ4j0qX83Vn9C8wtzrxXYIQFHtfpgPXhTLc8tV3k= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1735549885; c=relaxed/simple; bh=EElIB1gpaaG9REY1IEzvLAKdR3Au6Zf5cYxlQP4u0WQ=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=TC6arfYT00zcirjJ5suF+RlaeMc2VJeN6xVMZ0oC3yiFDOa2dLDFaVBDFusRc+13gtJr3GdJwcy9LHQ/eUTMQaUvI9+dXjWxJdMiDxSTNrK4khDaRD/z4MxWIxXX54kKq+VCj6AF9GbIftTr3GSdwXy6Mqx62WHgv9Yb4Wt7PIg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=lOlH7Qbn; arc=none smtp.client-ip=209.85.214.173 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="lOlH7Qbn" Received: by mail-pl1-f173.google.com with SMTP id d9443c01a7336-218c8aca5f1so142100345ad.0 for ; Mon, 30 Dec 2024 01:11:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1735549882; x=1736154682; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=m/+0uIvp7xUeTeFwO63TH1y++vtASeDM7X35qnuktEM=; b=lOlH7QbnulqHPJxKnPcnJAimaOzW9lVegyhVigXYwqFVrY+r3iO2Ti4KIzlvNyLPFS Wxc0rvJqxfHTGptm/YggxR2zntj0C6fYziFFSzJ+PuExG4vo5iYlSniLVBTym5FlmzuO cMNWK2sMgu5BjgEkSYssqT6H1JaG9VBo4u8y0VXjQcvnMmko/6k0KDc20ndo3mheYmPF fzBC5uOZWMPvpoIt33sewRzqCWHMHjjVrWIDyJ4nRaFrdDarnAnLPminkC0oFb2dlBpY hE3Z0N8TEi8P0J4F3xzyj5tNSLj9tWJVzWQuWeQvryp5r7K+sAnjYJZ7WJnPGlnZA66O 8OeQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1735549882; x=1736154682; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=m/+0uIvp7xUeTeFwO63TH1y++vtASeDM7X35qnuktEM=; b=krFiNoMvXMDZBFurbndh9taPLa37fkaSAK1xgHRstxwV7X0ZcoOimJxHGE/Yh0scIb 7dGUjzr7qCPCRc55QYdoXnk7PFY+H44GpZMGXW9qgy5gcUSn1Z1gpDGb8nN7qpTztzly /7CbhZ+YugtbcMUX4i6pl/s04Ru7GxJGbVeowRB/7/l6pYSnQ3lhOSiqDrYbaVAJunbK gxf8DxaRroGkO8hVroHBnXezqCACPAeuyLe+gNZwJjxCmSbZb6izP/KK0voP6/vge8Lm UxdfzSLw3IvpKQFGfGcS7JeIlMTddiObfi7z6jEOqeosNnV0YzCVDUUPrVrJN+BP5vZS Ee4w== X-Forwarded-Encrypted: i=1; AJvYcCW0ZvF9IdNOPGoh/bwpZ1qxC1JVtON11n+FK/nN1feGo2/H1AAzfPHod/vloMvmn/A0MeibrAKDexK+6/4=@vger.kernel.org X-Gm-Message-State: AOJu0Yx+jZy54+k2QBL7kpqFHcYyLr3HCEU2h/5UJ5B5Zyga6rDrPxXP 6ZBcRSNQcDPzLXopk1uF6OS/Kq/+Yy76VS48M+z5lfXdXO8XMq3+evc9YNsKl6I= X-Gm-Gg: ASbGncssa1bGMthWY4djhb7MKfiPhtJAvhr363rzuRxFCt1NvKcjGAlachwbV115X7u xukRu9zvjIH3AB4oa/Cw0Xtsy9JadLN0NKlfD8/+AoDJN4StR1ppsdACIvOzNBe7gELkH2GiSxB donXCHPPXRphnDmmhD9n1qLc0wrjPuUf/V5pehB0xSKNs7l9DDHccXkgbZ6xkfWIcOU9gm3Suni Tlv9SaSzucMPCu03R/pNm0wyHspo7JqjEe3YTzZtMwLl/3JbEqtotAeuENWqEZtkgSRpO2x2PvJ nemPyQWrpyAK+w4p+Q544w== X-Google-Smtp-Source: AGHT+IH93LLg/oVZ0Av8lq2Z9OPVl8eXrLxQFXcb3ZPfLzaeZOV1YT5zFyKOR468gBW6qJHDbn+qAQ== X-Received: by 2002:a17:903:947:b0:216:48f4:4f3d with SMTP id d9443c01a7336-219e6e9e004mr571689445ad.13.1735549882158; Mon, 30 Dec 2024 01:11:22 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-842aba72f7csm17057841a12.4.2024.12.30.01.11.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 30 Dec 2024 01:11:21 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, agordeev@linux.ibm.com, kevin.brodsky@arm.com, palmer@dabbelt.com, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com, arnd@arndb.de, will@kernel.org, aneesh.kumar@kernel.org, npiggin@gmail.com, dave.hansen@linux.intel.com, rppt@kernel.org, ryan.roberts@arm.com Cc: linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, linux-arch@vger.kernel.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, loongarch@lists.linux.dev, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-sh@vger.kernel.org, linux-um@lists.infradead.org, Qi Zheng Subject: [PATCH v4 13/15] mm: pgtable: introduce generic __tlb_remove_table() Date: Mon, 30 Dec 2024 17:07:48 +0800 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Several architectures (arm, arm64, riscv and x86) define exactly the same __tlb_remove_table(), just introduce generic __tlb_remove_table() to eliminate these duplications. The s390 __tlb_remove_table() is nearly the same, so also make s390 __tlb_remove_table() version generic. Signed-off-by: Qi Zheng Acked-by: Alexander Gordeev Acked-by: Andreas Larsson # sparc Reviewed-by: Kevin Brodsky --- arch/arm/include/asm/tlb.h | 9 --------- arch/arm64/include/asm/tlb.h | 7 ------- arch/powerpc/include/asm/tlb.h | 1 + arch/riscv/include/asm/tlb.h | 12 ------------ arch/s390/include/asm/tlb.h | 9 ++++----- arch/s390/mm/pgalloc.c | 7 ------- arch/sparc/include/asm/tlb_32.h | 1 + arch/sparc/include/asm/tlb_64.h | 1 + arch/x86/include/asm/tlb.h | 17 ----------------- include/asm-generic/tlb.h | 15 +++++++++++++-- 10 files changed, 20 insertions(+), 59 deletions(-) diff --git a/arch/arm/include/asm/tlb.h b/arch/arm/include/asm/tlb.h index 264ab635e807a..ea4fbe7b17f6f 100644 --- a/arch/arm/include/asm/tlb.h +++ b/arch/arm/include/asm/tlb.h @@ -27,15 +27,6 @@ #else /* !CONFIG_MMU */ =20 #include - -static inline void __tlb_remove_table(void *_table) -{ - struct ptdesc *ptdesc =3D (struct ptdesc *)_table; - - pagetable_dtor(ptdesc); - pagetable_free(ptdesc); -} - #include =20 static inline void diff --git a/arch/arm64/include/asm/tlb.h b/arch/arm64/include/asm/tlb.h index 93591a80b5bfb..8d762607285cc 100644 --- a/arch/arm64/include/asm/tlb.h +++ b/arch/arm64/include/asm/tlb.h @@ -10,13 +10,6 @@ =20 #include =20 -static inline void __tlb_remove_table(void *_table) -{ - struct ptdesc *ptdesc =3D (struct ptdesc *)_table; - - pagetable_dtor(ptdesc); - pagetable_free(ptdesc); -} =20 #define tlb_flush tlb_flush static void tlb_flush(struct mmu_gather *tlb); diff --git a/arch/powerpc/include/asm/tlb.h b/arch/powerpc/include/asm/tlb.h index 1ca7d4c4b90db..2058e8d3e0138 100644 --- a/arch/powerpc/include/asm/tlb.h +++ b/arch/powerpc/include/asm/tlb.h @@ -37,6 +37,7 @@ extern void tlb_flush(struct mmu_gather *tlb); */ #define tlb_needs_table_invalidate() radix_enabled() =20 +#define __HAVE_ARCH_TLB_REMOVE_TABLE /* Get the generic bits... */ #include =20 diff --git a/arch/riscv/include/asm/tlb.h b/arch/riscv/include/asm/tlb.h index ded8724b3c4f7..50b63b5c15bd8 100644 --- a/arch/riscv/include/asm/tlb.h +++ b/arch/riscv/include/asm/tlb.h @@ -10,18 +10,6 @@ struct mmu_gather; =20 static void tlb_flush(struct mmu_gather *tlb); =20 -#ifdef CONFIG_MMU - -static inline void __tlb_remove_table(void *table) -{ - struct ptdesc *ptdesc =3D (struct ptdesc *)table; - - pagetable_dtor(ptdesc); - pagetable_free(ptdesc); -} - -#endif /* CONFIG_MMU */ - #define tlb_flush tlb_flush #include =20 diff --git a/arch/s390/include/asm/tlb.h b/arch/s390/include/asm/tlb.h index 79df7c0932c56..da4a7d175f69c 100644 --- a/arch/s390/include/asm/tlb.h +++ b/arch/s390/include/asm/tlb.h @@ -22,7 +22,6 @@ * Pages used for the page tables is a different story. FIXME: more */ =20 -void __tlb_remove_table(void *_table); static inline void tlb_flush(struct mmu_gather *tlb); static inline bool __tlb_remove_page_size(struct mmu_gather *tlb, struct page *page, bool delay_rmap, int page_size); @@ -87,7 +86,7 @@ static inline void pte_free_tlb(struct mmu_gather *tlb, p= gtable_t pte, tlb->cleared_pmds =3D 1; if (mm_alloc_pgste(tlb->mm)) gmap_unlink(tlb->mm, (unsigned long *)pte, address); - tlb_remove_ptdesc(tlb, pte); + tlb_remove_ptdesc(tlb, virt_to_ptdesc(pte)); } =20 /* @@ -106,7 +105,7 @@ static inline void pmd_free_tlb(struct mmu_gather *tlb,= pmd_t *pmd, tlb->mm->context.flush_mm =3D 1; tlb->freed_tables =3D 1; tlb->cleared_puds =3D 1; - tlb_remove_ptdesc(tlb, pmd); + tlb_remove_ptdesc(tlb, virt_to_ptdesc(pmd)); } =20 /* @@ -124,7 +123,7 @@ static inline void pud_free_tlb(struct mmu_gather *tlb,= pud_t *pud, tlb->mm->context.flush_mm =3D 1; tlb->freed_tables =3D 1; tlb->cleared_p4ds =3D 1; - tlb_remove_ptdesc(tlb, pud); + tlb_remove_ptdesc(tlb, virt_to_ptdesc(pud)); } =20 /* @@ -142,7 +141,7 @@ static inline void p4d_free_tlb(struct mmu_gather *tlb,= p4d_t *p4d, __tlb_adjust_range(tlb, address, PAGE_SIZE); tlb->mm->context.flush_mm =3D 1; tlb->freed_tables =3D 1; - tlb_remove_ptdesc(tlb, p4d); + tlb_remove_ptdesc(tlb, virt_to_ptdesc(p4d)); } =20 #endif /* _S390_TLB_H */ diff --git a/arch/s390/mm/pgalloc.c b/arch/s390/mm/pgalloc.c index c73b89811a264..3e002dea6278f 100644 --- a/arch/s390/mm/pgalloc.c +++ b/arch/s390/mm/pgalloc.c @@ -193,13 +193,6 @@ void page_table_free(struct mm_struct *mm, unsigned lo= ng *table) pagetable_dtor_free(ptdesc); } =20 -void __tlb_remove_table(void *table) -{ - struct ptdesc *ptdesc =3D virt_to_ptdesc(table); - - pagetable_dtor_free(ptdesc); -} - #ifdef CONFIG_TRANSPARENT_HUGEPAGE static void pte_free_now(struct rcu_head *head) { diff --git a/arch/sparc/include/asm/tlb_32.h b/arch/sparc/include/asm/tlb_3= 2.h index 5cd28a8793e39..910254867dfbd 100644 --- a/arch/sparc/include/asm/tlb_32.h +++ b/arch/sparc/include/asm/tlb_32.h @@ -2,6 +2,7 @@ #ifndef _SPARC_TLB_H #define _SPARC_TLB_H =20 +#define __HAVE_ARCH_TLB_REMOVE_TABLE #include =20 #endif /* _SPARC_TLB_H */ diff --git a/arch/sparc/include/asm/tlb_64.h b/arch/sparc/include/asm/tlb_6= 4.h index 3037187482db7..1a6e694418e39 100644 --- a/arch/sparc/include/asm/tlb_64.h +++ b/arch/sparc/include/asm/tlb_64.h @@ -33,6 +33,7 @@ void flush_tlb_pending(void); #define tlb_needs_table_invalidate() (false) #endif =20 +#define __HAVE_ARCH_TLB_REMOVE_TABLE #include =20 #endif /* _SPARC64_TLB_H */ diff --git a/arch/x86/include/asm/tlb.h b/arch/x86/include/asm/tlb.h index f64730be5ad67..3858dbf75880e 100644 --- a/arch/x86/include/asm/tlb.h +++ b/arch/x86/include/asm/tlb.h @@ -20,23 +20,6 @@ static inline void tlb_flush(struct mmu_gather *tlb) flush_tlb_mm_range(tlb->mm, start, end, stride_shift, tlb->freed_tables); } =20 -/* - * While x86 architecture in general requires an IPI to perform TLB - * shootdown, enablement code for several hypervisors overrides - * .flush_tlb_others hook in pv_mmu_ops and implements it by issuing - * a hypercall. To keep software pagetable walkers safe in this case we - * switch to RCU based table free (MMU_GATHER_RCU_TABLE_FREE). See the com= ment - * below 'ifdef CONFIG_MMU_GATHER_RCU_TABLE_FREE' in include/asm-generic/t= lb.h - * for more details. - */ -static inline void __tlb_remove_table(void *table) -{ - struct ptdesc *ptdesc =3D (struct ptdesc *)table; - - pagetable_dtor(ptdesc); - pagetable_free(ptdesc); -} - #ifdef CONFIG_PT_RECLAIM static inline void __tlb_remove_table_one_rcu(struct rcu_head *head) { diff --git a/include/asm-generic/tlb.h b/include/asm-generic/tlb.h index 709830274b756..69de47c7ef3c5 100644 --- a/include/asm-generic/tlb.h +++ b/include/asm-generic/tlb.h @@ -153,8 +153,9 @@ * * Useful if your architecture has non-page page directories. * - * When used, an architecture is expected to provide __tlb_remove_table() - * which does the actual freeing of these pages. + * When used, an architecture is expected to provide __tlb_remove_table()= or + * use the generic __tlb_remove_table(), which does the actual freeing of= these + * pages. * * MMU_GATHER_RCU_TABLE_FREE * @@ -207,6 +208,16 @@ struct mmu_table_batch { #define MAX_TABLE_BATCH \ ((PAGE_SIZE - sizeof(struct mmu_table_batch)) / sizeof(void *)) =20 +#ifndef __HAVE_ARCH_TLB_REMOVE_TABLE +static inline void __tlb_remove_table(void *table) +{ + struct ptdesc *ptdesc =3D (struct ptdesc *)table; + + pagetable_dtor(ptdesc); + pagetable_free(ptdesc); +} +#endif + extern void tlb_remove_table(struct mmu_gather *tlb, void *table); =20 #else /* !CONFIG_MMU_GATHER_HAVE_TABLE_FREE */ --=20 2.20.1 From nobody Sun May 10 06:27:04 2026 Received: from mail-pl1-f178.google.com (mail-pl1-f178.google.com [209.85.214.178]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 054851A262A for ; Mon, 30 Dec 2024 09:11:35 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.178 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1735549896; cv=none; b=pY9eCqU0JczDBv9bCtf4NhCVoTvNp0wWklrs99b7T66ncK8j7b7hghCO4edRjKRxwWecEAYLunkfuhbG7YGxbs+xZaf05s+Al8MwjIlbNhUxZpY3SrRtLA4kqXR3caEB53CeLNdmvT8BXCt9qt35GXqYKfb4fRaZa7PhR3HuUKI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1735549896; c=relaxed/simple; bh=AOHzYz3e0W68bypaywk5PqwWiTbsh7X+wa84FHfFzTo=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=my/PJUI6/rU6u0oiFV9fYErMu1tmq4KsyhPF0hNHKTKP+VYfgQNmj+A1cW/l8X84lF/khw0oPoLWHB5tB3N6IEBfinEP0MrlCFGbl2ne9nqrGCCQo/tvDudgs7k+Qe4UzpK/+1D3xVCBrNsjZWjxHeIMPaA2TVrnHig1ftIk8aM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=dMGXvhJm; arc=none smtp.client-ip=209.85.214.178 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="dMGXvhJm" Received: by mail-pl1-f178.google.com with SMTP id d9443c01a7336-21669fd5c7cso116693625ad.3 for ; Mon, 30 Dec 2024 01:11:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1735549894; x=1736154694; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=VAJKsDdGNb+ALwQvMYs9Oz9Vq8VGrLJsA8GxB+kZmZc=; b=dMGXvhJmZw4f/XjeFPAOypYaJ1AeNxzuqtsV2p6mWczJPmfm9zdYnU3ZIXB8Y9vMeH ywM+JXATE4QRUnmfoMuK10N2mg9FnJkSay3ado5+OllS9iSY83jaERVs+Njr7nlkoIIp UygK4eILQ2UWIoPeQNeeWAlAY/Ya8/yMig+Z6V1ZJitEJKinQWfYiq+YSRG8izLJnDpD O7F2H8ddG4ikswjzAZ8Pfccf0A8fyI6BPEemW2TroujdF+t417cgbJriclhuXcz6lpgC EpOiL2JktxJkYb4K8498bnh+BfXaitQX0eJMMBMnmcuIQLF0Gh4Ci+zvCFyDYphl89fT 14NQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1735549894; x=1736154694; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=VAJKsDdGNb+ALwQvMYs9Oz9Vq8VGrLJsA8GxB+kZmZc=; b=Oj8XGb8mcaXbW2uoTm8n+nwwHQrkvv7WqsKjJ9SCk0Q4epVd67SjqY9KLSi904bDoL hsPy2F3vDMQZhkHf87CcWAqS5fqXJe1FC4EFWGo3FexhSZ7BAqSmD/V3XqriKKQr5/YD OkzS/q72BSKRpDo/jpVRxG1ecOGp0MveH3PM3N+sXh+jYFDg0noGcL4hP/JDDoIx9uYf wIQDtdeBWe1rjoyx5Yx/TRgzVM/DAgu4+5UjzuvSoof/aqjLCPeOTvGeDVXelVl/GZoc ZskOyitqeRJ5UPli8099GhT5adSdfYrFxZwLuKgy4CqfSb/DkBNvkumOJk5ahZicvwf7 oNcg== X-Forwarded-Encrypted: i=1; AJvYcCUL3EWGw/MkxXdaGjaxhtTlkUURr/JrgZa1p1y5/x+xIkt7vlYRQ9OmOzgCFPvdv0hA3XYHJ5cKbAWn8S4=@vger.kernel.org X-Gm-Message-State: AOJu0YzJCXLnORmT7YR8HNNdyiSNFrI1l3hC77hitdimnq94QOkw/mn5 wpnSXBrtjBwZvFXLL+cvK5qqfX/q4i6ZfO/1epEIbl0td5bdsskBUk662wet8E8= X-Gm-Gg: ASbGncveD2V8AWOzxhWHR65iD2zjP2NzoKdmPag91zwZy4Us++ZxW5cbgSc66baK5y5 UH2v2h4SOSm8mDoUX03JnhjpF5BExjXWpOYfAqWzePXs5LDzNp5pjbLqhzepZ05K7G/lTZCIUZn /TzT9IKVI7wj8g8o9MskPNurjxwJSGIxo/nnT6LFjECf2zNnkM8FY1/yayfOmu9xauHC8V9hflG CO/scQrh+pZfaItFBmaAnB7tMBFi5rHwVATjxxYQk7ajMtqFfA0jfULIKRwPLopr8tIC22hCGd0 aKgk7MqoYCAnrnfzW16qAw== X-Google-Smtp-Source: AGHT+IF2zawjY8HUYTCaMJHzWmoJ2TNFStaqczmOTvWfDCo9S9fZfm/YNlaFMm14SUWbTnX0T2oRFg== X-Received: by 2002:a05:6a00:600c:b0:728:b601:86ee with SMTP id d2e1a72fcca58-72abde82a17mr50548930b3a.16.1735549894520; Mon, 30 Dec 2024 01:11:34 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-842aba72f7csm17057841a12.4.2024.12.30.01.11.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 30 Dec 2024 01:11:34 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, agordeev@linux.ibm.com, kevin.brodsky@arm.com, palmer@dabbelt.com, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com, arnd@arndb.de, will@kernel.org, aneesh.kumar@kernel.org, npiggin@gmail.com, dave.hansen@linux.intel.com, rppt@kernel.org, ryan.roberts@arm.com Cc: linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, linux-arch@vger.kernel.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, loongarch@lists.linux.dev, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-sh@vger.kernel.org, linux-um@lists.infradead.org, Qi Zheng Subject: [PATCH v4 14/15] mm: pgtable: move __tlb_remove_table_one() in x86 to generic file Date: Mon, 30 Dec 2024 17:07:49 +0800 Message-Id: <286e9777dd266dc610de20120fae453b84d3a868.1735549103.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The __tlb_remove_table_one() in x86 does not contain architecture-specific content, so move it to the generic file. Signed-off-by: Qi Zheng Reviewed-by: Kevin Brodsky --- arch/x86/include/asm/tlb.h | 19 ------------------- mm/mmu_gather.c | 20 ++++++++++++++++++-- 2 files changed, 18 insertions(+), 21 deletions(-) diff --git a/arch/x86/include/asm/tlb.h b/arch/x86/include/asm/tlb.h index 3858dbf75880e..77f52bc1578a7 100644 --- a/arch/x86/include/asm/tlb.h +++ b/arch/x86/include/asm/tlb.h @@ -20,25 +20,6 @@ static inline void tlb_flush(struct mmu_gather *tlb) flush_tlb_mm_range(tlb->mm, start, end, stride_shift, tlb->freed_tables); } =20 -#ifdef CONFIG_PT_RECLAIM -static inline void __tlb_remove_table_one_rcu(struct rcu_head *head) -{ - struct ptdesc *ptdesc; - - ptdesc =3D container_of(head, struct ptdesc, pt_rcu_head); - __tlb_remove_table(ptdesc); -} - -static inline void __tlb_remove_table_one(void *table) -{ - struct ptdesc *ptdesc; - - ptdesc =3D table; - call_rcu(&ptdesc->pt_rcu_head, __tlb_remove_table_one_rcu); -} -#define __tlb_remove_table_one __tlb_remove_table_one -#endif /* CONFIG_PT_RECLAIM */ - static inline void invlpg(unsigned long addr) { asm volatile("invlpg (%0)" ::"r" (addr) : "memory"); diff --git a/mm/mmu_gather.c b/mm/mmu_gather.c index 1e21022bcf339..7aa6f18c500b2 100644 --- a/mm/mmu_gather.c +++ b/mm/mmu_gather.c @@ -311,13 +311,29 @@ static inline void tlb_table_invalidate(struct mmu_ga= ther *tlb) } } =20 -#ifndef __tlb_remove_table_one +#ifdef CONFIG_PT_RECLAIM +static inline void __tlb_remove_table_one_rcu(struct rcu_head *head) +{ + struct ptdesc *ptdesc; + + ptdesc =3D container_of(head, struct ptdesc, pt_rcu_head); + __tlb_remove_table(ptdesc); +} + +static inline void __tlb_remove_table_one(void *table) +{ + struct ptdesc *ptdesc; + + ptdesc =3D table; + call_rcu(&ptdesc->pt_rcu_head, __tlb_remove_table_one_rcu); +} +#else static inline void __tlb_remove_table_one(void *table) { tlb_remove_table_sync_one(); __tlb_remove_table(table); } -#endif +#endif /* CONFIG_PT_RECLAIM */ =20 static void tlb_remove_table_one(void *table) { --=20 2.20.1 From nobody Sun May 10 06:27:04 2026 Received: from mail-pl1-f177.google.com (mail-pl1-f177.google.com [209.85.214.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 73B681A2C25 for ; Mon, 30 Dec 2024 09:11:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.177 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1735549909; cv=none; b=crCgvHY+ot1f37VKFqSrjJqNRUYSf0SkYeibZw3DYyOZeVq5q/dRSrzSaanGXCr7JvZXILv2ZUuSRMNx2wL2Jm8gg3MSRYr8WlKgPHjT6Ac7F7Lz/sJ1WgoMtbOXuxGY9go7DU9xl3G7kwhAZ/YwxH1OAZp0TcWJPOwn0nwBYMY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1735549909; c=relaxed/simple; bh=q1hWJFvFfNeFAOErQnfZb6jq/jCJy3LyoYOS1Q4McDs=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=n5KQRljMKNAUr9XEg7owzET3W8fvK2y0A9Ajyb+//4WF5L9/xNza2zU2r1GC50S8NTVBAQOdV8sE2I6AJziPPjRV/ZqR24aVWB4F+OOJThEUYmU9H81Mf0NOVM/GdxkqTLx9cwZTQG/mM+fEyNR6q86xlNkz5AQXTrUX6YByn+E= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=DRjaqOGl; arc=none smtp.client-ip=209.85.214.177 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="DRjaqOGl" Received: by mail-pl1-f177.google.com with SMTP id d9443c01a7336-2163bd70069so13728065ad.0 for ; Mon, 30 Dec 2024 01:11:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1735549907; x=1736154707; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Uxx1VFA8X6F1Vml9+szJuo78n9B3sJITH2LTRfnVDm8=; b=DRjaqOGlV6hw64zv26N1z30EDiOrI+YnLf0gw9T+tGPQAu5Wac7dniLrGPK9VAmnEN +DvJKvoYVfSVQvQ2kRF/gH+UTGLCXne/2W/0H1hylVgjSn0yAcztGV4DoF+2oNHOeNtZ msarIzKohC1hFB/sJ6GPvkOeZ1NpIHP9W4Nvee3wPszY4rf4iUuuFgzacyuY1JlUuIsC vNpQ3kER45NRMuUR/eUci9rlozmPgxGmh4DBofm7U3DuxsTcCcJzKoPFm/GYTMnEBi1N Zmm0uItcdvQ/QisD6womS9SM91vOWA73OpXLN5/PsRwq3Ra/inu8EqoyKMSRW3NlEQ72 y3YQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1735549907; x=1736154707; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Uxx1VFA8X6F1Vml9+szJuo78n9B3sJITH2LTRfnVDm8=; b=ji9ed86vIpm9whaMrZj7gYlFmc7uZJzfJq+s9vYcaOT0bJloQTTrIs9JlOuB1A9IGt IBWFHuQjrIuoXb0Jpu/C4dyRJ4g6WWx6pXDqbT451VymPSo7t8nE92NtUP/bBhKlx0pj irbZi6OOAJpbJ/4SfBh6vi2+jYONPkXbjRmo6VNTttbQz5fycWjFtzRfD4WTpzYJekyS egTfm/VIQ92Vch0AG6DslO6phlEPYnARMFSxHitVTjUQbIAZKaoE/pvqXQaY9LeCY5Zm AN2aud4o+25DOfUjf6UwDVi9b/WAMh6AZHbiXdLBijmXsb8RCbeWSf0qayxZfPgojxz6 iJ6w== X-Forwarded-Encrypted: i=1; AJvYcCWtpAn5aEdi+u7y9r4mualDMEJIC+5ugAj103fAqYsB5kVALrBv4fG0toxLB2GvAICut7LRE/wi14VSy3o=@vger.kernel.org X-Gm-Message-State: AOJu0YxgIhL/co2JbohoHkSgxIxBdXwbgu6M4Zj8ucOzO0c+7SswlRjk gyweWCCx8AScVFEHNBepdd80HHuUFwy1FcrHT0QaHIhPbRwGcRQaKwfnxSclYa4= X-Gm-Gg: ASbGnct4cGVAp6vAKlxf01Ax7/9VBsl4QrFBBAZjTUhf48SsIKP2uNVgqMWlFRZ9ozr Z2Pc+IWJMnPYGtVQXq1j2Y3ogVesV7elTxBUfymxq4aWpWoX9asNT+51TrC+7vbeddfFGVLqYO+ mWoSlgtGN3diFBaUyzKzf+Y3TqVNs69xnb434IpIDdTR0aSwMkMeQkLnY89Mw76r8GZepyuzvpe 9UahC/8ow3m1of1wJQD5hzvk1hi6PJy6uedyonQIc6auVXbXzeifiwI8ZVYUTpUtXD5Dx8QVuk7 gzDQ5tCl8vVMiFS+NC4qlg== X-Google-Smtp-Source: AGHT+IH6enLQBRxhPiyC+b4rbFoH6ZPXIb+9j2XLkfNoZdfNkkGtY/WgMqtF7HfldrPWQopcc9iyNw== X-Received: by 2002:a05:6a20:4308:b0:1e0:c8d9:3382 with SMTP id adf61e73a8af0-1e5e0847084mr55644198637.45.1735549906952; Mon, 30 Dec 2024 01:11:46 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-842aba72f7csm17057841a12.4.2024.12.30.01.11.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 30 Dec 2024 01:11:46 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, agordeev@linux.ibm.com, kevin.brodsky@arm.com, palmer@dabbelt.com, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com, arnd@arndb.de, will@kernel.org, aneesh.kumar@kernel.org, npiggin@gmail.com, dave.hansen@linux.intel.com, rppt@kernel.org, ryan.roberts@arm.com Cc: linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, linux-arch@vger.kernel.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, loongarch@lists.linux.dev, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-sh@vger.kernel.org, linux-um@lists.infradead.org, Qi Zheng Subject: [PATCH v4 15/15] mm: pgtable: introduce generic pagetable_dtor_free() Date: Mon, 30 Dec 2024 17:07:50 +0800 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The pte_free(), pmd_free(), __pud_free() and __p4d_free() in asm-generic/pgalloc.h and the generic __tlb_remove_table() are basically the same, so let's introduce pagetable_dtor_free() to deduplicate them. In addition, the pagetable_dtor_free() in s390 does the same thing, so let's s390 also calls generic pagetable_dtor_free(). Signed-off-by: Qi Zheng Suggested-by: Peter Zijlstra (Intel) Acked-by: Alexander Gordeev Reviewed-by: Kevin Brodsky --- arch/s390/mm/pgalloc.c | 6 ------ include/asm-generic/pgalloc.h | 12 ++++-------- include/asm-generic/tlb.h | 3 +-- include/linux/mm.h | 6 ++++++ 4 files changed, 11 insertions(+), 16 deletions(-) diff --git a/arch/s390/mm/pgalloc.c b/arch/s390/mm/pgalloc.c index 3e002dea6278f..a4e7619020931 100644 --- a/arch/s390/mm/pgalloc.c +++ b/arch/s390/mm/pgalloc.c @@ -180,12 +180,6 @@ unsigned long *page_table_alloc(struct mm_struct *mm) return table; } =20 -static void pagetable_dtor_free(struct ptdesc *ptdesc) -{ - pagetable_dtor(ptdesc); - pagetable_free(ptdesc); -} - void page_table_free(struct mm_struct *mm, unsigned long *table) { struct ptdesc *ptdesc =3D virt_to_ptdesc(table); diff --git a/include/asm-generic/pgalloc.h b/include/asm-generic/pgalloc.h index 4afb346eae255..e3977ddca15e4 100644 --- a/include/asm-generic/pgalloc.h +++ b/include/asm-generic/pgalloc.h @@ -109,8 +109,7 @@ static inline void pte_free(struct mm_struct *mm, struc= t page *pte_page) { struct ptdesc *ptdesc =3D page_ptdesc(pte_page); =20 - pagetable_dtor(ptdesc); - pagetable_free(ptdesc); + pagetable_dtor_free(ptdesc); } =20 =20 @@ -153,8 +152,7 @@ static inline void pmd_free(struct mm_struct *mm, pmd_t= *pmd) struct ptdesc *ptdesc =3D virt_to_ptdesc(pmd); =20 BUG_ON((unsigned long)pmd & (PAGE_SIZE-1)); - pagetable_dtor(ptdesc); - pagetable_free(ptdesc); + pagetable_dtor_free(ptdesc); } #endif =20 @@ -202,8 +200,7 @@ static inline void __pud_free(struct mm_struct *mm, pud= _t *pud) struct ptdesc *ptdesc =3D virt_to_ptdesc(pud); =20 BUG_ON((unsigned long)pud & (PAGE_SIZE-1)); - pagetable_dtor(ptdesc); - pagetable_free(ptdesc); + pagetable_dtor_free(ptdesc); } =20 #ifndef __HAVE_ARCH_PUD_FREE @@ -248,8 +245,7 @@ static inline void __p4d_free(struct mm_struct *mm, p4d= _t *p4d) struct ptdesc *ptdesc =3D virt_to_ptdesc(p4d); =20 BUG_ON((unsigned long)p4d & (PAGE_SIZE-1)); - pagetable_dtor(ptdesc); - pagetable_free(ptdesc); + pagetable_dtor_free(ptdesc); } =20 #ifndef __HAVE_ARCH_P4D_FREE diff --git a/include/asm-generic/tlb.h b/include/asm-generic/tlb.h index 69de47c7ef3c5..a96d4b440f3da 100644 --- a/include/asm-generic/tlb.h +++ b/include/asm-generic/tlb.h @@ -213,8 +213,7 @@ static inline void __tlb_remove_table(void *table) { struct ptdesc *ptdesc =3D (struct ptdesc *)table; =20 - pagetable_dtor(ptdesc); - pagetable_free(ptdesc); + pagetable_dtor_free(ptdesc); } #endif =20 diff --git a/include/linux/mm.h b/include/linux/mm.h index cad11fa10c192..94078c488e904 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -3001,6 +3001,12 @@ static inline void pagetable_dtor(struct ptdesc *ptd= esc) lruvec_stat_sub_folio(folio, NR_PAGETABLE); } =20 +static inline void pagetable_dtor_free(struct ptdesc *ptdesc) +{ + pagetable_dtor(ptdesc); + pagetable_free(ptdesc); +} + static inline bool pagetable_pte_ctor(struct ptdesc *ptdesc) { struct folio *folio =3D ptdesc_folio(ptdesc); --=20 2.20.1