From nobody Wed Dec 17 15:51:21 2025 Received: from mail-pf1-f181.google.com (mail-pf1-f181.google.com [209.85.210.181]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 355927E0FF for ; Sat, 14 Dec 2024 09:03:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.181 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734167028; cv=none; b=EdqjMOqfiNkw6qi6DC2PNq96+3CCla1Gl1xuonX2E2U7DlQ8Ym9TXwtLyXkvNb7Jl/3R2KF9F2AkB17zjoFO//VF8WvdR7t6yscczrjfkew1sUtt043lJAtsrWHk7zhX+4/2Xa1GBHcgVb7sdN6S4vnY8tYgwhYu9j+5AlakxBQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734167028; c=relaxed/simple; bh=CnaAuCIjIGmiKSYtT+I1U505CPuTYPZahEmuA+VOoDM=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=JFq+hpk2NWX6SWy8/KqM8sdXAp7WYzumnXz1RIv+8BsqYvNA1R1ur+YspQ+tuW2j4g9jDXNU6SoM9Pz9ryrVezpr5dPnU2hNmp2OcET9owLVZQcQdZbu1nIUCswE18FANKogEr5EJ7Oe0K1Zj5kDH/OynaLFfJC2m2RMtBv6SSE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=VBI/9JXs; arc=none smtp.client-ip=209.85.210.181 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="VBI/9JXs" Received: by mail-pf1-f181.google.com with SMTP id d2e1a72fcca58-728e3826211so2067038b3a.0 for ; Sat, 14 Dec 2024 01:03:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1734167026; x=1734771826; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=/W0KMvrB9Jil9zRQ2f7XfdU3YONHwHkiAoiFljp+njo=; b=VBI/9JXsEC1O5sF7V/Xa2U8EJY7sZUHu3FGbqOMnSYj7R0NYsS93vL89WoAWWD2HnP e2qzs+EJG9bm3DwZXU/PrkNejJw+0xCQ48kvUREoERLEU+MdDOQHglX9r0bN+VQi123k 0GNI8AeQAYQ9j9kh8WhxFM1EcKH6Uzrycdrc+hdHSRXRvzbSJ/qaP10AcpWJeBp9+vsF Lh5IgSe3d5bkpHvEwRoJ0y1n4s82fhB+AFfJielgjE6e4Os+PwC3tGFOX+mSjZ/TSEQn jT6Xa7fsQ3oEXD60Ms1uSkack3jGeP8BWRvBSMpojqmnBM36Is7vfrf4a5X/zDsI4Bdd jXmw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734167026; x=1734771826; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=/W0KMvrB9Jil9zRQ2f7XfdU3YONHwHkiAoiFljp+njo=; b=loGXD6NpqJRHPqFlxODRYsprb9Jernie6v3v2qTexp2VxC4/xXFzFOTgh7rzQZ16RS seLEkG0rfPPAS9ZS4/IGvhVKehmqQFx0rIbU0UmqqY9i+ETCNk7IfDWxoD2kBC5khDn5 6ZqwLTNs44lJjjWvdGE4WZdRq/DkAZfhq6TT3+aMgTUNNSmTkdxlg53jW9JWdTEyX+Yg wtdJ9Xfu1WEqNk4mFHkfxHo8gt0ae+TPwaszlhW/kAMOYzQzxTGJCY9ZCnKsS+Ju9Wna uM0P2bgvQswLsp7aAJlCIUDMlVJuwgBIG1z8a3N4Y994QGYLPgUrTpbanxakq9xYG0FF saUA== X-Forwarded-Encrypted: i=1; AJvYcCUY1NPY2PMoPlBY7vYtxWFtKmcfRl0Xne3aTy3ye03rpAkCAd+U3UmWlXsfhYS1hIbb9YppFCA9Sw1Fnio=@vger.kernel.org X-Gm-Message-State: AOJu0YzxHU7lPuAwzamnotc81lm0yGGx06PSjVamB5W5OMy/oL9xvmn3 orq683t1J6Jr8YvGtG7v8azJlxdu60m3f/0lxahIvsTa9FUUVS5IkU84P4akQ2g= X-Gm-Gg: ASbGncuxM68n5WyDSRa+q2DYVCrQCilUoQxs/iA+aiH+SlDpuTtTB/900aVNpE50vi6 8zOLR0Pr0bNlZQREuqVrqFgv8yAUZQ47gz3X0krmySXH98n5q5De3EQlXlpJ+iPQYZ2ASpW9yTt dthHIu1eKHFQCCh9h9CeZ1oaR4kYfNoIgtLoWoc60imdjJi0qVgfwVBoOSDMekykQ43UD8rudMg ES3K/BrYCR807aBlp4GUByFvRZQPZmD/mDe6qTd1+t4ZVEd0sbRWOs6aiiTqSOgWfyWquzXWgHL XNKtufAyb+xnDxOM6iKZc9zHs69fYg== X-Google-Smtp-Source: AGHT+IFq7u9PWr4pIiPNWcOSLFRHni+qPQ4ESKUOWoSzAEQHr7L/XMr4iW3HNmhbkmKWJmb1Wl5pRQ== X-Received: by 2002:a05:6a21:a247:b0:1e1:afa9:d39b with SMTP id adf61e73a8af0-1e1dfc12584mr8746124637.7.1734167026495; Sat, 14 Dec 2024 01:03:46 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([2409:8a28:f4f:a9a4:5d46:4ef0:daa9:1db8]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-801d5a90ba0sm905731a12.16.2024.12.14.01.03.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 14 Dec 2024 01:03:46 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng Subject: [PATCH 01/12] Revert "mm: pgtable: make ptlock be freed by RCU" Date: Sat, 14 Dec 2024 17:02:47 +0800 Message-Id: <1fdb3ee32e6958ad82229941b2213ef76b7c4705.1734164094.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This reverts commit 2f3443770437e49abc39af26962d293851cbab6d. Signed-off-by: Qi Zheng Acked-by: Yu Zhao Suggested-by: Peter Zijlstra (Intel) --- include/linux/mm.h | 2 +- include/linux/mm_types.h | 9 +-------- mm/memory.c | 22 ++++++---------------- 3 files changed, 8 insertions(+), 25 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index e7902980439cc..5e73e53c34e9e 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2988,7 +2988,7 @@ void ptlock_free(struct ptdesc *ptdesc); =20 static inline spinlock_t *ptlock_ptr(struct ptdesc *ptdesc) { - return &(ptdesc->ptl->ptl); + return ptdesc->ptl; } #else /* ALLOC_SPLIT_PTLOCKS */ static inline void ptlock_cache_init(void) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index df8f5152644ec..5d8779997266e 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -434,13 +434,6 @@ FOLIO_MATCH(flags, _flags_2a); FOLIO_MATCH(compound_head, _head_2a); #undef FOLIO_MATCH =20 -#if ALLOC_SPLIT_PTLOCKS -struct pt_lock { - spinlock_t ptl; - struct rcu_head rcu; -}; -#endif - /** * struct ptdesc - Memory descriptor for page tables. * @__page_flags: Same as page flags. Powerpc only. @@ -485,7 +478,7 @@ struct ptdesc { union { unsigned long _pt_pad_2; #if ALLOC_SPLIT_PTLOCKS - struct pt_lock *ptl; + spinlock_t *ptl; #else spinlock_t ptl; #endif diff --git a/mm/memory.c b/mm/memory.c index d9af83dd86bbf..83765632e20b0 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -7041,34 +7041,24 @@ static struct kmem_cache *page_ptl_cachep; =20 void __init ptlock_cache_init(void) { - page_ptl_cachep =3D kmem_cache_create("page->ptl", sizeof(struct pt_lock)= , 0, + page_ptl_cachep =3D kmem_cache_create("page->ptl", sizeof(spinlock_t), 0, SLAB_PANIC, NULL); } =20 bool ptlock_alloc(struct ptdesc *ptdesc) { - struct pt_lock *pt_lock; + spinlock_t *ptl; =20 - pt_lock =3D kmem_cache_alloc(page_ptl_cachep, GFP_KERNEL); - if (!pt_lock) + ptl =3D kmem_cache_alloc(page_ptl_cachep, GFP_KERNEL); + if (!ptl) return false; - ptdesc->ptl =3D pt_lock; + ptdesc->ptl =3D ptl; return true; } =20 -static void ptlock_free_rcu(struct rcu_head *head) -{ - struct pt_lock *pt_lock; - - pt_lock =3D container_of(head, struct pt_lock, rcu); - kmem_cache_free(page_ptl_cachep, pt_lock); -} - void ptlock_free(struct ptdesc *ptdesc) { - struct pt_lock *pt_lock =3D ptdesc->ptl; - - call_rcu(&pt_lock->rcu, ptlock_free_rcu); + kmem_cache_free(page_ptl_cachep, ptdesc->ptl); } #endif =20 --=20 2.20.1 From nobody Wed Dec 17 15:51:21 2025 Received: from mail-pf1-f175.google.com (mail-pf1-f175.google.com [209.85.210.175]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2BCE07EF09 for ; Sat, 14 Dec 2024 09:03:55 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.175 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734167038; cv=none; b=OxaVS+cJoO+VuKkeKJT2fsS8iwQl7Qm47jpOzUJo5+KP0EtiKdrev00QZySuPRf6YIdTLrJ/012wZ+iIdNRYz2uVFFHlEoQDvmOUUuSAMEyYo8HNQfowcxV6IFMdmw7+xM1cvcQN43LhtwH6CHEdRV2vRgvPVhPsFeXkZpQvI/Q= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734167038; c=relaxed/simple; bh=BYquZoRYPXNRHDL38YoR5kg8KSza8ULPTlVHq7lYXtE=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=HGHMdkjCE9rESj2WtQoRsi98BkRenmvydwru6GvnRVpDChNvfj9RYXJJMGcHGn8qHLeY3AKnE+mu94qkksfoyDg9adO66ghEiF0Qv7qb0D68gpr5mQZRZIzm68WMLnzAcCk/hn842SlJfsyREzE95Q//21+W74wtrhRwbAve3k0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=NxBUg6eP; arc=none smtp.client-ip=209.85.210.175 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="NxBUg6eP" Received: by mail-pf1-f175.google.com with SMTP id d2e1a72fcca58-725dac69699so2193213b3a.0 for ; Sat, 14 Dec 2024 01:03:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1734167035; x=1734771835; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ogKjScRTvWjbxx3zCDWukawvbFfMJH57eI/cAcIUhW0=; b=NxBUg6eP3z/QA5465yrtLOjY+clr3fWDwXvvL7o1zwvtmNi6Iz8fPHKsOHeKOtUIDQ gy8Zl994lcu4eTcWfhhpvpCds3S4+H/T1eaYOXMQXaHRmGHwMdnJQBulMuPsAW9xlqyw 0aL9k9p1f8HD3w6dlbODRVuB22dITXZsGc59caDXxNvBWM3jV6iEpS1rthvlO8yh1Mnf DyZrAt2sM0yzxNKNiSXYmb6CQq29WOCbpX9gOZo9XvGlTQS36g54yqGXGR3K3XQn0F9Y fKi57jBnXOjPv9NaCfOpqqWTM/RkvnFfYFspIobJ6anHTVuXOCkJ+Rdyd6TWlmkWCDbr CPGw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734167035; x=1734771835; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ogKjScRTvWjbxx3zCDWukawvbFfMJH57eI/cAcIUhW0=; b=HpT+qESp745LA0zjDMvI/beLucygEN8vnVm36JLJziZSXFeu5nOjJdY+w8MvTixjjx NlWRrS7awj4D4HxtOKdx+84GjV6IVOar4F1L9y3jkheN2cdkQB+rLQoziTzDWUGJjVzN I16/UtzAfrdLyCcGCG9L84SStcmbZn2ZlsnNlRDdZTESDA7Td2gjjM4zY8EexFvfvIcf 4K05439fDvmjg+cF46KXldgg3j1LO7pHBbb1sOtfMF6wRQYzkNeYg657agsnzaFhCuJb erxAoO6d1HQZFS6UMjhTMzlGKfL22rMvPolgnqiKcQDCscm4Qyf0LNbsr67YZ49pL/+S R5kA== X-Forwarded-Encrypted: i=1; AJvYcCUExFaIZQ7I43pdzfxwngcNlpO4yMeLk2xLnXgW3+DTxKWu2FCMW+t6sFgsfUCyr64vujC07KiTNTtZ+fI=@vger.kernel.org X-Gm-Message-State: AOJu0YwQKLV+Nda4GL9WFsU1aft++Zx/OQaW6Kv+Hu4G8WkbpqsETgqK GJ7N9NIl20v4/4rzTfgQngr0TqEd7rgmGq5no8CVA/b6xJmqZ4C09AXShuAV5Wc= X-Gm-Gg: ASbGncsixeau5wUeBJpZXr5lc8fwE9nCkudPRSFxtUDOL1kpqjTOG6IVpL/XLY81aRY jS+LhPw6ov60TrYUWdvjRWC0BlAQXIBFaAUYQ63d4hyVl18dxGeZPzrayntseGY6Fmo1jp0N26A KohHVl+9Mpies48pASSNvZLtvPmTyQxEiyRciYIviiyp7ftD61x33LWcdKmH4H7gA9PVvA3LAQR B9GYuEJ0t8QCsdb0wLBm02SUgrIZa6pV1p+hQKwzqh4KIEZEydyHXFZbebmgqmSs2YkfSicImSs KkzAkP7d4PdrNXE5QbPUX23dsmPWjg== X-Google-Smtp-Source: AGHT+IH8cEofDdPO5AW+ExE/DB+fdraWijYPkixsUFC1j8JWg1c9+2I6Y5RryDFAZbl9xgKYU+lFhw== X-Received: by 2002:a05:6a20:9f05:b0:1d9:6c9c:75ea with SMTP id adf61e73a8af0-1e1dfbfbe78mr8233213637.5.1734167035315; Sat, 14 Dec 2024 01:03:55 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([2409:8a28:f4f:a9a4:5d46:4ef0:daa9:1db8]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-801d5a90ba0sm905731a12.16.2024.12.14.01.03.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 14 Dec 2024 01:03:54 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng Subject: [PATCH 02/12] mm: pgtable: introduce generic p4d_alloc_one() and p4d_free() Date: Sat, 14 Dec 2024 17:02:48 +0800 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Several architectures (arm64, riscv, x86) define p4d_alloc_one() as a wrapper for get_zeroed_page() and p4d_free() as a wrapper for free_page(). For these architectures, provide a generic implementation in asm-generic/pgalloc.h and convert them to use it. And like other levels of page tables, add statistics for P4D level page table. For s390, it also defines p4d_alloc_one() and p4d_free(), but it uses its own logic, so skip it. Signed-off-by: Qi Zheng Suggested-by: Peter Zijlstra (Intel) --- arch/arm64/include/asm/pgalloc.h | 15 ++++----- arch/riscv/include/asm/pgalloc.h | 25 ++++++--------- arch/x86/include/asm/pgalloc.h | 16 ++++------ arch/x86/mm/pgtable.c | 3 ++ include/asm-generic/pgalloc.h | 55 ++++++++++++++++++++++++++++++++ include/linux/mm.h | 16 ++++++++++ 6 files changed, 98 insertions(+), 32 deletions(-) diff --git a/arch/arm64/include/asm/pgalloc.h b/arch/arm64/include/asm/pgal= loc.h index e75422864d1bd..679c530549327 100644 --- a/arch/arm64/include/asm/pgalloc.h +++ b/arch/arm64/include/asm/pgalloc.h @@ -15,6 +15,8 @@ =20 #define __HAVE_ARCH_PGD_FREE #define __HAVE_ARCH_PUD_FREE +#define __HAVE_ARCH_P4D_ALLOC_ONE +#define __HAVE_ARCH_P4D_FREE #include =20 #define PGD_SIZE (PTRS_PER_PGD * sizeof(pgd_t)) @@ -87,19 +89,16 @@ static inline void pgd_populate(struct mm_struct *mm, p= gd_t *pgdp, p4d_t *p4dp) =20 static inline p4d_t *p4d_alloc_one(struct mm_struct *mm, unsigned long add= r) { - gfp_t gfp =3D GFP_PGTABLE_USER; + if (!pgtable_l5_enabled()) + return NULL; =20 - if (mm =3D=3D &init_mm) - gfp =3D GFP_PGTABLE_KERNEL; - return (p4d_t *)get_zeroed_page(gfp); + return __p4d_alloc_one(mm, addr); } =20 static inline void p4d_free(struct mm_struct *mm, p4d_t *p4d) { - if (!pgtable_l5_enabled()) - return; - BUG_ON((unsigned long)p4d & (PAGE_SIZE-1)); - free_page((unsigned long)p4d); + if (pgtable_l5_enabled()) + __p4d_free(mm, p4d); } =20 #define __p4d_free_tlb(tlb, p4d, addr) p4d_free((tlb)->mm, p4d) diff --git a/arch/riscv/include/asm/pgalloc.h b/arch/riscv/include/asm/pgal= loc.h index f52264304f772..bb6e1c5f1fb19 100644 --- a/arch/riscv/include/asm/pgalloc.h +++ b/arch/riscv/include/asm/pgalloc.h @@ -14,6 +14,8 @@ #ifdef CONFIG_MMU #define __HAVE_ARCH_PUD_ALLOC_ONE #define __HAVE_ARCH_PUD_FREE +#define __HAVE_ARCH_P4D_ALLOC_ONE +#define __HAVE_ARCH_P4D_FREE #include =20 static inline void riscv_tlb_remove_ptdesc(struct mmu_gather *tlb, void *p= t) @@ -118,21 +120,10 @@ static inline void __pud_free_tlb(struct mmu_gather *= tlb, pud_t *pud, #define p4d_alloc_one p4d_alloc_one static inline p4d_t *p4d_alloc_one(struct mm_struct *mm, unsigned long add= r) { - if (pgtable_l5_enabled) { - gfp_t gfp =3D GFP_PGTABLE_USER; - - if (mm =3D=3D &init_mm) - gfp =3D GFP_PGTABLE_KERNEL; - return (p4d_t *)get_zeroed_page(gfp); - } + if (!pgtable_l5_enabled) + return NULL; =20 - return NULL; -} - -static inline void __p4d_free(struct mm_struct *mm, p4d_t *p4d) -{ - BUG_ON((unsigned long)p4d & (PAGE_SIZE-1)); - free_page((unsigned long)p4d); + return __p4d_alloc_one(mm, addr); } =20 #define p4d_free p4d_free @@ -145,8 +136,12 @@ static inline void p4d_free(struct mm_struct *mm, p4d_= t *p4d) static inline void __p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d, unsigned long addr) { - if (pgtable_l5_enabled) + if (pgtable_l5_enabled) { + struct ptdesc *ptdesc =3D virt_to_ptdesc(p4d); + + pagetable_p4d_dtor(ptdesc); riscv_tlb_remove_ptdesc(tlb, virt_to_ptdesc(p4d)); + } } #endif /* __PAGETABLE_PMD_FOLDED */ =20 diff --git a/arch/x86/include/asm/pgalloc.h b/arch/x86/include/asm/pgalloc.h index dcd836b59bebd..d9bc6cae77c9e 100644 --- a/arch/x86/include/asm/pgalloc.h +++ b/arch/x86/include/asm/pgalloc.h @@ -8,6 +8,8 @@ =20 #define __HAVE_ARCH_PTE_ALLOC_ONE #define __HAVE_ARCH_PGD_FREE +#define __HAVE_ARCH_P4D_ALLOC_ONE +#define __HAVE_ARCH_P4D_FREE #include =20 static inline int __paravirt_pgd_alloc(struct mm_struct *mm) { return 0; } @@ -149,20 +151,16 @@ static inline void pgd_populate_safe(struct mm_struct= *mm, pgd_t *pgd, p4d_t *p4 =20 static inline p4d_t *p4d_alloc_one(struct mm_struct *mm, unsigned long add= r) { - gfp_t gfp =3D GFP_KERNEL_ACCOUNT; + if (!pgtable_l5_enabled()) + return NULL; =20 - if (mm =3D=3D &init_mm) - gfp &=3D ~__GFP_ACCOUNT; - return (p4d_t *)get_zeroed_page(gfp); + return __p4d_alloc_one(mm, addr); } =20 static inline void p4d_free(struct mm_struct *mm, p4d_t *p4d) { - if (!pgtable_l5_enabled()) - return; - - BUG_ON((unsigned long)p4d & (PAGE_SIZE-1)); - free_page((unsigned long)p4d); + if (pgtable_l5_enabled()) + return __p4d_free(mm, p4d); } =20 extern void ___p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d); diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c index 69a357b15974a..3d6e84da45b24 100644 --- a/arch/x86/mm/pgtable.c +++ b/arch/x86/mm/pgtable.c @@ -94,6 +94,9 @@ void ___pud_free_tlb(struct mmu_gather *tlb, pud_t *pud) #if CONFIG_PGTABLE_LEVELS > 4 void ___p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d) { + struct ptdesc *ptdesc =3D virt_to_ptdesc(p4d); + + pagetable_p4d_dtor(ptdesc); paravirt_release_p4d(__pa(p4d) >> PAGE_SHIFT); paravirt_tlb_remove_table(tlb, virt_to_page(p4d)); } diff --git a/include/asm-generic/pgalloc.h b/include/asm-generic/pgalloc.h index 7c48f5fbf8aa7..dbf61819b3581 100644 --- a/include/asm-generic/pgalloc.h +++ b/include/asm-generic/pgalloc.h @@ -215,6 +215,61 @@ static inline void pud_free(struct mm_struct *mm, pud_= t *pud) =20 #endif /* CONFIG_PGTABLE_LEVELS > 3 */ =20 +#if CONFIG_PGTABLE_LEVELS > 4 + +static inline p4d_t *__p4d_alloc_one_noprof(struct mm_struct *mm, unsigned= long addr) +{ + gfp_t gfp =3D GFP_PGTABLE_USER; + struct ptdesc *ptdesc; + + if (mm =3D=3D &init_mm) + gfp =3D GFP_PGTABLE_KERNEL; + gfp &=3D ~__GFP_HIGHMEM; + + ptdesc =3D pagetable_alloc_noprof(gfp, 0); + if (!ptdesc) + return NULL; + + pagetable_p4d_ctor(ptdesc); + return ptdesc_address(ptdesc); +} +#define __p4d_alloc_one(...) alloc_hooks(__p4d_alloc_one_noprof(__VA_ARGS_= _)) + +#ifndef __HAVE_ARCH_P4D_ALLOC_ONE +/** + * p4d_alloc_one - allocate memory for a P4D-level page table + * @mm: the mm_struct of the current context + * + * Allocate memory for a page table using %GFP_PGTABLE_USER for user conte= xt + * and %GFP_PGTABLE_KERNEL for kernel context. + * + * Return: pointer to the allocated memory or %NULL on error + */ +static inline p4d_t *p4d_alloc_one_noprof(struct mm_struct *mm, unsigned l= ong addr) +{ + return __p4d_alloc_one_noprof(mm, addr); +} +#define p4d_alloc_one(...) alloc_hooks(p4d_alloc_one_noprof(__VA_ARGS__)) +#endif + +static inline void __p4d_free(struct mm_struct *mm, p4d_t *p4d) +{ + struct ptdesc *ptdesc =3D virt_to_ptdesc(p4d); + + BUG_ON((unsigned long)p4d & (PAGE_SIZE-1)); + pagetable_p4d_dtor(ptdesc); + pagetable_free(ptdesc); +} + +#ifndef __HAVE_ARCH_P4D_FREE +static inline void p4d_free(struct mm_struct *mm, pud_t *p4d) +{ + __p4d_free(mm, p4d); +} +#endif + +#endif + #ifndef __HAVE_ARCH_PGD_FREE static inline void pgd_free(struct mm_struct *mm, pgd_t *pgd) { diff --git a/include/linux/mm.h b/include/linux/mm.h index 5e73e53c34e9e..807a12ed8ec96 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -3237,6 +3237,22 @@ static inline void pagetable_pud_dtor(struct ptdesc = *ptdesc) lruvec_stat_sub_folio(folio, NR_PAGETABLE); } =20 +static inline void pagetable_p4d_ctor(struct ptdesc *ptdesc) +{ + struct folio *folio =3D ptdesc_folio(ptdesc); + + __folio_set_pgtable(folio); + lruvec_stat_add_folio(folio, NR_PAGETABLE); +} + +static inline void pagetable_p4d_dtor(struct ptdesc *ptdesc) +{ + struct folio *folio =3D ptdesc_folio(ptdesc); + + __folio_clear_pgtable(folio); + lruvec_stat_sub_folio(folio, NR_PAGETABLE); +} + extern void __init pagecache_init(void); extern void free_initmem(void); =20 --=20 2.20.1 From nobody Wed Dec 17 15:51:21 2025 Received: from mail-pf1-f178.google.com (mail-pf1-f178.google.com [209.85.210.178]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id F3CF3186E58 for ; Sat, 14 Dec 2024 09:04:04 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.178 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734167046; cv=none; b=fVbK+PzugG+bQRD+LJE1MGPEH9T4pER/+m4OW+sMoNNvRvRef+c6yzePNL+EqUO9RZ794VJt16qC/HnhNUE7mW7wDAVreIYw9u1lZ5vsQCusylyS6zzbAMSigbib9N4YRM3xc+AokmZSdy4OlDoZraw3s3aa5Qu3KKl3Z/La0Gg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734167046; c=relaxed/simple; bh=1uZWvJuOb4bQfFe2ieD3kaHKgbuZ33TDPnehFQMQRrU=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=ZglGtvNChv1EahA+Qwqw1XSo7xeoX8qgd79ZxUwJ9NmgJbahgDLQD3LeD5nsHusuNg6tadoyiCCVyp0mxavuur3RwnYTT2h4YjlkwnqRqIjjtnF0cgFq05pLNtig6sN3r198fpzXCMFYX/E3uzzhos3QyQULScFtD1NgWkbcWK8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=gv8m8mur; arc=none smtp.client-ip=209.85.210.178 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="gv8m8mur" Received: by mail-pf1-f178.google.com with SMTP id d2e1a72fcca58-728e729562fso2081678b3a.0 for ; Sat, 14 Dec 2024 01:04:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1734167044; x=1734771844; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=pyTYWh3zj2l92o0qEOJPcU0U+om/laakNd04Rx8p+XM=; b=gv8m8murWGbU5tdHkAGdgQ39VDe+/aUpStOBUmcf9tmjQRCBdrj46nHyqmamOvR379 WLibRE+jURsk3cj/x61nnwLnoqEA46KIdz3w3hVD7nXO4pkbIes+VoCeU3snsTmfH2Bm 8HggWCCVXUiaO6rmXDm7SfZfwIntlHWZpuk3b+WqUJmhaHtC6cwFxqrpUw8TVFI43aMK /q5fH0rkWZloy13+6c9eS1PP2tuRujVTghaZfxlQFnqJaZSKCfMrTrZ9QiO0wfwZQw1c Ss8+0qLxqBs4+yPsGEueYejXMpjVXtDmPGsIDQ7h/PPjymreEYf0LRrVjMpQy22StgT6 cltg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734167044; x=1734771844; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=pyTYWh3zj2l92o0qEOJPcU0U+om/laakNd04Rx8p+XM=; b=o318ZqJdRhZcKLm8rFpT1X9dHR1ml5tVXECvxiQrn2UP/H0MB8S0TXs01+5DHj43dv cQVq84LRUjZBo3Oc1juvllELTkSileDQH3qEHrd9CNn4D4oedxaMNOjNN/+QUgjIk913 sx+Gz/WsdFwblUrqkpMyyofQEOmkZGwpHQSvMWwXMJ7LaGYnewtgfqs1lKoe4Gc3/eEm vxlA4JAYeYULJKuyVfrrqlbeJtIo5e9EGrp2Lv2oddmI0Enhq2VOGvTI5AT4+mPNYy9v EDyfH+9UTCZLsg5jSN7xl2cqKbPAwGJTY6qKMck4ewcS+EWXvlLUdeBewzflfKOLhS5x MNPg== X-Forwarded-Encrypted: i=1; AJvYcCVPsEZdv0YlfVS3zVt27dtJ7+2fjtNOGLTGZ4qONSz7+0LxAcVKwiP3eM68YXmdne348nRVJeUNvEGaIPY=@vger.kernel.org X-Gm-Message-State: AOJu0Yzf2wZZfdZ0hiRA3UMFrUV+VdPFHYRWsxG0xKJxHTEbRHUC2n6g KhSUlwE/ZmdMFvH42CUC/feDsGISK6I86zTf/qh4eNbxJbFh3w53Xj3PKlDMRDs= X-Gm-Gg: ASbGncuZ4HccxC7ycULXyLLCFmWWVajExxbShvK2emGzs3uKCSCo0uZNCv7mRpcsVG6 1zLA1j9RVNbQBI3RZ9M+4l8o5alhtPVh4lb2KZRVGx9MC+y7I77KxQymeGtofH3OVSJwM250Tk6 wUf1PMXokl0qsNX1FJYHA3bMARBY6Ic8q21e9zPsW8ms6wn14dYZYMqLu2+YsHOhYR/ps4Ymsl7 Xsv6D9rL8RSKPfvifCiuRoingmfc8qN5ly5CiJaFoqul888rx9QaCu2gAEVNvgVSPIrzgC7sqd/ I+fsVIwXx2fTzm+ENXTrPjA0uJPm8g== X-Google-Smtp-Source: AGHT+IFlpyRvtlHG2M4nIIBLJzVMLLLjWf8IFr835Xb0HMOnd3F4yHyM51MIPZOReaS2iWrLwbmGTw== X-Received: by 2002:a17:902:d481:b0:216:4b66:1f1f with SMTP id d9443c01a7336-21892a7c538mr78037165ad.54.1734167044262; Sat, 14 Dec 2024 01:04:04 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([2409:8a28:f4f:a9a4:5d46:4ef0:daa9:1db8]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-801d5a90ba0sm905731a12.16.2024.12.14.01.03.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 14 Dec 2024 01:04:03 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng , linux-arm-kernel@lists.infradead.org Subject: [PATCH 03/12] arm64: pgtable: use mmu gather to free p4d level page table Date: Sat, 14 Dec 2024 17:02:49 +0800 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Like other levels of page tables, also use mmu gather mechanism to free p4d level page table. Signed-off-by: Qi Zheng Cc: linux-arm-kernel@lists.infradead.org Suggested-by: Peter Zijlstra (Intel) --- arch/arm64/include/asm/pgalloc.h | 2 -- arch/arm64/include/asm/tlb.h | 14 ++++++++++++++ 2 files changed, 14 insertions(+), 2 deletions(-) diff --git a/arch/arm64/include/asm/pgalloc.h b/arch/arm64/include/asm/pgal= loc.h index 679c530549327..58cc260b6bee8 100644 --- a/arch/arm64/include/asm/pgalloc.h +++ b/arch/arm64/include/asm/pgalloc.h @@ -100,8 +100,6 @@ static inline void p4d_free(struct mm_struct *mm, p4d_t= *p4d) if (pgtable_l5_enabled()) __p4d_free(mm, p4d); } - -#define __p4d_free_tlb(tlb, p4d, addr) p4d_free((tlb)->mm, p4d) #else static inline void __pgd_populate(pgd_t *pgdp, phys_addr_t p4dp, pgdval_t = prot) { diff --git a/arch/arm64/include/asm/tlb.h b/arch/arm64/include/asm/tlb.h index a947c6e784ed2..445282cde9afb 100644 --- a/arch/arm64/include/asm/tlb.h +++ b/arch/arm64/include/asm/tlb.h @@ -111,4 +111,18 @@ static inline void __pud_free_tlb(struct mmu_gather *t= lb, pud_t *pudp, } #endif =20 +#if CONFIG_PGTABLE_LEVELS > 4 +static inline void __p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4dp, + unsigned long addr) +{ + struct ptdesc *ptdesc =3D virt_to_ptdesc(p4dp); + + if (!pgtable_l5_enabled()) + return; + + pagetable_p4d_dtor(ptdesc); + tlb_remove_ptdesc(tlb, ptdesc); +} +#endif + #endif --=20 2.20.1 From nobody Wed Dec 17 15:51:21 2025 Received: from mail-pj1-f46.google.com (mail-pj1-f46.google.com [209.85.216.46]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E462E139587 for ; Sat, 14 Dec 2024 09:04:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.46 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734167055; cv=none; b=Eetz3aWZUAMcF6T0h0gkc5Ya0IFMjNJPg+wmiIUP3Q8Qp/kvOzF0Fh3F34YcVNUTS/YzXHBCfFQZqaCCXFq5R9Q2ExwonuuJ+qT2a6n1iA6RIxb5ZLpeV82BW/CT5uv8f6Dqy0cH572a5Yi5+lXLD7BL1+LG2Qj8alqU37twUPc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734167055; c=relaxed/simple; bh=WunXdhU7qtQHmUT93eVkt+8JRouSIOc5FLEynmCEfmU=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=PWzpjT9N9f2UGnud/BoWlbPrpHl3Z7mS/qdpznmtI0ryxUiCTKNkOna63n7NPXwUz1X5hWvDf1BJN4AsdcY/mClcPKcYH47qDnzFUHOr5o/AtAP5fzOi9mn5oRBva9Q1YVFa/dWyqKDyUYNoEDn4sZHE8rRe5GQxSiPCU5T9WHQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=QX9pEfn0; arc=none smtp.client-ip=209.85.216.46 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="QX9pEfn0" Received: by mail-pj1-f46.google.com with SMTP id 98e67ed59e1d1-2f1459b6f84so1656847a91.0 for ; Sat, 14 Dec 2024 01:04:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1734167053; x=1734771853; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=PLGKRzIB8HXo0VMNVO8Y1nbnLVdnH3jBTxV4FtFLzdM=; b=QX9pEfn0hCffAO5eIM3JGCbr0eL2BLuIky+xnV1G9vki5vBIdJPLGhkiTvvQmWaezb faHyzqqGyUK2Xpg6uMaUlvIaMnEm2DVKuFlqnFRZVIo8i5TBlLfHY6vRYNkde6HK2U2C 3jzLLlH/n2OIN6ECiNS3+vJGd4fcbyPr7EhtqokLe0Zanx990yglNCX+MnIr4jS96hW3 IHF++IWi5zKcpoOLcInhoSGYR2Ro/pxefyPo5jZAQ4tmmJiWLuOzmXLDxNaX30sy+TIa HVVzq+G/E0cI/Bcg9jXpt8hNAk02Ar2LFV8UBCzbdsEvbH/rb67kVxvpSucIkY77jbNj fwxw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734167053; x=1734771853; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=PLGKRzIB8HXo0VMNVO8Y1nbnLVdnH3jBTxV4FtFLzdM=; b=KbgSpNS0/tOxfDPTgxZ2EXMvKutXrgbTB9az/eqFtAAwlo56m0vBjS/b1s+4l+2NED 3zzyOotAE688wn0RHMxLmy5LfcbSj+AcJo5OxqRmC2w92ugdeGFPjngYXii21jfAHpEt 6w9K/ScvJSl4Q+BSJZ/HsLEF+BoAMnRziTMz4G7ROjpN9GWhh3fy7+9cKLC0z1b5TMyI Y8dlJjsiEsnrTZUXStmRZK5q7kk+cB6TrlQAs/Uoxvk7sUYfaYs7Rdyj9hpSz+II791T q4mnM7g1Rj6eWdmbrjfgIu6svlp8Vh3ws3VAnTd5uWL6ovmMs1tm7RGGvBFxMv3kFT/D 3/1A== X-Forwarded-Encrypted: i=1; AJvYcCWD6liJ6a1FQTazVf1o236qVXw3eq+3z3IST1xbmntjmpn0b3tFe/45QweoF2/NFfb12IFm3+sTSTKuuVk=@vger.kernel.org X-Gm-Message-State: AOJu0YwXV/4RDyJHp0k5gt9C+KUuPusBo7n/xMDsM77iXyX4ZfLxBdY/ 8rv1Ns5bLL3BYUP+14tqYKmj8DlisByK8lxhNu2siEj8xXTJmYCL7kclDPTUQVM= X-Gm-Gg: ASbGncv+Nd3U4tUyvAlIEr88+Duzjb0B/7UuBDlQYZ7Kp0tEQUWUOcHQcIJc6zROUsq 4lYe5CYbQWBqy4icqytK0z4Zws91Xh4SZFol+bvOdz8NuADg/7fvh6Rzj6SlKYdEc2KK5fel+A5 RiIt11n+46pY6WA0gm6gaiEqoyQ0mssaZ15lE8l06kmVvqRW66Jy59qU6Jthv6vGe65JNpXh4lY Aa70M/EU4/cN8n0lFsuZJYoqmGz7SO590IKe2p/cfUq9M9CfRrqLzZ6HBoQ6lSnWH2HCJ9G41Z5 ee2R2FNlDEi/mV9vi0J6GqVyv49Dcw== X-Google-Smtp-Source: AGHT+IEUuhlzMDGPbhHcy72Znz6SKiudYIj5PWGCvj9lWM7sC/HnXgPB6JSnynFl46+kyeoAQCdvNg== X-Received: by 2002:a17:90b:3848:b0:2ee:d797:408b with SMTP id 98e67ed59e1d1-2f28fa55c45mr8606530a91.2.1734167053262; Sat, 14 Dec 2024 01:04:13 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([2409:8a28:f4f:a9a4:5d46:4ef0:daa9:1db8]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-801d5a90ba0sm905731a12.16.2024.12.14.01.04.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 14 Dec 2024 01:04:12 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng , linux-s390@vger.kernel.org Subject: [PATCH 04/12] s390: pgtable: add statistics for PUD and P4D level page table Date: Sat, 14 Dec 2024 17:02:50 +0800 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Like PMD and PTE level page table, also add statistics for PUD and P4D page table. Signed-off-by: Qi Zheng Cc: linux-s390@vger.kernel.org Suggested-by: Peter Zijlstra (Intel) --- arch/s390/include/asm/pgalloc.h | 29 +++++++++++++++++++------- arch/s390/include/asm/tlb.h | 37 +++++++++++++++++---------------- 2 files changed, 40 insertions(+), 26 deletions(-) diff --git a/arch/s390/include/asm/pgalloc.h b/arch/s390/include/asm/pgallo= c.h index 7b84ef6dc4b6d..a0c1ca5d8423c 100644 --- a/arch/s390/include/asm/pgalloc.h +++ b/arch/s390/include/asm/pgalloc.h @@ -53,29 +53,42 @@ static inline p4d_t *p4d_alloc_one(struct mm_struct *mm= , unsigned long address) { unsigned long *table =3D crst_table_alloc(mm); =20 - if (table) - crst_table_init(table, _REGION2_ENTRY_EMPTY); + if (!table) + return NULL; + crst_table_init(table, _REGION2_ENTRY_EMPTY); + pagetable_p4d_ctor(virt_to_ptdesc(table)); + return (p4d_t *) table; } =20 static inline void p4d_free(struct mm_struct *mm, p4d_t *p4d) { - if (!mm_p4d_folded(mm)) - crst_table_free(mm, (unsigned long *) p4d); + if (mm_p4d_folded(mm)) + return; + + pagetable_p4d_dtor(virt_to_ptdesc(p4d)); + crst_table_free(mm, (unsigned long *) p4d); } =20 static inline pud_t *pud_alloc_one(struct mm_struct *mm, unsigned long add= ress) { unsigned long *table =3D crst_table_alloc(mm); - if (table) - crst_table_init(table, _REGION3_ENTRY_EMPTY); + + if (!table) + return NULL; + crst_table_init(table, _REGION3_ENTRY_EMPTY); + pagetable_pud_ctor(virt_to_ptdesc(table)); + return (pud_t *) table; } =20 static inline void pud_free(struct mm_struct *mm, pud_t *pud) { - if (!mm_pud_folded(mm)) - crst_table_free(mm, (unsigned long *) pud); + if (mm_pud_folded(mm)) + return; + + pagetable_pud_dtor(virt_to_ptdesc(pud)); + crst_table_free(mm, (unsigned long *) pud); } =20 static inline pmd_t *pmd_alloc_one(struct mm_struct *mm, unsigned long vma= ddr) diff --git a/arch/s390/include/asm/tlb.h b/arch/s390/include/asm/tlb.h index e95b2c8081eb8..b946964afce8e 100644 --- a/arch/s390/include/asm/tlb.h +++ b/arch/s390/include/asm/tlb.h @@ -110,24 +110,6 @@ static inline void pmd_free_tlb(struct mmu_gather *tlb= , pmd_t *pmd, tlb_remove_ptdesc(tlb, pmd); } =20 -/* - * p4d_free_tlb frees a pud table and clears the CRSTE for the - * region second table entry from the tlb. - * If the mm uses a four level page table the single p4d is freed - * as the pgd. p4d_free_tlb checks the asce_limit against 8PB - * to avoid the double free of the p4d in this case. - */ -static inline void p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d, - unsigned long address) -{ - if (mm_p4d_folded(tlb->mm)) - return; - __tlb_adjust_range(tlb, address, PAGE_SIZE); - tlb->mm->context.flush_mm =3D 1; - tlb->freed_tables =3D 1; - tlb_remove_ptdesc(tlb, p4d); -} - /* * pud_free_tlb frees a pud table and clears the CRSTE for the * region third table entry from the tlb. @@ -140,11 +122,30 @@ static inline void pud_free_tlb(struct mmu_gather *tl= b, pud_t *pud, { if (mm_pud_folded(tlb->mm)) return; + pagetable_pud_dtor(virt_to_ptdesc(pud)); tlb->mm->context.flush_mm =3D 1; tlb->freed_tables =3D 1; tlb->cleared_p4ds =3D 1; tlb_remove_ptdesc(tlb, pud); } =20 +/* + * p4d_free_tlb frees a p4d table and clears the CRSTE for the + * region second table entry from the tlb. + * If the mm uses a four level page table the single p4d is freed + * as the pgd. p4d_free_tlb checks the asce_limit against 8PB + * to avoid the double free of the p4d in this case. + */ +static inline void p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d, + unsigned long address) +{ + if (mm_p4d_folded(tlb->mm)) + return; + pagetable_p4d_dtor(virt_to_ptdesc(p4d)); + __tlb_adjust_range(tlb, address, PAGE_SIZE); + tlb->mm->context.flush_mm =3D 1; + tlb->freed_tables =3D 1; + tlb_remove_ptdesc(tlb, p4d); +} =20 #endif /* _S390_TLB_H */ --=20 2.20.1 From nobody Wed Dec 17 15:51:21 2025 Received: from mail-pf1-f170.google.com (mail-pf1-f170.google.com [209.85.210.170]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B4398189902 for ; Sat, 14 Dec 2024 09:04:23 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.170 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734167065; cv=none; b=B92IXL/fXAMl3QItoqw2aqtCwZrx9tyoKK9kpYYwXZq2GxHMW9gXzl+jo3B75sjZeBAPPulaA9GsA/7+OnjYweGHgR1wogqvFfVVup22pGjIvtVDwwr8kUC+6Tc0ZRHvODF+aJWugZzR+4gViTw0xgRV2ywy136EYItN7pjhPSg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734167065; c=relaxed/simple; bh=paAQ4zb8a4c91nhEyfBrirgdMm3St3PMZUQmMh7Rqn4=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=gClvo68Tz+9E9BMELvZMHpMbhFQ/MjgO5M7McbMApc5/zKF2/eghUzTaKJvdnWPr8P7MlezBHi5q0znI3pdAHBuANqjXr8FYpfl6qPRPKvurjOAz/Ftxi+u4O76pIk8xLXXaHpnFoYdI15EcRRtDjPrMguEWK4Y1qaJG1rQxliQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=a0z1+Clm; arc=none smtp.client-ip=209.85.210.170 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="a0z1+Clm" Received: by mail-pf1-f170.google.com with SMTP id d2e1a72fcca58-725dc290c00so2908359b3a.0 for ; Sat, 14 Dec 2024 01:04:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1734167063; x=1734771863; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=B33p7H15CJn/2cs/WET0YT0xzjvXDMsJMJEC97/vu9s=; b=a0z1+Clmv/YChwoop4IGkv+cdj7bBUgnihNWBJQula24KQMZAfrWix1IodVRRDEwiL S6OP/JcSmMYdORdOn4njvl0gXYRxKwW3i8TTK6EWnCQxwxNrswkOpmGHf2WA51qIj4fg 7TJh6L3h9OMDzJyLT0FgyV8+f2ps/5X3uHvDr+4ZN2wq1WmAp+iGvQEn5TkvJOPBFtYf GymZ7HZ6YJNkMp1SW2mH+NKXU1TMJkQq2Z1EE7DSmGSesum3jaFlPEilBqvCRj9sp5PP PJ46tkZINEgHEvFcXH05jFjbHUvPtqJLiDhdAIr3efBZsSg3QvWKqpkiYoARncMLiIAP hNog== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734167063; x=1734771863; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=B33p7H15CJn/2cs/WET0YT0xzjvXDMsJMJEC97/vu9s=; b=HQMh4ilz1TitJ9ubg09aAYo0KNouny3N/EJwuuhIbHyxeXV635YVmtN3EEh/I1D+U4 ljihlvGkfjBSWk8LXItnVLaOjFA6gCAkQeVb2fASsOvjsyGSkTiOmQ4vDbJX/6qWTvgJ lzcgFegOjSSlT3jCemd423VtwQ1cPtYf9uU7uVyBZOgEC4MGqidRdfWS11knLXtl5yoA U64fpK14sdexro2VwuE5PJDHizC/OHx7s8dGN3N/Y+j91Cm60q2sQtwkbGDev4vaSEYd uHLG2LzBj6enp+2ufWFKa7Q2W7V2pL6c5HhSpqA9T5ISuJFJVqHf4iC5n71PvfQ7vYFX UQ+Q== X-Forwarded-Encrypted: i=1; AJvYcCXbBukmIQMTa+Gt2CiZKGg+GG8ei6y2GZI43DKZ3FUrCRSTqZHX/2yGxahV84IOKfdtQG9X1SmHDqAM0AU=@vger.kernel.org X-Gm-Message-State: AOJu0YylzjJKdzNSXD+Q8rmjw+RMhxUgq/hb3UluvBF0Ws2zLz5ViYTY kyj9qFEBQvWwf5cyuZCQN8e3mTbTEuVLQVh5b3z0M+DBV5Ei8M3EEComfSCG0aM= X-Gm-Gg: ASbGncv+C8y1F7sGRLgN2LMxvokZbP+M83huc8fFfDetVhl7mAF1dSFmcqJJDVPh41V RNLB7+Wea+ZSXT3Au2ava7b0KKyzI541cXRceOtPy3knAnzzfaTepva9bw01HVvhXfj8OhbhJjE SwGWnPhmfGWVZnASBdAqr5tZoaC+78PyfJRxORiNNByozWkSs3/PxfXue1fsIBtteb5SuwGRbHU M0VlxEqzQedY7JPWZZiNezRjfyVQhU5410rFW3jvKGHFaFhsBNTcd8s2cF+JmbR/3pCg7V1v1Au 803psf0GUvnvmPpugluOJa+mMeXcng== X-Google-Smtp-Source: AGHT+IGHGsg1Le1TWBXvKS/11BgmC8ZOwqij0qU2dY3aJLeleELqBt0qnn6Oaqqw31tJpFU41bENng== X-Received: by 2002:a05:6a20:d80f:b0:1e1:b220:63db with SMTP id adf61e73a8af0-1e1e00879d9mr8179608637.16.1734167062929; Sat, 14 Dec 2024 01:04:22 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([2409:8a28:f4f:a9a4:5d46:4ef0:daa9:1db8]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-801d5a90ba0sm905731a12.16.2024.12.14.01.04.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 14 Dec 2024 01:04:21 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng Subject: [PATCH 05/12] mm: pgtable: introduce pagetable_dtor() Date: Sat, 14 Dec 2024 17:02:51 +0800 Message-Id: <7b17aa89b669d715b48e94ead1717c20b805074f.1734164094.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The pagetable_p*_dtor() are exactly the same except for the handling of ptlock. If we make ptlock_free() handle the case where ptdesc->ptl is NULL and remove VM_BUG_ON_PAGE() from pmd_ptlock_free(), we can unify pagetable_p*_dtor() into one function. Let's introduce pagetable_dtor() to do this. Later, pagetable_dtor() will be moved to tlb_remove_ptdesc(), so that ptlock and page table pages can be freed together (regardless of whether RCU is used). This prevents the use-after-free problem where the ptlock is freed immediately but the page table pages is freed later via RCU. Signed-off-by: Qi Zheng Suggested-by: Peter Zijlstra (Intel) --- Documentation/mm/split_page_table_lock.rst | 4 +- arch/arm/include/asm/tlb.h | 4 +- arch/arm64/include/asm/tlb.h | 8 ++-- arch/csky/include/asm/pgalloc.h | 2 +- arch/hexagon/include/asm/pgalloc.h | 2 +- arch/loongarch/include/asm/pgalloc.h | 2 +- arch/m68k/include/asm/mcf_pgalloc.h | 4 +- arch/m68k/include/asm/sun3_pgalloc.h | 2 +- arch/m68k/mm/motorola.c | 2 +- arch/mips/include/asm/pgalloc.h | 2 +- arch/nios2/include/asm/pgalloc.h | 2 +- arch/openrisc/include/asm/pgalloc.h | 2 +- arch/powerpc/mm/book3s64/mmu_context.c | 2 +- arch/powerpc/mm/book3s64/pgtable.c | 2 +- arch/powerpc/mm/pgtable-frag.c | 4 +- arch/riscv/include/asm/pgalloc.h | 8 ++-- arch/riscv/mm/init.c | 4 +- arch/s390/include/asm/pgalloc.h | 6 +-- arch/s390/include/asm/tlb.h | 6 +-- arch/s390/mm/pgalloc.c | 2 +- arch/sh/include/asm/pgalloc.h | 2 +- arch/sparc/mm/init_64.c | 2 +- arch/sparc/mm/srmmu.c | 2 +- arch/um/include/asm/pgalloc.h | 6 +-- arch/x86/mm/pgtable.c | 12 ++--- include/asm-generic/pgalloc.h | 8 ++-- include/linux/mm.h | 52 ++++------------------ mm/memory.c | 3 +- 28 files changed, 62 insertions(+), 95 deletions(-) diff --git a/Documentation/mm/split_page_table_lock.rst b/Documentation/mm/= split_page_table_lock.rst index 581446d4a4eba..8e1ceb0a6619a 100644 --- a/Documentation/mm/split_page_table_lock.rst +++ b/Documentation/mm/split_page_table_lock.rst @@ -62,7 +62,7 @@ Support of split page table lock by an architecture =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D =20 There's no need in special enabling of PTE split page table lock: everythi= ng -required is done by pagetable_pte_ctor() and pagetable_pte_dtor(), which +required is done by pagetable_pte_ctor() and pagetable_dtor(), which must be called on PTE table allocation / freeing. =20 Make sure the architecture doesn't use slab allocator for page table @@ -73,7 +73,7 @@ PMD split lock only makes sense if you have more than two= page table levels. =20 PMD split lock enabling requires pagetable_pmd_ctor() call on PMD table -allocation and pagetable_pmd_dtor() on freeing. +allocation and pagetable_dtor() on freeing. =20 Allocation usually happens in pmd_alloc_one(), freeing in pmd_free() and pmd_free_tlb(), but make sure you cover all PMD table allocation / freeing diff --git a/arch/arm/include/asm/tlb.h b/arch/arm/include/asm/tlb.h index f40d06ad5d2a3..ef79bf1e8563f 100644 --- a/arch/arm/include/asm/tlb.h +++ b/arch/arm/include/asm/tlb.h @@ -41,7 +41,7 @@ __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pte, uns= igned long addr) { struct ptdesc *ptdesc =3D page_ptdesc(pte); =20 - pagetable_pte_dtor(ptdesc); + pagetable_dtor(ptdesc); =20 #ifndef CONFIG_ARM_LPAE /* @@ -61,7 +61,7 @@ __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmdp, unsig= ned long addr) #ifdef CONFIG_ARM_LPAE struct ptdesc *ptdesc =3D virt_to_ptdesc(pmdp); =20 - pagetable_pmd_dtor(ptdesc); + pagetable_dtor(ptdesc); tlb_remove_ptdesc(tlb, ptdesc); #endif } diff --git a/arch/arm64/include/asm/tlb.h b/arch/arm64/include/asm/tlb.h index 445282cde9afb..408d0f36a8a8f 100644 --- a/arch/arm64/include/asm/tlb.h +++ b/arch/arm64/include/asm/tlb.h @@ -82,7 +82,7 @@ static inline void __pte_free_tlb(struct mmu_gather *tlb,= pgtable_t pte, { struct ptdesc *ptdesc =3D page_ptdesc(pte); =20 - pagetable_pte_dtor(ptdesc); + pagetable_dtor(ptdesc); tlb_remove_ptdesc(tlb, ptdesc); } =20 @@ -92,7 +92,7 @@ static inline void __pmd_free_tlb(struct mmu_gather *tlb,= pmd_t *pmdp, { struct ptdesc *ptdesc =3D virt_to_ptdesc(pmdp); =20 - pagetable_pmd_dtor(ptdesc); + pagetable_dtor(ptdesc); tlb_remove_ptdesc(tlb, ptdesc); } #endif @@ -106,7 +106,7 @@ static inline void __pud_free_tlb(struct mmu_gather *tl= b, pud_t *pudp, if (!pgtable_l4_enabled()) return; =20 - pagetable_pud_dtor(ptdesc); + pagetable_dtor(ptdesc); tlb_remove_ptdesc(tlb, ptdesc); } #endif @@ -120,7 +120,7 @@ static inline void __p4d_free_tlb(struct mmu_gather *tl= b, p4d_t *p4dp, if (!pgtable_l5_enabled()) return; =20 - pagetable_p4d_dtor(ptdesc); + pagetable_dtor(ptdesc); tlb_remove_ptdesc(tlb, ptdesc); } #endif diff --git a/arch/csky/include/asm/pgalloc.h b/arch/csky/include/asm/pgallo= c.h index 9c84c9012e534..f1ce5b7b28f22 100644 --- a/arch/csky/include/asm/pgalloc.h +++ b/arch/csky/include/asm/pgalloc.h @@ -63,7 +63,7 @@ static inline pgd_t *pgd_alloc(struct mm_struct *mm) =20 #define __pte_free_tlb(tlb, pte, address) \ do { \ - pagetable_pte_dtor(page_ptdesc(pte)); \ + pagetable_dtor(page_ptdesc(pte)); \ tlb_remove_page_ptdesc(tlb, page_ptdesc(pte)); \ } while (0) =20 diff --git a/arch/hexagon/include/asm/pgalloc.h b/arch/hexagon/include/asm/= pgalloc.h index 55988625e6fbc..40e42a0e71673 100644 --- a/arch/hexagon/include/asm/pgalloc.h +++ b/arch/hexagon/include/asm/pgalloc.h @@ -89,7 +89,7 @@ static inline void pmd_populate_kernel(struct mm_struct *= mm, pmd_t *pmd, =20 #define __pte_free_tlb(tlb, pte, addr) \ do { \ - pagetable_pte_dtor((page_ptdesc(pte))); \ + pagetable_dtor((page_ptdesc(pte))); \ tlb_remove_page_ptdesc((tlb), (page_ptdesc(pte))); \ } while (0) =20 diff --git a/arch/loongarch/include/asm/pgalloc.h b/arch/loongarch/include/= asm/pgalloc.h index a7b9c9e73593d..7211dff8c969e 100644 --- a/arch/loongarch/include/asm/pgalloc.h +++ b/arch/loongarch/include/asm/pgalloc.h @@ -57,7 +57,7 @@ static inline pte_t *pte_alloc_one_kernel(struct mm_struc= t *mm) =20 #define __pte_free_tlb(tlb, pte, address) \ do { \ - pagetable_pte_dtor(page_ptdesc(pte)); \ + pagetable_dtor(page_ptdesc(pte)); \ tlb_remove_page_ptdesc((tlb), page_ptdesc(pte)); \ } while (0) =20 diff --git a/arch/m68k/include/asm/mcf_pgalloc.h b/arch/m68k/include/asm/mc= f_pgalloc.h index 302c5bf67179e..22d6c1fcabfb4 100644 --- a/arch/m68k/include/asm/mcf_pgalloc.h +++ b/arch/m68k/include/asm/mcf_pgalloc.h @@ -37,7 +37,7 @@ static inline void __pte_free_tlb(struct mmu_gather *tlb,= pgtable_t pgtable, { struct ptdesc *ptdesc =3D virt_to_ptdesc(pgtable); =20 - pagetable_pte_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); } =20 @@ -61,7 +61,7 @@ static inline void pte_free(struct mm_struct *mm, pgtable= _t pgtable) { struct ptdesc *ptdesc =3D virt_to_ptdesc(pgtable); =20 - pagetable_pte_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); } =20 diff --git a/arch/m68k/include/asm/sun3_pgalloc.h b/arch/m68k/include/asm/s= un3_pgalloc.h index 4a137eecb6fe4..2b626cb3ad0ae 100644 --- a/arch/m68k/include/asm/sun3_pgalloc.h +++ b/arch/m68k/include/asm/sun3_pgalloc.h @@ -19,7 +19,7 @@ extern const char bad_pmd_string[]; =20 #define __pte_free_tlb(tlb, pte, addr) \ do { \ - pagetable_pte_dtor(page_ptdesc(pte)); \ + pagetable_dtor(page_ptdesc(pte)); \ tlb_remove_page_ptdesc((tlb), page_ptdesc(pte)); \ } while (0) =20 diff --git a/arch/m68k/mm/motorola.c b/arch/m68k/mm/motorola.c index c1761d309fc61..81715cece70c6 100644 --- a/arch/m68k/mm/motorola.c +++ b/arch/m68k/mm/motorola.c @@ -201,7 +201,7 @@ int free_pointer_table(void *table, int type) list_del(dp); mmu_page_dtor((void *)page); if (type =3D=3D TABLE_PTE) - pagetable_pte_dtor(virt_to_ptdesc((void *)page)); + pagetable_dtor(virt_to_ptdesc((void *)page)); free_page (page); return 1; } else if (ptable_list[type].next !=3D dp) { diff --git a/arch/mips/include/asm/pgalloc.h b/arch/mips/include/asm/pgallo= c.h index f4440edcd8fe2..36d9805033c4b 100644 --- a/arch/mips/include/asm/pgalloc.h +++ b/arch/mips/include/asm/pgalloc.h @@ -56,7 +56,7 @@ static inline void pgd_free(struct mm_struct *mm, pgd_t *= pgd) =20 #define __pte_free_tlb(tlb, pte, address) \ do { \ - pagetable_pte_dtor(page_ptdesc(pte)); \ + pagetable_dtor(page_ptdesc(pte)); \ tlb_remove_page_ptdesc((tlb), page_ptdesc(pte)); \ } while (0) =20 diff --git a/arch/nios2/include/asm/pgalloc.h b/arch/nios2/include/asm/pgal= loc.h index ce6bb8e74271f..12a536b7bfbd4 100644 --- a/arch/nios2/include/asm/pgalloc.h +++ b/arch/nios2/include/asm/pgalloc.h @@ -30,7 +30,7 @@ extern pgd_t *pgd_alloc(struct mm_struct *mm); =20 #define __pte_free_tlb(tlb, pte, addr) \ do { \ - pagetable_pte_dtor(page_ptdesc(pte)); \ + pagetable_dtor(page_ptdesc(pte)); \ tlb_remove_page_ptdesc((tlb), (page_ptdesc(pte))); \ } while (0) =20 diff --git a/arch/openrisc/include/asm/pgalloc.h b/arch/openrisc/include/as= m/pgalloc.h index c6a73772a5466..596e2355824e3 100644 --- a/arch/openrisc/include/asm/pgalloc.h +++ b/arch/openrisc/include/asm/pgalloc.h @@ -68,7 +68,7 @@ extern pte_t *pte_alloc_one_kernel(struct mm_struct *mm); =20 #define __pte_free_tlb(tlb, pte, addr) \ do { \ - pagetable_pte_dtor(page_ptdesc(pte)); \ + pagetable_dtor(page_ptdesc(pte)); \ tlb_remove_page_ptdesc((tlb), (page_ptdesc(pte))); \ } while (0) =20 diff --git a/arch/powerpc/mm/book3s64/mmu_context.c b/arch/powerpc/mm/book3= s64/mmu_context.c index 1715b07c630c9..4e1e45420bd49 100644 --- a/arch/powerpc/mm/book3s64/mmu_context.c +++ b/arch/powerpc/mm/book3s64/mmu_context.c @@ -253,7 +253,7 @@ static void pmd_frag_destroy(void *pmd_frag) count =3D ((unsigned long)pmd_frag & ~PAGE_MASK) >> PMD_FRAG_SIZE_SHIFT; /* We allow PTE_FRAG_NR fragments from a PTE page */ if (atomic_sub_and_test(PMD_FRAG_NR - count, &ptdesc->pt_frag_refcount)) { - pagetable_pmd_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); } } diff --git a/arch/powerpc/mm/book3s64/pgtable.c b/arch/powerpc/mm/book3s64/= pgtable.c index 3745425280808..3f28e4acd920b 100644 --- a/arch/powerpc/mm/book3s64/pgtable.c +++ b/arch/powerpc/mm/book3s64/pgtable.c @@ -477,7 +477,7 @@ void pmd_fragment_free(unsigned long *pmd) =20 BUG_ON(atomic_read(&ptdesc->pt_frag_refcount) <=3D 0); if (atomic_dec_and_test(&ptdesc->pt_frag_refcount)) { - pagetable_pmd_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); } } diff --git a/arch/powerpc/mm/pgtable-frag.c b/arch/powerpc/mm/pgtable-frag.c index e89f64a0f24ae..713268ccb1a0e 100644 --- a/arch/powerpc/mm/pgtable-frag.c +++ b/arch/powerpc/mm/pgtable-frag.c @@ -25,7 +25,7 @@ void pte_frag_destroy(void *pte_frag) count =3D ((unsigned long)pte_frag & ~PAGE_MASK) >> PTE_FRAG_SIZE_SHIFT; /* We allow PTE_FRAG_NR fragments from a PTE page */ if (atomic_sub_and_test(PTE_FRAG_NR - count, &ptdesc->pt_frag_refcount)) { - pagetable_pte_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); } } @@ -111,7 +111,7 @@ static void pte_free_now(struct rcu_head *head) struct ptdesc *ptdesc; =20 ptdesc =3D container_of(head, struct ptdesc, pt_rcu_head); - pagetable_pte_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); } =20 diff --git a/arch/riscv/include/asm/pgalloc.h b/arch/riscv/include/asm/pgal= loc.h index bb6e1c5f1fb19..09c2eff571a49 100644 --- a/arch/riscv/include/asm/pgalloc.h +++ b/arch/riscv/include/asm/pgalloc.h @@ -112,7 +112,7 @@ static inline void __pud_free_tlb(struct mmu_gather *tl= b, pud_t *pud, if (pgtable_l4_enabled) { struct ptdesc *ptdesc =3D virt_to_ptdesc(pud); =20 - pagetable_pud_dtor(ptdesc); + pagetable_dtor(ptdesc); riscv_tlb_remove_ptdesc(tlb, ptdesc); } } @@ -139,7 +139,7 @@ static inline void __p4d_free_tlb(struct mmu_gather *tl= b, p4d_t *p4d, if (pgtable_l5_enabled) { struct ptdesc *ptdesc =3D virt_to_ptdesc(p4d); =20 - pagetable_p4d_dtor(ptdesc); + pagetable_dtor(ptdesc); riscv_tlb_remove_ptdesc(tlb, virt_to_ptdesc(p4d)); } } @@ -172,7 +172,7 @@ static inline void __pmd_free_tlb(struct mmu_gather *tl= b, pmd_t *pmd, { struct ptdesc *ptdesc =3D virt_to_ptdesc(pmd); =20 - pagetable_pmd_dtor(ptdesc); + pagetable_dtor(ptdesc); riscv_tlb_remove_ptdesc(tlb, ptdesc); } =20 @@ -183,7 +183,7 @@ static inline void __pte_free_tlb(struct mmu_gather *tl= b, pgtable_t pte, { struct ptdesc *ptdesc =3D page_ptdesc(pte); =20 - pagetable_pte_dtor(ptdesc); + pagetable_dtor(ptdesc); riscv_tlb_remove_ptdesc(tlb, ptdesc); } #endif /* CONFIG_MMU */ diff --git a/arch/riscv/mm/init.c b/arch/riscv/mm/init.c index fc53ce748c804..8d703fb51b1dc 100644 --- a/arch/riscv/mm/init.c +++ b/arch/riscv/mm/init.c @@ -1558,7 +1558,7 @@ static void __meminit free_pte_table(pte_t *pte_start= , pmd_t *pmd) return; } =20 - pagetable_pte_dtor(ptdesc); + pagetable_dtor(ptdesc); if (PageReserved(page)) free_reserved_page(page); else @@ -1580,7 +1580,7 @@ static void __meminit free_pmd_table(pmd_t *pmd_start= , pud_t *pud, bool is_vmemm } =20 if (!is_vmemmap) - pagetable_pmd_dtor(ptdesc); + pagetable_dtor(ptdesc); if (PageReserved(page)) free_reserved_page(page); else diff --git a/arch/s390/include/asm/pgalloc.h b/arch/s390/include/asm/pgallo= c.h index a0c1ca5d8423c..5fced6d3c36b0 100644 --- a/arch/s390/include/asm/pgalloc.h +++ b/arch/s390/include/asm/pgalloc.h @@ -66,7 +66,7 @@ static inline void p4d_free(struct mm_struct *mm, p4d_t *= p4d) if (mm_p4d_folded(mm)) return; =20 - pagetable_p4d_dtor(virt_to_ptdesc(p4d)); + pagetable_dtor(virt_to_ptdesc(p4d)); crst_table_free(mm, (unsigned long *) p4d); } =20 @@ -87,7 +87,7 @@ static inline void pud_free(struct mm_struct *mm, pud_t *= pud) if (mm_pud_folded(mm)) return; =20 - pagetable_pud_dtor(virt_to_ptdesc(pud)); + pagetable_dtor(virt_to_ptdesc(pud)); crst_table_free(mm, (unsigned long *) pud); } =20 @@ -109,7 +109,7 @@ static inline void pmd_free(struct mm_struct *mm, pmd_t= *pmd) { if (mm_pmd_folded(mm)) return; - pagetable_pmd_dtor(virt_to_ptdesc(pmd)); + pagetable_dtor(virt_to_ptdesc(pmd)); crst_table_free(mm, (unsigned long *) pmd); } =20 diff --git a/arch/s390/include/asm/tlb.h b/arch/s390/include/asm/tlb.h index b946964afce8e..74b6fba4c2ee3 100644 --- a/arch/s390/include/asm/tlb.h +++ b/arch/s390/include/asm/tlb.h @@ -102,7 +102,7 @@ static inline void pmd_free_tlb(struct mmu_gather *tlb,= pmd_t *pmd, { if (mm_pmd_folded(tlb->mm)) return; - pagetable_pmd_dtor(virt_to_ptdesc(pmd)); + pagetable_dtor(virt_to_ptdesc(pmd)); __tlb_adjust_range(tlb, address, PAGE_SIZE); tlb->mm->context.flush_mm =3D 1; tlb->freed_tables =3D 1; @@ -122,7 +122,7 @@ static inline void pud_free_tlb(struct mmu_gather *tlb,= pud_t *pud, { if (mm_pud_folded(tlb->mm)) return; - pagetable_pud_dtor(virt_to_ptdesc(pud)); + pagetable_dtor(virt_to_ptdesc(pud)); tlb->mm->context.flush_mm =3D 1; tlb->freed_tables =3D 1; tlb->cleared_p4ds =3D 1; @@ -141,7 +141,7 @@ static inline void p4d_free_tlb(struct mmu_gather *tlb,= p4d_t *p4d, { if (mm_p4d_folded(tlb->mm)) return; - pagetable_p4d_dtor(virt_to_ptdesc(p4d)); + pagetable_dtor(virt_to_ptdesc(p4d)); __tlb_adjust_range(tlb, address, PAGE_SIZE); tlb->mm->context.flush_mm =3D 1; tlb->freed_tables =3D 1; diff --git a/arch/s390/mm/pgalloc.c b/arch/s390/mm/pgalloc.c index 58696a0c4e4ac..569de24d33761 100644 --- a/arch/s390/mm/pgalloc.c +++ b/arch/s390/mm/pgalloc.c @@ -182,7 +182,7 @@ unsigned long *page_table_alloc(struct mm_struct *mm) =20 static void pagetable_pte_dtor_free(struct ptdesc *ptdesc) { - pagetable_pte_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); } =20 diff --git a/arch/sh/include/asm/pgalloc.h b/arch/sh/include/asm/pgalloc.h index 5d8577ab15911..96d938fdf2244 100644 --- a/arch/sh/include/asm/pgalloc.h +++ b/arch/sh/include/asm/pgalloc.h @@ -34,7 +34,7 @@ static inline void pmd_populate(struct mm_struct *mm, pmd= _t *pmd, =20 #define __pte_free_tlb(tlb, pte, addr) \ do { \ - pagetable_pte_dtor(page_ptdesc(pte)); \ + pagetable_dtor(page_ptdesc(pte)); \ tlb_remove_page_ptdesc((tlb), (page_ptdesc(pte))); \ } while (0) =20 diff --git a/arch/sparc/mm/init_64.c b/arch/sparc/mm/init_64.c index 21f8cbbd0581c..05882bca5b732 100644 --- a/arch/sparc/mm/init_64.c +++ b/arch/sparc/mm/init_64.c @@ -2915,7 +2915,7 @@ static void __pte_free(pgtable_t pte) { struct ptdesc *ptdesc =3D virt_to_ptdesc(pte); =20 - pagetable_pte_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); } =20 diff --git a/arch/sparc/mm/srmmu.c b/arch/sparc/mm/srmmu.c index 9df51a62333d6..e3a72c884b867 100644 --- a/arch/sparc/mm/srmmu.c +++ b/arch/sparc/mm/srmmu.c @@ -372,7 +372,7 @@ void pte_free(struct mm_struct *mm, pgtable_t ptep) page =3D pfn_to_page(__nocache_pa((unsigned long)ptep) >> PAGE_SHIFT); spin_lock(&mm->page_table_lock); if (page_ref_dec_return(page) =3D=3D 1) - pagetable_pte_dtor(page_ptdesc(page)); + pagetable_dtor(page_ptdesc(page)); spin_unlock(&mm->page_table_lock); =20 srmmu_free_nocache(ptep, SRMMU_PTE_TABLE_SIZE); diff --git a/arch/um/include/asm/pgalloc.h b/arch/um/include/asm/pgalloc.h index 04fb4e6969a46..f0af23c3aeb2b 100644 --- a/arch/um/include/asm/pgalloc.h +++ b/arch/um/include/asm/pgalloc.h @@ -27,7 +27,7 @@ extern pgd_t *pgd_alloc(struct mm_struct *); =20 #define __pte_free_tlb(tlb, pte, address) \ do { \ - pagetable_pte_dtor(page_ptdesc(pte)); \ + pagetable_dtor(page_ptdesc(pte)); \ tlb_remove_page_ptdesc((tlb), (page_ptdesc(pte))); \ } while (0) =20 @@ -35,7 +35,7 @@ do { \ =20 #define __pmd_free_tlb(tlb, pmd, address) \ do { \ - pagetable_pmd_dtor(virt_to_ptdesc(pmd)); \ + pagetable_dtor(virt_to_ptdesc(pmd)); \ tlb_remove_page_ptdesc((tlb), virt_to_ptdesc(pmd)); \ } while (0) =20 @@ -43,7 +43,7 @@ do { \ =20 #define __pud_free_tlb(tlb, pud, address) \ do { \ - pagetable_pud_dtor(virt_to_ptdesc(pud)); \ + pagetable_dtor(virt_to_ptdesc(pud)); \ tlb_remove_page_ptdesc((tlb), virt_to_ptdesc(pud)); \ } while (0) =20 diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c index 3d6e84da45b24..a6cd9660e29ec 100644 --- a/arch/x86/mm/pgtable.c +++ b/arch/x86/mm/pgtable.c @@ -60,7 +60,7 @@ early_param("userpte", setup_userpte); =20 void ___pte_free_tlb(struct mmu_gather *tlb, struct page *pte) { - pagetable_pte_dtor(page_ptdesc(pte)); + pagetable_dtor(page_ptdesc(pte)); paravirt_release_pte(page_to_pfn(pte)); paravirt_tlb_remove_table(tlb, pte); } @@ -77,7 +77,7 @@ void ___pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd) #ifdef CONFIG_X86_PAE tlb->need_flush_all =3D 1; #endif - pagetable_pmd_dtor(ptdesc); + pagetable_dtor(ptdesc); paravirt_tlb_remove_table(tlb, ptdesc_page(ptdesc)); } =20 @@ -86,7 +86,7 @@ void ___pud_free_tlb(struct mmu_gather *tlb, pud_t *pud) { struct ptdesc *ptdesc =3D virt_to_ptdesc(pud); =20 - pagetable_pud_dtor(ptdesc); + pagetable_dtor(ptdesc); paravirt_release_pud(__pa(pud) >> PAGE_SHIFT); paravirt_tlb_remove_table(tlb, virt_to_page(pud)); } @@ -96,7 +96,7 @@ void ___p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d) { struct ptdesc *ptdesc =3D virt_to_ptdesc(p4d); =20 - pagetable_p4d_dtor(ptdesc); + pagetable_dtor(ptdesc); paravirt_release_p4d(__pa(p4d) >> PAGE_SHIFT); paravirt_tlb_remove_table(tlb, virt_to_page(p4d)); } @@ -233,7 +233,7 @@ static void free_pmds(struct mm_struct *mm, pmd_t *pmds= [], int count) if (pmds[i]) { ptdesc =3D virt_to_ptdesc(pmds[i]); =20 - pagetable_pmd_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); mm_dec_nr_pmds(mm); } @@ -867,7 +867,7 @@ int pud_free_pmd_page(pud_t *pud, unsigned long addr) =20 free_page((unsigned long)pmd_sv); =20 - pagetable_pmd_dtor(virt_to_ptdesc(pmd)); + pagetable_dtor(virt_to_ptdesc(pmd)); free_page((unsigned long)pmd); =20 return 1; diff --git a/include/asm-generic/pgalloc.h b/include/asm-generic/pgalloc.h index dbf61819b3581..3673e9c29504e 100644 --- a/include/asm-generic/pgalloc.h +++ b/include/asm-generic/pgalloc.h @@ -109,7 +109,7 @@ static inline void pte_free(struct mm_struct *mm, struc= t page *pte_page) { struct ptdesc *ptdesc =3D page_ptdesc(pte_page); =20 - pagetable_pte_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); } =20 @@ -153,7 +153,7 @@ static inline void pmd_free(struct mm_struct *mm, pmd_t= *pmd) struct ptdesc *ptdesc =3D virt_to_ptdesc(pmd); =20 BUG_ON((unsigned long)pmd & (PAGE_SIZE-1)); - pagetable_pmd_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); } #endif @@ -202,7 +202,7 @@ static inline void __pud_free(struct mm_struct *mm, pud= _t *pud) struct ptdesc *ptdesc =3D virt_to_ptdesc(pud); =20 BUG_ON((unsigned long)pud & (PAGE_SIZE-1)); - pagetable_pud_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); } =20 @@ -257,7 +257,7 @@ static inline void __p4d_free(struct mm_struct *mm, p4d= _t *p4d) struct ptdesc *ptdesc =3D virt_to_ptdesc(p4d); =20 BUG_ON((unsigned long)p4d & (PAGE_SIZE-1)); - pagetable_p4d_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); } =20 diff --git a/include/linux/mm.h b/include/linux/mm.h index 807a12ed8ec96..497035a78849b 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -3055,6 +3055,15 @@ static inline bool ptlock_init(struct ptdesc *ptdesc= ) { return true; } static inline void ptlock_free(struct ptdesc *ptdesc) {} #endif /* defined(CONFIG_SPLIT_PTE_PTLOCKS) */ =20 +static inline void pagetable_dtor(struct ptdesc *ptdesc) +{ + struct folio *folio =3D ptdesc_folio(ptdesc); + + ptlock_free(ptdesc); + __folio_clear_pgtable(folio); + lruvec_stat_sub_folio(folio, NR_PAGETABLE); +} + static inline bool pagetable_pte_ctor(struct ptdesc *ptdesc) { struct folio *folio =3D ptdesc_folio(ptdesc); @@ -3066,15 +3075,6 @@ static inline bool pagetable_pte_ctor(struct ptdesc = *ptdesc) return true; } =20 -static inline void pagetable_pte_dtor(struct ptdesc *ptdesc) -{ - struct folio *folio =3D ptdesc_folio(ptdesc); - - ptlock_free(ptdesc); - __folio_clear_pgtable(folio); - lruvec_stat_sub_folio(folio, NR_PAGETABLE); -} - pte_t *___pte_offset_map(pmd_t *pmd, unsigned long addr, pmd_t *pmdvalp); static inline pte_t *__pte_offset_map(pmd_t *pmd, unsigned long addr, pmd_t *pmdvalp) @@ -3151,14 +3151,6 @@ static inline bool pmd_ptlock_init(struct ptdesc *pt= desc) return ptlock_init(ptdesc); } =20 -static inline void pmd_ptlock_free(struct ptdesc *ptdesc) -{ -#ifdef CONFIG_TRANSPARENT_HUGEPAGE - VM_BUG_ON_PAGE(ptdesc->pmd_huge_pte, ptdesc_page(ptdesc)); -#endif - ptlock_free(ptdesc); -} - #define pmd_huge_pte(mm, pmd) (pmd_ptdesc(pmd)->pmd_huge_pte) =20 #else @@ -3169,7 +3161,6 @@ static inline spinlock_t *pmd_lockptr(struct mm_struc= t *mm, pmd_t *pmd) } =20 static inline bool pmd_ptlock_init(struct ptdesc *ptdesc) { return true; } -static inline void pmd_ptlock_free(struct ptdesc *ptdesc) {} =20 #define pmd_huge_pte(mm, pmd) ((mm)->pmd_huge_pte) =20 @@ -3193,15 +3184,6 @@ static inline bool pagetable_pmd_ctor(struct ptdesc = *ptdesc) return true; } =20 -static inline void pagetable_pmd_dtor(struct ptdesc *ptdesc) -{ - struct folio *folio =3D ptdesc_folio(ptdesc); - - pmd_ptlock_free(ptdesc); - __folio_clear_pgtable(folio); - lruvec_stat_sub_folio(folio, NR_PAGETABLE); -} - /* * No scalability reason to split PUD locks yet, but follow the same patte= rn * as the PMD locks to make it easier if we decide to. The VM should not = be @@ -3229,14 +3211,6 @@ static inline void pagetable_pud_ctor(struct ptdesc = *ptdesc) lruvec_stat_add_folio(folio, NR_PAGETABLE); } =20 -static inline void pagetable_pud_dtor(struct ptdesc *ptdesc) -{ - struct folio *folio =3D ptdesc_folio(ptdesc); - - __folio_clear_pgtable(folio); - lruvec_stat_sub_folio(folio, NR_PAGETABLE); -} - static inline void pagetable_p4d_ctor(struct ptdesc *ptdesc) { struct folio *folio =3D ptdesc_folio(ptdesc); @@ -3245,14 +3219,6 @@ static inline void pagetable_p4d_ctor(struct ptdesc = *ptdesc) lruvec_stat_add_folio(folio, NR_PAGETABLE); } =20 -static inline void pagetable_p4d_dtor(struct ptdesc *ptdesc) -{ - struct folio *folio =3D ptdesc_folio(ptdesc); - - __folio_clear_pgtable(folio); - lruvec_stat_sub_folio(folio, NR_PAGETABLE); -} - extern void __init pagecache_init(void); extern void free_initmem(void); =20 diff --git a/mm/memory.c b/mm/memory.c index 83765632e20b0..c00fa95464b19 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -7058,7 +7058,8 @@ bool ptlock_alloc(struct ptdesc *ptdesc) =20 void ptlock_free(struct ptdesc *ptdesc) { - kmem_cache_free(page_ptl_cachep, ptdesc->ptl); + if (ptdesc->ptl) + kmem_cache_free(page_ptl_cachep, ptdesc->ptl); } #endif =20 --=20 2.20.1 From nobody Wed Dec 17 15:51:21 2025 Received: from mail-pf1-f181.google.com (mail-pf1-f181.google.com [209.85.210.181]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 59F1818E379 for ; Sat, 14 Dec 2024 09:04:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.181 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734167073; cv=none; b=jQ84mp0trSlZcnpx2U0CMSl15qZg1zS8hEbLuSwSTh05rdmc+4LIYAh5t0H1ggY7WVtvXFpNm/QJd9T/cW1FHBqB7iSo53TvEXRTxW/V7a0J92MTeNMyDqAM16aGRqYtKKMDA61SPEHqVgELbpWbY8LkxPev4wjpbdG6Nrty1M4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734167073; c=relaxed/simple; bh=2rLUjE/bBcdnW4BziWlyu8gvQ7w1P03MQD9OH0V2m1g=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=OB/cM9HBqJA010JUAJDjcSrRnBel536nv/rWAW2iGgqT36GjpjdETinWhlM3erpxeVSee89IKxuUPiMtYEthiVbep80RIw41Um3cLqn6NOs1aDXlOi2WkJeudtfCx/4oYtHjsymY3YBKK9zWklj7GvfIbRIekVIMpQhsa+0IgPY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=H2zFPTmS; arc=none smtp.client-ip=209.85.210.181 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="H2zFPTmS" Received: by mail-pf1-f181.google.com with SMTP id d2e1a72fcca58-728e1799d95so3058970b3a.2 for ; Sat, 14 Dec 2024 01:04:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1734167072; x=1734771872; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=12IiSMzMVIlfwdGc3SnaXSwRS1fKZAvHGusKFMV1b3k=; b=H2zFPTmSl4MeudeJgucox0hf3rswkD6kdoVvT4ke0Nbghtgbujsb54Ieyy5bPn2ri7 obes7A86jaS0dEn4RdD/MXrTtJdngzLxbklSqzY0GL6eYNcKD6E5489GQNZBN+P/FsFq WGClcmiBgFCDdlIxMiYm80DbDPvUaVMe9vdzHD5mRuA8Sbha8Jrbmjjh+3ugM8o1hDX0 oKf7J5hN3ZA0pcQgDzTvoBXIE539xdU3L/KHs6zK7e05b/0lckgd6IYDHYz89DFDDpt8 sjg1wP9meux4twRIScQcxXCLm7CqeQ6cl8Qlt+gIqcFAJl39DMAOcOyQeJZmMEghqaaW czkQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734167072; x=1734771872; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=12IiSMzMVIlfwdGc3SnaXSwRS1fKZAvHGusKFMV1b3k=; b=OB2KJC0Z9LYPduQShES5WKnBD79238srwrF1jyfBMF7tTNT6N8rXab4dAGg9XzlSa7 OmzdbtFj0gpuUkErFNigScp2e0QMkp/lPfvhm3aCnpuVMhlPxamNCMiPaVDh3cHy7yod OaUAbC5ikXcUBZhDkpE6hl5yZpzZY9gJq8FRB7s3twIec8Dx+PDhVNSLaQIwKn7VbRnY uYUZ6QrU0tlM9gzvpTVIxhFjXr4/dq+39Iv0SkM44gpG0EbxfxSeebMgki+IDd6mqlRt 7D2J6OQeNmSgRydblvjt1T2ReXUTJTY9TiubmPk3E6OX2BghGbqYaSt6BRzHCu/MtTmQ C6dA== X-Forwarded-Encrypted: i=1; AJvYcCUY2+9NCdDodBnsrZ32ARoyfElBCa1qJXo0znNJv6WFlosJXjt2cb5Oy5tOTG/8+UTjI3gedhkM3w3/jAo=@vger.kernel.org X-Gm-Message-State: AOJu0YwHk9io5PZwCL0cX/HwAqSBHfMM0HALHhyXdeFUTikeWiLs/e6e x8R48o4L56dw7T53Ih0gen9nJWEWa+TzGeyOJ3QxDwoBxfqES+qOk0YkgXDNmUk= X-Gm-Gg: ASbGncvu97MDeryXvDeVDaZvDgL2PZbALlIVO0TBAVYapf7MJMcrwluHByzWxgHN7Kt LxKAq4OdWemalJd6ZS2Nvudi3H54OHuAiSGz/wlMU0kRd9rd/SnlNrBfIk978G7ehrz9KzzwpP6 XUQI/03BF6K+8c82LaE0XYKIa2zM+gB8M71mPqmL250GetowJLpaydZzaqAR+Ypr354c6XcjYHZ qYVcD7d7ugbrVTUwTJ4qSkrp13mqQVVfValFMv3JfAS2fwXE+4uA2xM09cTi6fcfcSuW4xqzgVQ Z4dFDmKdvnFer53I0klnNp7mva9eMQ== X-Google-Smtp-Source: AGHT+IFvnXDbPrP66UsHi9s8CFtvVszYyEfjge+aAOFm/P/L2638kVYMzofXCSp9Xlrusy3CzYMSYA== X-Received: by 2002:a05:6a21:33a0:b0:1e0:d796:b079 with SMTP id adf61e73a8af0-1e1dfd55614mr8533024637.17.1734167071820; Sat, 14 Dec 2024 01:04:31 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([2409:8a28:f4f:a9a4:5d46:4ef0:daa9:1db8]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-801d5a90ba0sm905731a12.16.2024.12.14.01.04.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 14 Dec 2024 01:04:31 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng , linux-arm-kernel@lists.infradead.org Subject: [PATCH 06/12] arm: pgtable: move pagetable_dtor() to __tlb_remove_table() Date: Sat, 14 Dec 2024 17:02:52 +0800 Message-Id: <59fd7e0364e8dd293132d81b91b28694e70263e2.1734164094.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Move pagetable_dtor() to __tlb_remove_table(), so that ptlock and page table pages can be freed together (regardless of whether RCU is used). This prevents the use-after-free problem where the ptlock is freed immediately but the page table pages is freed later via RCU. Page tables shouldn't have swap cache, so use pagetable_free() instead of free_page_and_swap_cache() to free page table pages. Signed-off-by: Qi Zheng Cc: linux-arm-kernel@lists.infradead.org Suggested-by: Peter Zijlstra (Intel) --- arch/arm/include/asm/tlb.h | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/arch/arm/include/asm/tlb.h b/arch/arm/include/asm/tlb.h index ef79bf1e8563f..264ab635e807a 100644 --- a/arch/arm/include/asm/tlb.h +++ b/arch/arm/include/asm/tlb.h @@ -26,12 +26,14 @@ =20 #else /* !CONFIG_MMU */ =20 -#include #include =20 static inline void __tlb_remove_table(void *_table) { - free_page_and_swap_cache((struct page *)_table); + struct ptdesc *ptdesc =3D (struct ptdesc *)_table; + + pagetable_dtor(ptdesc); + pagetable_free(ptdesc); } =20 #include @@ -41,8 +43,6 @@ __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pte, uns= igned long addr) { struct ptdesc *ptdesc =3D page_ptdesc(pte); =20 - pagetable_dtor(ptdesc); - #ifndef CONFIG_ARM_LPAE /* * With the classic ARM MMU, a pte page has two corresponding pmd @@ -61,7 +61,6 @@ __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmdp, unsig= ned long addr) #ifdef CONFIG_ARM_LPAE struct ptdesc *ptdesc =3D virt_to_ptdesc(pmdp); =20 - pagetable_dtor(ptdesc); tlb_remove_ptdesc(tlb, ptdesc); #endif } --=20 2.20.1 From nobody Wed Dec 17 15:51:21 2025 Received: from mail-pl1-f182.google.com (mail-pl1-f182.google.com [209.85.214.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6973942A92 for ; Sat, 14 Dec 2024 09:04:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.182 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734167086; cv=none; b=n4zWft1itGKr6+l6X/KN0GGG5igWmLZDsUmMTP3EqpXEQ/FiaO8A2oylHCwcYI5eDxAhGHyNE6W+smLavoPfVsCEt+8wBkDqGPaMoaDTcUdgcw9/udji97+ngXvjqmr/qbD8awLnnf5sjmU/Fbg6od7nzyIGMWNGVxkFao/bDB4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734167086; c=relaxed/simple; bh=RtrI2tlYyIdcBio787YKVgk1cTNToRPh2j3uJiZRdZU=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=MtVlCXFqJMH6JCc50P92KDEVJvQKMYOihWcdntn+tFH4mckGu3q3ccuxU22XeoB637YzdzXKH5arEabQ9T0d8eyeV1IZyiilHF32qI/jWLz9cd/iGCr9BYMXNFW4zQTeqO2wsesDa+UreW4ZYBJTL1uDR/VEALJaagDV/LQpq0o= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=Axd3XqYA; arc=none smtp.client-ip=209.85.214.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="Axd3XqYA" Received: by mail-pl1-f182.google.com with SMTP id d9443c01a7336-2164b1f05caso24307105ad.3 for ; Sat, 14 Dec 2024 01:04:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1734167081; x=1734771881; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=+CB+suh8vMLKY78fggEtIk5gD9ivloFZvOHbrupXaAs=; b=Axd3XqYAMnwF827qRBo2Tl5EWXfoB0OI6Zp3tE3Ewo39i7y9aZw+dFB9ysDK/ZkCzl fZ0ZBVPNDLFjtbHSsB2r8ebKjw9HqZ56ZDhu5ypNLaT6dht5EwOzU482TZ71djBF7BlX a8GC2sFiYq4Z6iMwhWWCM0sXm1SXT1s6v7vp4AzMWQuGPXCMzim2LasCywRZvK9Xrz++ arPH9F7gwkjTM2lrdviUePW8YZ8IHLmd8Fm7qYNnTKyPjVcBakIVxdeYXhb+3Znmqn5X PCr56Dr6UcLeeoALdWwy5xagnC+RUZ9en0nVyvom1QTvOi2Np4ymglNyB78suR3ptM01 pjOg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734167081; x=1734771881; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=+CB+suh8vMLKY78fggEtIk5gD9ivloFZvOHbrupXaAs=; b=NCQYm1QdspBhjSqId2t+ShKdagLfJAKwGKD2CLGNiXpdw6chWp553PMdUELGIEZfg6 sIpW/HN3dFznR+4Lp8ve13nGQnW0SXlfmgzlSM25YrOxnLKwX8WXWRh9e0ujcrOb/EqR 59BTHbE32QP1091G8YMlkxSU0vAPo6i0pIYpY3q6M2SBAnpJHd2ZuRPypcoY4cpO5qox kqEPZrbwARhqzEbX26IVdel+exfAfhtwImXe/qptIb/23xIDctShiY9uKy7Mot+gN8bH 4iF3d4HgG+SxKLdWICvUqhzsHuCQJkAoCuTpsiC5WixLO+Oucqm+GOJTpbafote55kcP IhRg== X-Forwarded-Encrypted: i=1; AJvYcCUHQa35tnys64sLyQyJGMhrBF762uBSR+ZztVyhRWH5qsS3a348ZyE7/iwhlEKANDDldPOJObbDHiNpzUE=@vger.kernel.org X-Gm-Message-State: AOJu0Yw963yBo1/wwaAa7NiyrlWkSqKxUYQzTaGph9aImFNjffO+34JQ liKhMx+y4byRSRa5f0OHqg+z2YYhjSYWZBnL9u3nn0FhfDLFKkXAIXZHH7Y08/Y= X-Gm-Gg: ASbGncs+dzWoX2ItoOJzfbN+xWp89++GoJsV664vct0PlG7DoXxlAqocLX0ogAXyT/L p29J5Q/xlha+3VwhguoY29uFAl5QmUI43OVAOwmqHJ7sXDiEoKWosMGjteXigDyCTeSvoYMW+S9 aZaw2P30vDDYB7X4IemBmPWdquKgMzpDiulkqZP/7UALZaJNrEuyW8qP8+WbCi0J+o32b4a2GbJ HhsZt6QvuulpwsqctLs/Gp6k4oolKneEjIpoQyYbHHcb4XBJC6QY5GRd784gTv4fpBYGqk89G2+ l7kupSjrQtceHSXkGgqqehMXIPREDg== X-Google-Smtp-Source: AGHT+IE0QPwbAOQZHxlo/n3+wbqMrZuKXsuKTSsN4acRfuabgW+JUZvSc3827znPlMKVF1C6eV/6aw== X-Received: by 2002:a17:902:dac6:b0:215:8847:435c with SMTP id d9443c01a7336-21892982123mr70599045ad.12.1734167081724; Sat, 14 Dec 2024 01:04:41 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([2409:8a28:f4f:a9a4:5d46:4ef0:daa9:1db8]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-801d5a90ba0sm905731a12.16.2024.12.14.01.04.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 14 Dec 2024 01:04:40 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng , linux-arm-kernel@lists.infradead.org Subject: [PATCH 07/12] arm64: pgtable: move pagetable_dtor() to __tlb_remove_table() Date: Sat, 14 Dec 2024 17:02:53 +0800 Message-Id: <1781d3bab60aade7d64fbeb6682b9e73bd673d3c.1734164094.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Move pagetable_dtor() to __tlb_remove_table(), so that ptlock and page table pages can be freed together (regardless of whether RCU is used). This prevents the use-after-free problem where the ptlock is freed immediately but the page table pages is freed later via RCU. Page tables shouldn't have swap cache, so use pagetable_free() instead of free_page_and_swap_cache() to free page table pages. Signed-off-by: Qi Zheng Cc: linux-arm-kernel@lists.infradead.org Suggested-by: Peter Zijlstra (Intel) --- arch/arm64/include/asm/tlb.h | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-) diff --git a/arch/arm64/include/asm/tlb.h b/arch/arm64/include/asm/tlb.h index 408d0f36a8a8f..93591a80b5bfb 100644 --- a/arch/arm64/include/asm/tlb.h +++ b/arch/arm64/include/asm/tlb.h @@ -9,11 +9,13 @@ #define __ASM_TLB_H =20 #include -#include =20 static inline void __tlb_remove_table(void *_table) { - free_page_and_swap_cache((struct page *)_table); + struct ptdesc *ptdesc =3D (struct ptdesc *)_table; + + pagetable_dtor(ptdesc); + pagetable_free(ptdesc); } =20 #define tlb_flush tlb_flush @@ -82,7 +84,6 @@ static inline void __pte_free_tlb(struct mmu_gather *tlb,= pgtable_t pte, { struct ptdesc *ptdesc =3D page_ptdesc(pte); =20 - pagetable_dtor(ptdesc); tlb_remove_ptdesc(tlb, ptdesc); } =20 @@ -92,7 +93,6 @@ static inline void __pmd_free_tlb(struct mmu_gather *tlb,= pmd_t *pmdp, { struct ptdesc *ptdesc =3D virt_to_ptdesc(pmdp); =20 - pagetable_dtor(ptdesc); tlb_remove_ptdesc(tlb, ptdesc); } #endif @@ -106,7 +106,6 @@ static inline void __pud_free_tlb(struct mmu_gather *tl= b, pud_t *pudp, if (!pgtable_l4_enabled()) return; =20 - pagetable_dtor(ptdesc); tlb_remove_ptdesc(tlb, ptdesc); } #endif @@ -120,7 +119,6 @@ static inline void __p4d_free_tlb(struct mmu_gather *tl= b, p4d_t *p4dp, if (!pgtable_l5_enabled()) return; =20 - pagetable_dtor(ptdesc); tlb_remove_ptdesc(tlb, ptdesc); } #endif --=20 2.20.1 From nobody Wed Dec 17 15:51:21 2025 Received: from mail-pj1-f50.google.com (mail-pj1-f50.google.com [209.85.216.50]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5191619006F for ; Sat, 14 Dec 2024 09:04:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.50 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734167093; cv=none; b=HRjNFX+zFe0wfjPd1+nalmAW7PcIhJ5DAcwAhwE6PY0HH/ueVpJ3abDA9QMN6QpNRtyETKUBLPE8fVIX4e7/MHoZwpWd+u+MebXMlAs9HJheRWPuadAtxaOOp5Dftpedx6nSB8eA6IiLUqbTEGjFNnoncmDLIlHVyx51BhWCm+c= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734167093; c=relaxed/simple; bh=jFNmc4kjscGFNYYuudEQBS5d2YJZz2PscMxH+hm6jto=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=EC520ULqtBzPMh9I2ciXuCcX5G+C+kkloGTY4l99MnvcnGB/he2uw7ozzGlUQCvNbqAkHJzeUNbfa9YOODD3mnEdsLP1yQbbFLTdi+fiCdD1M+mvk7xSrLgeNqdq9GFdlYjnLCz0lGeTEwR0RCsCQ/KOgFYs0nrGhAKSdddvKVM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=cZW3/K2A; arc=none smtp.client-ip=209.85.216.50 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="cZW3/K2A" Received: by mail-pj1-f50.google.com with SMTP id 98e67ed59e1d1-2ee9a780de4so1655458a91.3 for ; Sat, 14 Dec 2024 01:04:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1734167090; x=1734771890; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=kxcjg+YJiLOjdKwLGOoVz8JWl+VadyRpr3tadtBHjWA=; b=cZW3/K2AaBgWOSre/2646RA7Q2SCn1FlCoJ6EWQqP9Kci4+XwHSyIQZarsfd527o6d p5SZunJpyzEDdG/tCOVg7RDvbNpv0ZCeBU7eDAVIeRBBGHNa+UFBcVOyj27fHXvguw47 KZBjfdT3Rw1R+GFrB/OpyOkG/1eYbTMtxgyElPIDKJNg6GfafAeppv5E2RTOUs8paoSX 0BHxZmCCE0WRqhzFzlZ1mHBVvuM1nuTbOhfAjWKvLJejKL/T37gpQOAO5uiyfI/fgRnX CYUhCCe9M5Db89udM8CDO5X8DFSRVw6EiqEDk+hvR98fBD5WCf/X7doKETDnW1N7PsyR 6pBA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734167090; x=1734771890; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=kxcjg+YJiLOjdKwLGOoVz8JWl+VadyRpr3tadtBHjWA=; b=mmopsmNz1H7E/n0DEJpQgNmYWNhannbHhrUDdjQ/VqewtHAFaxKGHej3TtENEqNXA5 IU3b0hOey6INeCCCag9BgPlE1SPcNTz+Qlj79yWjjzPuKPdf1KrIo14heHTntP55qgll Ebi/aJBuAuq9Sds4DMaqHWPmJpzUZgiibQ8zrkBB96+tak+Lj4AgfkkgAGvojok9/lAH /S3DCThufX7Evr/LlwiflZE1eBFr37wECxd0HKEz9ab3WYfjWUSAnAI0fYl7juk/aNJR Y3xHlWatIw6OLCnNNhBDuRWqce4iTWEqPi2pATgpkMym59sbyMaaoy4BSbwzY03KUC4f PBEQ== X-Forwarded-Encrypted: i=1; AJvYcCU+84p8KnMGT/yr9IeaRynekD/LUPCRH5Ae3XtDkWm+zVbEDb6TvzS2nIs0nsGCHKKG/Alg4eUBCvvyiaY=@vger.kernel.org X-Gm-Message-State: AOJu0YwxDOOJXjlJ29wzf+I5qy35GgAiapNudm3lyop64CvASaoEF9jZ 9gi+tvjg/4rSXRocnrey+2hsVYAGRfoBP1j6YNKcj3+TF4QrIbx+Ua71pTP/494= X-Gm-Gg: ASbGncsQTKT0xEXm4CQhsE6kIZIRiB1mA6bJgVI/VARBCThLbx/Nd7Wt2a5iAAmdro3 6DBLavA9VJD0cM5Jfa9Mq/+VwYk1BOYeYVdSPrB1m1iX2NUDS4AVcVVCPjypL7HJw29gueDQiNw 3MSwfpU7pQBB0fgyrso9vP49+3j9JgUyi9G/mVRr2XLC2o1DOPZIIEa4zFH/3767/gs++xNVeh2 2Q18fTQ2+2mbJR10NI5AldV6qsdImEk3kqPyIOcQM2FLG57sVQlDXz/lHCN4iUt88GwfvecESvt SKFn7N4feHrsWhUr75eKdFjY3rX6wA== X-Google-Smtp-Source: AGHT+IGmWNhert5ebnmEBEANehnSTDBWCLVNkF+841k1TH0tjB/D3zS21KwZ8s5YJg+vyRMQfoaR6w== X-Received: by 2002:a17:90b:3b52:b0:2ef:2d9f:8e55 with SMTP id 98e67ed59e1d1-2f28fd66b3dmr9591223a91.17.1734167090585; Sat, 14 Dec 2024 01:04:50 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([2409:8a28:f4f:a9a4:5d46:4ef0:daa9:1db8]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-801d5a90ba0sm905731a12.16.2024.12.14.01.04.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 14 Dec 2024 01:04:50 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng , linux-riscv@lists.infradead.org Subject: [PATCH 08/12] riscv: pgtable: move pagetable_dtor() to __tlb_remove_table() Date: Sat, 14 Dec 2024 17:02:54 +0800 Message-Id: <1e73cf87193d838c64e402896b4252875c212986.1734164094.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Move pagetable_dtor() to __tlb_remove_table(), so that ptlock and page table pages can be freed together (regardless of whether RCU is used). This prevents the use-after-free problem where the ptlock is freed immediately but the page table pages is freed later via RCU. Page tables shouldn't have swap cache, so use pagetable_free() instead of free_page_and_swap_cache() to free page table pages. By the way, move the comment above __tlb_remove_table() to riscv_tlb_remove_ptdesc(), it will be more appropriate. Signed-off-by: Qi Zheng Cc: linux-riscv@lists.infradead.org Suggested-by: Peter Zijlstra (Intel) --- arch/riscv/include/asm/pgalloc.h | 38 ++++++++++++++------------------ arch/riscv/include/asm/tlb.h | 14 ++++-------- 2 files changed, 21 insertions(+), 31 deletions(-) diff --git a/arch/riscv/include/asm/pgalloc.h b/arch/riscv/include/asm/pgal= loc.h index 09c2eff571a49..fc50d14010246 100644 --- a/arch/riscv/include/asm/pgalloc.h +++ b/arch/riscv/include/asm/pgalloc.h @@ -18,12 +18,22 @@ #define __HAVE_ARCH_P4D_FREE #include =20 +/* + * While riscv platforms with riscv_ipi_for_rfence as true require an IPI = to + * perform TLB shootdown, some platforms with riscv_ipi_for_rfence as fals= e use + * SBI to perform TLB shootdown. To keep software pagetable walkers safe i= n this + * case we switch to RCU based table free (MMU_GATHER_RCU_TABLE_FREE). See= the + * comment below 'ifdef CONFIG_MMU_GATHER_RCU_TABLE_FREE' in include/asm-g= eneric/tlb.h + * for more details. + */ static inline void riscv_tlb_remove_ptdesc(struct mmu_gather *tlb, void *p= t) { - if (riscv_use_sbi_for_rfence()) + if (riscv_use_sbi_for_rfence()) { tlb_remove_ptdesc(tlb, pt); - else + } else { + pagetable_dtor(pt); tlb_remove_page_ptdesc(tlb, pt); + } } =20 static inline void pmd_populate_kernel(struct mm_struct *mm, @@ -109,12 +119,8 @@ static inline void pud_free(struct mm_struct *mm, pud_= t *pud) static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pud, unsigned long addr) { - if (pgtable_l4_enabled) { - struct ptdesc *ptdesc =3D virt_to_ptdesc(pud); - - pagetable_dtor(ptdesc); - riscv_tlb_remove_ptdesc(tlb, ptdesc); - } + if (pgtable_l4_enabled) + riscv_tlb_remove_ptdesc(tlb, virt_to_ptdesc(pud)); } =20 #define p4d_alloc_one p4d_alloc_one @@ -136,12 +142,8 @@ static inline void p4d_free(struct mm_struct *mm, p4d_= t *p4d) static inline void __p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d, unsigned long addr) { - if (pgtable_l5_enabled) { - struct ptdesc *ptdesc =3D virt_to_ptdesc(p4d); - - pagetable_dtor(ptdesc); + if (pgtable_l5_enabled) riscv_tlb_remove_ptdesc(tlb, virt_to_ptdesc(p4d)); - } } #endif /* __PAGETABLE_PMD_FOLDED */ =20 @@ -170,10 +172,7 @@ static inline pgd_t *pgd_alloc(struct mm_struct *mm) static inline void __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd, unsigned long addr) { - struct ptdesc *ptdesc =3D virt_to_ptdesc(pmd); - - pagetable_dtor(ptdesc); - riscv_tlb_remove_ptdesc(tlb, ptdesc); + riscv_tlb_remove_ptdesc(tlb, virt_to_ptdesc(pmd)); } =20 #endif /* __PAGETABLE_PMD_FOLDED */ @@ -181,10 +180,7 @@ static inline void __pmd_free_tlb(struct mmu_gather *t= lb, pmd_t *pmd, static inline void __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pte, unsigned long addr) { - struct ptdesc *ptdesc =3D page_ptdesc(pte); - - pagetable_dtor(ptdesc); - riscv_tlb_remove_ptdesc(tlb, ptdesc); + riscv_tlb_remove_ptdesc(tlb, page_ptdesc(pte)); } #endif /* CONFIG_MMU */ =20 diff --git a/arch/riscv/include/asm/tlb.h b/arch/riscv/include/asm/tlb.h index 1f6c38420d8e0..ded8724b3c4f7 100644 --- a/arch/riscv/include/asm/tlb.h +++ b/arch/riscv/include/asm/tlb.h @@ -11,19 +11,13 @@ struct mmu_gather; static void tlb_flush(struct mmu_gather *tlb); =20 #ifdef CONFIG_MMU -#include =20 -/* - * While riscv platforms with riscv_ipi_for_rfence as true require an IPI = to - * perform TLB shootdown, some platforms with riscv_ipi_for_rfence as fals= e use - * SBI to perform TLB shootdown. To keep software pagetable walkers safe i= n this - * case we switch to RCU based table free (MMU_GATHER_RCU_TABLE_FREE). See= the - * comment below 'ifdef CONFIG_MMU_GATHER_RCU_TABLE_FREE' in include/asm-g= eneric/tlb.h - * for more details. - */ static inline void __tlb_remove_table(void *table) { - free_page_and_swap_cache(table); + struct ptdesc *ptdesc =3D (struct ptdesc *)table; + + pagetable_dtor(ptdesc); + pagetable_free(ptdesc); } =20 #endif /* CONFIG_MMU */ --=20 2.20.1 From nobody Wed Dec 17 15:51:21 2025 Received: from mail-pf1-f174.google.com (mail-pf1-f174.google.com [209.85.210.174]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 51029186E58 for ; Sat, 14 Dec 2024 09:05:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.174 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734167101; cv=none; b=Ja6dqM7t+G6QjTSftN9fiFnNnp6s/zneDxVslkZrvLEgM7a4+f0pS1QK/up5bAG38wp6nsLnHw+XVJMQL+sVkBRSVLC0BQ/SabfVU5EyyKIkIVkbwdbAx4Hd7k2c9rtBWMagzgr9cX9OZp4nDd+O33M46jRBAWoRRoCfDQqTPyo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734167101; c=relaxed/simple; bh=O1rd8xPAKupYKIoOHV8CV0w14aUMzX9ObgWsGmQJ/DM=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=APMBfI/5/42XbHaFd1luhjIkNK5k8CryiI7QyBuUp8CCwFS2LPI7xQz0mCiVQ0rJgpUeB2K1d2UQXoXERyNDUUpFZT4gye+GQneF+B8c4EluPngCLK/FTwlUaBtOVao5ZFx7Rw/a50WSUyXLfghz7lfZsgJygIvuDi+rYM1Xp+k= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=S+xwpjio; arc=none smtp.client-ip=209.85.210.174 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="S+xwpjio" Received: by mail-pf1-f174.google.com with SMTP id d2e1a72fcca58-728f1525565so2996047b3a.1 for ; Sat, 14 Dec 2024 01:05:00 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1734167099; x=1734771899; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=RMCudVMtlfI7I8C+EWzj20f+Ae2SGvA6pNu/2M7K/mw=; b=S+xwpjioxrmHWHedggmEIVICmQXCppfg4Q+La7y8+mQstfKeKxAj+FmzMDDXusjr0t 0IjfTXH/EoFNz20+g05D6IE+agUg6TRc+Y9dCyQNT3lquDHTi8jAhK2fQUMdQIzwEQ4+ Ho6BNaqV5W6JR2Oxq45uMvZDe3WmD3UXM4Y6dsDh0EBoVWYvG7M/yE0Uq3Kgu3P5ElaB 7hULBtqHiGNoHZC9Fvf+xJ3j699LSxph9ze9AffGN5R3g3aYrE8U5RITSWN4IHxHTh5i UllQ91I3kyrqcrtGXnMHbtWwth97FHMeFSzI+hNoj3VCqThq0TH6SzhA7dCoQQruu8oJ e6yw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734167099; x=1734771899; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=RMCudVMtlfI7I8C+EWzj20f+Ae2SGvA6pNu/2M7K/mw=; b=KT0PC9UkjnNgv7XFo7tLlR+rxAGTQEK8iVD/v8lKpBZ99E1WzAPhTm2INmQvXtyM+w vQBARVv1d6y8lihkHTe7470nYO+UjdtMIMvLpOrlca/DTw1c3T3Zc7kd26ylrCuXA75L 6YmGVSRoRSUQ7piTldpDISUJp6RsUkGzR0SshR0bF2SpZIXUxMuFbNSNwSJBVxRFrTxv P+NLVfGoIuFu2kUL4HGjnWO+6QkXVyS07ohZem7YKlrRlsEsRSJJsYe1FZDt4Z+0DP9L IvdYLaCFjFZ7fuphbCJ7HIdjJKvvxa8y9nFVFu9I5J3C71V6TBgdNy7r73eKfFwUeQ2X aDsQ== X-Forwarded-Encrypted: i=1; AJvYcCX7S71r444mqJJjYz1NdFW3ZNQrB9ydzZtMqw6kIUd01QXNrVoO2v2D40go4JeuJhbcDZua6R9vHkoe/eo=@vger.kernel.org X-Gm-Message-State: AOJu0YxeLkPcbND8vo2htrsiP8NFN9cKHJm1dIfle+3aaK2Vb0udkMAF oOTTfjYbhdjqD3UVSorj9jz5tGl2wKGQ4s/H8iO6IrNJ7imtHSWgD8/MmM4zfz8= X-Gm-Gg: ASbGncv3lMucHGlnVapRy62PilhUB/605Pyfj+/HHzxIYO+Bt2lbhG/HDmbuQSZCsZR g6Ow368dmZzkdGa8V6xJ5+IhXe2mwfgtqytcnvOz3oiOBMddx779OjR1vveCfpthgo2TgI1FTOI jZAN1/NIp4sro7eA1kpa9dqxJwI3Eu+Rfv+D9CMUOUVYYXINak1abv8Y5+5zg3Iv/6pHdhsuO0c a9EJblGu6E2wAQx8+konStLzpj+T9gC1ncsoXyfHp9kuGz7wK/2E6jYg4J8sVPe4bYpwSiW9Boi yWLSTCGwoGRMuQ294rEnY5TJwLmNQg== X-Google-Smtp-Source: AGHT+IHNImhO/lIwU1CxZtuKJT3uG8XCXYTC5vUDdEevSxX9VuCSFNAuRKGmdb9G/Urv3IN3FNQNtw== X-Received: by 2002:a05:6a20:d808:b0:1e1:c0d7:7502 with SMTP id adf61e73a8af0-1e1dfe1d265mr6847659637.37.1734167099625; Sat, 14 Dec 2024 01:04:59 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([2409:8a28:f4f:a9a4:5d46:4ef0:daa9:1db8]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-801d5a90ba0sm905731a12.16.2024.12.14.01.04.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 14 Dec 2024 01:04:59 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng , x86@kernel.org Subject: [PATCH 09/12] x86: pgtable: move pagetable_dtor() to __tlb_remove_table() Date: Sat, 14 Dec 2024 17:02:55 +0800 Message-Id: <01a0ce994f5d475230117b2991566cd7e2ff1dcf.1734164094.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Move pagetable_dtor() to __tlb_remove_table(), so that ptlock and page table pages can be freed together (regardless of whether RCU is used). This prevents the use-after-free problem where the ptlock is freed immediately but the page table pages is freed later via RCU. Page tables shouldn't have swap cache, so use pagetable_free() instead of free_page_and_swap_cache() to free page table pages. Signed-off-by: Qi Zheng Cc: x86@kernel.org Suggested-by: Peter Zijlstra (Intel) --- arch/x86/include/asm/tlb.h | 17 ++++++++++------- arch/x86/kernel/paravirt.c | 1 + arch/x86/mm/pgtable.c | 12 ++---------- 3 files changed, 13 insertions(+), 17 deletions(-) diff --git a/arch/x86/include/asm/tlb.h b/arch/x86/include/asm/tlb.h index 73f0786181cc9..f64730be5ad67 100644 --- a/arch/x86/include/asm/tlb.h +++ b/arch/x86/include/asm/tlb.h @@ -31,24 +31,27 @@ static inline void tlb_flush(struct mmu_gather *tlb) */ static inline void __tlb_remove_table(void *table) { - free_page_and_swap_cache(table); + struct ptdesc *ptdesc =3D (struct ptdesc *)table; + + pagetable_dtor(ptdesc); + pagetable_free(ptdesc); } =20 #ifdef CONFIG_PT_RECLAIM static inline void __tlb_remove_table_one_rcu(struct rcu_head *head) { - struct page *page; + struct ptdesc *ptdesc; =20 - page =3D container_of(head, struct page, rcu_head); - put_page(page); + ptdesc =3D container_of(head, struct ptdesc, pt_rcu_head); + __tlb_remove_table(ptdesc); } =20 static inline void __tlb_remove_table_one(void *table) { - struct page *page; + struct ptdesc *ptdesc; =20 - page =3D table; - call_rcu(&page->rcu_head, __tlb_remove_table_one_rcu); + ptdesc =3D table; + call_rcu(&ptdesc->pt_rcu_head, __tlb_remove_table_one_rcu); } #define __tlb_remove_table_one __tlb_remove_table_one #endif /* CONFIG_PT_RECLAIM */ diff --git a/arch/x86/kernel/paravirt.c b/arch/x86/kernel/paravirt.c index 7bdcf152778c0..46d5d325483b0 100644 --- a/arch/x86/kernel/paravirt.c +++ b/arch/x86/kernel/paravirt.c @@ -62,6 +62,7 @@ void __init native_pv_lock_init(void) #ifndef CONFIG_PT_RECLAIM static void native_tlb_remove_table(struct mmu_gather *tlb, void *table) { + pagetable_dtor(table); tlb_remove_page(tlb, table); } #else diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c index a6cd9660e29ec..a0b0e501ba663 100644 --- a/arch/x86/mm/pgtable.c +++ b/arch/x86/mm/pgtable.c @@ -23,6 +23,7 @@ EXPORT_SYMBOL(physical_mask); static inline void paravirt_tlb_remove_table(struct mmu_gather *tlb, void *table) { + pagetable_dtor(table); tlb_remove_page(tlb, table); } #else @@ -60,7 +61,6 @@ early_param("userpte", setup_userpte); =20 void ___pte_free_tlb(struct mmu_gather *tlb, struct page *pte) { - pagetable_dtor(page_ptdesc(pte)); paravirt_release_pte(page_to_pfn(pte)); paravirt_tlb_remove_table(tlb, pte); } @@ -68,7 +68,6 @@ void ___pte_free_tlb(struct mmu_gather *tlb, struct page = *pte) #if CONFIG_PGTABLE_LEVELS > 2 void ___pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd) { - struct ptdesc *ptdesc =3D virt_to_ptdesc(pmd); paravirt_release_pmd(__pa(pmd) >> PAGE_SHIFT); /* * NOTE! For PAE, any changes to the top page-directory-pointer-table @@ -77,16 +76,12 @@ void ___pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd) #ifdef CONFIG_X86_PAE tlb->need_flush_all =3D 1; #endif - pagetable_dtor(ptdesc); - paravirt_tlb_remove_table(tlb, ptdesc_page(ptdesc)); + paravirt_tlb_remove_table(tlb, virt_to_page(pmd)); } =20 #if CONFIG_PGTABLE_LEVELS > 3 void ___pud_free_tlb(struct mmu_gather *tlb, pud_t *pud) { - struct ptdesc *ptdesc =3D virt_to_ptdesc(pud); - - pagetable_dtor(ptdesc); paravirt_release_pud(__pa(pud) >> PAGE_SHIFT); paravirt_tlb_remove_table(tlb, virt_to_page(pud)); } @@ -94,9 +89,6 @@ void ___pud_free_tlb(struct mmu_gather *tlb, pud_t *pud) #if CONFIG_PGTABLE_LEVELS > 4 void ___p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d) { - struct ptdesc *ptdesc =3D virt_to_ptdesc(p4d); - - pagetable_dtor(ptdesc); paravirt_release_p4d(__pa(p4d) >> PAGE_SHIFT); paravirt_tlb_remove_table(tlb, virt_to_page(p4d)); } --=20 2.20.1 From nobody Wed Dec 17 15:51:21 2025 Received: from mail-pf1-f172.google.com (mail-pf1-f172.google.com [209.85.210.172]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3011A16F8F5 for ; Sat, 14 Dec 2024 09:05:10 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734167111; cv=none; b=GqNbqzW3dmJUPUDoPVDASccdESqu/y7DCJZE0WmsN5tEUJoQPewRp6mjbAs+185J1AmuDKZlstT6qIvVW42CDWL8qPikgB911eu5fpCNa9i68KAHIv5gWvdr+VJPdOX9/+Kk37PWAqS8VnhgkRzOd2w/4a4gZioqbtBzZ/nJTXo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734167111; c=relaxed/simple; bh=dUn22uFUF/7Q2aX8/sTNpsqlDbyw/AbPrg+Ofum3Ioc=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=SCSlbzE0O1c9o9T116b3UgyV+QU1E4hrhlXaxOZS/7up/0TDpcq/Ex7fXEHI5TowVSqlq8U+llEimBrKmZtiwDUr1rxoiTzlk3SVoZVUnn4yi/SM1iU6VMKyQz+SUrLOrpKF/L1Aeau9IafRUj1Kv4DlnGLGxvXktYQqfG2swdg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=EkJs1S40; arc=none smtp.client-ip=209.85.210.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="EkJs1S40" Received: by mail-pf1-f172.google.com with SMTP id d2e1a72fcca58-725ed193c9eso2117479b3a.1 for ; Sat, 14 Dec 2024 01:05:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1734167109; x=1734771909; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=eMqrm4pkcrZJ7adK/0JGqqs1dXiKNR49XOWaF/ar7JI=; b=EkJs1S40b36UBcQ5tUf9/vhozXD4UmwBoav0utIjGg4O6vI87BIvXEcRBESTFU0jxA kinWKUyncOnTcS0i9Fbv0ON6LZZX9H33oWjAbCGrS1HkbiftT8zr+ReTpScQbaInnsm2 dvBp553B55mShlO/mo39Tii4GpIsm9L5IYJT2RC1ZQ8SpkAfpLtJeJK1qvvf0+oXa6Ok T5sf1ovXb1U9w5v9psnSDheggAZUErRkndv/Bf8dg1aEDjfuwwLCl/Fq1vu+6SiXxogC F59pck4XW8mkJT9Y+MhBSTwlMxeCOmM6HK+O3ylpUZKdHyg16Hezfi5S7o7cmroCjbLA urTQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734167109; x=1734771909; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=eMqrm4pkcrZJ7adK/0JGqqs1dXiKNR49XOWaF/ar7JI=; b=dybYusX6vEmkiR7WsD/YLgBLVBdxeSfSpFM2TAjtZW73i11COC1W+La8LCBABYnIcy TOtb06QrxnkvfBeWtirtowaVOAkJGTFmugg+LeRrDIRdaaJG72T68Vndub89CwnhNjnU 1SSKUI4ZwusEEaUilYG/tF3vtvOAlPDJS9Kkq8yVZmLNBOO7d/pgiQHPt1f1S0nsjYGK bBu6aR5C1lBwaYKhsB//as0qWk/C3I44F+3qCtLBBlLnO7CqoGdWS5zV+51LYLROc3Fk FkYzmFwo+eBKFDGkictwxtM4yrcntz3/VI+4xbTPVrESVGx/+pwG7nUwfVM6oA+4r3/7 K2NA== X-Forwarded-Encrypted: i=1; AJvYcCUbR6zo7iZWyoI5r6XNgjf6zGT2TCNGpiNvNIMX4wLBKH1ZJDxGEt4uxHPfmKobqfCSG/45U7mEcfNHswI=@vger.kernel.org X-Gm-Message-State: AOJu0YxMNwtkp3o0Ep5tH90MYqzDboXy2pN6H0tM+iU0LvUK4pOHnyYg lZnvHOV48lggdfLGgE7FVh3JRl6xYDGvnl9lfNnl7Uu3mLr+j6vCXuK/uA7869c= X-Gm-Gg: ASbGncuOPMsdOR0sNbb4uDjUwm9peioJNUMxgwjU7YtsfhYvaPlsvsTQDYeuTU8gVXH C0/OMLVBVcVUlRBP79tu4lTizsuD7f9dB54U3pZME+8faWnUp96eqRhLPyHnI84gDcD+Kv2r+B5 RDflkvLcXhkUftCeLac7QSV1u0A0x6AcZzxOft1h8ioV1KRV4NJ0ss4a0kdh/JBMiOjqyldF1Yg JT9OYUEl2Gwd4YOJNbVO7WIq5FkbQmK3UnWs3GbrF/vVMI2Nu0jqxH9r9htMHeM0DjtkC8XrTA6 MZo033+MLRMjgaBdQ95IxiBWX7iWzQ== X-Google-Smtp-Source: AGHT+IE/1mbbP4uNRetgOIypjg2bvSDj3GWH/cGEc/xqzPLinfL2kdCDxCigFtQu4L9Hg/0wQ4NEdw== X-Received: by 2002:a05:6a20:a123:b0:1e1:96d9:a7db with SMTP id adf61e73a8af0-1e1dfbfc129mr8345681637.4.1734167109554; Sat, 14 Dec 2024 01:05:09 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([2409:8a28:f4f:a9a4:5d46:4ef0:daa9:1db8]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-801d5a90ba0sm905731a12.16.2024.12.14.01.05.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 14 Dec 2024 01:05:08 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng , linux-s390@vger.kernel.org Subject: [PATCH 10/12] s390: pgtable: also move pagetable_dtor() of PxD to __tlb_remove_table() Date: Sat, 14 Dec 2024 17:02:56 +0800 Message-Id: <1e87efe43e9b8b679384e947dd8503715dd9faf0.1734164094.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" In s390, the pagetable_dtor() of PTE has long been moved to __tlb_remove_table(). So similarly, also move the pagetable_dtor() of PMD|PUD|P4D to __tlb_remove_table(). This prevents the use-after-free problem where the ptlock is freed immediately but the page table pages is freed later via RCU. By the way, rename pagetable_pte_dtor_free() to pagetable_dtor_free(). Signed-off-by: Qi Zheng Cc: linux-s390@vger.kernel.org Suggested-by: Peter Zijlstra (Intel) --- arch/s390/include/asm/tlb.h | 3 --- arch/s390/mm/pgalloc.c | 14 ++++---------- 2 files changed, 4 insertions(+), 13 deletions(-) diff --git a/arch/s390/include/asm/tlb.h b/arch/s390/include/asm/tlb.h index 74b6fba4c2ee3..79df7c0932c56 100644 --- a/arch/s390/include/asm/tlb.h +++ b/arch/s390/include/asm/tlb.h @@ -102,7 +102,6 @@ static inline void pmd_free_tlb(struct mmu_gather *tlb,= pmd_t *pmd, { if (mm_pmd_folded(tlb->mm)) return; - pagetable_dtor(virt_to_ptdesc(pmd)); __tlb_adjust_range(tlb, address, PAGE_SIZE); tlb->mm->context.flush_mm =3D 1; tlb->freed_tables =3D 1; @@ -122,7 +121,6 @@ static inline void pud_free_tlb(struct mmu_gather *tlb,= pud_t *pud, { if (mm_pud_folded(tlb->mm)) return; - pagetable_dtor(virt_to_ptdesc(pud)); tlb->mm->context.flush_mm =3D 1; tlb->freed_tables =3D 1; tlb->cleared_p4ds =3D 1; @@ -141,7 +139,6 @@ static inline void p4d_free_tlb(struct mmu_gather *tlb,= p4d_t *p4d, { if (mm_p4d_folded(tlb->mm)) return; - pagetable_dtor(virt_to_ptdesc(p4d)); __tlb_adjust_range(tlb, address, PAGE_SIZE); tlb->mm->context.flush_mm =3D 1; tlb->freed_tables =3D 1; diff --git a/arch/s390/mm/pgalloc.c b/arch/s390/mm/pgalloc.c index 569de24d33761..c73b89811a264 100644 --- a/arch/s390/mm/pgalloc.c +++ b/arch/s390/mm/pgalloc.c @@ -180,7 +180,7 @@ unsigned long *page_table_alloc(struct mm_struct *mm) return table; } =20 -static void pagetable_pte_dtor_free(struct ptdesc *ptdesc) +static void pagetable_dtor_free(struct ptdesc *ptdesc) { pagetable_dtor(ptdesc); pagetable_free(ptdesc); @@ -190,20 +190,14 @@ void page_table_free(struct mm_struct *mm, unsigned l= ong *table) { struct ptdesc *ptdesc =3D virt_to_ptdesc(table); =20 - pagetable_pte_dtor_free(ptdesc); + pagetable_dtor_free(ptdesc); } =20 void __tlb_remove_table(void *table) { struct ptdesc *ptdesc =3D virt_to_ptdesc(table); - struct page *page =3D ptdesc_page(ptdesc); =20 - if (compound_order(page) =3D=3D CRST_ALLOC_ORDER) { - /* pmd, pud, or p4d */ - pagetable_free(ptdesc); - return; - } - pagetable_pte_dtor_free(ptdesc); + pagetable_dtor_free(ptdesc); } =20 #ifdef CONFIG_TRANSPARENT_HUGEPAGE @@ -211,7 +205,7 @@ static void pte_free_now(struct rcu_head *head) { struct ptdesc *ptdesc =3D container_of(head, struct ptdesc, pt_rcu_head); =20 - pagetable_pte_dtor_free(ptdesc); + pagetable_dtor_free(ptdesc); } =20 void pte_free_defer(struct mm_struct *mm, pgtable_t pgtable) --=20 2.20.1 From nobody Wed Dec 17 15:51:21 2025 Received: from mail-pl1-f179.google.com (mail-pl1-f179.google.com [209.85.214.179]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C804816F8F5 for ; Sat, 14 Dec 2024 09:05:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.179 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734167120; cv=none; b=ru5qiInYLkb3da8AGPlXTOjqsLewVZuGbgWCevtjcLrQHdVj/ww8BgNuM4yK6S7KZPwoUDn1wHWM+hSFZbE9FVfNDTD0c4x+jJhGefto0eKLHt5o0ipUMxRzB4G2FOUyH/o+tNwgHB+VFEhgJ+caXYbe1gxJPEnf07OdoO/dOPg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734167120; c=relaxed/simple; bh=jz2f+TXF9L6eH57WLv03lHqQHkdhz2aBBoef3FGpiyg=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=t6KV2vsdEMyTS8aX4Ld2zru11E3IyuJvVdAQNxHluTpF5NsH2ehGC/rUI1Cpdq4JY6YAsS5ZAtTjk1cYa2FglLC5UI4hIFK/YAGzBASqe9MC2GqiONxRBsbrtIuWiy1hiYYYzhuKteaKNHBhiqE17coUoGGUXd9G35uPJe1qmSk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=CjlPnAbc; arc=none smtp.client-ip=209.85.214.179 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="CjlPnAbc" Received: by mail-pl1-f179.google.com with SMTP id d9443c01a7336-21669fd5c7cso22631865ad.3 for ; Sat, 14 Dec 2024 01:05:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1734167118; x=1734771918; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=tGOtYoMgWgj4YJiYefYK2aPYu+m8suzmBGDTS71l9lQ=; b=CjlPnAbc1deakWC9wWLVtj+9u5M9eAWxpd58PQE3PwjWxs7YqQAGdNbsblUNevLBxu SmItK3a541WmTF4AV7vBun707O/UAOiFfEpY/DXNNDxgzymowLrNC3gEnGMEORiR33Gm ZIGr+ZoxK1N1Msq1DrXyZCVFDMidVeQ/Ga1/2j5F3CxTfUO79t0sfWwnyrpZIbziiUtc lBU6ck8FarO0qOiN+yY2wjio6Tx9KpNSgRH/9BbFmYojbkL6ReCKuWYDHJPp7DerJF9I 5R75h5iUo++F8K1hgFTjk6dhC4e78AyuWxLnjetqKUNCBiqu5BzbxLIcXPhvtR/VQk5H YIQQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734167118; x=1734771918; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=tGOtYoMgWgj4YJiYefYK2aPYu+m8suzmBGDTS71l9lQ=; b=HFXYtY4msWURNkCaM+ofqeOYV11YBxL+LlPJDI0rqUMGju1kxXjBDxp+J36039ymLg UkPeawyWaMAvtwUOyc0tUiPfaQLRlUk7M9UFl/GMndWS0Zg/cKxc7JJGKB7YETBQcXGl im2rTaSACe30si3B42j6Ans7CA1VKNF1kRw1Lk8PZetgTEsVcEvU//pFoHtqfNWJpX08 U34P9DmA58bGy7M7/lcFyeGlwGx7RPP2TfaHIBjX3ZjGR7JuDXZtgJB6RDdahnl1mejZ G+0cW0rWWE4t0pZiRYoCx2e06nMtdoDDJWx4lwuulQHBSyM2z0+3DwJZJ0EArjfnqAYw 2ahQ== X-Forwarded-Encrypted: i=1; AJvYcCWQR5ahhHR9bYpG7ep0GTRli48YUOWpYLY3xe5tIXYHxmxqiKwEqJd3dt7r16P3b+bwC+b7ZRCMUkdbbpQ=@vger.kernel.org X-Gm-Message-State: AOJu0YwW9hhkhxuyZvKpdlDHaPZ/0Dkiv+YSqrM9jK+eV/9ZJHD5YRg4 XFG3ye7tIQUnQWlqQ/n4/VvrqZ1uZ2UTVBTS2dBd2OBwOWhjVR254trh16KC/TA= X-Gm-Gg: ASbGncsFm/H3p0ES8+wlaX0OMgNV//lBtF3QmvbbsKZPKXKLGJ4naCk+bD3KJrpOKRI l10Yc+99omqNwwBVJK1FLsqsPSdEV9VGWMJlBMSgtZLW2E3TNAqZ9n5IyDTk0u5Zy4UJdv3A4CJ ifv4nF6IXTeZxkFXqVBc+rlInYyUerUM4tc3OFSCAJKrUUalnVbErHLsLyXFK6tFZ6e3VvGaYMm cBvNicIOmcxhtFe2TGtU9rl/EITOC0nc6V7tKdlrfjMPqL+qIHRKgsYhNfdnINNdnCPxCtYF4Em A3/dvj5RXy0ogCG2vBSswNvViLjVBQ== X-Google-Smtp-Source: AGHT+IGZK6CYdPWEqKPfkUj35U2xxEx7kipqiMJDua7xnmtA6POZpy/3kuQaFLc+awGAy7moPmF5qQ== X-Received: by 2002:a17:902:d481:b0:216:4b66:1f1f with SMTP id d9443c01a7336-21892a7c538mr78074805ad.54.1734167118105; Sat, 14 Dec 2024 01:05:18 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([2409:8a28:f4f:a9a4:5d46:4ef0:daa9:1db8]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-801d5a90ba0sm905731a12.16.2024.12.14.01.05.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 14 Dec 2024 01:05:17 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng Subject: [PATCH 11/12] mm: pgtable: introduce generic __tlb_remove_table() Date: Sat, 14 Dec 2024 17:02:57 +0800 Message-Id: <271e58cd4ab808c4f402539b76d5916924e2bc6f.1734164094.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Several architectures (arm, arm64, riscv, s390 and x86) define exactly the same __tlb_remove_table(), just introduce generic __tlb_remove_table() to eliminate these duplications. Signed-off-by: Qi Zheng Suggested-by: Peter Zijlstra (Intel) --- arch/arm/include/asm/tlb.h | 9 --------- arch/arm64/include/asm/tlb.h | 7 ------- arch/powerpc/include/asm/tlb.h | 1 + arch/riscv/include/asm/tlb.h | 12 ------------ arch/s390/include/asm/tlb.h | 1 - arch/s390/mm/pgalloc.c | 7 ------- arch/sparc/include/asm/tlb_32.h | 1 + arch/sparc/include/asm/tlb_64.h | 1 + arch/x86/include/asm/tlb.h | 17 ----------------- include/asm-generic/tlb.h | 15 +++++++++++++-- 10 files changed, 16 insertions(+), 55 deletions(-) diff --git a/arch/arm/include/asm/tlb.h b/arch/arm/include/asm/tlb.h index 264ab635e807a..ea4fbe7b17f6f 100644 --- a/arch/arm/include/asm/tlb.h +++ b/arch/arm/include/asm/tlb.h @@ -27,15 +27,6 @@ #else /* !CONFIG_MMU */ =20 #include - -static inline void __tlb_remove_table(void *_table) -{ - struct ptdesc *ptdesc =3D (struct ptdesc *)_table; - - pagetable_dtor(ptdesc); - pagetable_free(ptdesc); -} - #include =20 static inline void diff --git a/arch/arm64/include/asm/tlb.h b/arch/arm64/include/asm/tlb.h index 93591a80b5bfb..8d762607285cc 100644 --- a/arch/arm64/include/asm/tlb.h +++ b/arch/arm64/include/asm/tlb.h @@ -10,13 +10,6 @@ =20 #include =20 -static inline void __tlb_remove_table(void *_table) -{ - struct ptdesc *ptdesc =3D (struct ptdesc *)_table; - - pagetable_dtor(ptdesc); - pagetable_free(ptdesc); -} =20 #define tlb_flush tlb_flush static void tlb_flush(struct mmu_gather *tlb); diff --git a/arch/powerpc/include/asm/tlb.h b/arch/powerpc/include/asm/tlb.h index 1ca7d4c4b90db..2058e8d3e0138 100644 --- a/arch/powerpc/include/asm/tlb.h +++ b/arch/powerpc/include/asm/tlb.h @@ -37,6 +37,7 @@ extern void tlb_flush(struct mmu_gather *tlb); */ #define tlb_needs_table_invalidate() radix_enabled() =20 +#define __HAVE_ARCH_TLB_REMOVE_TABLE /* Get the generic bits... */ #include =20 diff --git a/arch/riscv/include/asm/tlb.h b/arch/riscv/include/asm/tlb.h index ded8724b3c4f7..50b63b5c15bd8 100644 --- a/arch/riscv/include/asm/tlb.h +++ b/arch/riscv/include/asm/tlb.h @@ -10,18 +10,6 @@ struct mmu_gather; =20 static void tlb_flush(struct mmu_gather *tlb); =20 -#ifdef CONFIG_MMU - -static inline void __tlb_remove_table(void *table) -{ - struct ptdesc *ptdesc =3D (struct ptdesc *)table; - - pagetable_dtor(ptdesc); - pagetable_free(ptdesc); -} - -#endif /* CONFIG_MMU */ - #define tlb_flush tlb_flush #include =20 diff --git a/arch/s390/include/asm/tlb.h b/arch/s390/include/asm/tlb.h index 79df7c0932c56..7052780740349 100644 --- a/arch/s390/include/asm/tlb.h +++ b/arch/s390/include/asm/tlb.h @@ -22,7 +22,6 @@ * Pages used for the page tables is a different story. FIXME: more */ =20 -void __tlb_remove_table(void *_table); static inline void tlb_flush(struct mmu_gather *tlb); static inline bool __tlb_remove_page_size(struct mmu_gather *tlb, struct page *page, bool delay_rmap, int page_size); diff --git a/arch/s390/mm/pgalloc.c b/arch/s390/mm/pgalloc.c index c73b89811a264..3e002dea6278f 100644 --- a/arch/s390/mm/pgalloc.c +++ b/arch/s390/mm/pgalloc.c @@ -193,13 +193,6 @@ void page_table_free(struct mm_struct *mm, unsigned lo= ng *table) pagetable_dtor_free(ptdesc); } =20 -void __tlb_remove_table(void *table) -{ - struct ptdesc *ptdesc =3D virt_to_ptdesc(table); - - pagetable_dtor_free(ptdesc); -} - #ifdef CONFIG_TRANSPARENT_HUGEPAGE static void pte_free_now(struct rcu_head *head) { diff --git a/arch/sparc/include/asm/tlb_32.h b/arch/sparc/include/asm/tlb_3= 2.h index 5cd28a8793e39..910254867dfbd 100644 --- a/arch/sparc/include/asm/tlb_32.h +++ b/arch/sparc/include/asm/tlb_32.h @@ -2,6 +2,7 @@ #ifndef _SPARC_TLB_H #define _SPARC_TLB_H =20 +#define __HAVE_ARCH_TLB_REMOVE_TABLE #include =20 #endif /* _SPARC_TLB_H */ diff --git a/arch/sparc/include/asm/tlb_64.h b/arch/sparc/include/asm/tlb_6= 4.h index 3037187482db7..1a6e694418e39 100644 --- a/arch/sparc/include/asm/tlb_64.h +++ b/arch/sparc/include/asm/tlb_64.h @@ -33,6 +33,7 @@ void flush_tlb_pending(void); #define tlb_needs_table_invalidate() (false) #endif =20 +#define __HAVE_ARCH_TLB_REMOVE_TABLE #include =20 #endif /* _SPARC64_TLB_H */ diff --git a/arch/x86/include/asm/tlb.h b/arch/x86/include/asm/tlb.h index f64730be5ad67..3858dbf75880e 100644 --- a/arch/x86/include/asm/tlb.h +++ b/arch/x86/include/asm/tlb.h @@ -20,23 +20,6 @@ static inline void tlb_flush(struct mmu_gather *tlb) flush_tlb_mm_range(tlb->mm, start, end, stride_shift, tlb->freed_tables); } =20 -/* - * While x86 architecture in general requires an IPI to perform TLB - * shootdown, enablement code for several hypervisors overrides - * .flush_tlb_others hook in pv_mmu_ops and implements it by issuing - * a hypercall. To keep software pagetable walkers safe in this case we - * switch to RCU based table free (MMU_GATHER_RCU_TABLE_FREE). See the com= ment - * below 'ifdef CONFIG_MMU_GATHER_RCU_TABLE_FREE' in include/asm-generic/t= lb.h - * for more details. - */ -static inline void __tlb_remove_table(void *table) -{ - struct ptdesc *ptdesc =3D (struct ptdesc *)table; - - pagetable_dtor(ptdesc); - pagetable_free(ptdesc); -} - #ifdef CONFIG_PT_RECLAIM static inline void __tlb_remove_table_one_rcu(struct rcu_head *head) { diff --git a/include/asm-generic/tlb.h b/include/asm-generic/tlb.h index 709830274b756..939a813023d7e 100644 --- a/include/asm-generic/tlb.h +++ b/include/asm-generic/tlb.h @@ -153,8 +153,9 @@ * * Useful if your architecture has non-page page directories. * - * When used, an architecture is expected to provide __tlb_remove_table() - * which does the actual freeing of these pages. + * When used, an architecture is expected to provide __tlb_remove_table()= or + * use the generic __tlb_remove_table(), which does the actual freeing of= these + * pages. * * MMU_GATHER_RCU_TABLE_FREE * @@ -207,6 +208,16 @@ struct mmu_table_batch { #define MAX_TABLE_BATCH \ ((PAGE_SIZE - sizeof(struct mmu_table_batch)) / sizeof(void *)) =20 +#ifndef __HAVE_ARCH_TLB_REMOVE_TABLE +static inline void __tlb_remove_table(void *_table) +{ + struct ptdesc *ptdesc =3D (struct ptdesc *)_table; + + pagetable_dtor(ptdesc); + pagetable_free(ptdesc); +} +#endif + extern void tlb_remove_table(struct mmu_gather *tlb, void *table); =20 #else /* !CONFIG_MMU_GATHER_HAVE_TABLE_FREE */ --=20 2.20.1 From nobody Wed Dec 17 15:51:21 2025 Received: from mail-pf1-f172.google.com (mail-pf1-f172.google.com [209.85.210.172]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5042E85931 for ; Sat, 14 Dec 2024 09:05:28 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734167129; cv=none; b=SJmDINgUbNU7fnlBhcoHkL1ZmV82SZ9AkGas/nTyWfdlaeJWMfMdDIzVSP5pGUPe3Ckuxb1wdINaJ4mkv1fcBn8E6SfnapbSZSIL1GrV+Qx0kYwU84Z7H7/bgIg9wL+RfFKdvNotkxzl8JC/pLkqTtqwwawJ54RTuPvC3UTIQQI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734167129; c=relaxed/simple; bh=AOHzYz3e0W68bypaywk5PqwWiTbsh7X+wa84FHfFzTo=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=rOeM5ln9VOpViWT8qMQJEvQOfzLNWDXp9AbmRaZ6TBISL7+sIIgp4l9QN67T/zSjGu8YgBBUB2kvWvPobUMcfTKwONW5AB0j4bSBu5ybjpUoLpyiT9TZyV2on4cFvebvT1IkOc5kBMRzCIluOgXM7KiYxIOO1fo1uFwnXYvzdZY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=gyELVjOM; arc=none smtp.client-ip=209.85.210.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="gyELVjOM" Received: by mail-pf1-f172.google.com with SMTP id d2e1a72fcca58-728f1e66418so2185630b3a.2 for ; Sat, 14 Dec 2024 01:05:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1734167128; x=1734771928; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=VAJKsDdGNb+ALwQvMYs9Oz9Vq8VGrLJsA8GxB+kZmZc=; b=gyELVjOMzOIe6ph/l41hPbeC2fX3S/ImoO4KpcUfbLSsIUgGwpHMolKP6qnQCvFiaa NFVlNjKqy3WA1ibWyq/hacbnUjeJRFKwxn6yuVwiT99Fldm2p3e4Lbr/uRb0QzhpxV4M UzCOKiworw+LsfLlDkZXRGn/LfxovlJChAgbDjnM6Xtw+qX3jJ0GlQhTvHvaQEKFtZPp nya9l/fQJMm1lLAe1t4ZHGyxuQnPNv6PQBG4M24TPG5OubOC5Tg0jUgaHUC7lddaHQTi UjNz5sjpMt5NvOuUaptEwtfWqm6D0UZlvmd/qTlTh6rOz8ZMTcHGXXaKBH9C0MWy7Kpf GTHg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734167128; x=1734771928; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=VAJKsDdGNb+ALwQvMYs9Oz9Vq8VGrLJsA8GxB+kZmZc=; b=M51zLxY4ogOHAhbOf6LJ2jluJFN2DQN5GF07es009ByNdqHQUq4q0miWXRvkc9wI/w CnOArAckFcMfWOgGaN7ViE0RRgn6jNitiCmv7EooPZtRrReweyUobk8rBVL/9rrW1B+5 Le50sHU6t5vexXU/trLedDLwo5AhYwf19jIlJ4r4HXatoJXl0GbCsivVOp1M0j8j15oq WrnCPIM/a+xcv6YoIY30uR3biYBpADEpilqjMELyBSvrjpXU1A90xfH0UP307Spug2hw ZFoMRRl7RDSQpCrLbZvYSdJgY/n3wo3r6kM2vyjxjKiKirhZNtO4173QBB+zlWvf0lsQ A7vA== X-Forwarded-Encrypted: i=1; AJvYcCVl1vjsBA5AJgxsGNvLhSeEyCxzh3eTUOZnjD2o0jw72GWKpoE+KrCV19tXGkmYxumkPXkMXr3X931hg4k=@vger.kernel.org X-Gm-Message-State: AOJu0YxWw8fXhxtDjckWxVDK4gCEmt+62UYNx5arJg8HpTcGdlT1A7E+ NHRLEscW21CSGmcrBWwltIU0xzlvDELphIYOyAhidRR0NtYJ2k7KlgpX5lFl9Unx0XLRT3wGY1q F X-Gm-Gg: ASbGncsHFVpx5EnQxSelkHa7khK6VXuPi6R4Hs7kzdX22ykr7sYNBSELDVtLccCHIiH BPgK1i90kBw9cuSVGoW85oEsmB+fllBhGCOQZrgnCeSvC43v/KNEtnxJu6wXpyz00PB7p8NKS3u cqbOdpxwMAwuNPVghIRzgBZg5nlEHh50k+se1OCt7jnwmqGtyD0fXwxmzpM4HyupjhAO5B8lA7J ATyhDhrf4QAwcpYM75LBxx8CivBnfKyvaY83SFiWtUGbuQ6jzwO1lPYpqXjyRgEJSJxdiv2yvMk EA18JkUvS9zcHG8XPuWhX0tBj8V1Bg== X-Google-Smtp-Source: AGHT+IFivM0byq9BUMDoda6VIidtD6lpjalZ4ywP7t2KopoDAAS1RcnNTtREGYaK+hQvD7u6okSdbQ== X-Received: by 2002:a05:6a20:1588:b0:1db:e536:158e with SMTP id adf61e73a8af0-1e1dfd91a2emr6805131637.22.1734167127753; Sat, 14 Dec 2024 01:05:27 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([2409:8a28:f4f:a9a4:5d46:4ef0:daa9:1db8]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-801d5a90ba0sm905731a12.16.2024.12.14.01.05.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 14 Dec 2024 01:05:26 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng Subject: [PATCH 12/12] mm: pgtable: move __tlb_remove_table_one() in x86 to generic file Date: Sat, 14 Dec 2024 17:02:58 +0800 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The __tlb_remove_table_one() in x86 does not contain architecture-specific content, so move it to the generic file. Signed-off-by: Qi Zheng Suggested-by: Peter Zijlstra (Intel) --- arch/x86/include/asm/tlb.h | 19 ------------------- mm/mmu_gather.c | 20 ++++++++++++++++++-- 2 files changed, 18 insertions(+), 21 deletions(-) diff --git a/arch/x86/include/asm/tlb.h b/arch/x86/include/asm/tlb.h index 3858dbf75880e..77f52bc1578a7 100644 --- a/arch/x86/include/asm/tlb.h +++ b/arch/x86/include/asm/tlb.h @@ -20,25 +20,6 @@ static inline void tlb_flush(struct mmu_gather *tlb) flush_tlb_mm_range(tlb->mm, start, end, stride_shift, tlb->freed_tables); } =20 -#ifdef CONFIG_PT_RECLAIM -static inline void __tlb_remove_table_one_rcu(struct rcu_head *head) -{ - struct ptdesc *ptdesc; - - ptdesc =3D container_of(head, struct ptdesc, pt_rcu_head); - __tlb_remove_table(ptdesc); -} - -static inline void __tlb_remove_table_one(void *table) -{ - struct ptdesc *ptdesc; - - ptdesc =3D table; - call_rcu(&ptdesc->pt_rcu_head, __tlb_remove_table_one_rcu); -} -#define __tlb_remove_table_one __tlb_remove_table_one -#endif /* CONFIG_PT_RECLAIM */ - static inline void invlpg(unsigned long addr) { asm volatile("invlpg (%0)" ::"r" (addr) : "memory"); diff --git a/mm/mmu_gather.c b/mm/mmu_gather.c index 1e21022bcf339..7aa6f18c500b2 100644 --- a/mm/mmu_gather.c +++ b/mm/mmu_gather.c @@ -311,13 +311,29 @@ static inline void tlb_table_invalidate(struct mmu_ga= ther *tlb) } } =20 -#ifndef __tlb_remove_table_one +#ifdef CONFIG_PT_RECLAIM +static inline void __tlb_remove_table_one_rcu(struct rcu_head *head) +{ + struct ptdesc *ptdesc; + + ptdesc =3D container_of(head, struct ptdesc, pt_rcu_head); + __tlb_remove_table(ptdesc); +} + +static inline void __tlb_remove_table_one(void *table) +{ + struct ptdesc *ptdesc; + + ptdesc =3D table; + call_rcu(&ptdesc->pt_rcu_head, __tlb_remove_table_one_rcu); +} +#else static inline void __tlb_remove_table_one(void *table) { tlb_remove_table_sync_one(); __tlb_remove_table(table); } -#endif +#endif /* CONFIG_PT_RECLAIM */ =20 static void tlb_remove_table_one(void *table) { --=20 2.20.1