From nobody Thu Dec 18 09:39:52 2025 Received: from mail-pl1-f181.google.com (mail-pl1-f181.google.com [209.85.214.181]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 33BBC157A5A for ; Wed, 18 Dec 2024 13:05:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.181 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734527116; cv=none; b=b/wC+ClF08e33iShQIJHI9N+XLDLGM4hd9Su1i89MsB9wwrs4gginl+jNvFeqSND9+6BI99zCAcXEUsI3WKFali5h6yMnpt8bIwKYQKE2TCwzOLiZukMHDfvgk6ajj14fV3sIgGcpVfBvYun+8tiN8GMg+6H+pzyIN5KwwZphw8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734527116; c=relaxed/simple; bh=Mv4aqDFHPoE9y9SKG/iLfdxgHrDObWP1H6FMZyzK5gU=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=eI60uxc20NPz3alZ0uqLzGpgJXNMtHcacq7Tlyp9wgAiGNLF5z5z87+q7ss1T2gMrByaqQ6eX/MDSzTIeZXfAzbhxCvSJPGAM+gX00LKbk6sjq+nAxD7LCtHfVr9EHnJmDvuCqxmv1TAcrNK1+C9409M8DA7dSomQqjrcErO3Lc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=kvyLDT3V; arc=none smtp.client-ip=209.85.214.181 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="kvyLDT3V" Received: by mail-pl1-f181.google.com with SMTP id d9443c01a7336-218c8aca5f1so23390275ad.0 for ; Wed, 18 Dec 2024 05:05:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1734527114; x=1735131914; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=37d3136MH/Lj3ZUUrUTIyNwrWsJuTyPww/s07XJQrz8=; b=kvyLDT3Vb9aNVznxjMdUs8pW/kuNGSQTiUuGxfHIsIexuEwf5sxSva4G6PEhIajUaI ryEhLAWyws3kPipQSt8Zm4JOND8DOYEL3QSrvTBq+8cZIzXiR0o+2foiQcT4JmXqJEMW eG6scdnK72EeOFM+VxnDlRv9z668q6DSaLCN12O9JQefRfQVTMqZnWKO0R15fFU4GYR1 j80AzWfn8nG9jiTE3Ykcs3yqJnRvhfNLiigskcaJzmoWuCNy715lMxDuNDSZtGyutHXD dmCTLmGHdCAhjDV/0mvnnCrKGVTaf3VJ3J/7ACfeBO6T2a4sQ+Ons1WrurDRbmQjJ4zI sJsA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734527114; x=1735131914; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=37d3136MH/Lj3ZUUrUTIyNwrWsJuTyPww/s07XJQrz8=; b=ahPytBfI0g+1Za2ZWXWHEHtaXyw+qDXSi3XhU69uNidyZApiCOi7XgAyAVzQPEAnEa w49DspESJyFZKWQnuHvvRtJd8jAyJRkHpGGBPxjzzpTeVlAn0rzek8i7hgWwKe5SHnLg qzVPS+FjWQSqwYjnJ5+2xVffbUCKc85GhOIb/lf3V8sfZc2wD+Rf77J+xQnMZYSr1zx6 kl43USpskahstvO37+PNgHY+7dBmsOSypVNqi8Av5yc/j9NYrXRopKXcHQPRWLMTxE99 GoDzwzHgQuLxHrtAbC3tcKECZWwbcfaSa+sR21Qut4G33h7x598MQL78ekp0hc1f5s1p opYw== X-Forwarded-Encrypted: i=1; AJvYcCXLWlaC3DCp3eyVlmfzls7SugcLkltHy8Co2lKE5N2xwE6rX7CYaQOCZ7q2CLGyiK2haC/yXbnG7a0BUq4=@vger.kernel.org X-Gm-Message-State: AOJu0YyPjlEFG9yVmxpkmlJvn6BO/Ox1J5sXi0trXtWnIwyzP2fFKMhq Uk31f2W/JUbYCTnIlRBvMKP8mRftUp9M2iOozmg6f4u+J9bDkYG5r6BaJO+G27M= X-Gm-Gg: ASbGnct2r241tUj5bwcS1xUNwvYW3diXpkB9sn13ONF7QQwYIpRDolVpBACx/bJan0G aWIRLse7Bn4L3PLzoOZJ3WIdrfsblh64LEPxA7LP3Ygcg+a5F9er51oxU+p41Ue3+MVWb+xa0Ia kMGGypPqbP1PCvWwyJFmIOa3mD1DdoT6ZDabvR5b6n4xYn1ya6xDJCFgvpM3WxgToETkX0KKu9T Jwt3e9pJmqMcsrlv07J+3ZCt9bnq52qPwhIUHHB91xv5frQfqt38rZBaVr9xnOGVMeQV05r59wZ ACQMBcvm3TI6gb3KL0LefA== X-Google-Smtp-Source: AGHT+IGVLgY8vFLZqjIL3oQPZehgU5XP1d4YxVu/+EapWtzKcY/+w347Rqmd9s+hVdjxdsfNE1spFQ== X-Received: by 2002:a17:902:ef0f:b0:216:6855:164 with SMTP id d9443c01a7336-218d7252ae5mr34612065ad.39.1734527114526; Wed, 18 Dec 2024 05:05:14 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.238]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-218a1db755dsm75751825ad.42.2024.12.18.05.05.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 18 Dec 2024 05:05:13 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng Subject: [PATCH v2 01/15] Revert "mm: pgtable: make ptlock be freed by RCU" Date: Wed, 18 Dec 2024 21:04:37 +0800 Message-Id: <914f1b9359d5547c08a30908e8c970fdf463e210.1734526570.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This reverts commit 2f3443770437e49abc39af26962d293851cbab6d. Signed-off-by: Qi Zheng --- include/linux/mm.h | 2 +- include/linux/mm_types.h | 9 +-------- mm/memory.c | 22 ++++++---------------- 3 files changed, 8 insertions(+), 25 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 024bfb3ca9152..9d58f89e9643e 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -3019,7 +3019,7 @@ void ptlock_free(struct ptdesc *ptdesc); =20 static inline spinlock_t *ptlock_ptr(struct ptdesc *ptdesc) { - return &(ptdesc->ptl->ptl); + return ptdesc->ptl; } #else /* ALLOC_SPLIT_PTLOCKS */ static inline void ptlock_cache_init(void) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index f73645627fd92..e2139af51e050 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -435,13 +435,6 @@ FOLIO_MATCH(flags, _flags_2a); FOLIO_MATCH(compound_head, _head_2a); #undef FOLIO_MATCH =20 -#if ALLOC_SPLIT_PTLOCKS -struct pt_lock { - spinlock_t ptl; - struct rcu_head rcu; -}; -#endif - /** * struct ptdesc - Memory descriptor for page tables. * @__page_flags: Same as page flags. Powerpc only. @@ -490,7 +483,7 @@ struct ptdesc { union { unsigned long _pt_pad_2; #if ALLOC_SPLIT_PTLOCKS - struct pt_lock *ptl; + spinlock_t *ptl; #else spinlock_t ptl; #endif diff --git a/mm/memory.c b/mm/memory.c index fed097ebedcbc..e064f5edb7c56 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -7087,34 +7087,24 @@ static struct kmem_cache *page_ptl_cachep; =20 void __init ptlock_cache_init(void) { - page_ptl_cachep =3D kmem_cache_create("page->ptl", sizeof(struct pt_lock)= , 0, + page_ptl_cachep =3D kmem_cache_create("page->ptl", sizeof(spinlock_t), 0, SLAB_PANIC, NULL); } =20 bool ptlock_alloc(struct ptdesc *ptdesc) { - struct pt_lock *pt_lock; + spinlock_t *ptl; =20 - pt_lock =3D kmem_cache_alloc(page_ptl_cachep, GFP_KERNEL); - if (!pt_lock) + ptl =3D kmem_cache_alloc(page_ptl_cachep, GFP_KERNEL); + if (!ptl) return false; - ptdesc->ptl =3D pt_lock; + ptdesc->ptl =3D ptl; return true; } =20 -static void ptlock_free_rcu(struct rcu_head *head) -{ - struct pt_lock *pt_lock; - - pt_lock =3D container_of(head, struct pt_lock, rcu); - kmem_cache_free(page_ptl_cachep, pt_lock); -} - void ptlock_free(struct ptdesc *ptdesc) { - struct pt_lock *pt_lock =3D ptdesc->ptl; - - call_rcu(&pt_lock->rcu, ptlock_free_rcu); + kmem_cache_free(page_ptl_cachep, ptdesc->ptl); } #endif =20 --=20 2.20.1 From nobody Thu Dec 18 09:39:52 2025 Received: from mail-pl1-f178.google.com (mail-pl1-f178.google.com [209.85.214.178]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 520281A23BC for ; Wed, 18 Dec 2024 13:05:23 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.178 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734527125; cv=none; b=Oy9xddlArLeqjlwknmt6Dyw5PIQx2nCHT//YlknclmGTsWvqd9HNyfSuaEIyGuAFdBbATBIoLboowx04ezKo4aKZto3L6ctEZZEULYvUceSeUMCuMnsw0mlp/kLztmCazS/CABf5vuedEAc2SpwDyNCbY7Jm+lGui44rqrbyl+o= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734527125; c=relaxed/simple; bh=Ti9Lo8T5GTRAmE8N1daT8ak1jSzvn9zg2U/AuMVKQNI=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=cVvDDzYPwOy0v8mdac9zBQQ+bf5+uZVbalP2CwqOUouRcyX6jkVXoEzfPiWQ0j7+2Knu0CsWQyHa0+fYKlFoZanT/6N6mmsZE/RAu2ZnX6IRwJDgawt86fash56ydXnrbp65/f+48eYiIfHa5k1zKs26pMVY7+RVjWAJSS/MRHM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=aHSMNh9s; arc=none smtp.client-ip=209.85.214.178 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="aHSMNh9s" Received: by mail-pl1-f178.google.com with SMTP id d9443c01a7336-2165cb60719so52140415ad.0 for ; Wed, 18 Dec 2024 05:05:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1734527122; x=1735131922; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=V1U3ICCbfOFNMbev/k3EdvOWA+vXPbG14S+K01SHhE0=; b=aHSMNh9soVzRp9U6ldwWI17HLY5/EFtdGAojIdw06/i0FgLXko50kAsyRWorhIecxw oR4M4U0fbjJBwur/zTPsO+cJeF6zqcFfM5/nYP0/5iBVjIDCUpoMQbSk1MrHJ8wYwGhF EplqxdaznfI3KsYlf1EV6lWJZUxrUW8x7i9KKIgzGr819MapSTQtRk7rxboQWFKg0qZ7 1yCLskHIwjy/fhdo/Dv60l/SsZIlSYHZndN2pXylERvGlPv9DBE4cXi1j8f8UaHmyVPr cUroOCu3MiouRELCyfWsChBNjaUjJtUf47UfNX7+4Fy8HD0gOs5Sd5mtxQdFTqNBAhw+ XC6A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734527123; x=1735131923; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=V1U3ICCbfOFNMbev/k3EdvOWA+vXPbG14S+K01SHhE0=; b=m/jcpbtoG5tVWazeH39lSYgGzNGSnrRBRa/XCRHjmFGGR7jrpy4QX+bE1ChN6msPzD HRbB1y9GzNikVDX00O/KbCQYqI2gmzAfHg8fvdjOK0P6OS+DwWzC/PuKYG0LGKR2AI9H j3ahSv39daGRX5ks4o0sRCzyqhymjCEV6xop35eRM+8PoP4XoUOQ/9EH8U7cIl6nc0Wx +gHpQW25m4Ajqn3xhLXMUdqc0PL4KbgZkBxZ9vYhONcVnEa5HgBkCYHKxvTDLA9JaHAx 8tAZYZEI3fLlq4+deTzS2uq6MrHoiY/N2ZYOgpIqucTj408FLjLkD0QkskvDxCyNVzKH ABaA== X-Forwarded-Encrypted: i=1; AJvYcCVg3HRzILYqzGx8FnxFNwKPeLfRnP9ZjEccIgqpmArABqns+lrEZqzhqmWocRkwGH9mYztiMs+0QgTccCA=@vger.kernel.org X-Gm-Message-State: AOJu0YxShhjI2RW9MB47dok1GgE0hm38EuQ90gHtqjTaXveZmSnW2t/x N9Sskzt9cVojN3Qu3xjdPgAnlEb+TYxCkjpH4YbFhTqC5QjsbHMZSensVyKjSho= X-Gm-Gg: ASbGncvUTVjkHV0n4UTA/0pNHEQmqIscaqrpYZ66m0TtizUr7aV9bbG9zKYWrhTxBdK zXYimN7rphg9vWMoOXLyGthJkOYnxq4p42YkRGyGb+EK9UYFiTezpzSGuSZ3c+dMF7BrAzfEZ0l hWCLv7N/JA3dtPiOnk4//qk8aL4LHXI2CTplgpp3WqH57EIJ1Ub7T4wHyBk8ye18eV74cE+XJGv uv1bVpM2tKBs+5tIfr4Y9qc5YLuOlThwQo0jSyoL9ywcKRI0jYyhUHQyb8yG0+3ck2Zh7IUujoI Y6vfOW6vApdQ+LdiLV+qZQ== X-Google-Smtp-Source: AGHT+IHf2m598v0e4o8VjiXswnpFdHX9IrqnwvlyprzzHZEN7MlhmaMV6JPA0vL7pLwTvqcblsUT+g== X-Received: by 2002:a17:902:fac5:b0:216:6769:9ed7 with SMTP id d9443c01a7336-218d724960fmr35428695ad.40.1734527122502; Wed, 18 Dec 2024 05:05:22 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.238]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-218a1db755dsm75751825ad.42.2024.12.18.05.05.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 18 Dec 2024 05:05:22 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng Subject: [PATCH v2 02/15] mm: pgtable: introduce generic p4d_alloc_one() and p4d_free() Date: Wed, 18 Dec 2024 21:04:38 +0800 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Several architectures (arm64, riscv, x86) define p4d_alloc_one() as a wrapper for get_zeroed_page() and p4d_free() as a wrapper for free_page(). For these architectures, provide a generic implementation in asm-generic/pgalloc.h and convert them to use it. And like other levels of page tables, add statistics for P4D level page table. For s390, it also defines p4d_alloc_one() and p4d_free(), but it uses its own logic, so skip it. Signed-off-by: Qi Zheng Originally-by: Peter Zijlstra (Intel) --- arch/arm64/include/asm/pgalloc.h | 15 ++++----- arch/riscv/include/asm/pgalloc.h | 25 ++++++--------- arch/x86/include/asm/pgalloc.h | 16 ++++------ arch/x86/mm/pgtable.c | 3 ++ include/asm-generic/pgalloc.h | 55 ++++++++++++++++++++++++++++++++ include/linux/mm.h | 16 ++++++++++ 6 files changed, 98 insertions(+), 32 deletions(-) diff --git a/arch/arm64/include/asm/pgalloc.h b/arch/arm64/include/asm/pgal= loc.h index e75422864d1bd..679c530549327 100644 --- a/arch/arm64/include/asm/pgalloc.h +++ b/arch/arm64/include/asm/pgalloc.h @@ -15,6 +15,8 @@ =20 #define __HAVE_ARCH_PGD_FREE #define __HAVE_ARCH_PUD_FREE +#define __HAVE_ARCH_P4D_ALLOC_ONE +#define __HAVE_ARCH_P4D_FREE #include =20 #define PGD_SIZE (PTRS_PER_PGD * sizeof(pgd_t)) @@ -87,19 +89,16 @@ static inline void pgd_populate(struct mm_struct *mm, p= gd_t *pgdp, p4d_t *p4dp) =20 static inline p4d_t *p4d_alloc_one(struct mm_struct *mm, unsigned long add= r) { - gfp_t gfp =3D GFP_PGTABLE_USER; + if (!pgtable_l5_enabled()) + return NULL; =20 - if (mm =3D=3D &init_mm) - gfp =3D GFP_PGTABLE_KERNEL; - return (p4d_t *)get_zeroed_page(gfp); + return __p4d_alloc_one(mm, addr); } =20 static inline void p4d_free(struct mm_struct *mm, p4d_t *p4d) { - if (!pgtable_l5_enabled()) - return; - BUG_ON((unsigned long)p4d & (PAGE_SIZE-1)); - free_page((unsigned long)p4d); + if (pgtable_l5_enabled()) + __p4d_free(mm, p4d); } =20 #define __p4d_free_tlb(tlb, p4d, addr) p4d_free((tlb)->mm, p4d) diff --git a/arch/riscv/include/asm/pgalloc.h b/arch/riscv/include/asm/pgal= loc.h index f52264304f772..bb6e1c5f1fb19 100644 --- a/arch/riscv/include/asm/pgalloc.h +++ b/arch/riscv/include/asm/pgalloc.h @@ -14,6 +14,8 @@ #ifdef CONFIG_MMU #define __HAVE_ARCH_PUD_ALLOC_ONE #define __HAVE_ARCH_PUD_FREE +#define __HAVE_ARCH_P4D_ALLOC_ONE +#define __HAVE_ARCH_P4D_FREE #include =20 static inline void riscv_tlb_remove_ptdesc(struct mmu_gather *tlb, void *p= t) @@ -118,21 +120,10 @@ static inline void __pud_free_tlb(struct mmu_gather *= tlb, pud_t *pud, #define p4d_alloc_one p4d_alloc_one static inline p4d_t *p4d_alloc_one(struct mm_struct *mm, unsigned long add= r) { - if (pgtable_l5_enabled) { - gfp_t gfp =3D GFP_PGTABLE_USER; - - if (mm =3D=3D &init_mm) - gfp =3D GFP_PGTABLE_KERNEL; - return (p4d_t *)get_zeroed_page(gfp); - } + if (!pgtable_l5_enabled) + return NULL; =20 - return NULL; -} - -static inline void __p4d_free(struct mm_struct *mm, p4d_t *p4d) -{ - BUG_ON((unsigned long)p4d & (PAGE_SIZE-1)); - free_page((unsigned long)p4d); + return __p4d_alloc_one(mm, addr); } =20 #define p4d_free p4d_free @@ -145,8 +136,12 @@ static inline void p4d_free(struct mm_struct *mm, p4d_= t *p4d) static inline void __p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d, unsigned long addr) { - if (pgtable_l5_enabled) + if (pgtable_l5_enabled) { + struct ptdesc *ptdesc =3D virt_to_ptdesc(p4d); + + pagetable_p4d_dtor(ptdesc); riscv_tlb_remove_ptdesc(tlb, virt_to_ptdesc(p4d)); + } } #endif /* __PAGETABLE_PMD_FOLDED */ =20 diff --git a/arch/x86/include/asm/pgalloc.h b/arch/x86/include/asm/pgalloc.h index dcd836b59bebd..d9bc6cae77c9e 100644 --- a/arch/x86/include/asm/pgalloc.h +++ b/arch/x86/include/asm/pgalloc.h @@ -8,6 +8,8 @@ =20 #define __HAVE_ARCH_PTE_ALLOC_ONE #define __HAVE_ARCH_PGD_FREE +#define __HAVE_ARCH_P4D_ALLOC_ONE +#define __HAVE_ARCH_P4D_FREE #include =20 static inline int __paravirt_pgd_alloc(struct mm_struct *mm) { return 0; } @@ -149,20 +151,16 @@ static inline void pgd_populate_safe(struct mm_struct= *mm, pgd_t *pgd, p4d_t *p4 =20 static inline p4d_t *p4d_alloc_one(struct mm_struct *mm, unsigned long add= r) { - gfp_t gfp =3D GFP_KERNEL_ACCOUNT; + if (!pgtable_l5_enabled()) + return NULL; =20 - if (mm =3D=3D &init_mm) - gfp &=3D ~__GFP_ACCOUNT; - return (p4d_t *)get_zeroed_page(gfp); + return __p4d_alloc_one(mm, addr); } =20 static inline void p4d_free(struct mm_struct *mm, p4d_t *p4d) { - if (!pgtable_l5_enabled()) - return; - - BUG_ON((unsigned long)p4d & (PAGE_SIZE-1)); - free_page((unsigned long)p4d); + if (pgtable_l5_enabled()) + return __p4d_free(mm, p4d); } =20 extern void ___p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d); diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c index 69a357b15974a..3d6e84da45b24 100644 --- a/arch/x86/mm/pgtable.c +++ b/arch/x86/mm/pgtable.c @@ -94,6 +94,9 @@ void ___pud_free_tlb(struct mmu_gather *tlb, pud_t *pud) #if CONFIG_PGTABLE_LEVELS > 4 void ___p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d) { + struct ptdesc *ptdesc =3D virt_to_ptdesc(p4d); + + pagetable_p4d_dtor(ptdesc); paravirt_release_p4d(__pa(p4d) >> PAGE_SHIFT); paravirt_tlb_remove_table(tlb, virt_to_page(p4d)); } diff --git a/include/asm-generic/pgalloc.h b/include/asm-generic/pgalloc.h index 7c48f5fbf8aa7..dbf61819b3581 100644 --- a/include/asm-generic/pgalloc.h +++ b/include/asm-generic/pgalloc.h @@ -215,6 +215,61 @@ static inline void pud_free(struct mm_struct *mm, pud_= t *pud) =20 #endif /* CONFIG_PGTABLE_LEVELS > 3 */ =20 +#if CONFIG_PGTABLE_LEVELS > 4 + +static inline p4d_t *__p4d_alloc_one_noprof(struct mm_struct *mm, unsigned= long addr) +{ + gfp_t gfp =3D GFP_PGTABLE_USER; + struct ptdesc *ptdesc; + + if (mm =3D=3D &init_mm) + gfp =3D GFP_PGTABLE_KERNEL; + gfp &=3D ~__GFP_HIGHMEM; + + ptdesc =3D pagetable_alloc_noprof(gfp, 0); + if (!ptdesc) + return NULL; + + pagetable_p4d_ctor(ptdesc); + return ptdesc_address(ptdesc); +} +#define __p4d_alloc_one(...) alloc_hooks(__p4d_alloc_one_noprof(__VA_ARGS_= _)) + +#ifndef __HAVE_ARCH_P4D_ALLOC_ONE +/** + * p4d_alloc_one - allocate memory for a P4D-level page table + * @mm: the mm_struct of the current context + * + * Allocate memory for a page table using %GFP_PGTABLE_USER for user conte= xt + * and %GFP_PGTABLE_KERNEL for kernel context. + * + * Return: pointer to the allocated memory or %NULL on error + */ +static inline p4d_t *p4d_alloc_one_noprof(struct mm_struct *mm, unsigned l= ong addr) +{ + return __p4d_alloc_one_noprof(mm, addr); +} +#define p4d_alloc_one(...) alloc_hooks(p4d_alloc_one_noprof(__VA_ARGS__)) +#endif + +static inline void __p4d_free(struct mm_struct *mm, p4d_t *p4d) +{ + struct ptdesc *ptdesc =3D virt_to_ptdesc(p4d); + + BUG_ON((unsigned long)p4d & (PAGE_SIZE-1)); + pagetable_p4d_dtor(ptdesc); + pagetable_free(ptdesc); +} + +#ifndef __HAVE_ARCH_P4D_FREE +static inline void p4d_free(struct mm_struct *mm, pud_t *p4d) +{ + __p4d_free(mm, p4d); +} +#endif + +#endif + #ifndef __HAVE_ARCH_PGD_FREE static inline void pgd_free(struct mm_struct *mm, pgd_t *pgd) { diff --git a/include/linux/mm.h b/include/linux/mm.h index 9d58f89e9643e..7168f46ad4770 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -3269,6 +3269,22 @@ static inline void pagetable_pud_dtor(struct ptdesc = *ptdesc) lruvec_stat_sub_folio(folio, NR_PAGETABLE); } =20 +static inline void pagetable_p4d_ctor(struct ptdesc *ptdesc) +{ + struct folio *folio =3D ptdesc_folio(ptdesc); + + __folio_set_pgtable(folio); + lruvec_stat_add_folio(folio, NR_PAGETABLE); +} + +static inline void pagetable_p4d_dtor(struct ptdesc *ptdesc) +{ + struct folio *folio =3D ptdesc_folio(ptdesc); + + __folio_clear_pgtable(folio); + lruvec_stat_sub_folio(folio, NR_PAGETABLE); +} + extern void __init pagecache_init(void); extern void free_initmem(void); =20 --=20 2.20.1 From nobody Thu Dec 18 09:39:52 2025 Received: from mail-pl1-f170.google.com (mail-pl1-f170.google.com [209.85.214.170]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 306211A9B45 for ; Wed, 18 Dec 2024 13:05:30 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.170 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734527133; cv=none; b=Rd8BWlEu4jEDT3X3ThsoedBvDz0QCWaijY/exvvuiUWE9/VhAZbQJwhacVSxWKZMQ6OI9Rns39DeZaMVZJBrSqkB7KJCe+I+y5y8F1T/+GbX/ulAiFT8SbMBbxtqDVJIw80hzJQoOu/C4kgMU6sscGGpiaZntrad/vFfLJQJ9Ww= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734527133; c=relaxed/simple; bh=0I+QujqJYcMfTXp+SDj87j/Ufo0cc3GvWv0DHo8HPg4=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=Uiq0OfZZjSOusC1hOB5F+sKNlGYhRrjXK0vTsa739KGQwrd5apEW7LDeolFUkACJ1rRzI5qSA2oyjA005VpubwpFz31fwOPKws4VZ0Ifz+2F9d1ct5wN8HjHnF2GbkoRbcH+ed+jqtrcCaQZe4iFYjJ6OoQfNPUc24no9X0ZZ/w= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=QwakrvzG; arc=none smtp.client-ip=209.85.214.170 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="QwakrvzG" Received: by mail-pl1-f170.google.com with SMTP id d9443c01a7336-215770613dbso43926635ad.2 for ; Wed, 18 Dec 2024 05:05:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1734527130; x=1735131930; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=hRj0wkIcDyLmKPONLgLEnODGJVOQ/lxkeDhrLguGMRU=; b=QwakrvzGeOtDAhoMk7lS6yrRYlUxBgr8N4lg/jpkFpGhWalPfk9OUnhnRVhI8NkNyO FHa4geNBLlRlYOYbYFt5V9RtMqN1z0Z9gvBByNeeWxCo5GmlxO4vVEx1Zi2A4uda3SSc NWOs/MAGVs4TgvltwLoBNar34n0Yw7lvh+EAkPwf6rLPMwWSO9M6Bg/2eSxXx0fhYWBy BH92oSNMD9Hvj6FrghwnciZwYzIIpQwJxd0z4EiULc50xNmJ4d/MQMsAy2nwt+EvFsLW oM6aNujn/fYwFDgNt+N5nJjewuyivw13vtBYUKu+qWSPOdtWFQXN0iHPPY9pQFvGZgvx RgJA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734527130; x=1735131930; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=hRj0wkIcDyLmKPONLgLEnODGJVOQ/lxkeDhrLguGMRU=; b=h1LZIYXutv5UVGKfNCULYXFMJkjdtAHUdPLrsuPVrAqmh0bDWFcibWyoZRYIzmUpYA 8VudZCnxHhzzikn3/vx6jh/Z8clsdoFA6bsMzP3DtEql0SUZCsV97ZOFpzOgy/OjlRdp KetVGUYnx94LYf1TITpojGQO+6TmTRasxqftT5mK70qrXf9p9lVlFp5LP8ja9J9Tf8zO CoTRepKMqQiHq29mFeIcoXkzk7j1D4bCPMINo85/lycrUqARKoZDjx18uKn16yuspcRn AayTqHwomlkFxXYyZtRBhgNRXN5cQSgwCJgteG8ZkyilYzL+xTQwe6+HICKKXDogkno7 pLTw== X-Forwarded-Encrypted: i=1; AJvYcCV08s6C1XLn/wGfvXNRI9URAJqaNGwLgjNHBmn/zYmNGWUrq6SAED679TFanPdSj5yageK7oUsIywA2LBk=@vger.kernel.org X-Gm-Message-State: AOJu0YyR9baf1xwvl33L6JGlOnRLtZz9MTohmA1zkkcC7nJlDq9ov1AP vcFppn9SY2c0uaYemCmL78U6s8HMLyh60nVHNOBRT2S49CLvn9ISF13JVJGOoZ0= X-Gm-Gg: ASbGncuaNU3trphVa6aY/Pua+8UTUHUycMQ8/U9hZJLoL6S+OnvFe91BONR3nLDyTdG Y/qiO83WicHpGv47u/rydOjLnPEWD9TyY9gUItrzudia+6kQwALZprkz4G4KMBZRPFp5Ftm5eoR ysZ6mN0XVhnf7ehGCWoQKszXYbINFtGnOexaSI28kenvdjZhQGmXk3ISZo5rI9fAJDXJPHuFr8G JDWUrouNto13//f5k/YUQPSloDEVF7YokD8602wRNZP0qpIwolRoPzTCp2e43C//4eP+hl7lhXO c5SIWn+WvgKjWPeneo/bPg== X-Google-Smtp-Source: AGHT+IEbMQUiOWkQtVsukFUm2ZLuTFY97EKfs/PZuqMM+WxRQxlasXxPYQ7Z03KXqwqNdfJrdRWLVA== X-Received: by 2002:a17:902:e5ca:b0:215:4394:40b5 with SMTP id d9443c01a7336-218d724abe7mr34796235ad.43.1734527130325; Wed, 18 Dec 2024 05:05:30 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.238]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-218a1db755dsm75751825ad.42.2024.12.18.05.05.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 18 Dec 2024 05:05:29 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng , linux-arm-kernel@lists.infradead.org Subject: [PATCH v2 03/15] arm64: pgtable: use mmu gather to free p4d level page table Date: Wed, 18 Dec 2024 21:04:39 +0800 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Like other levels of page tables, also use mmu gather mechanism to free p4d level page table. Signed-off-by: Qi Zheng Originally-by: Peter Zijlstra (Intel) Cc: linux-arm-kernel@lists.infradead.org --- arch/arm64/include/asm/pgalloc.h | 2 -- arch/arm64/include/asm/tlb.h | 14 ++++++++++++++ 2 files changed, 14 insertions(+), 2 deletions(-) diff --git a/arch/arm64/include/asm/pgalloc.h b/arch/arm64/include/asm/pgal= loc.h index 679c530549327..58cc260b6bee8 100644 --- a/arch/arm64/include/asm/pgalloc.h +++ b/arch/arm64/include/asm/pgalloc.h @@ -100,8 +100,6 @@ static inline void p4d_free(struct mm_struct *mm, p4d_t= *p4d) if (pgtable_l5_enabled()) __p4d_free(mm, p4d); } - -#define __p4d_free_tlb(tlb, p4d, addr) p4d_free((tlb)->mm, p4d) #else static inline void __pgd_populate(pgd_t *pgdp, phys_addr_t p4dp, pgdval_t = prot) { diff --git a/arch/arm64/include/asm/tlb.h b/arch/arm64/include/asm/tlb.h index a947c6e784ed2..445282cde9afb 100644 --- a/arch/arm64/include/asm/tlb.h +++ b/arch/arm64/include/asm/tlb.h @@ -111,4 +111,18 @@ static inline void __pud_free_tlb(struct mmu_gather *t= lb, pud_t *pudp, } #endif =20 +#if CONFIG_PGTABLE_LEVELS > 4 +static inline void __p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4dp, + unsigned long addr) +{ + struct ptdesc *ptdesc =3D virt_to_ptdesc(p4dp); + + if (!pgtable_l5_enabled()) + return; + + pagetable_p4d_dtor(ptdesc); + tlb_remove_ptdesc(tlb, ptdesc); +} +#endif + #endif --=20 2.20.1 From nobody Thu Dec 18 09:39:52 2025 Received: from mail-pl1-f169.google.com (mail-pl1-f169.google.com [209.85.214.169]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6EE021B0424 for ; Wed, 18 Dec 2024 13:05:39 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.169 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734527141; cv=none; b=swvuSY9gorLWzj/mg+zOv3OzCA59mtcSzgbEwEB+PtySTY786cLsq7+Y2Y6GVBoF1PnS60OaJ2ziZFvR9w2UalX5VubpsolWS/Cxg98bSaUbLZOlJLK5BDzhV6pmzaMcyGuU84BEapE/8zlsh0AhVr7311OKidE+ICI14UiBEsk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734527141; c=relaxed/simple; bh=gsn2MxaDHMIkVDeNJoAZ1GXzg4UbDMQdMp9AlgDJXWI=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=QDggprYVkBXIRRa61+0VcC7Bwc0/eeKZ42Ivla56z11HZCdRlq7mAhc0ck+lZoZ6m+e+ge4n5Cp+27Hzl9XENwaRZJ4bPLudoH3QCAfBBGOGdEA20wyHs3e8EtG+F6AeCZSnN74/oHS6LnkAu6fr0LJHkG1OkSsUzRvBd5KDzqQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=IivdJq3z; arc=none smtp.client-ip=209.85.214.169 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="IivdJq3z" Received: by mail-pl1-f169.google.com with SMTP id d9443c01a7336-21636268e43so78249855ad.2 for ; Wed, 18 Dec 2024 05:05:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1734527139; x=1735131939; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=xY41yoAggnrpKZDOvMIy0+FhpPRF9QFn+bDK7HKPzWw=; b=IivdJq3zwTbBWUsPy2MsCP/5HmRbrv/2yQa+fwmHxKVx0Yvj7bbyZ80AM6lvy9O285 cUHneJFlqIG4sTmzj56M7TR87h1kU76i285aRYZtELUEy9prvihgOoxnn1rzDquUwui5 IUAQuYQP90EfzAVlIp26nn/r5g74vfIWxi6zcway74EXyO2FM+W5y5bUscUWk5KJYWsy TEHR7M5ONoh7GK4a1bgRANf/YF0yBUh3XaaMGUQvYkdTLxq4+BXBk0+6RYinhkeb69xQ EdqWqNtultUYStJrcbEKa8esFMU2sF87VBupLBTawrjDcGA0XuK/I/0AqiD8xozFaW8M XuIg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734527139; x=1735131939; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=xY41yoAggnrpKZDOvMIy0+FhpPRF9QFn+bDK7HKPzWw=; b=tIQJT7ayTi46rW5E9qZdePUv9kZ0r9M/8KC2YrNGPo2ROqGpOh856tpfE5cpAIsanf I1we0MLe1CIsZqfN9eBAW5derH6+5MzKOd4QvzcDsr4TfQO7mgmJF6YW/aZt+c/Pl+me UBlgCteW9U30XLZJ65dOcZNXkkwTvIteL35AAL6cmpQ4UQagRDAJKm0PqKhUHKnXgQZL TZa03L7Ut1nsUjNxEpaVnYTY5Hj4Ro+qOQgTH0AB8DYubx0PN2p952PKuY3+AEc3kFoe pTUZ+Y8XVnQZVs2EE0w/jTNGghuc2NisOgEk76YxK8egJDTvVFyQMeEjSPCSkoYs14jU XyKg== X-Forwarded-Encrypted: i=1; AJvYcCWfhqzo4GE+LLFc8rR2NpuITp8a3EJWWhRvZluGK+VxU5e32uDUH+UQtJLc0glZzBQtBn271zf8O+3UCrI=@vger.kernel.org X-Gm-Message-State: AOJu0YyTFi5a4Zc9L04UgrTqok85tLB9m1n8TUboKoWOlsxFK6udQ+1/ sfj/WKY37T2b+fmdtAgOlFs9Ine3h70eQP5oN24XfxjcyxmKFc64I2XqiVQLz8Q= X-Gm-Gg: ASbGnctHjEnzSkURAzGgsWyjts80W2PtbDWHYz2xlCi11QIYMR2RV4IwL8gYr47+tLm 8v5p6/LVb019twnMG4LIIY8YXtw7GH0nex8G48aJ11G+/XGqwEd6eLlLu+85S+s59OvUdmK6BKI Uam+fq6oHZMKzViIJZL7pQIlorURz7NcJ4HmVldvk+58NQJv9/BTjQJ3/E5AFNp6cCUDO4Az7bD t/3uzeFuEgSK1V/ZZUbADCCT8mNFzDRU/jxAAhKytmZB1lkrOy0PTr9GERlbIpQ9mZfZ6wMufxU D/EU9cjh6PVExZmWfBVdNw== X-Google-Smtp-Source: AGHT+IEBex3adiI4KuoLgCHeY1D81j9Q6JTE0T6gwcWqf4Lz8nOg8PmUUMxAQoz8nfmNTBwOZklCxg== X-Received: by 2002:a17:903:2448:b0:216:6769:9eea with SMTP id d9443c01a7336-218d72477d0mr41450855ad.37.1734527138569; Wed, 18 Dec 2024 05:05:38 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.238]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-218a1db755dsm75751825ad.42.2024.12.18.05.05.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 18 Dec 2024 05:05:37 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng , linux-s390@vger.kernel.org Subject: [PATCH v2 04/15] s390: pgtable: add statistics for PUD and P4D level page table Date: Wed, 18 Dec 2024 21:04:40 +0800 Message-Id: <64401d867f3a965f60e9c8d09099b42f21749a47.1734526570.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Like PMD and PTE level page table, also add statistics for PUD and P4D page table. Signed-off-by: Qi Zheng Suggested-by: Peter Zijlstra (Intel) Cc: linux-s390@vger.kernel.org --- arch/s390/include/asm/pgalloc.h | 29 +++++++++++++++++++------- arch/s390/include/asm/tlb.h | 37 +++++++++++++++++---------------- 2 files changed, 40 insertions(+), 26 deletions(-) diff --git a/arch/s390/include/asm/pgalloc.h b/arch/s390/include/asm/pgallo= c.h index 7b84ef6dc4b6d..a0c1ca5d8423c 100644 --- a/arch/s390/include/asm/pgalloc.h +++ b/arch/s390/include/asm/pgalloc.h @@ -53,29 +53,42 @@ static inline p4d_t *p4d_alloc_one(struct mm_struct *mm= , unsigned long address) { unsigned long *table =3D crst_table_alloc(mm); =20 - if (table) - crst_table_init(table, _REGION2_ENTRY_EMPTY); + if (!table) + return NULL; + crst_table_init(table, _REGION2_ENTRY_EMPTY); + pagetable_p4d_ctor(virt_to_ptdesc(table)); + return (p4d_t *) table; } =20 static inline void p4d_free(struct mm_struct *mm, p4d_t *p4d) { - if (!mm_p4d_folded(mm)) - crst_table_free(mm, (unsigned long *) p4d); + if (mm_p4d_folded(mm)) + return; + + pagetable_p4d_dtor(virt_to_ptdesc(p4d)); + crst_table_free(mm, (unsigned long *) p4d); } =20 static inline pud_t *pud_alloc_one(struct mm_struct *mm, unsigned long add= ress) { unsigned long *table =3D crst_table_alloc(mm); - if (table) - crst_table_init(table, _REGION3_ENTRY_EMPTY); + + if (!table) + return NULL; + crst_table_init(table, _REGION3_ENTRY_EMPTY); + pagetable_pud_ctor(virt_to_ptdesc(table)); + return (pud_t *) table; } =20 static inline void pud_free(struct mm_struct *mm, pud_t *pud) { - if (!mm_pud_folded(mm)) - crst_table_free(mm, (unsigned long *) pud); + if (mm_pud_folded(mm)) + return; + + pagetable_pud_dtor(virt_to_ptdesc(pud)); + crst_table_free(mm, (unsigned long *) pud); } =20 static inline pmd_t *pmd_alloc_one(struct mm_struct *mm, unsigned long vma= ddr) diff --git a/arch/s390/include/asm/tlb.h b/arch/s390/include/asm/tlb.h index e95b2c8081eb8..b946964afce8e 100644 --- a/arch/s390/include/asm/tlb.h +++ b/arch/s390/include/asm/tlb.h @@ -110,24 +110,6 @@ static inline void pmd_free_tlb(struct mmu_gather *tlb= , pmd_t *pmd, tlb_remove_ptdesc(tlb, pmd); } =20 -/* - * p4d_free_tlb frees a pud table and clears the CRSTE for the - * region second table entry from the tlb. - * If the mm uses a four level page table the single p4d is freed - * as the pgd. p4d_free_tlb checks the asce_limit against 8PB - * to avoid the double free of the p4d in this case. - */ -static inline void p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d, - unsigned long address) -{ - if (mm_p4d_folded(tlb->mm)) - return; - __tlb_adjust_range(tlb, address, PAGE_SIZE); - tlb->mm->context.flush_mm =3D 1; - tlb->freed_tables =3D 1; - tlb_remove_ptdesc(tlb, p4d); -} - /* * pud_free_tlb frees a pud table and clears the CRSTE for the * region third table entry from the tlb. @@ -140,11 +122,30 @@ static inline void pud_free_tlb(struct mmu_gather *tl= b, pud_t *pud, { if (mm_pud_folded(tlb->mm)) return; + pagetable_pud_dtor(virt_to_ptdesc(pud)); tlb->mm->context.flush_mm =3D 1; tlb->freed_tables =3D 1; tlb->cleared_p4ds =3D 1; tlb_remove_ptdesc(tlb, pud); } =20 +/* + * p4d_free_tlb frees a p4d table and clears the CRSTE for the + * region second table entry from the tlb. + * If the mm uses a four level page table the single p4d is freed + * as the pgd. p4d_free_tlb checks the asce_limit against 8PB + * to avoid the double free of the p4d in this case. + */ +static inline void p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d, + unsigned long address) +{ + if (mm_p4d_folded(tlb->mm)) + return; + pagetable_p4d_dtor(virt_to_ptdesc(p4d)); + __tlb_adjust_range(tlb, address, PAGE_SIZE); + tlb->mm->context.flush_mm =3D 1; + tlb->freed_tables =3D 1; + tlb_remove_ptdesc(tlb, p4d); +} =20 #endif /* _S390_TLB_H */ --=20 2.20.1 From nobody Thu Dec 18 09:39:52 2025 Received: from mail-pl1-f182.google.com (mail-pl1-f182.google.com [209.85.214.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 77B221ACED3 for ; Wed, 18 Dec 2024 13:05:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.182 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734527148; cv=none; b=CoXVqEtXwSPCj189Z4CnrqnE5fgZtUxF3CbzTjDKVRRWCpq4pRI5njh6L8dk1XdY0QziIbNBgWGiWEm5JZ3KabtBZkOpsckXbILpNxiN8eaw+cQPPLZ8/3qt1EqN9EGFqnDKk3R1KPkmn9FfaEjeXFgW0GYjLR5VIoIUggY5cz0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734527148; c=relaxed/simple; bh=NRWp5O5U92hhVfmyy2N0brBAjNWMpLDO2isKg1Dlz3w=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=BYwSTzLT0NLFlM+pQ//ua4wPNLohKkeNofW84CNpsVdv/pvmONKje9pGtYmG3VP1YU1EJi7ubhlCYHulVZoSs6NYnGjPTB4dULtG/MW+p5BNTW2OtCuwi3I0jSRxVNqH0W3ZZ2appyal2ksIW8ZEOCv2Zb0WxLTELGH/HYgcxlk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=awUpxMlB; arc=none smtp.client-ip=209.85.214.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="awUpxMlB" Received: by mail-pl1-f182.google.com with SMTP id d9443c01a7336-216401de828so51710695ad.3 for ; Wed, 18 Dec 2024 05:05:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1734527146; x=1735131946; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=caVYIWOx6/5hl1UH4GU5wSnkiaHbuqRYtNqTt93ePYQ=; b=awUpxMlB5aWlN6BqrA+27gPKncPdc6EPx7u1R3ObzaoZxkjD9SJsZ2EFmai5g1q+pe iDSREho7rIbizPOLrwR3pydU1OJFllDw0K7DaaG6A92YDAY8UqtOA135a6FiSKGcR04+ TUDBM0nH5DLOWZj3AbnodaX2rIp9Ari/qAMx1/M6FQGaxwKf5iCCMpOww5U4u8YvxQVd 7JlfxiVtqMSmjrsL1MXnXRZea712Gr1HEzCGpWsPkslfq8E/2WsqPs//7jCK2qTfOtg4 TjAwFvpu7d9/MiZajzyHP19mVbG4VM0gQ5TmcyV9u0a/HCpGSRgQgNpViGJsExjOpIrY d6nQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734527146; x=1735131946; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=caVYIWOx6/5hl1UH4GU5wSnkiaHbuqRYtNqTt93ePYQ=; b=aFeunDyt7aiLe/M3yzj7KJ2B89FAroQJ0DeGc6vGrAXLZwF2yUtkdnE15uApW7GFFu LgcZtaDUj6wXtWNI4GrhhVNzbunR7HUuaX1duwOdNh5pbPkQ9T56yNKLTfXIETlIitQL Vja+cUBjVv25Vifk7cTNJZCaDnKaJn+nik8Wshd10Umu1QGIdTMBd2sfwBeaIufDP73P iQKUm3WXFfAZY5bFvku+cebdPwV2J3MXbhCNaugl2A8BcW5T5M7eca/+aYftDdAtbybN 9CsKJX40ElSAudU7yZK7i6sQyS2k/8t8bEi0xd7xRfwh0CTs7XMT6yu039rKFo7cAYMi vf/Q== X-Forwarded-Encrypted: i=1; AJvYcCUS02g5+rAvbMcYRSZPJbkaeGz2jYW0FXblMBhuaOJoXK+8PNwQRNQB1SS3cExA9+u/oFqv04EI1C5/AIk=@vger.kernel.org X-Gm-Message-State: AOJu0YybV4bPVMqk6jCQa2p7RAG+3vRjeR2Uja4OW1fWCYbz+D5sa3G2 QzvdxQHx4ywPe/WXj4ZJ20bQCEbw7M6XR3iKUuZM2c4GRmTOWFg8zjpe3xNEE08= X-Gm-Gg: ASbGncsxKBCMnQktpg7cDG/zqgi5BGbxdBSVfxiWrKhzz3k0H5CAKOuA2e4qsLl+La8 WKOs+DomLmD2s2pGH6T9pX/m8YY0ci15nRW9FswNWpAXwjiq7PuTBSvFA7jUbbCSi615JC90yM4 /0hTAHn6tbbP51jC8KiCPxYl0DVFfp36XrEkagF5nGuF9b7s70Ysi13aTUY5IcxNHGwf/vZpVcF QvV7NGf6fX/Pr8fM4X7FtQB4gAq6epSTat60utcrsRiH9KBvU4eXnJzKCuzgCJPXquovoyJw7rP pyN+cy7BvDbtlpVm4p3fFA== X-Google-Smtp-Source: AGHT+IHEov6aeh/QZkGUNU3nLpbBQcGuDGLxiZOzaTrM6oHehxavOqsLv6xgwuZxwOrOKdwEXRtt5g== X-Received: by 2002:a17:902:da8e:b0:215:6f5d:b756 with SMTP id d9443c01a7336-218d6fbe530mr37726365ad.7.1734527145692; Wed, 18 Dec 2024 05:05:45 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.238]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-218a1db755dsm75751825ad.42.2024.12.18.05.05.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 18 Dec 2024 05:05:45 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng Subject: [PATCH v2 05/15] mm: pgtable: introduce pagetable_dtor() Date: Wed, 18 Dec 2024 21:04:41 +0800 Message-Id: <0c00264b336eef6052c229b90bc9e6a32052ebff.1734526570.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The pagetable_p*_dtor() are exactly the same except for the handling of ptlock. If we make ptlock_free() handle the case where ptdesc->ptl is NULL and remove VM_BUG_ON_PAGE() from pmd_ptlock_free(), we can unify pagetable_p*_dtor() into one function. Let's introduce pagetable_dtor() to do this. Later, pagetable_dtor() will be moved to tlb_remove_ptdesc(), so that ptlock and page table pages can be freed together (regardless of whether RCU is used). This prevents the use-after-free problem where the ptlock is freed immediately but the page table pages is freed later via RCU. Signed-off-by: Qi Zheng Originally-by: Peter Zijlstra (Intel) --- Documentation/mm/split_page_table_lock.rst | 4 +- arch/arm/include/asm/tlb.h | 4 +- arch/arm64/include/asm/tlb.h | 8 ++-- arch/csky/include/asm/pgalloc.h | 2 +- arch/hexagon/include/asm/pgalloc.h | 2 +- arch/loongarch/include/asm/pgalloc.h | 2 +- arch/m68k/include/asm/mcf_pgalloc.h | 4 +- arch/m68k/include/asm/sun3_pgalloc.h | 2 +- arch/m68k/mm/motorola.c | 2 +- arch/mips/include/asm/pgalloc.h | 2 +- arch/nios2/include/asm/pgalloc.h | 2 +- arch/openrisc/include/asm/pgalloc.h | 2 +- arch/powerpc/mm/book3s64/mmu_context.c | 2 +- arch/powerpc/mm/book3s64/pgtable.c | 2 +- arch/powerpc/mm/pgtable-frag.c | 4 +- arch/riscv/include/asm/pgalloc.h | 8 ++-- arch/riscv/mm/init.c | 4 +- arch/s390/include/asm/pgalloc.h | 6 +-- arch/s390/include/asm/tlb.h | 6 +-- arch/s390/mm/pgalloc.c | 2 +- arch/sh/include/asm/pgalloc.h | 2 +- arch/sparc/mm/init_64.c | 2 +- arch/sparc/mm/srmmu.c | 2 +- arch/um/include/asm/pgalloc.h | 6 +-- arch/x86/mm/pgtable.c | 12 ++--- include/asm-generic/pgalloc.h | 8 ++-- include/linux/mm.h | 52 ++++------------------ mm/memory.c | 3 +- 28 files changed, 62 insertions(+), 95 deletions(-) diff --git a/Documentation/mm/split_page_table_lock.rst b/Documentation/mm/= split_page_table_lock.rst index 581446d4a4eba..8e1ceb0a6619a 100644 --- a/Documentation/mm/split_page_table_lock.rst +++ b/Documentation/mm/split_page_table_lock.rst @@ -62,7 +62,7 @@ Support of split page table lock by an architecture =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D =20 There's no need in special enabling of PTE split page table lock: everythi= ng -required is done by pagetable_pte_ctor() and pagetable_pte_dtor(), which +required is done by pagetable_pte_ctor() and pagetable_dtor(), which must be called on PTE table allocation / freeing. =20 Make sure the architecture doesn't use slab allocator for page table @@ -73,7 +73,7 @@ PMD split lock only makes sense if you have more than two= page table levels. =20 PMD split lock enabling requires pagetable_pmd_ctor() call on PMD table -allocation and pagetable_pmd_dtor() on freeing. +allocation and pagetable_dtor() on freeing. =20 Allocation usually happens in pmd_alloc_one(), freeing in pmd_free() and pmd_free_tlb(), but make sure you cover all PMD table allocation / freeing diff --git a/arch/arm/include/asm/tlb.h b/arch/arm/include/asm/tlb.h index f40d06ad5d2a3..ef79bf1e8563f 100644 --- a/arch/arm/include/asm/tlb.h +++ b/arch/arm/include/asm/tlb.h @@ -41,7 +41,7 @@ __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pte, uns= igned long addr) { struct ptdesc *ptdesc =3D page_ptdesc(pte); =20 - pagetable_pte_dtor(ptdesc); + pagetable_dtor(ptdesc); =20 #ifndef CONFIG_ARM_LPAE /* @@ -61,7 +61,7 @@ __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmdp, unsig= ned long addr) #ifdef CONFIG_ARM_LPAE struct ptdesc *ptdesc =3D virt_to_ptdesc(pmdp); =20 - pagetable_pmd_dtor(ptdesc); + pagetable_dtor(ptdesc); tlb_remove_ptdesc(tlb, ptdesc); #endif } diff --git a/arch/arm64/include/asm/tlb.h b/arch/arm64/include/asm/tlb.h index 445282cde9afb..408d0f36a8a8f 100644 --- a/arch/arm64/include/asm/tlb.h +++ b/arch/arm64/include/asm/tlb.h @@ -82,7 +82,7 @@ static inline void __pte_free_tlb(struct mmu_gather *tlb,= pgtable_t pte, { struct ptdesc *ptdesc =3D page_ptdesc(pte); =20 - pagetable_pte_dtor(ptdesc); + pagetable_dtor(ptdesc); tlb_remove_ptdesc(tlb, ptdesc); } =20 @@ -92,7 +92,7 @@ static inline void __pmd_free_tlb(struct mmu_gather *tlb,= pmd_t *pmdp, { struct ptdesc *ptdesc =3D virt_to_ptdesc(pmdp); =20 - pagetable_pmd_dtor(ptdesc); + pagetable_dtor(ptdesc); tlb_remove_ptdesc(tlb, ptdesc); } #endif @@ -106,7 +106,7 @@ static inline void __pud_free_tlb(struct mmu_gather *tl= b, pud_t *pudp, if (!pgtable_l4_enabled()) return; =20 - pagetable_pud_dtor(ptdesc); + pagetable_dtor(ptdesc); tlb_remove_ptdesc(tlb, ptdesc); } #endif @@ -120,7 +120,7 @@ static inline void __p4d_free_tlb(struct mmu_gather *tl= b, p4d_t *p4dp, if (!pgtable_l5_enabled()) return; =20 - pagetable_p4d_dtor(ptdesc); + pagetable_dtor(ptdesc); tlb_remove_ptdesc(tlb, ptdesc); } #endif diff --git a/arch/csky/include/asm/pgalloc.h b/arch/csky/include/asm/pgallo= c.h index 9c84c9012e534..f1ce5b7b28f22 100644 --- a/arch/csky/include/asm/pgalloc.h +++ b/arch/csky/include/asm/pgalloc.h @@ -63,7 +63,7 @@ static inline pgd_t *pgd_alloc(struct mm_struct *mm) =20 #define __pte_free_tlb(tlb, pte, address) \ do { \ - pagetable_pte_dtor(page_ptdesc(pte)); \ + pagetable_dtor(page_ptdesc(pte)); \ tlb_remove_page_ptdesc(tlb, page_ptdesc(pte)); \ } while (0) =20 diff --git a/arch/hexagon/include/asm/pgalloc.h b/arch/hexagon/include/asm/= pgalloc.h index 55988625e6fbc..40e42a0e71673 100644 --- a/arch/hexagon/include/asm/pgalloc.h +++ b/arch/hexagon/include/asm/pgalloc.h @@ -89,7 +89,7 @@ static inline void pmd_populate_kernel(struct mm_struct *= mm, pmd_t *pmd, =20 #define __pte_free_tlb(tlb, pte, addr) \ do { \ - pagetable_pte_dtor((page_ptdesc(pte))); \ + pagetable_dtor((page_ptdesc(pte))); \ tlb_remove_page_ptdesc((tlb), (page_ptdesc(pte))); \ } while (0) =20 diff --git a/arch/loongarch/include/asm/pgalloc.h b/arch/loongarch/include/= asm/pgalloc.h index a7b9c9e73593d..7211dff8c969e 100644 --- a/arch/loongarch/include/asm/pgalloc.h +++ b/arch/loongarch/include/asm/pgalloc.h @@ -57,7 +57,7 @@ static inline pte_t *pte_alloc_one_kernel(struct mm_struc= t *mm) =20 #define __pte_free_tlb(tlb, pte, address) \ do { \ - pagetable_pte_dtor(page_ptdesc(pte)); \ + pagetable_dtor(page_ptdesc(pte)); \ tlb_remove_page_ptdesc((tlb), page_ptdesc(pte)); \ } while (0) =20 diff --git a/arch/m68k/include/asm/mcf_pgalloc.h b/arch/m68k/include/asm/mc= f_pgalloc.h index 302c5bf67179e..22d6c1fcabfb4 100644 --- a/arch/m68k/include/asm/mcf_pgalloc.h +++ b/arch/m68k/include/asm/mcf_pgalloc.h @@ -37,7 +37,7 @@ static inline void __pte_free_tlb(struct mmu_gather *tlb,= pgtable_t pgtable, { struct ptdesc *ptdesc =3D virt_to_ptdesc(pgtable); =20 - pagetable_pte_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); } =20 @@ -61,7 +61,7 @@ static inline void pte_free(struct mm_struct *mm, pgtable= _t pgtable) { struct ptdesc *ptdesc =3D virt_to_ptdesc(pgtable); =20 - pagetable_pte_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); } =20 diff --git a/arch/m68k/include/asm/sun3_pgalloc.h b/arch/m68k/include/asm/s= un3_pgalloc.h index 4a137eecb6fe4..2b626cb3ad0ae 100644 --- a/arch/m68k/include/asm/sun3_pgalloc.h +++ b/arch/m68k/include/asm/sun3_pgalloc.h @@ -19,7 +19,7 @@ extern const char bad_pmd_string[]; =20 #define __pte_free_tlb(tlb, pte, addr) \ do { \ - pagetable_pte_dtor(page_ptdesc(pte)); \ + pagetable_dtor(page_ptdesc(pte)); \ tlb_remove_page_ptdesc((tlb), page_ptdesc(pte)); \ } while (0) =20 diff --git a/arch/m68k/mm/motorola.c b/arch/m68k/mm/motorola.c index c1761d309fc61..81715cece70c6 100644 --- a/arch/m68k/mm/motorola.c +++ b/arch/m68k/mm/motorola.c @@ -201,7 +201,7 @@ int free_pointer_table(void *table, int type) list_del(dp); mmu_page_dtor((void *)page); if (type =3D=3D TABLE_PTE) - pagetable_pte_dtor(virt_to_ptdesc((void *)page)); + pagetable_dtor(virt_to_ptdesc((void *)page)); free_page (page); return 1; } else if (ptable_list[type].next !=3D dp) { diff --git a/arch/mips/include/asm/pgalloc.h b/arch/mips/include/asm/pgallo= c.h index f4440edcd8fe2..36d9805033c4b 100644 --- a/arch/mips/include/asm/pgalloc.h +++ b/arch/mips/include/asm/pgalloc.h @@ -56,7 +56,7 @@ static inline void pgd_free(struct mm_struct *mm, pgd_t *= pgd) =20 #define __pte_free_tlb(tlb, pte, address) \ do { \ - pagetable_pte_dtor(page_ptdesc(pte)); \ + pagetable_dtor(page_ptdesc(pte)); \ tlb_remove_page_ptdesc((tlb), page_ptdesc(pte)); \ } while (0) =20 diff --git a/arch/nios2/include/asm/pgalloc.h b/arch/nios2/include/asm/pgal= loc.h index ce6bb8e74271f..12a536b7bfbd4 100644 --- a/arch/nios2/include/asm/pgalloc.h +++ b/arch/nios2/include/asm/pgalloc.h @@ -30,7 +30,7 @@ extern pgd_t *pgd_alloc(struct mm_struct *mm); =20 #define __pte_free_tlb(tlb, pte, addr) \ do { \ - pagetable_pte_dtor(page_ptdesc(pte)); \ + pagetable_dtor(page_ptdesc(pte)); \ tlb_remove_page_ptdesc((tlb), (page_ptdesc(pte))); \ } while (0) =20 diff --git a/arch/openrisc/include/asm/pgalloc.h b/arch/openrisc/include/as= m/pgalloc.h index c6a73772a5466..596e2355824e3 100644 --- a/arch/openrisc/include/asm/pgalloc.h +++ b/arch/openrisc/include/asm/pgalloc.h @@ -68,7 +68,7 @@ extern pte_t *pte_alloc_one_kernel(struct mm_struct *mm); =20 #define __pte_free_tlb(tlb, pte, addr) \ do { \ - pagetable_pte_dtor(page_ptdesc(pte)); \ + pagetable_dtor(page_ptdesc(pte)); \ tlb_remove_page_ptdesc((tlb), (page_ptdesc(pte))); \ } while (0) =20 diff --git a/arch/powerpc/mm/book3s64/mmu_context.c b/arch/powerpc/mm/book3= s64/mmu_context.c index 1715b07c630c9..4e1e45420bd49 100644 --- a/arch/powerpc/mm/book3s64/mmu_context.c +++ b/arch/powerpc/mm/book3s64/mmu_context.c @@ -253,7 +253,7 @@ static void pmd_frag_destroy(void *pmd_frag) count =3D ((unsigned long)pmd_frag & ~PAGE_MASK) >> PMD_FRAG_SIZE_SHIFT; /* We allow PTE_FRAG_NR fragments from a PTE page */ if (atomic_sub_and_test(PMD_FRAG_NR - count, &ptdesc->pt_frag_refcount)) { - pagetable_pmd_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); } } diff --git a/arch/powerpc/mm/book3s64/pgtable.c b/arch/powerpc/mm/book3s64/= pgtable.c index 3745425280808..3f28e4acd920b 100644 --- a/arch/powerpc/mm/book3s64/pgtable.c +++ b/arch/powerpc/mm/book3s64/pgtable.c @@ -477,7 +477,7 @@ void pmd_fragment_free(unsigned long *pmd) =20 BUG_ON(atomic_read(&ptdesc->pt_frag_refcount) <=3D 0); if (atomic_dec_and_test(&ptdesc->pt_frag_refcount)) { - pagetable_pmd_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); } } diff --git a/arch/powerpc/mm/pgtable-frag.c b/arch/powerpc/mm/pgtable-frag.c index e89f64a0f24ae..713268ccb1a0e 100644 --- a/arch/powerpc/mm/pgtable-frag.c +++ b/arch/powerpc/mm/pgtable-frag.c @@ -25,7 +25,7 @@ void pte_frag_destroy(void *pte_frag) count =3D ((unsigned long)pte_frag & ~PAGE_MASK) >> PTE_FRAG_SIZE_SHIFT; /* We allow PTE_FRAG_NR fragments from a PTE page */ if (atomic_sub_and_test(PTE_FRAG_NR - count, &ptdesc->pt_frag_refcount)) { - pagetable_pte_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); } } @@ -111,7 +111,7 @@ static void pte_free_now(struct rcu_head *head) struct ptdesc *ptdesc; =20 ptdesc =3D container_of(head, struct ptdesc, pt_rcu_head); - pagetable_pte_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); } =20 diff --git a/arch/riscv/include/asm/pgalloc.h b/arch/riscv/include/asm/pgal= loc.h index bb6e1c5f1fb19..09c2eff571a49 100644 --- a/arch/riscv/include/asm/pgalloc.h +++ b/arch/riscv/include/asm/pgalloc.h @@ -112,7 +112,7 @@ static inline void __pud_free_tlb(struct mmu_gather *tl= b, pud_t *pud, if (pgtable_l4_enabled) { struct ptdesc *ptdesc =3D virt_to_ptdesc(pud); =20 - pagetable_pud_dtor(ptdesc); + pagetable_dtor(ptdesc); riscv_tlb_remove_ptdesc(tlb, ptdesc); } } @@ -139,7 +139,7 @@ static inline void __p4d_free_tlb(struct mmu_gather *tl= b, p4d_t *p4d, if (pgtable_l5_enabled) { struct ptdesc *ptdesc =3D virt_to_ptdesc(p4d); =20 - pagetable_p4d_dtor(ptdesc); + pagetable_dtor(ptdesc); riscv_tlb_remove_ptdesc(tlb, virt_to_ptdesc(p4d)); } } @@ -172,7 +172,7 @@ static inline void __pmd_free_tlb(struct mmu_gather *tl= b, pmd_t *pmd, { struct ptdesc *ptdesc =3D virt_to_ptdesc(pmd); =20 - pagetable_pmd_dtor(ptdesc); + pagetable_dtor(ptdesc); riscv_tlb_remove_ptdesc(tlb, ptdesc); } =20 @@ -183,7 +183,7 @@ static inline void __pte_free_tlb(struct mmu_gather *tl= b, pgtable_t pte, { struct ptdesc *ptdesc =3D page_ptdesc(pte); =20 - pagetable_pte_dtor(ptdesc); + pagetable_dtor(ptdesc); riscv_tlb_remove_ptdesc(tlb, ptdesc); } #endif /* CONFIG_MMU */ diff --git a/arch/riscv/mm/init.c b/arch/riscv/mm/init.c index fc53ce748c804..8d703fb51b1dc 100644 --- a/arch/riscv/mm/init.c +++ b/arch/riscv/mm/init.c @@ -1558,7 +1558,7 @@ static void __meminit free_pte_table(pte_t *pte_start= , pmd_t *pmd) return; } =20 - pagetable_pte_dtor(ptdesc); + pagetable_dtor(ptdesc); if (PageReserved(page)) free_reserved_page(page); else @@ -1580,7 +1580,7 @@ static void __meminit free_pmd_table(pmd_t *pmd_start= , pud_t *pud, bool is_vmemm } =20 if (!is_vmemmap) - pagetable_pmd_dtor(ptdesc); + pagetable_dtor(ptdesc); if (PageReserved(page)) free_reserved_page(page); else diff --git a/arch/s390/include/asm/pgalloc.h b/arch/s390/include/asm/pgallo= c.h index a0c1ca5d8423c..5fced6d3c36b0 100644 --- a/arch/s390/include/asm/pgalloc.h +++ b/arch/s390/include/asm/pgalloc.h @@ -66,7 +66,7 @@ static inline void p4d_free(struct mm_struct *mm, p4d_t *= p4d) if (mm_p4d_folded(mm)) return; =20 - pagetable_p4d_dtor(virt_to_ptdesc(p4d)); + pagetable_dtor(virt_to_ptdesc(p4d)); crst_table_free(mm, (unsigned long *) p4d); } =20 @@ -87,7 +87,7 @@ static inline void pud_free(struct mm_struct *mm, pud_t *= pud) if (mm_pud_folded(mm)) return; =20 - pagetable_pud_dtor(virt_to_ptdesc(pud)); + pagetable_dtor(virt_to_ptdesc(pud)); crst_table_free(mm, (unsigned long *) pud); } =20 @@ -109,7 +109,7 @@ static inline void pmd_free(struct mm_struct *mm, pmd_t= *pmd) { if (mm_pmd_folded(mm)) return; - pagetable_pmd_dtor(virt_to_ptdesc(pmd)); + pagetable_dtor(virt_to_ptdesc(pmd)); crst_table_free(mm, (unsigned long *) pmd); } =20 diff --git a/arch/s390/include/asm/tlb.h b/arch/s390/include/asm/tlb.h index b946964afce8e..74b6fba4c2ee3 100644 --- a/arch/s390/include/asm/tlb.h +++ b/arch/s390/include/asm/tlb.h @@ -102,7 +102,7 @@ static inline void pmd_free_tlb(struct mmu_gather *tlb,= pmd_t *pmd, { if (mm_pmd_folded(tlb->mm)) return; - pagetable_pmd_dtor(virt_to_ptdesc(pmd)); + pagetable_dtor(virt_to_ptdesc(pmd)); __tlb_adjust_range(tlb, address, PAGE_SIZE); tlb->mm->context.flush_mm =3D 1; tlb->freed_tables =3D 1; @@ -122,7 +122,7 @@ static inline void pud_free_tlb(struct mmu_gather *tlb,= pud_t *pud, { if (mm_pud_folded(tlb->mm)) return; - pagetable_pud_dtor(virt_to_ptdesc(pud)); + pagetable_dtor(virt_to_ptdesc(pud)); tlb->mm->context.flush_mm =3D 1; tlb->freed_tables =3D 1; tlb->cleared_p4ds =3D 1; @@ -141,7 +141,7 @@ static inline void p4d_free_tlb(struct mmu_gather *tlb,= p4d_t *p4d, { if (mm_p4d_folded(tlb->mm)) return; - pagetable_p4d_dtor(virt_to_ptdesc(p4d)); + pagetable_dtor(virt_to_ptdesc(p4d)); __tlb_adjust_range(tlb, address, PAGE_SIZE); tlb->mm->context.flush_mm =3D 1; tlb->freed_tables =3D 1; diff --git a/arch/s390/mm/pgalloc.c b/arch/s390/mm/pgalloc.c index 58696a0c4e4ac..569de24d33761 100644 --- a/arch/s390/mm/pgalloc.c +++ b/arch/s390/mm/pgalloc.c @@ -182,7 +182,7 @@ unsigned long *page_table_alloc(struct mm_struct *mm) =20 static void pagetable_pte_dtor_free(struct ptdesc *ptdesc) { - pagetable_pte_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); } =20 diff --git a/arch/sh/include/asm/pgalloc.h b/arch/sh/include/asm/pgalloc.h index 5d8577ab15911..96d938fdf2244 100644 --- a/arch/sh/include/asm/pgalloc.h +++ b/arch/sh/include/asm/pgalloc.h @@ -34,7 +34,7 @@ static inline void pmd_populate(struct mm_struct *mm, pmd= _t *pmd, =20 #define __pte_free_tlb(tlb, pte, addr) \ do { \ - pagetable_pte_dtor(page_ptdesc(pte)); \ + pagetable_dtor(page_ptdesc(pte)); \ tlb_remove_page_ptdesc((tlb), (page_ptdesc(pte))); \ } while (0) =20 diff --git a/arch/sparc/mm/init_64.c b/arch/sparc/mm/init_64.c index 21f8cbbd0581c..05882bca5b732 100644 --- a/arch/sparc/mm/init_64.c +++ b/arch/sparc/mm/init_64.c @@ -2915,7 +2915,7 @@ static void __pte_free(pgtable_t pte) { struct ptdesc *ptdesc =3D virt_to_ptdesc(pte); =20 - pagetable_pte_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); } =20 diff --git a/arch/sparc/mm/srmmu.c b/arch/sparc/mm/srmmu.c index 9df51a62333d6..e3a72c884b867 100644 --- a/arch/sparc/mm/srmmu.c +++ b/arch/sparc/mm/srmmu.c @@ -372,7 +372,7 @@ void pte_free(struct mm_struct *mm, pgtable_t ptep) page =3D pfn_to_page(__nocache_pa((unsigned long)ptep) >> PAGE_SHIFT); spin_lock(&mm->page_table_lock); if (page_ref_dec_return(page) =3D=3D 1) - pagetable_pte_dtor(page_ptdesc(page)); + pagetable_dtor(page_ptdesc(page)); spin_unlock(&mm->page_table_lock); =20 srmmu_free_nocache(ptep, SRMMU_PTE_TABLE_SIZE); diff --git a/arch/um/include/asm/pgalloc.h b/arch/um/include/asm/pgalloc.h index 04fb4e6969a46..f0af23c3aeb2b 100644 --- a/arch/um/include/asm/pgalloc.h +++ b/arch/um/include/asm/pgalloc.h @@ -27,7 +27,7 @@ extern pgd_t *pgd_alloc(struct mm_struct *); =20 #define __pte_free_tlb(tlb, pte, address) \ do { \ - pagetable_pte_dtor(page_ptdesc(pte)); \ + pagetable_dtor(page_ptdesc(pte)); \ tlb_remove_page_ptdesc((tlb), (page_ptdesc(pte))); \ } while (0) =20 @@ -35,7 +35,7 @@ do { \ =20 #define __pmd_free_tlb(tlb, pmd, address) \ do { \ - pagetable_pmd_dtor(virt_to_ptdesc(pmd)); \ + pagetable_dtor(virt_to_ptdesc(pmd)); \ tlb_remove_page_ptdesc((tlb), virt_to_ptdesc(pmd)); \ } while (0) =20 @@ -43,7 +43,7 @@ do { \ =20 #define __pud_free_tlb(tlb, pud, address) \ do { \ - pagetable_pud_dtor(virt_to_ptdesc(pud)); \ + pagetable_dtor(virt_to_ptdesc(pud)); \ tlb_remove_page_ptdesc((tlb), virt_to_ptdesc(pud)); \ } while (0) =20 diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c index 3d6e84da45b24..a6cd9660e29ec 100644 --- a/arch/x86/mm/pgtable.c +++ b/arch/x86/mm/pgtable.c @@ -60,7 +60,7 @@ early_param("userpte", setup_userpte); =20 void ___pte_free_tlb(struct mmu_gather *tlb, struct page *pte) { - pagetable_pte_dtor(page_ptdesc(pte)); + pagetable_dtor(page_ptdesc(pte)); paravirt_release_pte(page_to_pfn(pte)); paravirt_tlb_remove_table(tlb, pte); } @@ -77,7 +77,7 @@ void ___pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd) #ifdef CONFIG_X86_PAE tlb->need_flush_all =3D 1; #endif - pagetable_pmd_dtor(ptdesc); + pagetable_dtor(ptdesc); paravirt_tlb_remove_table(tlb, ptdesc_page(ptdesc)); } =20 @@ -86,7 +86,7 @@ void ___pud_free_tlb(struct mmu_gather *tlb, pud_t *pud) { struct ptdesc *ptdesc =3D virt_to_ptdesc(pud); =20 - pagetable_pud_dtor(ptdesc); + pagetable_dtor(ptdesc); paravirt_release_pud(__pa(pud) >> PAGE_SHIFT); paravirt_tlb_remove_table(tlb, virt_to_page(pud)); } @@ -96,7 +96,7 @@ void ___p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d) { struct ptdesc *ptdesc =3D virt_to_ptdesc(p4d); =20 - pagetable_p4d_dtor(ptdesc); + pagetable_dtor(ptdesc); paravirt_release_p4d(__pa(p4d) >> PAGE_SHIFT); paravirt_tlb_remove_table(tlb, virt_to_page(p4d)); } @@ -233,7 +233,7 @@ static void free_pmds(struct mm_struct *mm, pmd_t *pmds= [], int count) if (pmds[i]) { ptdesc =3D virt_to_ptdesc(pmds[i]); =20 - pagetable_pmd_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); mm_dec_nr_pmds(mm); } @@ -867,7 +867,7 @@ int pud_free_pmd_page(pud_t *pud, unsigned long addr) =20 free_page((unsigned long)pmd_sv); =20 - pagetable_pmd_dtor(virt_to_ptdesc(pmd)); + pagetable_dtor(virt_to_ptdesc(pmd)); free_page((unsigned long)pmd); =20 return 1; diff --git a/include/asm-generic/pgalloc.h b/include/asm-generic/pgalloc.h index dbf61819b3581..3673e9c29504e 100644 --- a/include/asm-generic/pgalloc.h +++ b/include/asm-generic/pgalloc.h @@ -109,7 +109,7 @@ static inline void pte_free(struct mm_struct *mm, struc= t page *pte_page) { struct ptdesc *ptdesc =3D page_ptdesc(pte_page); =20 - pagetable_pte_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); } =20 @@ -153,7 +153,7 @@ static inline void pmd_free(struct mm_struct *mm, pmd_t= *pmd) struct ptdesc *ptdesc =3D virt_to_ptdesc(pmd); =20 BUG_ON((unsigned long)pmd & (PAGE_SIZE-1)); - pagetable_pmd_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); } #endif @@ -202,7 +202,7 @@ static inline void __pud_free(struct mm_struct *mm, pud= _t *pud) struct ptdesc *ptdesc =3D virt_to_ptdesc(pud); =20 BUG_ON((unsigned long)pud & (PAGE_SIZE-1)); - pagetable_pud_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); } =20 @@ -257,7 +257,7 @@ static inline void __p4d_free(struct mm_struct *mm, p4d= _t *p4d) struct ptdesc *ptdesc =3D virt_to_ptdesc(p4d); =20 BUG_ON((unsigned long)p4d & (PAGE_SIZE-1)); - pagetable_p4d_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); } =20 diff --git a/include/linux/mm.h b/include/linux/mm.h index 7168f46ad4770..736215c428293 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -3086,6 +3086,15 @@ static inline bool ptlock_init(struct ptdesc *ptdesc= ) { return true; } static inline void ptlock_free(struct ptdesc *ptdesc) {} #endif /* defined(CONFIG_SPLIT_PTE_PTLOCKS) */ =20 +static inline void pagetable_dtor(struct ptdesc *ptdesc) +{ + struct folio *folio =3D ptdesc_folio(ptdesc); + + ptlock_free(ptdesc); + __folio_clear_pgtable(folio); + lruvec_stat_sub_folio(folio, NR_PAGETABLE); +} + static inline bool pagetable_pte_ctor(struct ptdesc *ptdesc) { struct folio *folio =3D ptdesc_folio(ptdesc); @@ -3097,15 +3106,6 @@ static inline bool pagetable_pte_ctor(struct ptdesc = *ptdesc) return true; } =20 -static inline void pagetable_pte_dtor(struct ptdesc *ptdesc) -{ - struct folio *folio =3D ptdesc_folio(ptdesc); - - ptlock_free(ptdesc); - __folio_clear_pgtable(folio); - lruvec_stat_sub_folio(folio, NR_PAGETABLE); -} - pte_t *___pte_offset_map(pmd_t *pmd, unsigned long addr, pmd_t *pmdvalp); static inline pte_t *__pte_offset_map(pmd_t *pmd, unsigned long addr, pmd_t *pmdvalp) @@ -3182,14 +3182,6 @@ static inline bool pmd_ptlock_init(struct ptdesc *pt= desc) return ptlock_init(ptdesc); } =20 -static inline void pmd_ptlock_free(struct ptdesc *ptdesc) -{ -#ifdef CONFIG_TRANSPARENT_HUGEPAGE - VM_BUG_ON_PAGE(ptdesc->pmd_huge_pte, ptdesc_page(ptdesc)); -#endif - ptlock_free(ptdesc); -} - #define pmd_huge_pte(mm, pmd) (pmd_ptdesc(pmd)->pmd_huge_pte) =20 #else @@ -3200,7 +3192,6 @@ static inline spinlock_t *pmd_lockptr(struct mm_struc= t *mm, pmd_t *pmd) } =20 static inline bool pmd_ptlock_init(struct ptdesc *ptdesc) { return true; } -static inline void pmd_ptlock_free(struct ptdesc *ptdesc) {} =20 #define pmd_huge_pte(mm, pmd) ((mm)->pmd_huge_pte) =20 @@ -3225,15 +3216,6 @@ static inline bool pagetable_pmd_ctor(struct ptdesc = *ptdesc) return true; } =20 -static inline void pagetable_pmd_dtor(struct ptdesc *ptdesc) -{ - struct folio *folio =3D ptdesc_folio(ptdesc); - - pmd_ptlock_free(ptdesc); - __folio_clear_pgtable(folio); - lruvec_stat_sub_folio(folio, NR_PAGETABLE); -} - /* * No scalability reason to split PUD locks yet, but follow the same patte= rn * as the PMD locks to make it easier if we decide to. The VM should not = be @@ -3261,14 +3243,6 @@ static inline void pagetable_pud_ctor(struct ptdesc = *ptdesc) lruvec_stat_add_folio(folio, NR_PAGETABLE); } =20 -static inline void pagetable_pud_dtor(struct ptdesc *ptdesc) -{ - struct folio *folio =3D ptdesc_folio(ptdesc); - - __folio_clear_pgtable(folio); - lruvec_stat_sub_folio(folio, NR_PAGETABLE); -} - static inline void pagetable_p4d_ctor(struct ptdesc *ptdesc) { struct folio *folio =3D ptdesc_folio(ptdesc); @@ -3277,14 +3251,6 @@ static inline void pagetable_p4d_ctor(struct ptdesc = *ptdesc) lruvec_stat_add_folio(folio, NR_PAGETABLE); } =20 -static inline void pagetable_p4d_dtor(struct ptdesc *ptdesc) -{ - struct folio *folio =3D ptdesc_folio(ptdesc); - - __folio_clear_pgtable(folio); - lruvec_stat_sub_folio(folio, NR_PAGETABLE); -} - extern void __init pagecache_init(void); extern void free_initmem(void); =20 diff --git a/mm/memory.c b/mm/memory.c index e064f5edb7c56..e93ad42806478 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -7104,7 +7104,8 @@ bool ptlock_alloc(struct ptdesc *ptdesc) =20 void ptlock_free(struct ptdesc *ptdesc) { - kmem_cache_free(page_ptl_cachep, ptdesc->ptl); + if (ptdesc->ptl) + kmem_cache_free(page_ptl_cachep, ptdesc->ptl); } #endif =20 --=20 2.20.1 From nobody Thu Dec 18 09:39:52 2025 Received: from mail-pf1-f175.google.com (mail-pf1-f175.google.com [209.85.210.175]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B68971B042E for ; Wed, 18 Dec 2024 13:05:52 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.175 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734527154; cv=none; b=S0p/Igkosy+l7aVRm0uoa0yc1jv0S+dlIPwvWDqqF6t/4FMoC0Vm5uq0dYv3aBpk/EAWC617TJhp+q/P4KEO0/3aGBWVDYSQtxATZaKPKGfdx0J7HliTpvUz1EHGJRwDXvcVrBfyqb5vYTjkIBD/GSwehPZ7poI5T94ozu5R8Kw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734527154; c=relaxed/simple; bh=H/x1u07M3HXF9oVd+o3reuerQgd2pca8MMjlY49GFUc=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=QcCGSAsMAYsHlcOdYuo0rMrSl7nR13BVEVeUCxA4cJereHRBqq0qsa/isHiyKO2+9NQFge7qqY0sqwExcUuA83rD8tR/xuTHLNuelxunLI98c9TeB6rQDEhum4vDCBAKMFZY9/n7ksbKV355A312Tm7Jlqm9RD+mz5Rvh2zuE7E= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=LPmI5lFU; arc=none smtp.client-ip=209.85.210.175 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="LPmI5lFU" Received: by mail-pf1-f175.google.com with SMTP id d2e1a72fcca58-725d9f57d90so4827079b3a.1 for ; Wed, 18 Dec 2024 05:05:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1734527152; x=1735131952; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=qA0mhWnXo6upWemI3POPEc6VJyAkyxFJggkOB6WNq2k=; b=LPmI5lFUqZD0ocomrR73yFWz3Kr3g0JV5phlLfgkCyCoDr0MDFu2qY6roLwkAdzSMj O1CRVi2a0N4bkfZKQKo2ipFwVcqeRdjvGHXZ3reW4h6Oc6sjDgtZvqKwTzsa4hrQv92g zPpM2tQYq1+QG2LFkQb0vjgl4Cn8P+mH0lhVbnj1Ey5uNOt9xbOKx2tVr+SQSvvK/eFj v0arJZcLm3R704fc1vuCl7aoGxFAPxYdnha8/E1cpmrC2+zAVxaA3dwu7F1YyriYMZPI +ALjILfNTKc3tm6u0D5Ofl1RirbFWlodys+o3ywudohKMDUmhpmp8nlh9Ian9yI50C/L px/Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734527152; x=1735131952; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=qA0mhWnXo6upWemI3POPEc6VJyAkyxFJggkOB6WNq2k=; b=I8SSs3h5p5jxAWPmva+atKNlfKmIGMoVChA3+XGYQLTo4/mK2qyIRFVxR/wroIJcgQ 8US3MR0mAGj+b5pAAMcR2VDdezk0PGajeUInfX/tAI9BDMZhZFONTIgYfPGrLU4gzzjz D0HW+nRIiyAiv/0mRg7pojY7tue5Mi9qtOI1OXVtFxmsE2cGNpyu2EFnSHwZa5PV4Hry zCkD3QUPK49xT7ArlQMx2xM88rTlnEEa//JP8JHnU21Jw0UrutZizTrQvjZpgpEj2vqD cI6cUccI6YlfCfGUN5/+90hc9ucsX/5p/HNmxSTisaOVpaTjSzUbrnbuPIuDhXLMEVnN qldQ== X-Forwarded-Encrypted: i=1; AJvYcCUqm+gNOn7ydYXBmkWif7yS7a1sRpk+9v20/Wq19ggsEoNnWag21jFhDvR6iIfFCHQ7is3T4YGdsCQ1CLw=@vger.kernel.org X-Gm-Message-State: AOJu0YzWgZXwoqoXZXWec7LMFafzCw9j6EKnGWu6J/04derMd+X+Tluf /uITESQS6ZAFGdptuq+Ppeu+HRSUekxhGsc6wPu5xc9mINRihX8ehSfRkdCc/Ew= X-Gm-Gg: ASbGnctJCYybW00IBzzZJb5aqKwg+N9SmSM0SyQXyTcTdg4KmYXLStxMwxf+R0KMuiv G1FfxhaihBbGqtdRIu5CDIYqpq8tt1foB0Lo/BLLZfUuvYLCvZUhK4n87YY6zJNIv6bQURoBPyU PbqktP/CtLuxzK5cg7aCHWLBaquATil0EyamYdpPp/S5HzkkTFtqCUpIl0XlnbDvwHwv5WrNOPU msWMdOwX3wiZlNOkATMtq6JfmlBvbAxNjfdGfYv4GL++GZWXrz7jRMwK/ApZlw48dZIq19X9OBY bydusqotqkIhrof2bwoE2g== X-Google-Smtp-Source: AGHT+IFdeNYkaToIEDdUvkBTd8lP/0QrRZpL4HARu9WlaN/FvfyhXaaWSW9JhSu0jLr2tRkQ6EUdVw== X-Received: by 2002:a17:902:f68d:b0:215:b058:289c with SMTP id d9443c01a7336-218d70fe050mr36430515ad.8.1734527152123; Wed, 18 Dec 2024 05:05:52 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.238]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-218a1db755dsm75751825ad.42.2024.12.18.05.05.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 18 Dec 2024 05:05:51 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng , linux-arm-kernel@lists.infradead.org Subject: [PATCH v2 06/15] arm: pgtable: move pagetable_dtor() to __tlb_remove_table() Date: Wed, 18 Dec 2024 21:04:42 +0800 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Move pagetable_dtor() to __tlb_remove_table(), so that ptlock and page table pages can be freed together (regardless of whether RCU is used). This prevents the use-after-free problem where the ptlock is freed immediately but the page table pages is freed later via RCU. Page tables shouldn't have swap cache, so use pagetable_free() instead of free_page_and_swap_cache() to free page table pages. Signed-off-by: Qi Zheng Suggested-by: Peter Zijlstra (Intel) Cc: linux-arm-kernel@lists.infradead.org --- arch/arm/include/asm/tlb.h | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/arch/arm/include/asm/tlb.h b/arch/arm/include/asm/tlb.h index ef79bf1e8563f..264ab635e807a 100644 --- a/arch/arm/include/asm/tlb.h +++ b/arch/arm/include/asm/tlb.h @@ -26,12 +26,14 @@ =20 #else /* !CONFIG_MMU */ =20 -#include #include =20 static inline void __tlb_remove_table(void *_table) { - free_page_and_swap_cache((struct page *)_table); + struct ptdesc *ptdesc =3D (struct ptdesc *)_table; + + pagetable_dtor(ptdesc); + pagetable_free(ptdesc); } =20 #include @@ -41,8 +43,6 @@ __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pte, uns= igned long addr) { struct ptdesc *ptdesc =3D page_ptdesc(pte); =20 - pagetable_dtor(ptdesc); - #ifndef CONFIG_ARM_LPAE /* * With the classic ARM MMU, a pte page has two corresponding pmd @@ -61,7 +61,6 @@ __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmdp, unsig= ned long addr) #ifdef CONFIG_ARM_LPAE struct ptdesc *ptdesc =3D virt_to_ptdesc(pmdp); =20 - pagetable_dtor(ptdesc); tlb_remove_ptdesc(tlb, ptdesc); #endif } --=20 2.20.1 From nobody Thu Dec 18 09:39:52 2025 Received: from mail-pl1-f169.google.com (mail-pl1-f169.google.com [209.85.214.169]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 70F3C1ACEA4 for ; Wed, 18 Dec 2024 13:05:59 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.169 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734527160; cv=none; b=u2nroRtV4UdHnd4jeMxgKxCPTk7BAAFa2cWEaCelA9QMs2Y9tdIt2SUZv2VsBB+S+MRhmZ280gaMc+1adRWKqz6kD9j1fL9VxKVFcALHm/YuxGyGWsxQ2IHptRyAHRLNAyOldfD1PuIUb43Xt7dFxU5dns1A8R14p38DJv6WlZ4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734527160; c=relaxed/simple; bh=w9Tb6kkg2Q49UGEfUZpSUyo28N/wow0+21NvTYUEzY4=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=eoyReXOqdsicocdiD3e6SDjMPmAnWmqSjdmRhOP3xPaC6FN5E43p2R6x5X/0g2MW5/zd9zhcaU2zWbXib5i6CjzzvU6cjjepwKq9mD3Zvgl2bdOsJ5B2ikF5aLUfzQGyab8aIcI/8yUav1dLc8yp6s+98C1FtHb2EKsUQE1601Q= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=CVW+sNjP; arc=none smtp.client-ip=209.85.214.169 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="CVW+sNjP" Received: by mail-pl1-f169.google.com with SMTP id d9443c01a7336-2164b1f05caso59754195ad.3 for ; Wed, 18 Dec 2024 05:05:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1734527159; x=1735131959; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=0nJ0bdEaVLB0+13gaCQHdoBfFkuXXT3Pqta7GOjv8Bo=; b=CVW+sNjPZvAmtWDJaIy/lGCJUcCIMB4cYObcPLoNf5ONmwiIoEJt26j07aBce/XCkG HUjczFX0BDSaloaaHRC93wE8U/wPbngmblyKVrFa7oKcWB6SKbiOlO/GlRCAlOM1+6ml D+MjZSIxqpQpZdKzyKu9mL8kd8EHDsrb7KWr+CJeKGtpiWt7FbgKJHCwDvScQyBc8c7n rDNMt+F3/GHYcsyMgZDmnWsRm4MXK74r5mgIi2v8y7C2fPyHnEY3NhHc1fCY69ZeHHAz 3rTLfMd+lTpTbwLEoXNdA3cqlLbMKi7aaAKyqHOa9uaQNv0PEbqCRymtd+pzR0C2uv0o fTDA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734527159; x=1735131959; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=0nJ0bdEaVLB0+13gaCQHdoBfFkuXXT3Pqta7GOjv8Bo=; b=Hza1JT/ymzxjETsJ9AEY8y37jvIBhbSAG0UszmlkeMIbKa7eNNMsf8X1KwHYKWk5zV obn06o8LcuNA+JaaSBVgB08GyqLZKDbud3+YGv+sK2rs+QfyEjaC259xnMuQmeNJX8/H p2zDMvBgw9RTDloXRX3+MFlOq47eeXOVA69U1sGEnFt5C63QPvvLPafrWiPkX+uc+T0I Ag0QyejBDZscVFRfoqK0TAa5dYmbLQHlJN3npbYUXkelQkbMutLiFlYez6MOGmz4l0nY rPf+cZk5GcIh3Ui4WJwySJPMTI7b61nmbXLHaGvtzXH1GMZrerQ3QKB10r/FVxuiO+ti yimA== X-Forwarded-Encrypted: i=1; AJvYcCXUmuer3lePscQ8G+xhOPMY/pCNyztIanINfFx5HM4jfR9WFS8TzVuldsvq3N8TYQ9X9kYv4MbPJ6V0vRQ=@vger.kernel.org X-Gm-Message-State: AOJu0YyvBt4odePQ4dorGgOLijB/jz7hfGM+bxWEyhDDO9w4LTsnbqNX XwDuU5Iqk3etxlcXjp8qedoeWPNONr3/uIgSJUbdyXkExZL7e9v5b6uyx2JRTxY= X-Gm-Gg: ASbGnctk+CCt+mY3X1anSLECXWwt4tS5PMadHJrl0eeh/NDaKcT1eZtQt/I3Ipprkkn MoiX5aR+LoxSnTZwF/ZudknJh+Jq6GJ688YcrAywNqYhbQMOSwOytORdiqfGOJ6rSXhaUCJ+0Ld OZXmU3RQgNzApJVKiyAq+VsIpzlNhFnEhJ85V1u43sVVMt+LsPI01Unag8nQ3R6GlrRvapto8Ua Qkw5Zmk6ZstR1XIzbTRl8KayiqaLg28vdKqpSkBD6XFyOkAtYDv1IIprItzCqh1kj1vdDzeNJ8N Df6cAbg1Qj/eN3q6N0OSXQ== X-Google-Smtp-Source: AGHT+IFH9P+x/Mfn/YDKtSKvkH1s3GazvpYiwIZf55I8HG6lzpeJuaV9oSxqUwHwuTwe8mBCoFXtDw== X-Received: by 2002:a17:903:41cf:b0:216:4dfe:3ebd with SMTP id d9443c01a7336-218d725ebd2mr32859975ad.50.1734527158730; Wed, 18 Dec 2024 05:05:58 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.238]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-218a1db755dsm75751825ad.42.2024.12.18.05.05.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 18 Dec 2024 05:05:58 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng , linux-arm-kernel@lists.infradead.org Subject: [PATCH v2 07/15] arm64: pgtable: move pagetable_dtor() to __tlb_remove_table() Date: Wed, 18 Dec 2024 21:04:43 +0800 Message-Id: <634f0236b4b4effc528b06e108b7bda51cd0ea2c.1734526570.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Move pagetable_dtor() to __tlb_remove_table(), so that ptlock and page table pages can be freed together (regardless of whether RCU is used). This prevents the use-after-free problem where the ptlock is freed immediately but the page table pages is freed later via RCU. Page tables shouldn't have swap cache, so use pagetable_free() instead of free_page_and_swap_cache() to free page table pages. Signed-off-by: Qi Zheng Suggested-by: Peter Zijlstra (Intel) Cc: linux-arm-kernel@lists.infradead.org --- arch/arm64/include/asm/tlb.h | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-) diff --git a/arch/arm64/include/asm/tlb.h b/arch/arm64/include/asm/tlb.h index 408d0f36a8a8f..93591a80b5bfb 100644 --- a/arch/arm64/include/asm/tlb.h +++ b/arch/arm64/include/asm/tlb.h @@ -9,11 +9,13 @@ #define __ASM_TLB_H =20 #include -#include =20 static inline void __tlb_remove_table(void *_table) { - free_page_and_swap_cache((struct page *)_table); + struct ptdesc *ptdesc =3D (struct ptdesc *)_table; + + pagetable_dtor(ptdesc); + pagetable_free(ptdesc); } =20 #define tlb_flush tlb_flush @@ -82,7 +84,6 @@ static inline void __pte_free_tlb(struct mmu_gather *tlb,= pgtable_t pte, { struct ptdesc *ptdesc =3D page_ptdesc(pte); =20 - pagetable_dtor(ptdesc); tlb_remove_ptdesc(tlb, ptdesc); } =20 @@ -92,7 +93,6 @@ static inline void __pmd_free_tlb(struct mmu_gather *tlb,= pmd_t *pmdp, { struct ptdesc *ptdesc =3D virt_to_ptdesc(pmdp); =20 - pagetable_dtor(ptdesc); tlb_remove_ptdesc(tlb, ptdesc); } #endif @@ -106,7 +106,6 @@ static inline void __pud_free_tlb(struct mmu_gather *tl= b, pud_t *pudp, if (!pgtable_l4_enabled()) return; =20 - pagetable_dtor(ptdesc); tlb_remove_ptdesc(tlb, ptdesc); } #endif @@ -120,7 +119,6 @@ static inline void __p4d_free_tlb(struct mmu_gather *tl= b, p4d_t *p4dp, if (!pgtable_l5_enabled()) return; =20 - pagetable_dtor(ptdesc); tlb_remove_ptdesc(tlb, ptdesc); } #endif --=20 2.20.1 From nobody Thu Dec 18 09:39:52 2025 Received: from mail-pl1-f170.google.com (mail-pl1-f170.google.com [209.85.214.170]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1FE211B423A for ; Wed, 18 Dec 2024 13:06:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.170 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734527168; cv=none; b=cAyW/JIcglA733SwZZrtAuYI9A8jjP259hbA7xGxchkR4i56GDbgs7/KylNPiGI81Wum4Wan6G2WWmX6u0M8Ta8UyoAA7Lf3wKdHY7vRu8CJKnGmKN4/O0qt295H6Qjl3FfyQX4C8bnKB7DVeMjycdk5sYdeBFIp0So+Zv2BqPQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734527168; c=relaxed/simple; bh=HhoTKMNDVanOaoBYRYWMcNEW71QR3jhR9Nx/+t+mu9Y=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=UsWcz2ECS2Zd2jfdQN/ZCCmJ57FGamnaRA2SVo3Giz40sYJ1F+hw37nurtUDoDWIC8EfeXd9P2215lO9CbYpprgc9qGAUoKsyAG7yzcUStxMpgAnXMoxc6R2AZ5lJt+gGPCBOxsSJMX4UsBiRIyBUPP7hKZnMZfHAlcfpAX4Tjk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=jmqo6q1w; arc=none smtp.client-ip=209.85.214.170 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="jmqo6q1w" Received: by mail-pl1-f170.google.com with SMTP id d9443c01a7336-2164b1f05caso59756275ad.3 for ; Wed, 18 Dec 2024 05:06:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1734527165; x=1735131965; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=QtLDYyJyRe/cOzqFg0wwVyja+MW3+zRiiWZkoTiwjPI=; b=jmqo6q1wnD7Fy8ZNZBD1NLUwAqRLB/4lX898HbpLEpdbtdL4sn50x7QAnUr9I/+qwV fMwNvXauyhgNfYdufaUNf9WSzvJgGfCFYGjPyxYjI2bjH9pbtys4ooofx70s+rwIT045 BGBdHQTUXb3txtHzNU1eNmIeaP7qJLMbTvaitWmgiK2lFez7YWZalyJemt4HF7k9A8GQ XZzz3mti1ZzMpyl37uMVNyx4RtagpYspM/zVc4cWh+Sa+vS+LpP96qpV2DQDpx6BnJMl 6458xzMcu95fGqG6zWLTdX4sF3nv7yNOUV8b8unWICHRkWkPhm7QUfZSIipdrL05dYc7 Hlbg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734527165; x=1735131965; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=QtLDYyJyRe/cOzqFg0wwVyja+MW3+zRiiWZkoTiwjPI=; b=P6UkhcOv2I4ihvb07KPZ97AyxS5B6yh+T6UktsdBjTyZTtDpVAw9pbEcRAiuH+mDv3 gpRg48Rkw4LieHBdkLPLl3ogS0mXdhfYjiVSuXsp9QE27qq5EsA2DRzlAuvj2pZL+KCt eIHQjmUDA3bA8xCAhEGGZqBlvBwbnSzAGW+w1foH1uJWtez6cdDCNxsgnxIqpd/EJOHO QTDqyUAEBzgJa+TMVu+M0gkZMEB2U9sih2Wbti1JKcry/zRhIGTceuFvxRvUl/57Jggw 2VuMQtwE+BZZroWL96rmf6ozWman8lnQE/ROizF7b7F2vdFH5pi8OAqvP2UQCSg6D2Jf JKzA== X-Forwarded-Encrypted: i=1; AJvYcCW+aw5n3X+YE+bWs96mKGiKXGUvzYDNCUEB9HBh7vYybk6t6DE79Lc6yBCryMWf6mc3skyfJtIT8eqt+pI=@vger.kernel.org X-Gm-Message-State: AOJu0Yzc88RnjPMGcmTyK/CFtKyOhtyU7uz0w/LSLRWCmDvkVO14P6z8 g66QqL8n3621PcQJSlvco47Sok/mzWOhYV1XwSKhCUouUllSPMdkp9mQ8wQNack= X-Gm-Gg: ASbGncsBI0axGNRoebB/kVxYMaSNMhZB7cwYONHfKYW5pqYJe32u0BJkoAsblWP+foJ yKVWPEL3Y4FLHGeS+c67R60MBfz6QXjlpha3rNIvgrkACE6oEb26frQbK0B9YApp2jyxZ8XGPcY +7wgdYxGPzpZyCfObgwD2Z1RhyCGhGgxXc7zsXa9zCzLU9vGOB9h6OzgIGony9Yu+Wa+mDO+Ap8 0ZZ15VO9hBMBjDC7OcLwQvKD75qjbZvfpuPtMIJF5eJRp7Jt3XtcjEn8EH6gl/g8RMiFYV1Z527 Z6eDhZFNcqlyQbzm6M179A== X-Google-Smtp-Source: AGHT+IFCnYr+lY/2cKu1iNZU0iqph0CddR5mvkArPJVeCIaobzeDvVooRJFIjfOJ9AL+45aiVbBYIg== X-Received: by 2002:a17:902:c40d:b0:215:a05d:fb05 with SMTP id d9443c01a7336-218d72161camr43962385ad.32.1734527165431; Wed, 18 Dec 2024 05:06:05 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.238]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-218a1db755dsm75751825ad.42.2024.12.18.05.05.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 18 Dec 2024 05:06:04 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng , linux-riscv@lists.infradead.org Subject: [PATCH v2 08/15] riscv: pgtable: move pagetable_dtor() to __tlb_remove_table() Date: Wed, 18 Dec 2024 21:04:44 +0800 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Move pagetable_dtor() to __tlb_remove_table(), so that ptlock and page table pages can be freed together (regardless of whether RCU is used). This prevents the use-after-free problem where the ptlock is freed immediately but the page table pages is freed later via RCU. Page tables shouldn't have swap cache, so use pagetable_free() instead of free_page_and_swap_cache() to free page table pages. By the way, move the comment above __tlb_remove_table() to riscv_tlb_remove_ptdesc(), it will be more appropriate. Signed-off-by: Qi Zheng Suggested-by: Peter Zijlstra (Intel) Cc: linux-riscv@lists.infradead.org --- arch/riscv/include/asm/pgalloc.h | 38 ++++++++++++++------------------ arch/riscv/include/asm/tlb.h | 14 ++++-------- 2 files changed, 21 insertions(+), 31 deletions(-) diff --git a/arch/riscv/include/asm/pgalloc.h b/arch/riscv/include/asm/pgal= loc.h index 09c2eff571a49..fc50d14010246 100644 --- a/arch/riscv/include/asm/pgalloc.h +++ b/arch/riscv/include/asm/pgalloc.h @@ -18,12 +18,22 @@ #define __HAVE_ARCH_P4D_FREE #include =20 +/* + * While riscv platforms with riscv_ipi_for_rfence as true require an IPI = to + * perform TLB shootdown, some platforms with riscv_ipi_for_rfence as fals= e use + * SBI to perform TLB shootdown. To keep software pagetable walkers safe i= n this + * case we switch to RCU based table free (MMU_GATHER_RCU_TABLE_FREE). See= the + * comment below 'ifdef CONFIG_MMU_GATHER_RCU_TABLE_FREE' in include/asm-g= eneric/tlb.h + * for more details. + */ static inline void riscv_tlb_remove_ptdesc(struct mmu_gather *tlb, void *p= t) { - if (riscv_use_sbi_for_rfence()) + if (riscv_use_sbi_for_rfence()) { tlb_remove_ptdesc(tlb, pt); - else + } else { + pagetable_dtor(pt); tlb_remove_page_ptdesc(tlb, pt); + } } =20 static inline void pmd_populate_kernel(struct mm_struct *mm, @@ -109,12 +119,8 @@ static inline void pud_free(struct mm_struct *mm, pud_= t *pud) static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pud, unsigned long addr) { - if (pgtable_l4_enabled) { - struct ptdesc *ptdesc =3D virt_to_ptdesc(pud); - - pagetable_dtor(ptdesc); - riscv_tlb_remove_ptdesc(tlb, ptdesc); - } + if (pgtable_l4_enabled) + riscv_tlb_remove_ptdesc(tlb, virt_to_ptdesc(pud)); } =20 #define p4d_alloc_one p4d_alloc_one @@ -136,12 +142,8 @@ static inline void p4d_free(struct mm_struct *mm, p4d_= t *p4d) static inline void __p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d, unsigned long addr) { - if (pgtable_l5_enabled) { - struct ptdesc *ptdesc =3D virt_to_ptdesc(p4d); - - pagetable_dtor(ptdesc); + if (pgtable_l5_enabled) riscv_tlb_remove_ptdesc(tlb, virt_to_ptdesc(p4d)); - } } #endif /* __PAGETABLE_PMD_FOLDED */ =20 @@ -170,10 +172,7 @@ static inline pgd_t *pgd_alloc(struct mm_struct *mm) static inline void __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd, unsigned long addr) { - struct ptdesc *ptdesc =3D virt_to_ptdesc(pmd); - - pagetable_dtor(ptdesc); - riscv_tlb_remove_ptdesc(tlb, ptdesc); + riscv_tlb_remove_ptdesc(tlb, virt_to_ptdesc(pmd)); } =20 #endif /* __PAGETABLE_PMD_FOLDED */ @@ -181,10 +180,7 @@ static inline void __pmd_free_tlb(struct mmu_gather *t= lb, pmd_t *pmd, static inline void __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pte, unsigned long addr) { - struct ptdesc *ptdesc =3D page_ptdesc(pte); - - pagetable_dtor(ptdesc); - riscv_tlb_remove_ptdesc(tlb, ptdesc); + riscv_tlb_remove_ptdesc(tlb, page_ptdesc(pte)); } #endif /* CONFIG_MMU */ =20 diff --git a/arch/riscv/include/asm/tlb.h b/arch/riscv/include/asm/tlb.h index 1f6c38420d8e0..ded8724b3c4f7 100644 --- a/arch/riscv/include/asm/tlb.h +++ b/arch/riscv/include/asm/tlb.h @@ -11,19 +11,13 @@ struct mmu_gather; static void tlb_flush(struct mmu_gather *tlb); =20 #ifdef CONFIG_MMU -#include =20 -/* - * While riscv platforms with riscv_ipi_for_rfence as true require an IPI = to - * perform TLB shootdown, some platforms with riscv_ipi_for_rfence as fals= e use - * SBI to perform TLB shootdown. To keep software pagetable walkers safe i= n this - * case we switch to RCU based table free (MMU_GATHER_RCU_TABLE_FREE). See= the - * comment below 'ifdef CONFIG_MMU_GATHER_RCU_TABLE_FREE' in include/asm-g= eneric/tlb.h - * for more details. - */ static inline void __tlb_remove_table(void *table) { - free_page_and_swap_cache(table); + struct ptdesc *ptdesc =3D (struct ptdesc *)table; + + pagetable_dtor(ptdesc); + pagetable_free(ptdesc); } =20 #endif /* CONFIG_MMU */ --=20 2.20.1 From nobody Thu Dec 18 09:39:52 2025 Received: from mail-pl1-f180.google.com (mail-pl1-f180.google.com [209.85.214.180]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 683C21D9A63 for ; Wed, 18 Dec 2024 13:06:12 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.180 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734527174; cv=none; b=VuIcdt/Kuxsdon4IWLwv9aspKC9WdEcdZ0q38PF0TM6KtBDlDAaYPhMOFVW8XCnC9td1sVFdTEr1U3bU9lPqfltrBzhPiXCmGbpAciJBfa7zGWNHLNBJmyHWgKf3Y1u0NHxwFSz4MCAgPaSmWAdgQhlfXRz3Cw6UGGROTL3YUps= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734527174; c=relaxed/simple; bh=0DrkqJj4wQeX5ghzMY0LbiVqPzUVuk7cM17qWFCGuf4=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=k68eg7dzk0hvd2w4OlGsX1B/ZtrOvw/buuayue2EUfmwLXbYuxDVPBOE1/kfwdRxg8VgAFOjJIl15nVgvyyB0q4Eck63vBY4MJL3cF86pnm+w3M2ME0HT0xq6f40Onebnbs+nXs5JLvpxqGNR46mBolyo0P2+nHV6KvFjAQy9iM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=YOuTY/Bk; arc=none smtp.client-ip=209.85.214.180 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="YOuTY/Bk" Received: by mail-pl1-f180.google.com with SMTP id d9443c01a7336-216426b0865so56770525ad.0 for ; Wed, 18 Dec 2024 05:06:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1734527172; x=1735131972; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Rs6Lkouarv58ZoKb+WtHLVo9G63kikGgJGrD7ursbxc=; b=YOuTY/BkplvGenXkNyhw0r/LDzIgbDvBeTnEXlYPlDhkOE1ItQZbACM2T0qrT14zNn YMsEBjIEvdJgxbVKbo6zpGljjNcJcg9BnIanqN3zctgnrvX1DD9MmiaMHjG3i/r8PVzT 0a+H5hoLQZehJtdb3SbzPvuGGDklrPTdYmMw0ABk1AKe5o2fV+DIZAYAudRGOzpM5Sly EN7klHjcS6x3YLrJgtpOA19EeX0X5knaoDwoOv9K7ZFdeAa5QLciDdeGVg5yA6HUjnqA Jj8rzUNX1RT+loEFR/9NOVLr8wmQDWk0cgEUuqcfH7DzOJFdgklUML7W/H0ESLWIDSMU VHEA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734527172; x=1735131972; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Rs6Lkouarv58ZoKb+WtHLVo9G63kikGgJGrD7ursbxc=; b=Jp/Qp1xE37df9WevDjQXyXpAY2+cYk63qlcYJw3r9P9hR5EyHNwCknMNLkqmScuGVu v+ZZi/47ThWRzKzN9TP6tJdJUGkyr5iz6Cy+liBygFueTc7Z3QuDpKWn6Kt/q1i3Toak AJB9VIYZFK2LGgC1aJeDBXF68F6Bs5wnSmaqhtb5ZUEx1uD0h34P0ccuRnNdaWuyUV0Q RnPA9poAln4RVTaSRZxmE3SLIXR+isCIjiR9/Z3XIt7uJQfzv2DXRUlQDKOJZw4O6NpS 1bl42gacc4NfsJmwb0NEfws7nbwpaptCxAw+GYvhQ5TcGKDAtQwaOzIjIYSs+tzhDBpv ORHg== X-Forwarded-Encrypted: i=1; AJvYcCWL7R/VgNLDUHeIUSUKVziQeDHo1Uz3QwqUosuBbGzXINIbO4ZV1rKDfBDc/UjO8XsHBILvDIzSLNnQR7w=@vger.kernel.org X-Gm-Message-State: AOJu0YzXhU+uBsJxxxffwsJ1gwVnj0hM3v3TH+2ZF8QHaJtTUaktA9RO HF9ULDrIPyDIJnvlhTwQBJ1N5/J6B5VBu9ND6p/wHDeOlbyor7qyrwvzm4hg+bs= X-Gm-Gg: ASbGnctA+m4KhWXNNxnREao3e87qhEcXMjOIvhzJjx9RJPU/QhDaASBcijjwGy+cBi7 /oo4f1yiGI0lPuMh2lQ3TaRYY0PB4pYMmg6DcLsLQZUJFpDo9f3dlr8ILxpOOdH1uZ4rwAf2/Zu KeuWYj9H5soA4T46YohAicPiEXj3FzPfCsbruzVts61VofqN2RvtDGn5nB1ZrNPe3IArwfe8nZu rMcrW0IJxM4vycA1p4BO/i6gAfLcLB/xz1SY0DZlQUWVqvnwjgRTfn3PCiFFlqfLUGJi8Z/2lM0 UzEvMe4YUF2pSQU80EJxsQ== X-Google-Smtp-Source: AGHT+IEwCZCDbOJ1SqMAwJwdz2DwBH3whjOpJ7w3Lr+F6G2curZkg5Sqfpo28wJQmsWqUUWcuoXMow== X-Received: by 2002:a17:902:cec3:b0:212:6011:594a with SMTP id d9443c01a7336-218d6fbe5b2mr36617165ad.3.1734527171662; Wed, 18 Dec 2024 05:06:11 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.238]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-218a1db755dsm75751825ad.42.2024.12.18.05.06.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 18 Dec 2024 05:06:11 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng , x86@kernel.org Subject: [PATCH v2 09/15] x86: pgtable: move pagetable_dtor() to __tlb_remove_table() Date: Wed, 18 Dec 2024 21:04:45 +0800 Message-Id: <8c0d97d2a538cafb94cf30f5c94a84a0f93f2f4f.1734526570.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Move pagetable_dtor() to __tlb_remove_table(), so that ptlock and page table pages can be freed together (regardless of whether RCU is used). This prevents the use-after-free problem where the ptlock is freed immediately but the page table pages is freed later via RCU. Page tables shouldn't have swap cache, so use pagetable_free() instead of free_page_and_swap_cache() to free page table pages. Signed-off-by: Qi Zheng Suggested-by: Peter Zijlstra (Intel) Cc: x86@kernel.org --- arch/x86/include/asm/tlb.h | 17 ++++++++++------- arch/x86/kernel/paravirt.c | 1 + arch/x86/mm/pgtable.c | 12 ++---------- 3 files changed, 13 insertions(+), 17 deletions(-) diff --git a/arch/x86/include/asm/tlb.h b/arch/x86/include/asm/tlb.h index 73f0786181cc9..f64730be5ad67 100644 --- a/arch/x86/include/asm/tlb.h +++ b/arch/x86/include/asm/tlb.h @@ -31,24 +31,27 @@ static inline void tlb_flush(struct mmu_gather *tlb) */ static inline void __tlb_remove_table(void *table) { - free_page_and_swap_cache(table); + struct ptdesc *ptdesc =3D (struct ptdesc *)table; + + pagetable_dtor(ptdesc); + pagetable_free(ptdesc); } =20 #ifdef CONFIG_PT_RECLAIM static inline void __tlb_remove_table_one_rcu(struct rcu_head *head) { - struct page *page; + struct ptdesc *ptdesc; =20 - page =3D container_of(head, struct page, rcu_head); - put_page(page); + ptdesc =3D container_of(head, struct ptdesc, pt_rcu_head); + __tlb_remove_table(ptdesc); } =20 static inline void __tlb_remove_table_one(void *table) { - struct page *page; + struct ptdesc *ptdesc; =20 - page =3D table; - call_rcu(&page->rcu_head, __tlb_remove_table_one_rcu); + ptdesc =3D table; + call_rcu(&ptdesc->pt_rcu_head, __tlb_remove_table_one_rcu); } #define __tlb_remove_table_one __tlb_remove_table_one #endif /* CONFIG_PT_RECLAIM */ diff --git a/arch/x86/kernel/paravirt.c b/arch/x86/kernel/paravirt.c index 7bdcf152778c0..46d5d325483b0 100644 --- a/arch/x86/kernel/paravirt.c +++ b/arch/x86/kernel/paravirt.c @@ -62,6 +62,7 @@ void __init native_pv_lock_init(void) #ifndef CONFIG_PT_RECLAIM static void native_tlb_remove_table(struct mmu_gather *tlb, void *table) { + pagetable_dtor(table); tlb_remove_page(tlb, table); } #else diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c index a6cd9660e29ec..a0b0e501ba663 100644 --- a/arch/x86/mm/pgtable.c +++ b/arch/x86/mm/pgtable.c @@ -23,6 +23,7 @@ EXPORT_SYMBOL(physical_mask); static inline void paravirt_tlb_remove_table(struct mmu_gather *tlb, void *table) { + pagetable_dtor(table); tlb_remove_page(tlb, table); } #else @@ -60,7 +61,6 @@ early_param("userpte", setup_userpte); =20 void ___pte_free_tlb(struct mmu_gather *tlb, struct page *pte) { - pagetable_dtor(page_ptdesc(pte)); paravirt_release_pte(page_to_pfn(pte)); paravirt_tlb_remove_table(tlb, pte); } @@ -68,7 +68,6 @@ void ___pte_free_tlb(struct mmu_gather *tlb, struct page = *pte) #if CONFIG_PGTABLE_LEVELS > 2 void ___pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd) { - struct ptdesc *ptdesc =3D virt_to_ptdesc(pmd); paravirt_release_pmd(__pa(pmd) >> PAGE_SHIFT); /* * NOTE! For PAE, any changes to the top page-directory-pointer-table @@ -77,16 +76,12 @@ void ___pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd) #ifdef CONFIG_X86_PAE tlb->need_flush_all =3D 1; #endif - pagetable_dtor(ptdesc); - paravirt_tlb_remove_table(tlb, ptdesc_page(ptdesc)); + paravirt_tlb_remove_table(tlb, virt_to_page(pmd)); } =20 #if CONFIG_PGTABLE_LEVELS > 3 void ___pud_free_tlb(struct mmu_gather *tlb, pud_t *pud) { - struct ptdesc *ptdesc =3D virt_to_ptdesc(pud); - - pagetable_dtor(ptdesc); paravirt_release_pud(__pa(pud) >> PAGE_SHIFT); paravirt_tlb_remove_table(tlb, virt_to_page(pud)); } @@ -94,9 +89,6 @@ void ___pud_free_tlb(struct mmu_gather *tlb, pud_t *pud) #if CONFIG_PGTABLE_LEVELS > 4 void ___p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d) { - struct ptdesc *ptdesc =3D virt_to_ptdesc(p4d); - - pagetable_dtor(ptdesc); paravirt_release_p4d(__pa(p4d) >> PAGE_SHIFT); paravirt_tlb_remove_table(tlb, virt_to_page(p4d)); } --=20 2.20.1 From nobody Thu Dec 18 09:39:52 2025 Received: from mail-pl1-f182.google.com (mail-pl1-f182.google.com [209.85.214.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7C9E61B4258 for ; Wed, 18 Dec 2024 13:06:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.182 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734527182; cv=none; b=iFnmbt+FSlOns3qmnKsQTLIgQEMhq3J3JvPlQFb+Tpa7rUHJP9VtEL70KX/byZ6kFlM0wfmPElOqBn2QiM609Nqa9DdJO/uXxm2LWBsedzwF/uIYSOJFA65qY49DqwYg818+m+ptaL4Y8aqHeHI9LTPrOrUDegXUtg2/UYHIzv8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734527182; c=relaxed/simple; bh=EkdNyEu/xMej7Pyo7zL4QlgTElqHC2seEuKnGSW3Tk8=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=G9WtITU5fkff8zChL1H3AgWjx0dt7f5R1lc8uZiZhUp2kfAyNKJf7r34bys2Enj8tqQfbPFImbiS+XeQt11gE2H7xaoG08nHctw41hi0A/zthSg5nixT9QuhkCvvh8Vrlqf+L++y4U0ssmiCl/aNIskpYm5ej4h+2fOY1t3vpeM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=MH0oF+G1; arc=none smtp.client-ip=209.85.214.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="MH0oF+G1" Received: by mail-pl1-f182.google.com with SMTP id d9443c01a7336-2164b1f05caso59760315ad.3 for ; Wed, 18 Dec 2024 05:06:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1734527179; x=1735131979; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=3Ahoctd41Xu7L+7olWYCaTwfCpap8BqbvCHiRFQm2vo=; b=MH0oF+G1vSbumryty/6/5cVbka9T1WBVFiQ49sPhATE4mEsSnV4mzYmEGUHSrkqPCw t12Ni1OsgKWI4/MOsnwRZMZeIFt/VDBenHhFz4pfXGM4IhCIXlXj7o61h+nLpnnL8H+Z Oc7SFnpCxlrhpld3STpkcYj8kXoOjOqLwSZu64A52vgjYVM/FRmMRxT809cN1pBTxjH9 Q5fqENHAk1TaTLopc0Rm5r5n1MxrYeiosxbaZNzm1osWXXA8ByC/Ye9X9CIqazR9UqEA f4FFoOGPny6kkPAXZio+UcW0q4YDhG4+9Rd35LDB5NJW85CgTupRtltdCDRzIM1tueW7 1aOw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734527179; x=1735131979; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=3Ahoctd41Xu7L+7olWYCaTwfCpap8BqbvCHiRFQm2vo=; b=kNV99ZrVq4Tfbew4/gDjdS8V21vtYvRtnAE+rn2ISxP4rPmr5/a9DuaMQF51GKArUl w/X58kz8GczpLGZUR1Ckatd7vx/o9VkthL9DZU1wFeTZt52cHi68pQe7rzigL/rHAdsI 6NTJ0tbpk9ocTBXyPsAB/Z6qlBeIzWuOVV/MR0SQEcqSoSoqdDH+tt9UUsskMyUr2Oe1 NgXGGePt3+GDzN/5piTC3185TN2WDvdG4MwkX3PKSTVlbsADZaiTsgQpzxL56kr5jBlI MKDq0fSwd7NDLu6Xtml7REdT+gKa/DcjFQd7uB5AMDNMuAbkg0VL6Z7f2F5QfAdW8y5L Km1g== X-Forwarded-Encrypted: i=1; AJvYcCUYFMYUQZCb/MHhM7xewVPxmocWapQnh2Em8mfLHmDVNTKg6bsvE6BwzsuHGIfHf1PKpT1EEq0m/OkszCI=@vger.kernel.org X-Gm-Message-State: AOJu0Yy8IaJr7v5SlRnfCInVoU968IdElfJa5a57EtCogr0gTbGbaTzU 3g0zYnWCNblVCTNJNeY5zlWEA4m9GzlaL7totiC5uqvsQVLkxxmGHcLwqpgvWSE= X-Gm-Gg: ASbGncsISCOyUZElBlCXDg0v2ewNNQ3Jhq5FKjhkcoEdJmQVoyzcPFGooZEhBB7OOlw /N5xPB0GUyNhCmCMMDdCzyT4x2J/H3bc8fR7ecWeEcYQFQXJittgksK3KeYpFH/MZccyrBVyr1K WgmcIUmYwT64JvBaoWlIXrvaLrzZ69xAds+4W9Ae4enU0BcEkYK05NtpVp9PEPdMwEzLE/zF+kb 9RE2rmYMBm00XTCDIAfSm3LN/6ni23ufV3SpWGLf7UoxaLsDYAU3YKZfAW/9xc1baiGHsz87EkQ kjb9lWoNl3zYoVjh/aw2XQ== X-Google-Smtp-Source: AGHT+IErwrhCem5fzBI6a6wl06mNTQ9yKVlkmz7hDOEqq4HxfcsAp66u7+2R+nQ/CreR8BGTw2JG1g== X-Received: by 2002:a17:902:f646:b0:216:393b:23c9 with SMTP id d9443c01a7336-218d71fd765mr39074785ad.29.1734527178847; Wed, 18 Dec 2024 05:06:18 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.238]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-218a1db755dsm75751825ad.42.2024.12.18.05.06.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 18 Dec 2024 05:06:17 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng , linux-s390@vger.kernel.org Subject: [PATCH v2 10/15] s390: pgtable: also move pagetable_dtor() of PxD to __tlb_remove_table() Date: Wed, 18 Dec 2024 21:04:46 +0800 Message-Id: <0a6caf8305409173b5d41dccb6ecd46460cf9c1c.1734526570.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" In s390, the pagetable_dtor() of PTE has long been moved to __tlb_remove_table(). So similarly, also move the pagetable_dtor() of PMD|PUD|P4D to __tlb_remove_table(). This prevents the use-after-free problem where the ptlock is freed immediately but the page table pages is freed later via RCU. By the way, rename pagetable_pte_dtor_free() to pagetable_dtor_free(). Signed-off-by: Qi Zheng Suggested-by: Peter Zijlstra (Intel) Cc: linux-s390@vger.kernel.org --- arch/s390/include/asm/tlb.h | 3 --- arch/s390/mm/pgalloc.c | 14 ++++---------- 2 files changed, 4 insertions(+), 13 deletions(-) diff --git a/arch/s390/include/asm/tlb.h b/arch/s390/include/asm/tlb.h index 74b6fba4c2ee3..79df7c0932c56 100644 --- a/arch/s390/include/asm/tlb.h +++ b/arch/s390/include/asm/tlb.h @@ -102,7 +102,6 @@ static inline void pmd_free_tlb(struct mmu_gather *tlb,= pmd_t *pmd, { if (mm_pmd_folded(tlb->mm)) return; - pagetable_dtor(virt_to_ptdesc(pmd)); __tlb_adjust_range(tlb, address, PAGE_SIZE); tlb->mm->context.flush_mm =3D 1; tlb->freed_tables =3D 1; @@ -122,7 +121,6 @@ static inline void pud_free_tlb(struct mmu_gather *tlb,= pud_t *pud, { if (mm_pud_folded(tlb->mm)) return; - pagetable_dtor(virt_to_ptdesc(pud)); tlb->mm->context.flush_mm =3D 1; tlb->freed_tables =3D 1; tlb->cleared_p4ds =3D 1; @@ -141,7 +139,6 @@ static inline void p4d_free_tlb(struct mmu_gather *tlb,= p4d_t *p4d, { if (mm_p4d_folded(tlb->mm)) return; - pagetable_dtor(virt_to_ptdesc(p4d)); __tlb_adjust_range(tlb, address, PAGE_SIZE); tlb->mm->context.flush_mm =3D 1; tlb->freed_tables =3D 1; diff --git a/arch/s390/mm/pgalloc.c b/arch/s390/mm/pgalloc.c index 569de24d33761..c73b89811a264 100644 --- a/arch/s390/mm/pgalloc.c +++ b/arch/s390/mm/pgalloc.c @@ -180,7 +180,7 @@ unsigned long *page_table_alloc(struct mm_struct *mm) return table; } =20 -static void pagetable_pte_dtor_free(struct ptdesc *ptdesc) +static void pagetable_dtor_free(struct ptdesc *ptdesc) { pagetable_dtor(ptdesc); pagetable_free(ptdesc); @@ -190,20 +190,14 @@ void page_table_free(struct mm_struct *mm, unsigned l= ong *table) { struct ptdesc *ptdesc =3D virt_to_ptdesc(table); =20 - pagetable_pte_dtor_free(ptdesc); + pagetable_dtor_free(ptdesc); } =20 void __tlb_remove_table(void *table) { struct ptdesc *ptdesc =3D virt_to_ptdesc(table); - struct page *page =3D ptdesc_page(ptdesc); =20 - if (compound_order(page) =3D=3D CRST_ALLOC_ORDER) { - /* pmd, pud, or p4d */ - pagetable_free(ptdesc); - return; - } - pagetable_pte_dtor_free(ptdesc); + pagetable_dtor_free(ptdesc); } =20 #ifdef CONFIG_TRANSPARENT_HUGEPAGE @@ -211,7 +205,7 @@ static void pte_free_now(struct rcu_head *head) { struct ptdesc *ptdesc =3D container_of(head, struct ptdesc, pt_rcu_head); =20 - pagetable_pte_dtor_free(ptdesc); + pagetable_dtor_free(ptdesc); } =20 void pte_free_defer(struct mm_struct *mm, pgtable_t pgtable) --=20 2.20.1 From nobody Thu Dec 18 09:39:52 2025 Received: from mail-pl1-f172.google.com (mail-pl1-f172.google.com [209.85.214.172]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C72E61B0416 for ; Wed, 18 Dec 2024 13:06:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734527187; cv=none; b=CKHr4g9eJUVos7WZZvljMHOKYaEmn4HRse8H6dRgIIPjpe8eYyiolPUys9q6rxev7i/Ldb5v+mMcd93OeM9x7zKcwRAgtrvCpPCLoZ58qBe62zUFzsbUtPxQ79M97aB8yNKIc6DVkdqLHw8iLLVqKfdXXtpH6gdVOwF/faAJnGk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734527187; c=relaxed/simple; bh=B63fE8G305OqQFAlkrscfYj8+HDqUYWRJz9FPkmdlIQ=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=tXwrzfJeE4PIRBq6TVtowT0trktzoLxQH86ilDOT01OvVWDSzJDFQb3Yt6oA0d4qLUICdzPC+/YstW6vCyCVG7L0h8eyfwSpt0sOjficE84H/JJ5YFVmp7npvdrAfX6QJ5Ef+6fkyW1rLNp68Q6yj3A5ReBIYJExXSgPzRn7JdY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=VffQ9G4Y; arc=none smtp.client-ip=209.85.214.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="VffQ9G4Y" Received: by mail-pl1-f172.google.com with SMTP id d9443c01a7336-21636268e43so78266485ad.2 for ; Wed, 18 Dec 2024 05:06:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1734527185; x=1735131985; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=8jrhRkH9wk2k8zCLxFmzwNce2I+voVuTloSZdpUGAUc=; b=VffQ9G4YWwMWoyDH+S2mPxHhnVQMgg+m+FcIpnchk/bV727ml98Ixv9Q/j8zgz8B+Q AxziRGOwty/xihC3c4IromWqeqt/8TduxHL3RJ2PgzxRDBMUvF0Vbq34g08sBE3k+SrG hqff5dcj5cJT+dhjmB6wm67e6ewqhaEOXwQ0A5HTqHI/24LX03S12EpkWOFWK/wH9Ca6 v3iNhi9DNOTkMk4seECuQz29QrZIz/K4R2FgYwEv4RETHutDZIOvrYxuF+mtO8QgszmM luTYktITX9XLxCTYQujO1CaJZLsIG8AQjL67EOAcIcOUaIKhwYYZtgyi371rwVfM8rQU Ugew== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734527185; x=1735131985; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=8jrhRkH9wk2k8zCLxFmzwNce2I+voVuTloSZdpUGAUc=; b=D31f6ShPP5MwWwS6n8Gc0apksDVRPFS2M4rjIyPgmkapKsyqOTPwnEykPNTJq0JFZg AqqFkLj91x+cTWbFKFOcQ2/eLeAJylWbh4vJkp1TElgHCR2hJG8VvYWjf2yPJJAsziqc NllN/wSayokCnuqpUZRYWCeRd+vi3UPe0ezUjYMRpqsPeuTG2MzGw0KyO154I11tbwMz eL5+lFDj0up0AjvSHTaT6e1IPEkFDsbab/NqufuOJqTERxbEdP1FinSrlMLUbfIn3sef REBZMuRHFfWF6n2njj9mYQGeYFn/OjTXlPSC4ipPecBOd6c6IkSWLH1Mp7qFGpCwzA2+ oDrQ== X-Forwarded-Encrypted: i=1; AJvYcCWBVIFGogHWb1kd2z4pf++BR/WqEBmdPaaeKXQX7Z2fQ/Mw1XHlzWGVnukRIoRxkYIfzitIfhXh1qfgPUg=@vger.kernel.org X-Gm-Message-State: AOJu0YyfMQMm4RjUM9yHy8JqKaEGVF1PnUnp/B87CUciVVp/j1Ohwtan T9v87BuCelH6BqfBsclAddAFzAnG3ahjzBwRWEyFKURsxAiGMNDSd6W+Z7xVSt3MQCFdrg16RZa + X-Gm-Gg: ASbGncs1O3NPwoUY2Isvash+BWyhcApHh6w9Dj5mKvjlWhwHCGWUdmEC8rxTh6mT+Xu XHEF28PD58ewGw7AfDKeB1b5ITGjaqY6R5TuBqDK8WlheGgORHIdFhUCs5PS3YCC84/Jyt4fD8j ODzSQqeEjrQSPVtlhUNKZAGF7NBonUJYprQbff9r5wg1A+/YAHPHo97cavRCyaxmwikL76D0idS ZIu8XvtBfniVPkp8MQVJhzJfP+CZOWZK0sM0BdQaDPheKFL8OUQCzPhSSBcWjJ0WnIqazupQhO3 PZ4g7irDSCRKP4c2wteOPQ== X-Google-Smtp-Source: AGHT+IG/JUNAlV4GKtrGOWzVqsVky2vkpSpLF0iAE0qU7NCcidXao7F2hMgn6AIs6nUdS2ae8ITLQQ== X-Received: by 2002:a17:903:1105:b0:216:6769:9ee7 with SMTP id d9443c01a7336-218d7247707mr40539125ad.41.1734527185019; Wed, 18 Dec 2024 05:06:25 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.238]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-218a1db755dsm75751825ad.42.2024.12.18.05.06.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 18 Dec 2024 05:06:24 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng Subject: [PATCH v2 11/15] mm: pgtable: introduce generic __tlb_remove_table() Date: Wed, 18 Dec 2024 21:04:47 +0800 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Several architectures (arm, arm64, riscv, s390 and x86) define exactly the same __tlb_remove_table(), just introduce generic __tlb_remove_table() to eliminate these duplications. Signed-off-by: Qi Zheng --- arch/arm/include/asm/tlb.h | 9 --------- arch/arm64/include/asm/tlb.h | 7 ------- arch/powerpc/include/asm/tlb.h | 1 + arch/riscv/include/asm/tlb.h | 12 ------------ arch/s390/include/asm/tlb.h | 1 - arch/s390/mm/pgalloc.c | 7 ------- arch/sparc/include/asm/tlb_32.h | 1 + arch/sparc/include/asm/tlb_64.h | 1 + arch/x86/include/asm/tlb.h | 17 ----------------- include/asm-generic/tlb.h | 15 +++++++++++++-- 10 files changed, 16 insertions(+), 55 deletions(-) diff --git a/arch/arm/include/asm/tlb.h b/arch/arm/include/asm/tlb.h index 264ab635e807a..ea4fbe7b17f6f 100644 --- a/arch/arm/include/asm/tlb.h +++ b/arch/arm/include/asm/tlb.h @@ -27,15 +27,6 @@ #else /* !CONFIG_MMU */ =20 #include - -static inline void __tlb_remove_table(void *_table) -{ - struct ptdesc *ptdesc =3D (struct ptdesc *)_table; - - pagetable_dtor(ptdesc); - pagetable_free(ptdesc); -} - #include =20 static inline void diff --git a/arch/arm64/include/asm/tlb.h b/arch/arm64/include/asm/tlb.h index 93591a80b5bfb..8d762607285cc 100644 --- a/arch/arm64/include/asm/tlb.h +++ b/arch/arm64/include/asm/tlb.h @@ -10,13 +10,6 @@ =20 #include =20 -static inline void __tlb_remove_table(void *_table) -{ - struct ptdesc *ptdesc =3D (struct ptdesc *)_table; - - pagetable_dtor(ptdesc); - pagetable_free(ptdesc); -} =20 #define tlb_flush tlb_flush static void tlb_flush(struct mmu_gather *tlb); diff --git a/arch/powerpc/include/asm/tlb.h b/arch/powerpc/include/asm/tlb.h index 1ca7d4c4b90db..2058e8d3e0138 100644 --- a/arch/powerpc/include/asm/tlb.h +++ b/arch/powerpc/include/asm/tlb.h @@ -37,6 +37,7 @@ extern void tlb_flush(struct mmu_gather *tlb); */ #define tlb_needs_table_invalidate() radix_enabled() =20 +#define __HAVE_ARCH_TLB_REMOVE_TABLE /* Get the generic bits... */ #include =20 diff --git a/arch/riscv/include/asm/tlb.h b/arch/riscv/include/asm/tlb.h index ded8724b3c4f7..50b63b5c15bd8 100644 --- a/arch/riscv/include/asm/tlb.h +++ b/arch/riscv/include/asm/tlb.h @@ -10,18 +10,6 @@ struct mmu_gather; =20 static void tlb_flush(struct mmu_gather *tlb); =20 -#ifdef CONFIG_MMU - -static inline void __tlb_remove_table(void *table) -{ - struct ptdesc *ptdesc =3D (struct ptdesc *)table; - - pagetable_dtor(ptdesc); - pagetable_free(ptdesc); -} - -#endif /* CONFIG_MMU */ - #define tlb_flush tlb_flush #include =20 diff --git a/arch/s390/include/asm/tlb.h b/arch/s390/include/asm/tlb.h index 79df7c0932c56..7052780740349 100644 --- a/arch/s390/include/asm/tlb.h +++ b/arch/s390/include/asm/tlb.h @@ -22,7 +22,6 @@ * Pages used for the page tables is a different story. FIXME: more */ =20 -void __tlb_remove_table(void *_table); static inline void tlb_flush(struct mmu_gather *tlb); static inline bool __tlb_remove_page_size(struct mmu_gather *tlb, struct page *page, bool delay_rmap, int page_size); diff --git a/arch/s390/mm/pgalloc.c b/arch/s390/mm/pgalloc.c index c73b89811a264..3e002dea6278f 100644 --- a/arch/s390/mm/pgalloc.c +++ b/arch/s390/mm/pgalloc.c @@ -193,13 +193,6 @@ void page_table_free(struct mm_struct *mm, unsigned lo= ng *table) pagetable_dtor_free(ptdesc); } =20 -void __tlb_remove_table(void *table) -{ - struct ptdesc *ptdesc =3D virt_to_ptdesc(table); - - pagetable_dtor_free(ptdesc); -} - #ifdef CONFIG_TRANSPARENT_HUGEPAGE static void pte_free_now(struct rcu_head *head) { diff --git a/arch/sparc/include/asm/tlb_32.h b/arch/sparc/include/asm/tlb_3= 2.h index 5cd28a8793e39..910254867dfbd 100644 --- a/arch/sparc/include/asm/tlb_32.h +++ b/arch/sparc/include/asm/tlb_32.h @@ -2,6 +2,7 @@ #ifndef _SPARC_TLB_H #define _SPARC_TLB_H =20 +#define __HAVE_ARCH_TLB_REMOVE_TABLE #include =20 #endif /* _SPARC_TLB_H */ diff --git a/arch/sparc/include/asm/tlb_64.h b/arch/sparc/include/asm/tlb_6= 4.h index 3037187482db7..1a6e694418e39 100644 --- a/arch/sparc/include/asm/tlb_64.h +++ b/arch/sparc/include/asm/tlb_64.h @@ -33,6 +33,7 @@ void flush_tlb_pending(void); #define tlb_needs_table_invalidate() (false) #endif =20 +#define __HAVE_ARCH_TLB_REMOVE_TABLE #include =20 #endif /* _SPARC64_TLB_H */ diff --git a/arch/x86/include/asm/tlb.h b/arch/x86/include/asm/tlb.h index f64730be5ad67..3858dbf75880e 100644 --- a/arch/x86/include/asm/tlb.h +++ b/arch/x86/include/asm/tlb.h @@ -20,23 +20,6 @@ static inline void tlb_flush(struct mmu_gather *tlb) flush_tlb_mm_range(tlb->mm, start, end, stride_shift, tlb->freed_tables); } =20 -/* - * While x86 architecture in general requires an IPI to perform TLB - * shootdown, enablement code for several hypervisors overrides - * .flush_tlb_others hook in pv_mmu_ops and implements it by issuing - * a hypercall. To keep software pagetable walkers safe in this case we - * switch to RCU based table free (MMU_GATHER_RCU_TABLE_FREE). See the com= ment - * below 'ifdef CONFIG_MMU_GATHER_RCU_TABLE_FREE' in include/asm-generic/t= lb.h - * for more details. - */ -static inline void __tlb_remove_table(void *table) -{ - struct ptdesc *ptdesc =3D (struct ptdesc *)table; - - pagetable_dtor(ptdesc); - pagetable_free(ptdesc); -} - #ifdef CONFIG_PT_RECLAIM static inline void __tlb_remove_table_one_rcu(struct rcu_head *head) { diff --git a/include/asm-generic/tlb.h b/include/asm-generic/tlb.h index 709830274b756..69de47c7ef3c5 100644 --- a/include/asm-generic/tlb.h +++ b/include/asm-generic/tlb.h @@ -153,8 +153,9 @@ * * Useful if your architecture has non-page page directories. * - * When used, an architecture is expected to provide __tlb_remove_table() - * which does the actual freeing of these pages. + * When used, an architecture is expected to provide __tlb_remove_table()= or + * use the generic __tlb_remove_table(), which does the actual freeing of= these + * pages. * * MMU_GATHER_RCU_TABLE_FREE * @@ -207,6 +208,16 @@ struct mmu_table_batch { #define MAX_TABLE_BATCH \ ((PAGE_SIZE - sizeof(struct mmu_table_batch)) / sizeof(void *)) =20 +#ifndef __HAVE_ARCH_TLB_REMOVE_TABLE +static inline void __tlb_remove_table(void *table) +{ + struct ptdesc *ptdesc =3D (struct ptdesc *)table; + + pagetable_dtor(ptdesc); + pagetable_free(ptdesc); +} +#endif + extern void tlb_remove_table(struct mmu_gather *tlb, void *table); =20 #else /* !CONFIG_MMU_GATHER_HAVE_TABLE_FREE */ --=20 2.20.1 From nobody Thu Dec 18 09:39:52 2025 Received: from mail-pl1-f171.google.com (mail-pl1-f171.google.com [209.85.214.171]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C829B1B0416 for ; Wed, 18 Dec 2024 13:06:31 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.171 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734527193; cv=none; b=QyQxPrgX7nrAKoZ4ovzwFpJZxg5KwoqEDDrndylfPsdqaIrG6alshjYWq60bwxq++vT8wITz3WbkOpJZFxBvXXbXcho1huuVjcdz23uYEO/nhXPSN5elrzTPeyEaU90rxBk7wLC1k8YaekEgZ6R30CDDunbrD3g19riOPM6XLwA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734527193; c=relaxed/simple; bh=AOHzYz3e0W68bypaywk5PqwWiTbsh7X+wa84FHfFzTo=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=fim0k235BF18xmXG7QPX7X6+x0d8ceYbNwizIImCxBvjRSDfMPiOqoRolVXJYpb5ugLKe0P2J2D8BxVy2uRmllRU2wL8alKoGf+e+KKhIqyS1rk81xMLCdNfKeuxPGyfHRuMm1MWG8sXAvABpIlfP6EfHNTpGcJH/FqzMvDkjmE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=hBXJ02Mq; arc=none smtp.client-ip=209.85.214.171 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="hBXJ02Mq" Received: by mail-pl1-f171.google.com with SMTP id d9443c01a7336-21644e6140cso60661735ad.1 for ; Wed, 18 Dec 2024 05:06:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1734527191; x=1735131991; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=VAJKsDdGNb+ALwQvMYs9Oz9Vq8VGrLJsA8GxB+kZmZc=; b=hBXJ02MqMcI1ZorEciAWUscQzffpHD+Yo3C0Kg9nWL8HG3Rvbo1H53SSArhR7Ku+YW mZSWnyy8CYi3Ibd9J65Io6iWsRMplQMuTjUzHIEzg/9Fq40YZBwxrLvOwSgTJ7BcynVx rOTgLZ3oTxMV53A2rZB/uHQFeYOGannAm2VY2zi0gBH6yFMUghVIJH5skh2f6WiDSiP5 uEUI+XTAZhUW4xxKDOkoyo+nCTYOTkXN0yCnF2Ickd4fC6mDvfcL9oAGzo3h6C4RSNyX LZ3IrqQU8R7RTzhhEvKpTGpj6ESQC06fG8nFNuLgdH1EBTTMNNYVakXTr/Cr8zizwoz0 m0Dg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734527191; x=1735131991; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=VAJKsDdGNb+ALwQvMYs9Oz9Vq8VGrLJsA8GxB+kZmZc=; b=QQb9nBI64gMss8lC6sawFoH2Aae9Qz1STd/427FKeDoayP3RDFp7Egyfxs0uJZl0gS zMhAntxpdY03mv9hYrubEH+ge61Cw9Sm5G3JfpYFeQgtpCh2cDKMGYQshTnHMOipbcPb z4UpNEBDD0Qy6k3KdA8dZFLTX3sZ3yWX6skg8NvD90PvZWM24WViaetIQ0Qp42X4UfXy p/v0jXDUvn4CjVX4Uwgv+0gtaWb8U9wVBqb0b4CXpQ+xLvzx1PXTUdvRi7trGMPT/DrC DyDyokWRtV0u8DyBxPtySa2+z7PXP6Bd4hJ5hYEcJrB0IEe17X2zop0ZWYhRD2ikxPxL HZzA== X-Forwarded-Encrypted: i=1; AJvYcCVVs52JbeGhrQkuIAAv0FCwRnvEPTwvO7Ua2802OTpYLaHubtOUo2Q4oGPuNy0rE5q6YPmAIBKMt7oZ1kk=@vger.kernel.org X-Gm-Message-State: AOJu0Yyiib2AwN7rzbJdteX9fLPY9XKYSge41vpYvcAGY+Rf0xKqX+Y8 K6+BteC9paQxVb1youtK6bhxUnLB8DcHowDc+uMJs6B9G/z8fMesaFFYBq4w744= X-Gm-Gg: ASbGncsSYe1ogFIuqdXeoz5cl7f8oHHea/QeCN4DW3TL4UnjS2aXI5w5zJW4Za+xl0q N3dijSSQ9B2MyrWszK/NvF12kda7a5L0jq9q0mOlcsHMr+oDbpbRBo/PSUw/jHXA90QLGsvT2rl ca8zzJeFOxr/SA761ZFgWlFYwNbDyPINOyjTIaB/JwMLPalc5b+4GRQk1OB65TEzvg+OOVt5rvl NbGMkc80s2HiixhGDSyKEYjw2EP+NyCWWpRGEFclz9EaWLyz3M58WVtRIz3SRlfcsO5g8u2ZfR3 5UzBUS3kzolNubGcuexmew== X-Google-Smtp-Source: AGHT+IF0te8v/jB1TE9z7MkQrDAS+cXnBWDMEESZdiy37OaR/n/IJPlBOW2ORibeoxTxN5xCd8QOLA== X-Received: by 2002:a17:902:fa4e:b0:216:3732:ade3 with SMTP id d9443c01a7336-218d7268fcfmr30133425ad.35.1734527191112; Wed, 18 Dec 2024 05:06:31 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.238]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-218a1db755dsm75751825ad.42.2024.12.18.05.06.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 18 Dec 2024 05:06:30 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng Subject: [PATCH v2 12/15] mm: pgtable: move __tlb_remove_table_one() in x86 to generic file Date: Wed, 18 Dec 2024 21:04:48 +0800 Message-Id: <5129586354b398c26c80165dc9ebc7321ecd7a51.1734526570.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The __tlb_remove_table_one() in x86 does not contain architecture-specific content, so move it to the generic file. Signed-off-by: Qi Zheng --- arch/x86/include/asm/tlb.h | 19 ------------------- mm/mmu_gather.c | 20 ++++++++++++++++++-- 2 files changed, 18 insertions(+), 21 deletions(-) diff --git a/arch/x86/include/asm/tlb.h b/arch/x86/include/asm/tlb.h index 3858dbf75880e..77f52bc1578a7 100644 --- a/arch/x86/include/asm/tlb.h +++ b/arch/x86/include/asm/tlb.h @@ -20,25 +20,6 @@ static inline void tlb_flush(struct mmu_gather *tlb) flush_tlb_mm_range(tlb->mm, start, end, stride_shift, tlb->freed_tables); } =20 -#ifdef CONFIG_PT_RECLAIM -static inline void __tlb_remove_table_one_rcu(struct rcu_head *head) -{ - struct ptdesc *ptdesc; - - ptdesc =3D container_of(head, struct ptdesc, pt_rcu_head); - __tlb_remove_table(ptdesc); -} - -static inline void __tlb_remove_table_one(void *table) -{ - struct ptdesc *ptdesc; - - ptdesc =3D table; - call_rcu(&ptdesc->pt_rcu_head, __tlb_remove_table_one_rcu); -} -#define __tlb_remove_table_one __tlb_remove_table_one -#endif /* CONFIG_PT_RECLAIM */ - static inline void invlpg(unsigned long addr) { asm volatile("invlpg (%0)" ::"r" (addr) : "memory"); diff --git a/mm/mmu_gather.c b/mm/mmu_gather.c index 1e21022bcf339..7aa6f18c500b2 100644 --- a/mm/mmu_gather.c +++ b/mm/mmu_gather.c @@ -311,13 +311,29 @@ static inline void tlb_table_invalidate(struct mmu_ga= ther *tlb) } } =20 -#ifndef __tlb_remove_table_one +#ifdef CONFIG_PT_RECLAIM +static inline void __tlb_remove_table_one_rcu(struct rcu_head *head) +{ + struct ptdesc *ptdesc; + + ptdesc =3D container_of(head, struct ptdesc, pt_rcu_head); + __tlb_remove_table(ptdesc); +} + +static inline void __tlb_remove_table_one(void *table) +{ + struct ptdesc *ptdesc; + + ptdesc =3D table; + call_rcu(&ptdesc->pt_rcu_head, __tlb_remove_table_one_rcu); +} +#else static inline void __tlb_remove_table_one(void *table) { tlb_remove_table_sync_one(); __tlb_remove_table(table); } -#endif +#endif /* CONFIG_PT_RECLAIM */ =20 static void tlb_remove_table_one(void *table) { --=20 2.20.1 From nobody Thu Dec 18 09:39:52 2025 Received: from mail-pl1-f174.google.com (mail-pl1-f174.google.com [209.85.214.174]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D2E2F1B0416 for ; Wed, 18 Dec 2024 13:06:37 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.174 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734527199; cv=none; b=g2QwmhcgoBJR0nxB7u3xrF/MaMcmIFP05cLrLHAma8tNjTaYWq48Nj1WZTOufoYl8wx+8fFBJejszC1Gk/SF35lMaBTi8lYsrHRNFree5srPGfxdDGMfg9YHyDR57obWSIr/remLdsXxBHk3qhoUnycAdepZWebsK+Lt2YivlTU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734527199; c=relaxed/simple; bh=crW7V1IVbH9wJ2nwyeoBW3OR/UI9pcsSxVhMvHo/y3U=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=En1yfrCQvyK7ROo7lR2QFUjdyCdPnx9B4EcCLxXtP90RP0AEN2LohnE4pZOWDrrOHAq6hCvW/QZZrMJtDfW+t2Bx6LW60qsPd6TUXy5UBZWrNOEvu1hf7ccXxUrElYGC0wunQN7b14PcRw8aHDjKq1B/oOmD6UOWdbWIu480SmE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=O/LGAXgV; arc=none smtp.client-ip=209.85.214.174 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="O/LGAXgV" Received: by mail-pl1-f174.google.com with SMTP id d9443c01a7336-21675fd60feso66959035ad.2 for ; Wed, 18 Dec 2024 05:06:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1734527197; x=1735131997; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=qjP41dwIWVz3voDw1szUdJpl6GJGCX1n8EUO4uIA81k=; b=O/LGAXgVT0jS+RzWLZwk1rCE/36kxw7pc1Cbweadr2x7TXppaUs2ZxpMgmcq+n31t7 QjbHhXecsmXOct5nCoHxgNoH6dmzxy7+29KKhkj3Keo5HKg10DEXGrBknCLYxin9XQIc yDLFaUlj4EAIa1LAJs5Y75Swp5jtGYBWB4oA3I+i1Yp0+ekKVwervsxGoK6qhlKXfral i5K6ojJ7uaqzijo3zTAXgD+ulAG8okDFiRszqOwjlQCmTpMZedv1ZI+63AfcMTgS/WoD G/+YPAwPzYV2qE521aI67Zj3E206QAgwbWCu58m4t3y0z/aEsJhNGhS1XWj8eoHL8I2F 7owQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734527197; x=1735131997; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=qjP41dwIWVz3voDw1szUdJpl6GJGCX1n8EUO4uIA81k=; b=khTpidar0T7Zh/PlrrLS7p6Qh5SSzk91ZdaxiTqzq311znsAt6OtYHmg24AIB0uTws EW+r33cidKo0/kXzR9TV/O6heCb/dgmuz+6xzqY43vYebeQTSSc/OJhPR0BvvKwgI5nZ 4tVXq1dNibXxumNlaFbliAt2a8V1AFp29+CmQiZBko6CPY6VtAuxMZJrKqRy5iStxPef wXIVoTo2y/sAEjTAaTRxL3seeFmur7lxLUHygjYG0ZiWxdS+QVjdgInb6w5wpLrleUpV adw6hZR5mFiplgmWc1lATUeeSlVQ2weii4MZY3vjIjXy1NrmijlJx0I0ifhNFxANeufG eiXA== X-Forwarded-Encrypted: i=1; AJvYcCWgFjDoEv/EjXO458BjAD1Hyx4Nh3cEC0H6WBF5kZTwxr1mhIgdMiIfJJ0yb2QeYCVf3pMXiHqXTTxKrFg=@vger.kernel.org X-Gm-Message-State: AOJu0Yy+pW3imAu3brEr2u3w+EQUApQeuiD8AG7wR1cENV64t809jYiq rFc4QHWwTgQNmWw4hPJCzBoZMtvwGixkUO7m61zOuxcWJmKSW6D7uu7eOzwOHY0= X-Gm-Gg: ASbGncvtsdkYPJyIwQiKu9P/EsrmocMNdFyuHHdNH7ZEGvDEE+oTEWIR7CLdy6wh+1Q Aa2PAUQ1kylD9k4NssWDeigtIC5aycCrEhIHYSiiRSS10dwAR+RqTqsDcOqqqYkh3e9msdVzXtM 5aXBz+BuRZkFtZNSaWAHAoicsQf8MzLBpXEzca4GQbyjsBx21yPpP13iFpGBmLC/YGeEXHfjGQL 1DrTTxk6rOf5DCWUX/5FZrF7Hufsi+9nzEj8NKI+KH2euMkBwCBXLQKxU2xEgrSCph2ANgbtd2f i8Wtj7HJXPlyGwLnw3nVng== X-Google-Smtp-Source: AGHT+IGZLlII1E3uD2eJMNDRJL7Et07dogsfo2GqYIDSPjE3P+HLWVdTN80snitMlVb3E/0oGm+VVg== X-Received: by 2002:a17:902:e542:b0:216:7d7e:c010 with SMTP id d9443c01a7336-218d7252c97mr36826655ad.37.1734527197173; Wed, 18 Dec 2024 05:06:37 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.238]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-218a1db755dsm75751825ad.42.2024.12.18.05.06.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 18 Dec 2024 05:06:36 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng Subject: [PATCH v2 13/15] mm: pgtable: remove tlb_remove_page_ptdesc() Date: Wed, 18 Dec 2024 21:04:49 +0800 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Here we are explicitly dealing with struct page, and the following logic semms strange: tlb_remove_page_ptdesc((tlb), (page_ptdesc(pte))); tlb_remove_page_ptdesc --> tlb_remove_page(tlb, ptdesc_page(pt)); So remove tlb_remove_page_ptdesc() and make callers call tlb_remove_page() directly. Signed-off-by: Qi Zheng Originally-by: Peter Zijlstra (Intel) --- arch/csky/include/asm/pgalloc.h | 2 +- arch/hexagon/include/asm/pgalloc.h | 2 +- arch/loongarch/include/asm/pgalloc.h | 2 +- arch/m68k/include/asm/sun3_pgalloc.h | 2 +- arch/mips/include/asm/pgalloc.h | 2 +- arch/nios2/include/asm/pgalloc.h | 2 +- arch/openrisc/include/asm/pgalloc.h | 2 +- arch/riscv/include/asm/pgalloc.h | 2 +- arch/sh/include/asm/pgalloc.h | 2 +- arch/um/include/asm/pgalloc.h | 8 ++++---- include/asm-generic/tlb.h | 6 ------ 11 files changed, 13 insertions(+), 19 deletions(-) diff --git a/arch/csky/include/asm/pgalloc.h b/arch/csky/include/asm/pgallo= c.h index f1ce5b7b28f22..936a43a49e704 100644 --- a/arch/csky/include/asm/pgalloc.h +++ b/arch/csky/include/asm/pgalloc.h @@ -64,7 +64,7 @@ static inline pgd_t *pgd_alloc(struct mm_struct *mm) #define __pte_free_tlb(tlb, pte, address) \ do { \ pagetable_dtor(page_ptdesc(pte)); \ - tlb_remove_page_ptdesc(tlb, page_ptdesc(pte)); \ + tlb_remove_page(tlb, (pte)); \ } while (0) =20 extern void pagetable_init(void); diff --git a/arch/hexagon/include/asm/pgalloc.h b/arch/hexagon/include/asm/= pgalloc.h index 40e42a0e71673..8b1550498f1bf 100644 --- a/arch/hexagon/include/asm/pgalloc.h +++ b/arch/hexagon/include/asm/pgalloc.h @@ -90,7 +90,7 @@ static inline void pmd_populate_kernel(struct mm_struct *= mm, pmd_t *pmd, #define __pte_free_tlb(tlb, pte, addr) \ do { \ pagetable_dtor((page_ptdesc(pte))); \ - tlb_remove_page_ptdesc((tlb), (page_ptdesc(pte))); \ + tlb_remove_page((tlb), (pte)); \ } while (0) =20 #endif diff --git a/arch/loongarch/include/asm/pgalloc.h b/arch/loongarch/include/= asm/pgalloc.h index 7211dff8c969e..5a4f22aeb6189 100644 --- a/arch/loongarch/include/asm/pgalloc.h +++ b/arch/loongarch/include/asm/pgalloc.h @@ -58,7 +58,7 @@ static inline pte_t *pte_alloc_one_kernel(struct mm_struc= t *mm) #define __pte_free_tlb(tlb, pte, address) \ do { \ pagetable_dtor(page_ptdesc(pte)); \ - tlb_remove_page_ptdesc((tlb), page_ptdesc(pte)); \ + tlb_remove_page((tlb), (pte)); \ } while (0) =20 #ifndef __PAGETABLE_PMD_FOLDED diff --git a/arch/m68k/include/asm/sun3_pgalloc.h b/arch/m68k/include/asm/s= un3_pgalloc.h index 2b626cb3ad0ae..63d9f95f5e3dd 100644 --- a/arch/m68k/include/asm/sun3_pgalloc.h +++ b/arch/m68k/include/asm/sun3_pgalloc.h @@ -20,7 +20,7 @@ extern const char bad_pmd_string[]; #define __pte_free_tlb(tlb, pte, addr) \ do { \ pagetable_dtor(page_ptdesc(pte)); \ - tlb_remove_page_ptdesc((tlb), page_ptdesc(pte)); \ + tlb_remove_page((tlb), (pte)); \ } while (0) =20 static inline void pmd_populate_kernel(struct mm_struct *mm, pmd_t *pmd, p= te_t *pte) diff --git a/arch/mips/include/asm/pgalloc.h b/arch/mips/include/asm/pgallo= c.h index 36d9805033c4b..bbee21345154b 100644 --- a/arch/mips/include/asm/pgalloc.h +++ b/arch/mips/include/asm/pgalloc.h @@ -57,7 +57,7 @@ static inline void pgd_free(struct mm_struct *mm, pgd_t *= pgd) #define __pte_free_tlb(tlb, pte, address) \ do { \ pagetable_dtor(page_ptdesc(pte)); \ - tlb_remove_page_ptdesc((tlb), page_ptdesc(pte)); \ + tlb_remove_page((tlb), (pte)); \ } while (0) =20 #ifndef __PAGETABLE_PMD_FOLDED diff --git a/arch/nios2/include/asm/pgalloc.h b/arch/nios2/include/asm/pgal= loc.h index 12a536b7bfbd4..641cec8fb2a22 100644 --- a/arch/nios2/include/asm/pgalloc.h +++ b/arch/nios2/include/asm/pgalloc.h @@ -31,7 +31,7 @@ extern pgd_t *pgd_alloc(struct mm_struct *mm); #define __pte_free_tlb(tlb, pte, addr) \ do { \ pagetable_dtor(page_ptdesc(pte)); \ - tlb_remove_page_ptdesc((tlb), (page_ptdesc(pte))); \ + tlb_remove_page((tlb), (pte)); \ } while (0) =20 #endif /* _ASM_NIOS2_PGALLOC_H */ diff --git a/arch/openrisc/include/asm/pgalloc.h b/arch/openrisc/include/as= m/pgalloc.h index 596e2355824e3..e9b9bc53ece0b 100644 --- a/arch/openrisc/include/asm/pgalloc.h +++ b/arch/openrisc/include/asm/pgalloc.h @@ -69,7 +69,7 @@ extern pte_t *pte_alloc_one_kernel(struct mm_struct *mm); #define __pte_free_tlb(tlb, pte, addr) \ do { \ pagetable_dtor(page_ptdesc(pte)); \ - tlb_remove_page_ptdesc((tlb), (page_ptdesc(pte))); \ + tlb_remove_page((tlb), (pte)); \ } while (0) =20 #endif diff --git a/arch/riscv/include/asm/pgalloc.h b/arch/riscv/include/asm/pgal= loc.h index fc50d14010246..da782a80f8aad 100644 --- a/arch/riscv/include/asm/pgalloc.h +++ b/arch/riscv/include/asm/pgalloc.h @@ -32,7 +32,7 @@ static inline void riscv_tlb_remove_ptdesc(struct mmu_gat= her *tlb, void *pt) tlb_remove_ptdesc(tlb, pt); } else { pagetable_dtor(pt); - tlb_remove_page_ptdesc(tlb, pt); + tlb_remove_page(tlb, ptdesc_page((struct ptdesc *)pt)); } } =20 diff --git a/arch/sh/include/asm/pgalloc.h b/arch/sh/include/asm/pgalloc.h index 96d938fdf2244..43812b2363efd 100644 --- a/arch/sh/include/asm/pgalloc.h +++ b/arch/sh/include/asm/pgalloc.h @@ -35,7 +35,7 @@ static inline void pmd_populate(struct mm_struct *mm, pmd= _t *pmd, #define __pte_free_tlb(tlb, pte, addr) \ do { \ pagetable_dtor(page_ptdesc(pte)); \ - tlb_remove_page_ptdesc((tlb), (page_ptdesc(pte))); \ + tlb_remove_page((tlb), (pte)); \ } while (0) =20 #endif /* __ASM_SH_PGALLOC_H */ diff --git a/arch/um/include/asm/pgalloc.h b/arch/um/include/asm/pgalloc.h index f0af23c3aeb2b..98190c318a8e9 100644 --- a/arch/um/include/asm/pgalloc.h +++ b/arch/um/include/asm/pgalloc.h @@ -28,7 +28,7 @@ extern pgd_t *pgd_alloc(struct mm_struct *); #define __pte_free_tlb(tlb, pte, address) \ do { \ pagetable_dtor(page_ptdesc(pte)); \ - tlb_remove_page_ptdesc((tlb), (page_ptdesc(pte))); \ + tlb_remove_page((tlb), (pte)); \ } while (0) =20 #if CONFIG_PGTABLE_LEVELS > 2 @@ -36,15 +36,15 @@ do { \ #define __pmd_free_tlb(tlb, pmd, address) \ do { \ pagetable_dtor(virt_to_ptdesc(pmd)); \ - tlb_remove_page_ptdesc((tlb), virt_to_ptdesc(pmd)); \ + tlb_remove_page((tlb), virt_to_page(pmd)); \ } while (0) =20 #if CONFIG_PGTABLE_LEVELS > 3 =20 #define __pud_free_tlb(tlb, pud, address) \ do { \ - pagetable_dtor(virt_to_ptdesc(pud)); \ - tlb_remove_page_ptdesc((tlb), virt_to_ptdesc(pud)); \ + pagetable_dtor(virt_to_ptdesc(pud)); \ + tlb_remove_page((tlb), virt_to_page(pud)); \ } while (0) =20 #endif diff --git a/include/asm-generic/tlb.h b/include/asm-generic/tlb.h index 69de47c7ef3c5..8d6cfe5058543 100644 --- a/include/asm-generic/tlb.h +++ b/include/asm-generic/tlb.h @@ -504,12 +504,6 @@ static inline void tlb_remove_ptdesc(struct mmu_gather= *tlb, void *pt) tlb_remove_table(tlb, pt); } =20 -/* Like tlb_remove_ptdesc, but for page-like page directories. */ -static inline void tlb_remove_page_ptdesc(struct mmu_gather *tlb, struct p= tdesc *pt) -{ - tlb_remove_page(tlb, ptdesc_page(pt)); -} - static inline void tlb_change_page_size(struct mmu_gather *tlb, unsigned int page_size) { --=20 2.20.1 From nobody Thu Dec 18 09:39:52 2025 Received: from mail-pl1-f174.google.com (mail-pl1-f174.google.com [209.85.214.174]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DC5CD1B0416 for ; Wed, 18 Dec 2024 13:06:43 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.174 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734527205; cv=none; b=t52kOaXHcB5EpHdESz86xf8HYSMwQcs1IjP6sRS7GNWuijMsi+51vIgFmRMblVDJEixlGjysmYGWjLH5i/DwmJIuwAqrPoxdrstOMnEzozK/lzr1jWdiqnQsc0F2pSS/Y0R3lyHXCvMtnz+Bsv5G9mFKwaaB411VTCFeMx3qVgI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734527205; c=relaxed/simple; bh=Svs1pv7W8nli3pB3EWaqja1Gw7JXvPlzfs8XYKPSzj4=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=GfPXpJSSGM4Y5KTTdxras/twxXy7hqqi+9pPZ7HSAPQ5MEqlhdDVq6NFxOwJrsjaXIGaVK5ZdRKCyEr32b7M7Kf4zzsBxSzA8L9ycbtKBERHPWXYmBaNMYMkx1rGblT9e2wc9/1HTLCqVKfND3VedgFbEvp/bzvnfRI3LYEJtkM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=Bnaewscd; arc=none smtp.client-ip=209.85.214.174 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="Bnaewscd" Received: by mail-pl1-f174.google.com with SMTP id d9443c01a7336-2164b662090so51907345ad.1 for ; Wed, 18 Dec 2024 05:06:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1734527203; x=1735132003; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=oQG7I8CTZ7TxV9gz83QsafJIUScPDY5LnGxZvWTV1Wk=; b=BnaewscdtTEKgFEq2ufO9Bc4eN+bAandePOmzoAB5wDAqPSy27FD7gwN/Z5UKUuoHF aeiKxA8Ll0HyW/me4uevv15OmGGHT7kwGis+Xtab3bXunRY2bljJwW0LKPdcfRywHQXh oKpxklS1LpeGATMSHh21/m/Qd+5FS8XjOMHuRCwslOEMjpcGK1m+2GMAfnYk24RY2cw5 z1umM2CpQEO3kq9ZQVmdTjtLDF7SfhU3e84K2RZRd/XwvkpLsHtunX2iMqCVF5O1f+Bi fXWWGVekK0l6zkQdVRJ/ISS00eGPkkY8ZjrfyhL0koQk5B3MSTlALgG4iXm1P/TplMbW 1lhA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734527203; x=1735132003; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=oQG7I8CTZ7TxV9gz83QsafJIUScPDY5LnGxZvWTV1Wk=; b=MHFuC27FVKrwM7i2q/7iz0t35tK+oC0c9j4VVmDB2AXB6PkmQJyEFNeKWlZNOwUf5N FAGB0IbXrsrqUwExYq1wJM0n3tXldpWbRQ1+Mnew/kdF8lluQfXQECbSdUzYzZM60Suu Tuh9td0U1X8JLIsznbI/ZE+OMQj+T6ZLH9vmKH/bqu6WzQLbAlQQYHErU3G0kTG12k7r Xfm2uMKuH/X2WkvycG2qUHU8DQGewNwZk4ONyjMbY9oJwsc+NnWvjZXKFGcu4o7Q72b0 3ea36NV4DmjcbrVvaeda44vuodkGG2kGouIr118vhaCghCbmtMBmyujfmCyu+h81kfCu ab0A== X-Forwarded-Encrypted: i=1; AJvYcCVSrXBdqGhbcpTF3uptt12asRB2qrdraUQSGXxfmMjijHp/Kg9aBJzd9wBpzWUiyrBHUSvi+5JXapm1K+s=@vger.kernel.org X-Gm-Message-State: AOJu0YzBQN1g7XX9sBf9dNnRE4ul90uhpL+do24rsuAScErZD3Xfa8tD gAVLHxWhTBKWqYlLnrFlWfz759MtSKJ1YziUkxUbFOyrPXif1UysyrGBa6F/oco= X-Gm-Gg: ASbGncuwAyTlTRHYS9wq1/FPNXX/pMlL3tuxHL6YYsxg0ksmt81qMtHbMAzX+pjuWKc aHrsP3YR4zQCkUODmdpg25Qu9mOrneRn3xotk8rowtRhw5/Us43tr1kEEYyQOpLnV3B4kyHZChu 2zmpljHVyWTq5IGA8lnG8R/uLN16VLBov2jRDu9aTYX7NPzgENw0yn8T9mScYGR6XFepqVYWxqs vnfkR/bTFjwcqEGWvvxQiZQVQ0hLmtzFNQqSYxlxfbT2f1gIOXOGZg0iBnVLTQh3Idj6cIXUldb 2+LxShfkGpxor21RHOUnVA== X-Google-Smtp-Source: AGHT+IGqOVtOaXTsC11RmGC0pnVBsbhpRr7fSYpWbIozZcJ8TvZ8U4y91o/Td1tfWFL+H7ThjVOaNw== X-Received: by 2002:a17:902:d50b:b0:215:6816:632e with SMTP id d9443c01a7336-218d725bf5dmr42047505ad.48.1734527203288; Wed, 18 Dec 2024 05:06:43 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.238]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-218a1db755dsm75751825ad.42.2024.12.18.05.06.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 18 Dec 2024 05:06:42 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng Subject: [PATCH v2 14/15] mm: pgtable: remove tlb_remove_ptdesc() Date: Wed, 18 Dec 2024 21:04:50 +0800 Message-Id: <79b20df78eb81a9cd07e3048f2d4d63d98759b4a.1734526570.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Just like removing tlb_remove_page_ptdesc(), remove tlb_remove_ptdesc() as well, and make callers call tlb_remove_table() directly. Signed-off-by: Qi Zheng Originally-by: Peter Zijlstra (Intel) --- arch/arm/include/asm/tlb.h | 8 ++------ arch/arm64/include/asm/tlb.h | 16 ++++------------ arch/riscv/include/asm/pgalloc.h | 14 +++++++------- arch/s390/include/asm/tlb.h | 8 ++++---- include/asm-generic/tlb.h | 7 +------ mm/mmu_gather.c | 11 +++++------ 6 files changed, 23 insertions(+), 41 deletions(-) diff --git a/arch/arm/include/asm/tlb.h b/arch/arm/include/asm/tlb.h index ea4fbe7b17f6f..ac3881ec342f1 100644 --- a/arch/arm/include/asm/tlb.h +++ b/arch/arm/include/asm/tlb.h @@ -32,8 +32,6 @@ static inline void __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pte, unsigned long addr) { - struct ptdesc *ptdesc =3D page_ptdesc(pte); - #ifndef CONFIG_ARM_LPAE /* * With the classic ARM MMU, a pte page has two corresponding pmd @@ -43,16 +41,14 @@ __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pte, u= nsigned long addr) __tlb_adjust_range(tlb, addr - PAGE_SIZE, 2 * PAGE_SIZE); #endif =20 - tlb_remove_ptdesc(tlb, ptdesc); + tlb_remove_table(tlb, pte); } =20 static inline void __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmdp, unsigned long addr) { #ifdef CONFIG_ARM_LPAE - struct ptdesc *ptdesc =3D virt_to_ptdesc(pmdp); - - tlb_remove_ptdesc(tlb, ptdesc); + tlb_remove_table(tlb, virt_to_page(pmdp)); #endif } =20 diff --git a/arch/arm64/include/asm/tlb.h b/arch/arm64/include/asm/tlb.h index 8d762607285cc..4a60569fed696 100644 --- a/arch/arm64/include/asm/tlb.h +++ b/arch/arm64/include/asm/tlb.h @@ -75,18 +75,14 @@ static inline void tlb_flush(struct mmu_gather *tlb) static inline void __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pte, unsigned long addr) { - struct ptdesc *ptdesc =3D page_ptdesc(pte); - - tlb_remove_ptdesc(tlb, ptdesc); + tlb_remove_table(tlb, pte); } =20 #if CONFIG_PGTABLE_LEVELS > 2 static inline void __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmdp, unsigned long addr) { - struct ptdesc *ptdesc =3D virt_to_ptdesc(pmdp); - - tlb_remove_ptdesc(tlb, ptdesc); + tlb_remove_table(tlb, virt_to_page(pmdp)); } #endif =20 @@ -94,12 +90,10 @@ static inline void __pmd_free_tlb(struct mmu_gather *tl= b, pmd_t *pmdp, static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pudp, unsigned long addr) { - struct ptdesc *ptdesc =3D virt_to_ptdesc(pudp); - if (!pgtable_l4_enabled()) return; =20 - tlb_remove_ptdesc(tlb, ptdesc); + tlb_remove_table(tlb, virt_to_page(pudp)); } #endif =20 @@ -107,12 +101,10 @@ static inline void __pud_free_tlb(struct mmu_gather *= tlb, pud_t *pudp, static inline void __p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4dp, unsigned long addr) { - struct ptdesc *ptdesc =3D virt_to_ptdesc(p4dp); - if (!pgtable_l5_enabled()) return; =20 - tlb_remove_ptdesc(tlb, ptdesc); + tlb_remove_table(tlb, virt_to_page(p4dp)); } #endif =20 diff --git a/arch/riscv/include/asm/pgalloc.h b/arch/riscv/include/asm/pgal= loc.h index da782a80f8aad..f319e2c6776e4 100644 --- a/arch/riscv/include/asm/pgalloc.h +++ b/arch/riscv/include/asm/pgalloc.h @@ -29,10 +29,10 @@ static inline void riscv_tlb_remove_ptdesc(struct mmu_gather *tlb, void *p= t) { if (riscv_use_sbi_for_rfence()) { - tlb_remove_ptdesc(tlb, pt); + tlb_remove_table(tlb, pt); } else { - pagetable_dtor(pt); - tlb_remove_page(tlb, ptdesc_page((struct ptdesc *)pt)); + pagetable_dtor(page_ptdesc((struct page *)pt)); + tlb_remove_page(tlb, pt); } } =20 @@ -120,7 +120,7 @@ static inline void __pud_free_tlb(struct mmu_gather *tl= b, pud_t *pud, unsigned long addr) { if (pgtable_l4_enabled) - riscv_tlb_remove_ptdesc(tlb, virt_to_ptdesc(pud)); + riscv_tlb_remove_ptdesc(tlb, virt_to_page(pud)); } =20 #define p4d_alloc_one p4d_alloc_one @@ -143,7 +143,7 @@ static inline void __p4d_free_tlb(struct mmu_gather *tl= b, p4d_t *p4d, unsigned long addr) { if (pgtable_l5_enabled) - riscv_tlb_remove_ptdesc(tlb, virt_to_ptdesc(p4d)); + riscv_tlb_remove_ptdesc(tlb, virt_to_page(p4d)); } #endif /* __PAGETABLE_PMD_FOLDED */ =20 @@ -172,7 +172,7 @@ static inline pgd_t *pgd_alloc(struct mm_struct *mm) static inline void __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd, unsigned long addr) { - riscv_tlb_remove_ptdesc(tlb, virt_to_ptdesc(pmd)); + riscv_tlb_remove_ptdesc(tlb, virt_to_page(pmd)); } =20 #endif /* __PAGETABLE_PMD_FOLDED */ @@ -180,7 +180,7 @@ static inline void __pmd_free_tlb(struct mmu_gather *tl= b, pmd_t *pmd, static inline void __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pte, unsigned long addr) { - riscv_tlb_remove_ptdesc(tlb, page_ptdesc(pte)); + riscv_tlb_remove_ptdesc(tlb, pte); } #endif /* CONFIG_MMU */ =20 diff --git a/arch/s390/include/asm/tlb.h b/arch/s390/include/asm/tlb.h index 7052780740349..fba11949dd2ea 100644 --- a/arch/s390/include/asm/tlb.h +++ b/arch/s390/include/asm/tlb.h @@ -86,7 +86,7 @@ static inline void pte_free_tlb(struct mmu_gather *tlb, p= gtable_t pte, tlb->cleared_pmds =3D 1; if (mm_alloc_pgste(tlb->mm)) gmap_unlink(tlb->mm, (unsigned long *)pte, address); - tlb_remove_ptdesc(tlb, pte); + tlb_remove_table(tlb, pte); } =20 /* @@ -105,7 +105,7 @@ static inline void pmd_free_tlb(struct mmu_gather *tlb,= pmd_t *pmd, tlb->mm->context.flush_mm =3D 1; tlb->freed_tables =3D 1; tlb->cleared_puds =3D 1; - tlb_remove_ptdesc(tlb, pmd); + tlb_remove_table(tlb, pmd); } =20 /* @@ -123,7 +123,7 @@ static inline void pud_free_tlb(struct mmu_gather *tlb,= pud_t *pud, tlb->mm->context.flush_mm =3D 1; tlb->freed_tables =3D 1; tlb->cleared_p4ds =3D 1; - tlb_remove_ptdesc(tlb, pud); + tlb_remove_table(tlb, pud); } =20 /* @@ -141,7 +141,7 @@ static inline void p4d_free_tlb(struct mmu_gather *tlb,= p4d_t *p4d, __tlb_adjust_range(tlb, address, PAGE_SIZE); tlb->mm->context.flush_mm =3D 1; tlb->freed_tables =3D 1; - tlb_remove_ptdesc(tlb, p4d); + tlb_remove_table(tlb, p4d); } =20 #endif /* _S390_TLB_H */ diff --git a/include/asm-generic/tlb.h b/include/asm-generic/tlb.h index 8d6cfe5058543..583e95568f52b 100644 --- a/include/asm-generic/tlb.h +++ b/include/asm-generic/tlb.h @@ -211,7 +211,7 @@ struct mmu_table_batch { #ifndef __HAVE_ARCH_TLB_REMOVE_TABLE static inline void __tlb_remove_table(void *table) { - struct ptdesc *ptdesc =3D (struct ptdesc *)table; + struct ptdesc *ptdesc =3D page_ptdesc((struct page *)table); =20 pagetable_dtor(ptdesc); pagetable_free(ptdesc); @@ -499,11 +499,6 @@ static inline void tlb_remove_page(struct mmu_gather *= tlb, struct page *page) return tlb_remove_page_size(tlb, page, PAGE_SIZE); } =20 -static inline void tlb_remove_ptdesc(struct mmu_gather *tlb, void *pt) -{ - tlb_remove_table(tlb, pt); -} - static inline void tlb_change_page_size(struct mmu_gather *tlb, unsigned int page_size) { diff --git a/mm/mmu_gather.c b/mm/mmu_gather.c index 7aa6f18c500b2..c58ce4539c56f 100644 --- a/mm/mmu_gather.c +++ b/mm/mmu_gather.c @@ -314,18 +314,17 @@ static inline void tlb_table_invalidate(struct mmu_ga= ther *tlb) #ifdef CONFIG_PT_RECLAIM static inline void __tlb_remove_table_one_rcu(struct rcu_head *head) { - struct ptdesc *ptdesc; + struct page *page; =20 - ptdesc =3D container_of(head, struct ptdesc, pt_rcu_head); - __tlb_remove_table(ptdesc); + page =3D container_of(head, struct page, rcu_head); + __tlb_remove_table(page); } =20 static inline void __tlb_remove_table_one(void *table) { - struct ptdesc *ptdesc; + struct page *page =3D (struct page *)table; =20 - ptdesc =3D table; - call_rcu(&ptdesc->pt_rcu_head, __tlb_remove_table_one_rcu); + call_rcu(&page->rcu_head, __tlb_remove_table_one_rcu); } #else static inline void __tlb_remove_table_one(void *table) --=20 2.20.1 From nobody Thu Dec 18 09:39:52 2025 Received: from mail-pl1-f172.google.com (mail-pl1-f172.google.com [209.85.214.172]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3F1921B0416 for ; Wed, 18 Dec 2024 13:06:50 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734527211; cv=none; b=tSJrQE6hjSCVE7LQUzXrYK/KGKdmUNUnVxVK0PFAlOsEyOMb3JPdnXsAqH/BKUGw+N/eax5rLbGagYy+Iw56bPZg1mqTZoR1hqPkOTs6qoRB8tTYIEoUZk8FZvXkgknSQEcphCpH7FdnYifaPLfwBz5KPS0VcW2xRdgyvJRGyNg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734527211; c=relaxed/simple; bh=FWnScBZWDkJC1PU9kN58Q0gpTpGjLpyMzHDV709Ekfs=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=NBK0luxtqrpEl4YDf3lJ0wKvvoay9fzREm4XBH4ktfD8jPfhEaqctKSt3Z0h1waf7mauKD847CiYSvdvzbX4dy4VXiM6YiXPrEVJM1DfZzV6AWO+axShd+5cC81Gc8LbPOWN7YtUb10311Xvs1/YiN43N0qjmU+xnlttpV99+tw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=CxPTcnt6; arc=none smtp.client-ip=209.85.214.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="CxPTcnt6" Received: by mail-pl1-f172.google.com with SMTP id d9443c01a7336-2164b662090so51909455ad.1 for ; Wed, 18 Dec 2024 05:06:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1734527209; x=1735132009; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=KC7Ql8GT7M6xqj/drFleO18sYzKRSHwzPDd2hD5Jtik=; b=CxPTcnt6vWgAFFD4FCIXm7FpyxF5pJ5gaXYKleumX3a2WPRNZLejptUikq5W6ON2js kN1COofk55jd6S/wFc/34pdG/hSe1W8OCruZb5x/H3gU7vTJffPVYqzbHafd1w1A7bc5 9mTAo0o3eqVkAK1f0rYm+7yNvZV0/DAmZu6Y2hlujzB08Tb47ZROSAyoTTH5FiQ05/UI oRlm6s3lAT2HxJrIlRFASXxutCzthka2ZHriN7oikHDSgm+hsi+lmsbGU54S5AxQVkre jPweWyVP+JBPFsIJCEXN6VUd8CVrk+6ax3Ij9churU+Itie5jZxgOFKNfqB7uistl/YN 8EeQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734527209; x=1735132009; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=KC7Ql8GT7M6xqj/drFleO18sYzKRSHwzPDd2hD5Jtik=; b=u+smNNt67DkTMd90O6o781LyJ0ni6hfXTdzG5U5Mc/BDbmOW9kjHXxvXsrZ4r225ah sibivodQOZJyU2LJhBkMoHUSX0iKxTmmwiHSUpaMBF2f86P2Zsapan0GFz4nI8X+HeQS tRLIs1v10alDKP81QnJzTfi725kYX/xIHByViPSGl9FRykIh95J0EM95hpcRwm+LxsI6 8hJikZY3Ia10KNycetkwtuY+wtZypKxaGkiONj2TcjvghwztPWnKL637S9fN/munvOOe OMLrBob8nSTFBZbB7RPyDxqjvIrbz9tCKO3+otZIsQw2k8k5uISJL4uOFpVn4pQXoTQ7 PdJw== X-Forwarded-Encrypted: i=1; AJvYcCXLfpyL3YrOPBvWAGYL6/WwfMoFDEIDUXLZoeZkvCwuwTQ0d0oifno6/O8WiajH70x42K2uf1p7QbWDOzM=@vger.kernel.org X-Gm-Message-State: AOJu0YyTIISdw7pgaF2hT9k9534/byVAC7OCZgB97wANz+pNRrsjDPPQ wzNDC6mfUTYnQScPi7aL2eNlgm3oz6JpuU0BBFdfJjiSxw2TMkQtu9PBpOFoUkc= X-Gm-Gg: ASbGncsXuH2p4FdWHPrnKBYUV1rbkbR7mZeYw7OHQaEs5QdOTRnqnAY7dzb3xiVjYge BGVxqeKfMJBIgbyJ9NBYCMUqR+oJKqj/cjWuuZZ/h6Jy1mz6YsP/yWdUGk0k/UGuK93W+1Iqizk TLeprvBWBCIsiWjswCkjrOsz1WCPHR1j/RzBqIgLKucKK+gHxBTm+8ICHFWnGeglwcd+ap9Ze7Y YyZs5Gj8m+ZAu6nDoi3gJB79HkbQFQXPD4l+aTBY9t55CKdpY5ZmkXBofviNbJgFTpw6XzpyYNx ZaNRSXOnd3wUBKKvxTo3Xw== X-Google-Smtp-Source: AGHT+IFeQHKjf5x1j6HplQpPLqOMHr9gUGeYMwElc8Se13maebUbMGrfsZ8FVIlBe6kXPLktzNwjjA== X-Received: by 2002:a17:902:cccd:b0:215:6cb2:7877 with SMTP id d9443c01a7336-218d6fcc257mr40842835ad.4.1734527209409; Wed, 18 Dec 2024 05:06:49 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.238]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-218a1db755dsm75751825ad.42.2024.12.18.05.06.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 18 Dec 2024 05:06:49 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng Subject: [PATCH v2 15/15] mm: pgtable: introduce generic pagetable_dtor_free() Date: Wed, 18 Dec 2024 21:04:51 +0800 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The pte_free(), pmd_free(), __pud_free() and __p4d_free() in asm-generic/pgalloc.h and the generic __tlb_remove_table() are actually the same, so let's introduce pagetable_dtor_free() to deduplicate them. In addition, the pagetable_dtor_free() in s390 actually does the same thing, so let's s390 also calls generic pagetable_dtor_free(). Signed-off-by: Qi Zheng Suggested-by: Peter Zijlstra (Intel) --- arch/s390/mm/pgalloc.c | 18 ++++++------------ include/asm-generic/pgalloc.h | 23 ++++------------------- include/asm-generic/tlb.h | 5 +---- include/linux/mm.h | 8 ++++++++ 4 files changed, 19 insertions(+), 35 deletions(-) diff --git a/arch/s390/mm/pgalloc.c b/arch/s390/mm/pgalloc.c index 3e002dea6278f..1e0727be48eaf 100644 --- a/arch/s390/mm/pgalloc.c +++ b/arch/s390/mm/pgalloc.c @@ -180,32 +180,26 @@ unsigned long *page_table_alloc(struct mm_struct *mm) return table; } =20 -static void pagetable_dtor_free(struct ptdesc *ptdesc) -{ - pagetable_dtor(ptdesc); - pagetable_free(ptdesc); -} - void page_table_free(struct mm_struct *mm, unsigned long *table) { - struct ptdesc *ptdesc =3D virt_to_ptdesc(table); + struct page *page =3D virt_to_page(table); =20 - pagetable_dtor_free(ptdesc); + pagetable_dtor_free(page); } =20 #ifdef CONFIG_TRANSPARENT_HUGEPAGE static void pte_free_now(struct rcu_head *head) { - struct ptdesc *ptdesc =3D container_of(head, struct ptdesc, pt_rcu_head); + struct page *page =3D container_of(head, struct page, rcu_head); =20 - pagetable_dtor_free(ptdesc); + pagetable_dtor_free(page); } =20 void pte_free_defer(struct mm_struct *mm, pgtable_t pgtable) { - struct ptdesc *ptdesc =3D virt_to_ptdesc(pgtable); + struct page *page =3D virt_to_page(pgtable); =20 - call_rcu(&ptdesc->pt_rcu_head, pte_free_now); + call_rcu(&page->rcu_head, pte_free_now); /* * THPs are not allowed for KVM guests. Warn if pgste ever reaches here. * Turn to the generic pte_free_defer() version once gmap is removed. diff --git a/include/asm-generic/pgalloc.h b/include/asm-generic/pgalloc.h index 3673e9c29504e..370f5b579ff88 100644 --- a/include/asm-generic/pgalloc.h +++ b/include/asm-generic/pgalloc.h @@ -107,10 +107,7 @@ static inline pgtable_t pte_alloc_one_noprof(struct mm= _struct *mm) */ static inline void pte_free(struct mm_struct *mm, struct page *pte_page) { - struct ptdesc *ptdesc =3D page_ptdesc(pte_page); - - pagetable_dtor(ptdesc); - pagetable_free(ptdesc); + pagetable_dtor_free(pte_page); } =20 =20 @@ -150,11 +147,7 @@ static inline pmd_t *pmd_alloc_one_noprof(struct mm_st= ruct *mm, unsigned long ad #ifndef __HAVE_ARCH_PMD_FREE static inline void pmd_free(struct mm_struct *mm, pmd_t *pmd) { - struct ptdesc *ptdesc =3D virt_to_ptdesc(pmd); - - BUG_ON((unsigned long)pmd & (PAGE_SIZE-1)); - pagetable_dtor(ptdesc); - pagetable_free(ptdesc); + pagetable_dtor_free(virt_to_page(pmd)); } #endif =20 @@ -199,11 +192,7 @@ static inline pud_t *pud_alloc_one_noprof(struct mm_st= ruct *mm, unsigned long ad =20 static inline void __pud_free(struct mm_struct *mm, pud_t *pud) { - struct ptdesc *ptdesc =3D virt_to_ptdesc(pud); - - BUG_ON((unsigned long)pud & (PAGE_SIZE-1)); - pagetable_dtor(ptdesc); - pagetable_free(ptdesc); + pagetable_dtor_free(virt_to_page(pud)); } =20 #ifndef __HAVE_ARCH_PUD_FREE @@ -254,11 +243,7 @@ static inline p4d_t *p4d_alloc_one_noprof(struct mm_st= ruct *mm, unsigned long ad =20 static inline void __p4d_free(struct mm_struct *mm, p4d_t *p4d) { - struct ptdesc *ptdesc =3D virt_to_ptdesc(p4d); - - BUG_ON((unsigned long)p4d & (PAGE_SIZE-1)); - pagetable_dtor(ptdesc); - pagetable_free(ptdesc); + pagetable_dtor_free(virt_to_page(p4d)); } =20 #ifndef __HAVE_ARCH_P4D_FREE diff --git a/include/asm-generic/tlb.h b/include/asm-generic/tlb.h index 583e95568f52b..ef25169523602 100644 --- a/include/asm-generic/tlb.h +++ b/include/asm-generic/tlb.h @@ -211,10 +211,7 @@ struct mmu_table_batch { #ifndef __HAVE_ARCH_TLB_REMOVE_TABLE static inline void __tlb_remove_table(void *table) { - struct ptdesc *ptdesc =3D page_ptdesc((struct page *)table); - - pagetable_dtor(ptdesc); - pagetable_free(ptdesc); + pagetable_dtor_free(table); } #endif =20 diff --git a/include/linux/mm.h b/include/linux/mm.h index 736215c428293..f2d04eec66076 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -3095,6 +3095,14 @@ static inline void pagetable_dtor(struct ptdesc *ptd= esc) lruvec_stat_sub_folio(folio, NR_PAGETABLE); } =20 +static inline void pagetable_dtor_free(void *table) +{ + struct ptdesc *ptdesc =3D page_ptdesc((struct page *)table); + + pagetable_dtor(ptdesc); + pagetable_free(ptdesc); +} + static inline bool pagetable_pte_ctor(struct ptdesc *ptdesc) { struct folio *folio =3D ptdesc_folio(ptdesc); --=20 2.20.1