From nobody Thu Dec 18 23:20:46 2025 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A8792202F7C for ; Thu, 6 Mar 2025 22:45:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741301111; cv=none; b=cEWwn2+LrdDp+vCi6KNJV407ISi5wowzs1EyqFBJ9pr9uN31Ggw8ZKZQ0BIsmS+pnpFnaCudj55p3JsfsaN3jTE1M4I+oQ3dzhLwK5oo2rX++K1tp5vYfgCBdUIvdfRvNYOt/4mxK0OKl7owYFY/t085n7mhPya7DOiYfh65xmo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741301111; c=relaxed/simple; bh=6KyAmW1ynHK4gKKzbiuH5cEiOxuQlpK4quga9hHaa20=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=fnrXK5W9/YncRgE+yIJejsVze8UT89q6Mn5iGZmkxCb0QiBKaeDFGvnEyQrbYGzyiqqyeP+zuxDQ0Tjcw5oOAvn7TOxoIKBXrSJeXXa78Bkp8Mj5n+8rOw80gT4eEoZhdB60YblxReqsIirgYWeS+uH/Z/8o0DEowlDPo455HfI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=e4MPKx3w; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="e4MPKx3w" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1741301108; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=teoBvxfOPLZG+ZTVaUDHrQiabUlTMJDRM/IlVl3SrJk=; b=e4MPKx3wdW42onuazbBaIObbHmcGQtoFWg0jWj/+7G6l0llQZzmp86nSqdAFXqMZVV9S95 jNzoVxsP/tYoKMskxCODznXWz6TNe7jHgrFwdUZoKm8EQIPFehFrFzwZqOPP+E4ltmdBqf syXxYjtqGD0vf7IITz3GnJ0nXnPz93Q= Received: from mx-prod-mc-04.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-124-e9BIJS8YNq2T-zcJVrc-Tw-1; Thu, 06 Mar 2025 17:45:06 -0500 X-MC-Unique: e9BIJS8YNq2T-zcJVrc-Tw-1 X-Mimecast-MFC-AGG-ID: e9BIJS8YNq2T-zcJVrc-Tw_1741301105 Received: from mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.111]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-04.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id C1A0719560A3; Thu, 6 Mar 2025 22:45:04 +0000 (UTC) Received: from fedora.redhat.com (unknown [10.22.88.191]) by mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 4B98818009BC; Thu, 6 Mar 2025 22:45:02 +0000 (UTC) From: Luiz Capitulino To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, david@redhat.com, yuzhao@google.com, pasha.tatashin@soleen.com Cc: akpm@linux-foundation.org, hannes@cmpxchg.org, muchun.song@linux.dev, luizcap@redhat.com Subject: [PATCH v3 1/3] mm: page_ext: add an iteration API for page extensions Date: Thu, 6 Mar 2025 17:44:50 -0500 Message-ID: In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.111 Content-Type: text/plain; charset="utf-8" The page extension implementation assumes that all page extensions of a given page order are stored in the same memory section. The function page_ext_next() relies on this assumption by adding an offset to the current object to return the next adjacent page extension. This behavior works as expected for flatmem but fails for sparsemem when using 1G pages. The commit cf54f310d0d3 ("mm/hugetlb: use __GFP_COMP for gigantic folios") exposes this issue, making it possible for a crash when using page_owner or page_table_check page extensions. The problem is that for 1G pages, the page extensions may span memory section boundaries and be stored in different memory sections. This issue was not visible before commit cf54f310d0d3 ("mm/hugetlb: use __GFP_COMP for gigantic folios") because alloc_contig_pages() never passed more than MAX_PAGE_ORDER to post_alloc_hook(). However, the series introducing mentioned commit changed this behavior allowing the full 1G page order to be passed. Reproducer: 1. Build the kernel with CONFIG_SPARSEMEM=3Dy and table extensions support 2. Pass 'default_hugepagesz=3D1 page_owner=3Don' in the kernel command-line 3. Reserve one 1G page at run-time, this should crash (backtrace below) To address this issue, this commit introduces a new API for iterating through page extensions. The main iteration macro is for_each_page_ext() and it must be called with the RCU read lock taken. Here's an usage example: """ struct page_ext_iter iter; struct page_ext *page_ext; ... rcu_read_lock(); for_each_page_ext(page, 1 << order, page_ext, iter) { struct my_page_ext *obj =3D get_my_page_ext_obj(page_ext); ... } rcu_read_unlock(); """ The loop construct uses page_ext_iter_next() which checks to see if we have crossed sections in the iteration. In this case, page_ext_iter_next() retrieves the next page_ext object from another section. Thanks to David Hildenbrand for helping identify the root cause and providing suggestions on how to fix and optmize the solution (final implementation and bugs are all mine through). Lastly, here's the backtrace, without kasan you can get random crashes: [ 76.052526] BUG: KASAN: slab-out-of-bounds in __update_page_owner_handle= +0x238/0x298 [ 76.060283] Write of size 4 at addr ffff07ff96240038 by task tee/3598 [ 76.066714] [ 76.068203] CPU: 88 UID: 0 PID: 3598 Comm: tee Kdump: loaded Not tainted= 6.13.0-rep1 #3 [ 76.076202] Hardware name: WIWYNN Mt.Jade Server System B81.030Z1.0007/M= t.Jade Motherboard, BIOS 2.10.20220810 (SCP: 2.10.20220810) 2022/08/10 [ 76.088972] Call trace: [ 76.091411] show_stack+0x20/0x38 (C) [ 76.095073] dump_stack_lvl+0x80/0xf8 [ 76.098733] print_address_description.constprop.0+0x88/0x398 [ 76.104476] print_report+0xa8/0x278 [ 76.108041] kasan_report+0xa8/0xf8 [ 76.111520] __asan_report_store4_noabort+0x20/0x30 [ 76.116391] __update_page_owner_handle+0x238/0x298 [ 76.121259] __set_page_owner+0xdc/0x140 [ 76.125173] post_alloc_hook+0x190/0x1d8 [ 76.129090] alloc_contig_range_noprof+0x54c/0x890 [ 76.133874] alloc_contig_pages_noprof+0x35c/0x4a8 [ 76.138656] alloc_gigantic_folio.isra.0+0x2c0/0x368 [ 76.143616] only_alloc_fresh_hugetlb_folio.isra.0+0x24/0x150 [ 76.149353] alloc_pool_huge_folio+0x11c/0x1f8 [ 76.153787] set_max_huge_pages+0x364/0xca8 [ 76.157961] __nr_hugepages_store_common+0xb0/0x1a0 [ 76.162829] nr_hugepages_store+0x108/0x118 [ 76.167003] kobj_attr_store+0x3c/0x70 [ 76.170745] sysfs_kf_write+0xfc/0x188 [ 76.174492] kernfs_fop_write_iter+0x274/0x3e0 [ 76.178927] vfs_write+0x64c/0x8e0 [ 76.182323] ksys_write+0xf8/0x1f0 [ 76.185716] __arm64_sys_write+0x74/0xb0 [ 76.189630] invoke_syscall.constprop.0+0xd8/0x1e0 [ 76.194412] do_el0_svc+0x164/0x1e0 [ 76.197891] el0_svc+0x40/0xe0 [ 76.200939] el0t_64_sync_handler+0x144/0x168 [ 76.205287] el0t_64_sync+0x1ac/0x1b0 Fixes: cf54f310d0d3 ("mm/hugetlb: use __GFP_COMP for gigantic folios") Signed-off-by: Luiz Capitulino Acked-by: David Hildenbrand --- include/linux/page_ext.h | 93 ++++++++++++++++++++++++++++++++++++++++ mm/page_ext.c | 13 ++++++ 2 files changed, 106 insertions(+) diff --git a/include/linux/page_ext.h b/include/linux/page_ext.h index e4b48a0dda244..76c817162d2fb 100644 --- a/include/linux/page_ext.h +++ b/include/linux/page_ext.h @@ -3,6 +3,7 @@ #define __LINUX_PAGE_EXT_H =20 #include +#include #include =20 struct pglist_data; @@ -69,16 +70,31 @@ extern void page_ext_init(void); static inline void page_ext_init_flatmem_late(void) { } + +static inline bool page_ext_iter_next_fast_possible(unsigned long next_pfn) +{ + /* + * page_ext is allocated per memory section. Once we cross a + * memory section, we have to fetch the new pointer. + */ + return next_pfn % PAGES_PER_SECTION; +} #else extern void page_ext_init_flatmem(void); extern void page_ext_init_flatmem_late(void); static inline void page_ext_init(void) { } + +static inline bool page_ext_iter_next_fast_possible(unsigned long next_pfn) +{ + return true; +} #endif =20 extern struct page_ext *page_ext_get(const struct page *page); extern void page_ext_put(struct page_ext *page_ext); +extern struct page_ext *page_ext_lookup(unsigned long pfn); =20 static inline void *page_ext_data(struct page_ext *page_ext, struct page_ext_operations *ops) @@ -93,6 +109,83 @@ static inline struct page_ext *page_ext_next(struct pag= e_ext *curr) return next; } =20 +struct page_ext_iter { + unsigned long index; + unsigned long start_pfn; + struct page_ext *page_ext; +}; + +/** + * page_ext_iter_begin() - Prepare for iterating through page extensions. + * @iter: page extension iterator. + * @pfn: PFN of the page we're interested in. + * + * Must be called with RCU read lock taken. + * + * Return: NULL if no page_ext exists for this page. + */ +static inline struct page_ext *page_ext_iter_begin(struct page_ext_iter *i= ter, + unsigned long pfn) +{ + iter->index =3D 0; + iter->start_pfn =3D pfn; + iter->page_ext =3D page_ext_lookup(pfn); + + return iter->page_ext; +} + +/** + * page_ext_iter_next() - Get next page extension + * @iter: page extension iterator. + * + * Must be called with RCU read lock taken. + * + * Return: NULL if no next page_ext exists. + */ +static inline struct page_ext *page_ext_iter_next(struct page_ext_iter *it= er) +{ + unsigned long pfn; + + if (WARN_ON_ONCE(!iter->page_ext)) + return NULL; + + iter->index++; + pfn =3D iter->start_pfn + iter->index; + + if (page_ext_iter_next_fast_possible(pfn)) + iter->page_ext =3D page_ext_next(iter->page_ext); + else + iter->page_ext =3D page_ext_lookup(pfn); + + return iter->page_ext; +} + +/** + * page_ext_iter_get() - Get current page extension + * @iter: page extension iterator. + * + * Return: NULL if no page_ext exists for this iterator. + */ +static inline struct page_ext *page_ext_iter_get(const struct page_ext_ite= r *iter) +{ + return iter->page_ext; +} + +/** + * for_each_page_ext(): iterate through page_ext objects. + * @__page: the page we're interested in + * @__pgcount: how many pages to iterate through + * @__page_ext: struct page_ext pointer where the current page_ext + * object is returned + * @__iter: struct page_ext_iter object (defined in the stack) + * + * IMPORTANT: must be called with RCU read lock taken. + */ +#define for_each_page_ext(__page, __pgcount, __page_ext, __iter) \ + for (__page_ext =3D page_ext_iter_begin(&__iter, page_to_pfn(__page));\ + __page_ext && __iter.index < __pgcount; \ + __page_ext =3D page_ext_iter_next(&__iter)) + #else /* !CONFIG_PAGE_EXTENSION */ struct page_ext; =20 diff --git a/mm/page_ext.c b/mm/page_ext.c index 641d93f6af4c1..c351fdfe9e9a5 100644 --- a/mm/page_ext.c +++ b/mm/page_ext.c @@ -507,6 +507,19 @@ void __meminit pgdat_page_ext_init(struct pglist_data = *pgdat) =20 #endif =20 +/** + * page_ext_lookup() - Lookup a page extension for a PFN. + * @pfn: PFN of the page we're interested in. + * + * Must be called with RCU read lock taken and @pfn must be valid. + * + * Return: NULL if no page_ext exists for this page. + */ +struct page_ext *page_ext_lookup(unsigned long pfn) +{ + return lookup_page_ext(pfn_to_page(pfn)); +} + /** * page_ext_get() - Get the extended information for a page. * @page: The page we're interested in. --=20 2.48.1 From nobody Thu Dec 18 23:20:46 2025 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0909627CB0C for ; Thu, 6 Mar 2025 22:45:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741301124; cv=none; b=MbBYtaW8DCV3CR/XDXqdJeZ6kA9U3dZ7PJF9RPgcGMe9C/Mk4E6P3GY81pg227iQ28vvPTWgpgsg2uxUKacO7nzddta8WUQk0rWxUWENFGasghT88qlNjYkxq8JHw4wI3t2pXt7TyLunvmyfmt77WHL4tD8kvCziCXs4Yrbl4dY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741301124; c=relaxed/simple; bh=3OaQdiUuTcyEIe69qMe8cf9GJuNLKgFZ12pGgelBBDA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=L97CIQZRIcwSbVllzi67cdtxOiQCFfikt/+t7AOPaXn9iW+tsq1kQ1pbxx4dS9M7pBqEtB+PFOdQA8XkNruKKIniYBwDaCiuGwslT0Q7ygkKPvnndih2Lt6xAyK8CYhPKafCIc+/AhDWiWur4SrQQsXzr1g0ni61aY+ZsRfVA0A= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=L7Jb+1UQ; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="L7Jb+1UQ" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1741301121; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=igjG5KqbdWExRJ3eRwEZliK2f0M4oSjOk76qmgyzu2Y=; b=L7Jb+1UQn0RwutvR3DQ2j9F4eYwdVBBYLjxbsceDIM7DhQOQSv1xjKyz7AiD2QyvVfcGiw /dtVpGPGYXOyKfC+On3YMvTrtS6SMzBAl27TZ5Kq+WymW1lBuv0iGkbhIMxtGF8JPWDQby aqOqzyg5e7KozT+uqqdUmAMyiSzP3h0= Received: from mx-prod-mc-05.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-688-eFNtnSS5PtelNava3YdU2w-1; Thu, 06 Mar 2025 17:45:08 -0500 X-MC-Unique: eFNtnSS5PtelNava3YdU2w-1 X-Mimecast-MFC-AGG-ID: eFNtnSS5PtelNava3YdU2w_1741301107 Received: from mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.111]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-05.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id CB792195608A; Thu, 6 Mar 2025 22:45:06 +0000 (UTC) Received: from fedora.redhat.com (unknown [10.22.88.191]) by mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id E540B18001D3; Thu, 6 Mar 2025 22:45:04 +0000 (UTC) From: Luiz Capitulino To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, david@redhat.com, yuzhao@google.com, pasha.tatashin@soleen.com Cc: akpm@linux-foundation.org, hannes@cmpxchg.org, muchun.song@linux.dev, luizcap@redhat.com Subject: [PATCH v3 2/3] mm: page_table_check: use new iteration API Date: Thu, 6 Mar 2025 17:44:51 -0500 Message-ID: In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.111 Content-Type: text/plain; charset="utf-8" The page_ext_next() function assumes that page extension objects for a page order allocation always reside in the same memory section, which may not be true and could lead to crashes. Use the new page_ext iteration API instead. Fixes: cf54f310d0d3 ("mm/hugetlb: use __GFP_COMP for gigantic folios") Acked-by: David Hildenbrand Signed-off-by: Luiz Capitulino --- mm/page_table_check.c | 39 ++++++++++++--------------------------- 1 file changed, 12 insertions(+), 27 deletions(-) diff --git a/mm/page_table_check.c b/mm/page_table_check.c index 509c6ef8de400..e11bebf23e36f 100644 --- a/mm/page_table_check.c +++ b/mm/page_table_check.c @@ -62,24 +62,20 @@ static struct page_table_check *get_page_table_check(st= ruct page_ext *page_ext) */ static void page_table_check_clear(unsigned long pfn, unsigned long pgcnt) { + struct page_ext_iter iter; struct page_ext *page_ext; struct page *page; - unsigned long i; bool anon; =20 if (!pfn_valid(pfn)) return; =20 page =3D pfn_to_page(pfn); - page_ext =3D page_ext_get(page); - - if (!page_ext) - return; - BUG_ON(PageSlab(page)); anon =3D PageAnon(page); =20 - for (i =3D 0; i < pgcnt; i++) { + rcu_read_lock(); + for_each_page_ext(page, pgcnt, page_ext, iter) { struct page_table_check *ptc =3D get_page_table_check(page_ext); =20 if (anon) { @@ -89,9 +85,8 @@ static void page_table_check_clear(unsigned long pfn, uns= igned long pgcnt) BUG_ON(atomic_read(&ptc->anon_map_count)); BUG_ON(atomic_dec_return(&ptc->file_map_count) < 0); } - page_ext =3D page_ext_next(page_ext); } - page_ext_put(page_ext); + rcu_read_unlock(); } =20 /* @@ -102,24 +97,20 @@ static void page_table_check_clear(unsigned long pfn, = unsigned long pgcnt) static void page_table_check_set(unsigned long pfn, unsigned long pgcnt, bool rw) { + struct page_ext_iter iter; struct page_ext *page_ext; struct page *page; - unsigned long i; bool anon; =20 if (!pfn_valid(pfn)) return; =20 page =3D pfn_to_page(pfn); - page_ext =3D page_ext_get(page); - - if (!page_ext) - return; - BUG_ON(PageSlab(page)); anon =3D PageAnon(page); =20 - for (i =3D 0; i < pgcnt; i++) { + rcu_read_lock(); + for_each_page_ext(page, pgcnt, page_ext, iter) { struct page_table_check *ptc =3D get_page_table_check(page_ext); =20 if (anon) { @@ -129,9 +120,8 @@ static void page_table_check_set(unsigned long pfn, uns= igned long pgcnt, BUG_ON(atomic_read(&ptc->anon_map_count)); BUG_ON(atomic_inc_return(&ptc->file_map_count) < 0); } - page_ext =3D page_ext_next(page_ext); } - page_ext_put(page_ext); + rcu_read_unlock(); } =20 /* @@ -140,24 +130,19 @@ static void page_table_check_set(unsigned long pfn, u= nsigned long pgcnt, */ void __page_table_check_zero(struct page *page, unsigned int order) { + struct page_ext_iter iter; struct page_ext *page_ext; - unsigned long i; =20 BUG_ON(PageSlab(page)); =20 - page_ext =3D page_ext_get(page); - - if (!page_ext) - return; - - for (i =3D 0; i < (1ul << order); i++) { + rcu_read_lock(); + for_each_page_ext(page, 1 << order, page_ext, iter) { struct page_table_check *ptc =3D get_page_table_check(page_ext); =20 BUG_ON(atomic_read(&ptc->anon_map_count)); BUG_ON(atomic_read(&ptc->file_map_count)); - page_ext =3D page_ext_next(page_ext); } - page_ext_put(page_ext); + rcu_read_unlock(); } =20 void __page_table_check_pte_clear(struct mm_struct *mm, pte_t pte) --=20 2.48.1 From nobody Thu Dec 18 23:20:46 2025 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 46B2326E970 for ; Thu, 6 Mar 2025 22:45:34 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741301137; cv=none; b=D+PgfTpOUBNKQXkn4ZTc4TMiU4cDIsHpyVks5BPrmztRndL0P47Ma8+znVhsLUBXGH5mjmY+C2boXJ6nZzWw5/RsN8iuBgTx8d6DBE5DW3HV9PzEHifyeK4qgPU6iwbMbs9M46bomdm0LCtRn+us01hkEtuSKjlh6mlaY8sND3o= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741301137; c=relaxed/simple; bh=z3Mz6WoS2TV3GPZsEkpYUOvUgoXUcaqXReBJbzMbycw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=ESsLcEgJXJf9u8jfI1OqB/C1LWO2nlN4R/FCG/GHNM1ebtxY2Orzp6OHImkEJVq8ARcwgWnd6Q1rhSQTnkFe4qF1BjmUBPsYOQc17Q9ijBpVlHfkX4/Syz6nN6EURYI4V+VJFK8O3Cy2ugoHfo3eDnmPhmhzLEaSe0n7mYRFqUA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=VSLJM7nI; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="VSLJM7nI" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1741301134; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=WRxmppCKJxJfC19HA1OxBrFfA0HTaxCM6cQbJYX0rnQ=; b=VSLJM7nIBh5NES0QP1mVpZfpK/q4gD7+kYr+UthsI6vrHAIubfXXMEt0suLW+YQUZ79oO0 5FXSL6od9edV2VFxSI4EDikUueGjJLYi2AtKNFXUWjCH2Hsv2LklngHkgEpns4i9kOD6qE W2awN/usfbVbeHrogOl+2TEE9z5oOF4= Received: from mx-prod-mc-08.mail-002.prod.us-west-2.aws.redhat.com (ec2-35-165-154-97.us-west-2.compute.amazonaws.com [35.165.154.97]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-134-EFFnaN7fO9OoBTQOWk3X2g-1; Thu, 06 Mar 2025 17:45:10 -0500 X-MC-Unique: EFFnaN7fO9OoBTQOWk3X2g-1 X-Mimecast-MFC-AGG-ID: EFFnaN7fO9OoBTQOWk3X2g_1741301109 Received: from mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.111]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 54743180AF7A; Thu, 6 Mar 2025 22:45:09 +0000 (UTC) Received: from fedora.redhat.com (unknown [10.22.88.191]) by mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 1D81A18009BC; Thu, 6 Mar 2025 22:45:06 +0000 (UTC) From: Luiz Capitulino To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, david@redhat.com, yuzhao@google.com, pasha.tatashin@soleen.com Cc: akpm@linux-foundation.org, hannes@cmpxchg.org, muchun.song@linux.dev, luizcap@redhat.com Subject: [PATCH v3 3/3] mm: page_owner: use new iteration API Date: Thu, 6 Mar 2025 17:44:52 -0500 Message-ID: <93c80b040960fa2ebab4a9729073f77a30649862.1741301089.git.luizcap@redhat.com> In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.111 Content-Type: text/plain; charset="utf-8" The page_ext_next() function assumes that page extension objects for a page order allocation always reside in the same memory section, which may not be true and could lead to crashes. Use the new page_ext iteration API instead. Fixes: cf54f310d0d3 ("mm/hugetlb: use __GFP_COMP for gigantic folios") Signed-off-by: Luiz Capitulino Acked-by: David Hildenbrand --- mm/page_owner.c | 84 +++++++++++++++++++++++-------------------------- 1 file changed, 39 insertions(+), 45 deletions(-) diff --git a/mm/page_owner.c b/mm/page_owner.c index 2d6360eaccbb6..65adc66582d82 100644 --- a/mm/page_owner.c +++ b/mm/page_owner.c @@ -229,17 +229,19 @@ static void dec_stack_record_count(depot_stack_handle= _t handle, handle); } =20 -static inline void __update_page_owner_handle(struct page_ext *page_ext, +static inline void __update_page_owner_handle(struct page *page, depot_stack_handle_t handle, unsigned short order, gfp_t gfp_mask, short last_migrate_reason, u64 ts_nsec, pid_t pid, pid_t tgid, char *comm) { - int i; + struct page_ext_iter iter; + struct page_ext *page_ext; struct page_owner *page_owner; =20 - for (i =3D 0; i < (1 << order); i++) { + rcu_read_lock(); + for_each_page_ext(page, 1 << order, page_ext, iter) { page_owner =3D get_page_owner(page_ext); page_owner->handle =3D handle; page_owner->order =3D order; @@ -252,20 +254,22 @@ static inline void __update_page_owner_handle(struct = page_ext *page_ext, sizeof(page_owner->comm)); __set_bit(PAGE_EXT_OWNER, &page_ext->flags); __set_bit(PAGE_EXT_OWNER_ALLOCATED, &page_ext->flags); - page_ext =3D page_ext_next(page_ext); } + rcu_read_unlock(); } =20 -static inline void __update_page_owner_free_handle(struct page_ext *page_e= xt, +static inline void __update_page_owner_free_handle(struct page *page, depot_stack_handle_t handle, unsigned short order, pid_t pid, pid_t tgid, u64 free_ts_nsec) { - int i; + struct page_ext_iter iter; + struct page_ext *page_ext; struct page_owner *page_owner; =20 - for (i =3D 0; i < (1 << order); i++) { + rcu_read_lock(); + for_each_page_ext(page, 1 << order, page_ext, iter) { page_owner =3D get_page_owner(page_ext); /* Only __reset_page_owner() wants to clear the bit */ if (handle) { @@ -275,8 +279,8 @@ static inline void __update_page_owner_free_handle(stru= ct page_ext *page_ext, page_owner->free_ts_nsec =3D free_ts_nsec; page_owner->free_pid =3D current->pid; page_owner->free_tgid =3D current->tgid; - page_ext =3D page_ext_next(page_ext); } + rcu_read_unlock(); } =20 void __reset_page_owner(struct page *page, unsigned short order) @@ -293,11 +297,11 @@ void __reset_page_owner(struct page *page, unsigned s= hort order) =20 page_owner =3D get_page_owner(page_ext); alloc_handle =3D page_owner->handle; + page_ext_put(page_ext); =20 handle =3D save_stack(GFP_NOWAIT | __GFP_NOWARN); - __update_page_owner_free_handle(page_ext, handle, order, current->pid, + __update_page_owner_free_handle(page, handle, order, current->pid, current->tgid, free_ts_nsec); - page_ext_put(page_ext); =20 if (alloc_handle !=3D early_handle) /* @@ -313,19 +317,13 @@ void __reset_page_owner(struct page *page, unsigned s= hort order) noinline void __set_page_owner(struct page *page, unsigned short order, gfp_t gfp_mask) { - struct page_ext *page_ext; u64 ts_nsec =3D local_clock(); depot_stack_handle_t handle; =20 handle =3D save_stack(gfp_mask); - - page_ext =3D page_ext_get(page); - if (unlikely(!page_ext)) - return; - __update_page_owner_handle(page_ext, handle, order, gfp_mask, -1, + __update_page_owner_handle(page, handle, order, gfp_mask, -1, ts_nsec, current->pid, current->tgid, current->comm); - page_ext_put(page_ext); inc_stack_record_count(handle, gfp_mask, 1 << order); } =20 @@ -344,44 +342,42 @@ void __set_page_owner_migrate_reason(struct page *pag= e, int reason) =20 void __split_page_owner(struct page *page, int old_order, int new_order) { - int i; - struct page_ext *page_ext =3D page_ext_get(page); + struct page_ext_iter iter; + struct page_ext *page_ext; struct page_owner *page_owner; =20 - if (unlikely(!page_ext)) - return; - - for (i =3D 0; i < (1 << old_order); i++) { + rcu_read_lock(); + for_each_page_ext(page, 1 << old_order, page_ext, iter) { page_owner =3D get_page_owner(page_ext); page_owner->order =3D new_order; - page_ext =3D page_ext_next(page_ext); } - page_ext_put(page_ext); + rcu_read_unlock(); } =20 void __folio_copy_owner(struct folio *newfolio, struct folio *old) { - int i; - struct page_ext *old_ext; - struct page_ext *new_ext; + struct page_ext *page_ext; + struct page_ext_iter iter; struct page_owner *old_page_owner; struct page_owner *new_page_owner; depot_stack_handle_t migrate_handle; =20 - old_ext =3D page_ext_get(&old->page); - if (unlikely(!old_ext)) + page_ext =3D page_ext_get(&old->page); + if (unlikely(!page_ext)) return; =20 - new_ext =3D page_ext_get(&newfolio->page); - if (unlikely(!new_ext)) { - page_ext_put(old_ext); + old_page_owner =3D get_page_owner(page_ext); + page_ext_put(page_ext); + + page_ext =3D page_ext_get(&newfolio->page); + if (unlikely(!page_ext)) return; - } =20 - old_page_owner =3D get_page_owner(old_ext); - new_page_owner =3D get_page_owner(new_ext); + new_page_owner =3D get_page_owner(page_ext); + page_ext_put(page_ext); + migrate_handle =3D new_page_owner->handle; - __update_page_owner_handle(new_ext, old_page_owner->handle, + __update_page_owner_handle(&newfolio->page, old_page_owner->handle, old_page_owner->order, old_page_owner->gfp_mask, old_page_owner->last_migrate_reason, old_page_owner->ts_nsec, old_page_owner->pid, @@ -391,7 +387,7 @@ void __folio_copy_owner(struct folio *newfolio, struct = folio *old) * will be freed after migration. Keep them until then as they may be * useful. */ - __update_page_owner_free_handle(new_ext, 0, old_page_owner->order, + __update_page_owner_free_handle(&newfolio->page, 0, old_page_owner->order, old_page_owner->free_pid, old_page_owner->free_tgid, old_page_owner->free_ts_nsec); @@ -400,14 +396,12 @@ void __folio_copy_owner(struct folio *newfolio, struc= t folio *old) * for the new one and the old folio otherwise there will be an imbalance * when subtracting those pages from the stack. */ - for (i =3D 0; i < (1 << new_page_owner->order); i++) { + rcu_read_lock(); + for_each_page_ext(&old->page, 1 << new_page_owner->order, page_ext, iter)= { + old_page_owner =3D get_page_owner(page_ext); old_page_owner->handle =3D migrate_handle; - old_ext =3D page_ext_next(old_ext); - old_page_owner =3D get_page_owner(old_ext); } - - page_ext_put(new_ext); - page_ext_put(old_ext); + rcu_read_unlock(); } =20 void pagetypeinfo_showmixedcount_print(struct seq_file *m, @@ -813,7 +807,7 @@ static void init_pages_in_zone(pg_data_t *pgdat, struct= zone *zone) goto ext_put_continue; =20 /* Found early allocated page */ - __update_page_owner_handle(page_ext, early_handle, 0, 0, + __update_page_owner_handle(page, early_handle, 0, 0, -1, local_clock(), current->pid, current->tgid, current->comm); count++; --=20 2.48.1