From nobody Tue Jun 30 20:11:01 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8836FC433F5 for ; Mon, 10 Jan 2022 04:25:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238813AbiAJEZo (ORCPT ); Sun, 9 Jan 2022 23:25:44 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36580 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238518AbiAJEY3 (ORCPT ); Sun, 9 Jan 2022 23:24:29 -0500 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1ACC3C061756 for ; Sun, 9 Jan 2022 20:24:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=rKKpy0oluulkvLBfnA/9hhQL753tZQacHkRyC1WeJU0=; b=sGdCDh4ZVcVD4kmjTA+PY/SRmA Xy0yRyA5WDiywX9swJnbRDtdRVp/t9tnbO0FbJIbD8klaRw1FSmjZ6aQ3s87Zqfm83GSNw9+W6EBB jUEqlKQg87FdYHfliSejnX27WaWwrifLIdLGxqtuIivNfw8F47wH/NJVLGNkh9RiWTPp/jc+Mjpr/ 4OzgBeACGD0VX+bJqHy4Oh3gbqvvXQK2HdoX5rjTR92I/QwNL951f1izvcp4+2VZw0t66GoH8Pp2c dzDjq+odwp2UsO4gda+0abuQYve0KMcJcKKClan7zrqcmr/MWnR0NbwZfC0BJO5Z0xGFZMxeKcQML SErKzKMg==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1n6mE7-0025wS-Uo; Mon, 10 Jan 2022 04:24:11 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , John Hubbard , Christoph Hellwig , William Kucharski , linux-kernel@vger.kernel.org, Jason Gunthorpe Subject: [PATCH v2 01/28] gup: Remove for_each_compound_range() Date: Mon, 10 Jan 2022 04:23:39 +0000 Message-Id: <20220110042406.499429-2-willy@infradead.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220110042406.499429-1-willy@infradead.org> References: <20220110042406.499429-1-willy@infradead.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" This macro doesn't simplify the users; it's easier to just call compound_range_next() inside the loop. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Jason Gunthorpe Reviewed-by: John Hubbard Reviewed-by: William Kucharski --- mm/gup.c | 12 ++---------- 1 file changed, 2 insertions(+), 10 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index 2c51e9748a6a..7a07e0c00bf5 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -236,9 +236,6 @@ static inline void compound_range_next(unsigned long i,= unsigned long npages, struct page *next, *page; unsigned int nr =3D 1; =20 - if (i >=3D npages) - return; - next =3D *list + i; page =3D compound_head(next); if (PageCompound(page) && compound_order(page) >=3D 1) @@ -249,12 +246,6 @@ static inline void compound_range_next(unsigned long i= , unsigned long npages, *ntails =3D nr; } =20 -#define for_each_compound_range(__i, __list, __npages, __head, __ntails) \ - for (__i =3D 0, \ - compound_range_next(__i, __npages, __list, &(__head), &(__ntails)); \ - __i < __npages; __i +=3D __ntails, \ - compound_range_next(__i, __npages, __list, &(__head), &(__ntails))) - static inline void compound_next(unsigned long i, unsigned long npages, struct page **list, struct page **head, unsigned int *ntails) @@ -371,7 +362,8 @@ void unpin_user_page_range_dirty_lock(struct page *page= , unsigned long npages, struct page *head; unsigned int ntails; =20 - for_each_compound_range(index, &page, npages, head, ntails) { + for (index =3D 0; index < npages; index +=3D ntails) { + compound_range_next(index, npages, &page, &head, &ntails); if (make_dirty && !PageDirty(head)) set_page_dirty_lock(head); put_compound_head(head, ntails, FOLL_PIN); --=20 2.33.0 From nobody Tue Jun 30 20:11:01 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 12C17C433F5 for ; Mon, 10 Jan 2022 04:26:54 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238589AbiAJE0x (ORCPT ); Sun, 9 Jan 2022 23:26:53 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36668 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238658AbiAJEYy (ORCPT ); Sun, 9 Jan 2022 23:24:54 -0500 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 33239C06173F for ; Sun, 9 Jan 2022 20:24:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=8C3hUhjCGJ2Sdk9gK8xC03PTtTrcCZ/ERUofNnCUuq0=; b=bcS4FwNQbPo7ETR9B5dTcak8o1 MF2eSv6wiCnWc63DybJky7QZlDMkktmdQuJDgJcBCVo+tlYbuzp4WhHd1o9huxTeqgfkcCugweGYP gqTHByJSSStlrJazHskBOCGem9kIh1fzgaOvfDyCl9xN7pUCFyE5JowreUCy+8ESnLI40DXcyKRIX PFosyUxUrqzjI8XE4dsZU+x894HS9y3PwzR7QIMWUepl8eIIEhiJbJkAH29JfE6k94S4GTS1mQg6F mdIsdT/h+YjPWErGXlAFdxrKlkxuiREifCZtqAVSC/2Y0xG/mxTUSwrGWKB0nNmlq0XOi4Qmxh3tP Yp3tCFHw==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1n6mE8-0025wU-1G; Mon, 10 Jan 2022 04:24:12 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , John Hubbard , Christoph Hellwig , William Kucharski , linux-kernel@vger.kernel.org, Jason Gunthorpe Subject: [PATCH v2 02/28] gup: Remove for_each_compound_head() Date: Mon, 10 Jan 2022 04:23:40 +0000 Message-Id: <20220110042406.499429-3-willy@infradead.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220110042406.499429-1-willy@infradead.org> References: <20220110042406.499429-1-willy@infradead.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" This macro doesn't simplify the users; it's easier to just call compound_next() inside a standard loop. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Jason Gunthorpe Reviewed-by: John Hubbard Reviewed-by: William Kucharski --- mm/gup.c | 16 +++++----------- 1 file changed, 5 insertions(+), 11 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index 7a07e0c00bf5..86f8d843de72 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -253,9 +253,6 @@ static inline void compound_next(unsigned long i, unsig= ned long npages, struct page *page; unsigned int nr; =20 - if (i >=3D npages) - return; - page =3D compound_head(list[i]); for (nr =3D i + 1; nr < npages; nr++) { if (compound_head(list[nr]) !=3D page) @@ -266,12 +263,6 @@ static inline void compound_next(unsigned long i, unsi= gned long npages, *ntails =3D nr - i; } =20 -#define for_each_compound_head(__i, __list, __npages, __head, __ntails) \ - for (__i =3D 0, \ - compound_next(__i, __npages, __list, &(__head), &(__ntails)); \ - __i < __npages; __i +=3D __ntails, \ - compound_next(__i, __npages, __list, &(__head), &(__ntails))) - /** * unpin_user_pages_dirty_lock() - release and optionally dirty gup-pinned= pages * @pages: array of pages to be maybe marked dirty, and definitely releas= ed. @@ -306,7 +297,8 @@ void unpin_user_pages_dirty_lock(struct page **pages, u= nsigned long npages, return; } =20 - for_each_compound_head(index, pages, npages, head, ntails) { + for (index =3D 0; index < npages; index +=3D ntails) { + compound_next(index, npages, pages, &head, &ntails); /* * Checking PageDirty at this point may race with * clear_page_dirty_for_io(), but that's OK. Two key @@ -394,8 +386,10 @@ void unpin_user_pages(struct page **pages, unsigned lo= ng npages) if (WARN_ON(IS_ERR_VALUE(npages))) return; =20 - for_each_compound_head(index, pages, npages, head, ntails) + for (index =3D 0; index < npages; index +=3D ntails) { + compound_next(index, npages, pages, &head, &ntails); put_compound_head(head, ntails, FOLL_PIN); + } } EXPORT_SYMBOL(unpin_user_pages); =20 --=20 2.33.0 From nobody Tue Jun 30 20:11:01 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C34E3C433EF for ; Mon, 10 Jan 2022 04:27:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238783AbiAJE1N (ORCPT ); Sun, 9 Jan 2022 23:27:13 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36644 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238719AbiAJEZF (ORCPT ); Sun, 9 Jan 2022 23:25:05 -0500 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3F8A0C061212 for ; Sun, 9 Jan 2022 20:24:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=gT8RG9vD2iSbfvGg96Co6y7w74G3UgjYv7qBo/++MyI=; b=L2bhR2Cjml5DgbFsb0VbH/LASg hlrKrN5EDZSd+ik3iO94aKLBEhpESoh/ldGOKrbgbPmXGIu2qcbXCiAV8shqq+2/ODx3xlHB3+uR1 WXT2IWUHvpb3nAyxNS6UB8iQuPoLnEXLBQjD+gbex6c8IeDmYwqqlmyXJYD5WTjcNo12L6LLjE4i6 5IM8D1DQiY1HLOCzEqDN7ohzauoczPQSlIU0qIRibEvBiy4Aoezi+R2uI2P9jUtjmtREtxignvDIQ cPKhkILOzRhHzQBEAIQvme1db678BNuBgNhF3s0NoYWQDURb8fcVaWWKuI8yOQSqfYGyJIuDXyPx1 dFjKdG2A==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1n6mE8-0025wW-3h; Mon, 10 Jan 2022 04:24:12 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , John Hubbard , Christoph Hellwig , William Kucharski , linux-kernel@vger.kernel.org, Jason Gunthorpe Subject: [PATCH v2 03/28] gup: Change the calling convention for compound_range_next() Date: Mon, 10 Jan 2022 04:23:41 +0000 Message-Id: <20220110042406.499429-4-willy@infradead.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220110042406.499429-1-willy@infradead.org> References: <20220110042406.499429-1-willy@infradead.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Return the head page instead of storing it to a passed parameter. Pass the start page directly instead of passing a pointer to it. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Jason Gunthorpe Reviewed-by: John Hubbard Reviewed-by: William Kucharski --- mm/gup.c | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index 86f8d843de72..3c93d2fdf4da 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -229,21 +229,20 @@ void unpin_user_page(struct page *page) } EXPORT_SYMBOL(unpin_user_page); =20 -static inline void compound_range_next(unsigned long i, unsigned long npag= es, - struct page **list, struct page **head, - unsigned int *ntails) +static inline struct page *compound_range_next(unsigned long i, + unsigned long npages, struct page *start, unsigned int *ntails) { struct page *next, *page; unsigned int nr =3D 1; =20 - next =3D *list + i; + next =3D start + i; page =3D compound_head(next); if (PageCompound(page) && compound_order(page) >=3D 1) nr =3D min_t(unsigned int, page + compound_nr(page) - next, npages - i); =20 - *head =3D page; *ntails =3D nr; + return page; } =20 static inline void compound_next(unsigned long i, unsigned long npages, @@ -355,7 +354,7 @@ void unpin_user_page_range_dirty_lock(struct page *page= , unsigned long npages, unsigned int ntails; =20 for (index =3D 0; index < npages; index +=3D ntails) { - compound_range_next(index, npages, &page, &head, &ntails); + head =3D compound_range_next(index, npages, page, &ntails); if (make_dirty && !PageDirty(head)) set_page_dirty_lock(head); put_compound_head(head, ntails, FOLL_PIN); --=20 2.33.0 From nobody Tue Jun 30 20:11:01 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2D4E2C433EF for ; Mon, 10 Jan 2022 04:26:00 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238659AbiAJEZ4 (ORCPT ); Sun, 9 Jan 2022 23:25:56 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36602 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238533AbiAJEYd (ORCPT ); Sun, 9 Jan 2022 23:24:33 -0500 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1F803C061751 for ; Sun, 9 Jan 2022 20:24:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=zipY2Xef+bdp0dO6EMAI54lIjbt9nG2TQ8Ia2LQ9a/Q=; b=ns5OGy9Zz2OyKInKaLEalu2rr6 qg8S/j7UAKJxQzhpBFlCFvfgDQm/xaI4heaNVADhMk5hbH25uKPWE70CgIHr+u8FtdkeRgHDzldrm btpoe3/2uFb7+edVFD6tJjWaEj/OMkHQ22Xu/Bu/de0utTgIEQjHOvtK6+1wR6JmAwZww3Of1yb5v /MGjs6Jk2ANLvphPBpkFDvrreMxs5n11Hd9CsxU7X+KEfc9SL4iyLC8X53kQFaY785bAEQM6ng0Lu CpzNuR/qQh1XJpRTKtfDtTncReANru/+fgpdvIXzZc2dpTNsEN9TOtbOdOv5WHQLXxHehsufxIYfU BkVJoCOg==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1n6mE8-0025wY-68; Mon, 10 Jan 2022 04:24:12 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , John Hubbard , Christoph Hellwig , William Kucharski , linux-kernel@vger.kernel.org, Jason Gunthorpe Subject: [PATCH v2 04/28] gup: Optimise compound_range_next() Date: Mon, 10 Jan 2022 04:23:42 +0000 Message-Id: <20220110042406.499429-5-willy@infradead.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220110042406.499429-1-willy@infradead.org> References: <20220110042406.499429-1-willy@infradead.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" By definition, a compound page has an order >=3D 1, so the second half of the test was redundant. Also, this cannot be a tail page since it's the result of calling compound_head(), so use PageHead() instead of PageCompound(). Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Christoph Hellwig Reviewed-by: Jason Gunthorpe Reviewed-by: John Hubbard Reviewed-by: William Kucharski --- mm/gup.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/gup.c b/mm/gup.c index 3c93d2fdf4da..6eedca605b3d 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -237,7 +237,7 @@ static inline struct page *compound_range_next(unsigned= long i, =20 next =3D start + i; page =3D compound_head(next); - if (PageCompound(page) && compound_order(page) >=3D 1) + if (PageHead(page)) nr =3D min_t(unsigned int, page + compound_nr(page) - next, npages - i); =20 --=20 2.33.0 From nobody Tue Jun 30 20:11:01 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E0B24C433EF for ; Mon, 10 Jan 2022 04:25:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238644AbiAJEZx (ORCPT ); Sun, 9 Jan 2022 23:25:53 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36582 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238519AbiAJEY3 (ORCPT ); Sun, 9 Jan 2022 23:24:29 -0500 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6FE17C06173F for ; Sun, 9 Jan 2022 20:24:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=Z6FL4QqZXpHUsas/XSj8HAlMM6U3EcqhJKpaCEuMmN8=; b=gxWZW6abe4i+kvz3u9LtWrvatH BoiG6HjaV3GJHxMRnfZSa7H9MJrW92RwHaeh3xjnfLoiR0eUxaQ7XdWjA/BL6eJb0/QcWDZvVS8Rt DCaSPmPce+iympJJ2YASM5qmcfx5BuQSc65ohl4fkLmb+tdj5xAwUXehLGGdLSl619nzH1L/FHKl+ UDpwP47wln4ui7FvBWkA7D+nkmrPQjhcA/QvbT0pxWGDMVnveDBAJfolGSS/6tgIsLbRqfcL/616V gIejTrUhUfqYl3v3R+N0T6w0T8MRjR91OhMBPs1fbjvyix4twfVWIX6h6Z46/JLPO1L/n2goXg0KT xg9aX3Qw==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1n6mE8-0025wa-8a; Mon, 10 Jan 2022 04:24:12 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , John Hubbard , Christoph Hellwig , William Kucharski , linux-kernel@vger.kernel.org, Jason Gunthorpe Subject: [PATCH v2 05/28] gup: Change the calling convention for compound_next() Date: Mon, 10 Jan 2022 04:23:43 +0000 Message-Id: <20220110042406.499429-6-willy@infradead.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220110042406.499429-1-willy@infradead.org> References: <20220110042406.499429-1-willy@infradead.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Return the head page instead of storing it to a passed parameter. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Christoph Hellwig Reviewed-by: Jason Gunthorpe Reviewed-by: John Hubbard Reviewed-by: William Kucharski --- mm/gup.c | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index 6eedca605b3d..8a0ea220ced1 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -245,9 +245,8 @@ static inline struct page *compound_range_next(unsigned= long i, return page; } =20 -static inline void compound_next(unsigned long i, unsigned long npages, - struct page **list, struct page **head, - unsigned int *ntails) +static inline struct page *compound_next(unsigned long i, + unsigned long npages, struct page **list, unsigned int *ntails) { struct page *page; unsigned int nr; @@ -258,8 +257,8 @@ static inline void compound_next(unsigned long i, unsig= ned long npages, break; } =20 - *head =3D page; *ntails =3D nr - i; + return page; } =20 /** @@ -297,7 +296,7 @@ void unpin_user_pages_dirty_lock(struct page **pages, u= nsigned long npages, } =20 for (index =3D 0; index < npages; index +=3D ntails) { - compound_next(index, npages, pages, &head, &ntails); + head =3D compound_next(index, npages, pages, &ntails); /* * Checking PageDirty at this point may race with * clear_page_dirty_for_io(), but that's OK. Two key @@ -386,7 +385,7 @@ void unpin_user_pages(struct page **pages, unsigned lon= g npages) return; =20 for (index =3D 0; index < npages; index +=3D ntails) { - compound_next(index, npages, pages, &head, &ntails); + head =3D compound_next(index, npages, pages, &ntails); put_compound_head(head, ntails, FOLL_PIN); } } --=20 2.33.0 From nobody Tue Jun 30 20:11:01 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D8862C433F5 for ; Mon, 10 Jan 2022 04:26:50 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238498AbiAJE0t (ORCPT ); Sun, 9 Jan 2022 23:26:49 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36656 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238551AbiAJEYr (ORCPT ); Sun, 9 Jan 2022 23:24:47 -0500 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D3654C061201 for ; Sun, 9 Jan 2022 20:24:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=/mKomX3uXiZG7krGrOYnz6hJemX5OW1n+xjyCfpBJys=; b=a+pmaVeLJi8SRHS5L7Dmup8EfI 2Nsn3zM4iQfxtuZc0F+2ox0aylv0ROJxj6j+/0/YRSBk8bdqF/Js/jkheepU4xFCwlCqSafOo1kBC CmnDDcwxdecjjSCy5j4PCjipzr8m+UYAtqmsXVrU7MeC8Y2k5a8hOGFfDTSW0y1QshGRHnLBAynJ3 R/EcAV18PSYHlf7fr3+xRy51kcSev0z3bPVPQlqluI3IlARSIoqqlbaXvePozuIKaCyFsuN39gR1y 8t672VdTzDDI9o7KCcm08Si2loC7UJ4T3XmiHaept+7O6n4Dwkb8SCamVEAvgMjnYtMrbrPIiJdar lE1BtiUA==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1n6mE8-0025wc-B3; Mon, 10 Jan 2022 04:24:12 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , John Hubbard , Christoph Hellwig , William Kucharski , linux-kernel@vger.kernel.org, Jason Gunthorpe Subject: [PATCH v2 06/28] gup: Fix some contiguous memmap assumptions Date: Mon, 10 Jan 2022 04:23:44 +0000 Message-Id: <20220110042406.499429-7-willy@infradead.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220110042406.499429-1-willy@infradead.org> References: <20220110042406.499429-1-willy@infradead.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Several functions in gup.c assume that a compound page has virtually contiguous page structs. This isn't true for SPARSEMEM configs unless SPARSEMEM_VMEMMAP is also set. Fix them by using nth_page() instead of plain pointer arithmetic. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Christoph Hellwig Reviewed-by: Jason Gunthorpe Reviewed-by: John Hubbard Reviewed-by: William Kucharski --- mm/gup.c | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index 8a0ea220ced1..9c0a702a4e03 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -235,7 +235,7 @@ static inline struct page *compound_range_next(unsigned= long i, struct page *next, *page; unsigned int nr =3D 1; =20 - next =3D start + i; + next =3D nth_page(start, i); page =3D compound_head(next); if (PageHead(page)) nr =3D min_t(unsigned int, @@ -2430,8 +2430,8 @@ static int record_subpages(struct page *page, unsigne= d long addr, { int nr; =20 - for (nr =3D 0; addr !=3D end; addr +=3D PAGE_SIZE) - pages[nr++] =3D page++; + for (nr =3D 0; addr !=3D end; nr++, addr +=3D PAGE_SIZE) + pages[nr] =3D nth_page(page, nr); =20 return nr; } @@ -2466,7 +2466,7 @@ static int gup_hugepte(pte_t *ptep, unsigned long sz,= unsigned long addr, VM_BUG_ON(!pfn_valid(pte_pfn(pte))); =20 head =3D pte_page(pte); - page =3D head + ((addr & (sz-1)) >> PAGE_SHIFT); + page =3D nth_page(head, (addr & (sz-1)) >> PAGE_SHIFT); refs =3D record_subpages(page, addr, end, pages + *nr); =20 head =3D try_grab_compound_head(head, refs, flags); @@ -2526,7 +2526,7 @@ static int gup_huge_pmd(pmd_t orig, pmd_t *pmdp, unsi= gned long addr, pages, nr); } =20 - page =3D pmd_page(orig) + ((addr & ~PMD_MASK) >> PAGE_SHIFT); + page =3D nth_page(pmd_page(orig), (addr & ~PMD_MASK) >> PAGE_SHIFT); refs =3D record_subpages(page, addr, end, pages + *nr); =20 head =3D try_grab_compound_head(pmd_page(orig), refs, flags); @@ -2560,7 +2560,7 @@ static int gup_huge_pud(pud_t orig, pud_t *pudp, unsi= gned long addr, pages, nr); } =20 - page =3D pud_page(orig) + ((addr & ~PUD_MASK) >> PAGE_SHIFT); + page =3D nth_page(pud_page(orig), (addr & ~PUD_MASK) >> PAGE_SHIFT); refs =3D record_subpages(page, addr, end, pages + *nr); =20 head =3D try_grab_compound_head(pud_page(orig), refs, flags); @@ -2589,7 +2589,7 @@ static int gup_huge_pgd(pgd_t orig, pgd_t *pgdp, unsi= gned long addr, =20 BUILD_BUG_ON(pgd_devmap(orig)); =20 - page =3D pgd_page(orig) + ((addr & ~PGDIR_MASK) >> PAGE_SHIFT); + page =3D nth_page(pgd_page(orig), (addr & ~PGDIR_MASK) >> PAGE_SHIFT); refs =3D record_subpages(page, addr, end, pages + *nr); =20 head =3D try_grab_compound_head(pgd_page(orig), refs, flags); --=20 2.33.0 From nobody Tue Jun 30 20:11:01 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 97AB1C433F5 for ; Mon, 10 Jan 2022 04:26:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233420AbiAJE0m (ORCPT ); Sun, 9 Jan 2022 23:26:42 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36668 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238605AbiAJEYn (ORCPT ); Sun, 9 Jan 2022 23:24:43 -0500 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 13B3FC06175B for ; Sun, 9 Jan 2022 20:24:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=/DPJvh8nZY8O3CzKnWIX/byaa2jenc4w/Fleth+y20E=; b=gHzU4niHDL73CDvZ4y9ZrGch48 bMd5a3+9Z3TkUFgqpFk1IFK3Yu6nb+K2xMlltYCDn/3nNwCt58gKoG/cLHjhUoUH0s6MH/PH5GsOt 8xM+W9e6+2bFcXR9YEJMzKaqI5d6E2wpTwXYlSlGgf/PYNXXGRUYIxs9iLRgwoR3Jd5QhKxrkvT4B n7YcK3RyLBwZNGitu7ZYNkzKAeCx6UNIvZ5FikJ+1t9s6FxkfNsmyD6tV2wkKOyW7UaFwyLWhDK3V KyVFyPhmfKXNObz8PJkzqer7lRnp7CJiUO6sO4BFlN869hlAxZAwSfE4NJHkcKdjm6ePwcZ0UAebu s/Fm3+JQ==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1n6mE8-0025we-Dp; Mon, 10 Jan 2022 04:24:12 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , John Hubbard , Christoph Hellwig , William Kucharski , linux-kernel@vger.kernel.org, Jason Gunthorpe Subject: [PATCH v2 07/28] gup: Remove an assumption of a contiguous memmap Date: Mon, 10 Jan 2022 04:23:45 +0000 Message-Id: <20220110042406.499429-8-willy@infradead.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220110042406.499429-1-willy@infradead.org> References: <20220110042406.499429-1-willy@infradead.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" This assumption needs the inverse of nth_page(), which I've temporarily named page_nth() until someone comes up with a better name. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Christoph Hellwig Reviewed-by: Jason Gunthorpe Reviewed-by: John Hubbard Reviewed-by: William Kucharski --- include/linux/mm.h | 2 ++ mm/gup.c | 4 ++-- 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index d8b7d7ed14dd..f2f3400665a4 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -216,8 +216,10 @@ int overcommit_policy_handler(struct ctl_table *, int,= void *, size_t *, =20 #if defined(CONFIG_SPARSEMEM) && !defined(CONFIG_SPARSEMEM_VMEMMAP) #define nth_page(page,n) pfn_to_page(page_to_pfn((page)) + (n)) +#define page_nth(head, tail) (page_to_pfn(tail) - page_to_pfn(head)) #else #define nth_page(page,n) ((page) + (n)) +#define page_nth(head, tail) ((tail) - (head)) #endif =20 /* to align the pointer to the (next) page boundary */ diff --git a/mm/gup.c b/mm/gup.c index 9c0a702a4e03..afb638a30e44 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -238,8 +238,8 @@ static inline struct page *compound_range_next(unsigned= long i, next =3D nth_page(start, i); page =3D compound_head(next); if (PageHead(page)) - nr =3D min_t(unsigned int, - page + compound_nr(page) - next, npages - i); + nr =3D min_t(unsigned int, npages - i, + compound_nr(page) - page_nth(page, next)); =20 *ntails =3D nr; return page; --=20 2.33.0 From nobody Tue Jun 30 20:11:01 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 93F80C433F5 for ; Mon, 10 Jan 2022 04:27:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239014AbiAJE1d (ORCPT ); Sun, 9 Jan 2022 23:27:33 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36682 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238718AbiAJEZF (ORCPT ); Sun, 9 Jan 2022 23:25:05 -0500 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7F573C06118C for ; Sun, 9 Jan 2022 20:24:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=sKOYhN/P0cXC4qhPysyjZBfoAmLRLRFdOpaxGNZHEz8=; b=PJspN4WZf/k+qjE/mGfGFFrgkY cA9Kf5HN8fC0T5MXeLaNiI+h4JFXcp/9uKj/3uMyuTO3+G9NY0mZ3v0iR+rS7St765/En7TFFYyaK afHgj6DavPUc5K/Gsvjr8ejQPe2KTJly2O5vES3UKprH4ErwcIuKqvQP9L6OXTEv598D/7XyE9ljw 8hMsHnVRf1L1j0kB+nF0cImF8Sw9s2rRq0e6wHGDOJ1gVRMGSTi/oKvvNaCPc+7DovYWH/cF8mhFs r0XVqQPnvjqP4glZTdD9RFs9qMq2qgOLgKuwdzz8EG0OLCYNN7CpZuOPMKe/A7VZvKgLI4LWkuluJ 33unxJwQ==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1n6mE8-0025wg-GU; Mon, 10 Jan 2022 04:24:12 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , John Hubbard , Christoph Hellwig , William Kucharski , linux-kernel@vger.kernel.org, Jason Gunthorpe Subject: [PATCH v2 08/28] gup: Handle page split race more efficiently Date: Mon, 10 Jan 2022 04:23:46 +0000 Message-Id: <20220110042406.499429-9-willy@infradead.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220110042406.499429-1-willy@infradead.org> References: <20220110042406.499429-1-willy@infradead.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" If we hit the page split race, the current code returns NULL which will presumably trigger a retry under the mmap_lock. This isn't necessary; we can just retry the compound_head() lookup. This is a very minor optimisation of an unlikely path, but conceptually it matches (eg) the page cache RCU-protected lookup. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Christoph Hellwig Reviewed-by: Jason Gunthorpe Reviewed-by: John Hubbard Reviewed-by: William Kucharski --- mm/gup.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index afb638a30e44..dbb1b54d0def 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -68,7 +68,10 @@ static void put_page_refs(struct page *page, int refs) */ static inline struct page *try_get_compound_head(struct page *page, int re= fs) { - struct page *head =3D compound_head(page); + struct page *head; + +retry: + head =3D compound_head(page); =20 if (WARN_ON_ONCE(page_ref_count(head) < 0)) return NULL; @@ -86,7 +89,7 @@ static inline struct page *try_get_compound_head(struct p= age *page, int refs) */ if (unlikely(compound_head(page) !=3D head)) { put_page_refs(head, refs); - return NULL; + goto retry; } =20 return head; --=20 2.33.0 From nobody Tue Jun 30 20:11:01 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1C3A4C433F5 for ; Mon, 10 Jan 2022 04:26:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234874AbiAJE0b (ORCPT ); Sun, 9 Jan 2022 23:26:31 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36632 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233329AbiAJEYi (ORCPT ); Sun, 9 Jan 2022 23:24:38 -0500 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4FF4EC061748 for ; Sun, 9 Jan 2022 20:24:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=cNkGRRqE4MZ2IOY+aRtZ6O11ENnM+VUoRZ1VxqjqE3o=; b=XOj3bE4s2SziUPl4NNs9v1GOQC HevnwI8lrxL7DmnEAPUYAWy34kzK5npISqem7T56xj61z6L1q/Ddv/LabQ3++ka6+lsIEVo41JWsX 8rtSy+opZf39sTkTo8cAEMnv/XkqN6anRr/PyfCxHDwlLFFWajLlzFxx8mA+fso+lCsl+OSsqnfJa 6MJvBhMubiCav7PUMH1D4cVum7gqyi9YjODfwdYR0krwMxJO4ODeZhPC8iY0HPWKXqS5m8d63HWV4 CFSi3NBDsz+n8zSkyDHuxQ1sJpNTOR1oMnGth7hL9Z1Ux+EPuZAqRCG7JaoW4hwP7AQx5bDnFoUVp urm0Tg0g==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1n6mE8-0025wi-IQ; Mon, 10 Jan 2022 04:24:12 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , John Hubbard , Christoph Hellwig , William Kucharski , linux-kernel@vger.kernel.org, Jason Gunthorpe Subject: [PATCH v2 09/28] gup: Turn hpage_pincount_add() into page_pincount_add() Date: Mon, 10 Jan 2022 04:23:47 +0000 Message-Id: <20220110042406.499429-10-willy@infradead.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220110042406.499429-1-willy@infradead.org> References: <20220110042406.499429-1-willy@infradead.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Simplify try_grab_compound_head() and remove an unnecessary VM_BUG_ON by handling pages both with and without a pincount field in page_pincount_add(). Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Jason Gunthorpe Reviewed-by: John Hubbard Reviewed-by: William Kucharski --- mm/gup.c | 33 +++++++++++++++------------------ 1 file changed, 15 insertions(+), 18 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index dbb1b54d0def..3ed9907f3c8d 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -29,12 +29,23 @@ struct follow_page_context { unsigned int page_mask; }; =20 -static void hpage_pincount_add(struct page *page, int refs) +/* + * When pinning a compound page of order > 1 (which is what + * hpage_pincount_available() checks for), use an exact count to track + * it, via page_pincount_add/_sub(). + * + * However, be sure to *also* increment the normal page refcount field + * at least once, so that the page really is pinned. That's why the + * refcount from the earlier try_get_compound_head() is left intact. + */ +static void page_pincount_add(struct page *page, int refs) { - VM_BUG_ON_PAGE(!hpage_pincount_available(page), page); VM_BUG_ON_PAGE(page !=3D compound_head(page), page); =20 - atomic_add(refs, compound_pincount_ptr(page)); + if (hpage_pincount_available(page)) + atomic_add(refs, compound_pincount_ptr(page)); + else + page_ref_add(page, refs * (GUP_PIN_COUNTING_BIAS - 1)); } =20 static void hpage_pincount_sub(struct page *page, int refs) @@ -150,21 +161,7 @@ struct page *try_grab_compound_head(struct page *page, if (!page) return NULL; =20 - /* - * When pinning a compound page of order > 1 (which is what - * hpage_pincount_available() checks for), use an exact count to - * track it, via hpage_pincount_add/_sub(). - * - * However, be sure to *also* increment the normal page refcount - * field at least once, so that the page really is pinned. - * That's why the refcount from the earlier - * try_get_compound_head() is left intact. - */ - if (hpage_pincount_available(page)) - hpage_pincount_add(page, refs); - else - page_ref_add(page, refs * (GUP_PIN_COUNTING_BIAS - 1)); - + page_pincount_add(page, refs); mod_node_page_state(page_pgdat(page), NR_FOLL_PIN_ACQUIRED, refs); =20 --=20 2.33.0 From nobody Tue Jun 30 20:11:01 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C7196C433EF for ; Mon, 10 Jan 2022 04:27:07 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238609AbiAJE1D (ORCPT ); Sun, 9 Jan 2022 23:27:03 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36642 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238708AbiAJEZE (ORCPT ); Sun, 9 Jan 2022 23:25:04 -0500 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7B485C061757 for ; Sun, 9 Jan 2022 20:24:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=PMXingW4eQcetsP+Ankg7iWuo3nyPl2vlV/6qXVRFks=; b=n18ODInjFZJSyEpHvkEsh9YJTK +uPkafon+0J6706NZc+vLmkEIuqkh8Q8XhvMVc7lJ/Jo7aEh42HgZ+/eFQ9PytDBKBT4DdS9JBr0x 4bcO22N4r7Ibi0uEIDXR3WjTKCZT7Ut8zxXQHtquJq69Z2sQntuaHku+2aGnVbbjGw9+ulayIB2Q9 5CHQiTK+6y7fb7hnNmIhK8etwk5VKSFNN+dqOAKSvGIJOMiiUV2y/goH9vf7HGVi5vQ+iML8Dpg+5 Sddxp9xM5Df1U/mzGxDPb2Bt4oZE8U+Gw++ra1TUSorh+6nE+YgMyplf+BGDZxYlOsSnCpSZlOR9f qu4+B7jw==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1n6mE8-0025wk-KY; Mon, 10 Jan 2022 04:24:12 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , John Hubbard , Christoph Hellwig , William Kucharski , linux-kernel@vger.kernel.org, Jason Gunthorpe Subject: [PATCH v2 10/28] gup: Turn hpage_pincount_sub() into page_pincount_sub() Date: Mon, 10 Jan 2022 04:23:48 +0000 Message-Id: <20220110042406.499429-11-willy@infradead.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220110042406.499429-1-willy@infradead.org> References: <20220110042406.499429-1-willy@infradead.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Remove an unnecessary VM_BUG_ON by handling pages both with and without a pincount field in page_pincount_sub(). Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Jason Gunthorpe Reviewed-by: John Hubbard Reviewed-by: William Kucharski --- mm/gup.c | 15 +++++++-------- 1 file changed, 7 insertions(+), 8 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index 3ed9907f3c8d..aed48de3912e 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -48,12 +48,15 @@ static void page_pincount_add(struct page *page, int re= fs) page_ref_add(page, refs * (GUP_PIN_COUNTING_BIAS - 1)); } =20 -static void hpage_pincount_sub(struct page *page, int refs) +static int page_pincount_sub(struct page *page, int refs) { - VM_BUG_ON_PAGE(!hpage_pincount_available(page), page); VM_BUG_ON_PAGE(page !=3D compound_head(page), page); =20 - atomic_sub(refs, compound_pincount_ptr(page)); + if (hpage_pincount_available(page)) + atomic_sub(refs, compound_pincount_ptr(page)); + else + refs *=3D GUP_PIN_COUNTING_BIAS; + return refs; } =20 /* Equivalent to calling put_page() @refs times. */ @@ -177,11 +180,7 @@ static void put_compound_head(struct page *page, int r= efs, unsigned int flags) if (flags & FOLL_PIN) { mod_node_page_state(page_pgdat(page), NR_FOLL_PIN_RELEASED, refs); - - if (hpage_pincount_available(page)) - hpage_pincount_sub(page, refs); - else - refs *=3D GUP_PIN_COUNTING_BIAS; + refs =3D page_pincount_sub(page, refs); } =20 put_page_refs(page, refs); --=20 2.33.0 From nobody Tue Jun 30 20:11:01 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5FB86C4332F for ; Mon, 10 Jan 2022 04:24:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238561AbiAJEYf (ORCPT ); Sun, 9 Jan 2022 23:24:35 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36542 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238483AbiAJEYZ (ORCPT ); Sun, 9 Jan 2022 23:24:25 -0500 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BB8FDC061756 for ; Sun, 9 Jan 2022 20:24:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=vDqLbEnHUcTnSYJT/+8mqVyuYK0UxluPC/+3yXrlqME=; b=S68kUEDi9O306wA5NqLmOLyEqz ofgAfH/kLeUxCOEyNQjHk/dUz9c6MqDDSgGn9Jcgokac4kcE7pwJPLIAgEiMI7AdxS7mhPM7fJrwh iGls2n+779gV1uiExYzSEQ04Cnqu1bU14xGITfWI36NS7yD9ubEA9vztiAdURco2svLcOnMYinUtm BWU8R+MpN978d0ZJ+cldutqVeyp4Zk/aygpZWKzalb95VVbAaCEPrD6iO04KDtjvVPkPY/CKJzpCl WcFWM/rMEbNJa70OZpu3H4Pza5h2bShSsopOEgB73Jsy+jtg3nre+A5iiMqQUN1cMjZOhZEHiGw12 2WJMM5Eg==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1n6mE8-0025wm-Mz; Mon, 10 Jan 2022 04:24:12 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , John Hubbard , Christoph Hellwig , William Kucharski , linux-kernel@vger.kernel.org, Jason Gunthorpe Subject: [PATCH v2 11/28] mm: Make compound_pincount always available Date: Mon, 10 Jan 2022 04:23:49 +0000 Message-Id: <20220110042406.499429-12-willy@infradead.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220110042406.499429-1-willy@infradead.org> References: <20220110042406.499429-1-willy@infradead.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Move compound_pincount from the third page to the second page, which means it's available for all compound pages. That lets us delete hpage_pincount_available(). On 32-bit systems, there isn't enough space for both compound_pincount and compound_nr in the second page (it would collide with page->private, which is in use for pages in the swap cache), so revert the optimisation of storing both compound_order and compound_nr on 32-bit systems. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Christoph Hellwig Reviewed-by: Jason Gunthorpe Reviewed-by: John Hubbard Reviewed-by: William Kucharski --- Documentation/core-api/pin_user_pages.rst | 18 +++++++++--------- include/linux/mm.h | 21 ++++++++------------- include/linux/mm_types.h | 7 +++++-- mm/debug.c | 14 ++++---------- mm/gup.c | 18 ++++++++---------- mm/page_alloc.c | 3 +-- mm/rmap.c | 6 ++---- 7 files changed, 37 insertions(+), 50 deletions(-) diff --git a/Documentation/core-api/pin_user_pages.rst b/Documentation/core= -api/pin_user_pages.rst index fcf605be43d0..b18416f4500f 100644 --- a/Documentation/core-api/pin_user_pages.rst +++ b/Documentation/core-api/pin_user_pages.rst @@ -55,18 +55,18 @@ flags the caller provides. The caller is required to pa= ss in a non-null struct pages* array, and the function then pins pages by incrementing each by a s= pecial value: GUP_PIN_COUNTING_BIAS. =20 -For huge pages (and in fact, any compound page of more than 2 pages), the -GUP_PIN_COUNTING_BIAS scheme is not used. Instead, an exact form of pin co= unting -is achieved, by using the 3rd struct page in the compound page. A new stru= ct -page field, hpage_pinned_refcount, has been added in order to support this. +For compound pages, the GUP_PIN_COUNTING_BIAS scheme is not used. Instead, +an exact form of pin counting is achieved, by using the 2nd struct page +in the compound page. A new struct page field, compound_pincount, has +been added in order to support this. =20 This approach for compound pages avoids the counting upper limit problems = that are discussed below. Those limitations would have been aggravated severely= by huge pages, because each tail page adds a refcount to the head page. And in -fact, testing revealed that, without a separate hpage_pinned_refcount fiel= d, +fact, testing revealed that, without a separate compound_pincount field, page overflows were seen in some huge page stress tests. =20 -This also means that huge pages and compound pages (of order > 1) do not s= uffer +This also means that huge pages and compound pages do not suffer from the false positives problem that is mentioned below.:: =20 Function @@ -264,9 +264,9 @@ place.) Other diagnostics =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =20 -dump_page() has been enhanced slightly, to handle these new counting field= s, and -to better report on compound pages in general. Specifically, for compound = pages -with order > 1, the exact (hpage_pinned_refcount) pincount is reported. +dump_page() has been enhanced slightly, to handle these new counting +fields, and to better report on compound pages in general. Specifically, +for compound pages, the exact (compound_pincount) pincount is reported. =20 References =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D diff --git a/include/linux/mm.h b/include/linux/mm.h index f2f3400665a4..598be27d4d2e 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -929,17 +929,6 @@ static inline void destroy_compound_page(struct page *= page) compound_page_dtors[page[1].compound_dtor](page); } =20 -static inline bool hpage_pincount_available(struct page *page) -{ - /* - * Can the page->hpage_pinned_refcount field be used? That field is in - * the 3rd page of the compound page, so the smallest (2-page) compound - * pages cannot support it. - */ - page =3D compound_head(page); - return PageCompound(page) && compound_order(page) > 1; -} - static inline int head_compound_pincount(struct page *head) { return atomic_read(compound_pincount_ptr(head)); @@ -947,7 +936,7 @@ static inline int head_compound_pincount(struct page *h= ead) =20 static inline int compound_pincount(struct page *page) { - VM_BUG_ON_PAGE(!hpage_pincount_available(page), page); + VM_BUG_ON_PAGE(!PageCompound(page), page); page =3D compound_head(page); return head_compound_pincount(page); } @@ -955,7 +944,9 @@ static inline int compound_pincount(struct page *page) static inline void set_compound_order(struct page *page, unsigned int orde= r) { page[1].compound_order =3D order; +#ifdef CONFIG_64BIT page[1].compound_nr =3D 1U << order; +#endif } =20 /* Returns the number of pages in this potentially compound page. */ @@ -963,7 +954,11 @@ static inline unsigned long compound_nr(struct page *p= age) { if (!PageHead(page)) return 1; +#ifdef CONFIG_64BIT return page[1].compound_nr; +#else + return 1UL << compound_order(page); +#endif } =20 /* Returns the number of bytes in this potentially compound page. */ @@ -1325,7 +1320,7 @@ void unpin_user_pages(struct page **pages, unsigned l= ong npages); */ static inline bool page_maybe_dma_pinned(struct page *page) { - if (hpage_pincount_available(page)) + if (PageCompound(page)) return compound_pincount(page) > 0; =20 /* diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index c3a6e6209600..60e4595eaf63 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -150,11 +150,14 @@ struct page { unsigned char compound_dtor; unsigned char compound_order; atomic_t compound_mapcount; + atomic_t compound_pincount; +#ifdef CONFIG_64BIT unsigned int compound_nr; /* 1 << compound_order */ +#endif }; struct { /* Second tail page of compound page */ unsigned long _compound_pad_1; /* compound_head */ - atomic_t hpage_pinned_refcount; + unsigned long _compound_pad_2; /* For both global and memcg */ struct list_head deferred_list; }; @@ -311,7 +314,7 @@ static inline atomic_t *compound_mapcount_ptr(struct pa= ge *page) =20 static inline atomic_t *compound_pincount_ptr(struct page *page) { - return &page[2].hpage_pinned_refcount; + return &page[1].compound_pincount; } =20 /* diff --git a/mm/debug.c b/mm/debug.c index a05a39ff8fe4..7925fac2bd8e 100644 --- a/mm/debug.c +++ b/mm/debug.c @@ -92,16 +92,10 @@ static void __dump_page(struct page *page) page, page_ref_count(head), mapcount, mapping, page_to_pgoff(page), page_to_pfn(page)); if (compound) { - if (hpage_pincount_available(page)) { - pr_warn("head:%p order:%u compound_mapcount:%d compound_pincount:%d\n", - head, compound_order(head), - head_compound_mapcount(head), - head_compound_pincount(head)); - } else { - pr_warn("head:%p order:%u compound_mapcount:%d\n", - head, compound_order(head), - head_compound_mapcount(head)); - } + pr_warn("head:%p order:%u compound_mapcount:%d compound_pincount:%d\n", + head, compound_order(head), + head_compound_mapcount(head), + head_compound_pincount(head)); } =20 #ifdef CONFIG_MEMCG diff --git a/mm/gup.c b/mm/gup.c index aed48de3912e..1282d29357b7 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -30,9 +30,8 @@ struct follow_page_context { }; =20 /* - * When pinning a compound page of order > 1 (which is what - * hpage_pincount_available() checks for), use an exact count to track - * it, via page_pincount_add/_sub(). + * When pinning a compound page, use an exact count to track it, via + * page_pincount_add/_sub(). * * However, be sure to *also* increment the normal page refcount field * at least once, so that the page really is pinned. That's why the @@ -42,7 +41,7 @@ static void page_pincount_add(struct page *page, int refs) { VM_BUG_ON_PAGE(page !=3D compound_head(page), page); =20 - if (hpage_pincount_available(page)) + if (PageHead(page)) atomic_add(refs, compound_pincount_ptr(page)); else page_ref_add(page, refs * (GUP_PIN_COUNTING_BIAS - 1)); @@ -52,7 +51,7 @@ static int page_pincount_sub(struct page *page, int refs) { VM_BUG_ON_PAGE(page !=3D compound_head(page), page); =20 - if (hpage_pincount_available(page)) + if (PageHead(page)) atomic_sub(refs, compound_pincount_ptr(page)); else refs *=3D GUP_PIN_COUNTING_BIAS; @@ -129,12 +128,11 @@ static inline struct page *try_get_compound_head(stru= ct page *page, int refs) * * FOLL_GET: page's refcount will be incremented by @refs. * - * FOLL_PIN on compound pages that are > two pages long: page's refcoun= t will - * be incremented by @refs, and page[2].hpage_pinned_refcount will be - * incremented by @refs * GUP_PIN_COUNTING_BIAS. + * FOLL_PIN on compound pages: page's refcount will be incremented by + * @refs, and page[1].compound_pincount will be incremented by @refs. * - * FOLL_PIN on normal pages, or compound pages that are two pages long: - * page's refcount will be incremented by @refs * GUP_PIN_COUNTING_BIAS. + * FOLL_PIN on normal pages: page's refcount will be incremented by + * @refs * GUP_PIN_COUNTING_BIAS. * * Return: head page (with refcount appropriately incremented) for success= , or * NULL upon failure. If neither FOLL_GET nor FOLL_PIN was set, that's diff --git a/mm/page_alloc.c b/mm/page_alloc.c index c5952749ad40..6b030c0cb207 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -741,8 +741,7 @@ void prep_compound_page(struct page *page, unsigned int= order) set_compound_page_dtor(page, COMPOUND_PAGE_DTOR); set_compound_order(page, order); atomic_set(compound_mapcount_ptr(page), -1); - if (hpage_pincount_available(page)) - atomic_set(compound_pincount_ptr(page), 0); + atomic_set(compound_pincount_ptr(page), 0); } =20 #ifdef CONFIG_DEBUG_PAGEALLOC diff --git a/mm/rmap.c b/mm/rmap.c index 163ac4e6bcee..a44a32db4803 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1187,8 +1187,7 @@ void page_add_new_anon_rmap(struct page *page, VM_BUG_ON_PAGE(!PageTransHuge(page), page); /* increment count (starts at -1) */ atomic_set(compound_mapcount_ptr(page), 0); - if (hpage_pincount_available(page)) - atomic_set(compound_pincount_ptr(page), 0); + atomic_set(compound_pincount_ptr(page), 0); =20 __mod_lruvec_page_state(page, NR_ANON_THPS, nr); } else { @@ -2410,8 +2409,7 @@ void hugepage_add_new_anon_rmap(struct page *page, { BUG_ON(address < vma->vm_start || address >=3D vma->vm_end); atomic_set(compound_mapcount_ptr(page), 0); - if (hpage_pincount_available(page)) - atomic_set(compound_pincount_ptr(page), 0); + atomic_set(compound_pincount_ptr(page), 0); =20 __page_set_anon_rmap(page, vma, address, 1); } --=20 2.33.0 From nobody Tue Jun 30 20:11:01 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E9CFBC433F5 for ; Mon, 10 Jan 2022 04:26:24 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238676AbiAJE0Y (ORCPT ); Sun, 9 Jan 2022 23:26:24 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36626 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238569AbiAJEYg (ORCPT ); Sun, 9 Jan 2022 23:24:36 -0500 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6A5D9C061748 for ; Sun, 9 Jan 2022 20:24:36 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=pr6dwM9p32xFGkENVfW8ccHIrI0fREnA+ZakTJ73Pp8=; b=diLvaBxsmnGNc0FH5T2NoqikDX Q4Ygnsy6APbDxF2yhgPg/bUDT/ZZl5VKG73fA3Wb/qUY1yr/6zw06e9ozJDWRpziaMw2g2IVFIu6U QnHwvyE3ecy9hBMdhQhrtNPVBoIQRVfVsI3wVBFTXIjA17sjB/F36yzkUY1r2uMTg8eCIyjpcgs9o HERYwayFGxgm8bF0MKu0UBGFNSiWSAGYzZjJQKryikGkg7Sgy50thISU/Y4S9hK8usWe9Zha3lydG neZjS1K3jtVAYFKarLfjPxhFpKfl6hIaB5iteXtDUv2QkKsgaiFuxpgj50rTbXDia0Jtfj3fRnA79 jjHXkc7g==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1n6mE8-0025wo-Pi; Mon, 10 Jan 2022 04:24:12 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , John Hubbard , Christoph Hellwig , William Kucharski , linux-kernel@vger.kernel.org, Jason Gunthorpe Subject: [PATCH v2 12/28] mm: Add folio_put_refs() Date: Mon, 10 Jan 2022 04:23:50 +0000 Message-Id: <20220110042406.499429-13-willy@infradead.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220110042406.499429-1-willy@infradead.org> References: <20220110042406.499429-1-willy@infradead.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" This is like folio_put(), but puts N references at once instead of just one. It's like put_page_refs(), but does one atomic operation instead of two, and is available to more than just gup.c. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Christoph Hellwig Reviewed-by: Jason Gunthorpe Reviewed-by: John Hubbard Reviewed-by: William Kucharski --- include/linux/mm.h | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index 598be27d4d2e..bf9624ca61c3 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1234,6 +1234,26 @@ static inline void folio_put(struct folio *folio) __put_page(&folio->page); } =20 +/** + * folio_put_refs - Reduce the reference count on a folio. + * @folio: The folio. + * @refs: The number of references to reduce. + * + * If the folio's reference count reaches zero, the memory will be + * released back to the page allocator and may be used by another + * allocation immediately. Do not access the memory or the struct folio + * after calling folio_put_refs() unless you can be sure that these weren't + * the last references. + * + * Context: May be called in process or interrupt context, but not in NMI + * context. May be called while holding a spinlock. + */ +static inline void folio_put_refs(struct folio *folio, int refs) +{ + if (folio_ref_sub_and_test(folio, refs)) + __put_page(&folio->page); +} + static inline void put_page(struct page *page) { struct folio *folio =3D page_folio(page); --=20 2.33.0 From nobody Tue Jun 30 20:11:01 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id AFA10C433EF for ; Mon, 10 Jan 2022 04:27:12 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238777AbiAJE1L (ORCPT ); Sun, 9 Jan 2022 23:27:11 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36762 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238709AbiAJEZE (ORCPT ); Sun, 9 Jan 2022 23:25:04 -0500 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9E198C061759 for ; Sun, 9 Jan 2022 20:24:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=w+88rodVOkiUqMds1CDDO2rnbx+ZzvbF6EDnhZLZjHk=; b=MHde1Qleuqe7Nk4xqACcAx7HEl D8kJGZeV3c227mQCZyDaXVLZZyyzsX283wSRbmVtVux8gfh65uIkUe9PKbmG7LiN8Av1rGrz0b64n wzKrvVfSE8faTN1k0nH9mOXqLhoC2As+LH2ynMSk2GovVwLBt8PhprrxJGMwurHMoQN0j2e5UdE3d matAxpEXJXnrEz0GvItCgqQ/YSIzOpNc1FmFeBXxfCgj1n7Krq20kEMmHbbFFHodTHPqMzuBEJkbu +GVmJzrwFRfYm5SfWSxrlFJy1tI2FcZk9l8FDXj+mYCMLNpfQV739wpVskuIZrwIK6yfdqEQ/AdN7 fNaNMXMw==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1n6mE8-0025wq-SO; Mon, 10 Jan 2022 04:24:12 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , John Hubbard , Christoph Hellwig , William Kucharski , linux-kernel@vger.kernel.org, Jason Gunthorpe Subject: [PATCH v2 13/28] mm: Add folio_pincount_ptr() Date: Mon, 10 Jan 2022 04:23:51 +0000 Message-Id: <20220110042406.499429-14-willy@infradead.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220110042406.499429-1-willy@infradead.org> References: <20220110042406.499429-1-willy@infradead.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" This is the folio equivalent of compound_pincount_ptr(). Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Christoph Hellwig Reviewed-by: Jason Gunthorpe Reviewed-by: John Hubbard Reviewed-by: William Kucharski --- include/linux/mm_types.h | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 60e4595eaf63..34c7114ea9e9 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -312,6 +312,12 @@ static inline atomic_t *compound_mapcount_ptr(struct p= age *page) return &page[1].compound_mapcount; } =20 +static inline atomic_t *folio_pincount_ptr(struct folio *folio) +{ + struct page *tail =3D &folio->page + 1; + return &tail->compound_pincount; +} + static inline atomic_t *compound_pincount_ptr(struct page *page) { return &page[1].compound_pincount; --=20 2.33.0 From nobody Tue Jun 30 20:11:01 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D9AF5C433EF for ; Mon, 10 Jan 2022 04:24:27 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238508AbiAJEY0 (ORCPT ); Sun, 9 Jan 2022 23:24:26 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36538 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233306AbiAJEYZ (ORCPT ); Sun, 9 Jan 2022 23:24:25 -0500 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B1FA7C061748 for ; Sun, 9 Jan 2022 20:24:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=8lkIQ5pzUHptX+PQh07i+5GbGN5xDvd48mKak705GKI=; b=Lxrx0VGaelZR0XiQE/gKbFnugO prDvQ//ra4faWoP9k9x+hSjkLVWobcO2IlV2gT8KfuDeRx6xvoGP2AljqzpW5E9vj7OhN4bgWIFP4 JB39/sj6YfIja85XpZXV2AKYX0aY7i7/uAjwftPYg48TULXlaI8K5hdPfxN1/2uYBeUKllI2lzo+1 z8em1rbCoqk4oZ0b7ZGaPpEfzO0OWqkEIUZEZQvILtyeMWKnKXaEwFcTa7Nsgx6NVtx95GFS4vVcF I9QuY8oq4dgp/RLADuZ6z+wEXco9L1LLv5QmFvvAHvmcCeNhqYVXG/8ATpb6AvVgO5jnhedDHobxG zXBSVyWw==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1n6mE8-0025ws-Uo; Mon, 10 Jan 2022 04:24:12 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , John Hubbard , Christoph Hellwig , William Kucharski , linux-kernel@vger.kernel.org, Jason Gunthorpe Subject: [PATCH v2 14/28] mm: Convert page_maybe_dma_pinned() to use a folio Date: Mon, 10 Jan 2022 04:23:52 +0000 Message-Id: <20220110042406.499429-15-willy@infradead.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220110042406.499429-1-willy@infradead.org> References: <20220110042406.499429-1-willy@infradead.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Replace three calls to compound_head() with one. This removes the last user of compound_pincount(), so remove that helper too. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Christoph Hellwig Reviewed-by: Jason Gunthorpe Reviewed-by: John Hubbard Reviewed-by: William Kucharski --- include/linux/mm.h | 21 ++++++++------------- 1 file changed, 8 insertions(+), 13 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index bf9624ca61c3..d3769897c8ac 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -934,13 +934,6 @@ static inline int head_compound_pincount(struct page *= head) return atomic_read(compound_pincount_ptr(head)); } =20 -static inline int compound_pincount(struct page *page) -{ - VM_BUG_ON_PAGE(!PageCompound(page), page); - page =3D compound_head(page); - return head_compound_pincount(page); -} - static inline void set_compound_order(struct page *page, unsigned int orde= r) { page[1].compound_order =3D order; @@ -1340,18 +1333,20 @@ void unpin_user_pages(struct page **pages, unsigned= long npages); */ static inline bool page_maybe_dma_pinned(struct page *page) { - if (PageCompound(page)) - return compound_pincount(page) > 0; + struct folio *folio =3D page_folio(page); + + if (folio_test_large(folio)) + return atomic_read(folio_pincount_ptr(folio)) > 0; =20 /* - * page_ref_count() is signed. If that refcount overflows, then - * page_ref_count() returns a negative value, and callers will avoid + * folio_ref_count() is signed. If that refcount overflows, then + * folio_ref_count() returns a negative value, and callers will avoid * further incrementing the refcount. * - * Here, for that overflow case, use the signed bit to count a little + * Here, for that overflow case, use the sign bit to count a little * bit higher via unsigned math, and thus still get an accurate result. */ - return ((unsigned int)page_ref_count(compound_head(page))) >=3D + return ((unsigned int)folio_ref_count(folio)) >=3D GUP_PIN_COUNTING_BIAS; } =20 --=20 2.33.0 From nobody Tue Jun 30 20:11:01 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 84241C433F5 for ; Mon, 10 Jan 2022 04:25:10 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238545AbiAJEZI (ORCPT ); Sun, 9 Jan 2022 23:25:08 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36554 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238496AbiAJEYZ (ORCPT ); Sun, 9 Jan 2022 23:24:25 -0500 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D7CF1C06175D for ; Sun, 9 Jan 2022 20:24:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=3gcp8jAPpJyVoKpFTPIlauHZARxlFWOwXAJGDSUF0G0=; b=NwL/Za0kcScCPVLkGX/SBoUNFR 9vYY3W8cWSZdA7ql6pD8iJmt1qcMyUxWBoJcRxY3gwGammTlVvGKriKJRq8UpA5n1Y79Qo9Lq6q8x LGqen7wG86nmtFW02pR3n+s21BeEuGwk/+ajGfy+ANry+I5u2Z7VNk9IPOA2SHTL5tth1MEwxUspU JiILC990MGkTfNLzFaqXq1tPiEa4koChT79iA+id1Uc0NEEn4at4gcYbfOvvnjHln/f8Kt4OTgnVQ 4nJBVLyWtswepa/DIlbCEmyLb9J79XlDjyn6Z1LibcPXahZE+6FmTHKAs2opXPyXk4oZYpbtjLD0D b1VNC9bQ==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1n6mE9-0025wu-10; Mon, 10 Jan 2022 04:24:13 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , John Hubbard , Christoph Hellwig , William Kucharski , linux-kernel@vger.kernel.org, Jason Gunthorpe Subject: [PATCH v2 15/28] gup: Add try_get_folio() and try_grab_folio() Date: Mon, 10 Jan 2022 04:23:53 +0000 Message-Id: <20220110042406.499429-16-willy@infradead.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220110042406.499429-1-willy@infradead.org> References: <20220110042406.499429-1-willy@infradead.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Convert try_get_compound_head() into try_get_folio() and convert try_grab_compound_head() into try_grab_folio(). Also convert hpage_pincount_add() to folio_pincount_add(). Add a temporary try_grab_compound_head() wrapper around try_grab_folio() to let us convert callers individually. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Jason Gunthorpe Reviewed-by: William Kucharski --- mm/gup.c | 104 +++++++++++++++++++++++++------------------------- mm/internal.h | 5 +++ 2 files changed, 56 insertions(+), 53 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index 1282d29357b7..9e581201d679 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -30,21 +30,19 @@ struct follow_page_context { }; =20 /* - * When pinning a compound page, use an exact count to track it, via - * page_pincount_add/_sub(). + * When pinning a large folio, use an exact count to track it. * - * However, be sure to *also* increment the normal page refcount field - * at least once, so that the page really is pinned. That's why the - * refcount from the earlier try_get_compound_head() is left intact. + * However, be sure to *also* increment the normal folio refcount + * field at least once, so that the folio really is pinned. + * That's why the refcount from the earlier + * try_get_folio() is left intact. */ -static void page_pincount_add(struct page *page, int refs) +static void folio_pincount_add(struct folio *folio, int refs) { - VM_BUG_ON_PAGE(page !=3D compound_head(page), page); - - if (PageHead(page)) - atomic_add(refs, compound_pincount_ptr(page)); + if (folio_test_large(folio)) + atomic_add(refs, folio_pincount_ptr(folio)); else - page_ref_add(page, refs * (GUP_PIN_COUNTING_BIAS - 1)); + folio_ref_add(folio, refs * (GUP_PIN_COUNTING_BIAS - 1)); } =20 static int page_pincount_sub(struct page *page, int refs) @@ -76,75 +74,70 @@ static void put_page_refs(struct page *page, int refs) } =20 /* - * Return the compound head page with ref appropriately incremented, + * Return the folio with ref appropriately incremented, * or NULL if that failed. */ -static inline struct page *try_get_compound_head(struct page *page, int re= fs) +static inline struct folio *try_get_folio(struct page *page, int refs) { - struct page *head; + struct folio *folio; =20 retry: - head =3D compound_head(page); - - if (WARN_ON_ONCE(page_ref_count(head) < 0)) + folio =3D page_folio(page); + if (WARN_ON_ONCE(folio_ref_count(folio) < 0)) return NULL; - if (unlikely(!page_cache_add_speculative(head, refs))) + if (unlikely(!folio_ref_try_add_rcu(folio, refs))) return NULL; =20 /* - * At this point we have a stable reference to the head page; but it - * could be that between the compound_head() lookup and the refcount - * increment, the compound page was split, in which case we'd end up - * holding a reference on a page that has nothing to do with the page + * At this point we have a stable reference to the folio; but it + * could be that between calling page_folio() and the refcount + * increment, the folio was split, in which case we'd end up + * holding a reference on a folio that has nothing to do with the page * we were given anymore. - * So now that the head page is stable, recheck that the pages still - * belong together. + * So now that the folio is stable, recheck that the page still + * belongs to this folio. */ - if (unlikely(compound_head(page) !=3D head)) { - put_page_refs(head, refs); + if (unlikely(page_folio(page) !=3D folio)) { + folio_put_refs(folio, refs); goto retry; } =20 - return head; + return folio; } =20 /** - * try_grab_compound_head() - attempt to elevate a page's refcount, by a - * flags-dependent amount. - * - * Even though the name includes "compound_head", this function is still - * appropriate for callers that have a non-compound @page to get. - * + * try_grab_folio() - Attempt to get or pin a folio. * @page: pointer to page to be grabbed - * @refs: the value to (effectively) add to the page's refcount + * @refs: the value to (effectively) add to the folio's refcount * @flags: gup flags: these are the FOLL_* flag values. * * "grab" names in this file mean, "look at flags to decide whether to use - * FOLL_PIN or FOLL_GET behavior, when incrementing the page's refcount. + * FOLL_PIN or FOLL_GET behavior, when incrementing the folio's refcount. * * Either FOLL_PIN or FOLL_GET (or neither) must be set, but not both at t= he * same time. (That's true throughout the get_user_pages*() and * pin_user_pages*() APIs.) Cases: * - * FOLL_GET: page's refcount will be incremented by @refs. + * FOLL_GET: folio's refcount will be incremented by @refs. * - * FOLL_PIN on compound pages: page's refcount will be incremented by - * @refs, and page[1].compound_pincount will be incremented by @refs. + * FOLL_PIN on large folios: folio's refcount will be incremented by + * @refs, and its compound_pincount will be incremented by @refs. * - * FOLL_PIN on normal pages: page's refcount will be incremented by + * FOLL_PIN on single-page folios: folio's refcount will be incremented= by * @refs * GUP_PIN_COUNTING_BIAS. * - * Return: head page (with refcount appropriately incremented) for success= , or - * NULL upon failure. If neither FOLL_GET nor FOLL_PIN was set, that's - * considered failure, and furthermore, a likely bug in the caller, so a w= arning - * is also emitted. + * Return: The folio containing @page (with refcount appropriately + * incremented) for success, or NULL upon failure. If neither FOLL_GET + * nor FOLL_PIN was set, that's considered failure, and furthermore, + * a likely bug in the caller, so a warning is also emitted. */ -struct page *try_grab_compound_head(struct page *page, - int refs, unsigned int flags) +struct folio *try_grab_folio(struct page *page, int refs, unsigned int fla= gs) { if (flags & FOLL_GET) - return try_get_compound_head(page, refs); + return try_get_folio(page, refs); else if (flags & FOLL_PIN) { + struct folio *folio; + /* * Can't do FOLL_LONGTERM + FOLL_PIN gup fast path if not in a * right zone, so fail and let the caller fall back to the slow @@ -158,21 +151,26 @@ struct page *try_grab_compound_head(struct page *page, * CAUTION: Don't use compound_head() on the page before this * point, the result won't be stable. */ - page =3D try_get_compound_head(page, refs); - if (!page) + folio =3D try_get_folio(page, refs); + if (!folio) return NULL; =20 - page_pincount_add(page, refs); - mod_node_page_state(page_pgdat(page), NR_FOLL_PIN_ACQUIRED, - refs); + folio_pincount_add(folio, refs); + node_stat_mod_folio(folio, NR_FOLL_PIN_ACQUIRED, refs); =20 - return page; + return folio; } =20 WARN_ON_ONCE(1); return NULL; } =20 +struct page *try_grab_compound_head(struct page *page, + int refs, unsigned int flags) +{ + return &try_grab_folio(page, refs, flags)->page; +} + static void put_compound_head(struct page *page, int refs, unsigned int fl= ags) { if (flags & FOLL_PIN) { @@ -196,7 +194,7 @@ static void put_compound_head(struct page *page, int re= fs, unsigned int flags) * @flags: gup flags: these are the FOLL_* flag values. * * Either FOLL_PIN or FOLL_GET (or neither) may be set, but not both at th= e same - * time. Cases: please see the try_grab_compound_head() documentation, with + * time. Cases: please see the try_grab_folio() documentation, with * "refs=3D1". * * Return: true for success, or if no action was required (if neither FOLL= _PIN diff --git a/mm/internal.h b/mm/internal.h index 26af8a5a5be3..9a72d1ecdab4 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -723,4 +723,9 @@ void vunmap_range_noflush(unsigned long start, unsigned= long end); int numa_migrate_prep(struct page *page, struct vm_area_struct *vma, unsigned long addr, int page_nid, int *flags); =20 +/* + * mm/gup.c + */ +struct folio *try_grab_folio(struct page *page, int refs, unsigned int fla= gs); + #endif /* __MM_INTERNAL_H */ --=20 2.33.0 From nobody Tue Jun 30 20:11:01 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6E399C433EF for ; Mon, 10 Jan 2022 04:25:23 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238765AbiAJEZU (ORCPT ); Sun, 9 Jan 2022 23:25:20 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36558 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238499AbiAJEYZ (ORCPT ); Sun, 9 Jan 2022 23:24:25 -0500 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 94A3AC06175F for ; Sun, 9 Jan 2022 20:24:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=VLn5Ly3DUFPdVtS0Qgb2wjVa3EHIQfZkAoJn3cqa2FI=; b=WGawqvxoO+CJRv74HiwyOdh+PN b+iQoWlITUT1w+6C4IlF/R++bv9in4Be37KvHIuECtLahc6W0KIgt/GAMMqp7P7WpHE5mIrDwf8sb 3P9ICkI1fOtX8ZV4dS98NLmbQ3egwI/s4PJGcnr8Uyuy3Xe9dO0EQ+qWDdNxy0V+9orxqUSEwOdWb jIdk+NCjOkxSLoIa/SNHILdCRpzaiw9sS9cZC045N+jicqjrxbru+BnZsLIvtu1ZfVlWiDX7OdBG6 oDZ8rGVSiyu4DimzjxWlKuHI2T1zfLh9RCKaBvOV3Vk5kNcOW5GRc1nl6ks9KoTLb3NNF4HomPZld 2kCdk9AQ==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1n6mE9-0025ww-3i; Mon, 10 Jan 2022 04:24:13 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , John Hubbard , Christoph Hellwig , William Kucharski , linux-kernel@vger.kernel.org, Jason Gunthorpe Subject: [PATCH v2 16/28] mm: Remove page_cache_add_speculative() and page_cache_get_speculative() Date: Mon, 10 Jan 2022 04:23:54 +0000 Message-Id: <20220110042406.499429-17-willy@infradead.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220110042406.499429-1-willy@infradead.org> References: <20220110042406.499429-1-willy@infradead.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" These wrappers have no more callers, so delete them. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Jason Gunthorpe Reviewed-by: William Kucharski --- include/linux/mm.h | 7 +++---- include/linux/pagemap.h | 11 ----------- 2 files changed, 3 insertions(+), 15 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index d3769897c8ac..b249156f7cf1 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1291,10 +1291,9 @@ static inline void put_page(struct page *page) * applications that don't have huge page reference counts, this won't be = an * issue. * - * Locking: the lockless algorithm described in page_cache_get_speculative= () - * and page_cache_gup_pin_speculative() provides safe operation for - * get_user_pages and page_mkclean and other calls that race to set up page - * table entries. + * Locking: the lockless algorithm described in folio_try_get_rcu() + * provides safe operation for get_user_pages(), page_mkclean() and + * other calls that race to set up page table entries. */ #define GUP_PIN_COUNTING_BIAS (1U << 10) =20 diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index 704cb1b4b15d..4a63176b6417 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -283,17 +283,6 @@ static inline struct inode *folio_inode(struct folio *= folio) return folio->mapping->host; } =20 -static inline bool page_cache_add_speculative(struct page *page, int count) -{ - VM_BUG_ON_PAGE(PageTail(page), page); - return folio_ref_try_add_rcu((struct folio *)page, count); -} - -static inline bool page_cache_get_speculative(struct page *page) -{ - return page_cache_add_speculative(page, 1); -} - /** * folio_attach_private - Attach private data to a folio. * @folio: Folio to attach data to. --=20 2.33.0 From nobody Tue Jun 30 20:11:01 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 42863C433F5 for ; Mon, 10 Jan 2022 04:26:17 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238874AbiAJE0O (ORCPT ); Sun, 9 Jan 2022 23:26:14 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36606 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238543AbiAJEYe (ORCPT ); Sun, 9 Jan 2022 23:24:34 -0500 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E3215C061757 for ; Sun, 9 Jan 2022 20:24:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=Fb1nGJRKTWWZNwZ6nVvnMjBwiTsGQjxg+7diVdo/Ofc=; b=Dmsv6SlX9hxCR2dO33GsQIoQlw yIjfiUE/WRcLyJ7IjHBPc/JcqVXtuQYDNDvtGXi48grWo9YT8gUI/nmO5sGTVS13s+GDsjcop2UpS KbPsO1/RLw+Q8vlrvP0klzrRiJxitQQJsPEd6hCOm+wHihLZvovsyu6czW9xZrV5dntD+Ip9Fv2xn VwMphye2h29mC/1n3wonvq+9dKiNbz7VIhAKFfVxMk3cVVb+izq2ODz+CJM/OWZnmuyYuxAgWUY/D f4vktvA96JVvGLjqngotNfvJAw75IOBJIutxW/IWMPmVLlsvJAic9D0OSQgPAQ6hh6n+bYnE0fLgJ ealpxBSw==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1n6mE9-0025wy-6A; Mon, 10 Jan 2022 04:24:13 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , John Hubbard , Christoph Hellwig , William Kucharski , linux-kernel@vger.kernel.org, Jason Gunthorpe Subject: [PATCH v2 17/28] gup: Add gup_put_folio() Date: Mon, 10 Jan 2022 04:23:55 +0000 Message-Id: <20220110042406.499429-18-willy@infradead.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220110042406.499429-1-willy@infradead.org> References: <20220110042406.499429-1-willy@infradead.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Convert put_compound_head() to gup_put_folio() and hpage_pincount_sub() to folio_pincount_sub(). This removes the last call to put_page_refs(), so delete it. Add a temporary put_compound_head() wrapper which will be deleted by the end of this series. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Jason Gunthorpe Reviewed-by: William Kucharski --- mm/gup.c | 42 ++++++++++++++---------------------------- 1 file changed, 14 insertions(+), 28 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index 9e581201d679..719252fa0402 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -45,34 +45,15 @@ static void folio_pincount_add(struct folio *folio, int= refs) folio_ref_add(folio, refs * (GUP_PIN_COUNTING_BIAS - 1)); } =20 -static int page_pincount_sub(struct page *page, int refs) +static int folio_pincount_sub(struct folio *folio, int refs) { - VM_BUG_ON_PAGE(page !=3D compound_head(page), page); - - if (PageHead(page)) - atomic_sub(refs, compound_pincount_ptr(page)); + if (folio_test_large(folio)) + atomic_sub(refs, folio_pincount_ptr(folio)); else refs *=3D GUP_PIN_COUNTING_BIAS; return refs; } =20 -/* Equivalent to calling put_page() @refs times. */ -static void put_page_refs(struct page *page, int refs) -{ -#ifdef CONFIG_DEBUG_VM - if (VM_WARN_ON_ONCE_PAGE(page_ref_count(page) < refs, page)) - return; -#endif - - /* - * Calling put_page() for each ref is unnecessarily slow. Only the last - * ref needs a put_page(). - */ - if (refs > 1) - page_ref_sub(page, refs - 1); - put_page(page); -} - /* * Return the folio with ref appropriately incremented, * or NULL if that failed. @@ -171,15 +152,20 @@ struct page *try_grab_compound_head(struct page *page, return &try_grab_folio(page, refs, flags)->page; } =20 -static void put_compound_head(struct page *page, int refs, unsigned int fl= ags) +static void gup_put_folio(struct folio *folio, int refs, unsigned int flag= s) { if (flags & FOLL_PIN) { - mod_node_page_state(page_pgdat(page), NR_FOLL_PIN_RELEASED, - refs); - refs =3D page_pincount_sub(page, refs); + node_stat_mod_folio(folio, NR_FOLL_PIN_RELEASED, refs); + refs =3D folio_pincount_sub(folio, refs); } =20 - put_page_refs(page, refs); + folio_put_refs(folio, refs); +} + +static void put_compound_head(struct page *page, int refs, unsigned int fl= ags) +{ + VM_BUG_ON_PAGE(PageTail(page), page); + gup_put_folio((struct folio *)page, refs, flags); } =20 /** @@ -220,7 +206,7 @@ bool __must_check try_grab_page(struct page *page, unsi= gned int flags) */ void unpin_user_page(struct page *page) { - put_compound_head(compound_head(page), 1, FOLL_PIN); + gup_put_folio(page_folio(page), 1, FOLL_PIN); } EXPORT_SYMBOL(unpin_user_page); =20 --=20 2.33.0 From nobody Tue Jun 30 20:11:01 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 683ABC433EF for ; Mon, 10 Jan 2022 04:25:03 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238683AbiAJEZA (ORCPT ); Sun, 9 Jan 2022 23:25:00 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36546 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238493AbiAJEYZ (ORCPT ); Sun, 9 Jan 2022 23:24:25 -0500 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C4E97C061759 for ; Sun, 9 Jan 2022 20:24:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=gfxLedy5iy4I1tA8ZKlC8DvCnDAEDbxXKKrSeENY8cY=; b=JYCKGH1t+KQP7QX9HjamnWk5W2 mF/7x01TlcS93WH/d+/4Muv2r6t8WSpkVek6GeJL4C4uinGkDtDqksIAv98QraviByTXyG1hUmFcC o9Ah5t5TYuDzWB0c7fPsRId9euRKMDY6m29n97X+/Kfxm8Ubq1iYjHPa4LvvPx2ezmlzSKWhe2tfF wSdnBxdJqdbIBrJhavYNKpwOkVpafajUtsBR68f9Q7FE8GIGJgvjSrSSfFte8Nx/OOGDb1ZuuyAIv YqudSeZxTK7Z0UjRKtKUnhH0/BNlPrA0Kndwv0mV85kAjM4UkjSa3al2haF7Z15986OEdpXKtGmXA mMZp5low==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1n6mE9-0025x0-8q; Mon, 10 Jan 2022 04:24:13 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , John Hubbard , Christoph Hellwig , William Kucharski , linux-kernel@vger.kernel.org, Jason Gunthorpe Subject: [PATCH v2 18/28] hugetlb: Use try_grab_folio() instead of try_grab_compound_head() Date: Mon, 10 Jan 2022 04:23:56 +0000 Message-Id: <20220110042406.499429-19-willy@infradead.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220110042406.499429-1-willy@infradead.org> References: <20220110042406.499429-1-willy@infradead.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" follow_hugetlb_page() only cares about success or failure, so it doesn't need to know the type of the returned pointer, only whether it's NULL or not. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Jason Gunthorpe Reviewed-by: William Kucharski --- include/linux/mm.h | 3 --- mm/gup.c | 2 +- mm/hugetlb.c | 7 +++---- 3 files changed, 4 insertions(+), 8 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index b249156f7cf1..c103c6401ecd 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1195,9 +1195,6 @@ static inline void get_page(struct page *page) } =20 bool __must_check try_grab_page(struct page *page, unsigned int flags); -struct page *try_grab_compound_head(struct page *page, int refs, - unsigned int flags); - =20 static inline __must_check bool try_get_page(struct page *page) { diff --git a/mm/gup.c b/mm/gup.c index 719252fa0402..20703de2f107 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -146,7 +146,7 @@ struct folio *try_grab_folio(struct page *page, int ref= s, unsigned int flags) return NULL; } =20 -struct page *try_grab_compound_head(struct page *page, +static inline struct page *try_grab_compound_head(struct page *page, int refs, unsigned int flags) { return &try_grab_folio(page, refs, flags)->page; diff --git a/mm/hugetlb.c b/mm/hugetlb.c index abcd1785c629..ab67b13c4a71 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -6072,7 +6072,7 @@ long follow_hugetlb_page(struct mm_struct *mm, struct= vm_area_struct *vma, =20 if (pages) { /* - * try_grab_compound_head() should always succeed here, + * try_grab_folio() should always succeed here, * because: a) we hold the ptl lock, and b) we've just * checked that the huge page is present in the page * tables. If the huge page is present, then the tail @@ -6081,9 +6081,8 @@ long follow_hugetlb_page(struct mm_struct *mm, struct= vm_area_struct *vma, * any way. So this page must be available at this * point, unless the page refcount overflowed: */ - if (WARN_ON_ONCE(!try_grab_compound_head(pages[i], - refs, - flags))) { + if (WARN_ON_ONCE(!try_grab_folio(pages[i], refs, + flags))) { spin_unlock(ptl); remainder =3D 0; err =3D -ENOMEM; --=20 2.33.0 From nobody Tue Jun 30 20:11:01 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B0676C433F5 for ; Mon, 10 Jan 2022 04:25:07 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238725AbiAJEZF (ORCPT ); Sun, 9 Jan 2022 23:25:05 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36556 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238498AbiAJEYZ (ORCPT ); Sun, 9 Jan 2022 23:24:25 -0500 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 04EBCC06175E for ; Sun, 9 Jan 2022 20:24:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=fC4mfcf/XAHzYtj0vYWgXRylAg3qW7XjyC6jR63n0ws=; b=G8FxiLvPsae9GR/t85Spn5ZHFx YOtLGi7JPPDzlItCRlMqM6B76CuawBJ6HkD3SK7XQsfLpIrVcPhRbhsBvt0CCRGHMGkW7MJXtimh3 vnqAYCyMjelK5u8jFMEkFjMxjzOL7WoHidac+q474yFVCn47qhdjhoFhIdMEpIFd6t9LZV3Q6scZv +d5KpY89Mw0mw09jWK8SF+Ix0odXuP9dWx+xHAz7W8mgpPaz1hCCnPs+/RNEP1G97LlLqlnz6aqX3 4Ny9Lsn0lGlLjRQWNF4iiaiYJ73Q47X8bemQGmMp0HFLgAvtvFZdjZgTcIHNs+EPBU6zaylj6iih9 IEzUEUAg==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1n6mE9-0025x2-Al; Mon, 10 Jan 2022 04:24:13 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , John Hubbard , Christoph Hellwig , William Kucharski , linux-kernel@vger.kernel.org, Jason Gunthorpe Subject: [PATCH v2 19/28] gup: Convert try_grab_page() to call try_grab_folio() Date: Mon, 10 Jan 2022 04:23:57 +0000 Message-Id: <20220110042406.499429-20-willy@infradead.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220110042406.499429-1-willy@infradead.org> References: <20220110042406.499429-1-willy@infradead.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" try_grab_page() only cares about success or failure, not about the type returned. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Jason Gunthorpe Reviewed-by: William Kucharski --- mm/gup.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/gup.c b/mm/gup.c index 20703de2f107..c3e514172eaf 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -192,7 +192,7 @@ bool __must_check try_grab_page(struct page *page, unsi= gned int flags) if (!(flags & (FOLL_GET | FOLL_PIN))) return true; =20 - return try_grab_compound_head(page, 1, flags); + return try_grab_folio(page, 1, flags); } =20 /** --=20 2.33.0 From nobody Tue Jun 30 20:11:01 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 13EE4C43219 for ; Mon, 10 Jan 2022 04:24:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238579AbiAJEYh (ORCPT ); Sun, 9 Jan 2022 23:24:37 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36544 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238485AbiAJEYZ (ORCPT ); Sun, 9 Jan 2022 23:24:25 -0500 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C081BC061757 for ; Sun, 9 Jan 2022 20:24:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=yZ4KQo4q1BMM1IW+AzT2B5hxEwiTFmWyBI9LyNwadcs=; b=Y5qROpafFS6IdPE7SWUIpy6y8Z Ds+tDsyfKz6WZBH31pIe6DOFpJl+sL1ipeyGzZS6M32jWbWlVpPjrdc0wyfd3tfgdzi96/xLaCYPK 6IWJzByu6T19rwB9OaCPv52ko4Rp38fup0pf0Dn/rNQA93YVC5YF+L2YbELkCpRTT9VVzRlW2Set/ YCOnWIPsNbA16eeJW5Q/hsx4elwOPo+uFT9qegKtEVyIg7uDimB17Irt6Tsu6pozdXSx4r9XT13LR HDCAr9UF4wB8jm9nDjxjU2PPIK4G+ApcUMyf20j05eU8DoKnc8U15y26j3J/PkR6sHGJBD2QPxycG 630C9lUg==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1n6mE9-0025x4-Ce; Mon, 10 Jan 2022 04:24:13 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , John Hubbard , Christoph Hellwig , William Kucharski , linux-kernel@vger.kernel.org, Jason Gunthorpe Subject: [PATCH v2 20/28] gup: Convert gup_pte_range() to use a folio Date: Mon, 10 Jan 2022 04:23:58 +0000 Message-Id: <20220110042406.499429-21-willy@infradead.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220110042406.499429-1-willy@infradead.org> References: <20220110042406.499429-1-willy@infradead.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" We still call try_grab_folio() once per PTE; a future patch could optimise to just adjust the reference count for each page within the folio. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Jason Gunthorpe Reviewed-by: William Kucharski --- mm/gup.c | 16 +++++++--------- 1 file changed, 7 insertions(+), 9 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index c3e514172eaf..27cc097ec05d 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -2235,7 +2235,8 @@ static int gup_pte_range(pmd_t pmd, unsigned long add= r, unsigned long end, ptem =3D ptep =3D pte_offset_map(&pmd, addr); do { pte_t pte =3D ptep_get_lockless(ptep); - struct page *head, *page; + struct page *page; + struct folio *folio; =20 /* * Similar to the PMD case below, NUMA hinting must take slow @@ -2262,22 +2263,20 @@ static int gup_pte_range(pmd_t pmd, unsigned long a= ddr, unsigned long end, VM_BUG_ON(!pfn_valid(pte_pfn(pte))); page =3D pte_page(pte); =20 - head =3D try_grab_compound_head(page, 1, flags); - if (!head) + folio =3D try_grab_folio(page, 1, flags); + if (!folio) goto pte_unmap; =20 if (unlikely(page_is_secretmem(page))) { - put_compound_head(head, 1, flags); + gup_put_folio(folio, 1, flags); goto pte_unmap; } =20 if (unlikely(pte_val(pte) !=3D pte_val(*ptep))) { - put_compound_head(head, 1, flags); + gup_put_folio(folio, 1, flags); goto pte_unmap; } =20 - VM_BUG_ON_PAGE(compound_head(page) !=3D head, page); - /* * We need to make the page accessible if and only if we are * going to access its content (the FOLL_PIN case). Please @@ -2291,10 +2290,9 @@ static int gup_pte_range(pmd_t pmd, unsigned long ad= dr, unsigned long end, goto pte_unmap; } } - SetPageReferenced(page); + folio_set_referenced(folio); pages[*nr] =3D page; (*nr)++; - } while (ptep++, addr +=3D PAGE_SIZE, addr !=3D end); =20 ret =3D 1; --=20 2.33.0 From nobody Tue Jun 30 20:11:01 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 40783C433EF for ; Mon, 10 Jan 2022 04:24:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238539AbiAJEYd (ORCPT ); Sun, 9 Jan 2022 23:24:33 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36540 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233329AbiAJEYZ (ORCPT ); Sun, 9 Jan 2022 23:24:25 -0500 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B6766C061751 for ; Sun, 9 Jan 2022 20:24:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=dlgtKTjkERgxYua7qA7Yoi6vlUOwWisaGlBoko6Kg7w=; b=MdYehaqZTNoBT0pJr/dBzKu0xk rhxJqipWHBLzxGzgjGBSI9ARSQ1RlA4UFGaT5vyh+ZLD3n7X2WtXi/hIBHEVGqogTYY7hy7cYaNbD L5MM96NFsVLNhjaHZpA7rpoJPoiVDLQaQ+clb3yED1pcwzO53aJrART1VpQnwzz3IlH8mHxrQnI4v HCGbkVFR1Hd/dDw/RZ/wB1DfOJHhjE7+qw3/DKruUDpZ+rY+erQhJUTjkB1tFsoQIt9v4WCQiF9mS GSAMnbkin/NrUmK334/OwCMLSJDzanEXMKDv7Mwe92ldpljQ6RNp0MiNFKkdnOmL2uqPkJBfzkTE6 4x4BRuug==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1n6mE9-0025x6-F5; Mon, 10 Jan 2022 04:24:13 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , John Hubbard , Christoph Hellwig , William Kucharski , linux-kernel@vger.kernel.org, Jason Gunthorpe Subject: [PATCH v2 21/28] gup: Convert gup_hugepte() to use a folio Date: Mon, 10 Jan 2022 04:23:59 +0000 Message-Id: <20220110042406.499429-22-willy@infradead.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220110042406.499429-1-willy@infradead.org> References: <20220110042406.499429-1-willy@infradead.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" There should be little to no effect from this patch; just removing uses of some old APIs. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Jason Gunthorpe Reviewed-by: William Kucharski --- mm/gup.c | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index 27cc097ec05d..250326458df6 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -2428,7 +2428,8 @@ static int gup_hugepte(pte_t *ptep, unsigned long sz,= unsigned long addr, struct page **pages, int *nr) { unsigned long pte_end; - struct page *head, *page; + struct page *page; + struct folio *folio; pte_t pte; int refs; =20 @@ -2444,21 +2445,20 @@ static int gup_hugepte(pte_t *ptep, unsigned long s= z, unsigned long addr, /* hugepages are never "special" */ VM_BUG_ON(!pfn_valid(pte_pfn(pte))); =20 - head =3D pte_page(pte); - page =3D nth_page(head, (addr & (sz-1)) >> PAGE_SHIFT); + page =3D nth_page(pte_page(pte), (addr & (sz - 1)) >> PAGE_SHIFT); refs =3D record_subpages(page, addr, end, pages + *nr); =20 - head =3D try_grab_compound_head(head, refs, flags); - if (!head) + folio =3D try_grab_folio(page, refs, flags); + if (!folio) return 0; =20 if (unlikely(pte_val(pte) !=3D pte_val(*ptep))) { - put_compound_head(head, refs, flags); + gup_put_folio(folio, refs, flags); return 0; } =20 *nr +=3D refs; - SetPageReferenced(head); + folio_set_referenced(folio); return 1; } =20 --=20 2.33.0 From nobody Tue Jun 30 20:11:01 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A2016C433F5 for ; Mon, 10 Jan 2022 04:27:27 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238795AbiAJE1Z (ORCPT ); Sun, 9 Jan 2022 23:27:25 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36686 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238720AbiAJEZF (ORCPT ); Sun, 9 Jan 2022 23:25:05 -0500 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0F805C034000 for ; Sun, 9 Jan 2022 20:24:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=Tv81wfblYI2u4OnQaOMT5hu8NAr/0b4Pr52a1X1WL0g=; b=diks2I8M9hwZoO5nHFOXQp5WyE eA0oBnejOYqo+gENL+UsfMVy2fpxjWcrb7cx8F1VccqE0JvMXMYxPIwO1yv1SwfSxeISuecUznhZ7 bAIhQh40BWNiigAhbgDK/3iP4kW1Lnu4yIh/wLpKya93/tiE/PYCK52JWgDPxwa5XkP+PYAQL7W0m hLYS7ATN46P4pWbEIuy8Rm5w77AT4mexrKdF3SaxgC0ebQqzikpT8VZrNc77UhRWc0OZNbNp/Rqif D3rJTMFywPuaEVUm8fKfWuQftRHYCeN6D0Hg7QNvMqMprECBqONl3IDrBpubXzBu0TXHe840r9AYj F5VLmCtQ==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1n6mE9-0025x8-HW; Mon, 10 Jan 2022 04:24:13 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , John Hubbard , Christoph Hellwig , William Kucharski , linux-kernel@vger.kernel.org, Jason Gunthorpe Subject: [PATCH v2 22/28] gup: Convert gup_huge_pmd() to use a folio Date: Mon, 10 Jan 2022 04:24:00 +0000 Message-Id: <20220110042406.499429-23-willy@infradead.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220110042406.499429-1-willy@infradead.org> References: <20220110042406.499429-1-willy@infradead.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Use the new folio-based APIs. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Jason Gunthorpe Reviewed-by: William Kucharski --- mm/gup.c | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index 250326458df6..a006bce2d47b 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -2492,7 +2492,8 @@ static int gup_huge_pmd(pmd_t orig, pmd_t *pmdp, unsi= gned long addr, unsigned long end, unsigned int flags, struct page **pages, int *nr) { - struct page *head, *page; + struct page *page; + struct folio *folio; int refs; =20 if (!pmd_access_permitted(orig, flags & FOLL_WRITE)) @@ -2508,17 +2509,17 @@ static int gup_huge_pmd(pmd_t orig, pmd_t *pmdp, un= signed long addr, page =3D nth_page(pmd_page(orig), (addr & ~PMD_MASK) >> PAGE_SHIFT); refs =3D record_subpages(page, addr, end, pages + *nr); =20 - head =3D try_grab_compound_head(pmd_page(orig), refs, flags); - if (!head) + folio =3D try_grab_folio(page, refs, flags); + if (!folio) return 0; =20 if (unlikely(pmd_val(orig) !=3D pmd_val(*pmdp))) { - put_compound_head(head, refs, flags); + gup_put_folio(folio, refs, flags); return 0; } =20 *nr +=3D refs; - SetPageReferenced(head); + folio_set_referenced(folio); return 1; } =20 --=20 2.33.0 From nobody Tue Jun 30 20:11:01 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 986A7C433EF for ; Mon, 10 Jan 2022 04:26:27 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238697AbiAJE00 (ORCPT ); Sun, 9 Jan 2022 23:26:26 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36628 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238581AbiAJEYh (ORCPT ); Sun, 9 Jan 2022 23:24:37 -0500 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D2D54C06173F for ; Sun, 9 Jan 2022 20:24:36 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=toFUQqgYtl0v327PRslx3eBHdHJmUYmTsDw9nmAaME0=; b=dM3u7vcFyE2WKYv1dSy8MDTHG5 IGAL8q8VFYIIRashMmorLNTm3sfc4dsBwvsqXdwH3y6KCv4JHhMrUkNUaRhwrNh5yh725n0YhK8uw uW0u2BPiVjZpRxGA5+PijRHAsruBIUkNi6bZ6q+KJ4SsxA8TzXWqvvxvCsEhpH3lPDBG40DST1uQU kbK2dhclA8f1ozFQx3bUSlmcnxiaW/oifphqGtxqcIiXCiRKXu7tMuaANJJXGhBHcHxpfplm/Ptis Fx5YfJgoXVKWnN7dbFWK9ykw1dmIks3XqCyIjI/EA5z4yJI/wEO7F4IKRUdt4ZvZa29a/EBWNGFiz MrAhv56g==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1n6mE9-0025xA-Jy; Mon, 10 Jan 2022 04:24:13 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , John Hubbard , Christoph Hellwig , William Kucharski , linux-kernel@vger.kernel.org, Jason Gunthorpe Subject: [PATCH v2 23/28] gup: Convert gup_huge_pud() to use a folio Date: Mon, 10 Jan 2022 04:24:01 +0000 Message-Id: <20220110042406.499429-24-willy@infradead.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220110042406.499429-1-willy@infradead.org> References: <20220110042406.499429-1-willy@infradead.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Use the new folio-based APIs. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Jason Gunthorpe Reviewed-by: William Kucharski --- mm/gup.c | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index a006bce2d47b..7b7bf8361558 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -2527,7 +2527,8 @@ static int gup_huge_pud(pud_t orig, pud_t *pudp, unsi= gned long addr, unsigned long end, unsigned int flags, struct page **pages, int *nr) { - struct page *head, *page; + struct page *page; + struct folio *folio; int refs; =20 if (!pud_access_permitted(orig, flags & FOLL_WRITE)) @@ -2543,17 +2544,17 @@ static int gup_huge_pud(pud_t orig, pud_t *pudp, un= signed long addr, page =3D nth_page(pud_page(orig), (addr & ~PUD_MASK) >> PAGE_SHIFT); refs =3D record_subpages(page, addr, end, pages + *nr); =20 - head =3D try_grab_compound_head(pud_page(orig), refs, flags); - if (!head) + folio =3D try_grab_folio(page, refs, flags); + if (!folio) return 0; =20 if (unlikely(pud_val(orig) !=3D pud_val(*pudp))) { - put_compound_head(head, refs, flags); + gup_put_folio(folio, refs, flags); return 0; } =20 *nr +=3D refs; - SetPageReferenced(head); + folio_set_referenced(folio); return 1; } =20 --=20 2.33.0 From nobody Tue Jun 30 20:11:01 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id CA635C433EF for ; Mon, 10 Jan 2022 04:24:54 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238594AbiAJEYx (ORCPT ); Sun, 9 Jan 2022 23:24:53 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36550 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238494AbiAJEYZ (ORCPT ); Sun, 9 Jan 2022 23:24:25 -0500 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CE28BC06175B for ; Sun, 9 Jan 2022 20:24:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=FrPzWZ09xs9aa5csC+CTNKkHIKuSwjgWq1vjKtgzbWE=; b=PDZZlizvSkKJQNj2x7tsRWpb1N hdVKt7y+Xxh4qzyI+mWWsfjz8avSgFEZji3flVPQAM8N9XJFoZnAIjvu0gfC6AYOJS5I9+g7jBzhf +EcIZ/DFNZhcIBP6lqxrI9kuL1XbW49yqz7xGkLy4Jwwu4b+cLJym2/ff8s4QJMm00XgLqQGrsFy2 a6OeonVlXtwEvdKeN3+UTDGgJk8ikxpgZEHOHj8fC1C8EdX/yQ/+VNxk4EQnHstSUcksjT1ScuMjs N0aPifks38xolz+Si80N3UlHe2ElKqal3R/CjOy2CilLpPvco/iPkHGviOgvnv8LZopjAAEoVjon3 8DosvRHA==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1n6mE9-0025xC-MQ; Mon, 10 Jan 2022 04:24:13 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , John Hubbard , Christoph Hellwig , William Kucharski , linux-kernel@vger.kernel.org, Jason Gunthorpe Subject: [PATCH v2 24/28] gup: Convert gup_huge_pgd() to use a folio Date: Mon, 10 Jan 2022 04:24:02 +0000 Message-Id: <20220110042406.499429-25-willy@infradead.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220110042406.499429-1-willy@infradead.org> References: <20220110042406.499429-1-willy@infradead.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Use the new folio-based APIs. This was the last user of try_grab_compound_head(), so remove it. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Jason Gunthorpe Reviewed-by: William Kucharski --- mm/gup.c | 17 ++++++----------- 1 file changed, 6 insertions(+), 11 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index 7b7bf8361558..b5786e83c418 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -146,12 +146,6 @@ struct folio *try_grab_folio(struct page *page, int re= fs, unsigned int flags) return NULL; } =20 -static inline struct page *try_grab_compound_head(struct page *page, - int refs, unsigned int flags) -{ - return &try_grab_folio(page, refs, flags)->page; -} - static void gup_put_folio(struct folio *folio, int refs, unsigned int flag= s) { if (flags & FOLL_PIN) { @@ -2563,7 +2557,8 @@ static int gup_huge_pgd(pgd_t orig, pgd_t *pgdp, unsi= gned long addr, struct page **pages, int *nr) { int refs; - struct page *head, *page; + struct page *page; + struct folio *folio; =20 if (!pgd_access_permitted(orig, flags & FOLL_WRITE)) return 0; @@ -2573,17 +2568,17 @@ static int gup_huge_pgd(pgd_t orig, pgd_t *pgdp, un= signed long addr, page =3D nth_page(pgd_page(orig), (addr & ~PGDIR_MASK) >> PAGE_SHIFT); refs =3D record_subpages(page, addr, end, pages + *nr); =20 - head =3D try_grab_compound_head(pgd_page(orig), refs, flags); - if (!head) + folio =3D try_grab_folio(page, refs, flags); + if (!folio) return 0; =20 if (unlikely(pgd_val(orig) !=3D pgd_val(*pgdp))) { - put_compound_head(head, refs, flags); + gup_put_folio(folio, refs, flags); return 0; } =20 *nr +=3D refs; - SetPageReferenced(head); + folio_set_referenced(folio); return 1; } =20 --=20 2.33.0 From nobody Tue Jun 30 20:11:01 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id EA86AC433F5 for ; Mon, 10 Jan 2022 04:25:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233348AbiAJEZd (ORCPT ); Sun, 9 Jan 2022 23:25:33 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36568 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238509AbiAJEY1 (ORCPT ); Sun, 9 Jan 2022 23:24:27 -0500 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 73FA9C06173F for ; Sun, 9 Jan 2022 20:24:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=SzJAO82OEH7GJ/mx7wGbrx//DOWsumHlBiQh3Qh2tnQ=; b=fjA0ppGWoAmzrPEL5HhRkIA+hr fzlhnHNr4mbg+FEjXHmiirYvrOledQFtXWc+8V+wEmsSXldqHGRyDqFahNY1GgB+Djy/o6DrumHiW a9LAONzmOJGdPQ7c1X+jl+BpOYhqLCvUzgUkFc5bKyNcnzdG1wEL80ExG+BOgn/L3HoDTLwJ00VOW hhsvdm2fk05IAB25GVH39Ip69aba3Bk3Ba2n7Zx7f0L9KpUOCn6Dn9tZ9Gaa2nSJrlxOzyUSIkITB zzm50sqecdqw8qZaSbFuY/V8Qovp1Z8QLFT4/kSUZZ8KBmhTI7AWTA9RGWJC0tgQscXCShYQFp65Z wSTTeGFg==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1n6mE9-0025xE-OM; Mon, 10 Jan 2022 04:24:13 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , John Hubbard , Christoph Hellwig , William Kucharski , linux-kernel@vger.kernel.org, Jason Gunthorpe Subject: [PATCH v2 25/28] gup: Convert compound_next() to gup_folio_next() Date: Mon, 10 Jan 2022 04:24:03 +0000 Message-Id: <20220110042406.499429-26-willy@infradead.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220110042406.499429-1-willy@infradead.org> References: <20220110042406.499429-1-willy@infradead.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Convert both callers to work on folios instead of pages. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Jason Gunthorpe Reviewed-by: William Kucharski --- mm/gup.c | 41 ++++++++++++++++++++++------------------- 1 file changed, 22 insertions(+), 19 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index b5786e83c418..0cf2d5fd8d2d 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -220,20 +220,20 @@ static inline struct page *compound_range_next(unsign= ed long i, return page; } =20 -static inline struct page *compound_next(unsigned long i, +static inline struct folio *gup_folio_next(unsigned long i, unsigned long npages, struct page **list, unsigned int *ntails) { - struct page *page; + struct folio *folio; unsigned int nr; =20 - page =3D compound_head(list[i]); + folio =3D page_folio(list[i]); for (nr =3D i + 1; nr < npages; nr++) { - if (compound_head(list[nr]) !=3D page) + if (page_folio(list[nr]) !=3D folio) break; } =20 *ntails =3D nr - i; - return page; + return folio; } =20 /** @@ -261,17 +261,17 @@ static inline struct page *compound_next(unsigned lon= g i, void unpin_user_pages_dirty_lock(struct page **pages, unsigned long npages, bool make_dirty) { - unsigned long index; - struct page *head; - unsigned int ntails; + unsigned long i; + struct folio *folio; + unsigned int nr; =20 if (!make_dirty) { unpin_user_pages(pages, npages); return; } =20 - for (index =3D 0; index < npages; index +=3D ntails) { - head =3D compound_next(index, npages, pages, &ntails); + for (i =3D 0; i < npages; i +=3D nr) { + folio =3D gup_folio_next(i, npages, pages, &nr); /* * Checking PageDirty at this point may race with * clear_page_dirty_for_io(), but that's OK. Two key @@ -292,9 +292,12 @@ void unpin_user_pages_dirty_lock(struct page **pages, = unsigned long npages, * written back, so it gets written back again in the * next writeback cycle. This is harmless. */ - if (!PageDirty(head)) - set_page_dirty_lock(head); - put_compound_head(head, ntails, FOLL_PIN); + if (!folio_test_dirty(folio)) { + folio_lock(folio); + folio_mark_dirty(folio); + folio_unlock(folio); + } + gup_put_folio(folio, nr, FOLL_PIN); } } EXPORT_SYMBOL(unpin_user_pages_dirty_lock); @@ -347,9 +350,9 @@ EXPORT_SYMBOL(unpin_user_page_range_dirty_lock); */ void unpin_user_pages(struct page **pages, unsigned long npages) { - unsigned long index; - struct page *head; - unsigned int ntails; + unsigned long i; + struct folio *folio; + unsigned int nr; =20 /* * If this WARN_ON() fires, then the system *might* be leaking pages (by @@ -359,9 +362,9 @@ void unpin_user_pages(struct page **pages, unsigned lon= g npages) if (WARN_ON(IS_ERR_VALUE(npages))) return; =20 - for (index =3D 0; index < npages; index +=3D ntails) { - head =3D compound_next(index, npages, pages, &ntails); - put_compound_head(head, ntails, FOLL_PIN); + for (i =3D 0; i < npages; i +=3D nr) { + folio =3D gup_folio_next(i, npages, pages, &nr); + gup_put_folio(folio, nr, FOLL_PIN); } } EXPORT_SYMBOL(unpin_user_pages); --=20 2.33.0 From nobody Tue Jun 30 20:11:01 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5A924C433EF for ; Mon, 10 Jan 2022 04:24:51 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238587AbiAJEYs (ORCPT ); Sun, 9 Jan 2022 23:24:48 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36548 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238492AbiAJEYZ (ORCPT ); Sun, 9 Jan 2022 23:24:25 -0500 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C9353C06175A for ; Sun, 9 Jan 2022 20:24:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=Z2gQhd94zLvZOxg2P3olHAv1sEvF2BSvVZSPWfTVDGI=; b=SZFQzxmRdm+Os236gX9J2RY9g8 ckaQRRqi/4gGgNiUzDOpo+Mjlm1JMQ98qgXAULO+b56mRwhJWsQPtarcgcozdlMQtgJPjUrqev1ee dkhmh9+NEytrAX2GgYwm34QOpOrOYIGR6IRK72602NpYlhbosHIwe3m5WeOXbQbjDr2DXAtZFsTv5 Wf1tYpEl641n2TRVis0AorZG7PFLpKfda9BjAiHgxleT6FYlwiSLRg7ZOMPlHmCv192Sre5GH6XTj 3QzhO5KG4nhoQc8iIjTzVKrlSItQNzoJQKIkOQU0vBw1vWD+e4ghX4G5GfqLC1r5M6sRjkwQbMhE6 qrVmW3Hw==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1n6mE9-0025xG-S7; Mon, 10 Jan 2022 04:24:13 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , John Hubbard , Christoph Hellwig , William Kucharski , linux-kernel@vger.kernel.org, Jason Gunthorpe Subject: [PATCH v2 26/28] gup: Convert compound_range_next() to gup_folio_range_next() Date: Mon, 10 Jan 2022 04:24:04 +0000 Message-Id: <20220110042406.499429-27-willy@infradead.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220110042406.499429-1-willy@infradead.org> References: <20220110042406.499429-1-willy@infradead.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Convert the only caller to work on folios instead of pages. This removes the last caller of put_compound_head(), so delete it. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Jason Gunthorpe Reviewed-by: William Kucharski --- include/linux/mm.h | 4 ++-- mm/gup.c | 38 ++++++++++++++++++-------------------- 2 files changed, 20 insertions(+), 22 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index c103c6401ecd..1ddb0a55b5ca 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -216,10 +216,10 @@ int overcommit_policy_handler(struct ctl_table *, int= , void *, size_t *, =20 #if defined(CONFIG_SPARSEMEM) && !defined(CONFIG_SPARSEMEM_VMEMMAP) #define nth_page(page,n) pfn_to_page(page_to_pfn((page)) + (n)) -#define page_nth(head, tail) (page_to_pfn(tail) - page_to_pfn(head)) +#define folio_nth(folio, page) (page_to_pfn(page) - folio_pfn(folio)) #else #define nth_page(page,n) ((page) + (n)) -#define page_nth(head, tail) ((tail) - (head)) +#define folio_nth(folio, tail) ((tail) - &(folio)->page) #endif =20 /* to align the pointer to the (next) page boundary */ diff --git a/mm/gup.c b/mm/gup.c index 0cf2d5fd8d2d..1cdd5f2887a8 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -156,12 +156,6 @@ static void gup_put_folio(struct folio *folio, int ref= s, unsigned int flags) folio_put_refs(folio, refs); } =20 -static void put_compound_head(struct page *page, int refs, unsigned int fl= ags) -{ - VM_BUG_ON_PAGE(PageTail(page), page); - gup_put_folio((struct folio *)page, refs, flags); -} - /** * try_grab_page() - elevate a page's refcount by a flag-dependent amount * @@ -204,20 +198,21 @@ void unpin_user_page(struct page *page) } EXPORT_SYMBOL(unpin_user_page); =20 -static inline struct page *compound_range_next(unsigned long i, +static inline struct folio *gup_folio_range_next(unsigned long i, unsigned long npages, struct page *start, unsigned int *ntails) { - struct page *next, *page; + struct page *next; + struct folio *folio; unsigned int nr =3D 1; =20 next =3D nth_page(start, i); - page =3D compound_head(next); - if (PageHead(page)) + folio =3D page_folio(next); + if (folio_test_large(folio)) nr =3D min_t(unsigned int, npages - i, - compound_nr(page) - page_nth(page, next)); + folio_nr_pages(folio) - folio_nth(folio, next)); =20 *ntails =3D nr; - return page; + return folio; } =20 static inline struct folio *gup_folio_next(unsigned long i, @@ -326,15 +321,18 @@ EXPORT_SYMBOL(unpin_user_pages_dirty_lock); void unpin_user_page_range_dirty_lock(struct page *page, unsigned long npa= ges, bool make_dirty) { - unsigned long index; - struct page *head; - unsigned int ntails; + unsigned long i; + struct folio *folio; + unsigned int nr; =20 - for (index =3D 0; index < npages; index +=3D ntails) { - head =3D compound_range_next(index, npages, page, &ntails); - if (make_dirty && !PageDirty(head)) - set_page_dirty_lock(head); - put_compound_head(head, ntails, FOLL_PIN); + for (i =3D 0; i < npages; i +=3D nr) { + folio =3D gup_folio_range_next(i, npages, page, &nr); + if (make_dirty && !folio_test_dirty(folio)) { + folio_lock(folio); + folio_mark_dirty(folio); + folio_unlock(folio); + } + gup_put_folio(folio, nr, FOLL_PIN); } } EXPORT_SYMBOL(unpin_user_page_range_dirty_lock); --=20 2.33.0 From nobody Tue Jun 30 20:11:01 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 87B4FC433FE for ; Mon, 10 Jan 2022 04:24:46 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238584AbiAJEYo (ORCPT ); Sun, 9 Jan 2022 23:24:44 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36552 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238495AbiAJEYZ (ORCPT ); Sun, 9 Jan 2022 23:24:25 -0500 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D302CC06175C for ; Sun, 9 Jan 2022 20:24:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=cvgoV6iZTRFNzicbxb1xEs3MoVjFdOdlhKJq+daLZeY=; b=QaV5vrd9M3+Jx4m/wixZx8OTTL Kxj5Bb1AaFaoQo8k3WogjzA1Vyddx1okmixBW/P1DTs4XgAk4mAe6MWzihj5eYcleH78bpYhTPdQH DuN67oRLI0aUgReH/FnnUnU4UZ6Zd39EadpnRXBwsH0iIoBWZQAAg/HhQbGbqheiMK9IqVriQkYnj hTfD3AGBFXanP8VBqimr3O3Bdroe0JsrERS2+hBC7+oX1kA+tNQ/Q1cWWg6lOrzfxFAaf9iakUqQe 75J2OWyZzKXjai/c5Bw5stgexuKiLxbhMhqEDnT2HWHUWu0X4jPrXmxYpy+aAMf5HBHFUZ9URN+qG OcgIuu0g==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1n6mE9-0025xI-V4; Mon, 10 Jan 2022 04:24:13 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , John Hubbard , Christoph Hellwig , William Kucharski , linux-kernel@vger.kernel.org, Jason Gunthorpe Subject: [PATCH v2 27/28] mm: Add isolate_lru_folio() Date: Mon, 10 Jan 2022 04:24:05 +0000 Message-Id: <20220110042406.499429-28-willy@infradead.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220110042406.499429-1-willy@infradead.org> References: <20220110042406.499429-1-willy@infradead.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Turn isolate_lru_page() into a wrapper around isolate_lru_folio(). TestClearPageLRU() would have always failed on a tail page, so returning -EBUSY is the same behaviour. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Jason Gunthorpe Reviewed-by: William Kucharski --- arch/powerpc/include/asm/mmu_context.h | 1 - mm/folio-compat.c | 8 +++++ mm/internal.h | 3 +- mm/vmscan.c | 43 ++++++++++++-------------- 4 files changed, 29 insertions(+), 26 deletions(-) diff --git a/arch/powerpc/include/asm/mmu_context.h b/arch/powerpc/include/= asm/mmu_context.h index 9ba6b585337f..b9cab0a11421 100644 --- a/arch/powerpc/include/asm/mmu_context.h +++ b/arch/powerpc/include/asm/mmu_context.h @@ -21,7 +21,6 @@ extern void destroy_context(struct mm_struct *mm); #ifdef CONFIG_SPAPR_TCE_IOMMU struct mm_iommu_table_group_mem_t; =20 -extern int isolate_lru_page(struct page *page); /* from internal.h */ extern bool mm_iommu_preregistered(struct mm_struct *mm); extern long mm_iommu_new(struct mm_struct *mm, unsigned long ua, unsigned long entries, diff --git a/mm/folio-compat.c b/mm/folio-compat.c index 749555a232a8..782e766cd1ee 100644 --- a/mm/folio-compat.c +++ b/mm/folio-compat.c @@ -7,6 +7,7 @@ #include #include #include +#include "internal.h" =20 struct address_space *page_mapping(struct page *page) { @@ -151,3 +152,10 @@ int try_to_release_page(struct page *page, gfp_t gfp) return filemap_release_folio(page_folio(page), gfp); } EXPORT_SYMBOL(try_to_release_page); + +int isolate_lru_page(struct page *page) +{ + if (WARN_RATELIMIT(PageTail(page), "trying to isolate tail page")) + return -EBUSY; + return isolate_lru_folio((struct folio *)page); +} diff --git a/mm/internal.h b/mm/internal.h index 9a72d1ecdab4..8b90db90e7f2 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -157,7 +157,8 @@ extern unsigned long highest_memmap_pfn; /* * in mm/vmscan.c: */ -extern int isolate_lru_page(struct page *page); +int isolate_lru_page(struct page *page); +int isolate_lru_folio(struct folio *folio); extern void putback_lru_page(struct page *page); extern void reclaim_throttle(pg_data_t *pgdat, enum vmscan_throttle_state = reason); =20 diff --git a/mm/vmscan.c b/mm/vmscan.c index fb9584641ac7..ac2f5b76cdb2 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -2168,45 +2168,40 @@ static unsigned long isolate_lru_pages(unsigned lon= g nr_to_scan, } =20 /** - * isolate_lru_page - tries to isolate a page from its LRU list - * @page: page to isolate from its LRU list + * isolate_lru_folio - Try to isolate a folio from its LRU list. + * @folio: Folio to isolate from its LRU list. * - * Isolates a @page from an LRU list, clears PageLRU and adjusts the - * vmstat statistic corresponding to whatever LRU list the page was on. + * Isolate a @folio from an LRU list and adjust the vmstat statistic + * corresponding to whatever LRU list the folio was on. * - * Returns 0 if the page was removed from an LRU list. - * Returns -EBUSY if the page was not on an LRU list. - * - * The returned page will have PageLRU() cleared. If it was found on - * the active list, it will have PageActive set. If it was found on - * the unevictable list, it will have the PageUnevictable bit set. That fl= ag + * The folio will have its LRU flag cleared. If it was found on the + * active list, it will have the Active flag set. If it was found on the + * unevictable list, it will have the Unevictable flag set. These flags * may need to be cleared by the caller before letting the page go. * - * The vmstat statistic corresponding to the list on which the page was - * found will be decremented. - * - * Restrictions: + * Context: * * (1) Must be called with an elevated refcount on the page. This is a - * fundamental difference from isolate_lru_pages (which is called + * fundamental difference from isolate_lru_pages() (which is called * without a stable reference). - * (2) the lru_lock must not be held. - * (3) interrupts must be enabled. + * (2) The lru_lock must not be held. + * (3) Interrupts must be enabled. + * + * Return: 0 if the folio was removed from an LRU list. + * -EBUSY if the folio was not on an LRU list. */ -int isolate_lru_page(struct page *page) +int isolate_lru_folio(struct folio *folio) { - struct folio *folio =3D page_folio(page); int ret =3D -EBUSY; =20 - VM_BUG_ON_PAGE(!page_count(page), page); - WARN_RATELIMIT(PageTail(page), "trying to isolate tail page"); + VM_BUG_ON_FOLIO(!folio_ref_count(folio), folio); =20 - if (TestClearPageLRU(page)) { + if (folio_test_clear_lru(folio)) { struct lruvec *lruvec; =20 - get_page(page); + folio_get(folio); lruvec =3D folio_lruvec_lock_irq(folio); - del_page_from_lru_list(page, lruvec); + lruvec_del_folio(lruvec, folio); unlock_page_lruvec_irq(lruvec); ret =3D 0; } --=20 2.33.0 From nobody Tue Jun 30 20:11:01 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 97DD2C4332F for ; Mon, 10 Jan 2022 04:24:32 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238525AbiAJEYa (ORCPT ); Sun, 9 Jan 2022 23:24:30 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36536 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233144AbiAJEYZ (ORCPT ); Sun, 9 Jan 2022 23:24:25 -0500 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AB18AC06173F for ; Sun, 9 Jan 2022 20:24:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=bF3YwsJlDdayaosPsW7OCN1/4CCh76lSoZ3+feMmOkc=; b=PQk/7QjPqGUvXtrFs0Kr76GObb Xng+45VgzfG8GyAOUqUyRYL+YY0Khbhmk3/VWOl13W9NIKy2s9O7y0iZBYM4LlIDEA4SvuqHJGkwi 85c5ycIXougd1QOFneGEldcJQhSiMm/Xtu4pnGjK20d3hXYGKnFctxHQwWtGUG5dXWVItZ8lqqSkB bPcdmokKh7Q4sCD7wcdLHuR3+2VWirUQ8rUzwDF8NtvWwG+vIc0L6dFZu6Lw7TW0ejqjEo15pXiGK 8SkQfW8Y6XX/KmtV8BDF6iArC0FmXD1TEXPzz6tXqNQ1rvSniEvUx6bDIMtVHm0Knx8fPxDmUzRKX 803c6OaA==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1n6mEA-0025xL-1X; Mon, 10 Jan 2022 04:24:14 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , John Hubbard , Christoph Hellwig , William Kucharski , linux-kernel@vger.kernel.org, Jason Gunthorpe Subject: [PATCH v2 28/28] gup: Convert check_and_migrate_movable_pages() to use a folio Date: Mon, 10 Jan 2022 04:24:06 +0000 Message-Id: <20220110042406.499429-29-willy@infradead.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220110042406.499429-1-willy@infradead.org> References: <20220110042406.499429-1-willy@infradead.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Switch from head pages to folios. This removes an assumption that THPs are the only way to have a high-order page. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Jason Gunthorpe Reviewed-by: William Kucharski --- mm/gup.c | 28 ++++++++++++++-------------- 1 file changed, 14 insertions(+), 14 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index 1cdd5f2887a8..b2d109626c44 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -1801,41 +1801,41 @@ static long check_and_migrate_movable_pages(unsigne= d long nr_pages, bool drain_allow =3D true; LIST_HEAD(movable_page_list); long ret =3D 0; - struct page *prev_head =3D NULL; - struct page *head; + struct folio *folio, *prev_folio =3D NULL; struct migration_target_control mtc =3D { .nid =3D NUMA_NO_NODE, .gfp_mask =3D GFP_USER | __GFP_NOWARN, }; =20 for (i =3D 0; i < nr_pages; i++) { - head =3D compound_head(pages[i]); - if (head =3D=3D prev_head) + folio =3D page_folio(pages[i]); + if (folio =3D=3D prev_folio) continue; - prev_head =3D head; + prev_folio =3D folio; /* * If we get a movable page, since we are going to be pinning * these entries, try to move them out if possible. */ - if (!is_pinnable_page(head)) { - if (PageHuge(head)) { - if (!isolate_huge_page(head, &movable_page_list)) + if (!is_pinnable_page(&folio->page)) { + if (folio_test_hugetlb(folio)) { + if (!isolate_huge_page(&folio->page, + &movable_page_list)) isolation_error_count++; } else { - if (!PageLRU(head) && drain_allow) { + if (!folio_test_lru(folio) && drain_allow) { lru_add_drain_all(); drain_allow =3D false; } =20 - if (isolate_lru_page(head)) { + if (isolate_lru_folio(folio)) { isolation_error_count++; continue; } - list_add_tail(&head->lru, &movable_page_list); - mod_node_page_state(page_pgdat(head), + list_add_tail(&folio->lru, &movable_page_list); + node_stat_mod_folio(folio, NR_ISOLATED_ANON + - page_is_file_lru(head), - thp_nr_pages(head)); + folio_is_file_lru(folio), + folio_nr_pages(folio)); } } } --=20 2.33.0