From nobody Wed Jul 1 13:27:41 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id BEE78C43217 for ; Tue, 21 Dec 2021 15:01:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235776AbhLUPBw (ORCPT ); Tue, 21 Dec 2021 10:01:52 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53302 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235750AbhLUPBq (ORCPT ); Tue, 21 Dec 2021 10:01:46 -0500 Received: from mail-qk1-x732.google.com (mail-qk1-x732.google.com [IPv6:2607:f8b0:4864:20::732]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E51F0C061574 for ; Tue, 21 Dec 2021 07:01:45 -0800 (PST) Received: by mail-qk1-x732.google.com with SMTP id 131so1524342qkk.2 for ; Tue, 21 Dec 2021 07:01:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=VWjTX7nyk2kx8oM7xeRq6hwgPVlUZA1U2SvwycLjyC4=; b=lgtZiYGORboj3gotQhfaYY0bpKv8j7J53Ax30ReWADNOLR3pP3G9Z7zz0Dj3k7Uk5e 0ux/jWhBkcDDWZNbsnxvxvWqzhDnWsFJL6QgUavAiI/eka9aWQq8xoxVtqUlS9RGSBUW nlTTru8N+9u/o7uUhKBpJshNUteAtIguMa1UgZsfWlJD78EVWpHWIFS2slz+8JXVVG1f JLyUy/6E5WopWvHFpq8EHOeA9IBe+56qANZgmDPrZtgk0pC+n3iGJQHqc4n2berXKDeA jCzCBlAGXkRImCHI7kzOxNqbzac8AssrtfSH3cOrvGK1iK8PCqdcWvyNr7g3q0q6Tx+5 tI5w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=VWjTX7nyk2kx8oM7xeRq6hwgPVlUZA1U2SvwycLjyC4=; b=4DaieFA3z01QI0cNy07tGfMrvG9rKfkse7rzHN07H46Csugy31xybbUS6NAZqZRCWx fK8tEBExjKRa3zjJ7z2K38tvKBIOOZxKI/LzMsCkpiamb0Y6xs57Fv1XWW0zr2Ka71n+ 7JKO/yilCCzetP7ugCilsLPl9hERiR+eL+1/sOppblg/lwbzJNY1cUZH7ernoMt4BVm+ foFofMmVPsN0j2SUYLMIWkNWJ9CBq8VPLTLD6FH8SXRs3byqTrkjTing8zETWxJTasg8 7bMrIr1rBQ9EylvJoL2k5DgnpuwjLQ6AsGdb39MEiGrgISK94rp06hJijCOjnM0wkB8i oUCw== X-Gm-Message-State: AOAM532hqOWoCF2wXoW3IxFAYPgIrmyn+x3IAqSH3F0cllW2++b0VTWc MpDnJTI8lV6irwUmvvoCjVwmWA== X-Google-Smtp-Source: ABdhPJw9luqKLc24vjtBqjzZvm0/teGU0ZGMeaGDdyuIHDvuLAMXiYA2USkVhgRDMEYtHXrjRz3seg== X-Received: by 2002:a37:2c03:: with SMTP id s3mr2285026qkh.83.1640098904994; Tue, 21 Dec 2021 07:01:44 -0800 (PST) Received: from soleen.c.googlers.com.com (189.216.85.34.bc.googleusercontent.com. [34.85.216.189]) by smtp.gmail.com with ESMTPSA id d4sm1991371qkn.79.2021.12.21.07.01.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 21 Dec 2021 07:01:44 -0800 (PST) From: Pasha Tatashin To: pasha.tatashin@soleen.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-m68k@lists.linux-m68k.org, anshuman.khandual@arm.com, willy@infradead.org, akpm@linux-foundation.org, william.kucharski@oracle.com, mike.kravetz@oracle.com, vbabka@suse.cz, geert@linux-m68k.org, schmitzmic@gmail.com, rostedt@goodmis.org, mingo@redhat.com, hannes@cmpxchg.org, guro@fb.com, songmuchun@bytedance.com, weixugc@google.com, gthelen@google.com, rientjes@google.com, pjt@google.com Subject: [PATCH v2 1/9] mm: add overflow and underflow checks for page->_refcount Date: Tue, 21 Dec 2021 15:01:32 +0000 Message-Id: <20211221150140.988298-2-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.34.1.307.g9b7440fafd-goog In-Reply-To: <20211221150140.988298-1-pasha.tatashin@soleen.com> References: <20211221150140.988298-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" The problems with page->_refcount are hard to debug, because usually when they are detected, the damage has occurred a long time ago. Yet, the problems with invalid page refcount may be catastrophic and lead to memory corruptions. Reduce the scope of when the _refcount problems manifest themselves by adding checks for underflows and overflows into functions that modify _refcount. Use atomic_fetch_* functions to get the old values of the _refcount, and use it to check for overflow/underflow. Signed-off-by: Pasha Tatashin --- include/linux/page_ref.h | 59 +++++++++++++++++++++++++++++----------- 1 file changed, 43 insertions(+), 16 deletions(-) diff --git a/include/linux/page_ref.h b/include/linux/page_ref.h index 2e677e6ad09f..fe4864f7f69c 100644 --- a/include/linux/page_ref.h +++ b/include/linux/page_ref.h @@ -117,7 +117,10 @@ static inline void init_page_count(struct page *page) =20 static inline void page_ref_add(struct page *page, int nr) { - atomic_add(nr, &page->_refcount); + int old_val =3D atomic_fetch_add(nr, &page->_refcount); + int new_val =3D old_val + nr; + + VM_BUG_ON_PAGE((unsigned int)new_val < (unsigned int)old_val, page); if (page_ref_tracepoint_active(page_ref_mod)) __page_ref_mod(page, nr); } @@ -129,7 +132,10 @@ static inline void folio_ref_add(struct folio *folio, = int nr) =20 static inline void page_ref_sub(struct page *page, int nr) { - atomic_sub(nr, &page->_refcount); + int old_val =3D atomic_fetch_sub(nr, &page->_refcount); + int new_val =3D old_val - nr; + + VM_BUG_ON_PAGE((unsigned int)new_val > (unsigned int)old_val, page); if (page_ref_tracepoint_active(page_ref_mod)) __page_ref_mod(page, -nr); } @@ -141,11 +147,13 @@ static inline void folio_ref_sub(struct folio *folio,= int nr) =20 static inline int page_ref_sub_return(struct page *page, int nr) { - int ret =3D atomic_sub_return(nr, &page->_refcount); + int old_val =3D atomic_fetch_sub(nr, &page->_refcount); + int new_val =3D old_val - nr; =20 + VM_BUG_ON_PAGE((unsigned int)new_val > (unsigned int)old_val, page); if (page_ref_tracepoint_active(page_ref_mod_and_return)) - __page_ref_mod_and_return(page, -nr, ret); - return ret; + __page_ref_mod_and_return(page, -nr, new_val); + return new_val; } =20 static inline int folio_ref_sub_return(struct folio *folio, int nr) @@ -155,7 +163,10 @@ static inline int folio_ref_sub_return(struct folio *f= olio, int nr) =20 static inline void page_ref_inc(struct page *page) { - atomic_inc(&page->_refcount); + int old_val =3D atomic_fetch_inc(&page->_refcount); + int new_val =3D old_val + 1; + + VM_BUG_ON_PAGE((unsigned int)new_val < (unsigned int)old_val, page); if (page_ref_tracepoint_active(page_ref_mod)) __page_ref_mod(page, 1); } @@ -167,7 +178,10 @@ static inline void folio_ref_inc(struct folio *folio) =20 static inline void page_ref_dec(struct page *page) { - atomic_dec(&page->_refcount); + int old_val =3D atomic_fetch_dec(&page->_refcount); + int new_val =3D old_val - 1; + + VM_BUG_ON_PAGE((unsigned int)new_val > (unsigned int)old_val, page); if (page_ref_tracepoint_active(page_ref_mod)) __page_ref_mod(page, -1); } @@ -179,8 +193,11 @@ static inline void folio_ref_dec(struct folio *folio) =20 static inline int page_ref_sub_and_test(struct page *page, int nr) { - int ret =3D atomic_sub_and_test(nr, &page->_refcount); + int old_val =3D atomic_fetch_sub(nr, &page->_refcount); + int new_val =3D old_val - nr; + int ret =3D new_val =3D=3D 0; =20 + VM_BUG_ON_PAGE((unsigned int)new_val > (unsigned int)old_val, page); if (page_ref_tracepoint_active(page_ref_mod_and_test)) __page_ref_mod_and_test(page, -nr, ret); return ret; @@ -193,11 +210,13 @@ static inline int folio_ref_sub_and_test(struct folio= *folio, int nr) =20 static inline int page_ref_inc_return(struct page *page) { - int ret =3D atomic_inc_return(&page->_refcount); + int old_val =3D atomic_fetch_inc(&page->_refcount); + int new_val =3D old_val + 1; =20 + VM_BUG_ON_PAGE((unsigned int)new_val < (unsigned int)old_val, page); if (page_ref_tracepoint_active(page_ref_mod_and_return)) - __page_ref_mod_and_return(page, 1, ret); - return ret; + __page_ref_mod_and_return(page, 1, new_val); + return new_val; } =20 static inline int folio_ref_inc_return(struct folio *folio) @@ -207,8 +226,11 @@ static inline int folio_ref_inc_return(struct folio *f= olio) =20 static inline int page_ref_dec_and_test(struct page *page) { - int ret =3D atomic_dec_and_test(&page->_refcount); + int old_val =3D atomic_fetch_dec(&page->_refcount); + int new_val =3D old_val - 1; + int ret =3D new_val =3D=3D 0; =20 + VM_BUG_ON_PAGE((unsigned int)new_val > (unsigned int)old_val, page); if (page_ref_tracepoint_active(page_ref_mod_and_test)) __page_ref_mod_and_test(page, -1, ret); return ret; @@ -221,11 +243,13 @@ static inline int folio_ref_dec_and_test(struct folio= *folio) =20 static inline int page_ref_dec_return(struct page *page) { - int ret =3D atomic_dec_return(&page->_refcount); + int old_val =3D atomic_fetch_dec(&page->_refcount); + int new_val =3D old_val - 1; =20 + VM_BUG_ON_PAGE((unsigned int)new_val > (unsigned int)old_val, page); if (page_ref_tracepoint_active(page_ref_mod_and_return)) - __page_ref_mod_and_return(page, -1, ret); - return ret; + __page_ref_mod_and_return(page, -1, new_val); + return new_val; } =20 static inline int folio_ref_dec_return(struct folio *folio) @@ -235,8 +259,11 @@ static inline int folio_ref_dec_return(struct folio *f= olio) =20 static inline bool page_ref_add_unless(struct page *page, int nr, int u) { - bool ret =3D atomic_add_unless(&page->_refcount, nr, u); + int old_val =3D atomic_fetch_add_unless(&page->_refcount, nr, u); + int new_val =3D old_val + nr; + int ret =3D old_val !=3D u; =20 + VM_BUG_ON_PAGE(ret && (unsigned int)new_val < (unsigned int)old_val, page= ); if (page_ref_tracepoint_active(page_ref_mod_unless)) __page_ref_mod_unless(page, nr, ret); return ret; --=20 2.34.1.307.g9b7440fafd-goog From nobody Wed Jul 1 13:27:41 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 01584C433EF for ; Tue, 21 Dec 2021 15:01:55 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238697AbhLUPBy (ORCPT ); Tue, 21 Dec 2021 10:01:54 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53310 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235774AbhLUPBr (ORCPT ); Tue, 21 Dec 2021 10:01:47 -0500 Received: from mail-qk1-x735.google.com (mail-qk1-x735.google.com [IPv6:2607:f8b0:4864:20::735]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EA838C061746 for ; Tue, 21 Dec 2021 07:01:46 -0800 (PST) Received: by mail-qk1-x735.google.com with SMTP id 131so1524439qkk.2 for ; Tue, 21 Dec 2021 07:01:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=8OLKCLK70MdlV7lY3Fm4nJPND3+4SBjM5tfYfr3at+o=; b=J7/VHH0JOS8YGb7cCfiDWVyL9L52RvtQXQMwPLoGbG4THf4d7TZ+nr3GN+lZxvr3gt DR+3zRsmjf8qmyeuY98lOO6DG0APhdsS3T9PbV7zB70pmM2ErA+pPzEESJOwAv5Fi81w KtsaENFiwW8dBAVRiZ7W+I6iah2RetGECM6ejHg3WUYuyUdI9g0vOUEOBGedrKnZPVaQ rSd6Yw4YzULM473hI0xFGj7CPWywVbUSzXVqRK4jNNnUPE9VstMypT1I0KCXBUKyb9WL V/LJtDyszKXavEJ85nP31HaFTBUoUKK4UZeO7tPUy+R2KUqthGLCG7zbzK/iMb56yo+U SmxQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=8OLKCLK70MdlV7lY3Fm4nJPND3+4SBjM5tfYfr3at+o=; b=WmX4rtzoqHvMqM3EUQMzffoFIVgElUIIs/tk9UU4TaYI1JD4hdrgmyJ5+xYVfDXBBI kvEvPEhdkkcebIeL81uzwmcpdBbjPFjseU3HmxoqNvoFtNfBRaKzQWiIAC3DodJonTGP EhPu6y5WQcEmAc/lZb8NlOcNLA5GGwn4VFSboCQM0vWwKc8HQPJerprZ6WgPjGV8xZMk r6tsC3TQk3gwmp25okTu5mWXPr+TV2SuMkh4DAVX2jevkZ/rABBLWptZ1X1qvNuLcSzN tLx4K8udyFXGT+Sr4PJNCfABJqM52tyOQoK8AvOurRSnISBv2RuSdPnjBAC0Odv+3CAa qQ0A== X-Gm-Message-State: AOAM5322ZaEnC1Jnrz/+TIjJ4N3gmxRxiwfo0t2wagNZJVFlKkxXheY4 yTXKVNbm81tMN37Mcv85M/An+Q== X-Google-Smtp-Source: ABdhPJzN9SRSZZItQKCqO558WN2Y7tM5Mh2bBAZZpVvE6EZpGwsiLYjKEfjD3vbJIkTtFd7w2FLsYQ== X-Received: by 2002:a05:620a:c4f:: with SMTP id u15mr2250058qki.565.1640098905805; Tue, 21 Dec 2021 07:01:45 -0800 (PST) Received: from soleen.c.googlers.com.com (189.216.85.34.bc.googleusercontent.com. [34.85.216.189]) by smtp.gmail.com with ESMTPSA id d4sm1991371qkn.79.2021.12.21.07.01.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 21 Dec 2021 07:01:45 -0800 (PST) From: Pasha Tatashin To: pasha.tatashin@soleen.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-m68k@lists.linux-m68k.org, anshuman.khandual@arm.com, willy@infradead.org, akpm@linux-foundation.org, william.kucharski@oracle.com, mike.kravetz@oracle.com, vbabka@suse.cz, geert@linux-m68k.org, schmitzmic@gmail.com, rostedt@goodmis.org, mingo@redhat.com, hannes@cmpxchg.org, guro@fb.com, songmuchun@bytedance.com, weixugc@google.com, gthelen@google.com, rientjes@google.com, pjt@google.com Subject: [PATCH v2 2/9] mm: Avoid using set_page_count() in set_page_recounted() Date: Tue, 21 Dec 2021 15:01:33 +0000 Message-Id: <20211221150140.988298-3-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.34.1.307.g9b7440fafd-goog In-Reply-To: <20211221150140.988298-1-pasha.tatashin@soleen.com> References: <20211221150140.988298-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" set_page_refcounted() converts a non-refcounted page that has (page->_refcount =3D=3D 0) into a refcounted page by setting _refcount to 1. The current apporach uses the following logic: VM_BUG_ON_PAGE(page_ref_count(page), page); set_page_count(page, 1); However, if _refcount changes from 0 to 1 between the VM_BUG_ON_PAGE() and set_page_count() we can break _refcount, which can cause other problems such as memory corruptions. Instead, use a safer method: increment _refcount first and verify that at increment time it was indeed 1. refcnt =3D page_ref_inc_return(page); VM_BUG_ON_PAGE(refcnt !=3D 1, page); Use page_ref_inc_return() to avoid unconditionally overwriting the _refcount value with set_page_count(), and check the return value. Signed-off-by: Pasha Tatashin --- mm/internal.h | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/mm/internal.h b/mm/internal.h index deb9bda18e59..4d45ef2ffea6 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -162,9 +162,11 @@ static inline bool page_evictable(struct page *page) */ static inline void set_page_refcounted(struct page *page) { + int refcnt; + VM_BUG_ON_PAGE(PageTail(page), page); - VM_BUG_ON_PAGE(page_ref_count(page), page); - set_page_count(page, 1); + refcnt =3D page_ref_inc_return(page); + VM_BUG_ON_PAGE(refcnt !=3D 1, page); } =20 extern unsigned long highest_memmap_pfn; --=20 2.34.1.307.g9b7440fafd-goog From nobody Wed Jul 1 13:27:41 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 87DD0C4332F for ; Tue, 21 Dec 2021 15:01:57 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232308AbhLUPB4 (ORCPT ); Tue, 21 Dec 2021 10:01:56 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53320 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235796AbhLUPBs (ORCPT ); Tue, 21 Dec 2021 10:01:48 -0500 Received: from mail-qk1-x72a.google.com (mail-qk1-x72a.google.com [IPv6:2607:f8b0:4864:20::72a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B667AC061747 for ; Tue, 21 Dec 2021 07:01:47 -0800 (PST) Received: by mail-qk1-x72a.google.com with SMTP id 69so265169qkd.6 for ; Tue, 21 Dec 2021 07:01:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=/Bz+RH2sd3d5dh0W3emMfNr+x1w1TZodqurj+f1M1QQ=; b=KidDaTtrQ8tAIOl8DDgN3FOj8v1GQoe8AsMNbfRPu+zCoyTqxLCHTwzt9TbJbGovo0 6Q9nLCCLveS3W+IiLFrcheP6UIKLC/0UIEGvzCdIJEB3tS03dwPcqR5y4Gx65v2xT1pJ 9DSwSVolFxtdTKHj0Ik+YdvPhw6379bJal5U2mQLhnKWK+y+akA/COr5nZyW37G/V5Pt LnhAmG0s96vB1BlwKg3qDz836SVwtKNrueM8zE6fy78U14axejFvjq5Gfq1OVGF5Z/7C Wk9q1LEkgVQYUNYNyzxS2vRVpOaUifCNJ5eGJPO6psx4JzNMhN5x+6NRyXbvV/t0qYjL UO7w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=/Bz+RH2sd3d5dh0W3emMfNr+x1w1TZodqurj+f1M1QQ=; b=TugVr/aQs5XJNCfx3akmZjTAqb/d/q/ahxDQsmt6ELDXl0DL9d0hoyyX5Qlui6IYAk 7ZIMeEq+M7HAc8BR8D0fxfikC5DqoJnJyQLfbj1TSXmC3xuhS9Daj2/ZXuhCXgnD0/6r q78vPMP7skapIydEH+RpfUeW68TGO+z5y8n2JCwX/59LqV1IAA7Z+dPxoDodzRpXD7oq 5Euz0AHIou7BlL4/WLK7eV+WU6UPU+IHPn1cWV6L1L5mizKHM/jVtkG0xw6pRzxs4c4l BSxp7XyDenXDN/GVBodQrYPAP1ko7uW5AKTjEdKhwc34v2ojuTCW0Rf4ceTgeV/X3hRW Krcw== X-Gm-Message-State: AOAM531zGGE32vg9ab3T5Ph239hph3JzHppA3D+pQQLVp7S0+oN+L9WU HJ+l9CRJovtXm1vukJx8mc6ggg== X-Google-Smtp-Source: ABdhPJwWtWwvP9UVXTIMJbtTvAxC6RyjKtLEk/a3kJ8GOYA632y7N+nkICuXAEpwnyyJVHEAouzJYA== X-Received: by 2002:a37:60a:: with SMTP id 10mr1540467qkg.19.1640098906882; Tue, 21 Dec 2021 07:01:46 -0800 (PST) Received: from soleen.c.googlers.com.com (189.216.85.34.bc.googleusercontent.com. [34.85.216.189]) by smtp.gmail.com with ESMTPSA id d4sm1991371qkn.79.2021.12.21.07.01.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 21 Dec 2021 07:01:46 -0800 (PST) From: Pasha Tatashin To: pasha.tatashin@soleen.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-m68k@lists.linux-m68k.org, anshuman.khandual@arm.com, willy@infradead.org, akpm@linux-foundation.org, william.kucharski@oracle.com, mike.kravetz@oracle.com, vbabka@suse.cz, geert@linux-m68k.org, schmitzmic@gmail.com, rostedt@goodmis.org, mingo@redhat.com, hannes@cmpxchg.org, guro@fb.com, songmuchun@bytedance.com, weixugc@google.com, gthelen@google.com, rientjes@google.com, pjt@google.com Subject: [PATCH v2 3/9] mm: remove set_page_count() from page_frag_alloc_align Date: Tue, 21 Dec 2021 15:01:34 +0000 Message-Id: <20211221150140.988298-4-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.34.1.307.g9b7440fafd-goog In-Reply-To: <20211221150140.988298-1-pasha.tatashin@soleen.com> References: <20211221150140.988298-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" set_page_count() unconditionally resets the value of _ref_count and that is dangerous, as it is not programmatically verified. Instead we rely on comments like: "OK, page count is 0, we can safely set it". Add a new refcount function: page_ref_add_return() to return the new refcount value after adding to it. Use the return value to verify that the _ref_count was indeed the expected one. Signed-off-by: Pasha Tatashin --- include/linux/page_ref.h | 11 +++++++++++ mm/page_alloc.c | 6 ++++-- 2 files changed, 15 insertions(+), 2 deletions(-) diff --git a/include/linux/page_ref.h b/include/linux/page_ref.h index fe4864f7f69c..03e21ce2f1bd 100644 --- a/include/linux/page_ref.h +++ b/include/linux/page_ref.h @@ -115,6 +115,17 @@ static inline void init_page_count(struct page *page) set_page_count(page, 1); } =20 +static inline int page_ref_add_return(struct page *page, int nr) +{ + int old_val =3D atomic_fetch_add(nr, &page->_refcount); + int new_val =3D old_val + nr; + + VM_BUG_ON_PAGE((unsigned int)new_val < (unsigned int)old_val, page); + if (page_ref_tracepoint_active(page_ref_mod_and_return)) + __page_ref_mod_and_return(page, nr, new_val); + return new_val; +} + static inline void page_ref_add(struct page *page, int nr) { int old_val =3D atomic_fetch_add(nr, &page->_refcount); diff --git a/mm/page_alloc.c b/mm/page_alloc.c index edfd6c81af82..b5554767b9de 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -5523,6 +5523,7 @@ void *page_frag_alloc_align(struct page_frag_cache *n= c, unsigned int size =3D PAGE_SIZE; struct page *page; int offset; + int refcnt; =20 if (unlikely(!nc->va)) { refill: @@ -5561,8 +5562,9 @@ void *page_frag_alloc_align(struct page_frag_cache *n= c, /* if size can vary use size else just use PAGE_SIZE */ size =3D nc->size; #endif - /* OK, page count is 0, we can safely set it */ - set_page_count(page, PAGE_FRAG_CACHE_MAX_SIZE + 1); + /* page count is 0, set it to PAGE_FRAG_CACHE_MAX_SIZE + 1 */ + refcnt =3D page_ref_add_return(page, PAGE_FRAG_CACHE_MAX_SIZE + 1); + VM_BUG_ON_PAGE(refcnt !=3D PAGE_FRAG_CACHE_MAX_SIZE + 1, page); =20 /* reset page count bias and offset to start of new frag */ nc->pagecnt_bias =3D PAGE_FRAG_CACHE_MAX_SIZE + 1; --=20 2.34.1.307.g9b7440fafd-goog From nobody Wed Jul 1 13:27:41 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id CC991C433EF for ; Tue, 21 Dec 2021 15:02:01 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238809AbhLUPCA (ORCPT ); Tue, 21 Dec 2021 10:02:00 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53332 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235819AbhLUPBu (ORCPT ); Tue, 21 Dec 2021 10:01:50 -0500 Received: from mail-qt1-x835.google.com (mail-qt1-x835.google.com [IPv6:2607:f8b0:4864:20::835]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 91972C061756 for ; Tue, 21 Dec 2021 07:01:48 -0800 (PST) Received: by mail-qt1-x835.google.com with SMTP id l17so603577qtk.7 for ; Tue, 21 Dec 2021 07:01:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=z2VhwZJp+CYNkMvAFgED+Ch239FJE+mqbH+kwAZRtjs=; b=Isqh/qtjAM6CifMsfjDdrpzprBY88bVNe++QqbYGtWKvcFBuirzAI8eSLjED3Yr8q+ K7oDPwFcOSairohgieMo5fJaUSk3ooSZuchaU6ouPnieVyBf+Z59FeGZDN+/F0u1b5Nl GrT5P6lCepWEXlpHRGEpkNm6qhtg9jSC7cSG1N4qf4C6+XirC+wMJAFVTFCPxeHMeONP cVrb23Wr6/HQ33ydNc0n+tMa8LJY0iVIUicZuVRXqOe0MxFIZmGEdMuYvosUMV3neE7k 9Scs7dNJ/vV1gn2dv0CVxKWBB5vpDAPau9zo11fySBO4QlfQCVXtinoeU+NScgXZ/d8T jm0Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=z2VhwZJp+CYNkMvAFgED+Ch239FJE+mqbH+kwAZRtjs=; b=PpYUEDU2pYQ29Y4xjDB1saC742nDW4l2FzFz//p9NWnYvTlEefoxIVhk8cj8GQEA3o Wnn5T8P5r8sRf67EffZEDP17qbLrnxiqDHIWRqpE0P+fIbbUsD4iOP7QbI2qU3jVeAmM wEx5QiJ39CIPF/AlcyptbcwZlrN2m1MxFQMyiL5ta6B0kkodozETWSV2vxTLqe6DHe2h WkiuvtrBSn//3dF8ox9l9VjT8ULC6dGzZGieOaqZWABjieP10ZNnjtikT9wsbK8DRfYR bWnDMgtD+E1BcmFoq4iYyOEm8nij7cprCseTVgUy7PUeS6o9LJk8JLGNyyxLR6QnCS6F dzTA== X-Gm-Message-State: AOAM533tCfNQVLTtkg02qupvJmwBAVxcK2gxsaQ5h58/1QHNWis7B/kq 0oOSEt+zHi2YBVXlt4vZ/sFJ2w== X-Google-Smtp-Source: ABdhPJz6AlBZ8QhXsFxyNCcVEnd/n/MyhBcR1fXsh3p0kndyZdbgogIw17CApmd9zqCEKn07vnlowQ== X-Received: by 2002:ac8:5a51:: with SMTP id o17mr2530376qta.180.1640098907723; Tue, 21 Dec 2021 07:01:47 -0800 (PST) Received: from soleen.c.googlers.com.com (189.216.85.34.bc.googleusercontent.com. [34.85.216.189]) by smtp.gmail.com with ESMTPSA id d4sm1991371qkn.79.2021.12.21.07.01.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 21 Dec 2021 07:01:47 -0800 (PST) From: Pasha Tatashin To: pasha.tatashin@soleen.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-m68k@lists.linux-m68k.org, anshuman.khandual@arm.com, willy@infradead.org, akpm@linux-foundation.org, william.kucharski@oracle.com, mike.kravetz@oracle.com, vbabka@suse.cz, geert@linux-m68k.org, schmitzmic@gmail.com, rostedt@goodmis.org, mingo@redhat.com, hannes@cmpxchg.org, guro@fb.com, songmuchun@bytedance.com, weixugc@google.com, gthelen@google.com, rientjes@google.com, pjt@google.com Subject: [PATCH v2 4/9] mm: avoid using set_page_count() when pages are freed into allocator Date: Tue, 21 Dec 2021 15:01:35 +0000 Message-Id: <20211221150140.988298-5-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.34.1.307.g9b7440fafd-goog In-Reply-To: <20211221150140.988298-1-pasha.tatashin@soleen.com> References: <20211221150140.988298-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" When struct pages are first initialized the page->_refcount field is set 1. However, later when pages are freed into allocator we set _refcount to 0 via set_page_count(). Unconditionally resetting _refcount is dangerous. Instead use page_ref_dec_return(), and verify that the _refcount is what is expected. Signed-off-by: Pasha Tatashin --- mm/page_alloc.c | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index b5554767b9de..13d989d62012 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -1667,6 +1667,7 @@ void __free_pages_core(struct page *page, unsigned in= t order) unsigned int nr_pages =3D 1 << order; struct page *p =3D page; unsigned int loop; + int refcnt; =20 /* * When initializing the memmap, __init_single_page() sets the refcount @@ -1677,10 +1678,12 @@ void __free_pages_core(struct page *page, unsigned = int order) for (loop =3D 0; loop < (nr_pages - 1); loop++, p++) { prefetchw(p + 1); __ClearPageReserved(p); - set_page_count(p, 0); + refcnt =3D page_ref_dec_return(p); + VM_BUG_ON_PAGE(refcnt, p); } __ClearPageReserved(p); - set_page_count(p, 0); + refcnt =3D page_ref_dec_return(p); + VM_BUG_ON_PAGE(refcnt, p); =20 atomic_long_add(nr_pages, &page_zone(page)->managed_pages); =20 @@ -2252,10 +2255,12 @@ void __init init_cma_reserved_pageblock(struct page= *page) { unsigned i =3D pageblock_nr_pages; struct page *p =3D page; + int refcnt; =20 do { __ClearPageReserved(p); - set_page_count(p, 0); + refcnt =3D page_ref_dec_return(p); + VM_BUG_ON_PAGE(refcnt, p); } while (++p, --i); =20 set_pageblock_migratetype(page, MIGRATE_CMA); --=20 2.34.1.307.g9b7440fafd-goog From nobody Wed Jul 1 13:27:41 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9E294C4321E for ; Tue, 21 Dec 2021 15:01:59 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238803AbhLUPB6 (ORCPT ); Tue, 21 Dec 2021 10:01:58 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53334 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235821AbhLUPBu (ORCPT ); Tue, 21 Dec 2021 10:01:50 -0500 Received: from mail-qk1-x735.google.com (mail-qk1-x735.google.com [IPv6:2607:f8b0:4864:20::735]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7E19AC061574 for ; Tue, 21 Dec 2021 07:01:49 -0800 (PST) Received: by mail-qk1-x735.google.com with SMTP id m186so12784692qkb.4 for ; Tue, 21 Dec 2021 07:01:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=SsQjilbSmtOxN4/+kaclrIIaX0+LAsp7vHqyKiKMlRY=; b=gcBvZoUtrYF3NYCTKHXzdceQiMgG3aqp8QQC/1YCPMu//H6PPmBh6cbnqiASj5UIOO 2wJEeGwU4eOhh4IP+XQ974yhSApwdUidA9SYQadh0fRVAuP4IbUKVo6lqjggb2oOK5lq DM9iFM+R8v7OAN4Xa6Lb1xPTsXCEgta3V/iUyRbPzgcSGntUN2RXWivv3j6pSrJn7CpH Pmj2c2F8EET1EyM0FQ3c/4EfrGh5jpd71L68gnn3IL+M+OT4a47k1HGc4YJWxVgW0tYd njGijwXT7U5gG64oLXvjZ4CFri64I8WnnI91/PXtSVFrpKzw5l/A1IG64P1bXl6rX+p8 uPcQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=SsQjilbSmtOxN4/+kaclrIIaX0+LAsp7vHqyKiKMlRY=; b=vbISf+j1MEqF9HyezZDV4oSvHtGQhWcXptQiuRoz+XSUB+93/ICHDgEZtfS9LcHi7L cPD+/8E4LMDiMtjGN7x3D1j70OMrdlggWan8jwHrW0JqQvWx8k9CWSBaZVIXnEhEzFT8 UfQ9fv1KDKGrQKv1XwOYoB1BNxi1irIDh9cvAFq5GXat8efs4pvqjF2LLqwpi1Dq+kSJ 8e9CFYu8rG0qD4deJtZbbNpJI9KZPjyyBZB9S04B0eadprjWEhMwIAEt8RP6e3T8J/NQ ZoH1+FguSykuUJ0R4zIWy5qUf3qYoFSlaiKZShcsYuFVYdPXSr7UiKy6PAAVmGWo0ZW8 db3Q== X-Gm-Message-State: AOAM530TwNr+Qvnfnaf1rNdbC14tejSQFIuteSGOPuKXnOoEYty0sFkE iJul6+u8um/zxa/1PKlUD4ehGw== X-Google-Smtp-Source: ABdhPJzRSE2PQv8kZTNTCLI9TYbronVdzGDNuPGOGxLks2rlUw8onAlSVNgM6PFYon6DnS7EEo9SMg== X-Received: by 2002:a05:620a:2989:: with SMTP id r9mr2208491qkp.630.1640098908598; Tue, 21 Dec 2021 07:01:48 -0800 (PST) Received: from soleen.c.googlers.com.com (189.216.85.34.bc.googleusercontent.com. [34.85.216.189]) by smtp.gmail.com with ESMTPSA id d4sm1991371qkn.79.2021.12.21.07.01.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 21 Dec 2021 07:01:48 -0800 (PST) From: Pasha Tatashin To: pasha.tatashin@soleen.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-m68k@lists.linux-m68k.org, anshuman.khandual@arm.com, willy@infradead.org, akpm@linux-foundation.org, william.kucharski@oracle.com, mike.kravetz@oracle.com, vbabka@suse.cz, geert@linux-m68k.org, schmitzmic@gmail.com, rostedt@goodmis.org, mingo@redhat.com, hannes@cmpxchg.org, guro@fb.com, songmuchun@bytedance.com, weixugc@google.com, gthelen@google.com, rientjes@google.com, pjt@google.com Subject: [PATCH v2 5/9] mm: rename init_page_count() -> page_ref_init() Date: Tue, 21 Dec 2021 15:01:36 +0000 Message-Id: <20211221150140.988298-6-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.34.1.307.g9b7440fafd-goog In-Reply-To: <20211221150140.988298-1-pasha.tatashin@soleen.com> References: <20211221150140.988298-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Now, that set_page_count() is not called from outside anymore and about to be removed, init_page_count() is the only function that is going to be used to unconditionally set _refcount, however it is restricted to set it only to 1. Make init_page_count() aligned with the other page_ref_* functions by renaming it. Signed-off-by: Pasha Tatashin Acked-by: Geert Uytterhoeven --- arch/m68k/mm/motorola.c | 2 +- include/linux/mm.h | 2 +- include/linux/page_ref.h | 10 +++++++--- mm/page_alloc.c | 2 +- 4 files changed, 10 insertions(+), 6 deletions(-) diff --git a/arch/m68k/mm/motorola.c b/arch/m68k/mm/motorola.c index ecbe948f4c1a..dd3b77d03d5c 100644 --- a/arch/m68k/mm/motorola.c +++ b/arch/m68k/mm/motorola.c @@ -133,7 +133,7 @@ void __init init_pointer_table(void *table, int type) =20 /* unreserve the page so it's possible to free that page */ __ClearPageReserved(PD_PAGE(dp)); - init_page_count(PD_PAGE(dp)); + page_ref_init(PD_PAGE(dp)); =20 return; } diff --git a/include/linux/mm.h b/include/linux/mm.h index d211a06784d5..fae3b6ef66a5 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2451,7 +2451,7 @@ extern void reserve_bootmem_region(phys_addr_t start,= phys_addr_t end); static inline void free_reserved_page(struct page *page) { ClearPageReserved(page); - init_page_count(page); + page_ref_init(page); __free_page(page); adjust_managed_page_count(page, 1); } diff --git a/include/linux/page_ref.h b/include/linux/page_ref.h index 03e21ce2f1bd..1af12a0d7ba1 100644 --- a/include/linux/page_ref.h +++ b/include/linux/page_ref.h @@ -107,10 +107,14 @@ static inline void folio_set_count(struct folio *foli= o, int v) } =20 /* - * Setup the page count before being freed into the page allocator for - * the first time (boot or memory hotplug) + * Setup the page refcount to one before being freed into the page allocat= or. + * The memory might not be initialized and therefore there cannot be any + * assumptions about the current value of page->_refcount. This call shoul= d be + * done during boot when memory is being initialized, during memory hotplug + * when new memory is added, or when a previous reserved memory is unreser= ved + * this is the first time kernel take control of the given memory. */ -static inline void init_page_count(struct page *page) +static inline void page_ref_init(struct page *page) { set_page_count(page, 1); } diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 13d989d62012..000c057a2d24 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -1569,7 +1569,7 @@ static void __meminit __init_single_page(struct page = *page, unsigned long pfn, { mm_zero_struct_page(page); set_page_links(page, zone, nid, pfn); - init_page_count(page); + page_ref_init(page); page_mapcount_reset(page); page_cpupid_reset_last(page); page_kasan_tag_reset(page); --=20 2.34.1.307.g9b7440fafd-goog From nobody Wed Jul 1 13:27:41 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E4F3DC433FE for ; Tue, 21 Dec 2021 15:02:03 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238778AbhLUPCD (ORCPT ); Tue, 21 Dec 2021 10:02:03 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53348 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235930AbhLUPBv (ORCPT ); Tue, 21 Dec 2021 10:01:51 -0500 Received: from mail-qk1-x72e.google.com (mail-qk1-x72e.google.com [IPv6:2607:f8b0:4864:20::72e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 848BFC06173F for ; Tue, 21 Dec 2021 07:01:50 -0800 (PST) Received: by mail-qk1-x72e.google.com with SMTP id t6so12792834qkg.1 for ; Tue, 21 Dec 2021 07:01:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=3o2eHohDdMg61Zg9MpGxB9pEXtCaHvT+otQKTxX/uSo=; b=iuzU+b7NbW+JYMr1V/VFN1cbVqPFRX5aWHIoVZDiIgnseTjlV3RZHt4inZvvgmNRiS 01ywWKPt4olBDuMB/G1vZsXnq/5JNtN4cFf8rNAE9i2LIfxY8HvWDs3HsMMlzj5yggBf HSVm+l9Bs5QEbBB3MPZJ3jXuWn97SEV6rFoB7RSNQB1liZdXprh5UII555VqXNchYERw pCJRILLIx2kG3kjiUNQoY5L80OVihCOmxL+8yCEyIfKzyECxruuJtip8pYmKzD68tVYb osDvRwFMs5lrz2LGtgcutefqKs0k+tqH4+ohnMQJh2FuFy5z6Rv12pl6VXlcLzmpsd/4 EBNQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=3o2eHohDdMg61Zg9MpGxB9pEXtCaHvT+otQKTxX/uSo=; b=41I87rRxN9c6w9aOxEv+l/qhDF8ib5erHlo0WyVnyP+3RU3QGuG2D5IRMRe76tqwTo siUhfSlmdOFYREwg0lJwL/fJHELFlqRPyNLyXcO+Z4Yz8eZlh8vLBMmpXHuxodI90qJw 4RLaocCITb/4MuU3LoQukuAZJqyT9vk91GY6oty0gRJz1JiFutTYBb1/tMHVvKzs7GZ9 Wd4rh087pZPcX744bNvk9K7gfeyeW6OLwrDmKfg0Bwq3ycNNjE3307FG6/m9qhIjy0s5 tBa03kVCWKMY/r75fF7ryWsadU5ralERipKd8s0/oN8DvkmOqQ2Lzm9ab3Svhz9c13rE hqnw== X-Gm-Message-State: AOAM532nwqEiXz5orFgCb9+NNPr+LgKb1MSqq+D72rCIF3lJ44sbC4He i5GP6ZHlHYa38Me4VqpiqTbI6A== X-Google-Smtp-Source: ABdhPJzlLOcPJEIg41oZHOlIl3LtbdhfBgCsn83TjJQh/PpUyX+gZqSSqFtrvHTI7G/6ZdMWfpiZ0w== X-Received: by 2002:a05:620a:710:: with SMTP id 16mr2197192qkc.379.1640098909644; Tue, 21 Dec 2021 07:01:49 -0800 (PST) Received: from soleen.c.googlers.com.com (189.216.85.34.bc.googleusercontent.com. [34.85.216.189]) by smtp.gmail.com with ESMTPSA id d4sm1991371qkn.79.2021.12.21.07.01.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 21 Dec 2021 07:01:49 -0800 (PST) From: Pasha Tatashin To: pasha.tatashin@soleen.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-m68k@lists.linux-m68k.org, anshuman.khandual@arm.com, willy@infradead.org, akpm@linux-foundation.org, william.kucharski@oracle.com, mike.kravetz@oracle.com, vbabka@suse.cz, geert@linux-m68k.org, schmitzmic@gmail.com, rostedt@goodmis.org, mingo@redhat.com, hannes@cmpxchg.org, guro@fb.com, songmuchun@bytedance.com, weixugc@google.com, gthelen@google.com, rientjes@google.com, pjt@google.com Subject: [PATCH v2 6/9] mm: remove set_page_count() Date: Tue, 21 Dec 2021 15:01:37 +0000 Message-Id: <20211221150140.988298-7-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.34.1.307.g9b7440fafd-goog In-Reply-To: <20211221150140.988298-1-pasha.tatashin@soleen.com> References: <20211221150140.988298-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" set_page_count() is dangerous because it resets _refcount to an arbitrary value. Instead we now initialize _refcount to 1 only once, and the rest of the time we are using add/dec/cmpxchg to have a contiguous track of the counter. Remove set_page_count() and add new tracing hooks to page_ref_init(). Signed-off-by: Pasha Tatashin --- include/linux/page_ref.h | 27 ++++++++----------- include/trace/events/page_ref.h | 46 ++++++++++++++++++++++++++++----- mm/debug_page_ref.c | 8 +++--- 3 files changed, 54 insertions(+), 27 deletions(-) diff --git a/include/linux/page_ref.h b/include/linux/page_ref.h index 1af12a0d7ba1..d7316881626c 100644 --- a/include/linux/page_ref.h +++ b/include/linux/page_ref.h @@ -7,7 +7,7 @@ #include #include =20 -DECLARE_TRACEPOINT(page_ref_set); +DECLARE_TRACEPOINT(page_ref_init); DECLARE_TRACEPOINT(page_ref_mod); DECLARE_TRACEPOINT(page_ref_mod_and_test); DECLARE_TRACEPOINT(page_ref_mod_and_return); @@ -26,7 +26,7 @@ DECLARE_TRACEPOINT(page_ref_unfreeze); */ #define page_ref_tracepoint_active(t) tracepoint_enabled(t) =20 -extern void __page_ref_set(struct page *page, int v); +extern void __page_ref_init(struct page *page); extern void __page_ref_mod(struct page *page, int v); extern void __page_ref_mod_and_test(struct page *page, int v, int ret); extern void __page_ref_mod_and_return(struct page *page, int v, int ret); @@ -38,7 +38,7 @@ extern void __page_ref_unfreeze(struct page *page, int v); =20 #define page_ref_tracepoint_active(t) false =20 -static inline void __page_ref_set(struct page *page, int v) +static inline void __page_ref_init(struct page *page) { } static inline void __page_ref_mod(struct page *page, int v) @@ -94,18 +94,6 @@ static inline int page_count(const struct page *page) return folio_ref_count(page_folio(page)); } =20 -static inline void set_page_count(struct page *page, int v) -{ - atomic_set(&page->_refcount, v); - if (page_ref_tracepoint_active(page_ref_set)) - __page_ref_set(page, v); -} - -static inline void folio_set_count(struct folio *folio, int v) -{ - set_page_count(&folio->page, v); -} - /* * Setup the page refcount to one before being freed into the page allocat= or. * The memory might not be initialized and therefore there cannot be any @@ -116,7 +104,14 @@ static inline void folio_set_count(struct folio *folio= , int v) */ static inline void page_ref_init(struct page *page) { - set_page_count(page, 1); + atomic_set(&page->_refcount, 1); + if (page_ref_tracepoint_active(page_ref_init)) + __page_ref_init(page); +} + +static inline void folio_ref_init(struct folio *folio) +{ + page_ref_init(&folio->page); } =20 static inline int page_ref_add_return(struct page *page, int nr) diff --git a/include/trace/events/page_ref.h b/include/trace/events/page_re= f.h index 8a99c1cd417b..87551bb1df9e 100644 --- a/include/trace/events/page_ref.h +++ b/include/trace/events/page_ref.h @@ -10,6 +10,45 @@ #include #include =20 +DECLARE_EVENT_CLASS(page_ref_init_template, + + TP_PROTO(struct page *page), + + TP_ARGS(page), + + TP_STRUCT__entry( + __field(unsigned long, pfn) + __field(unsigned long, flags) + __field(int, count) + __field(int, mapcount) + __field(void *, mapping) + __field(int, mt) + __field(int, val) + ), + + TP_fast_assign( + __entry->pfn =3D page_to_pfn(page); + __entry->flags =3D page->flags; + __entry->count =3D page_ref_count(page); + __entry->mapcount =3D page_mapcount(page); + __entry->mapping =3D page->mapping; + __entry->mt =3D get_pageblock_migratetype(page); + ), + + TP_printk("pfn=3D0x%lx flags=3D%s count=3D%d mapcount=3D%d mapping=3D%p m= t=3D%d", + __entry->pfn, + show_page_flags(__entry->flags & PAGEFLAGS_MASK), + __entry->count, + __entry->mapcount, __entry->mapping, __entry->mt) +); + +DEFINE_EVENT(page_ref_init_template, page_ref_init, + + TP_PROTO(struct page *page), + + TP_ARGS(page) +); + DECLARE_EVENT_CLASS(page_ref_mod_template, =20 TP_PROTO(struct page *page, int v), @@ -44,13 +83,6 @@ DECLARE_EVENT_CLASS(page_ref_mod_template, __entry->val) ); =20 -DEFINE_EVENT(page_ref_mod_template, page_ref_set, - - TP_PROTO(struct page *page, int v), - - TP_ARGS(page, v) -); - DEFINE_EVENT(page_ref_mod_template, page_ref_mod, =20 TP_PROTO(struct page *page, int v), diff --git a/mm/debug_page_ref.c b/mm/debug_page_ref.c index f3b2c9d3ece2..e32149734122 100644 --- a/mm/debug_page_ref.c +++ b/mm/debug_page_ref.c @@ -5,12 +5,12 @@ #define CREATE_TRACE_POINTS #include =20 -void __page_ref_set(struct page *page, int v) +void __page_ref_init(struct page *page) { - trace_page_ref_set(page, v); + trace_page_ref_init(page); } -EXPORT_SYMBOL(__page_ref_set); -EXPORT_TRACEPOINT_SYMBOL(page_ref_set); +EXPORT_SYMBOL(__page_ref_init); +EXPORT_TRACEPOINT_SYMBOL(page_ref_init); =20 void __page_ref_mod(struct page *page, int v) { --=20 2.34.1.307.g9b7440fafd-goog From nobody Wed Jul 1 13:27:41 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7104FC433F5 for ; Tue, 21 Dec 2021 15:02:05 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238812AbhLUPCE (ORCPT ); Tue, 21 Dec 2021 10:02:04 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53356 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235909AbhLUPBx (ORCPT ); Tue, 21 Dec 2021 10:01:53 -0500 Received: from mail-qv1-xf2c.google.com (mail-qv1-xf2c.google.com [IPv6:2607:f8b0:4864:20::f2c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BE483C061747 for ; Tue, 21 Dec 2021 07:01:52 -0800 (PST) Received: by mail-qv1-xf2c.google.com with SMTP id fo11so12669196qvb.4 for ; Tue, 21 Dec 2021 07:01:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=jaGLmQY52APl8v98fQta2llNqqHuAoOAllJQBEPRHxk=; b=eBNsLd7ARojSEaIKr1fNBnwoyaIGKyZ1YHAKdIfcl9DOJfyYTkckGPZbs6qzYnm8KF UJGzZZh8frzcL5dLuDfj2s2helG8nS8rsopxSfDzz9tw/cLdXPv2mCjPvrstIh/YQPYO MbaoqWmM6lsBlKCaN1uZQkuIZ41kY/Gcg1uwL/9zkiAhhYTohOSNbbZBjIn2BSzk9Nm0 q/s7BnOn/guBwmkQKFZUpYCqMI3XtWH6SDbIOS1QqygMsL4z0JmE0xbqcgBZAaN02AU7 azLIyYmAUcTHEF5X85XZmwKKXfDKkG6YkmrdODo9qNwJSKd0aZg2I6SaJzvz7h66dyAp JdJw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=jaGLmQY52APl8v98fQta2llNqqHuAoOAllJQBEPRHxk=; b=nxmwcr2RRv8oPLisMfkYz8gPmWQ3D+wGB1T5w5SlTq+IUFhcqX9aXJDX+9B0NN9HEi +BcGwmnqffBLnTao5T/MUOdMLYIFIX6ElhmF4GTla2Dn+GT4lh7mxgoxM006/phaB+n+ WxH3LRUhJQ0bF6YwVTAkKaslC03Ewtb/LXCsk6nhXm/Vqv/bIqew/C0mxAhl8UVMOj4S FfaeuOUV1K8p5owG3z5JNOdB7CONUbGibLS5/BuCzu5iSVTUSJdPP8x+Rwn2LUPJnyGC UabDF5pnj7B5JIXHkxrioErybDBltvieRKJZ5MbKyFcXg2cjNfD779ODUJemv1uQcqJt tFFw== X-Gm-Message-State: AOAM531TAkZjTYaJmqNVfHoxeYxkZkt73px6mOvUJ/YG58yAcj6Qe/oI +TmGlvNVyhxw3QJR3Z2+rf1w/g== X-Google-Smtp-Source: ABdhPJwMlFRiSYhQumZA31hM2R0ePuuAlQ2ZhyLXodIAC4YlYRRVox4HfZF1uSX5tXZr7Hd0UOKJ3A== X-Received: by 2002:a0c:8e08:: with SMTP id v8mr2635635qvb.93.1640098910549; Tue, 21 Dec 2021 07:01:50 -0800 (PST) Received: from soleen.c.googlers.com.com (189.216.85.34.bc.googleusercontent.com. [34.85.216.189]) by smtp.gmail.com with ESMTPSA id d4sm1991371qkn.79.2021.12.21.07.01.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 21 Dec 2021 07:01:50 -0800 (PST) From: Pasha Tatashin To: pasha.tatashin@soleen.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-m68k@lists.linux-m68k.org, anshuman.khandual@arm.com, willy@infradead.org, akpm@linux-foundation.org, william.kucharski@oracle.com, mike.kravetz@oracle.com, vbabka@suse.cz, geert@linux-m68k.org, schmitzmic@gmail.com, rostedt@goodmis.org, mingo@redhat.com, hannes@cmpxchg.org, guro@fb.com, songmuchun@bytedance.com, weixugc@google.com, gthelen@google.com, rientjes@google.com, pjt@google.com Subject: [PATCH v2 7/9] mm: simplify page_ref_* functions Date: Tue, 21 Dec 2021 15:01:38 +0000 Message-Id: <20211221150140.988298-8-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.34.1.307.g9b7440fafd-goog In-Reply-To: <20211221150140.988298-1-pasha.tatashin@soleen.com> References: <20211221150140.988298-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Now, that we are using atomic_fetch* variants to add/sub/inc/dec page _refcount, it makes sense to combined page_ref_* return and non return functions. Also remove some extra trace points for non-return variants. This improves the tracability by always recording the new _refcount value after the modifications has occurred. Signed-off-by: Pasha Tatashin --- include/linux/page_ref.h | 102 +++++++++----------------------- include/trace/events/page_ref.h | 18 +----- mm/debug_page_ref.c | 14 ----- 3 files changed, 31 insertions(+), 103 deletions(-) diff --git a/include/linux/page_ref.h b/include/linux/page_ref.h index d7316881626c..243fc60ae6c8 100644 --- a/include/linux/page_ref.h +++ b/include/linux/page_ref.h @@ -8,8 +8,6 @@ #include =20 DECLARE_TRACEPOINT(page_ref_init); -DECLARE_TRACEPOINT(page_ref_mod); -DECLARE_TRACEPOINT(page_ref_mod_and_test); DECLARE_TRACEPOINT(page_ref_mod_and_return); DECLARE_TRACEPOINT(page_ref_mod_unless); DECLARE_TRACEPOINT(page_ref_freeze); @@ -27,8 +25,6 @@ DECLARE_TRACEPOINT(page_ref_unfreeze); #define page_ref_tracepoint_active(t) tracepoint_enabled(t) =20 extern void __page_ref_init(struct page *page); -extern void __page_ref_mod(struct page *page, int v); -extern void __page_ref_mod_and_test(struct page *page, int v, int ret); extern void __page_ref_mod_and_return(struct page *page, int v, int ret); extern void __page_ref_mod_unless(struct page *page, int v, int u); extern void __page_ref_freeze(struct page *page, int v, int ret); @@ -41,12 +37,6 @@ extern void __page_ref_unfreeze(struct page *page, int v= ); static inline void __page_ref_init(struct page *page) { } -static inline void __page_ref_mod(struct page *page, int v) -{ -} -static inline void __page_ref_mod_and_test(struct page *page, int v, int r= et) -{ -} static inline void __page_ref_mod_and_return(struct page *page, int v, int= ret) { } @@ -127,12 +117,7 @@ static inline int page_ref_add_return(struct page *pag= e, int nr) =20 static inline void page_ref_add(struct page *page, int nr) { - int old_val =3D atomic_fetch_add(nr, &page->_refcount); - int new_val =3D old_val + nr; - - VM_BUG_ON_PAGE((unsigned int)new_val < (unsigned int)old_val, page); - if (page_ref_tracepoint_active(page_ref_mod)) - __page_ref_mod(page, nr); + page_ref_add_return(page, nr); } =20 static inline void folio_ref_add(struct folio *folio, int nr) @@ -140,30 +125,25 @@ static inline void folio_ref_add(struct folio *folio,= int nr) page_ref_add(&folio->page, nr); } =20 -static inline void page_ref_sub(struct page *page, int nr) +static inline int page_ref_sub_return(struct page *page, int nr) { int old_val =3D atomic_fetch_sub(nr, &page->_refcount); int new_val =3D old_val - nr; =20 VM_BUG_ON_PAGE((unsigned int)new_val > (unsigned int)old_val, page); - if (page_ref_tracepoint_active(page_ref_mod)) - __page_ref_mod(page, -nr); + if (page_ref_tracepoint_active(page_ref_mod_and_return)) + __page_ref_mod_and_return(page, -nr, new_val); + return new_val; } =20 -static inline void folio_ref_sub(struct folio *folio, int nr) +static inline void page_ref_sub(struct page *page, int nr) { - page_ref_sub(&folio->page, nr); + page_ref_sub_return(page, nr); } =20 -static inline int page_ref_sub_return(struct page *page, int nr) +static inline void folio_ref_sub(struct folio *folio, int nr) { - int old_val =3D atomic_fetch_sub(nr, &page->_refcount); - int new_val =3D old_val - nr; - - VM_BUG_ON_PAGE((unsigned int)new_val > (unsigned int)old_val, page); - if (page_ref_tracepoint_active(page_ref_mod_and_return)) - __page_ref_mod_and_return(page, -nr, new_val); - return new_val; + page_ref_sub(&folio->page, nr); } =20 static inline int folio_ref_sub_return(struct folio *folio, int nr) @@ -171,14 +151,20 @@ static inline int folio_ref_sub_return(struct folio *= folio, int nr) return page_ref_sub_return(&folio->page, nr); } =20 -static inline void page_ref_inc(struct page *page) +static inline int page_ref_inc_return(struct page *page) { int old_val =3D atomic_fetch_inc(&page->_refcount); int new_val =3D old_val + 1; =20 VM_BUG_ON_PAGE((unsigned int)new_val < (unsigned int)old_val, page); - if (page_ref_tracepoint_active(page_ref_mod)) - __page_ref_mod(page, 1); + if (page_ref_tracepoint_active(page_ref_mod_and_return)) + __page_ref_mod_and_return(page, 1, new_val); + return new_val; +} + +static inline void page_ref_inc(struct page *page) +{ + page_ref_inc_return(page); } =20 static inline void folio_ref_inc(struct folio *folio) @@ -186,14 +172,20 @@ static inline void folio_ref_inc(struct folio *folio) page_ref_inc(&folio->page); } =20 -static inline void page_ref_dec(struct page *page) +static inline int page_ref_dec_return(struct page *page) { int old_val =3D atomic_fetch_dec(&page->_refcount); int new_val =3D old_val - 1; =20 VM_BUG_ON_PAGE((unsigned int)new_val > (unsigned int)old_val, page); - if (page_ref_tracepoint_active(page_ref_mod)) - __page_ref_mod(page, -1); + if (page_ref_tracepoint_active(page_ref_mod_and_return)) + __page_ref_mod_and_return(page, -1, new_val); + return new_val; +} + +static inline void page_ref_dec(struct page *page) +{ + page_ref_dec_return(page); } =20 static inline void folio_ref_dec(struct folio *folio) @@ -203,14 +195,7 @@ static inline void folio_ref_dec(struct folio *folio) =20 static inline int page_ref_sub_and_test(struct page *page, int nr) { - int old_val =3D atomic_fetch_sub(nr, &page->_refcount); - int new_val =3D old_val - nr; - int ret =3D new_val =3D=3D 0; - - VM_BUG_ON_PAGE((unsigned int)new_val > (unsigned int)old_val, page); - if (page_ref_tracepoint_active(page_ref_mod_and_test)) - __page_ref_mod_and_test(page, -nr, ret); - return ret; + return page_ref_sub_return(page, nr) =3D=3D 0; } =20 static inline int folio_ref_sub_and_test(struct folio *folio, int nr) @@ -218,17 +203,6 @@ static inline int folio_ref_sub_and_test(struct folio = *folio, int nr) return page_ref_sub_and_test(&folio->page, nr); } =20 -static inline int page_ref_inc_return(struct page *page) -{ - int old_val =3D atomic_fetch_inc(&page->_refcount); - int new_val =3D old_val + 1; - - VM_BUG_ON_PAGE((unsigned int)new_val < (unsigned int)old_val, page); - if (page_ref_tracepoint_active(page_ref_mod_and_return)) - __page_ref_mod_and_return(page, 1, new_val); - return new_val; -} - static inline int folio_ref_inc_return(struct folio *folio) { return page_ref_inc_return(&folio->page); @@ -236,14 +210,7 @@ static inline int folio_ref_inc_return(struct folio *f= olio) =20 static inline int page_ref_dec_and_test(struct page *page) { - int old_val =3D atomic_fetch_dec(&page->_refcount); - int new_val =3D old_val - 1; - int ret =3D new_val =3D=3D 0; - - VM_BUG_ON_PAGE((unsigned int)new_val > (unsigned int)old_val, page); - if (page_ref_tracepoint_active(page_ref_mod_and_test)) - __page_ref_mod_and_test(page, -1, ret); - return ret; + return page_ref_dec_return(page) =3D=3D 0; } =20 static inline int folio_ref_dec_and_test(struct folio *folio) @@ -251,17 +218,6 @@ static inline int folio_ref_dec_and_test(struct folio = *folio) return page_ref_dec_and_test(&folio->page); } =20 -static inline int page_ref_dec_return(struct page *page) -{ - int old_val =3D atomic_fetch_dec(&page->_refcount); - int new_val =3D old_val - 1; - - VM_BUG_ON_PAGE((unsigned int)new_val > (unsigned int)old_val, page); - if (page_ref_tracepoint_active(page_ref_mod_and_return)) - __page_ref_mod_and_return(page, -1, new_val); - return new_val; -} - static inline int folio_ref_dec_return(struct folio *folio) { return page_ref_dec_return(&folio->page); diff --git a/include/trace/events/page_ref.h b/include/trace/events/page_re= f.h index 87551bb1df9e..35cd795aa7c6 100644 --- a/include/trace/events/page_ref.h +++ b/include/trace/events/page_ref.h @@ -49,7 +49,7 @@ DEFINE_EVENT(page_ref_init_template, page_ref_init, TP_ARGS(page) ); =20 -DECLARE_EVENT_CLASS(page_ref_mod_template, +DECLARE_EVENT_CLASS(page_ref_unfreeze_template, =20 TP_PROTO(struct page *page, int v), =20 @@ -83,13 +83,6 @@ DECLARE_EVENT_CLASS(page_ref_mod_template, __entry->val) ); =20 -DEFINE_EVENT(page_ref_mod_template, page_ref_mod, - - TP_PROTO(struct page *page, int v), - - TP_ARGS(page, v) -); - DECLARE_EVENT_CLASS(page_ref_mod_and_test_template, =20 TP_PROTO(struct page *page, int v, int ret), @@ -126,13 +119,6 @@ DECLARE_EVENT_CLASS(page_ref_mod_and_test_template, __entry->val, __entry->ret) ); =20 -DEFINE_EVENT(page_ref_mod_and_test_template, page_ref_mod_and_test, - - TP_PROTO(struct page *page, int v, int ret), - - TP_ARGS(page, v, ret) -); - DEFINE_EVENT(page_ref_mod_and_test_template, page_ref_mod_and_return, =20 TP_PROTO(struct page *page, int v, int ret), @@ -154,7 +140,7 @@ DEFINE_EVENT(page_ref_mod_and_test_template, page_ref_f= reeze, TP_ARGS(page, v, ret) ); =20 -DEFINE_EVENT(page_ref_mod_template, page_ref_unfreeze, +DEFINE_EVENT(page_ref_unfreeze_template, page_ref_unfreeze, =20 TP_PROTO(struct page *page, int v), =20 diff --git a/mm/debug_page_ref.c b/mm/debug_page_ref.c index e32149734122..1de9d93cca25 100644 --- a/mm/debug_page_ref.c +++ b/mm/debug_page_ref.c @@ -12,20 +12,6 @@ void __page_ref_init(struct page *page) EXPORT_SYMBOL(__page_ref_init); EXPORT_TRACEPOINT_SYMBOL(page_ref_init); =20 -void __page_ref_mod(struct page *page, int v) -{ - trace_page_ref_mod(page, v); -} -EXPORT_SYMBOL(__page_ref_mod); -EXPORT_TRACEPOINT_SYMBOL(page_ref_mod); - -void __page_ref_mod_and_test(struct page *page, int v, int ret) -{ - trace_page_ref_mod_and_test(page, v, ret); -} -EXPORT_SYMBOL(__page_ref_mod_and_test); -EXPORT_TRACEPOINT_SYMBOL(page_ref_mod_and_test); - void __page_ref_mod_and_return(struct page *page, int v, int ret) { trace_page_ref_mod_and_return(page, v, ret); --=20 2.34.1.307.g9b7440fafd-goog From nobody Wed Jul 1 13:27:41 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id DF851C433EF for ; Tue, 21 Dec 2021 15:02:09 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238857AbhLUPCI (ORCPT ); Tue, 21 Dec 2021 10:02:08 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53338 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238687AbhLUPBz (ORCPT ); Tue, 21 Dec 2021 10:01:55 -0500 Received: from mail-qt1-x82a.google.com (mail-qt1-x82a.google.com [IPv6:2607:f8b0:4864:20::82a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3D512C06175B for ; Tue, 21 Dec 2021 07:01:53 -0800 (PST) Received: by mail-qt1-x82a.google.com with SMTP id q14so13049695qtx.10 for ; Tue, 21 Dec 2021 07:01:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=9Ho7P1Z1WqP7ryS19lrI0k8d2U4f/eXw/1oJbaqWceE=; b=Gq4vEZXFfCrx18wyS4Qq27rBeHdcAXH69jksx+OkwhSowQMlXtH4nFIJdXpj0jffND GMRkTkEsROTk+Q0fN/zQv/aA8Nw+i6ac14Mx8BBAW3gONlgYyoUM4BzZ5Oc3F3mY6x7O q2mF8Q2KFRGrCnnGXI/fEWXmh5E5q0wklFMVekfIC7hRBna0SIm50MbZ9WVxjPTgp+3W z4EzflFfMl0tNruyp4qrk3cUMkdYdVCQx7xv+fIJKz/0uYLLCSDlawMXFO6s0/kqT9Ff MS3Y8+2ZU5HfUKGCMZQ9wgByhNWhrueYcDvj4Yj9qoCYc+RLj/eBDzEmztt+cmbowjXB mpMw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=9Ho7P1Z1WqP7ryS19lrI0k8d2U4f/eXw/1oJbaqWceE=; b=eUueI9EgZa9AvXJrOKUJJ88rMQ46sOWYRxr6WyYtG0xekv5oiIUP/2R38az5Nskwr0 9c908blbzrJIBy8CrO3QVavDv4CdiFvxVf2pcoiTbuk/rwwW8450WcFsVZj4tEzrnsK8 6E8pJU86xFFgT0zfWmYRgK08+8EWJG5MeO8fWCoN5KeNF6N5NDeeOaVG2t7nQ+QCc7/f HpSZ4LGvGjMjBzKLj3wGbs+OeeMSKB72DD27E+XqaoHB7bBaIK6WUTaB33gdF9C3Go0h NVKqHwziV/PuaUVXm1Tt8jyBITxcP8adM1dWOC5l9mrod+HuFrVcL/8v6RLb8IgfP+t0 tksQ== X-Gm-Message-State: AOAM531hH0tH2tXBWAS0Dyd4Hc51hxP/OnTigim7z0mPu1ZqkiAd7ROk UK7GbYKsqnO8HNpLLtsNTRtKYg== X-Google-Smtp-Source: ABdhPJwpEZVujf1Pk3vgH5fkaQH3jWmAOGB6Xqm6lM1kpQ6rkwA951oHxisyuwNEIFAkYBGCXlnQ6w== X-Received: by 2002:ac8:7f4b:: with SMTP id g11mr2566406qtk.4.1640098912343; Tue, 21 Dec 2021 07:01:52 -0800 (PST) Received: from soleen.c.googlers.com.com (189.216.85.34.bc.googleusercontent.com. [34.85.216.189]) by smtp.gmail.com with ESMTPSA id d4sm1991371qkn.79.2021.12.21.07.01.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 21 Dec 2021 07:01:51 -0800 (PST) From: Pasha Tatashin To: pasha.tatashin@soleen.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-m68k@lists.linux-m68k.org, anshuman.khandual@arm.com, willy@infradead.org, akpm@linux-foundation.org, william.kucharski@oracle.com, mike.kravetz@oracle.com, vbabka@suse.cz, geert@linux-m68k.org, schmitzmic@gmail.com, rostedt@goodmis.org, mingo@redhat.com, hannes@cmpxchg.org, guro@fb.com, songmuchun@bytedance.com, weixugc@google.com, gthelen@google.com, rientjes@google.com, pjt@google.com Subject: [PATCH v2 8/9] mm: do not use atomic_set_release in page_ref_unfreeze() Date: Tue, 21 Dec 2021 15:01:39 +0000 Message-Id: <20211221150140.988298-9-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.34.1.307.g9b7440fafd-goog In-Reply-To: <20211221150140.988298-1-pasha.tatashin@soleen.com> References: <20211221150140.988298-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" In we set the old _refcount value after verifying that the old value was indeed 0. VM_BUG_ON_PAGE(page_count(page) !=3D 0, page); < the _refcount may change here> atomic_set_release(&page->_refcount, count); To avoid the smal gap where _refcount may change lets verify the time of the _refcount at the time of the set operation. Use atomic_xchg_release() and at the set time verify that the value was 0. Signed-off-by: Pasha Tatashin --- include/linux/page_ref.h | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/include/linux/page_ref.h b/include/linux/page_ref.h index 243fc60ae6c8..9efabeff4e06 100644 --- a/include/linux/page_ref.h +++ b/include/linux/page_ref.h @@ -322,10 +322,9 @@ static inline int folio_ref_freeze(struct folio *folio= , int count) =20 static inline void page_ref_unfreeze(struct page *page, int count) { - VM_BUG_ON_PAGE(page_count(page) !=3D 0, page); - VM_BUG_ON(count =3D=3D 0); + int old_val =3D atomic_xchg_release(&page->_refcount, count); =20 - atomic_set_release(&page->_refcount, count); + VM_BUG_ON_PAGE(count =3D=3D 0 || old_val !=3D 0, page); if (page_ref_tracepoint_active(page_ref_unfreeze)) __page_ref_unfreeze(page, count); } --=20 2.34.1.307.g9b7440fafd-goog From nobody Wed Jul 1 13:27:41 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 40FE9C433FE for ; Tue, 21 Dec 2021 15:02:07 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238841AbhLUPCG (ORCPT ); Tue, 21 Dec 2021 10:02:06 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53326 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235801AbhLUPBz (ORCPT ); Tue, 21 Dec 2021 10:01:55 -0500 Received: from mail-qv1-xf34.google.com (mail-qv1-xf34.google.com [IPv6:2607:f8b0:4864:20::f34]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1BFAFC06175E for ; Tue, 21 Dec 2021 07:01:54 -0800 (PST) Received: by mail-qv1-xf34.google.com with SMTP id q4so6015864qvh.9 for ; Tue, 21 Dec 2021 07:01:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=ujiBeyeY0ZhWFNLdTCKueYw4ysMT9Aw/7apPybkWG1s=; b=PA3Q/yDh7W+DgSnKOTeVJUsVJL2iHQHGtRuXH0oQMCfi3NgBzbAlRtnkbB9VslFuln JgUXIsrq1dntIqfnKtVub3TAjFSBwovezuGjEs3vddMSz4B5E6iP75xXidJlVHdWmWRJ K2Hka8iaYZ5H/DI+OAzA2Wp0g/1hRTlVUi1AkgHDGaUEHi4zkn+tCPyqgXJyaUlevaHs LbEjFWY6ySLuQAHYvgf2ziB9zfVBqb9otAVbfhkvHbGoUQrnySsFwdw6XlQqKyE9RhVo XBcZRWgYNo4vWOlPwLvSnUALA/P7VH0q92uNKhHlNB67+Ut18sXhqjf6Ed+1ptUU0jls bK+Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ujiBeyeY0ZhWFNLdTCKueYw4ysMT9Aw/7apPybkWG1s=; b=V+lPZyT9K9VRm2aYtrJfnT7NnAdoXHohzfRfab9p4792SiC1hEQ2PGIl0ArWQ7iv8T +UxILcBw8IhbqTXCZPO1nCK8HuggtSEC3o3S4pkxaqDKUpkHQKNM07GcH3Ax8LNmaWDR Ft+v7hmpV3H8VOy1MMIdR5Lb4sWQssILw2gTEN/FZzT/rkn3I1tGoDeJgllMQtLYQmFx 059OAGfqztXPCgFZVoO4cE7waDpinnc3tBp5rBZb7C8ZqPoDv7H9WjtSZgPqjDp8zjz9 z5KXTdMo/bB1VAN7fwe+DWWWu7vncnADi+T/1b1iqRBfncTp7ew1HaOqBgVHYr3xvOdw pO7g== X-Gm-Message-State: AOAM533nuupxgPzRo2SqOGXYOW8NpBVHcD4QDXMbFtGTW6xp+KWcgIU3 JNbCSALClOSsc+wPeyeOpXdXaA== X-Google-Smtp-Source: ABdhPJwstE7x+0HpgXRt/GfFm68tAcQrAuhOR2pvSynOcMLtQsVanoM/IfrenS7mms+J84bIphIm9Q== X-Received: by 2002:ad4:5aad:: with SMTP id u13mr2841845qvg.46.1640098913240; Tue, 21 Dec 2021 07:01:53 -0800 (PST) Received: from soleen.c.googlers.com.com (189.216.85.34.bc.googleusercontent.com. [34.85.216.189]) by smtp.gmail.com with ESMTPSA id d4sm1991371qkn.79.2021.12.21.07.01.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 21 Dec 2021 07:01:52 -0800 (PST) From: Pasha Tatashin To: pasha.tatashin@soleen.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-m68k@lists.linux-m68k.org, anshuman.khandual@arm.com, willy@infradead.org, akpm@linux-foundation.org, william.kucharski@oracle.com, mike.kravetz@oracle.com, vbabka@suse.cz, geert@linux-m68k.org, schmitzmic@gmail.com, rostedt@goodmis.org, mingo@redhat.com, hannes@cmpxchg.org, guro@fb.com, songmuchun@bytedance.com, weixugc@google.com, gthelen@google.com, rientjes@google.com, pjt@google.com Subject: [PATCH v2 9/9] mm: use atomic_cmpxchg_acquire in page_ref_freeze(). Date: Tue, 21 Dec 2021 15:01:40 +0000 Message-Id: <20211221150140.988298-10-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.34.1.307.g9b7440fafd-goog In-Reply-To: <20211221150140.988298-1-pasha.tatashin@soleen.com> References: <20211221150140.988298-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" page_ref_freeze and page_ref_unfreeze are designed to be used as a pair. They protect critical sections where struct page can be modified. page_ref_unfreeze() is protected by _release() atomic operation, but page_ref_freeze() is not as it is assumed that cmpxch provides the full barrier. Instead, use the appropriate atomic_cmpxchg_acquire() to ensure that memory model is excplicitly followed. Signed-off-by: Pasha Tatashin --- include/linux/page_ref.h | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/include/linux/page_ref.h b/include/linux/page_ref.h index 9efabeff4e06..45be731d8919 100644 --- a/include/linux/page_ref.h +++ b/include/linux/page_ref.h @@ -308,7 +308,8 @@ static inline bool folio_try_get_rcu(struct folio *foli= o) =20 static inline int page_ref_freeze(struct page *page, int count) { - int ret =3D likely(atomic_cmpxchg(&page->_refcount, count, 0) =3D=3D coun= t); + int old_val =3D atomic_cmpxchg_acquire(&page->_refcount, count, 0); + int ret =3D likely(old_val =3D=3D count); =20 if (page_ref_tracepoint_active(page_ref_freeze)) __page_ref_freeze(page, count, ret); --=20 2.34.1.307.g9b7440fafd-goog