From nobody Wed Jul 1 13:26:37 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 73D5DC63697 for ; Wed, 26 Jan 2022 18:34:41 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S244053AbiAZSek (ORCPT ); Wed, 26 Jan 2022 13:34:40 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60120 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238302AbiAZSee (ORCPT ); Wed, 26 Jan 2022 13:34:34 -0500 Received: from mail-qv1-xf2d.google.com (mail-qv1-xf2d.google.com [IPv6:2607:f8b0:4864:20::f2d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D4D81C06161C for ; Wed, 26 Jan 2022 10:34:33 -0800 (PST) Received: by mail-qv1-xf2d.google.com with SMTP id o9so581009qvy.13 for ; Wed, 26 Jan 2022 10:34:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=5WscGeHykgFKvtNA2a3xQVpoktH8d9hYnkzSqx4X754=; b=c5KOiDhIQTjr76e99jrvXjP+zQIipG3qWrlyoBFSGPZvpXa3a24bsABSirLKilejPC HkTjj42IFw8t4AGBMUZWfJ7deIiZuar7QUetQ8j4b4ofpD9nqn3+4cZ+Qs8vY95+vQF3 0YiNinlkA9NmsHXyZHGsJIIXnG2UIJ6XjZZ7qxMufoPDrAK4ntjA7Duo9NfAdnsNtZjp s/Q8JPMpXFQvDIwl2QI3Apz9+P5MxV3P6qHkV3oq9kortiYH97cGqcZNdyBFGhEUXVIX TdnOrdrUYAxm1YWbVGQ0v05o4Zwi7U4IYlcDrSNMwEf1C1RR/aG//gsOJJPFR3WepWN6 bd7Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=5WscGeHykgFKvtNA2a3xQVpoktH8d9hYnkzSqx4X754=; b=kQLoE3TR3fVpzSs0sPBC6pKvtgGYuhm6E7xkrfuT0TYPD0FC6zm431flpHy43nBTVk afsR4oV6L+km3lKYf/Qd7wA3IAw9ULC6zwmQ3cZOjVy6NMnqDhUbQ5vKq0iqiuf26A8x 2l6tPNhKHEdeVb+/MyGX0ZSJQmSPb4m4xXzvEZ62iJ6SI43ZHcBMqV3H8tZG7EtTwYbc ej30J7/P4YIdO266pgg/DCd26Ii1f3EEvl5dvJs9iy6gbLALXHQpWXcvXJ7G8HhZMjFm 2QaBP97mbLM99cldmuSB/iMaGdWOhISz7SY13kOt10zxfcdxVncqmqg0vO141SByQ+Ju 6mSw== X-Gm-Message-State: AOAM5306Jsm22FA2juqyqrfpR7MH+A+isTBd/T5c2ip0eKDb5weGqzav rVMbnWt/Ub+is3smGwORWkDKEg== X-Google-Smtp-Source: ABdhPJwFL8yN6047PgmxSq7+98SLjxRErvrti9/csMHETfIdMupPBgY4rmaoYYVPLw9Ssr5QtQmqpw== X-Received: by 2002:a05:6214:2b06:: with SMTP id jx6mr15297174qvb.117.1643222072958; Wed, 26 Jan 2022 10:34:32 -0800 (PST) Received: from soleen.c.googlers.com.com (189.216.85.34.bc.googleusercontent.com. [34.85.216.189]) by smtp.gmail.com with ESMTPSA id u17sm35886qki.12.2022.01.26.10.34.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 26 Jan 2022 10:34:32 -0800 (PST) From: Pasha Tatashin To: pasha.tatashin@soleen.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-m68k@lists.linux-m68k.org, anshuman.khandual@arm.com, willy@infradead.org, akpm@linux-foundation.org, william.kucharski@oracle.com, mike.kravetz@oracle.com, vbabka@suse.cz, geert@linux-m68k.org, schmitzmic@gmail.com, rostedt@goodmis.org, mingo@redhat.com, hannes@cmpxchg.org, guro@fb.com, songmuchun@bytedance.com, weixugc@google.com, gthelen@google.com, rientjes@google.com, pjt@google.com, hughd@google.com Subject: [PATCH v3 1/9] mm: add overflow and underflow checks for page->_refcount Date: Wed, 26 Jan 2022 18:34:21 +0000 Message-Id: <20220126183429.1840447-2-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.35.0.rc0.227.g00780c9af4-goog In-Reply-To: <20220126183429.1840447-1-pasha.tatashin@soleen.com> References: <20220126183429.1840447-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" The problems with page->_refcount are hard to debug, because usually when they are detected, the damage has occurred a long time ago. Yet, the problems with invalid page refcount may be catastrophic and lead to memory corruptions. Reduce the scope of when the _refcount problems manifest themselves by adding checks for underflows and overflows into functions that modify _refcount. Use atomic_fetch_* functions to get the old values of the _refcount, and use it to check for overflow/underflow. Signed-off-by: Pasha Tatashin --- include/linux/page_ref.h | 59 +++++++++++++++++++++++++++++----------- 1 file changed, 43 insertions(+), 16 deletions(-) diff --git a/include/linux/page_ref.h b/include/linux/page_ref.h index 2e677e6ad09f..fe4864f7f69c 100644 --- a/include/linux/page_ref.h +++ b/include/linux/page_ref.h @@ -117,7 +117,10 @@ static inline void init_page_count(struct page *page) =20 static inline void page_ref_add(struct page *page, int nr) { - atomic_add(nr, &page->_refcount); + int old_val =3D atomic_fetch_add(nr, &page->_refcount); + int new_val =3D old_val + nr; + + VM_BUG_ON_PAGE((unsigned int)new_val < (unsigned int)old_val, page); if (page_ref_tracepoint_active(page_ref_mod)) __page_ref_mod(page, nr); } @@ -129,7 +132,10 @@ static inline void folio_ref_add(struct folio *folio, = int nr) =20 static inline void page_ref_sub(struct page *page, int nr) { - atomic_sub(nr, &page->_refcount); + int old_val =3D atomic_fetch_sub(nr, &page->_refcount); + int new_val =3D old_val - nr; + + VM_BUG_ON_PAGE((unsigned int)new_val > (unsigned int)old_val, page); if (page_ref_tracepoint_active(page_ref_mod)) __page_ref_mod(page, -nr); } @@ -141,11 +147,13 @@ static inline void folio_ref_sub(struct folio *folio,= int nr) =20 static inline int page_ref_sub_return(struct page *page, int nr) { - int ret =3D atomic_sub_return(nr, &page->_refcount); + int old_val =3D atomic_fetch_sub(nr, &page->_refcount); + int new_val =3D old_val - nr; =20 + VM_BUG_ON_PAGE((unsigned int)new_val > (unsigned int)old_val, page); if (page_ref_tracepoint_active(page_ref_mod_and_return)) - __page_ref_mod_and_return(page, -nr, ret); - return ret; + __page_ref_mod_and_return(page, -nr, new_val); + return new_val; } =20 static inline int folio_ref_sub_return(struct folio *folio, int nr) @@ -155,7 +163,10 @@ static inline int folio_ref_sub_return(struct folio *f= olio, int nr) =20 static inline void page_ref_inc(struct page *page) { - atomic_inc(&page->_refcount); + int old_val =3D atomic_fetch_inc(&page->_refcount); + int new_val =3D old_val + 1; + + VM_BUG_ON_PAGE((unsigned int)new_val < (unsigned int)old_val, page); if (page_ref_tracepoint_active(page_ref_mod)) __page_ref_mod(page, 1); } @@ -167,7 +178,10 @@ static inline void folio_ref_inc(struct folio *folio) =20 static inline void page_ref_dec(struct page *page) { - atomic_dec(&page->_refcount); + int old_val =3D atomic_fetch_dec(&page->_refcount); + int new_val =3D old_val - 1; + + VM_BUG_ON_PAGE((unsigned int)new_val > (unsigned int)old_val, page); if (page_ref_tracepoint_active(page_ref_mod)) __page_ref_mod(page, -1); } @@ -179,8 +193,11 @@ static inline void folio_ref_dec(struct folio *folio) =20 static inline int page_ref_sub_and_test(struct page *page, int nr) { - int ret =3D atomic_sub_and_test(nr, &page->_refcount); + int old_val =3D atomic_fetch_sub(nr, &page->_refcount); + int new_val =3D old_val - nr; + int ret =3D new_val =3D=3D 0; =20 + VM_BUG_ON_PAGE((unsigned int)new_val > (unsigned int)old_val, page); if (page_ref_tracepoint_active(page_ref_mod_and_test)) __page_ref_mod_and_test(page, -nr, ret); return ret; @@ -193,11 +210,13 @@ static inline int folio_ref_sub_and_test(struct folio= *folio, int nr) =20 static inline int page_ref_inc_return(struct page *page) { - int ret =3D atomic_inc_return(&page->_refcount); + int old_val =3D atomic_fetch_inc(&page->_refcount); + int new_val =3D old_val + 1; =20 + VM_BUG_ON_PAGE((unsigned int)new_val < (unsigned int)old_val, page); if (page_ref_tracepoint_active(page_ref_mod_and_return)) - __page_ref_mod_and_return(page, 1, ret); - return ret; + __page_ref_mod_and_return(page, 1, new_val); + return new_val; } =20 static inline int folio_ref_inc_return(struct folio *folio) @@ -207,8 +226,11 @@ static inline int folio_ref_inc_return(struct folio *f= olio) =20 static inline int page_ref_dec_and_test(struct page *page) { - int ret =3D atomic_dec_and_test(&page->_refcount); + int old_val =3D atomic_fetch_dec(&page->_refcount); + int new_val =3D old_val - 1; + int ret =3D new_val =3D=3D 0; =20 + VM_BUG_ON_PAGE((unsigned int)new_val > (unsigned int)old_val, page); if (page_ref_tracepoint_active(page_ref_mod_and_test)) __page_ref_mod_and_test(page, -1, ret); return ret; @@ -221,11 +243,13 @@ static inline int folio_ref_dec_and_test(struct folio= *folio) =20 static inline int page_ref_dec_return(struct page *page) { - int ret =3D atomic_dec_return(&page->_refcount); + int old_val =3D atomic_fetch_dec(&page->_refcount); + int new_val =3D old_val - 1; =20 + VM_BUG_ON_PAGE((unsigned int)new_val > (unsigned int)old_val, page); if (page_ref_tracepoint_active(page_ref_mod_and_return)) - __page_ref_mod_and_return(page, -1, ret); - return ret; + __page_ref_mod_and_return(page, -1, new_val); + return new_val; } =20 static inline int folio_ref_dec_return(struct folio *folio) @@ -235,8 +259,11 @@ static inline int folio_ref_dec_return(struct folio *f= olio) =20 static inline bool page_ref_add_unless(struct page *page, int nr, int u) { - bool ret =3D atomic_add_unless(&page->_refcount, nr, u); + int old_val =3D atomic_fetch_add_unless(&page->_refcount, nr, u); + int new_val =3D old_val + nr; + int ret =3D old_val !=3D u; =20 + VM_BUG_ON_PAGE(ret && (unsigned int)new_val < (unsigned int)old_val, page= ); if (page_ref_tracepoint_active(page_ref_mod_unless)) __page_ref_mod_unless(page, nr, ret); return ret; --=20 2.35.0.rc0.227.g00780c9af4-goog From nobody Wed Jul 1 13:26:37 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id AB2BAC28CF5 for ; Wed, 26 Jan 2022 18:34:48 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S244175AbiAZSer (ORCPT ); Wed, 26 Jan 2022 13:34:47 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60132 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239191AbiAZSee (ORCPT ); Wed, 26 Jan 2022 13:34:34 -0500 Received: from mail-qv1-xf31.google.com (mail-qv1-xf31.google.com [IPv6:2607:f8b0:4864:20::f31]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A73ECC061748 for ; Wed, 26 Jan 2022 10:34:34 -0800 (PST) Received: by mail-qv1-xf31.google.com with SMTP id o9so581051qvy.13 for ; Wed, 26 Jan 2022 10:34:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=4mw64hU1ZpbwSPVpRDpj2TUUxEzGleIpOUNlQVlyvBg=; b=F3hPkfc5vl0lCp4Fc8VE6hdllDFjQ/KJ4ChxD81oXql/kI+Wqg+5Eg6mJp84q5ARi0 bwTA2AeDywIsvmiI0bsQwfYhuw5oHutsV5TEGXWen0Fswl2QklzgqTABOo6pfqUX6H4A 9iw9QJC5YKH4ek5QFHZ7USKm0q6pF0Hv/5ucF13RYA0PqDHQMv+uzuEM91G9R/l4ehP2 FKA2+uQkBZtYXRHaOK3gVvGZ6YGKSt/5ZzAzKnPdoMy8jM7rcvPnHIKlRB/z3n0fM5tC qw27euVm+4l4GXriH7OI+46l5IztN1E0Q7uHd4+RVdjriFD0RFh1sJb4k12yDj0T6ywZ gvCQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=4mw64hU1ZpbwSPVpRDpj2TUUxEzGleIpOUNlQVlyvBg=; b=uhuhv8lBVBAjcEKfZkdnB8nrwc05NcXJtHF/0GS4rn9S+JWxmKdVvWD3sblPgMm5tL 2kv0Bb2iY3E2fZXtaq5u5uMLavDRI1B4/VVZvKzhgUKL6roCc1AYaZdm8ReBVGiHpJsm +xby56VQhNGPDWv+6wwfF4PVXchvkEteHZH+56fEjuYcaQNtzNS+ZZl0Aj2IN8A6ByDV bXKPzXyUJZOS5NYo06jB/6+fhUS6KT7rVHoaYdLCgSQgxEnkPPg+prJ/26kfEo0xVWz3 Af39ajc25SW+mOM7cGrr9ePpIchxo+dVma3+CG67DaocZZRvNtZLWL7us4bAIbAbCGxv FCRw== X-Gm-Message-State: AOAM5316lfaaHuoF/IPlAfV5HYt9U1yaGJjja6q3qtAgfzFjQSzN+m/9 GuTehCcwGQcV2NeCU1SsuGHoTw== X-Google-Smtp-Source: ABdhPJwb0TuIX2qSbyMJVsMfaoSMdbbLW4f4sRMAfDNGo3TL8yjF/ORtCOVu1ucqYV6uSVM24X6+vg== X-Received: by 2002:a05:6214:401b:: with SMTP id kd27mr77991qvb.22.1643222073781; Wed, 26 Jan 2022 10:34:33 -0800 (PST) Received: from soleen.c.googlers.com.com (189.216.85.34.bc.googleusercontent.com. [34.85.216.189]) by smtp.gmail.com with ESMTPSA id u17sm35886qki.12.2022.01.26.10.34.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 26 Jan 2022 10:34:33 -0800 (PST) From: Pasha Tatashin To: pasha.tatashin@soleen.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-m68k@lists.linux-m68k.org, anshuman.khandual@arm.com, willy@infradead.org, akpm@linux-foundation.org, william.kucharski@oracle.com, mike.kravetz@oracle.com, vbabka@suse.cz, geert@linux-m68k.org, schmitzmic@gmail.com, rostedt@goodmis.org, mingo@redhat.com, hannes@cmpxchg.org, guro@fb.com, songmuchun@bytedance.com, weixugc@google.com, gthelen@google.com, rientjes@google.com, pjt@google.com, hughd@google.com Subject: [PATCH v3 2/9] mm: Avoid using set_page_count() in set_page_recounted() Date: Wed, 26 Jan 2022 18:34:22 +0000 Message-Id: <20220126183429.1840447-3-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.35.0.rc0.227.g00780c9af4-goog In-Reply-To: <20220126183429.1840447-1-pasha.tatashin@soleen.com> References: <20220126183429.1840447-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" set_page_refcounted() converts a non-refcounted page that has (page->_refcount =3D=3D 0) into a refcounted page by setting _refcount to 1. The current apporach uses the following logic: VM_BUG_ON_PAGE(page_ref_count(page), page); set_page_count(page, 1); However, if _refcount changes from 0 to 1 between the VM_BUG_ON_PAGE() and set_page_count() we can break _refcount, which can cause other problems such as memory corruptions. Instead, use a safer method: increment _refcount first and verify that at increment time it was indeed 1. refcnt =3D page_ref_inc_return(page); VM_BUG_ON_PAGE(refcnt !=3D 1, page); Use page_ref_inc_return() to avoid unconditionally overwriting the _refcount value with set_page_count(), and check the return value. Signed-off-by: Pasha Tatashin --- mm/internal.h | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/mm/internal.h b/mm/internal.h index 4c2d06a2f50b..6b74f7f32613 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -141,9 +141,11 @@ static inline bool page_evictable(struct page *page) */ static inline void set_page_refcounted(struct page *page) { + int refcnt; + VM_BUG_ON_PAGE(PageTail(page), page); - VM_BUG_ON_PAGE(page_ref_count(page), page); - set_page_count(page, 1); + refcnt =3D page_ref_inc_return(page); + VM_BUG_ON_PAGE(refcnt !=3D 1, page); } =20 extern unsigned long highest_memmap_pfn; --=20 2.35.0.rc0.227.g00780c9af4-goog From nobody Wed Jul 1 13:26:37 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D51BBC2BA4C for ; Wed, 26 Jan 2022 18:34:49 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S244179AbiAZSet (ORCPT ); Wed, 26 Jan 2022 13:34:49 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60144 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240486AbiAZSei (ORCPT ); Wed, 26 Jan 2022 13:34:38 -0500 Received: from mail-qk1-x733.google.com (mail-qk1-x733.google.com [IPv6:2607:f8b0:4864:20::733]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7CE38C06174E for ; Wed, 26 Jan 2022 10:34:35 -0800 (PST) Received: by mail-qk1-x733.google.com with SMTP id q5so413082qkc.1 for ; Wed, 26 Jan 2022 10:34:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=6rktNgcbrbK1gQESupfA4dHtrh4dYlEFYGvtlevr0bg=; b=ZOPvyP2LJa1z3UAKsmFkGIvb9SGcukKRw1eJDnZ03Qy7wZ4AeqMOKXH3MMBQGrbIrD /K1gtEKq72kR4D2ZLluXJ7N65IkST6vYrjwupXq4pYplOy24Hwrbt019FVcqjyTfs4uM lsxV8J1HoLT678mTxAyiXcz4eRp1gSd1iPLhXoXTOfQioYdf5YHp5LPIvXTIzhL/PYZg tFa4DDUfUv2/nhf4ixyqPQOrpe0bQFsJHGraZDElfYhxqzwN7HggCEhbvvXk+a5O2CJ8 zuTcV5TSk5C5M3X8ND+XcwEmPXrCXMwHne1ra/PCaFUMgtU1czwB78aFNGC/vAvu+kVG NUuA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=6rktNgcbrbK1gQESupfA4dHtrh4dYlEFYGvtlevr0bg=; b=qcgpYmyS2N4AnmEcO1pat3E+5k037wD/XaswA9LDJ5l+WQi11CJwSOoJAcevhzEVNN f7Sf8gxN0OT7FfgCKsyaWG98OHWNri8ypCmLPRPFZqLJX2o6FdFjmOvmlkjmKklVBDfC v3HLYckTOtnDGuNDmVe9fPi3iYliWdUM9uyYG4RX4/7JXST+gPUDDHYmnjO/xnh449Ge CYPuEMZfIjql7T1NBaVNRjxeXm2WEo7Rv7OZbduTgm4xSdxV91TgnDmBbMVnwutRrFWM U4OxazgUboZ0nbYYAbD87Slu9BixKkEq5r+ty+jaozbrHulNMFmVzYRDEcdwgUS/Jccf y/Cw== X-Gm-Message-State: AOAM530HR6vnsAFG/K9c+eedXVR0ugOgAqDkOOSb8qysemJBzMTXNnD9 wx6aWgJAfUn96sDG8FXjBj4xBA== X-Google-Smtp-Source: ABdhPJyOZcmiDwXUXWvvn+PHKnj59WKkQW/1KnbYA3kyw9zl35YOaxOviGVfeae+pb/kVHJjKl9WDQ== X-Received: by 2002:a05:620a:25ca:: with SMTP id y10mr59059qko.546.1643222074652; Wed, 26 Jan 2022 10:34:34 -0800 (PST) Received: from soleen.c.googlers.com.com (189.216.85.34.bc.googleusercontent.com. [34.85.216.189]) by smtp.gmail.com with ESMTPSA id u17sm35886qki.12.2022.01.26.10.34.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 26 Jan 2022 10:34:34 -0800 (PST) From: Pasha Tatashin To: pasha.tatashin@soleen.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-m68k@lists.linux-m68k.org, anshuman.khandual@arm.com, willy@infradead.org, akpm@linux-foundation.org, william.kucharski@oracle.com, mike.kravetz@oracle.com, vbabka@suse.cz, geert@linux-m68k.org, schmitzmic@gmail.com, rostedt@goodmis.org, mingo@redhat.com, hannes@cmpxchg.org, guro@fb.com, songmuchun@bytedance.com, weixugc@google.com, gthelen@google.com, rientjes@google.com, pjt@google.com, hughd@google.com Subject: [PATCH v3 3/9] mm: remove set_page_count() from page_frag_alloc_align Date: Wed, 26 Jan 2022 18:34:23 +0000 Message-Id: <20220126183429.1840447-4-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.35.0.rc0.227.g00780c9af4-goog In-Reply-To: <20220126183429.1840447-1-pasha.tatashin@soleen.com> References: <20220126183429.1840447-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" set_page_count() unconditionally resets the value of _ref_count and that is dangerous, as it is not programmatically verified. Instead we rely on comments like: "OK, page count is 0, we can safely set it". Add a new refcount function: page_ref_add_return() to return the new refcount value after adding to it. Use the return value to verify that the _ref_count was indeed the expected one. Signed-off-by: Pasha Tatashin --- include/linux/page_ref.h | 11 +++++++++++ mm/page_alloc.c | 6 ++++-- 2 files changed, 15 insertions(+), 2 deletions(-) diff --git a/include/linux/page_ref.h b/include/linux/page_ref.h index fe4864f7f69c..03e21ce2f1bd 100644 --- a/include/linux/page_ref.h +++ b/include/linux/page_ref.h @@ -115,6 +115,17 @@ static inline void init_page_count(struct page *page) set_page_count(page, 1); } =20 +static inline int page_ref_add_return(struct page *page, int nr) +{ + int old_val =3D atomic_fetch_add(nr, &page->_refcount); + int new_val =3D old_val + nr; + + VM_BUG_ON_PAGE((unsigned int)new_val < (unsigned int)old_val, page); + if (page_ref_tracepoint_active(page_ref_mod_and_return)) + __page_ref_mod_and_return(page, nr, new_val); + return new_val; +} + static inline void page_ref_add(struct page *page, int nr) { int old_val =3D atomic_fetch_add(nr, &page->_refcount); diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 8dd6399bafb5..5a9167bda279 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -5528,6 +5528,7 @@ void *page_frag_alloc_align(struct page_frag_cache *n= c, unsigned int size =3D PAGE_SIZE; struct page *page; int offset; + int refcnt; =20 if (unlikely(!nc->va)) { refill: @@ -5566,8 +5567,9 @@ void *page_frag_alloc_align(struct page_frag_cache *n= c, /* if size can vary use size else just use PAGE_SIZE */ size =3D nc->size; #endif - /* OK, page count is 0, we can safely set it */ - set_page_count(page, PAGE_FRAG_CACHE_MAX_SIZE + 1); + /* page count is 0, set it to PAGE_FRAG_CACHE_MAX_SIZE + 1 */ + refcnt =3D page_ref_add_return(page, PAGE_FRAG_CACHE_MAX_SIZE + 1); + VM_BUG_ON_PAGE(refcnt !=3D PAGE_FRAG_CACHE_MAX_SIZE + 1, page); =20 /* reset page count bias and offset to start of new frag */ nc->pagecnt_bias =3D PAGE_FRAG_CACHE_MAX_SIZE + 1; --=20 2.35.0.rc0.227.g00780c9af4-goog From nobody Wed Jul 1 13:26:37 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B00D0C636C9 for ; Wed, 26 Jan 2022 18:34:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238302AbiAZSem (ORCPT ); Wed, 26 Jan 2022 13:34:42 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60150 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S241849AbiAZSei (ORCPT ); Wed, 26 Jan 2022 13:34:38 -0500 Received: from mail-qv1-xf30.google.com (mail-qv1-xf30.google.com [IPv6:2607:f8b0:4864:20::f30]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5A8BFC061755 for ; Wed, 26 Jan 2022 10:34:37 -0800 (PST) Received: by mail-qv1-xf30.google.com with SMTP id k4so619703qvt.6 for ; Wed, 26 Jan 2022 10:34:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=Ie3wyi01fH7CgmQYiqIN+tfxdxDxn1n1+kzHm4Q9RTE=; b=Bh400EnOT890ni7sht/44ozXW/6kzxp07ZHx7PHxWuPEpO5UWUblrE8yWYYEvyOZpy MEeDREPxYzJitlr3ychWp4Zzd53yjtbV1lTT6U4sXKY5QOs8eT0Ao/oZIAnwcHjyATZz tvRrEJaOpgFl+PLRLGNYDmv7E1xAUepdJHaTNhjB3fefVxY+fVJFRhn6o2dRH5BjqRJo syBZkBNB2cBomk/aVeT/u9aFQAkG7UcRC0tnPO4eZmPSZcT6zo03CZ/wyTukDnwagYtN RZjniCisA268LqO5pXzQ5ABPwHsfQoiYEfCJwAa8+UqYzT3ImXNuyo7FY6tV1pzS48q6 1hxQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Ie3wyi01fH7CgmQYiqIN+tfxdxDxn1n1+kzHm4Q9RTE=; b=PJ61KNsnoyNP98n2lrokeiNRIgLhqS7BOIo89p+RAAT56oCTmIwXRr3OpUTgNfWO/0 Xl7Zd9BMcAFtANwyRx2uFXcmMpPmjYcIk/JcVR4CH1OtQlx7E/ir+eSg2PjHD0XeICXe sk2N7dCA3t5MjDg7jS7qSgyDsztwwtZO+3QtKjLrgzrCxpL3w0CqCvrFp425U7+uUaGf 29AX9Un/MHzgd2YsSER4zuMY8luBceBuU0LPY3+aZHRYz0UyaKAThtR6SCfN1zD/DeiX +FPqFmL3ceC57390psp3djnkuIcjDHWsZZJmRKTcsEFxdCVzGpENQ2y5n/U4Y8J2Mjyl 8+AA== X-Gm-Message-State: AOAM530T3IdQ7NXeci4j3vKQhDoQfZCAyM0Kdi4WrTmP0acdYZFrG6mH SP6sFpKUX09CREAAMF84rNAnug== X-Google-Smtp-Source: ABdhPJzoIdGrjBW0d47PH4jGXBlBszWnfJdkQI7JvZmOWO27z83YtLTGz3Vk6XtP2izWkdLA7bnMTg== X-Received: by 2002:a05:6214:21c4:: with SMTP id d4mr22950102qvh.90.1643222076515; Wed, 26 Jan 2022 10:34:36 -0800 (PST) Received: from soleen.c.googlers.com.com (189.216.85.34.bc.googleusercontent.com. [34.85.216.189]) by smtp.gmail.com with ESMTPSA id u17sm35886qki.12.2022.01.26.10.34.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 26 Jan 2022 10:34:35 -0800 (PST) From: Pasha Tatashin To: pasha.tatashin@soleen.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-m68k@lists.linux-m68k.org, anshuman.khandual@arm.com, willy@infradead.org, akpm@linux-foundation.org, william.kucharski@oracle.com, mike.kravetz@oracle.com, vbabka@suse.cz, geert@linux-m68k.org, schmitzmic@gmail.com, rostedt@goodmis.org, mingo@redhat.com, hannes@cmpxchg.org, guro@fb.com, songmuchun@bytedance.com, weixugc@google.com, gthelen@google.com, rientjes@google.com, pjt@google.com, hughd@google.com Subject: [PATCH v3 4/9] mm: avoid using set_page_count() when pages are freed into allocator Date: Wed, 26 Jan 2022 18:34:24 +0000 Message-Id: <20220126183429.1840447-5-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.35.0.rc0.227.g00780c9af4-goog In-Reply-To: <20220126183429.1840447-1-pasha.tatashin@soleen.com> References: <20220126183429.1840447-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" When struct pages are first initialized the page->_refcount field is set 1. However, later when pages are freed into allocator we set _refcount to 0 via set_page_count(). Unconditionally resetting _refcount is dangerous. Instead use page_ref_dec_return(), and verify that the _refcount is what is expected. Signed-off-by: Pasha Tatashin --- mm/page_alloc.c | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 5a9167bda279..0fa100152a2a 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -1668,6 +1668,7 @@ void __free_pages_core(struct page *page, unsigned in= t order) unsigned int nr_pages =3D 1 << order; struct page *p =3D page; unsigned int loop; + int refcnt; =20 /* * When initializing the memmap, __init_single_page() sets the refcount @@ -1678,10 +1679,12 @@ void __free_pages_core(struct page *page, unsigned = int order) for (loop =3D 0; loop < (nr_pages - 1); loop++, p++) { prefetchw(p + 1); __ClearPageReserved(p); - set_page_count(p, 0); + refcnt =3D page_ref_dec_return(p); + VM_BUG_ON_PAGE(refcnt, p); } __ClearPageReserved(p); - set_page_count(p, 0); + refcnt =3D page_ref_dec_return(p); + VM_BUG_ON_PAGE(refcnt, p); =20 atomic_long_add(nr_pages, &page_zone(page)->managed_pages); =20 @@ -2253,10 +2256,12 @@ void __init init_cma_reserved_pageblock(struct page= *page) { unsigned i =3D pageblock_nr_pages; struct page *p =3D page; + int refcnt; =20 do { __ClearPageReserved(p); - set_page_count(p, 0); + refcnt =3D page_ref_dec_return(p); + VM_BUG_ON_PAGE(refcnt, p); } while (++p, --i); =20 set_pageblock_migratetype(page, MIGRATE_CMA); --=20 2.35.0.rc0.227.g00780c9af4-goog From nobody Wed Jul 1 13:26:37 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0EE02C28CF5 for ; Wed, 26 Jan 2022 18:34:46 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S244176AbiAZSeo (ORCPT ); Wed, 26 Jan 2022 13:34:44 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60144 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S242305AbiAZSei (ORCPT ); Wed, 26 Jan 2022 13:34:38 -0500 Received: from mail-qk1-x729.google.com (mail-qk1-x729.google.com [IPv6:2607:f8b0:4864:20::729]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 44780C061756 for ; Wed, 26 Jan 2022 10:34:38 -0800 (PST) Received: by mail-qk1-x729.google.com with SMTP id b22so353380qkk.12 for ; Wed, 26 Jan 2022 10:34:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=aBUwytSQMfy2PxxuYrxKtq3rfi9+4lfxfuQGWEZYavU=; b=l3wPcJdVsThu2Fd/44toWOV3aZBf2u1UKyJRPPbZryrarAUeut+CJwDZ8ZBn8Iu6Z1 9gfTp4Bv075j/BwF+Ngy6Ay6sMzIDMcqa4X8TTgaUs+Im6V7S3ofLK7W93oJ0z4lz3xr Qq5pM6XuZ4+WM30Swh3KI2BP+9imuy0GGJRJc3Y7t/ct2om5THUg1HsY771nUugjwhNB o6yhRFb/OKjGZiUe0daItrOPgtpA/FyZgE2RgrIndndd8D7dpKNP5udwWejZmB+vBbc1 b7IDp70VdgRBYRugGCONiyNlFJLAo0NVlGsvjEU/xzgfMrMAzq/Q9HVB0NvZ4Aa8v4++ +MWA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=aBUwytSQMfy2PxxuYrxKtq3rfi9+4lfxfuQGWEZYavU=; b=M2VKgdYNCzOYnEn1JH39Au5T3cdsKnRhbhkh1AUeMDNM/lDvEPXUvfHu6miKhVlX9n o/pAj27Ey+m3kG6ud1zy6b+qJB1oKvImhTiCamyCpBIvPjO7Uc+n2Yvw/RvHcsnibpYF rl87Sq5jrm2jyIIjlzIe/4aRLrxoDYJGHmT3nulwRl2PddeqrctefcMPItZSWL1s+/cD YJ9Ey5wRPPKvXgAZMoBd1bo1ERA0MXQZ8SK+XmutE9ads/sL9DjACZF9ts2pMvz5AZYN TyaVV9OvDRve6XAbnqdQ++EpHBgGz+JK2cwlHLm/TexXKXgXExt4V4Lt4BDtmnO4vAEC 6foA== X-Gm-Message-State: AOAM531Hksuk5IxDnB1qzhcHFsyi0zCsW/8LmBiVX7mHgytg/5bnGj7r QU1iOBsuBQp56YuvZXTemhSP7g== X-Google-Smtp-Source: ABdhPJzoa7OWWtfZhqM2BvNvH16AEuWpcEepXXHuHzqnh7Po9LG5Z02BFoetP3Q074/fbmgo2SkhZA== X-Received: by 2002:a05:620a:ecb:: with SMTP id x11mr18649qkm.399.1643222077431; Wed, 26 Jan 2022 10:34:37 -0800 (PST) Received: from soleen.c.googlers.com.com (189.216.85.34.bc.googleusercontent.com. [34.85.216.189]) by smtp.gmail.com with ESMTPSA id u17sm35886qki.12.2022.01.26.10.34.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 26 Jan 2022 10:34:37 -0800 (PST) From: Pasha Tatashin To: pasha.tatashin@soleen.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-m68k@lists.linux-m68k.org, anshuman.khandual@arm.com, willy@infradead.org, akpm@linux-foundation.org, william.kucharski@oracle.com, mike.kravetz@oracle.com, vbabka@suse.cz, geert@linux-m68k.org, schmitzmic@gmail.com, rostedt@goodmis.org, mingo@redhat.com, hannes@cmpxchg.org, guro@fb.com, songmuchun@bytedance.com, weixugc@google.com, gthelen@google.com, rientjes@google.com, pjt@google.com, hughd@google.com Subject: [PATCH v3 5/9] mm: rename init_page_count() -> page_ref_init() Date: Wed, 26 Jan 2022 18:34:25 +0000 Message-Id: <20220126183429.1840447-6-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.35.0.rc0.227.g00780c9af4-goog In-Reply-To: <20220126183429.1840447-1-pasha.tatashin@soleen.com> References: <20220126183429.1840447-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Now, that set_page_count() is not called from outside anymore and about to be removed, init_page_count() is the only function that is going to be used to unconditionally set _refcount, however it is restricted to set it only to 1. Make init_page_count() aligned with the other page_ref_* functions by renaming it. Signed-off-by: Pasha Tatashin Acked-by: Geert Uytterhoeven --- arch/m68k/mm/motorola.c | 2 +- include/linux/mm.h | 2 +- include/linux/page_ref.h | 10 +++++++--- mm/page_alloc.c | 2 +- 4 files changed, 10 insertions(+), 6 deletions(-) diff --git a/arch/m68k/mm/motorola.c b/arch/m68k/mm/motorola.c index ecbe948f4c1a..dd3b77d03d5c 100644 --- a/arch/m68k/mm/motorola.c +++ b/arch/m68k/mm/motorola.c @@ -133,7 +133,7 @@ void __init init_pointer_table(void *table, int type) =20 /* unreserve the page so it's possible to free that page */ __ClearPageReserved(PD_PAGE(dp)); - init_page_count(PD_PAGE(dp)); + page_ref_init(PD_PAGE(dp)); =20 return; } diff --git a/include/linux/mm.h b/include/linux/mm.h index 45bcd6f78141..cd8b9a592235 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2467,7 +2467,7 @@ extern void reserve_bootmem_region(phys_addr_t start,= phys_addr_t end); static inline void free_reserved_page(struct page *page) { ClearPageReserved(page); - init_page_count(page); + page_ref_init(page); __free_page(page); adjust_managed_page_count(page, 1); } diff --git a/include/linux/page_ref.h b/include/linux/page_ref.h index 03e21ce2f1bd..1af12a0d7ba1 100644 --- a/include/linux/page_ref.h +++ b/include/linux/page_ref.h @@ -107,10 +107,14 @@ static inline void folio_set_count(struct folio *foli= o, int v) } =20 /* - * Setup the page count before being freed into the page allocator for - * the first time (boot or memory hotplug) + * Setup the page refcount to one before being freed into the page allocat= or. + * The memory might not be initialized and therefore there cannot be any + * assumptions about the current value of page->_refcount. This call shoul= d be + * done during boot when memory is being initialized, during memory hotplug + * when new memory is added, or when a previous reserved memory is unreser= ved + * this is the first time kernel take control of the given memory. */ -static inline void init_page_count(struct page *page) +static inline void page_ref_init(struct page *page) { set_page_count(page, 1); } diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 0fa100152a2a..cbe444d74e8a 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -1570,7 +1570,7 @@ static void __meminit __init_single_page(struct page = *page, unsigned long pfn, { mm_zero_struct_page(page); set_page_links(page, zone, nid, pfn); - init_page_count(page); + page_ref_init(page); page_mapcount_reset(page); page_cpupid_reset_last(page); page_kasan_tag_reset(page); --=20 2.35.0.rc0.227.g00780c9af4-goog From nobody Wed Jul 1 13:26:37 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 318A6C63682 for ; Wed, 26 Jan 2022 18:34:51 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S244182AbiAZSeu (ORCPT ); Wed, 26 Jan 2022 13:34:50 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60160 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240802AbiAZSej (ORCPT ); Wed, 26 Jan 2022 13:34:39 -0500 Received: from mail-qv1-xf32.google.com (mail-qv1-xf32.google.com [IPv6:2607:f8b0:4864:20::f32]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 35C00C06173B for ; Wed, 26 Jan 2022 10:34:39 -0800 (PST) Received: by mail-qv1-xf32.google.com with SMTP id i19so592075qvx.12 for ; Wed, 26 Jan 2022 10:34:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=gYivpEDsQjEq78GhHcAcDgg59f4QbW6Q5APhuV5RVYI=; b=PE3E80IyrXq8BCsBStLnLyvmcgNmV/BpCZmgYh+K7CEQaynk18nntY72LuFtCe7+ha e9s+aKdASKSnS081OwbL3xJFK1jhEuWqFt7PsB5bp5pa9tjcBRynTMdFH8CXbWj4B/xd k+VNwwpTTB2t0ppy9CwDHCTpZaP/pQjGE5H5oXAmrqW5EAJCmoWqnD34gpvK+1bH5TnQ ldZmu8X1VXTq40kLf8Y8enLhHNbaWFRCjbfITCbKAIwgxTWSbWYykxQEAYsCkIhPPYlD 2T3JlwV/pLOZxZ2PzS1Z+mkkWt91mujqKbavrDfOPwxP2roA7Yddmbz1UckgKSb4hThF JZcw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=gYivpEDsQjEq78GhHcAcDgg59f4QbW6Q5APhuV5RVYI=; b=cKus5X0pcr8/LJhSJ4oIcmBDWsxwLNzjq3TwqC9SjUtzGE/n7kt8DR59iCtbw0xhcE 6JhqV2TqYTLOQK2D33f1qETbHCDwrAOR207GP8PTx94vmxwEMyqgywmG6JEhcV+lKMR1 isoNsfKGlS8IkZIv05yCsGRMmHV4btiV1VKSf0O+9EmHY6uhKpljzEZn0gAQ2Op2yRJ6 vMBoPajXVNxKoCFom28D2OaoWRrIkSMnfgWUoT5mjM78CALuVsOHUFyLYcRWg5J2wL2M RrjZId33G8QBpFQkcYzF1PZ7JczUe18AH/cHdnUE88dDmeFbmOPmoTQJ0nF7ne9AinXp EQ/A== X-Gm-Message-State: AOAM531SUdnOHRKKbSIjlQ7VlNM1nF2EFhoBW7IUydswXXsI0H71GiIz 2S+XUPJMF1/XGqOHBXVokqPcyw== X-Google-Smtp-Source: ABdhPJwprdDGrDVv4RqpUnk5EiygoP6cVKM2Clr2Z/fCbgxkukmcUlI8BNQaQsir3IwuPZgHzOc71A== X-Received: by 2002:a05:6214:d6e:: with SMTP id 14mr22475789qvs.63.1643222078322; Wed, 26 Jan 2022 10:34:38 -0800 (PST) Received: from soleen.c.googlers.com.com (189.216.85.34.bc.googleusercontent.com. [34.85.216.189]) by smtp.gmail.com with ESMTPSA id u17sm35886qki.12.2022.01.26.10.34.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 26 Jan 2022 10:34:37 -0800 (PST) From: Pasha Tatashin To: pasha.tatashin@soleen.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-m68k@lists.linux-m68k.org, anshuman.khandual@arm.com, willy@infradead.org, akpm@linux-foundation.org, william.kucharski@oracle.com, mike.kravetz@oracle.com, vbabka@suse.cz, geert@linux-m68k.org, schmitzmic@gmail.com, rostedt@goodmis.org, mingo@redhat.com, hannes@cmpxchg.org, guro@fb.com, songmuchun@bytedance.com, weixugc@google.com, gthelen@google.com, rientjes@google.com, pjt@google.com, hughd@google.com Subject: [PATCH v3 6/9] mm: remove set_page_count() Date: Wed, 26 Jan 2022 18:34:26 +0000 Message-Id: <20220126183429.1840447-7-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.35.0.rc0.227.g00780c9af4-goog In-Reply-To: <20220126183429.1840447-1-pasha.tatashin@soleen.com> References: <20220126183429.1840447-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" set_page_count() is dangerous because it resets _refcount to an arbitrary value. Instead we now initialize _refcount to 1 only once, and the rest of the time we are using add/dec/cmpxchg to have a contiguous track of the counter. Remove set_page_count() and add new tracing hooks to page_ref_init(). Signed-off-by: Pasha Tatashin --- include/linux/page_ref.h | 27 ++++++++----------- include/trace/events/page_ref.h | 46 ++++++++++++++++++++++++++++----- mm/debug_page_ref.c | 8 +++--- 3 files changed, 54 insertions(+), 27 deletions(-) diff --git a/include/linux/page_ref.h b/include/linux/page_ref.h index 1af12a0d7ba1..d7316881626c 100644 --- a/include/linux/page_ref.h +++ b/include/linux/page_ref.h @@ -7,7 +7,7 @@ #include #include =20 -DECLARE_TRACEPOINT(page_ref_set); +DECLARE_TRACEPOINT(page_ref_init); DECLARE_TRACEPOINT(page_ref_mod); DECLARE_TRACEPOINT(page_ref_mod_and_test); DECLARE_TRACEPOINT(page_ref_mod_and_return); @@ -26,7 +26,7 @@ DECLARE_TRACEPOINT(page_ref_unfreeze); */ #define page_ref_tracepoint_active(t) tracepoint_enabled(t) =20 -extern void __page_ref_set(struct page *page, int v); +extern void __page_ref_init(struct page *page); extern void __page_ref_mod(struct page *page, int v); extern void __page_ref_mod_and_test(struct page *page, int v, int ret); extern void __page_ref_mod_and_return(struct page *page, int v, int ret); @@ -38,7 +38,7 @@ extern void __page_ref_unfreeze(struct page *page, int v); =20 #define page_ref_tracepoint_active(t) false =20 -static inline void __page_ref_set(struct page *page, int v) +static inline void __page_ref_init(struct page *page) { } static inline void __page_ref_mod(struct page *page, int v) @@ -94,18 +94,6 @@ static inline int page_count(const struct page *page) return folio_ref_count(page_folio(page)); } =20 -static inline void set_page_count(struct page *page, int v) -{ - atomic_set(&page->_refcount, v); - if (page_ref_tracepoint_active(page_ref_set)) - __page_ref_set(page, v); -} - -static inline void folio_set_count(struct folio *folio, int v) -{ - set_page_count(&folio->page, v); -} - /* * Setup the page refcount to one before being freed into the page allocat= or. * The memory might not be initialized and therefore there cannot be any @@ -116,7 +104,14 @@ static inline void folio_set_count(struct folio *folio= , int v) */ static inline void page_ref_init(struct page *page) { - set_page_count(page, 1); + atomic_set(&page->_refcount, 1); + if (page_ref_tracepoint_active(page_ref_init)) + __page_ref_init(page); +} + +static inline void folio_ref_init(struct folio *folio) +{ + page_ref_init(&folio->page); } =20 static inline int page_ref_add_return(struct page *page, int nr) diff --git a/include/trace/events/page_ref.h b/include/trace/events/page_re= f.h index 8a99c1cd417b..87551bb1df9e 100644 --- a/include/trace/events/page_ref.h +++ b/include/trace/events/page_ref.h @@ -10,6 +10,45 @@ #include #include =20 +DECLARE_EVENT_CLASS(page_ref_init_template, + + TP_PROTO(struct page *page), + + TP_ARGS(page), + + TP_STRUCT__entry( + __field(unsigned long, pfn) + __field(unsigned long, flags) + __field(int, count) + __field(int, mapcount) + __field(void *, mapping) + __field(int, mt) + __field(int, val) + ), + + TP_fast_assign( + __entry->pfn =3D page_to_pfn(page); + __entry->flags =3D page->flags; + __entry->count =3D page_ref_count(page); + __entry->mapcount =3D page_mapcount(page); + __entry->mapping =3D page->mapping; + __entry->mt =3D get_pageblock_migratetype(page); + ), + + TP_printk("pfn=3D0x%lx flags=3D%s count=3D%d mapcount=3D%d mapping=3D%p m= t=3D%d", + __entry->pfn, + show_page_flags(__entry->flags & PAGEFLAGS_MASK), + __entry->count, + __entry->mapcount, __entry->mapping, __entry->mt) +); + +DEFINE_EVENT(page_ref_init_template, page_ref_init, + + TP_PROTO(struct page *page), + + TP_ARGS(page) +); + DECLARE_EVENT_CLASS(page_ref_mod_template, =20 TP_PROTO(struct page *page, int v), @@ -44,13 +83,6 @@ DECLARE_EVENT_CLASS(page_ref_mod_template, __entry->val) ); =20 -DEFINE_EVENT(page_ref_mod_template, page_ref_set, - - TP_PROTO(struct page *page, int v), - - TP_ARGS(page, v) -); - DEFINE_EVENT(page_ref_mod_template, page_ref_mod, =20 TP_PROTO(struct page *page, int v), diff --git a/mm/debug_page_ref.c b/mm/debug_page_ref.c index f3b2c9d3ece2..e32149734122 100644 --- a/mm/debug_page_ref.c +++ b/mm/debug_page_ref.c @@ -5,12 +5,12 @@ #define CREATE_TRACE_POINTS #include =20 -void __page_ref_set(struct page *page, int v) +void __page_ref_init(struct page *page) { - trace_page_ref_set(page, v); + trace_page_ref_init(page); } -EXPORT_SYMBOL(__page_ref_set); -EXPORT_TRACEPOINT_SYMBOL(page_ref_set); +EXPORT_SYMBOL(__page_ref_init); +EXPORT_TRACEPOINT_SYMBOL(page_ref_init); =20 void __page_ref_mod(struct page *page, int v) { --=20 2.35.0.rc0.227.g00780c9af4-goog From nobody Wed Jul 1 13:26:37 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id AC290C63682 for ; Wed, 26 Jan 2022 18:34:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S244212AbiAZSew (ORCPT ); Wed, 26 Jan 2022 13:34:52 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60166 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S243388AbiAZSek (ORCPT ); Wed, 26 Jan 2022 13:34:40 -0500 Received: from mail-qv1-xf33.google.com (mail-qv1-xf33.google.com [IPv6:2607:f8b0:4864:20::f33]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 108A5C06161C for ; Wed, 26 Jan 2022 10:34:40 -0800 (PST) Received: by mail-qv1-xf33.google.com with SMTP id d8so644559qvv.2 for ; Wed, 26 Jan 2022 10:34:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=C+eeUKO5CVQ7ifoeaB9j3JsCJ7+YnH2kUfg22hZzWck=; b=XlIIm3wFS2VYZgNZKO5PJxQyYem5gwT2tE11aOfFUFxbLG8yhzwPtal/KkTwyPV6jq hq7je5EMTfT6eIOKF0UtJyZGr1AEA80ucW0fZFZ+kF+1ZAhRp7G19Ib/V2EsejOfjrI1 0k35qJFNU3ZaGcaTggA7hfMLQZc+gq+K8Fh2KbqC7/Dwdv9U7IVlrAAUt0EeGa5cz6ae SDb/wUBW1tkvRbnTEIoKDAiSr7Qy5GL0TJ9Eq+cgV9VvvI5FZtpeXiyi9roHEEO0sGY7 owIqf2CsLI7NGOad5+EtsSD7l8LN6widewoojQRFBiDTVsa0S9IXPM5RqIvlqslly+iL xLCw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=C+eeUKO5CVQ7ifoeaB9j3JsCJ7+YnH2kUfg22hZzWck=; b=UVE/jluTClGLedVZ6ppMpXNz7gK8WjUpKWOGCqQE+sMmnU7rNTrRuuSN0xE2gNr9G8 xPXaOEXpWpu/3RqgbJDmpwfInRfXfIP3J/DKUdEyoSA6QWX5juOnSpgjg8LmQIrRDe7q xNY2oA0Nf2A7Fc9GbOUnree2cj02VCLh6muRGoAeQJV9ytdz1SfKjgyDqcWZv44wwlo/ Xj4gsHRX3KuKvIhzJ3Sxmknc1RZ+/ABcszBqtHhdT8XPxyZEmU+lZc8yZel5I4wAsEgZ xNSv+kckrf7aeLc3I/aoRCR4M4KJCaLmR2WzYEOzer8JaKWQNtQ4dmAkil7TwPuGgU8f QC3w== X-Gm-Message-State: AOAM532E2Tr3rr/JVXfzNEYTkXHKTzxJCu4MJkGPKTIJUlbyRUjS0+Ey p+z/lAmJCDJjwkjQOV8n7J3zpg== X-Google-Smtp-Source: ABdhPJwtx1adUUMZjurVfu5lfu5E+GD/aExk1KJah16hTOTHhksPValmJe7VDZJFddjF2UR8dJc5hg== X-Received: by 2002:a05:6214:c82:: with SMTP id r2mr6785664qvr.81.1643222079173; Wed, 26 Jan 2022 10:34:39 -0800 (PST) Received: from soleen.c.googlers.com.com (189.216.85.34.bc.googleusercontent.com. [34.85.216.189]) by smtp.gmail.com with ESMTPSA id u17sm35886qki.12.2022.01.26.10.34.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 26 Jan 2022 10:34:38 -0800 (PST) From: Pasha Tatashin To: pasha.tatashin@soleen.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-m68k@lists.linux-m68k.org, anshuman.khandual@arm.com, willy@infradead.org, akpm@linux-foundation.org, william.kucharski@oracle.com, mike.kravetz@oracle.com, vbabka@suse.cz, geert@linux-m68k.org, schmitzmic@gmail.com, rostedt@goodmis.org, mingo@redhat.com, hannes@cmpxchg.org, guro@fb.com, songmuchun@bytedance.com, weixugc@google.com, gthelen@google.com, rientjes@google.com, pjt@google.com, hughd@google.com Subject: [PATCH v3 7/9] mm: simplify page_ref_* functions Date: Wed, 26 Jan 2022 18:34:27 +0000 Message-Id: <20220126183429.1840447-8-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.35.0.rc0.227.g00780c9af4-goog In-Reply-To: <20220126183429.1840447-1-pasha.tatashin@soleen.com> References: <20220126183429.1840447-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Now, that we are using atomic_fetch* variants to add/sub/inc/dec page _refcount, it makes sense to combined page_ref_* return and non return functions. Also remove some extra trace points for non-return variants. This improves the tracability by always recording the new _refcount value after the modifications has occurred. Signed-off-by: Pasha Tatashin --- include/linux/page_ref.h | 102 +++++++++----------------------- include/trace/events/page_ref.h | 18 +----- mm/debug_page_ref.c | 14 ----- 3 files changed, 31 insertions(+), 103 deletions(-) diff --git a/include/linux/page_ref.h b/include/linux/page_ref.h index d7316881626c..243fc60ae6c8 100644 --- a/include/linux/page_ref.h +++ b/include/linux/page_ref.h @@ -8,8 +8,6 @@ #include =20 DECLARE_TRACEPOINT(page_ref_init); -DECLARE_TRACEPOINT(page_ref_mod); -DECLARE_TRACEPOINT(page_ref_mod_and_test); DECLARE_TRACEPOINT(page_ref_mod_and_return); DECLARE_TRACEPOINT(page_ref_mod_unless); DECLARE_TRACEPOINT(page_ref_freeze); @@ -27,8 +25,6 @@ DECLARE_TRACEPOINT(page_ref_unfreeze); #define page_ref_tracepoint_active(t) tracepoint_enabled(t) =20 extern void __page_ref_init(struct page *page); -extern void __page_ref_mod(struct page *page, int v); -extern void __page_ref_mod_and_test(struct page *page, int v, int ret); extern void __page_ref_mod_and_return(struct page *page, int v, int ret); extern void __page_ref_mod_unless(struct page *page, int v, int u); extern void __page_ref_freeze(struct page *page, int v, int ret); @@ -41,12 +37,6 @@ extern void __page_ref_unfreeze(struct page *page, int v= ); static inline void __page_ref_init(struct page *page) { } -static inline void __page_ref_mod(struct page *page, int v) -{ -} -static inline void __page_ref_mod_and_test(struct page *page, int v, int r= et) -{ -} static inline void __page_ref_mod_and_return(struct page *page, int v, int= ret) { } @@ -127,12 +117,7 @@ static inline int page_ref_add_return(struct page *pag= e, int nr) =20 static inline void page_ref_add(struct page *page, int nr) { - int old_val =3D atomic_fetch_add(nr, &page->_refcount); - int new_val =3D old_val + nr; - - VM_BUG_ON_PAGE((unsigned int)new_val < (unsigned int)old_val, page); - if (page_ref_tracepoint_active(page_ref_mod)) - __page_ref_mod(page, nr); + page_ref_add_return(page, nr); } =20 static inline void folio_ref_add(struct folio *folio, int nr) @@ -140,30 +125,25 @@ static inline void folio_ref_add(struct folio *folio,= int nr) page_ref_add(&folio->page, nr); } =20 -static inline void page_ref_sub(struct page *page, int nr) +static inline int page_ref_sub_return(struct page *page, int nr) { int old_val =3D atomic_fetch_sub(nr, &page->_refcount); int new_val =3D old_val - nr; =20 VM_BUG_ON_PAGE((unsigned int)new_val > (unsigned int)old_val, page); - if (page_ref_tracepoint_active(page_ref_mod)) - __page_ref_mod(page, -nr); + if (page_ref_tracepoint_active(page_ref_mod_and_return)) + __page_ref_mod_and_return(page, -nr, new_val); + return new_val; } =20 -static inline void folio_ref_sub(struct folio *folio, int nr) +static inline void page_ref_sub(struct page *page, int nr) { - page_ref_sub(&folio->page, nr); + page_ref_sub_return(page, nr); } =20 -static inline int page_ref_sub_return(struct page *page, int nr) +static inline void folio_ref_sub(struct folio *folio, int nr) { - int old_val =3D atomic_fetch_sub(nr, &page->_refcount); - int new_val =3D old_val - nr; - - VM_BUG_ON_PAGE((unsigned int)new_val > (unsigned int)old_val, page); - if (page_ref_tracepoint_active(page_ref_mod_and_return)) - __page_ref_mod_and_return(page, -nr, new_val); - return new_val; + page_ref_sub(&folio->page, nr); } =20 static inline int folio_ref_sub_return(struct folio *folio, int nr) @@ -171,14 +151,20 @@ static inline int folio_ref_sub_return(struct folio *= folio, int nr) return page_ref_sub_return(&folio->page, nr); } =20 -static inline void page_ref_inc(struct page *page) +static inline int page_ref_inc_return(struct page *page) { int old_val =3D atomic_fetch_inc(&page->_refcount); int new_val =3D old_val + 1; =20 VM_BUG_ON_PAGE((unsigned int)new_val < (unsigned int)old_val, page); - if (page_ref_tracepoint_active(page_ref_mod)) - __page_ref_mod(page, 1); + if (page_ref_tracepoint_active(page_ref_mod_and_return)) + __page_ref_mod_and_return(page, 1, new_val); + return new_val; +} + +static inline void page_ref_inc(struct page *page) +{ + page_ref_inc_return(page); } =20 static inline void folio_ref_inc(struct folio *folio) @@ -186,14 +172,20 @@ static inline void folio_ref_inc(struct folio *folio) page_ref_inc(&folio->page); } =20 -static inline void page_ref_dec(struct page *page) +static inline int page_ref_dec_return(struct page *page) { int old_val =3D atomic_fetch_dec(&page->_refcount); int new_val =3D old_val - 1; =20 VM_BUG_ON_PAGE((unsigned int)new_val > (unsigned int)old_val, page); - if (page_ref_tracepoint_active(page_ref_mod)) - __page_ref_mod(page, -1); + if (page_ref_tracepoint_active(page_ref_mod_and_return)) + __page_ref_mod_and_return(page, -1, new_val); + return new_val; +} + +static inline void page_ref_dec(struct page *page) +{ + page_ref_dec_return(page); } =20 static inline void folio_ref_dec(struct folio *folio) @@ -203,14 +195,7 @@ static inline void folio_ref_dec(struct folio *folio) =20 static inline int page_ref_sub_and_test(struct page *page, int nr) { - int old_val =3D atomic_fetch_sub(nr, &page->_refcount); - int new_val =3D old_val - nr; - int ret =3D new_val =3D=3D 0; - - VM_BUG_ON_PAGE((unsigned int)new_val > (unsigned int)old_val, page); - if (page_ref_tracepoint_active(page_ref_mod_and_test)) - __page_ref_mod_and_test(page, -nr, ret); - return ret; + return page_ref_sub_return(page, nr) =3D=3D 0; } =20 static inline int folio_ref_sub_and_test(struct folio *folio, int nr) @@ -218,17 +203,6 @@ static inline int folio_ref_sub_and_test(struct folio = *folio, int nr) return page_ref_sub_and_test(&folio->page, nr); } =20 -static inline int page_ref_inc_return(struct page *page) -{ - int old_val =3D atomic_fetch_inc(&page->_refcount); - int new_val =3D old_val + 1; - - VM_BUG_ON_PAGE((unsigned int)new_val < (unsigned int)old_val, page); - if (page_ref_tracepoint_active(page_ref_mod_and_return)) - __page_ref_mod_and_return(page, 1, new_val); - return new_val; -} - static inline int folio_ref_inc_return(struct folio *folio) { return page_ref_inc_return(&folio->page); @@ -236,14 +210,7 @@ static inline int folio_ref_inc_return(struct folio *f= olio) =20 static inline int page_ref_dec_and_test(struct page *page) { - int old_val =3D atomic_fetch_dec(&page->_refcount); - int new_val =3D old_val - 1; - int ret =3D new_val =3D=3D 0; - - VM_BUG_ON_PAGE((unsigned int)new_val > (unsigned int)old_val, page); - if (page_ref_tracepoint_active(page_ref_mod_and_test)) - __page_ref_mod_and_test(page, -1, ret); - return ret; + return page_ref_dec_return(page) =3D=3D 0; } =20 static inline int folio_ref_dec_and_test(struct folio *folio) @@ -251,17 +218,6 @@ static inline int folio_ref_dec_and_test(struct folio = *folio) return page_ref_dec_and_test(&folio->page); } =20 -static inline int page_ref_dec_return(struct page *page) -{ - int old_val =3D atomic_fetch_dec(&page->_refcount); - int new_val =3D old_val - 1; - - VM_BUG_ON_PAGE((unsigned int)new_val > (unsigned int)old_val, page); - if (page_ref_tracepoint_active(page_ref_mod_and_return)) - __page_ref_mod_and_return(page, -1, new_val); - return new_val; -} - static inline int folio_ref_dec_return(struct folio *folio) { return page_ref_dec_return(&folio->page); diff --git a/include/trace/events/page_ref.h b/include/trace/events/page_re= f.h index 87551bb1df9e..35cd795aa7c6 100644 --- a/include/trace/events/page_ref.h +++ b/include/trace/events/page_ref.h @@ -49,7 +49,7 @@ DEFINE_EVENT(page_ref_init_template, page_ref_init, TP_ARGS(page) ); =20 -DECLARE_EVENT_CLASS(page_ref_mod_template, +DECLARE_EVENT_CLASS(page_ref_unfreeze_template, =20 TP_PROTO(struct page *page, int v), =20 @@ -83,13 +83,6 @@ DECLARE_EVENT_CLASS(page_ref_mod_template, __entry->val) ); =20 -DEFINE_EVENT(page_ref_mod_template, page_ref_mod, - - TP_PROTO(struct page *page, int v), - - TP_ARGS(page, v) -); - DECLARE_EVENT_CLASS(page_ref_mod_and_test_template, =20 TP_PROTO(struct page *page, int v, int ret), @@ -126,13 +119,6 @@ DECLARE_EVENT_CLASS(page_ref_mod_and_test_template, __entry->val, __entry->ret) ); =20 -DEFINE_EVENT(page_ref_mod_and_test_template, page_ref_mod_and_test, - - TP_PROTO(struct page *page, int v, int ret), - - TP_ARGS(page, v, ret) -); - DEFINE_EVENT(page_ref_mod_and_test_template, page_ref_mod_and_return, =20 TP_PROTO(struct page *page, int v, int ret), @@ -154,7 +140,7 @@ DEFINE_EVENT(page_ref_mod_and_test_template, page_ref_f= reeze, TP_ARGS(page, v, ret) ); =20 -DEFINE_EVENT(page_ref_mod_template, page_ref_unfreeze, +DEFINE_EVENT(page_ref_unfreeze_template, page_ref_unfreeze, =20 TP_PROTO(struct page *page, int v), =20 diff --git a/mm/debug_page_ref.c b/mm/debug_page_ref.c index e32149734122..1de9d93cca25 100644 --- a/mm/debug_page_ref.c +++ b/mm/debug_page_ref.c @@ -12,20 +12,6 @@ void __page_ref_init(struct page *page) EXPORT_SYMBOL(__page_ref_init); EXPORT_TRACEPOINT_SYMBOL(page_ref_init); =20 -void __page_ref_mod(struct page *page, int v) -{ - trace_page_ref_mod(page, v); -} -EXPORT_SYMBOL(__page_ref_mod); -EXPORT_TRACEPOINT_SYMBOL(page_ref_mod); - -void __page_ref_mod_and_test(struct page *page, int v, int ret) -{ - trace_page_ref_mod_and_test(page, v, ret); -} -EXPORT_SYMBOL(__page_ref_mod_and_test); -EXPORT_TRACEPOINT_SYMBOL(page_ref_mod_and_test); - void __page_ref_mod_and_return(struct page *page, int v, int ret) { trace_page_ref_mod_and_return(page, v, ret); --=20 2.35.0.rc0.227.g00780c9af4-goog From nobody Wed Jul 1 13:26:37 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8A74FC28CF5 for ; Wed, 26 Jan 2022 18:34:57 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S244203AbiAZSe4 (ORCPT ); Wed, 26 Jan 2022 13:34:56 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60178 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S244098AbiAZSel (ORCPT ); Wed, 26 Jan 2022 13:34:41 -0500 Received: from mail-qv1-xf36.google.com (mail-qv1-xf36.google.com [IPv6:2607:f8b0:4864:20::f36]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DE541C061748 for ; Wed, 26 Jan 2022 10:34:40 -0800 (PST) Received: by mail-qv1-xf36.google.com with SMTP id i19so592138qvx.12 for ; Wed, 26 Jan 2022 10:34:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=9qGtluafW49WVeKC1+T5xXe9Gi8hCvJzE0QRf1Gctb0=; b=WLC8zAaBt+pGLb2dprhgeTFhzFiOz0Dwgv2sKq4nVV6mme5Ik3TLKMC25II36bZZ5S nkiFdbElwzoNvpJfm4IKKcbz4QHq3OId66HMHWWICIgu3RfIP3Y28sd06cMaFCGtpsoS og++vqB9AwLOYf7hTuIUOpHHMLg2tMzq5LGyO4/ee6B06PVN72Mnxy/Dau54Nz815L0m CT7F1/eD+VSzR33uglaTP7hK7ASfdfmOKDZ4FSRaQjQRm6Pxa7fA7N4bO8kgwtKy7d54 bCo3neAgZVI3g2RjtaU1yvqnnOVddQ/gjMInuoUFYJvOFvmOj49aqKxoHSPnVsaElEQ5 B3Mw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=9qGtluafW49WVeKC1+T5xXe9Gi8hCvJzE0QRf1Gctb0=; b=AyOO/Ntbj6Y5jki7ORtR7U6jg+X5xipBtc4d7cXXeL2GCiZKd3u0y9KLTbtaSqdkY3 WMZsUXSrxiTbG12u2RfwPjpQCFAH/BaaRNY8wYzfTkYb9nWhBRkRSAe6/7gAX+WQo17y QnJAs/x4o9soeHiw1twQcuLmiQ1QctanhtMX90NwQQmjspF5jdFHd+b2VyWl8KBMSdaN rNWTgnqoobxVXDh2ZZvZqv2Zquu3XfaNvk6Tiq73KJXG48MrfRtMJCV6KWdkK0J4VaLZ WzhvvnzYZlpPR9bxXYs3HIacZze0NNRkswz+AiGToaduv+QGLhzDhkjdcnuryRFmfcTN 3Ydg== X-Gm-Message-State: AOAM531mP4KIM+Wuc5iFtbNI6bO9iRCtSrweM/htb1Tt4Z+NlcN+5ozQ HOlbCRtPaDrN+wPlbpuQMD0K2Q== X-Google-Smtp-Source: ABdhPJxJtoQHYrpWNf8C4q9GS25k4xHwcCuWK9f4FFFmUET3LankryAhv5s7uODB4uZa4iJOyGzaCg== X-Received: by 2002:a05:6214:1ccb:: with SMTP id g11mr25729042qvd.97.1643222080024; Wed, 26 Jan 2022 10:34:40 -0800 (PST) Received: from soleen.c.googlers.com.com (189.216.85.34.bc.googleusercontent.com. [34.85.216.189]) by smtp.gmail.com with ESMTPSA id u17sm35886qki.12.2022.01.26.10.34.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 26 Jan 2022 10:34:39 -0800 (PST) From: Pasha Tatashin To: pasha.tatashin@soleen.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-m68k@lists.linux-m68k.org, anshuman.khandual@arm.com, willy@infradead.org, akpm@linux-foundation.org, william.kucharski@oracle.com, mike.kravetz@oracle.com, vbabka@suse.cz, geert@linux-m68k.org, schmitzmic@gmail.com, rostedt@goodmis.org, mingo@redhat.com, hannes@cmpxchg.org, guro@fb.com, songmuchun@bytedance.com, weixugc@google.com, gthelen@google.com, rientjes@google.com, pjt@google.com, hughd@google.com Subject: [PATCH v3 8/9] mm: do not use atomic_set_release in page_ref_unfreeze() Date: Wed, 26 Jan 2022 18:34:28 +0000 Message-Id: <20220126183429.1840447-9-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.35.0.rc0.227.g00780c9af4-goog In-Reply-To: <20220126183429.1840447-1-pasha.tatashin@soleen.com> References: <20220126183429.1840447-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" In we set the old _refcount value after verifying that the old value was indeed 0. VM_BUG_ON_PAGE(page_count(page) !=3D 0, page); < the _refcount may change here> atomic_set_release(&page->_refcount, count); To avoid the smal gap where _refcount may change lets verify the time of the _refcount at the time of the set operation. Use atomic_xchg_release() and at the set time verify that the value was 0. Signed-off-by: Pasha Tatashin --- include/linux/page_ref.h | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/include/linux/page_ref.h b/include/linux/page_ref.h index 243fc60ae6c8..9efabeff4e06 100644 --- a/include/linux/page_ref.h +++ b/include/linux/page_ref.h @@ -322,10 +322,9 @@ static inline int folio_ref_freeze(struct folio *folio= , int count) =20 static inline void page_ref_unfreeze(struct page *page, int count) { - VM_BUG_ON_PAGE(page_count(page) !=3D 0, page); - VM_BUG_ON(count =3D=3D 0); + int old_val =3D atomic_xchg_release(&page->_refcount, count); =20 - atomic_set_release(&page->_refcount, count); + VM_BUG_ON_PAGE(count =3D=3D 0 || old_val !=3D 0, page); if (page_ref_tracepoint_active(page_ref_unfreeze)) __page_ref_unfreeze(page, count); } --=20 2.35.0.rc0.227.g00780c9af4-goog From nobody Wed Jul 1 13:26:37 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B6C41C28CF5 for ; Wed, 26 Jan 2022 18:34:59 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S244215AbiAZSe6 (ORCPT ); Wed, 26 Jan 2022 13:34:58 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60180 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S244138AbiAZSel (ORCPT ); Wed, 26 Jan 2022 13:34:41 -0500 Received: from mail-qv1-xf2c.google.com (mail-qv1-xf2c.google.com [IPv6:2607:f8b0:4864:20::f2c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BE8BCC061749 for ; Wed, 26 Jan 2022 10:34:41 -0800 (PST) Received: by mail-qv1-xf2c.google.com with SMTP id k9so607751qvv.9 for ; Wed, 26 Jan 2022 10:34:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=l8kTegpTqMksygyMNd7O9r7qJ6myjdhmrXkVSbv/IzA=; b=dcgO/hZ6elZPBdqxd4MboPCo3EmolMnIS887io5RaZKDoH7H3DsIKjWe4xL0M92rAN NGAs7DZ4N0Cm3Mu1BqA/u4YPeEnAejo0/rDUXvOGHwdQqDTzgDduz/YPu8cTYag6DNR2 pZks0Czhk7nN87F+uqzoh7CXPfDmtA2bXmMEn5cLDfgC74eDx4rtj9zlO5y9Gz/6RdQY QYtkCpcN6yotcPm74Tik/4EYy8j0Ppcfy9AOeQyVai8riP0oJXWDwXI7oyJNNCbj4jxC H/r9ApPE7qNwyGQk9KJq0YaO8bT0ni9gz3Tm8obbCQhHb9KIqrUiniiwj2Db7oc6jaKC ByKQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=l8kTegpTqMksygyMNd7O9r7qJ6myjdhmrXkVSbv/IzA=; b=ZzSUiejvpBeUd/M4O9kbf5kFlrVbqCzvE+C5r+54cAC4sLQWQHRevJlfQZf6KPM2hh Dohz8WCslGFsd6OmV5dVPaB9NHBstzOK37BySm3ZqsppTFyfuLPzX+U7FKmN1Q/JZ/FX bEQRbK6iUpjMgwKqeQ655EULOsIQL9Y5vLCOZcoAIMU8kgoq1cmFTGiFjnn8BDg2e3AJ YtZ98r+5Be96fRsMyxJYAANQ2d9FvCJzIGJeAiOPKW+6VTj4+PYDwI6Iztv5dv3FFfXZ C8yWw4VEQ12PcsJmV4dttJguyJnYy6zOKlrkf7jtZSk+VHa6WliSXAblCfm1o/6pCCqo FfCA== X-Gm-Message-State: AOAM532MYRnKuU3mZuDmZ1lrSQ/y6VA+OKvmOPFrTqncq+FHYNX0yBT0 6ziiqUFAJazs/ECO0cHk7tOl4A== X-Google-Smtp-Source: ABdhPJx1uUL1QguAQjuOjaCUHuB84L0yLxyE8YceJm8x5PIc7mOwOmi6RJ2+VMz5b+XHNn1KPA5orA== X-Received: by 2002:ad4:5ca4:: with SMTP id q4mr24120qvh.64.1643222080942; Wed, 26 Jan 2022 10:34:40 -0800 (PST) Received: from soleen.c.googlers.com.com (189.216.85.34.bc.googleusercontent.com. [34.85.216.189]) by smtp.gmail.com with ESMTPSA id u17sm35886qki.12.2022.01.26.10.34.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 26 Jan 2022 10:34:40 -0800 (PST) From: Pasha Tatashin To: pasha.tatashin@soleen.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-m68k@lists.linux-m68k.org, anshuman.khandual@arm.com, willy@infradead.org, akpm@linux-foundation.org, william.kucharski@oracle.com, mike.kravetz@oracle.com, vbabka@suse.cz, geert@linux-m68k.org, schmitzmic@gmail.com, rostedt@goodmis.org, mingo@redhat.com, hannes@cmpxchg.org, guro@fb.com, songmuchun@bytedance.com, weixugc@google.com, gthelen@google.com, rientjes@google.com, pjt@google.com, hughd@google.com Subject: [PATCH v3 9/9] mm: use atomic_cmpxchg_acquire in page_ref_freeze(). Date: Wed, 26 Jan 2022 18:34:29 +0000 Message-Id: <20220126183429.1840447-10-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.35.0.rc0.227.g00780c9af4-goog In-Reply-To: <20220126183429.1840447-1-pasha.tatashin@soleen.com> References: <20220126183429.1840447-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" page_ref_freeze and page_ref_unfreeze are designed to be used as a pair. They protect critical sections where struct page can be modified. page_ref_unfreeze() is protected by _release() atomic operation, but page_ref_freeze() is not as it is assumed that cmpxch provides the full barrier. Instead, use the appropriate atomic_cmpxchg_acquire() to ensure that memory model is excplicitly followed. Signed-off-by: Pasha Tatashin --- include/linux/page_ref.h | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/include/linux/page_ref.h b/include/linux/page_ref.h index 9efabeff4e06..45be731d8919 100644 --- a/include/linux/page_ref.h +++ b/include/linux/page_ref.h @@ -308,7 +308,8 @@ static inline bool folio_try_get_rcu(struct folio *foli= o) =20 static inline int page_ref_freeze(struct page *page, int count) { - int ret =3D likely(atomic_cmpxchg(&page->_refcount, count, 0) =3D=3D coun= t); + int old_val =3D atomic_cmpxchg_acquire(&page->_refcount, count, 0); + int ret =3D likely(old_val =3D=3D count); =20 if (page_ref_tracepoint_active(page_ref_freeze)) __page_ref_freeze(page, count, ret); --=20 2.35.0.rc0.227.g00780c9af4-goog