From nobody Wed Sep 17 23:54:39 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B7C7BC4332F for ; Wed, 14 Dec 2022 02:52:26 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237096AbiLNCwZ (ORCPT ); Tue, 13 Dec 2022 21:52:25 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46232 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236890AbiLNCwL (ORCPT ); Tue, 13 Dec 2022 21:52:11 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D6F6217A8D for ; Tue, 13 Dec 2022 18:51:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1670986286; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=YvXYXT7+pZNjN3bsp9+0AEDOU4OMWFYmE3SAXAaEld0=; b=JbmO1LSzu+tuhIo6GXcle8PSPwzWYI4jx7oGBCJWKpgLiv+0a1/L5R7FweGmLTJhsdJlpI OrYIV3LU+QbiwRY9nit1vw2kqUaN7WkfXGnut4H28ejtC8R/HB9mNJRuNskvaxX8O0URkm NLeebrY0RbHh5pEOlpXmJ/JaWhvM69w= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-290-pa7vWdvINQiKRuTjDpGF-w-1; Tue, 13 Dec 2022 21:51:22 -0500 X-MC-Unique: pa7vWdvINQiKRuTjDpGF-w-1 Received: from smtp.corp.redhat.com (int-mx10.intmail.prod.int.rdu2.redhat.com [10.11.54.10]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 327A0806002; Wed, 14 Dec 2022 02:51:22 +0000 (UTC) Received: from localhost (ovpn-8-24.pek2.redhat.com [10.72.8.24]) by smtp.corp.redhat.com (Postfix) with ESMTP id 4FA9A492B00; Wed, 14 Dec 2022 02:51:20 +0000 (UTC) From: Ming Lei To: Jens Axboe , Tejun Heo Cc: linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, Zhong Jinghua , Yu Kuai , Dennis Zhou , Ming Lei Subject: [PATCH 1/3] lib/percpu-refcount: support to exit refcount automatically during releasing Date: Wed, 14 Dec 2022 10:50:59 +0800 Message-Id: <20221214025101.1268437-2-ming.lei@redhat.com> In-Reply-To: <20221214025101.1268437-1-ming.lei@redhat.com> References: <20221214025101.1268437-1-ming.lei@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.1 on 10.11.54.10 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" We only have two users in which percpu_ref_exit() is called from ->release(). Add flag of PERCPU_REF_AUTO_EXIT for avoiding to call percpu_ref_exit() from ->release() directly since we need to drain ->release() in percpu_ref_exit() for fixing use-after-free. Signed-off-by: Ming Lei --- include/linux/percpu-refcount.h | 21 +++++++++++++++++++-- lib/percpu-refcount.c | 9 ++++++--- 2 files changed, 25 insertions(+), 5 deletions(-) diff --git a/include/linux/percpu-refcount.h b/include/linux/percpu-refcoun= t.h index d73a1c08c3e3..006c6aae261e 100644 --- a/include/linux/percpu-refcount.h +++ b/include/linux/percpu-refcount.h @@ -90,6 +90,11 @@ enum { * Allow switching from atomic mode to percpu mode. */ PERCPU_REF_ALLOW_REINIT =3D 1 << 2, + + /* + * call percpu_ref_exit() when releasing + */ + PERCPU_REF_AUTO_EXIT =3D 1 << 3, }; =20 struct percpu_ref_data { @@ -98,6 +103,7 @@ struct percpu_ref_data { percpu_ref_func_t *confirm_switch; bool force_atomic:1; bool allow_reinit:1; + bool auto_exit:1; struct rcu_head rcu; struct percpu_ref *ref; }; @@ -331,8 +337,19 @@ static inline void percpu_ref_put_many(struct percpu_r= ef *ref, unsigned long nr) =20 if (__ref_is_percpu(ref, &percpu_count)) this_cpu_sub(*percpu_count, nr); - else if (unlikely(atomic_long_sub_and_test(nr, &ref->data->count))) - ref->data->release(ref); + else { + struct percpu_ref_data *data =3D ref->data; + struct percpu_ref copy =3D *ref; + bool release =3D false; + + if (unlikely(atomic_long_sub_and_test(nr, &data->count))) { + data->release(ref); + release =3D true; + } + + if (release && data->auto_exit) + percpu_ref_exit(©); + } =20 rcu_read_unlock(); } diff --git a/lib/percpu-refcount.c b/lib/percpu-refcount.c index 668f6aa6a75d..c0cadf92948f 100644 --- a/lib/percpu-refcount.c +++ b/lib/percpu-refcount.c @@ -82,6 +82,7 @@ int percpu_ref_init(struct percpu_ref *ref, percpu_ref_fu= nc_t *release, =20 data->force_atomic =3D flags & PERCPU_REF_INIT_ATOMIC; data->allow_reinit =3D flags & PERCPU_REF_ALLOW_REINIT; + data->auto_exit =3D flags & PERCPU_REF_AUTO_EXIT; =20 if (flags & (PERCPU_REF_INIT_ATOMIC | PERCPU_REF_INIT_DEAD)) { ref->percpu_count_ptr |=3D __PERCPU_REF_ATOMIC; @@ -123,9 +124,11 @@ static void __percpu_ref_exit(struct percpu_ref *ref) * * This function exits @ref. The caller is responsible for ensuring that * @ref is no longer in active use. The usual places to invoke this - * function from are the @ref->release() callback or in init failure path - * where percpu_ref_init() succeeded but other parts of the initialization - * of the embedding object failed. + * function from are where the refcounter is confirmed as idle or in init + * failure path where percpu_ref_init() succeeded but other parts of the + * initialization of the embedding object failed. For caller which needs + * to call percpu_ref_exit() in ->release, please pass PERCPU_REF_AUTO_EXIT + * to percpu_ref_init(). */ void percpu_ref_exit(struct percpu_ref *ref) { --=20 2.38.1 From nobody Wed Sep 17 23:54:39 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 62FEAC4332F for ; Wed, 14 Dec 2022 02:52:19 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237047AbiLNCwR (ORCPT ); Tue, 13 Dec 2022 21:52:17 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46256 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236731AbiLNCwK (ORCPT ); Tue, 13 Dec 2022 21:52:10 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 29B5E22538 for ; Tue, 13 Dec 2022 18:51:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1670986290; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=BYOjfeiBRlo83yXPXn1TZ8OQl1UZe5v7B1C1yln8/XY=; b=W4GYjbyu3o+FAp6/iO7b7tMJAVf3nLarRQVI5BwllOD7hByuMqz5Im3y8b3l/pdgaWUk6u 0xRS2ROvDVZI22cW66wWgMY8UkRUI46eoAhAUbwltNT69eYWq0A/FuB8JnD2WOX1qGPKA+ 38w4G+SQFrUoH6a/zaHBJ0jICDHt1sM= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-261-TswRycF3MN-7n1xDGsI2kQ-1; Tue, 13 Dec 2022 21:51:27 -0500 X-MC-Unique: TswRycF3MN-7n1xDGsI2kQ-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 787843832781; Wed, 14 Dec 2022 02:51:26 +0000 (UTC) Received: from localhost (ovpn-8-24.pek2.redhat.com [10.72.8.24]) by smtp.corp.redhat.com (Postfix) with ESMTP id 8F6B51121314; Wed, 14 Dec 2022 02:51:25 +0000 (UTC) From: Ming Lei To: Jens Axboe , Tejun Heo Cc: linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, Zhong Jinghua , Yu Kuai , Dennis Zhou , Ming Lei Subject: [PATCH 2/3] lib/percpu-refcount: apply PERCPU_REF_AUTO_EXIT Date: Wed, 14 Dec 2022 10:51:00 +0800 Message-Id: <20221214025101.1268437-3-ming.lei@redhat.com> In-Reply-To: <20221214025101.1268437-1-ming.lei@redhat.com> References: <20221214025101.1268437-1-ming.lei@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.1 on 10.11.54.3 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Apply the added new flag of PERCPU_REF_AUTO_EXIT, so that users needn't to call percpu_ref_exit() in ->release() any more. Signed-off-by: Ming Lei --- drivers/infiniband/ulp/rtrs/rtrs-srv.c | 4 ++-- mm/memcontrol.c | 5 ++--- 2 files changed, 4 insertions(+), 5 deletions(-) diff --git a/drivers/infiniband/ulp/rtrs/rtrs-srv.c b/drivers/infiniband/ul= p/rtrs/rtrs-srv.c index 22d7ba05e9fe..07d1af3fad28 100644 --- a/drivers/infiniband/ulp/rtrs/rtrs-srv.c +++ b/drivers/infiniband/ulp/rtrs/rtrs-srv.c @@ -122,7 +122,6 @@ static inline void rtrs_srv_inflight_ref_release(struct= percpu_ref *ref) struct rtrs_srv_path, ids_inflight_ref); =20 - percpu_ref_exit(&srv_path->ids_inflight_ref); complete(&srv_path->complete_done); } =20 @@ -147,7 +146,8 @@ static int rtrs_srv_alloc_ops_ids(struct rtrs_srv_path = *srv_path) } =20 ret =3D percpu_ref_init(&srv_path->ids_inflight_ref, - rtrs_srv_inflight_ref_release, 0, GFP_KERNEL); + rtrs_srv_inflight_ref_release, + PERCPU_REF_AUTO_EXIT, GFP_KERNEL); if (ret) { pr_err("Percpu reference init failed\n"); goto err; diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 266a1ab05434..eaca7d16b143 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -294,7 +294,6 @@ static void obj_cgroup_release(struct percpu_ref *ref) list_del(&objcg->list); spin_unlock_irqrestore(&objcg_lock, flags); =20 - percpu_ref_exit(ref); kfree_rcu(objcg, rcu); } =20 @@ -307,8 +306,8 @@ static struct obj_cgroup *obj_cgroup_alloc(void) if (!objcg) return NULL; =20 - ret =3D percpu_ref_init(&objcg->refcnt, obj_cgroup_release, 0, - GFP_KERNEL); + ret =3D percpu_ref_init(&objcg->refcnt, obj_cgroup_release, + PERCPU_REF_AUTO_EXIT, GFP_KERNEL); if (ret) { kfree(objcg); return NULL; --=20 2.38.1 From nobody Wed Sep 17 23:54:39 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B38E3C4332F for ; Wed, 14 Dec 2022 02:52:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237195AbiLNCwg (ORCPT ); Tue, 13 Dec 2022 21:52:36 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46262 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236608AbiLNCwV (ORCPT ); Tue, 13 Dec 2022 21:52:21 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6DADA17AB5 for ; Tue, 13 Dec 2022 18:51:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1670986292; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=QJTsGV35CYOraEoRISqpKpr/2uwkfrZs9x/0odiJU0M=; b=cfca+OB7wUvQs+nnyPs5gOIEsfk3zUtlcqHM4lL17r//GII9ubIz5fkVI8epPrbtv94c0b QS3LwDYSuKwb7ZL1meRlHktYti/n4Jbkvr2pZDNSIek/0r0NL2AMFylT4q8XFf77UVfc8c MaggVteFr8nKv3IT2BzPxzUmWBQ/3qo= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-279-J8NTIGfpNlG_jTgAlDFv2Q-1; Tue, 13 Dec 2022 21:51:31 -0500 X-MC-Unique: J8NTIGfpNlG_jTgAlDFv2Q-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id C42023832786; Wed, 14 Dec 2022 02:51:30 +0000 (UTC) Received: from localhost (ovpn-8-24.pek2.redhat.com [10.72.8.24]) by smtp.corp.redhat.com (Postfix) with ESMTP id ECD2540ED784; Wed, 14 Dec 2022 02:51:29 +0000 (UTC) From: Ming Lei To: Jens Axboe , Tejun Heo Cc: linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, Zhong Jinghua , Yu Kuai , Dennis Zhou , Ming Lei Subject: [PATCH 3/3] lib/percpu-refcount: drain ->release() in perpcu_ref_exit() Date: Wed, 14 Dec 2022 10:51:01 +0800 Message-Id: <20221214025101.1268437-4-ming.lei@redhat.com> In-Reply-To: <20221214025101.1268437-1-ming.lei@redhat.com> References: <20221214025101.1268437-1-ming.lei@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.1 on 10.11.54.2 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" The pattern of wait_event(percpu_ref_is_zero()) has been used in several kernel components, and this way actually has the following risk: - percpu_ref_is_zero() can be returned just between atomic_long_sub_and_test() and ref->data->release(ref) - given the refcount is found as zero, percpu_ref_exit() could be called, and the host data structure is freed - then use-after-free is triggered in ->release() when the user host data structure is freed after percpu_ref_exit() returns Reported-by: Zhong Jinghua Fixes: 2b0d3d3e4fcf ("percpu_ref: reduce memory footprint of percpu_ref in = fast path") Signed-off-by: Ming Lei --- include/linux/percpu-refcount.h | 41 ++++++++++++++++++++++----------- lib/percpu-refcount.c | 22 ++++++++++++++++++ 2 files changed, 50 insertions(+), 13 deletions(-) diff --git a/include/linux/percpu-refcount.h b/include/linux/percpu-refcoun= t.h index 006c6aae261e..6ef29ebffd58 100644 --- a/include/linux/percpu-refcount.h +++ b/include/linux/percpu-refcount.h @@ -55,6 +55,7 @@ #include #include #include +#include =20 struct percpu_ref; typedef void (percpu_ref_func_t)(struct percpu_ref *); @@ -104,6 +105,7 @@ struct percpu_ref_data { bool force_atomic:1; bool allow_reinit:1; bool auto_exit:1; + bool being_release:1; struct rcu_head rcu; struct percpu_ref *ref; }; @@ -137,6 +139,7 @@ void percpu_ref_kill_and_confirm(struct percpu_ref *ref, void percpu_ref_resurrect(struct percpu_ref *ref); void percpu_ref_reinit(struct percpu_ref *ref); bool percpu_ref_is_zero(struct percpu_ref *ref); +wait_queue_head_t *percpu_ref_get_switch_waitq(void); =20 /** * percpu_ref_kill - drop the initial ref @@ -319,6 +322,29 @@ static inline bool percpu_ref_tryget_live(struct percp= u_ref *ref) return ret; } =20 +/* Internal helper, please do not call it outside */ +static inline void __percpu_ref_put_many(struct percpu_ref *ref, + unsigned long nr) +{ + struct percpu_ref_data *data =3D ref->data; + struct percpu_ref copy =3D *ref; + bool release =3D false; + + data->being_release =3D 1; + if (unlikely(atomic_long_sub_and_test(nr, &data->count))) { + data->release(ref); + release =3D true; + } + data->being_release =3D 0; + + if (release) { + if (data->auto_exit) + percpu_ref_exit(©); + /* re-use switch waitq for ack the release done */ + wake_up_all(percpu_ref_get_switch_waitq()); + } +} + /** * percpu_ref_put_many - decrement a percpu refcount * @ref: percpu_ref to put @@ -337,19 +363,8 @@ static inline void percpu_ref_put_many(struct percpu_r= ef *ref, unsigned long nr) =20 if (__ref_is_percpu(ref, &percpu_count)) this_cpu_sub(*percpu_count, nr); - else { - struct percpu_ref_data *data =3D ref->data; - struct percpu_ref copy =3D *ref; - bool release =3D false; - - if (unlikely(atomic_long_sub_and_test(nr, &data->count))) { - data->release(ref); - release =3D true; - } - - if (release && data->auto_exit) - percpu_ref_exit(©); - } + else + __percpu_ref_put_many(ref, nr); =20 rcu_read_unlock(); } diff --git a/lib/percpu-refcount.c b/lib/percpu-refcount.c index c0cadf92948f..fd50eda233ed 100644 --- a/lib/percpu-refcount.c +++ b/lib/percpu-refcount.c @@ -140,6 +140,22 @@ void percpu_ref_exit(struct percpu_ref *ref) if (!data) return; =20 + /* + * We may reach here because wait_event(percpu_ref_is_zero()) + * returns, and ->release() may not be completed or even started + * ye, then use-after-free is caused, so drain ->release() here + */ + if (!data->auto_exit) { + /* + * Order reading the atomic count in percpu_ref_is_zero + * and reading data->being_release. The counter pair is + * the one implied in atomic_long_sub_and_test() called + * from __percpu_ref_put_many(). + */ + smp_rmb(); + wait_event(percpu_ref_switch_waitq, !data->being_release); + } + spin_lock_irqsave(&percpu_ref_switch_lock, flags); ref->percpu_count_ptr |=3D atomic_long_read(&ref->data->count) << __PERCPU_REF_FLAG_BITS; @@ -480,3 +496,9 @@ void percpu_ref_resurrect(struct percpu_ref *ref) spin_unlock_irqrestore(&percpu_ref_switch_lock, flags); } EXPORT_SYMBOL_GPL(percpu_ref_resurrect); + +wait_queue_head_t *percpu_ref_get_switch_waitq() +{ + return &percpu_ref_switch_waitq; +} +EXPORT_SYMBOL_GPL(percpu_ref_get_switch_waitq); --=20 2.38.1