From nobody Tue Dec 23 14:23:47 2025 Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 295DF16EC19 for ; Mon, 13 Jan 2025 04:35:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=205.220.177.32 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1736742960; cv=none; b=oYF3XsQxZdfnHifKafOmvhJChuv75WDVxeeke+c3bzUkWBnlwj87yneS54yOIeata2nvivOsipszRxZnHHr3iu/p3tX/GysksMsYoGEGXbJuphCg7EpOMkwXkP+lcja2ois/9YvcxWSQvk1Vux56SH7HGyb7yUppAFiphxn0Ty8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1736742960; c=relaxed/simple; bh=msxzXHuX/TSxCIPc+2ekBjZNruySqTTF+eV2Ab5HEMk=; h=From:To:Cc:Subject:Date:Message-Id:MIME-Version; b=FoYpkIFvLhk3NZ7gCGSgkKMhZwriwcKg6HyyvlzwuvFOSQHT4jlx1o6+tw9jDh/RUq0RfOb/YoyKcKYSvKOxSogMJokoofWi4w0JZDii2sW/zWqarf706d7pXCJfpFkfyv1hg4IZHW2moePRziGS5BKrVanADLVohb6sZ/EOIaw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=oracle.com; spf=pass smtp.mailfrom=oracle.com; dkim=pass (2048-bit key) header.d=oracle.com header.i=@oracle.com header.b=oekZpsGL; arc=none smtp.client-ip=205.220.177.32 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=oracle.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=oracle.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=oracle.com header.i=@oracle.com header.b="oekZpsGL" Received: from pps.filterd (m0246631.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 50D0Qv0J007802; Mon, 13 Jan 2025 04:35:48 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=cc :content-transfer-encoding:date:from:message-id:mime-version :subject:to; s=corp-2023-11-20; bh=PW6MQwkj/m/GUXKmnck58KKN8M0vw +7md10tJlTJhw4=; b=oekZpsGLMpxiqA4AoHGsWuME5L9QP7YrFtiSOr2T1suZ0 l13r732O5Ymm3YaRAQnPukg7SofkckMd7bWVckk/Njow1oD7NP6DarrWfRPDWYyP 4aYjmB3ol+aysGtf2LRzU3J6YXfX/rLbIE8dr5TQAuA0EgiLr8Cz1SwOjfVNBV3f hSLW0g4R28cbHS1lOAELm2xBLKPB+F8klE7kguvQTGoTMYN+WM7VtqvW6i7SCd+h t68B1H7dPX96U2dvlJ6YlRShWAuRPwDXnEew1pEFFDt+pBoxG6o7e5l4B+mQKHdZ ZguD79MGX82h8s2kY1tgcRzIzfkrNUMNPZyo9flkA== Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.appoci.oracle.com [138.1.114.2]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 443fe2axpc-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 13 Jan 2025 04:35:48 +0000 (GMT) Received: from pps.filterd (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (8.18.1.2/8.18.1.2) with ESMTP id 50D2bR1U029862; Mon, 13 Jan 2025 04:35:47 GMT Received: from localhost.localdomain (dhcp-10-191-133-23.vpn.oracle.com [10.191.133.23]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTP id 443f36h7sx-1; Mon, 13 Jan 2025 04:35:46 +0000 From: Imran Khan To: tj@kernel.org, jiangshanlai@gmail.com Cc: linux-kernel@vger.kernel.org Subject: [RFC PATCH] workqueue: introduce queue_delayed_work_on_offline_safe. Date: Mon, 13 Jan 2025 15:35:40 +1100 Message-Id: <20250113043540.2396426-1-imran.f.khan@oracle.com> X-Mailer: git-send-email 2.34.1 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1057,Hydra:6.0.680,FMLib:17.12.68.34 definitions=2025-01-13_01,2025-01-10_03,2024-11-22_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 mlxscore=0 suspectscore=0 bulkscore=0 malwarescore=0 adultscore=0 spamscore=0 mlxlogscore=999 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2411120000 definitions=main-2501130036 X-Proofpoint-ORIG-GUID: K1HlAc3_F_Q_vmeyqKb5SSuUoSQDck5S X-Proofpoint-GUID: K1HlAc3_F_Q_vmeyqKb5SSuUoSQDck5S Content-Type: text/plain; charset="utf-8" Currently users of queue_delayed_work_on, need to ensure that specified cpu is and remains online. The failure to do so may result in delayed_work getting queued on an offlined cpu and hence never getting executed. The current users of queue_delayed_work_on, seem to ensure the above mentioned criteria but for those, unknown amongst current users or new users, who can't confirm to this we need another interface. So introduce queue_delayed_work_on_offline_safe, which explicitly ensures that specified cpu is and remains online. If the specified cpu is already offline or if that can't be confirmed (due to failure in acquiring hotplug lock), delayed_work.timer is not queued on specified cpu. Signed-off-by: Imran Khan --- I have kept the patch as RFC because from mailing list, I could not find any users, of queue_delayed_work_on, that is ending up queuing dwork on an offlined CPU. We have some in-house code that is running into this problem, and currently we are fixing it on caller side of queue_delayed_work_on. Other users who run into this issue, can also use the approach of fixing it on caller side or we can use the interface introduced here for such use cases. include/linux/workqueue.h | 3 ++ kernel/workqueue.c | 67 ++++++++++++++++++++++++++++++++++++--- 2 files changed, 66 insertions(+), 4 deletions(-) diff --git a/include/linux/workqueue.h b/include/linux/workqueue.h index b0dc957c3e560..57f39807f3bf1 100644 --- a/include/linux/workqueue.h +++ b/include/linux/workqueue.h @@ -589,6 +589,9 @@ extern bool queue_work_node(int node, struct workqueue_= struct *wq, struct work_struct *work); extern bool queue_delayed_work_on(int cpu, struct workqueue_struct *wq, struct delayed_work *work, unsigned long delay); +extern bool queue_delayed_work_on_offline_safe(int cpu, + struct workqueue_struct *wq, struct delayed_work *work, + unsigned long delay); extern bool mod_delayed_work_on(int cpu, struct workqueue_struct *wq, struct delayed_work *dwork, unsigned long delay); extern bool queue_rcu_work(struct workqueue_struct *wq, struct rcu_work *r= work); diff --git a/kernel/workqueue.c b/kernel/workqueue.c index 8e0bb3c608239..f1b6320f675a6 100644 --- a/kernel/workqueue.c +++ b/kernel/workqueue.c @@ -2487,7 +2487,8 @@ void delayed_work_timer_fn(struct timer_list *t) EXPORT_SYMBOL(delayed_work_timer_fn); =20 static void __queue_delayed_work(int cpu, struct workqueue_struct *wq, - struct delayed_work *dwork, unsigned long delay) + struct delayed_work *dwork, unsigned long delay, + bool offline_safe) { struct timer_list *timer =3D &dwork->timer; struct work_struct *work =3D &dwork->work; @@ -2521,8 +2522,22 @@ static void __queue_delayed_work(int cpu, struct wor= kqueue_struct *wq, } else { if (likely(cpu =3D=3D WORK_CPU_UNBOUND)) add_timer_global(timer); - else + else if (likely(!offline_safe)) add_timer_on(timer, cpu); + else { /* offline_safe */ + if (unlikely(!cpu_online(cpu))) + /* cpu is already offline*/ + add_timer_global(timer); + else if (cpus_read_trylock()) { + if (likely(cpu_online(cpu))) + add_timer_on(timer, cpu); + else + add_timer_global(timer); + cpus_read_unlock(); + } else + /* could not get hotplug lock*/ + add_timer_global(timer); + } } } =20 @@ -2549,7 +2564,7 @@ bool queue_delayed_work_on(int cpu, struct workqueue_= struct *wq, =20 if (!test_and_set_bit(WORK_STRUCT_PENDING_BIT, work_data_bits(work)) && !clear_pending_if_disabled(work)) { - __queue_delayed_work(cpu, wq, dwork, delay); + __queue_delayed_work(cpu, wq, dwork, delay, false); ret =3D true; } =20 @@ -2558,6 +2573,50 @@ bool queue_delayed_work_on(int cpu, struct workqueue= _struct *wq, } EXPORT_SYMBOL(queue_delayed_work_on); =20 +/** + * queue_delayed_work_on_offline_safe - queue work on specific online CPU = after + * delay, + * + * @cpu: CPU number to execute work on + * @wq: workqueue to use + * @dwork: work to queue + * @delay: number of jiffies to wait before queueing + * + * same as queue_delayed_work_on, but checks and ensures that specified @c= pu + * is online. If @cpu is found to be offline or if its online status can't + * be confirmed queue @dwork on any CPU. + * cpus_read_trylock is used to ensure @cpu remains online, but we don't b= lock + * if hptplug lock can't be taken. So there is good chance that even when + * specified @cpu was online, we queued @dwork to some other CPU, because + * cpu_read_trylock returned failure. On the plus side this interface + * makes sure that we don't endup putting @dwork on offlined @cpu and + * thus possibly losing it forever. + * + * Return: %false if @work was already on a queue, %true otherwise. If + * @delay is zero and @dwork is idle, it will be scheduled for immediate + * execution. + */ +bool queue_delayed_work_on_offline_safe(int cpu, struct workqueue_struct *= wq, + struct delayed_work *dwork, unsigned long delay) +{ + struct work_struct *work =3D &dwork->work; + bool ret =3D false; + unsigned long irq_flags; + + /* read the comment in __queue_work() */ + local_irq_save(irq_flags); + + if (!test_and_set_bit(WORK_STRUCT_PENDING_BIT, work_data_bits(work)) && + !clear_pending_if_disabled(work)) { + __queue_delayed_work(cpu, wq, dwork, delay, true); + ret =3D true; + } + + local_irq_restore(irq_flags); + return ret; +} +EXPORT_SYMBOL(queue_delayed_work_on_offline_safe); + /** * mod_delayed_work_on - modify delay of or queue a delayed work on specif= ic CPU * @cpu: CPU number to execute work on @@ -2585,7 +2644,7 @@ bool mod_delayed_work_on(int cpu, struct workqueue_st= ruct *wq, ret =3D work_grab_pending(&dwork->work, WORK_CANCEL_DELAYED, &irq_flags); =20 if (!clear_pending_if_disabled(&dwork->work)) - __queue_delayed_work(cpu, wq, dwork, delay); + __queue_delayed_work(cpu, wq, dwork, delay, false); =20 local_irq_restore(irq_flags); return ret; base-commit: 2b88851f583d3c4e40bcd40cfe1965241ec229dd --=20 2.34.1