From nobody Mon Dec 29 16:52:19 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E6C3CC61D97 for ; Fri, 24 Nov 2023 21:12:50 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229703AbjKXVMj (ORCPT ); Fri, 24 Nov 2023 16:12:39 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52762 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229557AbjKXVMh (ORCPT ); Fri, 24 Nov 2023 16:12:37 -0500 Received: from mx0a-0031df01.pphosted.com (mx0a-0031df01.pphosted.com [205.220.168.131]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DF25A1702; Fri, 24 Nov 2023 13:12:43 -0800 (PST) Received: from pps.filterd (m0279864.ppops.net [127.0.0.1]) by mx0a-0031df01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 3AOKk3Wc015910; Fri, 24 Nov 2023 21:12:35 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=quicinc.com; h=from : to : cc : subject : date : message-id : mime-version : content-type; s=qcppdkim1; bh=aJVTqvZRGJ1DiRSJOwioDvRJ9BIm61Q4/UiydkhVbLw=; b=gVKrpSKn56U+Dg4Ai7cyMWUoh2RHsEIzAMLjg9MbRBE5yViRW5mkZLWMAmTmsTq83BpX i9U09OX5sfs4S1Xm/FxYnJPhp2i0R8nPJqi+5KiJeNPSQvkAIpk+RjtNBpDTFfCFnXmv q6liQzKYz/gXavNnGJhYnHx3YwaYHxFcpHiSOlot4vEi/7wl9JaJAni5FaGlw02PJUAj iMx4EAU4m4njQS77AxVp7FiLZEibfXA8xvD7pOfnvrh6Zb6x3aiiYQkhXPYPZI3FCNwu GicM73oZEXemxXwLWDnO9FX5snrPMoolTh3Sr9wET8ObFIqdAzRG9veLyh6k2eD8Rgw1 eg== Received: from nasanppmta05.qualcomm.com (i-global254.qualcomm.com [199.106.103.254]) by mx0a-0031df01.pphosted.com (PPS) with ESMTPS id 3ujhh4tfs6-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 24 Nov 2023 21:12:35 +0000 Received: from nasanex01c.na.qualcomm.com (nasanex01c.na.qualcomm.com [10.45.79.139]) by NASANPPMTA05.qualcomm.com (8.17.1.5/8.17.1.5) with ESMTPS id 3AOLCYjm029283 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 24 Nov 2023 21:12:34 GMT Received: from hu-mojha-hyd.qualcomm.com (10.80.80.8) by nasanex01c.na.qualcomm.com (10.45.79.139) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1118.40; Fri, 24 Nov 2023 13:12:31 -0800 From: Mukesh Ojha To: , , CC: , , , Mukesh Ojha Subject: [PATCH v4] PM / devfreq: Synchronize device_monitor_[start/stop] Date: Sat, 25 Nov 2023 02:41:58 +0530 Message-ID: <1700860318-4025-1-git-send-email-quic_mojha@quicinc.com> X-Mailer: git-send-email 2.7.4 MIME-Version: 1.0 X-Originating-IP: [10.80.80.8] X-ClientProxiedBy: nasanex01b.na.qualcomm.com (10.46.141.250) To nasanex01c.na.qualcomm.com (10.45.79.139) X-QCInternal: smtphost X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=5800 signatures=585085 X-Proofpoint-GUID: f_1IhZyzynzKrAhZLiydFRCSJ6XN-GUH X-Proofpoint-ORIG-GUID: f_1IhZyzynzKrAhZLiydFRCSJ6XN-GUH X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.272,Aquarius:18.0.987,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2023-11-24_08,2023-11-22_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 suspectscore=0 mlxlogscore=999 priorityscore=1501 adultscore=0 lowpriorityscore=0 malwarescore=0 mlxscore=0 clxscore=1015 phishscore=0 bulkscore=0 impostorscore=0 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2311060000 definitions=main-2311240165 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" There is a chance if a frequent switch of the governor done in a loop result in timer list corruption where timer cancel being done from two place one from cancel_delayed_work_sync() and followed by expire_timers() can be seen from the traces[1]. while true do echo "simple_ondemand" > /sys/class/devfreq/1d84000.ufshc/governor echo "performance" > /sys/class/devfreq/1d84000.ufshc/governor done It looks to be issue with devfreq driver where device_monitor_[start/stop] need to synchronized so that delayed work should get corrupted while it is either being queued or running or being cancelled. Let's use polling flag and devfreq lock to synchronize the queueing the timer instance twice and work data being corrupted. [1] ... .. -0 [003] 9436.209662: timer_cancel timer=3D0xffffff80444f0428 -0 [003] 9436.209664: timer_expire_entry timer=3D0xffffff8044= 4f0428 now=3D0x10022da1c function=3D__typeid__ZTSFvP10timer_listE_global_= addr baseclk=3D0x10022da1c -0 [003] 9436.209718: timer_expire_exit timer=3D0xffffff80444= f0428 kworker/u16:6-14217 [003] 9436.209863: timer_start timer=3D0xffffff= 80444f0428 function=3D__typeid__ZTSFvP10timer_listE_global_addr expires= =3D0x10022da2b now=3D0x10022da1c flags=3D182452227 vendor.xxxyyy.ha-1593 [004] 9436.209888: timer_cancel timer=3D0xfff= fff80444f0428 vendor.xxxyyy.ha-1593 [004] 9436.216390: timer_init timer=3D0xfffff= f80444f0428 vendor.xxxyyy.ha-1593 [004] 9436.216392: timer_start timer=3D0xffff= ff80444f0428 function=3D__typeid__ZTSFvP10timer_listE_global_addr expires= =3D0x10022da2c now=3D0x10022da1d flags=3D186646532 vendor.xxxyyy.ha-1593 [005] 9436.220992: timer_cancel timer=3D0xfff= fff80444f0428 xxxyyyTraceManag-7795 [004] 9436.261641: timer_cancel timer=3D0xfff= fff80444f0428 [2] 9436.261653][ C4] Unable to handle kernel paging request at virtual add= ress dead00000000012a [ 9436.261664][ C4] Mem abort info: [ 9436.261666][ C4] ESR =3D 0x96000044 [ 9436.261669][ C4] EC =3D 0x25: DABT (current EL), IL =3D 32 bits [ 9436.261671][ C4] SET =3D 0, FnV =3D 0 [ 9436.261673][ C4] EA =3D 0, S1PTW =3D 0 [ 9436.261675][ C4] Data abort info: [ 9436.261677][ C4] ISV =3D 0, ISS =3D 0x00000044 [ 9436.261680][ C4] CM =3D 0, WnR =3D 1 [ 9436.261682][ C4] [dead00000000012a] address between user and kernel a= ddress ranges [ 9436.261685][ C4] Internal error: Oops: 96000044 [#1] PREEMPT SMP [ 9436.261701][ C4] Skip md ftrace buffer dump for: 0x3a982d0 ... [ 9436.262138][ C4] CPU: 4 PID: 7795 Comm: TraceManag Tainted: G S = W O 5.10.149-android12-9-o-g17f915d29d0c #1 [ 9436.262141][ C4] Hardware name: Qualcomm Technologies, Inc. (DT) [ 9436.262144][ C4] pstate: 22400085 (nzCv daIf +PAN -UAO +TCO BTYPE=3D-= -) [ 9436.262161][ C4] pc : expire_timers+0x9c/0x438 [ 9436.262164][ C4] lr : expire_timers+0x2a4/0x438 [ 9436.262168][ C4] sp : ffffffc010023dd0 [ 9436.262171][ C4] x29: ffffffc010023df0 x28: ffffffd0636fdc18 [ 9436.262178][ C4] x27: ffffffd063569dd0 x26: ffffffd063536008 [ 9436.262182][ C4] x25: 0000000000000001 x24: ffffff88f7c69280 [ 9436.262185][ C4] x23: 00000000000000e0 x22: dead000000000122 [ 9436.262188][ C4] x21: 000000010022da29 x20: ffffff8af72b4e80 [ 9436.262191][ C4] x19: ffffffc010023e50 x18: ffffffc010025038 [ 9436.262195][ C4] x17: 0000000000000240 x16: 0000000000000201 [ 9436.262199][ C4] x15: ffffffffffffffff x14: ffffff889f3c3100 [ 9436.262203][ C4] x13: ffffff889f3c3100 x12: 00000000049f56b8 [ 9436.262207][ C4] x11: 00000000049f56b8 x10: 00000000ffffffff [ 9436.262212][ C4] x9 : ffffffc010023e50 x8 : dead000000000122 [ 9436.262216][ C4] x7 : ffffffffffffffff x6 : ffffffc0100239d8 [ 9436.262220][ C4] x5 : 0000000000000000 x4 : 0000000000000101 [ 9436.262223][ C4] x3 : 0000000000000080 x2 : ffffff889edc155c [ 9436.262227][ C4] x1 : ffffff8001005200 x0 : ffffff80444f0428 [ 9436.262232][ C4] Call trace: [ 9436.262236][ C4] expire_timers+0x9c/0x438 [ 9436.262240][ C4] __run_timers+0x1f0/0x330 [ 9436.262245][ C4] run_timer_softirq+0x28/0x58 [ 9436.262255][ C4] efi_header_end+0x168/0x5ec [ 9436.262265][ C4] __irq_exit_rcu+0x108/0x124 [ 9436.262274][ C4] __handle_domain_irq+0x118/0x1e4 [ 9436.262282][ C4] gic_handle_irq.30369+0x6c/0x2bc [ 9436.262286][ C4] el0_irq_naked+0x60/0x6c Reported-by: Joyyoung Huang Signed-off-by: Mukesh Ojha --- Huang, Would be looking for your tested-by.. -Mukesh Changes in v4: https://lore.kernel.org/lkml/1700238027-20518-1-git-send-ema= il-quic_mojha@quicinc.com/ - Mistakenly put cancel work under devfreq lock which could result in dead= lock reported by [Joyyoung Huang] https://lore.kernel.org/lkml/KL1PR02MB8141D1A307457AF69EBB6AFBA3B8A@KL1P= R02MB8141.apcprd02.prod.outlook.com/ Changes in v3: https://lore.kernel.org/lkml/1700235522-31105-1-git-send-ema= il-quic_mojha@quicinc.com/ - Remove the unexpected 'twice' from the subject. Changes in v2: https://lore.kernel.org/lkml/1699957648-31299-1-git-send-ema= il-quic_mojha@quicinc.com/ - Changed subject. - Added lock to avoid work data corruption due to parallel calls to devfreq_monitor_start while work is queued in flight. - Added lock to cover the same as above case while the work is being cancelled. - Added Reported-by for similar issue reported at https://lore.kernel.org/lkml/SEYPR02MB565398175FA093AC3E63EE7BA3B0A@SEYP= R02MB5653.apcprd02.prod.outlook.com/ drivers/devfreq/devfreq.c | 24 ++++++++++++++++++++++-- 1 file changed, 22 insertions(+), 2 deletions(-) diff --git a/drivers/devfreq/devfreq.c b/drivers/devfreq/devfreq.c index b3a68d5833bd..cb1c24721a37 100644 --- a/drivers/devfreq/devfreq.c +++ b/drivers/devfreq/devfreq.c @@ -461,10 +461,14 @@ static void devfreq_monitor(struct work_struct *work) if (err) dev_err(&devfreq->dev, "dvfs failed with (%d) error\n", err); =20 + if (devfreq->stop_polling) + goto out; + queue_delayed_work(devfreq_wq, &devfreq->work, msecs_to_jiffies(devfreq->profile->polling_ms)); - mutex_unlock(&devfreq->lock); =20 +out: + mutex_unlock(&devfreq->lock); trace_devfreq_monitor(devfreq); } =20 @@ -483,6 +487,10 @@ void devfreq_monitor_start(struct devfreq *devfreq) if (IS_SUPPORTED_FLAG(devfreq->governor->flags, IRQ_DRIVEN)) return; =20 + mutex_lock(&devfreq->lock); + if (delayed_work_pending(&devfreq->work)) + goto out; + switch (devfreq->profile->timer) { case DEVFREQ_TIMER_DEFERRABLE: INIT_DEFERRABLE_WORK(&devfreq->work, devfreq_monitor); @@ -491,12 +499,16 @@ void devfreq_monitor_start(struct devfreq *devfreq) INIT_DELAYED_WORK(&devfreq->work, devfreq_monitor); break; default: - return; + goto out; } =20 if (devfreq->profile->polling_ms) queue_delayed_work(devfreq_wq, &devfreq->work, msecs_to_jiffies(devfreq->profile->polling_ms)); + +out: + devfreq->stop_polling =3D false; + mutex_unlock(&devfreq->lock); } EXPORT_SYMBOL(devfreq_monitor_start); =20 @@ -513,6 +525,14 @@ void devfreq_monitor_stop(struct devfreq *devfreq) if (IS_SUPPORTED_FLAG(devfreq->governor->flags, IRQ_DRIVEN)) return; =20 + mutex_lock(&devfreq->lock); + if (devfreq->stop_polling) { + mutex_unlock(&devfreq->lock); + return; + } + + devfreq->stop_polling =3D true; + mutex_unlock(&devfreq->lock); cancel_delayed_work_sync(&devfreq->work); } EXPORT_SYMBOL(devfreq_monitor_stop); --=20 2.7.4