From nobody Fri Dec 19 20:13:20 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1554154038101253.64976707277424; Mon, 1 Apr 2019 14:27:18 -0700 (PDT) Received: from localhost ([127.0.0.1]:44075 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hB4SY-0000V0-QS for importer@patchew.org; Mon, 01 Apr 2019 17:27:14 -0400 Received: from eggs.gnu.org ([209.51.188.92]:41492) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hB45D-000204-6N for qemu-devel@nongnu.org; Mon, 01 Apr 2019 17:03:08 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1hB45B-0003pl-ST for qemu-devel@nongnu.org; Mon, 01 Apr 2019 17:03:07 -0400 Received: from mx0a-001b2d01.pphosted.com ([148.163.156.1]:44860) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1hB45B-0003ie-EG for qemu-devel@nongnu.org; Mon, 01 Apr 2019 17:03:05 -0400 Received: from pps.filterd (m0098396.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x31L1rqJ122295 for ; Mon, 1 Apr 2019 17:02:54 -0400 Received: from e33.co.us.ibm.com (e33.co.us.ibm.com [32.97.110.151]) by mx0a-001b2d01.pphosted.com with ESMTP id 2rksyg8sjj-1 (version=TLSv1.2 cipher=AES256-GCM-SHA384 bits=256 verify=NOT) for ; Mon, 01 Apr 2019 17:02:53 -0400 Received: from localhost by e33.co.us.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Mon, 1 Apr 2019 22:02:53 +0100 Received: from b03cxnp08025.gho.boulder.ibm.com (9.17.130.17) by e33.co.us.ibm.com (192.168.1.133) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; (version=TLSv1/SSLv3 cipher=AES256-GCM-SHA384 bits=256/256) Mon, 1 Apr 2019 22:02:50 +0100 Received: from b03ledav002.gho.boulder.ibm.com (b03ledav002.gho.boulder.ibm.com [9.17.130.233]) by b03cxnp08025.gho.boulder.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id x31L2n4E26607830 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 1 Apr 2019 21:02:49 GMT Received: from b03ledav002.gho.boulder.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 3707113604F; Mon, 1 Apr 2019 21:02:49 +0000 (GMT) Received: from b03ledav002.gho.boulder.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 00B67136053; Mon, 1 Apr 2019 21:02:48 +0000 (GMT) Received: from localhost (unknown [9.80.94.43]) by b03ledav002.gho.boulder.ibm.com (Postfix) with ESMTP; Mon, 1 Apr 2019 21:02:48 +0000 (GMT) From: Michael Roth To: qemu-devel@nongnu.org Date: Mon, 1 Apr 2019 15:59:46 -0500 X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190401210011.16009-1-mdroth@linux.vnet.ibm.com> References: <20190401210011.16009-1-mdroth@linux.vnet.ibm.com> X-TM-AS-GCONF: 00 x-cbid: 19040121-0036-0000-0000-00000AA31E14 X-IBM-SpamModules-Scores: X-IBM-SpamModules-Versions: BY=3.00010857; HX=3.00000242; KW=3.00000007; PH=3.00000004; SC=3.00000283; SDB=6.01182941; UDB=6.00619269; IPR=6.00963683; MB=3.00026249; MTD=3.00000008; XFM=3.00000015; UTC=2019-04-01 21:02:51 X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 19040121-0037-0000-0000-00004B3CB210 Message-Id: <20190401210011.16009-73-mdroth@linux.vnet.ibm.com> X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:, , definitions=2019-04-01_06:, , signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 malwarescore=0 suspectscore=1 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1904010136 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 3.x [generic] X-Received-From: 148.163.156.1 Subject: [Qemu-devel] [PATCH 72/97] mirror: fix dead-lock X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Vladimir Sementsov-Ogievskiy , qemu-stable@nongnu.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: Vladimir Sementsov-Ogievskiy Let start from the beginning: Commit b9e413dd375 (in 2.9) "block: explicitly acquire aiocontext in aio callbacks that need it" added pairs of aio_context_acquire/release to mirror_write_complete and mirror_read_complete, when they were aio callbacks for blk_aio_* calls. Then, commit 2e1990b26e5 (in 3.0) "block/mirror: Convert to coroutines" dropped these blk_aio_* calls, than mirror_write_complete and mirror_read_complete are not callbacks more, and don't need additional aiocontext acquiring. Furthermore, mirror_read_complete calls blk_co_pwritev inside these pair of aio_context_acquire/release, which leads to the following dead-lock with mirror: (gdb) info thr Id Target Id Frame 3 Thread (LWP 145412) "qemu-system-x86" syscall () 2 Thread (LWP 145416) "qemu-system-x86" __lll_lock_wait () * 1 Thread (LWP 145411) "qemu-system-x86" __lll_lock_wait () (gdb) bt #0 __lll_lock_wait () #1 _L_lock_812 () #2 __GI___pthread_mutex_lock #3 qemu_mutex_lock_impl (mutex=3D0x561032dce420 , file=3D0x5610327d8654 "util/main-loop.c", line=3D236) at util/qemu-thread-posix.c:66 #4 qemu_mutex_lock_iothread_impl #5 os_host_main_loop_wait (timeout=3D480116000) at util/main-loop.c:236 #6 main_loop_wait (nonblocking=3D0) at util/main-loop.c:497 #7 main_loop () at vl.c:1892 #8 main Printing contents of qemu_global_mutex, I see that "__owner =3D 145416", so, thr1 is main loop, and now it wants BQL, which is owned by thr2. (gdb) thr 2 (gdb) bt #0 __lll_lock_wait () #1 _L_lock_870 () #2 __GI___pthread_mutex_lock #3 qemu_mutex_lock_impl (mutex=3D0x561034d25dc0, ... #4 aio_context_acquire (ctx=3D0x561034d25d60) #5 dma_blk_cb #6 dma_blk_io #7 dma_blk_read #8 ide_dma_cb #9 bmdma_cmd_writeb #10 bmdma_write #11 memory_region_write_accessor #12 access_with_adjusted_size #15 flatview_write #16 address_space_write #17 address_space_rw #18 kvm_handle_io #19 kvm_cpu_exec #20 qemu_kvm_cpu_thread_fn #21 qemu_thread_start #22 start_thread #23 clone () Printing mutex in fr 2, I see "__owner =3D 145411", so thr2 wants aio context mutex, which is owned by thr1. Classic dead-lock. Then, let's check that aio context is hold by mirror coroutine: just print coroutine stack of first tracked request in mirror job target: (gdb) [...] (gdb) qemu coroutine 0x561035dd0860 #0 qemu_coroutine_switch #1 qemu_coroutine_yield #2 qemu_co_mutex_lock_slowpath #3 qemu_co_mutex_lock #4 qcow2_co_pwritev #5 bdrv_driver_pwritev #6 bdrv_aligned_pwritev #7 bdrv_co_pwritev #8 blk_co_pwritev #9 mirror_read_complete () at block/mirror.c:232 #10 mirror_co_read () at block/mirror.c:370 #11 coroutine_trampoline #12 __start_context Yes it is mirror_read_complete calling blk_co_pwritev after acquiring aio context. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Max Reitz Signed-off-by: Kevin Wolf (cherry picked from commit d12ade5732c4d2d293735a39b4bd943da8d64db6) Signed-off-by: Michael Roth --- block/mirror.c | 13 +++++-------- 1 file changed, 5 insertions(+), 8 deletions(-) diff --git a/block/mirror.c b/block/mirror.c index dd5ca02b09..b097853983 100644 --- a/block/mirror.c +++ b/block/mirror.c @@ -198,7 +198,6 @@ static void coroutine_fn mirror_write_complete(MirrorOp= *op, int ret) { MirrorBlockJob *s =3D op->s; =20 - aio_context_acquire(blk_get_aio_context(s->common.blk)); if (ret < 0) { BlockErrorAction action; =20 @@ -208,15 +207,14 @@ static void coroutine_fn mirror_write_complete(Mirror= Op *op, int ret) s->ret =3D ret; } } + mirror_iteration_done(op, ret); - aio_context_release(blk_get_aio_context(s->common.blk)); } =20 static void coroutine_fn mirror_read_complete(MirrorOp *op, int ret) { MirrorBlockJob *s =3D op->s; =20 - aio_context_acquire(blk_get_aio_context(s->common.blk)); if (ret < 0) { BlockErrorAction action; =20 @@ -227,12 +225,11 @@ static void coroutine_fn mirror_read_complete(MirrorO= p *op, int ret) } =20 mirror_iteration_done(op, ret); - } else { - ret =3D blk_co_pwritev(s->target, op->offset, - op->qiov.size, &op->qiov, 0); - mirror_write_complete(op, ret); + return; } - aio_context_release(blk_get_aio_context(s->common.blk)); + + ret =3D blk_co_pwritev(s->target, op->offset, op->qiov.size, &op->qiov= , 0); + mirror_write_complete(op, ret); } =20 /* Clip bytes relative to offset to not exceed end-of-file */ --=20 2.17.1