From nobody Thu Dec 18 13:15:31 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1543858332201868.6112364985947; Mon, 3 Dec 2018 09:32:12 -0800 (PST) Received: from localhost ([::1]:50932 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gTs4o-0002UI-PW for importer@patchew.org; Mon, 03 Dec 2018 12:32:10 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:54050) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gTrYE-0007do-Vj for qemu-devel@nongnu.org; Mon, 03 Dec 2018 11:58:32 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gTrYD-0008K0-MI for qemu-devel@nongnu.org; Mon, 03 Dec 2018 11:58:30 -0500 Received: from mx1.redhat.com ([209.132.183.28]:40558) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1gTrYA-0008Co-VU; Mon, 03 Dec 2018 11:58:27 -0500 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 3B59630DDBA1; Mon, 3 Dec 2018 16:58:26 +0000 (UTC) Received: from localhost.localdomain.com (unknown [10.36.118.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id 638F317B8A; Mon, 3 Dec 2018 16:58:24 +0000 (UTC) From: Kevin Wolf To: qemu-block@nongnu.org Date: Mon, 3 Dec 2018 17:58:09 +0100 Message-Id: <20181203165810.14509-2-kwolf@redhat.com> In-Reply-To: <20181203165810.14509-1-kwolf@redhat.com> References: <20181203165810.14509-1-kwolf@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.49]); Mon, 03 Dec 2018 16:58:26 +0000 (UTC) Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PULL 1/2] mirror: fix dead-lock X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kwolf@redhat.com, peter.maydell@linaro.org, qemu-devel@nongnu.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Type: text/plain; charset="utf-8" From: Vladimir Sementsov-Ogievskiy Let start from the beginning: Commit b9e413dd375 (in 2.9) "block: explicitly acquire aiocontext in aio callbacks that need it" added pairs of aio_context_acquire/release to mirror_write_complete and mirror_read_complete, when they were aio callbacks for blk_aio_* calls. Then, commit 2e1990b26e5 (in 3.0) "block/mirror: Convert to coroutines" dropped these blk_aio_* calls, than mirror_write_complete and mirror_read_complete are not callbacks more, and don't need additional aiocontext acquiring. Furthermore, mirror_read_complete calls blk_co_pwritev inside these pair of aio_context_acquire/release, which leads to the following dead-lock with mirror: (gdb) info thr Id Target Id Frame 3 Thread (LWP 145412) "qemu-system-x86" syscall () 2 Thread (LWP 145416) "qemu-system-x86" __lll_lock_wait () * 1 Thread (LWP 145411) "qemu-system-x86" __lll_lock_wait () (gdb) bt #0 __lll_lock_wait () #1 _L_lock_812 () #2 __GI___pthread_mutex_lock #3 qemu_mutex_lock_impl (mutex=3D0x561032dce420 , file=3D0x5610327d8654 "util/main-loop.c", line=3D236) at util/qemu-thread-posix.c:66 #4 qemu_mutex_lock_iothread_impl #5 os_host_main_loop_wait (timeout=3D480116000) at util/main-loop.c:236 #6 main_loop_wait (nonblocking=3D0) at util/main-loop.c:497 #7 main_loop () at vl.c:1892 #8 main Printing contents of qemu_global_mutex, I see that "__owner =3D 145416", so, thr1 is main loop, and now it wants BQL, which is owned by thr2. (gdb) thr 2 (gdb) bt #0 __lll_lock_wait () #1 _L_lock_870 () #2 __GI___pthread_mutex_lock #3 qemu_mutex_lock_impl (mutex=3D0x561034d25dc0, ... #4 aio_context_acquire (ctx=3D0x561034d25d60) #5 dma_blk_cb #6 dma_blk_io #7 dma_blk_read #8 ide_dma_cb #9 bmdma_cmd_writeb #10 bmdma_write #11 memory_region_write_accessor #12 access_with_adjusted_size #15 flatview_write #16 address_space_write #17 address_space_rw #18 kvm_handle_io #19 kvm_cpu_exec #20 qemu_kvm_cpu_thread_fn #21 qemu_thread_start #22 start_thread #23 clone () Printing mutex in fr 2, I see "__owner =3D 145411", so thr2 wants aio context mutex, which is owned by thr1. Classic dead-lock. Then, let's check that aio context is hold by mirror coroutine: just print coroutine stack of first tracked request in mirror job target: (gdb) [...] (gdb) qemu coroutine 0x561035dd0860 #0 qemu_coroutine_switch #1 qemu_coroutine_yield #2 qemu_co_mutex_lock_slowpath #3 qemu_co_mutex_lock #4 qcow2_co_pwritev #5 bdrv_driver_pwritev #6 bdrv_aligned_pwritev #7 bdrv_co_pwritev #8 blk_co_pwritev #9 mirror_read_complete () at block/mirror.c:232 #10 mirror_co_read () at block/mirror.c:370 #11 coroutine_trampoline #12 __start_context Yes it is mirror_read_complete calling blk_co_pwritev after acquiring aio context. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Max Reitz Signed-off-by: Kevin Wolf --- block/mirror.c | 13 +++++-------- 1 file changed, 5 insertions(+), 8 deletions(-) diff --git a/block/mirror.c b/block/mirror.c index 56d9ef7474..8f52c6215d 100644 --- a/block/mirror.c +++ b/block/mirror.c @@ -199,7 +199,6 @@ static void coroutine_fn mirror_write_complete(MirrorOp= *op, int ret) { MirrorBlockJob *s =3D op->s; =20 - aio_context_acquire(blk_get_aio_context(s->common.blk)); if (ret < 0) { BlockErrorAction action; =20 @@ -209,15 +208,14 @@ static void coroutine_fn mirror_write_complete(Mirror= Op *op, int ret) s->ret =3D ret; } } + mirror_iteration_done(op, ret); - aio_context_release(blk_get_aio_context(s->common.blk)); } =20 static void coroutine_fn mirror_read_complete(MirrorOp *op, int ret) { MirrorBlockJob *s =3D op->s; =20 - aio_context_acquire(blk_get_aio_context(s->common.blk)); if (ret < 0) { BlockErrorAction action; =20 @@ -228,12 +226,11 @@ static void coroutine_fn mirror_read_complete(MirrorO= p *op, int ret) } =20 mirror_iteration_done(op, ret); - } else { - ret =3D blk_co_pwritev(s->target, op->offset, - op->qiov.size, &op->qiov, 0); - mirror_write_complete(op, ret); + return; } - aio_context_release(blk_get_aio_context(s->common.blk)); + + ret =3D blk_co_pwritev(s->target, op->offset, op->qiov.size, &op->qiov= , 0); + mirror_write_complete(op, ret); } =20 /* Clip bytes relative to offset to not exceed end-of-file */ --=20 2.19.2 From nobody Thu Dec 18 13:15:31 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1543858557172137.93676349520433; Mon, 3 Dec 2018 09:35:57 -0800 (PST) Received: from localhost ([::1]:50962 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gTs8R-0005XP-U7 for importer@patchew.org; Mon, 03 Dec 2018 12:35:55 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:54073) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gTrYH-0007gI-0n for qemu-devel@nongnu.org; Mon, 03 Dec 2018 11:58:35 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gTrYF-0008LW-AP for qemu-devel@nongnu.org; Mon, 03 Dec 2018 11:58:32 -0500 Received: from mx1.redhat.com ([209.132.183.28]:42548) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1gTrYC-0008E2-Cp; Mon, 03 Dec 2018 11:58:28 -0500 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id A0FE989AC4; Mon, 3 Dec 2018 16:58:27 +0000 (UTC) Received: from localhost.localdomain.com (unknown [10.36.118.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id 834FC60E3F; Mon, 3 Dec 2018 16:58:26 +0000 (UTC) From: Kevin Wolf To: qemu-block@nongnu.org Date: Mon, 3 Dec 2018 17:58:10 +0100 Message-Id: <20181203165810.14509-3-kwolf@redhat.com> In-Reply-To: <20181203165810.14509-1-kwolf@redhat.com> References: <20181203165810.14509-1-kwolf@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.26]); Mon, 03 Dec 2018 16:58:27 +0000 (UTC) Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PULL 2/2] iotests: simple mirror test with kvm on 1G image X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kwolf@redhat.com, peter.maydell@linaro.org, qemu-devel@nongnu.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Type: text/plain; charset="utf-8" From: Vladimir Sementsov-Ogievskiy This test is broken without previous commit fixing dead-lock in mirror. Signed-off-by: Vladimir Sementsov-Ogievskiy Signed-off-by: Max Reitz Acked-by: Vladimir Sementsov-Ogievskiy Signed-off-by: Kevin Wolf --- tests/qemu-iotests/235 | 76 ++++++++++++++++++++++++++++++++++++++ tests/qemu-iotests/235.out | 3 ++ tests/qemu-iotests/group | 1 + 3 files changed, 80 insertions(+) create mode 100755 tests/qemu-iotests/235 create mode 100644 tests/qemu-iotests/235.out diff --git a/tests/qemu-iotests/235 b/tests/qemu-iotests/235 new file mode 100755 index 0000000000..da044ed34e --- /dev/null +++ b/tests/qemu-iotests/235 @@ -0,0 +1,76 @@ +#!/usr/bin/env python +# +# Simple mirror test +# +# Copyright (c) 2018 Virtuozzo International GmbH. All rights reserved. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . +# + +import sys +import os +import iotests +from iotests import qemu_img_create, qemu_io, file_path, log + +sys.path.append(os.path.join(os.path.dirname(__file__), '..', '..', 'scrip= ts')) + +from qemu import QEMUMachine + +# Note: +# This test was added to check that mirror dead-lock was fixed (see previo= us +# commit before this test addition). +# And it didn't reproduce if at least one of the following: +# 1. use small image size +# 2. use raw format (not qcow2) +# 3. drop kvm and use iotests.VM() (maybe, because of qtest) (however, it = still +# reproduces, if just drop kvm, but gdb failed to produce full backtrac= es +# for me) +# 4. add iothread + +size =3D 1 * 1024 * 1024 * 1024 + +iotests.verify_image_format(supported_fmts=3D['qcow2']) + +disk =3D file_path('disk') + +# prepare source image +qemu_img_create('-f', iotests.imgfmt, '-o', 'preallocation=3Dmetadata', di= sk, + str(size)) + +vm =3D QEMUMachine(iotests.qemu_prog) +vm.add_args('-machine', 'pc,accel=3Dkvm') +vm.add_args('-drive', 'id=3Dsrc,file=3D' + disk) +vm.launch() + +log(vm.qmp('object-add', qom_type=3D'throttle-group', id=3D'tg0', + props=3D{ 'x-bps-total': size })) + +log(vm.qmp('blockdev-add', + **{ 'node-name': 'target', + 'driver': 'throttle', + 'throttle-group': 'tg0', + 'file': { + 'driver': 'null-co', + 'size': size + } })) + +log(vm.qmp('blockdev-mirror', device=3D'src', target=3D'target', sync=3D'f= ull')) + +try: + vm.event_wait('BLOCK_JOB_READY', timeout=3D10.0) +except: + vm.shutdown() + raise + +vm.shutdown() diff --git a/tests/qemu-iotests/235.out b/tests/qemu-iotests/235.out new file mode 100644 index 0000000000..39db621e04 --- /dev/null +++ b/tests/qemu-iotests/235.out @@ -0,0 +1,3 @@ +{"return": {}} +{"return": {}} +{"return": {}} diff --git a/tests/qemu-iotests/group b/tests/qemu-iotests/group index 8c56a0ad11..61a6d98ebd 100644 --- a/tests/qemu-iotests/group +++ b/tests/qemu-iotests/group @@ -232,3 +232,4 @@ 232 auto quick 233 auto quick 234 auto quick migration +235 auto quick --=20 2.19.2