From nobody Sat May 18 23:55:24 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1547475581268629.4581272779928; Mon, 14 Jan 2019 06:19:41 -0800 (PST) Received: from localhost ([127.0.0.1]:37145 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gj35Y-0004yv-4k for importer@patchew.org; Mon, 14 Jan 2019 09:19:40 -0500 Received: from eggs.gnu.org ([209.51.188.92]:46240) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gj2jV-0005Eq-JS for qemu-devel@nongnu.org; Mon, 14 Jan 2019 08:56:54 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gj2Ut-0006hw-U9 for qemu-devel@nongnu.org; Mon, 14 Jan 2019 08:41:48 -0500 Received: from mx1.redhat.com ([209.132.183.28]:51322) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1gj2Us-0006h7-OZ; Mon, 14 Jan 2019 08:41:46 -0500 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 0365EC057F99; Mon, 14 Jan 2019 13:33:08 +0000 (UTC) Received: from localhost (unknown [10.36.118.30]) by smtp.corp.redhat.com (Postfix) with ESMTP id 14529608FA; Mon, 14 Jan 2019 13:33:04 +0000 (UTC) From: Stefan Hajnoczi To: qemu-devel@nongnu.org Date: Mon, 14 Jan 2019 13:32:56 +0000 Message-Id: <20190114133257.30299-2-stefanha@redhat.com> In-Reply-To: <20190114133257.30299-1-stefanha@redhat.com> References: <20190114133257.30299-1-stefanha@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.32]); Mon, 14 Jan 2019 13:33:08 +0000 (UTC) Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PATCH v2 1/2] throttle-groups: fix restart coroutine iothread race X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Alberto Garcia , qemu-block@nongnu.org, Max Reitz , Stefan Hajnoczi , Paolo Bonzini Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Type: text/plain; charset="utf-8" The following QMP command leads to a crash when iothreads are used: { 'execute': 'device_del', 'arguments': {'id': 'data'} } The backtrace involves the queue restart coroutine where tgm->throttle_state is a NULL pointer because throttle_group_unregister_tgm() has already been called: (gdb) bt full #0 0x00005585a7a3b378 in qemu_mutex_lock_impl (mutex=3D0xffffffffffffffd= 0, file=3D0x5585a7bb3d54 "block/throttle-groups.c", line=3D412) at util/qem= u-thread-posix.c:64 err =3D __PRETTY_FUNCTION__ =3D "qemu_mutex_lock_impl" __func__ =3D "qemu_mutex_lock_impl" #1 0x00005585a79be074 in throttle_group_restart_queue_entry (opaque=3D0x= 5585a9de4eb0) at block/throttle-groups.c:412 _f =3D data =3D 0x5585a9de4eb0 tgm =3D 0x5585a9079440 ts =3D 0x0 tg =3D 0xffffffffffffff98 is_write =3D false empty_queue =3D 255 This coroutine should not execute in the iothread after the throttle group member has been unregistered! The root cause is that the device_del code path schedules the restart coroutine in the iothread while holding the AioContext lock. Therefore the iothread cannot execute the coroutine until after device_del releases the lock - by this time it's too late. This patch adds a reference count to ThrottleGroupMember so we can synchronously wait for restart coroutines to complete. Once they are done it is safe to unregister the ThrottleGroupMember. Signed-off-by: Stefan Hajnoczi Reviewed-by: Alberto Garcia --- v2: * Use atomic operations instead of AioContext lock [Paolo] --- include/block/throttle-groups.h | 5 +++++ block/throttle-groups.c | 9 +++++++++ 2 files changed, 14 insertions(+) diff --git a/include/block/throttle-groups.h b/include/block/throttle-group= s.h index e2fd0513c4..712a8e64b4 100644 --- a/include/block/throttle-groups.h +++ b/include/block/throttle-groups.h @@ -43,6 +43,11 @@ typedef struct ThrottleGroupMember { */ unsigned int io_limits_disabled; =20 + /* Number of pending throttle_group_restart_queue_entry() coroutines. + * Accessed with atomic operations. + */ + unsigned int restart_pending; + /* The following fields are protected by the ThrottleGroup lock. * See the ThrottleGroup documentation for details. * throttle_state tells us if I/O limits are configured. */ diff --git a/block/throttle-groups.c b/block/throttle-groups.c index 5d8213a443..a5a2037924 100644 --- a/block/throttle-groups.c +++ b/block/throttle-groups.c @@ -415,6 +415,9 @@ static void coroutine_fn throttle_group_restart_queue_e= ntry(void *opaque) } =20 g_free(data); + + atomic_dec(&tgm->restart_pending); + aio_wait_kick(); } =20 static void throttle_group_restart_queue(ThrottleGroupMember *tgm, bool is= _write) @@ -430,6 +433,8 @@ static void throttle_group_restart_queue(ThrottleGroupM= ember *tgm, bool is_write * be no timer pending on this tgm at this point */ assert(!timer_pending(tgm->throttle_timers.timers[is_write])); =20 + atomic_inc(&tgm->restart_pending); + co =3D qemu_coroutine_create(throttle_group_restart_queue_entry, rd); aio_co_enter(tgm->aio_context, co); } @@ -538,6 +543,7 @@ void throttle_group_register_tgm(ThrottleGroupMember *t= gm, =20 tgm->throttle_state =3D ts; tgm->aio_context =3D ctx; + atomic_set(&tgm->restart_pending, 0); =20 qemu_mutex_lock(&tg->lock); /* If the ThrottleGroup is new set this ThrottleGroupMember as the tok= en */ @@ -584,6 +590,9 @@ void throttle_group_unregister_tgm(ThrottleGroupMember = *tgm) return; } =20 + /* Wait for throttle_group_restart_queue_entry() coroutines to finish = */ + AIO_WAIT_WHILE(tgm->aio_context, atomic_read(&tgm->restart_pending) > = 0); + qemu_mutex_lock(&tg->lock); for (i =3D 0; i < 2; i++) { assert(tgm->pending_reqs[i] =3D=3D 0); --=20 2.20.1 From nobody Sat May 18 23:55:24 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (209.51.188.17 [209.51.188.17]) by mx.zohomail.com with SMTPS id 1547472887492867.6446225552564; Mon, 14 Jan 2019 05:34:47 -0800 (PST) Received: from localhost ([127.0.0.1]:54143 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gj2Nt-000746-Re for importer@patchew.org; Mon, 14 Jan 2019 08:34:33 -0500 Received: from eggs.gnu.org ([209.51.188.92]:40708) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gj2Mf-0006d7-GC for qemu-devel@nongnu.org; Mon, 14 Jan 2019 08:33:18 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gj2Me-0002Ye-9B for qemu-devel@nongnu.org; Mon, 14 Jan 2019 08:33:17 -0500 Received: from mx1.redhat.com ([209.132.183.28]:45948) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1gj2Mb-0002Vn-O2; Mon, 14 Jan 2019 08:33:13 -0500 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.phx2.redhat.com [10.5.11.23]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 7D9952D7E2; Mon, 14 Jan 2019 13:33:12 +0000 (UTC) Received: from localhost (unknown [10.36.118.30]) by smtp.corp.redhat.com (Postfix) with ESMTP id 5D2A318502; Mon, 14 Jan 2019 13:33:09 +0000 (UTC) From: Stefan Hajnoczi To: qemu-devel@nongnu.org Date: Mon, 14 Jan 2019 13:32:57 +0000 Message-Id: <20190114133257.30299-3-stefanha@redhat.com> In-Reply-To: <20190114133257.30299-1-stefanha@redhat.com> References: <20190114133257.30299-1-stefanha@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.23 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.30]); Mon, 14 Jan 2019 13:33:12 +0000 (UTC) Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PATCH v2 2/2] iotests: add 238 for throttling tgm unregister iothread segfault X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Alberto Garcia , qemu-block@nongnu.org, Max Reitz , Stefan Hajnoczi , Paolo Bonzini Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Type: text/plain; charset="utf-8" Hot-unplug a scsi-hd using an iothread. The previous patch fixes a segfault in this scenario. This patch adds a regression test. Suggested-by: Alberto Garcia Suggested-by: Kevin Wolf Signed-off-by: Stefan Hajnoczi Reviewed-by: Alberto Garcia --- v2: * Added qemu-iotests test case [Kevin, Alberto] --- tests/qemu-iotests/238 | 47 ++++++++++++++++++++++++++++++++++++++ tests/qemu-iotests/238.out | 6 +++++ tests/qemu-iotests/group | 1 + 3 files changed, 54 insertions(+) create mode 100755 tests/qemu-iotests/238 create mode 100644 tests/qemu-iotests/238.out diff --git a/tests/qemu-iotests/238 b/tests/qemu-iotests/238 new file mode 100755 index 0000000000..f81ee1112f --- /dev/null +++ b/tests/qemu-iotests/238 @@ -0,0 +1,47 @@ +#!/usr/bin/env python +# +# Regression test for throttle group member unregister segfault with iothr= ead +# +# Copyright (c) 2019 Red Hat, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . +# + +import sys +import os +import iotests +from iotests import log + +sys.path.append(os.path.join(os.path.dirname(__file__), '..', '..', 'scrip= ts')) + +from qemu import QEMUMachine + +if iotests.qemu_default_machine =3D=3D 's390-ccw-virtio': + virtio_scsi_device =3D 'virtio-scsi-ccw' +else: + virtio_scsi_device =3D 'virtio-scsi-pci' + +vm =3D QEMUMachine(iotests.qemu_prog) +vm.add_args('-machine', 'accel=3Dkvm') +vm.launch() + +log(vm.qmp('blockdev-add', node_name=3D'hd0', driver=3D'null-co')) +log(vm.qmp('object-add', qom_type=3D'iothread', id=3D'iothread0')) +log(vm.qmp('device_add', id=3D'scsi0', driver=3Dvirtio_scsi_device, iothre= ad=3D'iothread0')) +log(vm.qmp('device_add', id=3D'scsi-hd0', driver=3D'scsi-hd', drive=3D'hd0= ')) +log(vm.qmp('block_set_io_throttle', id=3D'scsi-hd0', bps=3D0, bps_rd=3D0, = bps_wr=3D0, + iops=3D1000, iops_rd=3D0, iops_wr=3D0, conv_keys=3DFalse)) +log(vm.qmp('device_del', id=3D'scsi-hd0')) + +vm.shutdown() diff --git a/tests/qemu-iotests/238.out b/tests/qemu-iotests/238.out new file mode 100644 index 0000000000..4de840ba8c --- /dev/null +++ b/tests/qemu-iotests/238.out @@ -0,0 +1,6 @@ +{"return": {}} +{"return": {}} +{"return": {}} +{"return": {}} +{"return": {}} +{"return": {}} diff --git a/tests/qemu-iotests/group b/tests/qemu-iotests/group index 61a6d98ebd..1f55bcf4e6 100644 --- a/tests/qemu-iotests/group +++ b/tests/qemu-iotests/group @@ -233,3 +233,4 @@ 233 auto quick 234 auto quick migration 235 auto quick +238 auto quick --=20 2.20.1