From nobody Fri May 17 22:04:42 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1652369796; cv=none; d=zohomail.com; s=zohoarc; b=nY0zlUWginCJ2pUCd0e8gLgXY1frBoaCkEr4p0IRgL0kJT9hj49RRJ8Bjh3fewx/ej4vmGNdeDa3VEtan089ET6zu6rE7yA6WNsmtBUG+y5d1uGCJAWluppar4vVJtBYWiEqh5y6g/OlbgVTENNbbmwWvucSkyX8ortLtFwJ3CQ= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1652369796; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=VFCLKVLT3fV8vMr/G2rFQwUvAvQqi6JrN6pAE+j38Q0=; b=lv2f2NssgHcnB/BPIyBZx3X9Y5ywF0tFMEUvEJqgCEfbOlslMrInmvyTj5AxKBao5gIHZ+4Gr5BnnQgFUKul9U069HksR4g6rzSvXcskgqzBUfsVu88wndgN/BS0/Ck0+GevMvSwcEOmChyJc6rU8vN7bD17DYMdUlNqYdfbtMY= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 16523697966331021.3995517972915; Thu, 12 May 2022 08:36:36 -0700 (PDT) Received: from localhost ([::1]:50022 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1npAri-0000mc-D2 for importer@patchew.org; Thu, 12 May 2022 11:36:34 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:38924) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1npApY-00069G-GZ for qemu-devel@nongnu.org; Thu, 12 May 2022 11:34:21 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]:49910) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1npApW-0003Pv-2d for qemu-devel@nongnu.org; Thu, 12 May 2022 11:34:19 -0400 Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-633-YSBgTecQMOGRacbW3y2kig-1; Thu, 12 May 2022 11:34:14 -0400 Received: from smtp.corp.redhat.com (int-mx10.intmail.prod.int.rdu2.redhat.com [10.11.54.10]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id C3874296A607; Thu, 12 May 2022 15:34:13 +0000 (UTC) Received: from merkur.fritz.box (unknown [10.39.194.111]) by smtp.corp.redhat.com (Postfix) with ESMTP id 1599C401E9D; Thu, 12 May 2022 15:34:12 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1652369657; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=VFCLKVLT3fV8vMr/G2rFQwUvAvQqi6JrN6pAE+j38Q0=; b=fhDApdPgXKIOVbMQFrc1/jiV6rghbb4fM45RhF7zojtTjIeOrvgmMwov1kr+6mxTM5ZdL7 gILKC4tyRNKFYmq/By2zx4eJDbbTlqkoXRyI31KcdabKNZoMEEmG7d8qOpOGeJviDT5wSX xqZqQLHXLzrurIMUQoCZ/mk4bcdC+zk= X-MC-Unique: YSBgTecQMOGRacbW3y2kig-1 From: Kevin Wolf To: qemu-block@nongnu.org Cc: kwolf@redhat.com, qemu-devel@nongnu.org Subject: [PULL 01/10] coroutine: Rename qemu_coroutine_inc/dec_pool_size() Date: Thu, 12 May 2022 17:33:52 +0200 Message-Id: <20220512153401.202863-2-kwolf@redhat.com> In-Reply-To: <20220512153401.202863-1-kwolf@redhat.com> References: <20220512153401.202863-1-kwolf@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 2.85 on 10.11.54.10 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.129.124; envelope-from=kwolf@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -28 X-Spam_score: -2.9 X-Spam_bar: -- X-Spam_report: (-2.9 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.082, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1652369797297100003 Content-Type: text/plain; charset="utf-8" It's true that these functions currently affect the batch size in which coroutines are reused (i.e. moved from the global release pool to the allocation pool of a specific thread), but this is a bug and will be fixed in a separate patch. In fact, the comment in the header file already just promises that it influences the pool size, so reflect this in the name of the functions. As a nice side effect, the shorter function name makes some line wrapping unnecessary. Cc: qemu-stable@nongnu.org Signed-off-by: Kevin Wolf Message-Id: <20220510151020.105528-2-kwolf@redhat.com> Signed-off-by: Kevin Wolf --- include/qemu/coroutine.h | 6 +++--- hw/block/virtio-blk.c | 6 ++---- util/qemu-coroutine.c | 4 ++-- 3 files changed, 7 insertions(+), 9 deletions(-) diff --git a/include/qemu/coroutine.h b/include/qemu/coroutine.h index 284571badb..031cf23711 100644 --- a/include/qemu/coroutine.h +++ b/include/qemu/coroutine.h @@ -334,12 +334,12 @@ void coroutine_fn yield_until_fd_readable(int fd); /** * Increase coroutine pool size */ -void qemu_coroutine_increase_pool_batch_size(unsigned int additional_pool_= size); +void qemu_coroutine_inc_pool_size(unsigned int additional_pool_size); =20 /** - * Devcrease coroutine pool size + * Decrease coroutine pool size */ -void qemu_coroutine_decrease_pool_batch_size(unsigned int additional_pool_= size); +void qemu_coroutine_dec_pool_size(unsigned int additional_pool_size); =20 #include "qemu/lockable.h" =20 diff --git a/hw/block/virtio-blk.c b/hw/block/virtio-blk.c index 540c38f829..6a1cc41877 100644 --- a/hw/block/virtio-blk.c +++ b/hw/block/virtio-blk.c @@ -1215,8 +1215,7 @@ static void virtio_blk_device_realize(DeviceState *de= v, Error **errp) for (i =3D 0; i < conf->num_queues; i++) { virtio_add_queue(vdev, conf->queue_size, virtio_blk_handle_output); } - qemu_coroutine_increase_pool_batch_size(conf->num_queues * conf->queue= _size - / 2); + qemu_coroutine_inc_pool_size(conf->num_queues * conf->queue_size / 2); virtio_blk_data_plane_create(vdev, conf, &s->dataplane, &err); if (err !=3D NULL) { error_propagate(errp, err); @@ -1253,8 +1252,7 @@ static void virtio_blk_device_unrealize(DeviceState *= dev) for (i =3D 0; i < conf->num_queues; i++) { virtio_del_queue(vdev, i); } - qemu_coroutine_decrease_pool_batch_size(conf->num_queues * conf->queue= _size - / 2); + qemu_coroutine_dec_pool_size(conf->num_queues * conf->queue_size / 2); qemu_del_vm_change_state_handler(s->change); blockdev_mark_auto_del(s->blk); virtio_cleanup(vdev); diff --git a/util/qemu-coroutine.c b/util/qemu-coroutine.c index f3e8300c8d..ea23929a74 100644 --- a/util/qemu-coroutine.c +++ b/util/qemu-coroutine.c @@ -212,12 +212,12 @@ AioContext *coroutine_fn qemu_coroutine_get_aio_conte= xt(Coroutine *co) return co->ctx; } =20 -void qemu_coroutine_increase_pool_batch_size(unsigned int additional_pool_= size) +void qemu_coroutine_inc_pool_size(unsigned int additional_pool_size) { qatomic_add(&pool_batch_size, additional_pool_size); } =20 -void qemu_coroutine_decrease_pool_batch_size(unsigned int removing_pool_si= ze) +void qemu_coroutine_dec_pool_size(unsigned int removing_pool_size) { qatomic_sub(&pool_batch_size, removing_pool_size); } --=20 2.35.3 From nobody Fri May 17 22:04:42 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1652370014; cv=none; d=zohomail.com; s=zohoarc; b=AijDomjeWKmfgvZ1QJ+tlO9q6Iowyym0l0wh9JZAY8DcmAfWKc/5AvRLjA+PHHwWokRy3FEZpiJ1J36dO9AN8skum8eFRGLevNUVn7C1m50E2xQcL0nMlLEp82ir9zMBaK3TksEe6SHoUlfcrmcPncObwnbxQM5auIGkZggcO1U= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1652370014; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=e+/FCWiRghZmMPb+/DZSTzz2CkIYLtYwWqJCHxx/Stw=; b=ZW7PY9DWHij2JkzvQLpx3yWBUgH2JK1UVhewVA2A5sNTZ/IlQErFSsLRFpyQzozFXHUVDgsyIDBdjok+9vm8tm5MYbQfUakYgKQ1VwDvMgqfO1PEh5yeefgacMt3TnsVm5qk7eCD8rKX8qk7qRo/Y41XGHSRHm8htEWuj9hNzN4= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 165237001498466.18938204485733; Thu, 12 May 2022 08:40:14 -0700 (PDT) Received: from localhost ([::1]:59322 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1npAvF-0007KA-3V for importer@patchew.org; Thu, 12 May 2022 11:40:13 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:38928) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1npApY-00069I-Gv for qemu-devel@nongnu.org; Thu, 12 May 2022 11:34:21 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]:39889) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1npApU-0003PW-Rr for qemu-devel@nongnu.org; Thu, 12 May 2022 11:34:20 -0400 Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-625-hA5QFP4SOty_M-xhVWQOcA-1; Thu, 12 May 2022 11:34:15 -0400 Received: from smtp.corp.redhat.com (int-mx10.intmail.prod.int.rdu2.redhat.com [10.11.54.10]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id AA39B296A60A; Thu, 12 May 2022 15:34:14 +0000 (UTC) Received: from merkur.fritz.box (unknown [10.39.194.111]) by smtp.corp.redhat.com (Postfix) with ESMTP id 0FE5241373D; Thu, 12 May 2022 15:34:13 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1652369656; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=e+/FCWiRghZmMPb+/DZSTzz2CkIYLtYwWqJCHxx/Stw=; b=Xw4ozkOTyoaPLt6g1CwuGCF89yOgrmtpIi+Fjxkf89w5MA0hCS9FKQkcAxbc9HlDWodo32 VjdMttie7GXssnhWgCowbsocm4vdgAhp4PIRKNPvX2q7r8yOetnGs3iEvOri8Xg1rLHCoC TJ+La2PfNarEN/b4Zeu16nlBrQMEIjk= X-MC-Unique: hA5QFP4SOty_M-xhVWQOcA-1 From: Kevin Wolf To: qemu-block@nongnu.org Cc: kwolf@redhat.com, qemu-devel@nongnu.org Subject: [PULL 02/10] coroutine: Revert to constant batch size Date: Thu, 12 May 2022 17:33:53 +0200 Message-Id: <20220512153401.202863-3-kwolf@redhat.com> In-Reply-To: <20220512153401.202863-1-kwolf@redhat.com> References: <20220512153401.202863-1-kwolf@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 2.85 on 10.11.54.10 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.129.124; envelope-from=kwolf@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -28 X-Spam_score: -2.9 X-Spam_bar: -- X-Spam_report: (-2.9 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.082, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1652370016835100001 Content-Type: text/plain; charset="utf-8" Commit 4c41c69e changed the way the coroutine pool is sized because for virtio-blk devices with a large queue size and heavy I/O, it was just too small and caused coroutines to be deleted and reallocated soon afterwards. The change made the size dynamic based on the number of queues and the queue size of virtio-blk devices. There are two important numbers here: Slightly simplified, when a coroutine terminates, it is generally stored in the global release pool up to a certain pool size, and if the pool is full, it is freed. Conversely, when allocating a new coroutine, the coroutines in the release pool are reused if the pool already has reached a certain minimum size (the batch size), otherwise we allocate new coroutines. The problem after commit 4c41c69e is that it not only increases the maximum pool size (which is the intended effect), but also the batch size for reusing coroutines (which is a bug). It means that in cases with many devices and/or a large queue size (which defaults to the number of vcpus for virtio-blk-pci), many thousand coroutines could be sitting in the release pool without being reused. This is not only a waste of memory and allocations, but it actually makes the QEMU process likely to hit the vm.max_map_count limit on Linux because each coroutine requires two mappings (its stack and the guard page for the stack), causing it to abort() in qemu_alloc_stack() because when the limit is hit, mprotect() starts to fail with ENOMEM. In order to fix the problem, change the batch size back to 64 to avoid uselessly accumulating coroutines in the release pool, but keep the dynamic maximum pool size so that coroutines aren't freed too early in heavy I/O scenarios. Note that this fix doesn't strictly make it impossible to hit the limit, but this would only happen if most of the coroutines are actually in use at the same time, not just sitting in a pool. This is the same behaviour as we already had before commit 4c41c69e. Fully preventing this would require allowing qemu_coroutine_create() to return an error, but it doesn't seem to be a scenario that people hit in practice. Cc: qemu-stable@nongnu.org Resolves: https://bugzilla.redhat.com/show_bug.cgi?id=3D2079938 Fixes: 4c41c69e05fe28c0f95f8abd2ebf407e95a4f04b Signed-off-by: Kevin Wolf Message-Id: <20220510151020.105528-3-kwolf@redhat.com> Tested-by: Hiroki Narukawa Signed-off-by: Kevin Wolf --- util/qemu-coroutine.c | 22 ++++++++++++++-------- 1 file changed, 14 insertions(+), 8 deletions(-) diff --git a/util/qemu-coroutine.c b/util/qemu-coroutine.c index ea23929a74..4a8bd63ef0 100644 --- a/util/qemu-coroutine.c +++ b/util/qemu-coroutine.c @@ -21,14 +21,20 @@ #include "qemu/coroutine-tls.h" #include "block/aio.h" =20 -/** Initial batch size is 64, and is increased on demand */ +/** + * The minimal batch size is always 64, coroutines from the release_pool a= re + * reused as soon as there are 64 coroutines in it. The maximum pool size = starts + * with 64 and is increased on demand so that coroutines are not deleted e= ven if + * they are not immediately reused. + */ enum { - POOL_INITIAL_BATCH_SIZE =3D 64, + POOL_MIN_BATCH_SIZE =3D 64, + POOL_INITIAL_MAX_SIZE =3D 64, }; =20 /** Free list to speed up creation */ static QSLIST_HEAD(, Coroutine) release_pool =3D QSLIST_HEAD_INITIALIZER(p= ool); -static unsigned int pool_batch_size =3D POOL_INITIAL_BATCH_SIZE; +static unsigned int pool_max_size =3D POOL_INITIAL_MAX_SIZE; static unsigned int release_pool_size; =20 typedef QSLIST_HEAD(, Coroutine) CoroutineQSList; @@ -57,7 +63,7 @@ Coroutine *qemu_coroutine_create(CoroutineEntry *entry, v= oid *opaque) =20 co =3D QSLIST_FIRST(alloc_pool); if (!co) { - if (release_pool_size > qatomic_read(&pool_batch_size)) { + if (release_pool_size > POOL_MIN_BATCH_SIZE) { /* Slow path; a good place to register the destructor, too= . */ Notifier *notifier =3D get_ptr_coroutine_pool_cleanup_noti= fier(); if (!notifier->notify) { @@ -95,12 +101,12 @@ static void coroutine_delete(Coroutine *co) co->caller =3D NULL; =20 if (CONFIG_COROUTINE_POOL) { - if (release_pool_size < qatomic_read(&pool_batch_size) * 2) { + if (release_pool_size < qatomic_read(&pool_max_size) * 2) { QSLIST_INSERT_HEAD_ATOMIC(&release_pool, co, pool_next); qatomic_inc(&release_pool_size); return; } - if (get_alloc_pool_size() < qatomic_read(&pool_batch_size)) { + if (get_alloc_pool_size() < qatomic_read(&pool_max_size)) { QSLIST_INSERT_HEAD(get_ptr_alloc_pool(), co, pool_next); set_alloc_pool_size(get_alloc_pool_size() + 1); return; @@ -214,10 +220,10 @@ AioContext *coroutine_fn qemu_coroutine_get_aio_conte= xt(Coroutine *co) =20 void qemu_coroutine_inc_pool_size(unsigned int additional_pool_size) { - qatomic_add(&pool_batch_size, additional_pool_size); + qatomic_add(&pool_max_size, additional_pool_size); } =20 void qemu_coroutine_dec_pool_size(unsigned int removing_pool_size) { - qatomic_sub(&pool_batch_size, removing_pool_size); + qatomic_sub(&pool_max_size, removing_pool_size); } --=20 2.35.3 From nobody Fri May 17 22:04:42 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1652370042; cv=none; d=zohomail.com; s=zohoarc; b=ea9Rpbf+rMMN4oELF6ILv/QQw4N6+hrlRjDXCBN+pJv7/Dn+vd5/KttPXMX1wKBfGomCrs6dicOlMHgchjn3XxK2UG7iwBYIwVz7yLPxJhvymCUMg2CGac/AnnmlR4mCEHCiwIJw+G0g3MB8MXnfhEu4SyvCjU+1eilwTIrWvQE= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1652370042; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=qjGNCvj1Zken4M74xajCTiZhE3E89VrsFke8eY/QFUg=; b=AmAbqOE8rPZds/oRpwmfdcnKtJjYHnNFyMplI+/1wMKz7zdOcxa3lXSWw7gv4+qBgrQT3PT/Z7zhhk+/f/hC1UmNtnfgzPWFgpexYiDQerwuZcSme+wfNKopfCPvX6IRVkdCSRzY3mEzGfJuJsrAA/f0Go5aM/O1sh+92o+/nAk= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1652370042980567.0581229814396; Thu, 12 May 2022 08:40:42 -0700 (PDT) Received: from localhost ([::1]:60068 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1npAvh-0007qA-Ch for importer@patchew.org; Thu, 12 May 2022 11:40:41 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:39016) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1npApb-0006Ad-DB for qemu-devel@nongnu.org; Thu, 12 May 2022 11:34:25 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]:28166) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1npApY-0003Qa-8d for qemu-devel@nongnu.org; Thu, 12 May 2022 11:34:22 -0400 Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-131-jK06WjVbPL6i6-ANRS8z_Q-1; Thu, 12 May 2022 11:34:15 -0400 Received: from smtp.corp.redhat.com (int-mx10.intmail.prod.int.rdu2.redhat.com [10.11.54.10]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 99F02185A79C; Thu, 12 May 2022 15:34:15 +0000 (UTC) Received: from merkur.fritz.box (unknown [10.39.194.111]) by smtp.corp.redhat.com (Postfix) with ESMTP id E5C88401E9D; Thu, 12 May 2022 15:34:14 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1652369659; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=qjGNCvj1Zken4M74xajCTiZhE3E89VrsFke8eY/QFUg=; b=ir1OVlzgPTxM+LDYsvpMikuZ42zdd54ZCOud/L1llwlDAAiqqVCMGYTh82mnjcF0I1wAkh QisJUhqDkhHViOP8KR7hJOpMwf1+ztAI0tl0QnTePKvzYENfQHBgF78dmF/8SEjIf8LVJ/ oj9JrxepcUk1udCA2JtrpacalkE6Wc4= X-MC-Unique: jK06WjVbPL6i6-ANRS8z_Q-1 From: Kevin Wolf To: qemu-block@nongnu.org Cc: kwolf@redhat.com, qemu-devel@nongnu.org Subject: [PULL 03/10] iotests/testrunner: Flush after run_test() Date: Thu, 12 May 2022 17:33:54 +0200 Message-Id: <20220512153401.202863-4-kwolf@redhat.com> In-Reply-To: <20220512153401.202863-1-kwolf@redhat.com> References: <20220512153401.202863-1-kwolf@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 2.85 on 10.11.54.10 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.129.124; envelope-from=kwolf@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -28 X-Spam_score: -2.9 X-Spam_bar: -- X-Spam_report: (-2.9 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.082, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1652370044918100001 Content-Type: text/plain; charset="utf-8" From: Hanna Reitz When stdout is not a terminal, the buffer may not be flushed at each end of line, so we should flush after each test is done. This is especially apparent when run by check-block, in two ways: First, when running make check-block -jX with X > 1, progress indication was missing, even though testrunner.py does theoretically print each test's status once it has been run, even in multi-processing mode. Flushing after each test restores this progress indication. Second, sometimes make check-block failed altogether, with an error message that "too few tests [were] run". I presume that's because one worker process in the job pool did not get to flush its stdout before the main process exited, and so meson did not get to see that worker's test results. In any case, by flushing at the end of run_test(), the problem has disappeared for me. Signed-off-by: Hanna Reitz Message-Id: <20220506134215.10086-1-hreitz@redhat.com> Reviewed-by: Eric Blake Signed-off-by: Kevin Wolf --- tests/qemu-iotests/testrunner.py | 1 + 1 file changed, 1 insertion(+) diff --git a/tests/qemu-iotests/testrunner.py b/tests/qemu-iotests/testrunn= er.py index aae70a8341..10d9e8ef27 100644 --- a/tests/qemu-iotests/testrunner.py +++ b/tests/qemu-iotests/testrunner.py @@ -378,6 +378,7 @@ def run_test(self, test: str, else: print(res.casenotrun) =20 + sys.stdout.flush() return res =20 def run_tests(self, tests: List[str], jobs: int =3D 1) -> bool: --=20 2.35.3 From nobody Fri May 17 22:04:42 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1652370140; cv=none; d=zohomail.com; s=zohoarc; b=AyQ9uIQT6EJd2rXDULSol7uSm1S09Hpxhte1VLswd6roVSnUCjRef0kuic0nVomfnrG+sRti/L2Y0IK2XToInoAVOe0u5UpC4B5lytwU/EFeX7mYTU67TKMGNWZAiOVIAvlVx5znW52TWaDCb0NSiprzODNuynYzoQzouIaMZSE= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1652370140; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=YDr9YOkC3Q1s86mGL+/paUkGs6rnqmco0SjHyGvLG/A=; b=OIcFAmhF4IWbpLj6hgKK3dGA2JN6n7uR4rWkTWNYC14aoIgG0oKIwtVaLWdMwVt6JXqYgDxWQkL5Ivts/YNBykIGN2ZnFG0YKnzZFgnE45V2oM0PGEuohUYtZanNXPAMvT4FHV5+Gev4cxJmA3Zic0gXuanFCOSQMBe6A9fbfCI= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1652370140778341.3579903195043; Thu, 12 May 2022 08:42:20 -0700 (PDT) Received: from localhost ([::1]:37728 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1npAxH-0003fT-JL for importer@patchew.org; Thu, 12 May 2022 11:42:19 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:38974) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1npApZ-0006AX-PU for qemu-devel@nongnu.org; Thu, 12 May 2022 11:34:25 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]:28088) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1npApX-0003QS-NW for qemu-devel@nongnu.org; Thu, 12 May 2022 11:34:21 -0400 Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-185-wX_xHkvOMEqBOeBydKRfgA-1; Thu, 12 May 2022 11:34:16 -0400 Received: from smtp.corp.redhat.com (int-mx10.intmail.prod.int.rdu2.redhat.com [10.11.54.10]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 82AE580B90A; Thu, 12 May 2022 15:34:16 +0000 (UTC) Received: from merkur.fritz.box (unknown [10.39.194.111]) by smtp.corp.redhat.com (Postfix) with ESMTP id D6757401E9D; Thu, 12 May 2022 15:34:15 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1652369659; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=YDr9YOkC3Q1s86mGL+/paUkGs6rnqmco0SjHyGvLG/A=; b=aMm5d0iTjXlOS5ssZeGnSQaDPqNTJPxjLNUfeijQJtOtgqyUPnlio3FUoRnhP/8g7zw94R rjhwMrNHo5C4cuN73OS5NBothC6kpk9rmao80NcNUSSPoe+7nkt6MunfNz2n0Weu4GUC+1 JJ/ulnAJBvTAzUTXJkA62pAIS37fqAM= X-MC-Unique: wX_xHkvOMEqBOeBydKRfgA-1 From: Kevin Wolf To: qemu-block@nongnu.org Cc: kwolf@redhat.com, qemu-devel@nongnu.org Subject: [PULL 04/10] tests/qemu-iotests: print intent to run a test in TAP mode Date: Thu, 12 May 2022 17:33:55 +0200 Message-Id: <20220512153401.202863-5-kwolf@redhat.com> In-Reply-To: <20220512153401.202863-1-kwolf@redhat.com> References: <20220512153401.202863-1-kwolf@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 2.85 on 10.11.54.10 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.129.124; envelope-from=kwolf@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -28 X-Spam_score: -2.9 X-Spam_bar: -- X-Spam_report: (-2.9 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.082, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1652370141911100001 From: Daniel P. Berrang=C3=A9 When running I/O tests using TAP output mode, we get a single TAP test with a sub-test reported for each I/O test that is run. The output looks something like this: 1..123 ok qcow2 011 ok qcow2 012 ok qcow2 013 ok qcow2 217 ... If everything runs or fails normally this is fine, but periodically we have been seeing the test harness abort early before all 123 tests have been run, just leaving a fairly useless message like TAP parsing error: Too few tests run (expected 123, got 107) we have no idea which tests were running at the time the test harness abruptly exited. This change causes us to print a message about our intent to run each test, so we have a record of what is active at the time the harness exits abnormally. 1..123 # running qcow2 011 ok qcow2 011 # running qcow2 012 ok qcow2 012 # running qcow2 013 ok qcow2 013 # running qcow2 217 ok qcow2 217 ... Signed-off-by: Daniel P. Berrang=C3=A9 Message-Id: <20220509124134.867431-2-berrange@redhat.com> Reviewed-by: Thomas Huth Signed-off-by: Kevin Wolf --- tests/qemu-iotests/testrunner.py | 3 +++ 1 file changed, 3 insertions(+) diff --git a/tests/qemu-iotests/testrunner.py b/tests/qemu-iotests/testrunn= er.py index 10d9e8ef27..5a771da86e 100644 --- a/tests/qemu-iotests/testrunner.py +++ b/tests/qemu-iotests/testrunner.py @@ -361,6 +361,9 @@ def run_test(self, test: str, starttime=3Dstart, lasttime=3Dlast_el, end =3D '\n' if mp else '\r') + else: + testname =3D os.path.basename(test) + print(f'# running {self.env.imgfmt} {testname}') =20 res =3D self.do_run_test(test, mp) =20 --=20 2.35.3 From nobody Fri May 17 22:04:42 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1652370055; cv=none; d=zohomail.com; s=zohoarc; b=ltZMCfSiAqL+hjAfwR3323PcRRo80A+7LVnOWdBhzMXzYgBFZr+JDDQHchZZkp6X+gwV6US8ikv47zVSFJ1EbekW14NrxHINpcmjdBLK8dg0y/DOO3leXAaYtn6AYheq4ewZS4ino50pD92HiubDev303+dr3BF1X0YDv/l3ing= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1652370055; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=n28Cd9eeQ1fEIqgdq9yvf+4QfWs332/Z2K07s3gYTqs=; b=HMJ74hkSdXi8unTnWS9mzWe5dkEV5ywg09mkOwNXlH3kl5iF4gxXz5g01CsRkVbBdmpnTFI7UZ4Vn4qhn06+fWvRTNGp/dCpqM1Z2edWTgny1nk2n/4q038RrdG2v35hjAJo/W1s+x/Syr4NEOihQdKAKsZdHJ9o4DAetHytph8= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1652370055515899.1669044817048; Thu, 12 May 2022 08:40:55 -0700 (PDT) Received: from localhost ([::1]:60642 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1npAvu-0008Hj-F5 for importer@patchew.org; Thu, 12 May 2022 11:40:54 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:39024) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1npApb-0006Ag-E0 for qemu-devel@nongnu.org; Thu, 12 May 2022 11:34:25 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]:49064) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1npApY-0003QW-8I for qemu-devel@nongnu.org; Thu, 12 May 2022 11:34:22 -0400 Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-263-zpnq4FnVNT6MmjjV-qHaKw-1; Thu, 12 May 2022 11:34:17 -0400 Received: from smtp.corp.redhat.com (int-mx10.intmail.prod.int.rdu2.redhat.com [10.11.54.10]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 5E504100F84A; Thu, 12 May 2022 15:34:17 +0000 (UTC) Received: from merkur.fritz.box (unknown [10.39.194.111]) by smtp.corp.redhat.com (Postfix) with ESMTP id BBC20401E9D; Thu, 12 May 2022 15:34:16 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1652369659; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=n28Cd9eeQ1fEIqgdq9yvf+4QfWs332/Z2K07s3gYTqs=; b=E3qH+TSW4na5ulbUd1kw3kd1ZXvemztW5qXcuxSTfpkILi+forNLgKhQl0q8X9Y6PDAOhx C7vh/bRX8EaQJS7ZAGcWWGi43k50sZXk5aVpCXmFLa9ZfSfNz3XVzlS59NCb8Le7KkJ+Qe 6bmq9e/pXXitRIMJ+P921le6pUNCf8c= X-MC-Unique: zpnq4FnVNT6MmjjV-qHaKw-1 From: Kevin Wolf To: qemu-block@nongnu.org Cc: kwolf@redhat.com, qemu-devel@nongnu.org Subject: [PULL 05/10] .gitlab-ci.d: export meson testlog.txt as an artifact Date: Thu, 12 May 2022 17:33:56 +0200 Message-Id: <20220512153401.202863-6-kwolf@redhat.com> In-Reply-To: <20220512153401.202863-1-kwolf@redhat.com> References: <20220512153401.202863-1-kwolf@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 2.85 on 10.11.54.10 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.129.124; envelope-from=kwolf@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -28 X-Spam_score: -2.9 X-Spam_bar: -- X-Spam_report: (-2.9 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.082, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1652370056953100001 From: Daniel P. Berrang=C3=A9 When running 'make check' we only get a summary of progress on the console. Fortunately meson/ninja have saved the raw test output to a logfile. Exposing this log will make it easier to debug failures that happen in CI. Signed-off-by: Daniel P. Berrang=C3=A9 Message-Id: <20220509124134.867431-3-berrange@redhat.com> Reviewed-by: Thomas Huth Reviewed-by: Philippe Mathieu-Daud=C3=A9 Signed-off-by: Kevin Wolf --- .gitlab-ci.d/buildtest-template.yml | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/.gitlab-ci.d/buildtest-template.yml b/.gitlab-ci.d/buildtest-t= emplate.yml index 2c7980a4f6..dc6d67aacf 100644 --- a/.gitlab-ci.d/buildtest-template.yml +++ b/.gitlab-ci.d/buildtest-template.yml @@ -26,7 +26,7 @@ make -j"$JOBS" $MAKE_CHECK_ARGS ; fi =20 -.native_test_job_template: +.common_test_job_template: stage: test image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest script: @@ -37,8 +37,16 @@ # Avoid recompiling by hiding ninja with NINJA=3D":" - make NINJA=3D":" $MAKE_CHECK_ARGS =20 +.native_test_job_template: + extends: .common_test_job_template + artifacts: + name: "$CI_JOB_NAME-$CI_COMMIT_REF_SLUG" + expire_in: 7 days + paths: + - build/meson-logs/testlog.txt + .avocado_test_job_template: - extends: .native_test_job_template + extends: .common_test_job_template cache: key: "${CI_JOB_NAME}-cache" paths: --=20 2.35.3 From nobody Fri May 17 22:04:42 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1652370374; cv=none; d=zohomail.com; s=zohoarc; b=Wuoom9perfDbFjhwA4VWKm7PczrFrNrvbgxyMgmVhE1//QF+8DSywoo0j4AVImG0OWZG9rR7cQjv/hi+Se4NkcXaMjTKYWc21D2/4uFgvuyaxJJtIak0gU7KPCn1xJ48G/7hc2zWQYvUVF268Qut45q9SN6zILHfC19LKEouzkI= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1652370374; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=8o+ph8BTyXCS9KvKvIQa4Ljx49+bdnjo25PLmYJa1MY=; b=ZodFxY9IAZThZ8yJEdzomMFf9sJXBCzbxMKGwSHHsmty8BeSvxPGMaDM0E+bkWJETaL/ZVCHD76MteneCGcje1zrM3CfHyuxfSIynLSYhWi3oLnHH+mK4K3m9Mu0SHx1fpYS08ipVMgNFv0Ybi5LAbu19ONkhJAr7xlJD5Qa/Tg= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1652370374468172.68837641786035; Thu, 12 May 2022 08:46:14 -0700 (PDT) Received: from localhost ([::1]:41992 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1npB13-0006qp-8t for importer@patchew.org; Thu, 12 May 2022 11:46:13 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:39064) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1npApc-0006Ak-Rz for qemu-devel@nongnu.org; Thu, 12 May 2022 11:34:25 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:34021) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1npApb-0003RW-4t for qemu-devel@nongnu.org; Thu, 12 May 2022 11:34:24 -0400 Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-620-V156yKbEN8-qTedH0XdpXA-1; Thu, 12 May 2022 11:34:18 -0400 Received: from smtp.corp.redhat.com (int-mx10.intmail.prod.int.rdu2.redhat.com [10.11.54.10]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 3D330185A79C; Thu, 12 May 2022 15:34:18 +0000 (UTC) Received: from merkur.fritz.box (unknown [10.39.194.111]) by smtp.corp.redhat.com (Postfix) with ESMTP id 97EF1416156; Thu, 12 May 2022 15:34:17 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1652369661; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=8o+ph8BTyXCS9KvKvIQa4Ljx49+bdnjo25PLmYJa1MY=; b=NziVAq0tYddOn63yvsMqBOudPygRwsgGi3V+Dl+Egao3CjiJwylVSgPecPpMwovpzZwa8C a89pp7w0VL0Q09EvM4GCkYsKCChJlZ5l/qhBCM6WxgYpD6ahU0W9ZAv5EQW84eb21ru/hn SyI62ZVOWwa+n8SuTQRdI3xtOOguE/Q= X-MC-Unique: V156yKbEN8-qTedH0XdpXA-1 From: Kevin Wolf To: qemu-block@nongnu.org Cc: kwolf@redhat.com, qemu-devel@nongnu.org Subject: [PULL 06/10] hw/block/fdc: Prevent end-of-track overrun (CVE-2021-3507) Date: Thu, 12 May 2022 17:33:57 +0200 Message-Id: <20220512153401.202863-7-kwolf@redhat.com> In-Reply-To: <20220512153401.202863-1-kwolf@redhat.com> References: <20220512153401.202863-1-kwolf@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 2.85 on 10.11.54.10 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.133.124; envelope-from=kwolf@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -28 X-Spam_score: -2.9 X-Spam_bar: -- X-Spam_report: (-2.9 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.082, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1652370375043100001 From: Philippe Mathieu-Daud=C3=A9 Per the 82078 datasheet, if the end-of-track (EOT byte in the FIFO) is more than the number of sectors per side, the command is terminated unsuccessfully: * 5.2.5 DATA TRANSFER TERMINATION The 82078 supports terminal count explicitly through the TC pin and implicitly through the underrun/over- run and end-of-track (EOT) functions. For full sector transfers, the EOT parameter can define the last sector to be transferred in a single or multisector transfer. If the last sector to be transferred is a par- tial sector, the host can stop transferring the data in mid-sector, and the 82078 will continue to complete the sector as if a hardware TC was received. The only difference between these implicit functions and TC is that they return "abnormal termination" result status. Such status indications can be ignored if they were expected. * 6.1.3 READ TRACK This command terminates when the EOT specified number of sectors have been read. If the 82078 does not find an I D Address Mark on the diskette after the second=C2=B7 occurrence of a pulse on the INDX# pin, then it sets the IC code in Status Regis- ter 0 to "01" (Abnormal termination), sets the MA bit in Status Register 1 to "1", and terminates the com- mand. * 6.1.6 VERIFY Refer to Table 6-6 and Table 6-7 for information concerning the values of MT and EC versus SC and EOT value. * Table 6=C2=B76. Result Phase Table * Table 6-7. Verify Command Result Phase Table Fix by aborting the transfer when EOT > # Sectors Per Side. Cc: qemu-stable@nongnu.org Cc: Herv=C3=A9 Poussineau Fixes: baca51faff0 ("floppy driver: disk geometry auto detect") Reported-by: Alexander Bulekov Resolves: https://gitlab.com/qemu-project/qemu/-/issues/339 Signed-off-by: Philippe Mathieu-Daud=C3=A9 Message-Id: <20211118115733.4038610-2-philmd@redhat.com> Reviewed-by: Hanna Reitz Signed-off-by: Kevin Wolf --- hw/block/fdc.c | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/hw/block/fdc.c b/hw/block/fdc.c index 347875a0cd..57bb355794 100644 --- a/hw/block/fdc.c +++ b/hw/block/fdc.c @@ -1530,6 +1530,14 @@ static void fdctrl_start_transfer(FDCtrl *fdctrl, in= t direction) int tmp; fdctrl->data_len =3D 128 << (fdctrl->fifo[5] > 7 ? 7 : fdctrl->fif= o[5]); tmp =3D (fdctrl->fifo[6] - ks + 1); + if (tmp < 0) { + FLOPPY_DPRINTF("invalid EOT: %d\n", tmp); + fdctrl_stop_transfer(fdctrl, FD_SR0_ABNTERM, FD_SR1_MA, 0x00); + fdctrl->fifo[3] =3D kt; + fdctrl->fifo[4] =3D kh; + fdctrl->fifo[5] =3D ks; + return; + } if (fdctrl->fifo[0] & 0x80) tmp +=3D fdctrl->fifo[6]; fdctrl->data_len *=3D tmp; --=20 2.35.3 From nobody Fri May 17 22:04:42 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1652369830; cv=none; d=zohomail.com; s=zohoarc; b=ID+rZSoNsz7XRIhKU6S8wQVzH6AUnlnuRahflLcqQxtluSL0+F7AbsZu1dBhDiPPmJ8Pxr05kx8SsVma65KeA6ATZ3mSTvamoECsIHzTo0VZhYD+VqlVsMptuB4eqs+1ccnQGZ65/Hn7oPZYaQiShIEFXHArdO1//YlyruMNVuE= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1652369830; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=LlHp2QFgV12afbC9iBiyFGCbFkmU937vOMmvo5pijbs=; b=GRRI/Z4P15l02vlC2sW7ut0O5ZQGfUMXvDnw3Ef0Jwz2EVn+qV+203bqdbnWGMnE6U8eUI4aIm20RcLlCnTdEVYUJXqNR+zjJDBTojh776a1m9F9MjeF8+71v3WcYoIvfd1AZX7K8XmYmYz8QzNaQZdx3VKNCNyeNIMfWbKEuJ0= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1652369830056111.03120272790557; Thu, 12 May 2022 08:37:10 -0700 (PDT) Received: from localhost ([::1]:51466 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1npAsG-0001jx-SD for importer@patchew.org; Thu, 12 May 2022 11:37:08 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:39072) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1npApd-0006Al-5s for qemu-devel@nongnu.org; Thu, 12 May 2022 11:34:25 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:36572) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1npApb-0003Rf-6U for qemu-devel@nongnu.org; Thu, 12 May 2022 11:34:24 -0400 Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-90-b2_ITH0INZ6lUl7eba_0kA-1; Thu, 12 May 2022 11:34:19 -0400 Received: from smtp.corp.redhat.com (int-mx10.intmail.prod.int.rdu2.redhat.com [10.11.54.10]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 1E2658038E3; Thu, 12 May 2022 15:34:19 +0000 (UTC) Received: from merkur.fritz.box (unknown [10.39.194.111]) by smtp.corp.redhat.com (Postfix) with ESMTP id 774A0401E9D; Thu, 12 May 2022 15:34:18 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1652369662; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=LlHp2QFgV12afbC9iBiyFGCbFkmU937vOMmvo5pijbs=; b=SbQhodAZASbJmEURjhxXpC7BzvRbOegm65VsqQnUDGZaiH0L2IrYEEyLpSRsSG5APUOJvU zzYF/1NMln8FHhvE4Dr3oX31maK4fjSsu6SQwQN+Y9CcSLTSNPzd3BaVbl0L+ggW495hui VzA/fN3kmc+891TCC/dWE4+2dMxn4gU= X-MC-Unique: b2_ITH0INZ6lUl7eba_0kA-1 From: Kevin Wolf To: qemu-block@nongnu.org Cc: kwolf@redhat.com, qemu-devel@nongnu.org Subject: [PULL 07/10] tests/qtest/fdc-test: Add a regression test for CVE-2021-3507 Date: Thu, 12 May 2022 17:33:58 +0200 Message-Id: <20220512153401.202863-8-kwolf@redhat.com> In-Reply-To: <20220512153401.202863-1-kwolf@redhat.com> References: <20220512153401.202863-1-kwolf@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 2.85 on 10.11.54.10 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.133.124; envelope-from=kwolf@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -28 X-Spam_score: -2.9 X-Spam_bar: -- X-Spam_report: (-2.9 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.082, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1652369831603100002 From: Philippe Mathieu-Daud=C3=A9 Add the reproducer from https://gitlab.com/qemu-project/qemu/-/issues/339 Without the previous commit, when running 'make check-qtest-i386' with QEMU configured with '--enable-sanitizers' we get: =3D=3D4028352=3D=3DERROR: AddressSanitizer: heap-buffer-overflow on addre= ss 0x619000062a00 at pc 0x5626d03c491a bp 0x7ffdb4199410 sp 0x7ffdb4198bc0 READ of size 786432 at 0x619000062a00 thread T0 #0 0x5626d03c4919 in __asan_memcpy (qemu-system-i386+0x1e65919) #1 0x5626d1c023cc in flatview_write_continue softmmu/physmem.c:2787:13 #2 0x5626d1bf0c0f in flatview_write softmmu/physmem.c:2822:14 #3 0x5626d1bf0798 in address_space_write softmmu/physmem.c:2914:18 #4 0x5626d1bf0f37 in address_space_rw softmmu/physmem.c:2924:16 #5 0x5626d1bf14c8 in cpu_physical_memory_rw softmmu/physmem.c:2933:5 #6 0x5626d0bd5649 in cpu_physical_memory_write include/exec/cpu-commo= n.h:82:5 #7 0x5626d0bd0a07 in i8257_dma_write_memory hw/dma/i8257.c:452:9 #8 0x5626d09f825d in fdctrl_transfer_handler hw/block/fdc.c:1616:13 #9 0x5626d0a048b4 in fdctrl_start_transfer hw/block/fdc.c:1539:13 #10 0x5626d09f4c3e in fdctrl_write_data hw/block/fdc.c:2266:13 #11 0x5626d09f22f7 in fdctrl_write hw/block/fdc.c:829:9 #12 0x5626d1c20bc5 in portio_write softmmu/ioport.c:207:17 0x619000062a00 is located 0 bytes to the right of 512-byte region [0x6190= 00062800,0x619000062a00) allocated by thread T0 here: #0 0x5626d03c66ec in posix_memalign (qemu-system-i386+0x1e676ec) #1 0x5626d2b988d4 in qemu_try_memalign util/oslib-posix.c:210:11 #2 0x5626d2b98b0c in qemu_memalign util/oslib-posix.c:226:27 #3 0x5626d09fbaf0 in fdctrl_realize_common hw/block/fdc.c:2341:20 #4 0x5626d0a150ed in isabus_fdc_realize hw/block/fdc-isa.c:113:5 #5 0x5626d2367935 in device_set_realized hw/core/qdev.c:531:13 SUMMARY: AddressSanitizer: heap-buffer-overflow (qemu-system-i386+0x1e659= 19) in __asan_memcpy Shadow bytes around the buggy address: 0x0c32800044f0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x0c3280004500: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x0c3280004510: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x0c3280004520: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x0c3280004530: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 =3D>0x0c3280004540:[fa]fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x0c3280004550: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x0c3280004560: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x0c3280004570: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x0c3280004580: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x0c3280004590: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd Shadow byte legend (one shadow byte represents 8 application bytes): Addressable: 00 Heap left redzone: fa Freed heap region: fd =3D=3D4028352=3D=3DABORTING [ kwolf: Added snapshot=3Don to prevent write file lock failure ] Reported-by: Alexander Bulekov Signed-off-by: Philippe Mathieu-Daud=C3=A9 Reviewed-by: Alexander Bulekov Signed-off-by: Kevin Wolf --- tests/qtest/fdc-test.c | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) diff --git a/tests/qtest/fdc-test.c b/tests/qtest/fdc-test.c index 0b3c2c0d52..52ade90a7d 100644 --- a/tests/qtest/fdc-test.c +++ b/tests/qtest/fdc-test.c @@ -582,6 +582,26 @@ static void test_cve_2021_20196(void) qtest_quit(s); } =20 +static void test_cve_2021_3507(void) +{ + QTestState *s; + + s =3D qtest_initf("-nographic -m 32M -nodefaults " + "-drive file=3D%s,format=3Draw,if=3Dfloppy,snapshot=3D= on", + test_image); + qtest_outl(s, 0x9, 0x0a0206); + qtest_outw(s, 0x3f4, 0x1600); + qtest_outw(s, 0x3f4, 0x0000); + qtest_outw(s, 0x3f4, 0x0000); + qtest_outw(s, 0x3f4, 0x0000); + qtest_outw(s, 0x3f4, 0x0200); + qtest_outw(s, 0x3f4, 0x0200); + qtest_outw(s, 0x3f4, 0x0000); + qtest_outw(s, 0x3f4, 0x0000); + qtest_outw(s, 0x3f4, 0x0000); + qtest_quit(s); +} + int main(int argc, char **argv) { int fd; @@ -613,6 +633,7 @@ int main(int argc, char **argv) qtest_add_func("/fdc/read_no_dma_19", test_read_no_dma_19); qtest_add_func("/fdc/fuzz-registers", fuzz_registers); qtest_add_func("/fdc/fuzz/cve_2021_20196", test_cve_2021_20196); + qtest_add_func("/fdc/fuzz/cve_2021_3507", test_cve_2021_3507); =20 ret =3D g_test_run(); =20 --=20 2.35.3 From nobody Fri May 17 22:04:42 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1652369829; cv=none; d=zohomail.com; s=zohoarc; b=AalaCCU75sMMN24LWDbZ5Wvrd8jiJ+II03wS4xTCYsoDy9ALNpkMuk6JJEaGmmxZ8iiJifcrNjJgrX4zCnIApCybtf4LuO2OykoqizFbMPhYHJH1PalMm6EjxEpNhpxX98vqcS43HS7AlE5jkWqY3HZbCYorwQTsDqQ5Mkgt/ck= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1652369829; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=S/ah9paPVxbBVZphrAhEc0BYKrdp2jOfBKGneKB97Rs=; b=SOELIXyDJoYKpQyL8tLSs1pjAmrAtjAqWWpSAfIBUiUyJGmiTKi18+ejbJiFp66S5rsRKxlxW4G3NBOTuY//V0X+9ogPnUB35MSZKwn85GtS/hW8GkFo1HglGGn/AVKhnbg72AXfzNyPiviG/LF+EkJMq3ep6Vr/PCVuCwYvteU= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1652369829931821.4352187187598; Thu, 12 May 2022 08:37:09 -0700 (PDT) Received: from localhost ([::1]:51456 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1npAsG-0001jV-Ua for importer@patchew.org; Thu, 12 May 2022 11:37:08 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:39062) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1npApc-0006Aj-Ll for qemu-devel@nongnu.org; Thu, 12 May 2022 11:34:25 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]:33614) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1npApb-0003RS-3F for qemu-devel@nongnu.org; Thu, 12 May 2022 11:34:24 -0400 Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-47-FWAdukNeO0iLOeZB7RSywQ-1; Thu, 12 May 2022 11:34:20 -0400 Received: from smtp.corp.redhat.com (int-mx10.intmail.prod.int.rdu2.redhat.com [10.11.54.10]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id F0C9F802A5B; Thu, 12 May 2022 15:34:19 +0000 (UTC) Received: from merkur.fritz.box (unknown [10.39.194.111]) by smtp.corp.redhat.com (Postfix) with ESMTP id 5AE5C416156; Thu, 12 May 2022 15:34:19 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1652369661; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=S/ah9paPVxbBVZphrAhEc0BYKrdp2jOfBKGneKB97Rs=; b=QGWZV1XzRBWK/cdeoF8MfOssS9n0wbI/WvE9ExCcuIkPqyokjNzQhWcMYX3gYp6AM0+Rrm cpufRLd712V7lBun/wsYg7g0B9S4ndWdGSXfgoUCAM2sXjRZ6/I75RvoNWVz9Uhd6L6BW1 u8DxHz2BOmdU4dDEw2RwepJW5OqYfts= X-MC-Unique: FWAdukNeO0iLOeZB7RSywQ-1 From: Kevin Wolf To: qemu-block@nongnu.org Cc: kwolf@redhat.com, qemu-devel@nongnu.org Subject: [PULL 08/10] qemu-nbd: Pass max connections to blockdev layer Date: Thu, 12 May 2022 17:33:59 +0200 Message-Id: <20220512153401.202863-9-kwolf@redhat.com> In-Reply-To: <20220512153401.202863-1-kwolf@redhat.com> References: <20220512153401.202863-1-kwolf@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 2.85 on 10.11.54.10 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.129.124; envelope-from=kwolf@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -28 X-Spam_score: -2.9 X-Spam_bar: -- X-Spam_report: (-2.9 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.082, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1652369831602100001 Content-Type: text/plain; charset="utf-8" From: Eric Blake The next patch wants to adjust whether the NBD server code advertises MULTI_CONN based on whether it is known if the server limits to exactly one client. For a server started by QMP, this information is obtained through nbd_server_start (which can support more than one export); but for qemu-nbd (which supports exactly one export), it is controlled only by the command-line option -e/--shared. Since we already have a hook function used by qemu-nbd, it's easiest to just alter its signature to fit our needs. Signed-off-by: Eric Blake Message-Id: <20220512004924.417153-2-eblake@redhat.com> Signed-off-by: Kevin Wolf --- include/block/nbd.h | 2 +- blockdev-nbd.c | 8 ++++---- qemu-nbd.c | 2 +- 3 files changed, 6 insertions(+), 6 deletions(-) diff --git a/include/block/nbd.h b/include/block/nbd.h index a98eb665da..c5a29ce1c6 100644 --- a/include/block/nbd.h +++ b/include/block/nbd.h @@ -344,7 +344,7 @@ void nbd_client_new(QIOChannelSocket *sioc, void nbd_client_get(NBDClient *client); void nbd_client_put(NBDClient *client); =20 -void nbd_server_is_qemu_nbd(bool value); +void nbd_server_is_qemu_nbd(int max_connections); bool nbd_server_is_running(void); void nbd_server_start(SocketAddress *addr, const char *tls_creds, const char *tls_authz, uint32_t max_connections, diff --git a/blockdev-nbd.c b/blockdev-nbd.c index 7f6531cba0..711e0e72bd 100644 --- a/blockdev-nbd.c +++ b/blockdev-nbd.c @@ -30,18 +30,18 @@ typedef struct NBDServerData { } NBDServerData; =20 static NBDServerData *nbd_server; -static bool is_qemu_nbd; +static int qemu_nbd_connections =3D -1; /* Non-negative if this is qemu-nb= d */ =20 static void nbd_update_server_watch(NBDServerData *s); =20 -void nbd_server_is_qemu_nbd(bool value) +void nbd_server_is_qemu_nbd(int max_connections) { - is_qemu_nbd =3D value; + qemu_nbd_connections =3D max_connections; } =20 bool nbd_server_is_running(void) { - return nbd_server || is_qemu_nbd; + return nbd_server || qemu_nbd_connections >=3D 0; } =20 static void nbd_blockdev_client_closed(NBDClient *client, bool ignored) diff --git a/qemu-nbd.c b/qemu-nbd.c index 2382b5042a..0cd5aa6f02 100644 --- a/qemu-nbd.c +++ b/qemu-nbd.c @@ -1095,7 +1095,7 @@ int main(int argc, char **argv) =20 bs->detect_zeroes =3D detect_zeroes; =20 - nbd_server_is_qemu_nbd(true); + nbd_server_is_qemu_nbd(shared); =20 export_opts =3D g_new(BlockExportOptions, 1); *export_opts =3D (BlockExportOptions) { --=20 2.35.3 From nobody Fri May 17 22:04:42 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1652370788; cv=none; d=zohomail.com; s=zohoarc; b=POjZuJOzcNZP6HNiPLhmnrCzijaUVFVavniNXnQD0UWr+o1sbo/w0Oy21KcTpsUOpGCdcw2AKtZbHlqN6iJq9rgnbHWYKPBD+9Ym94dkd9R1BSwpjxp72weiNJTh+3sorUtefwAKGFcMY70a8QuvI0UdOwHyil77yaRFjA85VZs= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1652370788; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=VbTpvtLEVzwjRk2+8JuUmLmJDFvkRzxxk82q6SYJGQw=; b=OnzU3kf38WSI7PO+0hX1Vkf/t+7u3t/00TpFdvD9JA8Erdq1+CgFrD1oyPadO2dLIKwEtAFbQrAiexhWS1HzTfskOP9kuWNNCoTfy4zGxkdxVC5GoD3TnxmrPr7jwaQkzXco9IDuVOduIAvuIDrb3+2clnv6t2mi9PhCiSaAA58= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1652370788741433.89632312085223; Thu, 12 May 2022 08:53:08 -0700 (PDT) Received: from localhost ([::1]:58860 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1npB7i-0001Xp-Sd for importer@patchew.org; Thu, 12 May 2022 11:53:06 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:39142) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1npApj-0006DT-0s for qemu-devel@nongnu.org; Thu, 12 May 2022 11:34:31 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:41642) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1npApf-0003TA-6W for qemu-devel@nongnu.org; Thu, 12 May 2022 11:34:29 -0400 Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-152-HYhUE1duPJeAMAlckRmZ3A-1; Thu, 12 May 2022 11:34:21 -0400 Received: from smtp.corp.redhat.com (int-mx10.intmail.prod.int.rdu2.redhat.com [10.11.54.10]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id CF04929324AC; Thu, 12 May 2022 15:34:20 +0000 (UTC) Received: from merkur.fritz.box (unknown [10.39.194.111]) by smtp.corp.redhat.com (Postfix) with ESMTP id 36295416156; Thu, 12 May 2022 15:34:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1652369666; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=VbTpvtLEVzwjRk2+8JuUmLmJDFvkRzxxk82q6SYJGQw=; b=cSUI/1KsPWny3lk4bHAN3TrpyqHdjyWlSH39Tpxhy3l6wstZdsXvmiBij7LFUCB1DHoq/v +EGfhvL39mCDk9CLWQfl+fpfVafpCWCToRv0lj7ZwoNRtWiSub859FFpCb+RxwlTpRTjHW A2U/FLA2LqroohaN3PP00MeJ/elCe88= X-MC-Unique: HYhUE1duPJeAMAlckRmZ3A-1 From: Kevin Wolf To: qemu-block@nongnu.org Cc: kwolf@redhat.com, qemu-devel@nongnu.org Subject: [PULL 09/10] nbd/server: Allow MULTI_CONN for shared writable exports Date: Thu, 12 May 2022 17:34:00 +0200 Message-Id: <20220512153401.202863-10-kwolf@redhat.com> In-Reply-To: <20220512153401.202863-1-kwolf@redhat.com> References: <20220512153401.202863-1-kwolf@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 2.85 on 10.11.54.10 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.133.124; envelope-from=kwolf@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -28 X-Spam_score: -2.9 X-Spam_bar: -- X-Spam_report: (-2.9 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.082, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1652370790656100001 Content-Type: text/plain; charset="utf-8" From: Eric Blake According to the NBD spec, a server that advertises NBD_FLAG_CAN_MULTI_CONN promises that multiple client connections will not see any cache inconsistencies: when properly separated by a single flush, actions performed by one client will be visible to another client, regardless of which client did the flush. We always satisfy these conditions in qemu - even when we support multiple clients, ALL clients go through a single point of reference into the block layer, with no local caching. The effect of one client is instantly visible to the next client. Even if our backend were a network device, we argue that any multi-path caching effects that would cause inconsistencies in back-to-back actions not seeing the effect of previous actions would be a bug in that backend, and not the fault of caching in qemu. As such, it is safe to unconditionally advertise CAN_MULTI_CONN for any qemu NBD server situation that supports parallel clients. Note, however, that we don't want to advertise CAN_MULTI_CONN when we know that a second client cannot connect (for historical reasons, qemu-nbd defaults to a single connection while nbd-server-add and QMP commands default to unlimited connections; but we already have existing means to let either style of NBD server creation alter those defaults). This is visible by no longer advertising MULTI_CONN for 'qemu-nbd -r' without -e, as in the iotest nbd-qemu-allocation. The harder part of this patch is setting up an iotest to demonstrate behavior of multiple NBD clients to a single server. It might be possible with parallel qemu-io processes, but I found it easier to do in python with the help of libnbd, and help from Nir and Vladimir in writing the test. Signed-off-by: Eric Blake Suggested-by: Nir Soffer Suggested-by: Vladimir Sementsov-Ogievskiy Message-Id: <20220512004924.417153-3-eblake@redhat.com> Signed-off-by: Kevin Wolf --- qapi/block-export.json | 8 +- docs/interop/nbd.txt | 1 + docs/tools/qemu-nbd.rst | 3 +- include/block/nbd.h | 3 +- blockdev-nbd.c | 5 + nbd/server.c | 10 +- MAINTAINERS | 1 + tests/qemu-iotests/tests/nbd-multiconn | 145 ++++++++++++++++++ tests/qemu-iotests/tests/nbd-multiconn.out | 5 + .../tests/nbd-qemu-allocation.out | 2 +- 10 files changed, 172 insertions(+), 11 deletions(-) create mode 100755 tests/qemu-iotests/tests/nbd-multiconn create mode 100644 tests/qemu-iotests/tests/nbd-multiconn.out diff --git a/qapi/block-export.json b/qapi/block-export.json index 1de16d2589..7776248435 100644 --- a/qapi/block-export.json +++ b/qapi/block-export.json @@ -22,7 +22,9 @@ # recreated on the fly while the NBD server is active. # If missing, it will default to denying access (since 4.0). # @max-connections: The maximum number of connections to allow at the same -# time, 0 for unlimited. (since 5.2; default: 0) +# time, 0 for unlimited. Setting this to 1 also stops +# the server from advertising multiple client support +# (since 5.2; default: 0) # # Since: 4.2 ## @@ -51,7 +53,9 @@ # recreated on the fly while the NBD server is active. # If missing, it will default to denying access (since 4.0). # @max-connections: The maximum number of connections to allow at the same -# time, 0 for unlimited. (since 5.2; default: 0) +# time, 0 for unlimited. Setting this to 1 also stops +# the server from advertising multiple client support +# (since 5.2; default: 0). # # Returns: error if the server is already running. # diff --git a/docs/interop/nbd.txt b/docs/interop/nbd.txt index bdb0f2a41a..f5ca25174a 100644 --- a/docs/interop/nbd.txt +++ b/docs/interop/nbd.txt @@ -68,3 +68,4 @@ NBD_CMD_BLOCK_STATUS for "qemu:dirty-bitmap:", NBD_CMD_CA= CHE * 4.2: NBD_FLAG_CAN_MULTI_CONN for shareable read-only exports, NBD_CMD_FLAG_FAST_ZERO * 5.2: NBD_CMD_BLOCK_STATUS for "qemu:allocation-depth" +* 7.1: NBD_FLAG_CAN_MULTI_CONN for shareable writable exports diff --git a/docs/tools/qemu-nbd.rst b/docs/tools/qemu-nbd.rst index 4c950f6199..8e08a29e89 100644 --- a/docs/tools/qemu-nbd.rst +++ b/docs/tools/qemu-nbd.rst @@ -139,8 +139,7 @@ driver options if :option:`--image-opts` is specified. .. option:: -e, --shared=3DNUM =20 Allow up to *NUM* clients to share the device (default - ``1``), 0 for unlimited. Safe for readers, but for now, - consistency is not guaranteed between multiple writers. + ``1``), 0 for unlimited. =20 .. option:: -t, --persistent =20 diff --git a/include/block/nbd.h b/include/block/nbd.h index c5a29ce1c6..c74b7a9d2e 100644 --- a/include/block/nbd.h +++ b/include/block/nbd.h @@ -1,5 +1,5 @@ /* - * Copyright (C) 2016-2020 Red Hat, Inc. + * Copyright (C) 2016-2022 Red Hat, Inc. * Copyright (C) 2005 Anthony Liguori * * Network Block Device @@ -346,6 +346,7 @@ void nbd_client_put(NBDClient *client); =20 void nbd_server_is_qemu_nbd(int max_connections); bool nbd_server_is_running(void); +int nbd_server_max_connections(void); void nbd_server_start(SocketAddress *addr, const char *tls_creds, const char *tls_authz, uint32_t max_connections, Error **errp); diff --git a/blockdev-nbd.c b/blockdev-nbd.c index 711e0e72bd..012256bb02 100644 --- a/blockdev-nbd.c +++ b/blockdev-nbd.c @@ -44,6 +44,11 @@ bool nbd_server_is_running(void) return nbd_server || qemu_nbd_connections >=3D 0; } =20 +int nbd_server_max_connections(void) +{ + return nbd_server ? nbd_server->max_connections : qemu_nbd_connections; +} + static void nbd_blockdev_client_closed(NBDClient *client, bool ignored) { nbd_client_put(client); diff --git a/nbd/server.c b/nbd/server.c index 4cdbc062c1..213e00e761 100644 --- a/nbd/server.c +++ b/nbd/server.c @@ -1,5 +1,5 @@ /* - * Copyright (C) 2016-2021 Red Hat, Inc. + * Copyright (C) 2016-2022 Red Hat, Inc. * Copyright (C) 2005 Anthony Liguori * * Network Block Device Server Side @@ -1642,7 +1642,6 @@ static int nbd_export_create(BlockExport *blk_exp, Bl= ockExportOptions *exp_args, int64_t size; uint64_t perm, shared_perm; bool readonly =3D !exp_args->writable; - bool shared =3D !exp_args->writable; BlockDirtyBitmapOrStrList *bitmaps; size_t i; int ret; @@ -1693,11 +1692,12 @@ static int nbd_export_create(BlockExport *blk_exp, = BlockExportOptions *exp_args, exp->description =3D g_strdup(arg->description); exp->nbdflags =3D (NBD_FLAG_HAS_FLAGS | NBD_FLAG_SEND_FLUSH | NBD_FLAG_SEND_FUA | NBD_FLAG_SEND_CACHE); + + if (nbd_server_max_connections() !=3D 1) { + exp->nbdflags |=3D NBD_FLAG_CAN_MULTI_CONN; + } if (readonly) { exp->nbdflags |=3D NBD_FLAG_READ_ONLY; - if (shared) { - exp->nbdflags |=3D NBD_FLAG_CAN_MULTI_CONN; - } } else { exp->nbdflags |=3D (NBD_FLAG_SEND_TRIM | NBD_FLAG_SEND_WRITE_ZEROE= S | NBD_FLAG_SEND_FAST_ZERO); diff --git a/MAINTAINERS b/MAINTAINERS index 571556d279..fbc0662627 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -3367,6 +3367,7 @@ F: qemu-nbd.* F: blockdev-nbd.c F: docs/interop/nbd.txt F: docs/tools/qemu-nbd.rst +F: tests/qemu-iotests/tests/*nbd* T: git https://repo.or.cz/qemu/ericb.git nbd T: git https://src.openvz.org/scm/~vsementsov/qemu.git nbd =20 diff --git a/tests/qemu-iotests/tests/nbd-multiconn b/tests/qemu-iotests/te= sts/nbd-multiconn new file mode 100755 index 0000000000..b121f2e363 --- /dev/null +++ b/tests/qemu-iotests/tests/nbd-multiconn @@ -0,0 +1,145 @@ +#!/usr/bin/env python3 +# group: rw auto quick +# +# Test cases for NBD multi-conn advertisement +# +# Copyright (C) 2022 Red Hat, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . + +import os +from contextlib import contextmanager +import iotests +from iotests import qemu_img_create, qemu_io + + +disk =3D os.path.join(iotests.test_dir, 'disk') +size =3D '4M' +nbd_sock =3D os.path.join(iotests.sock_dir, 'nbd_sock') +nbd_uri =3D 'nbd+unix:///{}?socket=3D' + nbd_sock + + +@contextmanager +def open_nbd(export_name): + h =3D nbd.NBD() + try: + h.connect_uri(nbd_uri.format(export_name)) + yield h + finally: + h.shutdown() + +class TestNbdMulticonn(iotests.QMPTestCase): + def setUp(self): + qemu_img_create('-f', iotests.imgfmt, disk, size) + qemu_io('-c', 'w -P 1 0 2M', '-c', 'w -P 2 2M 2M', disk) + + self.vm =3D iotests.VM() + self.vm.launch() + result =3D self.vm.qmp('blockdev-add', { + 'driver': 'qcow2', + 'node-name': 'n', + 'file': {'driver': 'file', 'filename': disk} + }) + self.assert_qmp(result, 'return', {}) + + def tearDown(self): + self.vm.shutdown() + os.remove(disk) + try: + os.remove(nbd_sock) + except OSError: + pass + + @contextmanager + def run_server(self, max_connections=3DNone): + args =3D { + 'addr': { + 'type': 'unix', + 'data': {'path': nbd_sock} + } + } + if max_connections is not None: + args['max-connections'] =3D max_connections + + result =3D self.vm.qmp('nbd-server-start', args) + self.assert_qmp(result, 'return', {}) + yield + + result =3D self.vm.qmp('nbd-server-stop') + self.assert_qmp(result, 'return', {}) + + def add_export(self, name, writable=3DNone): + args =3D { + 'type': 'nbd', + 'id': name, + 'node-name': 'n', + 'name': name, + } + if writable is not None: + args['writable'] =3D writable + + result =3D self.vm.qmp('block-export-add', args) + self.assert_qmp(result, 'return', {}) + + def test_default_settings(self): + with self.run_server(): + self.add_export('r') + self.add_export('w', writable=3DTrue) + with open_nbd('r') as h: + self.assertTrue(h.can_multi_conn()) + with open_nbd('w') as h: + self.assertTrue(h.can_multi_conn()) + + def test_limited_connections(self): + with self.run_server(max_connections=3D1): + self.add_export('r') + self.add_export('w', writable=3DTrue) + with open_nbd('r') as h: + self.assertFalse(h.can_multi_conn()) + with open_nbd('w') as h: + self.assertFalse(h.can_multi_conn()) + + def test_parallel_writes(self): + with self.run_server(): + self.add_export('w', writable=3DTrue) + + clients =3D [nbd.NBD() for _ in range(3)] + for c in clients: + c.connect_uri(nbd_uri.format('w')) + self.assertTrue(c.can_multi_conn()) + + initial_data =3D clients[0].pread(1024 * 1024, 0) + self.assertEqual(initial_data, b'\x01' * 1024 * 1024) + + updated_data =3D b'\x03' * 1024 * 1024 + clients[1].pwrite(updated_data, 0) + clients[2].flush() + current_data =3D clients[0].pread(1024 * 1024, 0) + + self.assertEqual(updated_data, current_data) + + for i in range(3): + clients[i].shutdown() + + +if __name__ =3D=3D '__main__': + try: + # Easier to use libnbd than to try and set up parallel + # 'qemu-nbd --list' or 'qemu-io' processes, but not all systems + # have libnbd installed. + import nbd # type: ignore + + iotests.main(supported_fmts=3D['qcow2']) + except ImportError: + iotests.notrun('libnbd not installed') diff --git a/tests/qemu-iotests/tests/nbd-multiconn.out b/tests/qemu-iotest= s/tests/nbd-multiconn.out new file mode 100644 index 0000000000..8d7e996700 --- /dev/null +++ b/tests/qemu-iotests/tests/nbd-multiconn.out @@ -0,0 +1,5 @@ +... +---------------------------------------------------------------------- +Ran 3 tests + +OK diff --git a/tests/qemu-iotests/tests/nbd-qemu-allocation.out b/tests/qemu-= iotests/tests/nbd-qemu-allocation.out index 0bf1abb063..9d938db24e 100644 --- a/tests/qemu-iotests/tests/nbd-qemu-allocation.out +++ b/tests/qemu-iotests/tests/nbd-qemu-allocation.out @@ -17,7 +17,7 @@ wrote 2097152/2097152 bytes at offset 1048576 exports available: 1 export: '' size: 4194304 - flags: 0x58f ( readonly flush fua df multi cache ) + flags: 0x48f ( readonly flush fua df cache ) min block: 1 opt block: 4096 max block: 33554432 --=20 2.35.3 From nobody Fri May 17 22:04:42 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1652370511; cv=none; d=zohomail.com; s=zohoarc; b=nBLQgbFqnsBnKMmoi+KKvX4mLX4cJX8RYIdinrAVFv0nNiqdYymkXXO5UeuJghCOHQMuaZACxlkoJUVDfmcrYI8vKkYbFPvCMicpPkcdOy7fB9WI4jrSsoalJNuM3ipmC4oFNFlvxZUOB8A024FkMs5Zjga6ARmlQEUoanxzytw= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1652370511; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=Qudi18RG3hZqO+lwyHf43ie+4OpDI9j9yU5BQHMoDvM=; b=mU/iPJz+ZBY138NfP2khD72wlQ8qWzYpZmZsoSYHVM0gENZiaMtWcE9tEGWFhEsTSLJ+QWlRKwl+4jGnpwO6frKqWFGmX6nDk7nCZ2AE6KblTZRC/iAURhk0VYbmHRbwWJAQkd1zQ4sFfqcPSlsAtnWjlJpKHjf7Vw2kYt9/Zqw= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1652370511551885.3328041834878; Thu, 12 May 2022 08:48:31 -0700 (PDT) Received: from localhost ([::1]:50416 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1npB3G-00045Z-1A for importer@patchew.org; Thu, 12 May 2022 11:48:30 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:39094) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1npApe-0006Ca-9b for qemu-devel@nongnu.org; Thu, 12 May 2022 11:34:27 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]:32820) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1npApb-0003S5-RF for qemu-devel@nongnu.org; Thu, 12 May 2022 11:34:25 -0400 Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-297-Z5o6fluWM1qqzhcMrd11GA-1; Thu, 12 May 2022 11:34:22 -0400 Received: from smtp.corp.redhat.com (int-mx10.intmail.prod.int.rdu2.redhat.com [10.11.54.10]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id A94CB8039D7; Thu, 12 May 2022 15:34:21 +0000 (UTC) Received: from merkur.fritz.box (unknown [10.39.194.111]) by smtp.corp.redhat.com (Postfix) with ESMTP id 144CD416156; Thu, 12 May 2022 15:34:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1652369663; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Qudi18RG3hZqO+lwyHf43ie+4OpDI9j9yU5BQHMoDvM=; b=KAarz1yNRDaCfVzkYDb+9ZLgcvm/srhsXtzyG1u+aSuHzilswqe8dnV/VEOcoKEWrG4B+3 HYudQh61+wliLgcwc9usUKKTwKLlrGPDXOat+uBpET95HVfQ7VEQIHJIwo+ZFVjpiPMJw2 +peM/UzAS3vpIo0TOTNE0xeN1fEJ0GQ= X-MC-Unique: Z5o6fluWM1qqzhcMrd11GA-1 From: Kevin Wolf To: qemu-block@nongnu.org Cc: kwolf@redhat.com, qemu-devel@nongnu.org Subject: [PULL 10/10] qemu-iotests: inline common.config into common.rc Date: Thu, 12 May 2022 17:34:01 +0200 Message-Id: <20220512153401.202863-11-kwolf@redhat.com> In-Reply-To: <20220512153401.202863-1-kwolf@redhat.com> References: <20220512153401.202863-1-kwolf@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 2.85 on 10.11.54.10 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.129.124; envelope-from=kwolf@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -28 X-Spam_score: -2.9 X-Spam_bar: -- X-Spam_report: (-2.9 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.082, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1652370512589100001 Content-Type: text/plain; charset="utf-8" From: Paolo Bonzini common.rc has some complicated logic to find the common.config that dates back to xfstests and is completely unnecessary now. Just include the contents of the file. Signed-off-by: Paolo Bonzini Message-Id: <20220505094723.732116-1-pbonzini@redhat.com> Signed-off-by: Kevin Wolf --- tests/qemu-iotests/common.config | 41 -------------------------------- tests/qemu-iotests/common.rc | 31 ++++++++++++++---------- 2 files changed, 19 insertions(+), 53 deletions(-) delete mode 100644 tests/qemu-iotests/common.config diff --git a/tests/qemu-iotests/common.config b/tests/qemu-iotests/common.c= onfig deleted file mode 100644 index 9bd1a5a6fc..0000000000 --- a/tests/qemu-iotests/common.config +++ /dev/null @@ -1,41 +0,0 @@ -#!/usr/bin/env bash -# -# Copyright (C) 2009 Red Hat, Inc. -# Copyright (c) 2000-2003,2006 Silicon Graphics, Inc. All Rights Reserved. -# -# This program is free software; you can redistribute it and/or -# modify it under the terms of the GNU General Public License as -# published by the Free Software Foundation. -# -# This program is distributed in the hope that it would be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with this program. If not, see . -# -# all tests should use a common language setting to prevent golden -# output mismatches. -export LANG=3DC - -PATH=3D".:$PATH" - -HOSTOS=3D$(uname -s) -arch=3D$(uname -m) -[[ "$arch" =3D~ "ppc64" ]] && qemu_arch=3Dppc64 || qemu_arch=3D"$arch" - -# make sure we have a standard umask -umask 022 - -_optstr_add() -{ - if [ -n "$1" ]; then - echo "$1,$2" - else - echo "$2" - fi -} - -# make sure this script returns success -true diff --git a/tests/qemu-iotests/common.rc b/tests/qemu-iotests/common.rc index 227e0a5be9..165b54a61e 100644 --- a/tests/qemu-iotests/common.rc +++ b/tests/qemu-iotests/common.rc @@ -17,6 +17,17 @@ # along with this program. If not, see . # =20 +export LANG=3DC + +PATH=3D".:$PATH" + +HOSTOS=3D$(uname -s) +arch=3D$(uname -m) +[[ "$arch" =3D~ "ppc64" ]] && qemu_arch=3Dppc64 || qemu_arch=3D"$arch" + +# make sure we have a standard umask +umask 022 + # bail out, setting up .notrun file _notrun() { @@ -120,18 +131,14 @@ peek_file_raw() dd if=3D"$1" bs=3D1 skip=3D"$2" count=3D"$3" status=3Dnone } =20 -config=3Dcommon.config -test -f $config || config=3D../common.config -if ! test -f $config -then - echo "$0: failed to find common.config" - exit 1 -fi -if ! . $config - then - echo "$0: failed to source common.config" - exit 1 -fi +_optstr_add() +{ + if [ -n "$1" ]; then + echo "$1,$2" + else + echo "$2" + fi +} =20 # Set the variables to the empty string to turn Valgrind off # for specific processes, e.g. --=20 2.35.3