From nobody Fri Sep 20 22:16:17 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1651837800; cv=none; d=zohomail.com; s=zohoarc; b=SsDbiaNISUshNaxfo7iFbgCK8XHrqccW0yAH1PEOdTRK7HAYAMe5HnCJ5Ql7h5G7We3JmQail5FrDHZAJClZYSKxMxNc+pBwITaRVLWEKwfZwCcd5Tm0ASWA7+TwDH5Ct2uSrJcV02uMDzHvvPCLCurhiBnhs6pHAVUD3+/COoo= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1651837800; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=yilwTR+1kWyyUvLDlj/r4gDsYioy2urpeSLRDjc7FSw=; b=aya2iICxISR6TDHfUcIjnlpjfaE5FeuBYjzDL272yIqmLX91UDewJqWbB5vDfpRhQj5+MhkUztdQe4ScO3mNTzOG5/DKayGXk61+YzY5msnvpAZ49F5FHQ0WwrZUjDU4mfEtz3OrRvpfbZSb6fmCmjF0IkYs/e+L62vC8qjl0qM= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1651837800161210.42918916709664; Fri, 6 May 2022 04:50:00 -0700 (PDT) Received: from localhost ([::1]:48882 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nmwT8-0004Sh-Lb for importer@patchew.org; Fri, 06 May 2022 07:49:58 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:36170) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nmwQd-0002KU-Jh for qemu-devel@nongnu.org; Fri, 06 May 2022 07:47:23 -0400 Received: from us-smtp-delivery-74.mimecast.com ([170.10.129.74]:31816) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nmwQa-0008KR-1T for qemu-devel@nongnu.org; Fri, 06 May 2022 07:47:21 -0400 Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-211-9lA5KmkENNGpGUXa75gclA-1; Fri, 06 May 2022 07:47:15 -0400 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 473DA2999B4A for ; Fri, 6 May 2022 11:47:15 +0000 (UTC) Received: from virtlab701.virt.lab.eng.bos.redhat.com (virtlab701.virt.lab.eng.bos.redhat.com [10.19.152.228]) by smtp.corp.redhat.com (Postfix) with ESMTP id 28A6F9E75; Fri, 6 May 2022 11:47:15 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1651837636; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=yilwTR+1kWyyUvLDlj/r4gDsYioy2urpeSLRDjc7FSw=; b=O9VmxorYJXLF8ipzy7ApkexVuSULOXXjx0OD2Ix9sJ16b8Hm/9a5RjPc2YuNNeo+0VKsp6 d66eBLEIWJBWgLPY18GBCQXs4dWTw0m24xLqthnC/Ew2O0M8aLb4gH6+1k5N/oVKOfETY9 +A4tktOf0/RD6kFH9B6amxeM+sWAGWU= X-MC-Unique: 9lA5KmkENNGpGUXa75gclA-1 From: Paolo Bonzini To: qemu-devel@nongnu.org Cc: stefanha@redhat.com, ldoktor@redhat.com Subject: [PATCH 1/2] thread-pool: optimize scheduling of completion bottom half Date: Fri, 6 May 2022 07:47:10 -0400 Message-Id: <20220506114711.1398662-2-pbonzini@redhat.com> In-Reply-To: <20220506114711.1398662-1-pbonzini@redhat.com> References: <20220506114711.1398662-1-pbonzini@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 2.79 on 10.11.54.5 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.129.74; envelope-from=pbonzini@redhat.com; helo=us-smtp-delivery-74.mimecast.com X-Spam_score_int: -28 X-Spam_score: -2.9 X-Spam_bar: -- X-Spam_report: (-2.9 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.082, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1651837800610100003 Content-Type: text/plain; charset="utf-8" The completion bottom half was scheduled within the pool->lock critical section. That actually results in worse performance, because the worker thread can run its own small critical section and go to sleep before the bottom half starts running. Note that this simple change does not produce an improvement without changing the thread pool QemuSemaphore to a condition variable. Signed-off-by: Paolo Bonzini --- util/thread-pool.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/util/thread-pool.c b/util/thread-pool.c index d763cea505..7e9e2c178b 100644 --- a/util/thread-pool.c +++ b/util/thread-pool.c @@ -108,9 +108,8 @@ static void *worker_thread(void *opaque) smp_wmb(); req->state =3D THREAD_DONE; =20 - qemu_mutex_lock(&pool->lock); - qemu_bh_schedule(pool->completion_bh); + qemu_mutex_lock(&pool->lock); } =20 pool->cur_threads--; --=20 2.31.1 From nobody Fri Sep 20 22:16:17 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1651837800; cv=none; d=zohomail.com; s=zohoarc; b=bygiSmb8ZOzwS5vQUATCAAs3bNb+hQShTTZBjkqkitv6r3etHlQD49AFghPX/vZX1Xa2rlbW/Nj/Y3XXdeBHBEvnBuXprFUfGWM/Nc1/GXcVis5d73DuuwwRqOI85ij9Rv8oKLdH0re71qNQ1CJrw1Thu53vgPRQKGZGOgOkVbQ= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1651837800; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=oYPejR+xm7AfJPjZe039oam3tIUkevBzq5/pi33NtEc=; b=I7uYWT3JVg7iScBqdnlDvdyN0rA6bCqPgfL4q1LjwQuJeVATUGhpxT9o8ZscZr1mG9yB2Uequ3v7NAdtIiTys6AnYmRIZN9DjszRTreSG6pJSiRe2g1gP1bnHTbfBLkhJepiD4plbGHb1jLOYSBlgI/vVMDbgKPAbt9S/HCdEOE= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1651837800105810.1279926456932; Fri, 6 May 2022 04:50:00 -0700 (PDT) Received: from localhost ([::1]:48796 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nmwT8-0004PI-8b for importer@patchew.org; Fri, 06 May 2022 07:49:58 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:36176) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nmwQd-0002KW-Iw for qemu-devel@nongnu.org; Fri, 06 May 2022 07:47:23 -0400 Received: from us-smtp-delivery-74.mimecast.com ([170.10.133.74]:36596) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nmwQZ-0008KV-Pg for qemu-devel@nongnu.org; Fri, 06 May 2022 07:47:21 -0400 Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-642-eG9koSJIMWWzw3v4Vfunhg-1; Fri, 06 May 2022 07:47:15 -0400 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 6FE5318A6582 for ; Fri, 6 May 2022 11:47:15 +0000 (UTC) Received: from virtlab701.virt.lab.eng.bos.redhat.com (virtlab701.virt.lab.eng.bos.redhat.com [10.19.152.228]) by smtp.corp.redhat.com (Postfix) with ESMTP id 4FCB29E75; Fri, 6 May 2022 11:47:15 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1651837637; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=oYPejR+xm7AfJPjZe039oam3tIUkevBzq5/pi33NtEc=; b=DpOLcAas61ha68DYs1NUYTz0PDQuWQCRAlnzfbSzVAi23+9AAWHUi6DfxLgOp8frTWU2YB R402HPGA/YMBSMSKUtqLpimQE54Bcib12RpWwZhVlKak3Kzz3DhCJ1ojHiQPUXnAHHxnPp 9QKWwJOFsPWDCwFe4qZy2rU3XNqB/J8= X-MC-Unique: eG9koSJIMWWzw3v4Vfunhg-1 From: Paolo Bonzini To: qemu-devel@nongnu.org Cc: stefanha@redhat.com, ldoktor@redhat.com Subject: [PATCH 2/2] thread-pool: replace semaphore with condition variable Date: Fri, 6 May 2022 07:47:11 -0400 Message-Id: <20220506114711.1398662-3-pbonzini@redhat.com> In-Reply-To: <20220506114711.1398662-1-pbonzini@redhat.com> References: <20220506114711.1398662-1-pbonzini@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 2.79 on 10.11.54.5 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.133.74; envelope-from=pbonzini@redhat.com; helo=us-smtp-delivery-74.mimecast.com X-Spam_score_int: -28 X-Spam_score: -2.9 X-Spam_bar: -- X-Spam_report: (-2.9 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.082, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1651837800509100001 Since commit f9fc8932b1 ("thread-posix: remove the posix semaphore support", 2022-04-06) QemuSemaphore has its own mutex and condition variable; this adds unnecessary overhead on I/O with small block sizes. Check the QTAILQ directly instead of adding the indirection of a semaphore's count. Using a semaphore has not been necessary since qemu_cond_timedwait was introduced; the new code has to be careful about spurious wakeups but it is simpler, for example thread_pool_cancel does not have to worry about synchronizing the semaphore count with the number of elements of pool->request_list. Note that the return value of qemu_cond_timedwait (0 for timeout, 1 for signal or spurious wakeup) is different from that of qemu_sem_timedwait (-1 for timeout, 0 for success). Reported-by: Luk=C3=A1=C5=A1 Doktor Suggested-by: Stefan Hajnoczi Reviewed-by: Stefan Hajnoczi Signed-off-by: Paolo Bonzini --- util/thread-pool.c | 31 ++++++++++++------------------- 1 file changed, 12 insertions(+), 19 deletions(-) diff --git a/util/thread-pool.c b/util/thread-pool.c index 7e9e2c178b..5511706de8 100644 --- a/util/thread-pool.c +++ b/util/thread-pool.c @@ -57,7 +57,7 @@ struct ThreadPool { QEMUBH *completion_bh; QemuMutex lock; QemuCond worker_stopped; - QemuSemaphore sem; + QemuCond request_cond; int max_threads; QEMUBH *new_thread_bh; =20 @@ -85,15 +85,14 @@ static void *worker_thread(void *opaque) ThreadPoolElement *req; int ret; =20 - do { + if (QTAILQ_EMPTY(&pool->request_list)) { pool->idle_threads++; - qemu_mutex_unlock(&pool->lock); - ret =3D qemu_sem_timedwait(&pool->sem, 10000); - qemu_mutex_lock(&pool->lock); + ret =3D qemu_cond_timedwait(&pool->request_cond, &pool->lock, = 10000); pool->idle_threads--; - } while (ret =3D=3D -1 && !QTAILQ_EMPTY(&pool->request_list)); - if (ret =3D=3D -1 || pool->stopping) { - break; + if (!ret && QTAILQ_EMPTY(&pool->request_list)) { + break; + } + continue; } =20 req =3D QTAILQ_FIRST(&pool->request_list); @@ -210,13 +209,7 @@ static void thread_pool_cancel(BlockAIOCB *acb) trace_thread_pool_cancel(elem, elem->common.opaque); =20 QEMU_LOCK_GUARD(&pool->lock); - if (elem->state =3D=3D THREAD_QUEUED && - /* No thread has yet started working on elem. we can try to "steal" - * the item from the worker if we can get a signal from the - * semaphore. Because this is non-blocking, we can do it with - * the lock taken and ensure that elem will remain THREAD_QUEUED. - */ - qemu_sem_timedwait(&pool->sem, 0) =3D=3D 0) { + if (elem->state =3D=3D THREAD_QUEUED) { QTAILQ_REMOVE(&pool->request_list, elem, reqs); qemu_bh_schedule(pool->completion_bh); =20 @@ -261,7 +254,7 @@ BlockAIOCB *thread_pool_submit_aio(ThreadPool *pool, } QTAILQ_INSERT_TAIL(&pool->request_list, req, reqs); qemu_mutex_unlock(&pool->lock); - qemu_sem_post(&pool->sem); + qemu_cond_signal(&pool->request_cond); return &req->common; } =20 @@ -304,7 +297,7 @@ static void thread_pool_init_one(ThreadPool *pool, AioC= ontext *ctx) pool->completion_bh =3D aio_bh_new(ctx, thread_pool_completion_bh, poo= l); qemu_mutex_init(&pool->lock); qemu_cond_init(&pool->worker_stopped); - qemu_sem_init(&pool->sem, 0); + qemu_cond_init(&pool->request_cond); pool->max_threads =3D 64; pool->new_thread_bh =3D aio_bh_new(ctx, spawn_thread_bh_fn, pool); =20 @@ -336,15 +329,15 @@ void thread_pool_free(ThreadPool *pool) =20 /* Wait for worker threads to terminate */ pool->stopping =3D true; + qemu_cond_broadcast(&pool->request_cond); while (pool->cur_threads > 0) { - qemu_sem_post(&pool->sem); qemu_cond_wait(&pool->worker_stopped, &pool->lock); } =20 qemu_mutex_unlock(&pool->lock); =20 qemu_bh_delete(pool->completion_bh); - qemu_sem_destroy(&pool->sem); + qemu_cond_destroy(&pool->request_cond); qemu_cond_destroy(&pool->worker_stopped); qemu_mutex_destroy(&pool->lock); g_free(pool); --=20 2.31.1