From nobody Wed May 15 15:34:55 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of redhat.com designates 170.10.129.124 as permitted sender) client-ip=170.10.129.124; envelope-from=libvir-list-bounces@redhat.com; helo=us-smtp-delivery-124.mimecast.com; Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=libvir-list-bounces@redhat.com; dmarc=fail(p=none dis=none) header.from=suse.de Return-Path: Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by mx.zohomail.com with SMTPS id 1647103083616180.86033292581124; Sat, 12 Mar 2022 08:38:03 -0800 (PST) Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-651-9cTIjXGQN7mmo0ObEDwf1w-1; Sat, 12 Mar 2022 11:37:10 -0500 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 3C6673C01BAC; Sat, 12 Mar 2022 16:37:08 +0000 (UTC) Received: from mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com [10.30.29.100]) by smtp.corp.redhat.com (Postfix) with ESMTP id BB16A145FBA0; Sat, 12 Mar 2022 16:37:05 +0000 (UTC) Received: from mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (localhost [IPv6:::1]) by mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (Postfix) with ESMTP id 74915194035B; Sat, 12 Mar 2022 16:37:04 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) by mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (Postfix) with ESMTP id 5CDA61940341 for ; Sat, 12 Mar 2022 16:37:03 +0000 (UTC) Received: by smtp.corp.redhat.com (Postfix) id F0A522029F91; Sat, 12 Mar 2022 16:37:02 +0000 (UTC) Received: from mimecast-mx02.redhat.com (mimecast05.extmail.prod.ext.rdu2.redhat.com [10.11.55.21]) by smtp.corp.redhat.com (Postfix) with ESMTPS id EB9962029F90 for ; Sat, 12 Mar 2022 16:36:59 +0000 (UTC) Received: from us-smtp-1.mimecast.com (us-smtp-1.mimecast.com [207.211.31.81]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 3BC64800882 for ; Sat, 12 Mar 2022 16:36:59 +0000 (UTC) Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.220.28]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-599-ZVyv9UN3O3WhyKPSj9ndIg-1; Sat, 12 Mar 2022 11:36:57 -0500 Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id 64B33210F7; Sat, 12 Mar 2022 16:30:10 +0000 (UTC) Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id 2362E13AB4; Sat, 12 Mar 2022 16:30:10 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id xCwrBpLKLGJSJgAAMHmgww (envelope-from ); Sat, 12 Mar 2022 16:30:10 +0000 X-MC-Unique: 9cTIjXGQN7mmo0ObEDwf1w-1 X-Original-To: libvir-list@listman.corp.redhat.com X-MC-Unique: ZVyv9UN3O3WhyKPSj9ndIg-1 From: Claudio Fontana To: =?UTF-8?q?Daniel=20P=20=2E=20Berrang=C3=A9?= Subject: [libvirt RFC] virFile: new VIR_FILE_WRAPPER_BIG_PIPE to improve performance Date: Sat, 12 Mar 2022 17:30:01 +0100 Message-Id: <20220312163001.3811-1-cfontana@suse.de> MIME-Version: 1.0 X-Mimecast-Impersonation-Protect: Policy=CLT - Impersonation Protection Definition; Similar Internal Domain=false; Similar Monitored External Domain=false; Custom External Domain=false; Mimecast External Domain=false; Newly Observed Domain=false; Internal User Name=false; Custom Display Name List=false; Reply-to Address Mismatch=false; Targeted Threat Dictionary=false; Mimecast Threat Dictionary=false; Custom Threat Dictionary=false X-Scanned-By: MIMEDefang 2.78 on 10.11.54.4 X-BeenThere: libvir-list@redhat.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Development discussions about the libvirt library & tools List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: libvir-list@redhat.com, Claudio Fontana Errors-To: libvir-list-bounces@redhat.com Sender: "libvir-list" X-Scanned-By: MIMEDefang 2.85 on 10.11.54.7 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=libvir-list-bounces@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable X-ZM-MESSAGEID: 1647103084471100001 Content-Type: text/plain; charset="utf-8" the first user is the qemu driver, virsh save/resume would slow to a crawl with a default pipe size (64k). This improves the situation by 400%. Going through io_helper still seems to incur in some penalty (~15%-ish) compared with direct qemu migration to a nc socket to a file. Signed-off-by: Claudio Fontana --- src/qemu/qemu_driver.c | 6 +++--- src/qemu/qemu_saveimage.c | 11 ++++++----- src/util/virfile.c | 12 ++++++++++++ src/util/virfile.h | 1 + 4 files changed, 22 insertions(+), 8 deletions(-) Hello, I initially thought this to be a qemu performance issue, so you can find the discussion about this in qemu-devel: "Re: bad virsh save /dev/null performance (600 MiB/s max)" https://lists.gnu.org/archive/html/qemu-devel/2022-03/msg03142.html RFC since need to validate idea, and it is only lightly tested: save - about 400% benefit in throughput, getting around 20 Gbps to /dev= /null, and around 13 Gbps to a ramdisk. By comparison, direct qemu migration to a nc socket is around 24Gbps. restore - not tested, _should_ also benefit in the "bypass_cache" case coredump - not tested, _should_ also benefit like for save Thanks for your comments and review, Claudio diff --git a/src/qemu/qemu_driver.c b/src/qemu/qemu_driver.c index c1b3bd8536..be248c1e92 100644 --- a/src/qemu/qemu_driver.c +++ b/src/qemu/qemu_driver.c @@ -3044,7 +3044,7 @@ doCoreDump(virQEMUDriver *driver, virFileWrapperFd *wrapperFd =3D NULL; int directFlag =3D 0; bool needUnlink =3D false; - unsigned int flags =3D VIR_FILE_WRAPPER_NON_BLOCKING; + unsigned int wrapperFlags =3D VIR_FILE_WRAPPER_NON_BLOCKING | VIR_FILE= _WRAPPER_BIG_PIPE; const char *memory_dump_format =3D NULL; g_autoptr(virQEMUDriverConfig) cfg =3D virQEMUDriverGetConfig(driver); g_autoptr(virCommand) compressor =3D NULL; @@ -3059,7 +3059,7 @@ doCoreDump(virQEMUDriver *driver, =20 /* Create an empty file with appropriate ownership. */ if (dump_flags & VIR_DUMP_BYPASS_CACHE) { - flags |=3D VIR_FILE_WRAPPER_BYPASS_CACHE; + wrapperFlags |=3D VIR_FILE_WRAPPER_BYPASS_CACHE; directFlag =3D virFileDirectFdFlag(); if (directFlag < 0) { virReportError(VIR_ERR_OPERATION_FAILED, "%s", @@ -3072,7 +3072,7 @@ doCoreDump(virQEMUDriver *driver, &needUnlink)) < 0) goto cleanup; =20 - if (!(wrapperFd =3D virFileWrapperFdNew(&fd, path, flags))) + if (!(wrapperFd =3D virFileWrapperFdNew(&fd, path, wrapperFlags))) goto cleanup; =20 if (dump_flags & VIR_DUMP_MEMORY_ONLY) { diff --git a/src/qemu/qemu_saveimage.c b/src/qemu/qemu_saveimage.c index c0139041eb..1b522a1542 100644 --- a/src/qemu/qemu_saveimage.c +++ b/src/qemu/qemu_saveimage.c @@ -267,7 +267,7 @@ qemuSaveImageCreate(virQEMUDriver *driver, int fd =3D -1; int directFlag =3D 0; virFileWrapperFd *wrapperFd =3D NULL; - unsigned int wrapperFlags =3D VIR_FILE_WRAPPER_NON_BLOCKING; + unsigned int wrapperFlags =3D VIR_FILE_WRAPPER_NON_BLOCKING | VIR_FILE= _WRAPPER_BIG_PIPE; =20 /* Obtain the file handle. */ if ((flags & VIR_DOMAIN_SAVE_BYPASS_CACHE)) { @@ -463,10 +463,11 @@ qemuSaveImageOpen(virQEMUDriver *driver, if ((fd =3D qemuDomainOpenFile(cfg, NULL, path, oflags, NULL)) < 0) return -1; =20 - if (bypass_cache && - !(*wrapperFd =3D virFileWrapperFdNew(&fd, path, - VIR_FILE_WRAPPER_BYPASS_CACHE))) - return -1; + if (bypass_cache) { + unsigned int wrapperFlags =3D VIR_FILE_WRAPPER_BYPASS_CACHE | VIR_= FILE_WRAPPER_BIG_PIPE; + if (!(*wrapperFd =3D virFileWrapperFdNew(&fd, path, wrapperFlags))) + return -1; + } =20 data =3D g_new0(virQEMUSaveData, 1); =20 diff --git a/src/util/virfile.c b/src/util/virfile.c index a04f888e06..fdacd17890 100644 --- a/src/util/virfile.c +++ b/src/util/virfile.c @@ -282,6 +282,18 @@ virFileWrapperFdNew(int *fd, const char *name, unsigne= d int flags) =20 ret->cmd =3D virCommandNewArgList(iohelper_path, name, NULL); =20 + if (flags & VIR_FILE_WRAPPER_BIG_PIPE) { + /* + * virsh save/resume would slow to a crawl with a default pipe siz= e (usually 64k). + * This improves the situation by 400%, although going through io_= helper still incurs + * in a performance penalty compared with a direct qemu migration = to a socket. + */ + int pipe_sz, rv =3D virFileReadValueInt(&pipe_sz, "/proc/sys/fs/pi= pe-max-size"); + if (rv !=3D 0) { + pipe_sz =3D 1024 * 1024; /* common default for pipe-max-size */ + } + fcntl(pipefd[output ? 0 : 1], F_SETPIPE_SZ, pipe_sz); + } if (output) { virCommandSetInputFD(ret->cmd, pipefd[0]); virCommandSetOutputFD(ret->cmd, fd); diff --git a/src/util/virfile.h b/src/util/virfile.h index b04386f6e6..8383c4b069 100644 --- a/src/util/virfile.h +++ b/src/util/virfile.h @@ -107,6 +107,7 @@ int virFileDirectFdFlag(void); typedef enum { VIR_FILE_WRAPPER_BYPASS_CACHE =3D (1 << 0), VIR_FILE_WRAPPER_NON_BLOCKING =3D (1 << 1), + VIR_FILE_WRAPPER_BIG_PIPE =3D (1 << 2), } virFileWrapperFdFlags; =20 virFileWrapperFd *virFileWrapperFdNew(int *fd, --=20 2.26.2