From nobody Fri Mar 29 12:35:46 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of redhat.com designates 205.139.110.61 as permitted sender) client-ip=205.139.110.61; envelope-from=libvir-list-bounces@redhat.com; helo=us-smtp-delivery-1.mimecast.com; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of redhat.com designates 205.139.110.61 as permitted sender) smtp.mailfrom=libvir-list-bounces@redhat.com; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1574944175; cv=none; d=zohomail.com; s=zohoarc; b=NUT4Y3L835S06ZNDPliEP5t8HbVvwFGtw7vu4lR/3mDRa9qpc6Xx6/cNxPxETJ7Pvpnsi8Stwj/RuZXdyMHNL6Tl7HRUQezSVHaDhRurI8lf9lNsHJEE4c5BUDKeEKF2L+1/6hyDrmoyDh4fQq8ydxsfG2CrtLD9dqNf0/Xc3zI= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1574944175; h=Content-Type:Content-Transfer-Encoding:Date:From:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:Sender:Subject:To; bh=m99b42TnnGnm0gBmk6mtDSmFjRhzowbi40c45Hirk4o=; b=hcQ6PTxVKZZ8H7olZQZRaJHDAbENFigxeXPDNwPh+zEv9fPDzMfjT7335bJuPhF8f7609Il6Fm5NxRP3a+dTQryukVHhvXzIYx4Tz4j+CyTft432fzIag1ZBnmPrqLV8m+LgB4I5rtT9wI/uO3mytQrhnaoAqivB3w1nzAj4JPM= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of redhat.com designates 205.139.110.61 as permitted sender) smtp.mailfrom=libvir-list-bounces@redhat.com; dmarc=pass header.from= (p=none dis=none) header.from= Return-Path: Received: from us-smtp-delivery-1.mimecast.com (us-smtp-1.mimecast.com [205.139.110.61]) by mx.zohomail.com with SMTPS id 1574944175578432.0379197108889; Thu, 28 Nov 2019 04:29:35 -0800 (PST) Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-330-DHbsrh3WNFSK8A1gH09nHw-1; Thu, 28 Nov 2019 07:29:32 -0500 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id D0959800EBA; Thu, 28 Nov 2019 12:29:25 +0000 (UTC) Received: from colo-mx.corp.redhat.com (colo-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.21]) by smtp.corp.redhat.com (Postfix) with ESMTPS id B0EAF5D9E1; Thu, 28 Nov 2019 12:29:23 +0000 (UTC) Received: from lists01.pubmisc.prod.ext.phx2.redhat.com (lists01.pubmisc.prod.ext.phx2.redhat.com [10.5.19.33]) by colo-mx.corp.redhat.com (Postfix) with ESMTP id 821DC4BB65; Thu, 28 Nov 2019 12:29:17 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) by lists01.pubmisc.prod.ext.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id xASCRUOH004723 for ; Thu, 28 Nov 2019 07:27:30 -0500 Received: by smtp.corp.redhat.com (Postfix) id 910AD5DA2C; Thu, 28 Nov 2019 12:27:30 +0000 (UTC) Received: from domokun.gsslab.fab.redhat.com (dhcp-94.gsslab.fab.redhat.com [10.33.9.94]) by smtp.corp.redhat.com (Postfix) with ESMTP id CE7A05D9E2; Thu, 28 Nov 2019 12:27:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1574944174; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:list-id:list-help: list-unsubscribe:list-subscribe:list-post; bh=m99b42TnnGnm0gBmk6mtDSmFjRhzowbi40c45Hirk4o=; b=U5KhdbrkIJ6vdZ2kTrUqrf1FmuZs5ib+2D3+c8kolQYdOM5CQ/qXfd3HPuWJWxAcowdWCS GNfdIigBC9jcT2E8Cc5k3ggMPIbuTTGeXWqgz/xsbgB9JBH0Jt1IE2thWZGZjSxg+RUlsm Dig6VnYkwjJlH7sLdkC3tvNztL2zOHI= From: =?UTF-8?q?Daniel=20P=2E=20Berrang=C3=A9?= To: libvir-list@redhat.com Date: Thu, 28 Nov 2019 12:27:24 +0000 Message-Id: <20191128122724.10576-1-berrange@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 X-loop: libvir-list@redhat.com Subject: [libvirt] [PATCH] qemu: make 'xz' image compression viable by using -3 X-BeenThere: libvir-list@redhat.com X-Mailman-Version: 2.1.12 Precedence: junk List-Id: Development discussions about the libvirt library & tools List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: libvir-list-bounces@redhat.com Errors-To: libvir-list-bounces@redhat.com X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 X-MC-Unique: DHbsrh3WNFSK8A1gH09nHw-1 X-Mimecast-Spam-Score: 0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-ZohoMail-DKIM: pass (identity @redhat.com) For managed save we can choose between various compression methods. I randomly tested the 'xz' program on a 8 GB guest and was surprised to have to wait > 50 minutes for it to finish compressing, with 'xz' burning 100% cpu for the entire time. Despite the impressive compression, this is completely useless in the real world as it is far too long to wait to save the VM. The 'xz' binary defaults to '-6' optimization level which aims for high compression, with moderate memory usage, at the expense of speed. This change switches it to use the '-3' optimization level which is documented as being the one that optimizes speed at expense of compression. Even with this, it will still outperform all the other options in terms of compression level. It is a little less than x4 faster than '-6' which means it starts to be a viable choice to use 'xz' for people who really want best compression. The test results on a 1 GB, fairly freshly booted VM are as follows format | save | restore size =3D=3D=3D=3D=3D=3D=3D+=3D=3D=3D=3D=3D=3D=3D+=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D raw | 05s | 1s | 428 MB lzop | 05s | 3s | 160 MB gzip | 29s | 5s | 118 MB bz2 | 54s | 22s | 114 MB xz | 4m37s | 13s | 86 MB xz -3 | 1m20s | 12s | 95 MB Based on this we can say * For moderate compression with no noticable loss in speed =3D> use lzop * For high compression with moderate loss in speed =3D> use gzip * For best compression with significant loss in speed =3D> use xz Signed-off-by: Daniel P. Berrang=C3=A9 Reviewed-by: Michal Privoznik --- src/qemu/qemu_driver.c | 46 ++++++++++++++++++++++----------------- src/qemu/qemu_migration.c | 28 ++++++++---------------- src/qemu/qemu_migration.h | 2 +- 3 files changed, 36 insertions(+), 40 deletions(-) diff --git a/src/qemu/qemu_driver.c b/src/qemu/qemu_driver.c index 8c2670d377..ba50701d3a 100644 --- a/src/qemu/qemu_driver.c +++ b/src/qemu/qemu_driver.c @@ -3177,7 +3177,7 @@ qemuDomainSaveMemory(virQEMUDriverPtr driver, virDomainObjPtr vm, const char *path, virQEMUSaveDataPtr data, - const char *compressedpath, + virCommandPtr compressor, unsigned int flags, qemuDomainAsyncJob asyncJob) { @@ -3216,7 +3216,7 @@ qemuDomainSaveMemory(virQEMUDriverPtr driver, goto cleanup; =20 /* Perform the migration */ - if (qemuMigrationSrcToFile(driver, vm, fd, compressedpath, asyncJob) <= 0) + if (qemuMigrationSrcToFile(driver, vm, fd, compressor, asyncJob) < 0) goto cleanup; =20 /* Touch up file header to mark image complete. */ @@ -3259,7 +3259,7 @@ qemuDomainSaveMemory(virQEMUDriverPtr driver, static int qemuDomainSaveInternal(virQEMUDriverPtr driver, virDomainObjPtr vm, const char *path, - int compressed, const char *compressedpath, + int compressed, virCommandPtr compressor, const char *xmlin, unsigned int flags) { g_autofree char *xml =3D NULL; @@ -3345,7 +3345,7 @@ qemuDomainSaveInternal(virQEMUDriverPtr driver, goto endjob; xml =3D NULL; =20 - ret =3D qemuDomainSaveMemory(driver, vm, path, data, compressedpath, + ret =3D qemuDomainSaveMemory(driver, vm, path, data, compressor, flags, QEMU_ASYNC_JOB_SAVE); if (ret < 0) goto endjob; @@ -3406,13 +3406,14 @@ qemuDomainSaveInternal(virQEMUDriverPtr driver, */ static int ATTRIBUTE_NONNULL(2) qemuGetCompressionProgram(const char *imageFormat, - char **compresspath, + virCommandPtr *compressor, const char *styleFormat, bool use_raw_on_fail) { int ret; + const char *prog; =20 - *compresspath =3D NULL; + *compressor =3D NULL; =20 if (!imageFormat) return QEMU_SAVE_FORMAT_RAW; @@ -3423,9 +3424,14 @@ qemuGetCompressionProgram(const char *imageFormat, if (ret =3D=3D QEMU_SAVE_FORMAT_RAW) return QEMU_SAVE_FORMAT_RAW; =20 - if (!(*compresspath =3D virFindFileInPath(imageFormat))) + if (!(prog =3D virFindFileInPath(imageFormat))) goto error; =20 + *compressor =3D virCommandNew(prog); + virCommandAddArg(*compressor, "-c"); + if (ret =3D=3D QEMU_SAVE_FORMAT_XZ) + virCommandAddArg(*compressor, "-3"); + return ret; =20 error: @@ -3466,7 +3472,7 @@ qemuDomainSaveFlags(virDomainPtr dom, const char *pat= h, const char *dxml, { virQEMUDriverPtr driver =3D dom->conn->privateData; int compressed; - g_autofree char *compressedpath =3D NULL; + g_autoptr(virCommand) compressor =3D NULL; int ret =3D -1; virDomainObjPtr vm =3D NULL; g_autoptr(virQEMUDriverConfig) cfg =3D NULL; @@ -3477,7 +3483,7 @@ qemuDomainSaveFlags(virDomainPtr dom, const char *pat= h, const char *dxml, =20 cfg =3D virQEMUDriverGetConfig(driver); if ((compressed =3D qemuGetCompressionProgram(cfg->saveImageFormat, - &compressedpath, + &compressor, "save", false)) < 0) goto cleanup; =20 @@ -3491,7 +3497,7 @@ qemuDomainSaveFlags(virDomainPtr dom, const char *pat= h, const char *dxml, goto cleanup; =20 ret =3D qemuDomainSaveInternal(driver, vm, path, compressed, - compressedpath, dxml, flags); + compressor, dxml, flags); =20 cleanup: virDomainObjEndAPI(&vm); @@ -3522,7 +3528,7 @@ qemuDomainManagedSave(virDomainPtr dom, unsigned int = flags) virQEMUDriverPtr driver =3D dom->conn->privateData; g_autoptr(virQEMUDriverConfig) cfg =3D NULL; int compressed; - g_autofree char *compressedpath =3D NULL; + g_autoptr(virCommand) compressor =3D NULL; virDomainObjPtr vm; g_autofree char *name =3D NULL; int ret =3D -1; @@ -3548,7 +3554,7 @@ qemuDomainManagedSave(virDomainPtr dom, unsigned int = flags) =20 cfg =3D virQEMUDriverGetConfig(driver); if ((compressed =3D qemuGetCompressionProgram(cfg->saveImageFormat, - &compressedpath, + &compressor, "save", false)) < 0) goto cleanup; =20 @@ -3558,7 +3564,7 @@ qemuDomainManagedSave(virDomainPtr dom, unsigned int = flags) VIR_INFO("Saving state of domain '%s' to '%s'", vm->def->name, name); =20 ret =3D qemuDomainSaveInternal(driver, vm, name, compressed, - compressedpath, NULL, flags); + compressor, NULL, flags); if (ret =3D=3D 0) vm->hasManagedSave =3D true; =20 @@ -3754,14 +3760,14 @@ doCoreDump(virQEMUDriverPtr driver, unsigned int flags =3D VIR_FILE_WRAPPER_NON_BLOCKING; const char *memory_dump_format =3D NULL; g_autoptr(virQEMUDriverConfig) cfg =3D virQEMUDriverGetConfig(driver); - g_autofree char *compressedpath =3D NULL; + g_autoptr(virCommand) compressor =3D NULL; =20 /* We reuse "save" flag for "dump" here. Then, we can support the same * format in "save" and "dump". This path doesn't need the compression * program to exist and can ignore the return value - it only cares to - * get the compressedpath */ + * get the compressor */ ignore_value(qemuGetCompressionProgram(cfg->dumpImageFormat, - &compressedpath, + &compressor, "dump", true)); =20 /* Create an empty file with appropriate ownership. */ @@ -3809,7 +3815,7 @@ doCoreDump(virQEMUDriverPtr driver, if (!qemuMigrationSrcIsAllowed(driver, vm, false, 0)) goto cleanup; =20 - ret =3D qemuMigrationSrcToFile(driver, vm, fd, compressedpath, + ret =3D qemuMigrationSrcToFile(driver, vm, fd, compressor, QEMU_ASYNC_JOB_DUMP); } =20 @@ -15586,7 +15592,7 @@ qemuDomainSnapshotCreateActiveExternal(virQEMUDrive= rPtr driver, int thaw =3D 0; /* 1 if freeze succeeded, -1 if freeze failed */ bool pmsuspended =3D false; int compressed; - g_autofree char *compressedpath =3D NULL; + g_autoptr(virCommand) compressor =3D NULL; virQEMUSaveDataPtr data =3D NULL; =20 /* If quiesce was requested, then issue a freeze command, and a @@ -15656,7 +15662,7 @@ qemuDomainSnapshotCreateActiveExternal(virQEMUDrive= rPtr driver, JOB_MASK(QEMU_JOB_MIGRATION_OP))= ); =20 if ((compressed =3D qemuGetCompressionProgram(cfg->snapshotImageFo= rmat, - &compressedpath, + &compressor, "snapshot", false)) < = 0) goto cleanup; =20 @@ -15673,7 +15679,7 @@ qemuDomainSnapshotCreateActiveExternal(virQEMUDrive= rPtr driver, xml =3D NULL; =20 if ((ret =3D qemuDomainSaveMemory(driver, vm, snapdef->file, data, - compressedpath, 0, + compressor, 0, QEMU_ASYNC_JOB_SNAPSHOT)) < 0) goto cleanup; =20 diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c index c38bf342d7..dabdda2715 100644 --- a/src/qemu/qemu_migration.c +++ b/src/qemu/qemu_migration.c @@ -5170,13 +5170,12 @@ qemuMigrationDstFinish(virQEMUDriverPtr driver, int qemuMigrationSrcToFile(virQEMUDriverPtr driver, virDomainObjPtr vm, int fd, - const char *compressor, + virCommandPtr compressor, qemuDomainAsyncJob asyncJob) { qemuDomainObjPrivatePtr priv =3D vm->privateData; int rc; int ret =3D -1; - virCommandPtr cmd =3D NULL; int pipeFD[2] =3D { -1, -1 }; unsigned long saveMigBandwidth =3D priv->migMaxBandwidth; char *errbuf =3D NULL; @@ -5221,25 +5220,17 @@ qemuMigrationSrcToFile(virQEMUDriverPtr driver, vir= DomainObjPtr vm, QEMU_MONITOR_MIGRATE_BACKGROUND, fd); } else { - const char *prog =3D compressor; - const char *args[] =3D { - prog, - "-c", - NULL - }; - - cmd =3D virCommandNewArgs(args); - virCommandSetInputFD(cmd, pipeFD[0]); - virCommandSetOutputFD(cmd, &fd); - virCommandSetErrorBuffer(cmd, &errbuf); - virCommandDoAsyncIO(cmd); + virCommandSetInputFD(compressor, pipeFD[0]); + virCommandSetOutputFD(compressor, &fd); + virCommandSetErrorBuffer(compressor, &errbuf); + virCommandDoAsyncIO(compressor); if (virSetCloseExec(pipeFD[1]) < 0) { virReportSystemError(errno, "%s", _("Unable to set cloexec flag")); ignore_value(qemuDomainObjExitMonitor(driver, vm)); goto cleanup; } - if (virCommandRunAsync(cmd, NULL) < 0) { + if (virCommandRunAsync(compressor, NULL) < 0) { ignore_value(qemuDomainObjExitMonitor(driver, vm)); goto cleanup; } @@ -5260,7 +5251,7 @@ qemuMigrationSrcToFile(virQEMUDriverPtr driver, virDo= mainObjPtr vm, if (rc < 0) { if (rc =3D=3D -2) { virErrorPreserveLast(&orig_err); - virCommandAbort(cmd); + virCommandAbort(compressor); if (virDomainObjIsActive(vm) && qemuDomainObjEnterMonitorAsync(driver, vm, asyncJob) =3D= =3D 0) { qemuMonitorMigrateCancel(priv->mon); @@ -5270,7 +5261,7 @@ qemuMigrationSrcToFile(virQEMUDriverPtr driver, virDo= mainObjPtr vm, goto cleanup; } =20 - if (cmd && virCommandWait(cmd, NULL) < 0) + if (compressor && virCommandWait(compressor, NULL) < 0) goto cleanup; =20 qemuDomainEventEmitJobCompleted(driver, vm); @@ -5290,10 +5281,9 @@ qemuMigrationSrcToFile(virQEMUDriverPtr driver, virD= omainObjPtr vm, =20 VIR_FORCE_CLOSE(pipeFD[0]); VIR_FORCE_CLOSE(pipeFD[1]); - if (cmd) { + if (errbuf) { VIR_DEBUG("Compression binary stderr: %s", NULLSTR(errbuf)); VIR_FREE(errbuf); - virCommandFree(cmd); } =20 virErrorRestore(&orig_err); diff --git a/src/qemu/qemu_migration.h b/src/qemu/qemu_migration.h index 2daae65191..d98fe9f80a 100644 --- a/src/qemu/qemu_migration.h +++ b/src/qemu/qemu_migration.h @@ -204,7 +204,7 @@ int qemuMigrationSrcToFile(virQEMUDriverPtr driver, virDomainObjPtr vm, int fd, - const char *compressor, + virCommandPtr compressor, qemuDomainAsyncJob asyncJob) ATTRIBUTE_NONNULL(1) ATTRIBUTE_NONNULL(2) G_GNUC_WARN_UNUSED_RESULT; =20 --=20 2.21.0 -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list