From nobody Sun Feb 8 22:05:35 2026 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of redhat.com designates 209.132.183.28 as permitted sender) client-ip=209.132.183.28; envelope-from=libvir-list-bounces@redhat.com; helo=mx1.redhat.com; Authentication-Results: mx.zoho.com; spf=pass (zoho.com: domain of redhat.com designates 209.132.183.28 as permitted sender) smtp.mailfrom=libvir-list-bounces@redhat.com; Return-Path: Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by mx.zohomail.com with SMTPS id 1492090397730413.18417818193666; Thu, 13 Apr 2017 06:33:17 -0700 (PDT) Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 94D5C37B2C7; Thu, 13 Apr 2017 13:33:15 +0000 (UTC) Received: from colo-mx.corp.redhat.com (colo-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.20]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 65BD6C05AE; Thu, 13 Apr 2017 13:33:15 +0000 (UTC) Received: from lists01.pubmisc.prod.ext.phx2.redhat.com (lists01.pubmisc.prod.ext.phx2.redhat.com [10.5.19.33]) by colo-mx.corp.redhat.com (Postfix) with ESMTP id B8AA718523FC; Thu, 13 Apr 2017 13:32:49 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.phx2.redhat.com [10.5.11.15]) by lists01.pubmisc.prod.ext.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id v3DDWfgJ012919 for ; Thu, 13 Apr 2017 09:32:41 -0400 Received: by smtp.corp.redhat.com (Postfix) id 5162B7A42C; Thu, 13 Apr 2017 13:32:41 +0000 (UTC) Received: from moe.brq.redhat.com (dhcp129-131.brq.redhat.com [10.34.129.131]) by smtp.corp.redhat.com (Postfix) with ESMTP id CE0AB7A43F for ; Thu, 13 Apr 2017 13:32:40 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com 94D5C37B2C7 Authentication-Results: ext-mx05.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=redhat.com Authentication-Results: ext-mx05.extmail.prod.ext.phx2.redhat.com; spf=pass smtp.mailfrom=libvir-list-bounces@redhat.com DKIM-Filter: OpenDKIM Filter v2.11.0 mx1.redhat.com 94D5C37B2C7 From: Michal Privoznik To: libvir-list@redhat.com Date: Thu, 13 Apr 2017 15:31:42 +0200 Message-Id: <9e810ccd72c350ecf538ecb0880eda89240c48a2.1492089792.git.mprivozn@redhat.com> In-Reply-To: References: In-Reply-To: References: X-Scanned-By: MIMEDefang 2.79 on 10.5.11.15 X-loop: libvir-list@redhat.com Subject: [libvirt] [PATCH 34/38] fdstream: Implement sparse stream X-BeenThere: libvir-list@redhat.com X-Mailman-Version: 2.1.12 Precedence: junk List-Id: Development discussions about the libvirt library & tools List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Sender: libvir-list-bounces@redhat.com Errors-To: libvir-list-bounces@redhat.com X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.29]); Thu, 13 Apr 2017 13:33:16 +0000 (UTC) X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Type: text/plain; charset="utf-8" Implement virStreamSkip and virStreamInData callbacks. These callbacks do no magic, just skip a hole or detect whether we are in a data section of a file or in a hole and how much bytes can we read until section changes. Signed-off-by: Michal Privoznik --- src/storage/storage_util.c | 4 +- src/util/virfdstream.c | 234 +++++++++++++++++++++++++++++++++++++++++= ---- src/util/virfdstream.h | 1 + 3 files changed, 216 insertions(+), 23 deletions(-) diff --git a/src/storage/storage_util.c b/src/storage/storage_util.c index a2d89af..3576435 100644 --- a/src/storage/storage_util.c +++ b/src/storage/storage_util.c @@ -2427,7 +2427,7 @@ virStorageBackendVolUploadLocal(virConnectPtr conn AT= TRIBUTE_UNUSED, /* Not using O_CREAT because the file is required to already exist at * this point */ ret =3D virFDStreamOpenBlockDevice(stream, target_path, - offset, len, O_WRONLY); + offset, len, false, O_WRONLY); =20 cleanup: VIR_FREE(path); @@ -2465,7 +2465,7 @@ virStorageBackendVolDownloadLocal(virConnectPtr conn = ATTRIBUTE_UNUSED, } =20 ret =3D virFDStreamOpenBlockDevice(stream, target_path, - offset, len, O_RDONLY); + offset, len, false, O_RDONLY); =20 cleanup: VIR_FREE(path); diff --git a/src/util/virfdstream.c b/src/util/virfdstream.c index efd9199..e9b5962 100644 --- a/src/util/virfdstream.c +++ b/src/util/virfdstream.c @@ -51,6 +51,7 @@ VIR_LOG_INIT("fdstream"); =20 typedef enum { VIR_FDSTREAM_MSG_TYPE_DATA, + VIR_FDSTREAM_MSG_TYPE_SKIP, } virFDStreamMsgType; =20 typedef struct _virFDStreamMsg virFDStreamMsg; @@ -66,6 +67,9 @@ struct _virFDStreamMsg { size_t len; size_t offset; } data; + struct { + size_t len; + } skip; } stream; }; =20 @@ -175,6 +179,9 @@ virFDStreamMsgFree(virFDStreamMsgPtr msg) case VIR_FDSTREAM_MSG_TYPE_DATA: VIR_FREE(msg->stream.data.buf); break; + case VIR_FDSTREAM_MSG_TYPE_SKIP: + /* nada */ + break; } =20 VIR_FREE(msg); @@ -361,6 +368,7 @@ typedef virFDStreamThreadData *virFDStreamThreadDataPtr; struct _virFDStreamThreadData { virStreamPtr st; size_t length; + bool sparse; int fdin; char *fdinname; int fdout; @@ -383,32 +391,66 @@ virFDStreamThreadDataFree(virFDStreamThreadDataPtr da= ta) =20 static ssize_t virFDStreamThreadDoRead(virFDStreamDataPtr fdst, + bool sparse, const int fdin, const char *fdinname, + size_t *dataLen, size_t buflen) { virFDStreamMsgPtr msg =3D NULL; + int inData =3D 0; + unsigned long long sectionLen =3D 0; char *buf =3D NULL; ssize_t got; =20 + if (sparse && *dataLen =3D=3D 0) { + if (virFileInData(fdin, &inData, §ionLen) < 0) + goto error; + + if (inData) + *dataLen =3D sectionLen; + } + if (VIR_ALLOC(msg) < 0) goto error; =20 - if (VIR_ALLOC_N(buf, buflen) < 0) - goto error; - - if ((got =3D saferead(fdin, buf, buflen)) < 0) { - virReportSystemError(errno, - _("Unable to read %s"), - fdinname); - goto error; + if (sparse && *dataLen =3D=3D 0) { + msg->type =3D VIR_FDSTREAM_MSG_TYPE_SKIP; + msg->stream.skip.len =3D sectionLen; + got =3D sectionLen; + + /* HACK. The message queue is one directional. So caller + * cannot make us skip the hole. Do that for them instead. */ + if (sectionLen && + lseek(fdin, sectionLen, SEEK_CUR) =3D=3D (off_t) -1) { + virReportSystemError(errno, + _("unable to seek in %s"), + fdinname); + goto error; + } + } else { + if (sparse && + buflen > *dataLen) + buflen =3D *dataLen; + + if (VIR_ALLOC_N(buf, buflen) < 0) + goto error; + + if ((got =3D saferead(fdin, buf, buflen)) < 0) { + virReportSystemError(errno, + _("Unable to read %s"), + fdinname); + goto error; + } + + msg->type =3D VIR_FDSTREAM_MSG_TYPE_DATA; + msg->stream.data.buf =3D buf; + msg->stream.data.len =3D got; + buf =3D NULL; + if (sparse) + *dataLen -=3D got; } =20 - msg->type =3D VIR_FDSTREAM_MSG_TYPE_DATA; - msg->stream.data.buf =3D buf; - msg->stream.data.len =3D got; - buf =3D NULL; - virFDStreamMsgQueuePush(fdst, msg); msg =3D NULL; =20 @@ -423,11 +465,13 @@ virFDStreamThreadDoRead(virFDStreamDataPtr fdst, =20 static ssize_t virFDStreamThreadDoWrite(virFDStreamDataPtr fdst, + bool sparse, const int fdout, const char *fdoutname) { ssize_t got; virFDStreamMsgPtr msg =3D fdst->msg; + off_t off; bool pop =3D false; =20 switch (msg->type) { @@ -446,6 +490,32 @@ virFDStreamThreadDoWrite(virFDStreamDataPtr fdst, =20 pop =3D msg->stream.data.offset =3D=3D msg->stream.data.len; break; + + case VIR_FDSTREAM_MSG_TYPE_SKIP: + if (!sparse) { + virReportError(VIR_ERR_INTERNAL_ERROR, "%s", + _("unexpected stream skip")); + return -1; + } + + got =3D msg->stream.skip.len; + off =3D lseek(fdout, got, SEEK_CUR); + if (off =3D=3D (off_t) -1) { + virReportSystemError(errno, + _("unable to seek in %s"), + fdoutname); + return -1; + } + + if (ftruncate(fdout, off) < 0) { + virReportSystemError(errno, + _("unable to truncate %s"), + fdoutname); + return -1; + } + + pop =3D true; + break; } =20 if (pop) { @@ -463,6 +533,7 @@ virFDStreamThread(void *opaque) virFDStreamThreadDataPtr data =3D opaque; virStreamPtr st =3D data->st; size_t length =3D data->length; + bool sparse =3D data->sparse; int fdin =3D data->fdin; char *fdinname =3D data->fdinname; int fdout =3D data->fdout; @@ -471,6 +542,7 @@ virFDStreamThread(void *opaque) bool doRead =3D fdst->threadDoRead; size_t buflen =3D 256 * 1024; size_t total =3D 0; + size_t dataLen =3D 0; =20 virObjectRef(fdst); virObjectLock(fdst); @@ -505,9 +577,9 @@ virFDStreamThread(void *opaque) } =20 if (doRead) - got =3D virFDStreamThreadDoRead(fdst, fdin, fdinname, buflen); + got =3D virFDStreamThreadDoRead(fdst, sparse, fdin, fdinname, = &dataLen, buflen); else - got =3D virFDStreamThreadDoWrite(fdst, fdout, fdoutname); + got =3D virFDStreamThreadDoWrite(fdst, sparse, fdout, fdoutnam= e); =20 if (got < 0) goto error; @@ -773,6 +845,14 @@ static int virFDStreamRead(virStreamPtr st, char *byte= s, size_t nbytes) } } =20 + /* Shortcut, if the stream is in the trailing hole, + * return 0 immediately. */ + if (msg->type =3D=3D VIR_FDSTREAM_MSG_TYPE_SKIP && + msg->stream.skip.len =3D=3D 0) { + ret =3D 0; + goto cleanup; + } + if (msg->type !=3D VIR_FDSTREAM_MSG_TYPE_DATA) { /* Nope, nope, I'm outta here */ virReportError(VIR_ERR_INTERNAL_ERROR, "%s", @@ -823,11 +903,120 @@ static int virFDStreamRead(virStreamPtr st, char *by= tes, size_t nbytes) } =20 =20 +static int +virFDStreamSkip(virStreamPtr st, + unsigned long long length) +{ + virFDStreamDataPtr fdst =3D st->privateData; + virFDStreamMsgPtr msg =3D NULL; + off_t off; + int ret =3D -1; + + virObjectLock(fdst); + if (fdst->length) { + if (length > fdst->length - fdst->offset) + length =3D fdst->length - fdst->offset; + fdst->offset +=3D length; + } + + if (fdst->thread) { + /* Things are a bit complicated here. But bear with me. If FDStrea= m is + * in a read mode, then if the message at the queue head is SKIP, = just + * pop it. The thread has lseek()-ed anyway. If however, the FDStr= eam + * is in write mode, then tell the thread to do the lseek() for us. + * Under no circumstances we can do the lseek() ourselves here. We + * might mess up file position for the thread. */ + if (fdst->threadDoRead) { + msg =3D fdst->msg; + if (msg->type !=3D VIR_FDSTREAM_MSG_TYPE_SKIP) { + virReportError(VIR_ERR_INTERNAL_ERROR, "%s", + _("Invalid stream skip")); + goto cleanup; + } + + virFDStreamMsgQueuePop(fdst); + } else { + if (VIR_ALLOC(msg) < 0) + goto cleanup; + + msg->type =3D VIR_FDSTREAM_MSG_TYPE_SKIP; + msg->stream.skip.len =3D length; + virFDStreamMsgQueuePush(fdst, msg); + msg =3D NULL; + } + } else { + off =3D lseek(fdst->fd, length, SEEK_CUR); + if (off =3D=3D (off_t) -1) { + virReportSystemError(errno, "%s", + _("unable to seek")); + goto cleanup; + } + + if (ftruncate(fdst->fd, off) < 0) { + virReportSystemError(errno, "%s", + _("unable to truncate")); + goto cleanup; + } + } + + ret =3D 0; + cleanup: + virObjectUnlock(fdst); + virFDStreamMsgFree(msg); + return ret; +} + + +static int +virFDStreamInData(virStreamPtr st, + int *inData, + unsigned long long *length) +{ + virFDStreamDataPtr fdst =3D st->privateData; + int ret =3D -1; + + virObjectLock(fdst); + + if (fdst->thread) { + virFDStreamMsgPtr msg; + + while (!(msg =3D fdst->msg)) { + if (fdst->threadQuit) { + *inData =3D *length =3D 0; + ret =3D 0; + goto cleanup; + } else { + virObjectUnlock(fdst); + virCondSignal(&fdst->threadCond); + virObjectLock(fdst); + } + } + + if (msg->type =3D=3D VIR_FDSTREAM_MSG_TYPE_DATA) { + *inData =3D 1; + *length =3D msg->stream.data.len - msg->stream.data.offset; + } else { + *inData =3D 0; + *length =3D msg->stream.skip.len; + } + ret =3D 0; + } else { + ret =3D virFileInData(fdst->fd, inData, length); + } + + cleanup: + virObjectUnlock(fdst); + return ret; +} + + static virStreamDriver virFDStreamDrv =3D { .streamSend =3D virFDStreamWrite, .streamRecv =3D virFDStreamRead, .streamFinish =3D virFDStreamClose, .streamAbort =3D virFDStreamAbort, + .streamSkip =3D virFDStreamSkip, + .streamInData =3D virFDStreamInData, .streamEventAddCallback =3D virFDStreamAddCallback, .streamEventUpdateCallback =3D virFDStreamUpdateCallback, .streamEventRemoveCallback =3D virFDStreamRemoveCallback @@ -969,7 +1158,8 @@ virFDStreamOpenFileInternal(virStreamPtr st, unsigned long long length, int oflags, int mode, - bool forceIOHelper) + bool forceIOHelper, + bool sparse) { int fd =3D -1; struct stat sb; @@ -1026,6 +1216,7 @@ virFDStreamOpenFileInternal(virStreamPtr st, =20 threadData->st =3D virObjectRef(st); threadData->length =3D length; + threadData->sparse =3D sparse; =20 if ((oflags & O_ACCMODE) =3D=3D O_RDONLY) { threadData->fdin =3D fd; @@ -1067,7 +1258,7 @@ int virFDStreamOpenFile(virStreamPtr st, } return virFDStreamOpenFileInternal(st, path, offset, length, - oflags, 0, false); + oflags, 0, false, false); } =20 int virFDStreamCreateFile(virStreamPtr st, @@ -1080,7 +1271,7 @@ int virFDStreamCreateFile(virStreamPtr st, return virFDStreamOpenFileInternal(st, path, offset, length, oflags | O_CREAT, mode, - false); + false, false); } =20 #ifdef HAVE_CFMAKERAW @@ -1096,7 +1287,7 @@ int virFDStreamOpenPTY(virStreamPtr st, if (virFDStreamOpenFileInternal(st, path, offset, length, oflags | O_CREAT, 0, - false) < 0) + false, false) < 0) return -1; =20 fdst =3D st->privateData; @@ -1133,7 +1324,7 @@ int virFDStreamOpenPTY(virStreamPtr st, return virFDStreamOpenFileInternal(st, path, offset, length, oflags | O_CREAT, 0, - false); + false, false); } #endif /* !HAVE_CFMAKERAW */ =20 @@ -1141,11 +1332,12 @@ int virFDStreamOpenBlockDevice(virStreamPtr st, const char *path, unsigned long long offset, unsigned long long length, + bool sparse, int oflags) { return virFDStreamOpenFileInternal(st, path, offset, length, - oflags, 0, true); + oflags, 0, true, sparse); } =20 int virFDStreamSetInternalCloseCb(virStreamPtr st, diff --git a/src/util/virfdstream.h b/src/util/virfdstream.h index 34c4c3f..887c991 100644 --- a/src/util/virfdstream.h +++ b/src/util/virfdstream.h @@ -59,6 +59,7 @@ int virFDStreamOpenBlockDevice(virStreamPtr st, const char *path, unsigned long long offset, unsigned long long length, + bool sparse, int oflags); =20 int virFDStreamSetInternalCloseCb(virStreamPtr st, --=20 2.10.2 -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list