From nobody Fri May 3 18:25:12 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zoho.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1494950207981820.1125220452442; Tue, 16 May 2017 08:56:47 -0700 (PDT) Received: from localhost ([::1]:42891 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dAeq6-0004qu-H0 for importer@patchew.org; Tue, 16 May 2017 11:56:46 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:46644) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dAeo0-0003JR-Hb for qemu-devel@nongnu.org; Tue, 16 May 2017 11:54:37 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dAenx-0001XG-EY for qemu-devel@nongnu.org; Tue, 16 May 2017 11:54:36 -0400 Received: from mx1.redhat.com ([209.132.183.28]:52628) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1dAenu-0001W7-E5; Tue, 16 May 2017 11:54:30 -0400 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 3A7C84621C; Tue, 16 May 2017 15:54:24 +0000 (UTC) Received: from localhost (ovpn-116-10.phx2.redhat.com [10.3.116.10]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 8D98958823; Tue, 16 May 2017 15:54:23 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com 3A7C84621C Authentication-Results: ext-mx05.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=redhat.com Authentication-Results: ext-mx05.extmail.prod.ext.phx2.redhat.com; spf=pass smtp.mailfrom=jcody@redhat.com DKIM-Filter: OpenDKIM Filter v2.11.0 mx1.redhat.com 3A7C84621C From: Jeff Cody To: qemu-block@nongnu.org Date: Tue, 16 May 2017 11:54:13 -0400 Message-Id: <20170516155420.10106-2-jcody@redhat.com> In-Reply-To: <20170516155420.10106-1-jcody@redhat.com> References: <20170516155420.10106-1-jcody@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.29]); Tue, 16 May 2017 15:54:29 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PULL 1/8] block: curl: Allow passing cookies via QCryptoSecret X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org, jcody@redhat.com, Peter Krempa , qemu-devel@nongnu.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: Peter Krempa Since cookies can contain sensitive data (session ID, etc ...) it is desired to hide them from the prying eyes of users. Add a possibility to pass them via the secret infrastructure. Resolves: https://bugzilla.redhat.com/show_bug.cgi?id=3D1447413 Signed-off-by: Peter Krempa Reviewed-by: Eric Blake Reviewed-by: Jeff Cody Message-id: f4a22cdebdd0bca6a13a43a2a6deead7f2ec4bb3.1493906281.git.pkrempa= @redhat.com Signed-off-by: Jeff Cody --- block/curl.c | 24 +++++++++++++++++++++++- qapi/block-core.json | 12 ++++++++++-- 2 files changed, 33 insertions(+), 3 deletions(-) diff --git a/block/curl.c b/block/curl.c index aa6e8cc..4382234 100644 --- a/block/curl.c +++ b/block/curl.c @@ -85,6 +85,7 @@ static CURLMcode __curl_multi_socket_action(CURLM *multi_= handle, #define CURL_BLOCK_OPT_SSLVERIFY "sslverify" #define CURL_BLOCK_OPT_TIMEOUT "timeout" #define CURL_BLOCK_OPT_COOKIE "cookie" +#define CURL_BLOCK_OPT_COOKIE_SECRET "cookie-secret" #define CURL_BLOCK_OPT_USERNAME "username" #define CURL_BLOCK_OPT_PASSWORD_SECRET "password-secret" #define CURL_BLOCK_OPT_PROXY_USERNAME "proxy-username" @@ -624,6 +625,11 @@ static QemuOptsList runtime_opts =3D { .help =3D "Pass the cookie or list of cookies with each reques= t" }, { + .name =3D CURL_BLOCK_OPT_COOKIE_SECRET, + .type =3D QEMU_OPT_STRING, + .help =3D "ID of secret used as cookie passed with each reques= t" + }, + { .name =3D CURL_BLOCK_OPT_USERNAME, .type =3D QEMU_OPT_STRING, .help =3D "Username for HTTP auth" @@ -657,6 +663,7 @@ static int curl_open(BlockDriverState *bs, QDict *optio= ns, int flags, Error *local_err =3D NULL; const char *file; const char *cookie; + const char *cookie_secret; double d; const char *secretid; const char *protocol_delimiter; @@ -693,7 +700,22 @@ static int curl_open(BlockDriverState *bs, QDict *opti= ons, int flags, s->sslverify =3D qemu_opt_get_bool(opts, CURL_BLOCK_OPT_SSLVERIFY, tru= e); =20 cookie =3D qemu_opt_get(opts, CURL_BLOCK_OPT_COOKIE); - s->cookie =3D g_strdup(cookie); + cookie_secret =3D qemu_opt_get(opts, CURL_BLOCK_OPT_COOKIE_SECRET); + + if (cookie && cookie_secret) { + error_setg(errp, + "curl driver cannot handle both cookie and cookie secre= t"); + goto out_noclean; + } + + if (cookie_secret) { + s->cookie =3D qcrypto_secret_lookup_as_utf8(cookie_secret, errp); + if (!s->cookie) { + goto out_noclean; + } + } else { + s->cookie =3D g_strdup(cookie); + } =20 file =3D qemu_opt_get(opts, CURL_BLOCK_OPT_URL); if (file =3D=3D NULL) { diff --git a/qapi/block-core.json b/qapi/block-core.json index 6b974b9..ea0b3e8 100644 --- a/qapi/block-core.json +++ b/qapi/block-core.json @@ -2813,11 +2813,15 @@ # "name1=3Dcontent1; name2=3Dcontent2;" as explained by # CURLOPT_COOKIE(3). Defaults to no cookies. # +# @cookie-secret: ID of a QCryptoSecret object providing the cookie data i= n a +# secure way. See @cookie for the format. (since 2.10) +# # Since: 2.9 ## { 'struct': 'BlockdevOptionsCurlHttp', 'base': 'BlockdevOptionsCurlBase', - 'data': { '*cookie': 'str' } } + 'data': { '*cookie': 'str', + '*cookie-secret': 'str'} } =20 ## # @BlockdevOptionsCurlHttps: @@ -2832,12 +2836,16 @@ # @sslverify: Whether to verify the SSL certificate's validity (defaults= to # true) # +# @cookie-secret: ID of a QCryptoSecret object providing the cookie data i= n a +# secure way. See @cookie for the format. (since 2.10) +# # Since: 2.9 ## { 'struct': 'BlockdevOptionsCurlHttps', 'base': 'BlockdevOptionsCurlBase', 'data': { '*cookie': 'str', - '*sslverify': 'bool' } } + '*sslverify': 'bool', + '*cookie-secret': 'str'} } =20 ## # @BlockdevOptionsCurlFtp: --=20 2.9.3 From nobody Fri May 3 18:25:12 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zoho.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1494950177287237.6249235972142; Tue, 16 May 2017 08:56:17 -0700 (PDT) Received: from localhost ([::1]:42889 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dAepb-0004TP-KK for importer@patchew.org; Tue, 16 May 2017 11:56:15 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:46603) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dAenv-0003FM-Mc for qemu-devel@nongnu.org; Tue, 16 May 2017 11:54:32 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dAenu-0001WY-VC for qemu-devel@nongnu.org; Tue, 16 May 2017 11:54:31 -0400 Received: from mx1.redhat.com ([209.132.183.28]:39690) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1dAens-0001VQ-Vq; Tue, 16 May 2017 11:54:29 -0400 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 8B18380480; Tue, 16 May 2017 15:54:27 +0000 (UTC) Received: from localhost (ovpn-116-10.phx2.redhat.com [10.3.116.10]) by smtp.corp.redhat.com (Postfix) with ESMTPS id E5C5CA92B1; Tue, 16 May 2017 15:54:24 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com 8B18380480 Authentication-Results: ext-mx04.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=redhat.com Authentication-Results: ext-mx04.extmail.prod.ext.phx2.redhat.com; spf=pass smtp.mailfrom=jcody@redhat.com DKIM-Filter: OpenDKIM Filter v2.11.0 mx1.redhat.com 8B18380480 From: Jeff Cody To: qemu-block@nongnu.org Date: Tue, 16 May 2017 11:54:14 -0400 Message-Id: <20170516155420.10106-3-jcody@redhat.com> In-Reply-To: <20170516155420.10106-1-jcody@redhat.com> References: <20170516155420.10106-1-jcody@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.28]); Tue, 16 May 2017 15:54:27 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PULL 2/8] curl: strengthen assertion in curl_clean_state X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org, jcody@redhat.com, qemu-stable@nongnu.org, qemu-devel@nongnu.org, Paolo Bonzini Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: Paolo Bonzini curl_clean_state should only be called after all AIOCBs have been completed. This is not so obvious for the call from curl_detach_aio_contex= t, so assert that. Cc: qemu-stable@nongnu.org Reviewed-by: Jeff Cody Signed-off-by: Paolo Bonzini Reviewed-by: Max Reitz Message-id: 20170515100059.15795-2-pbonzini@redhat.com Signed-off-by: Jeff Cody --- block/curl.c | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/block/curl.c b/block/curl.c index 4382234..562340f 100644 --- a/block/curl.c +++ b/block/curl.c @@ -533,6 +533,11 @@ static CURLState *curl_init_state(BlockDriverState *bs= , BDRVCURLState *s) =20 static void curl_clean_state(CURLState *s) { + int j; + for (j =3D 0; j < CURL_NUM_ACB; j++) { + assert(!s->acb[j]); + } + if (s->s->multi) curl_multi_remove_handle(s->s->multi, s->curl); =20 --=20 2.9.3 From nobody Fri May 3 18:25:12 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zoho.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1494950329292611.4987074671891; Tue, 16 May 2017 08:58:49 -0700 (PDT) Received: from localhost ([::1]:42906 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dAes1-0006oD-W6 for importer@patchew.org; Tue, 16 May 2017 11:58:46 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:46779) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dAeoC-0003U8-CC for qemu-devel@nongnu.org; Tue, 16 May 2017 11:54:49 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dAeoB-0001dU-C7 for qemu-devel@nongnu.org; Tue, 16 May 2017 11:54:48 -0400 Received: from mx1.redhat.com ([209.132.183.28]:52956) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1dAeo8-0001a5-3s; Tue, 16 May 2017 11:54:44 -0400 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id A6AFD37E72; Tue, 16 May 2017 15:54:27 +0000 (UTC) Received: from localhost (ovpn-116-10.phx2.redhat.com [10.3.116.10]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 3EF09A92AD; Tue, 16 May 2017 15:54:27 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com A6AFD37E72 Authentication-Results: ext-mx05.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=redhat.com Authentication-Results: ext-mx05.extmail.prod.ext.phx2.redhat.com; spf=pass smtp.mailfrom=jcody@redhat.com DKIM-Filter: OpenDKIM Filter v2.11.0 mx1.redhat.com A6AFD37E72 From: Jeff Cody To: qemu-block@nongnu.org Date: Tue, 16 May 2017 11:54:15 -0400 Message-Id: <20170516155420.10106-4-jcody@redhat.com> In-Reply-To: <20170516155420.10106-1-jcody@redhat.com> References: <20170516155420.10106-1-jcody@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.29]); Tue, 16 May 2017 15:54:38 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PULL 3/8] curl: never invoke callbacks with s->mutex held X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org, jcody@redhat.com, qemu-stable@nongnu.org, qemu-devel@nongnu.org, Paolo Bonzini Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: Paolo Bonzini All curl callbacks go through curl_multi_do, and hence are called with s->mutex held. Note that with comments, and make curl_read_cb drop the lock before invoking the callback. Likewise for curl_find_buf, where the callback can be invoked by the caller. Cc: qemu-stable@nongnu.org Reviewed-by: Jeff Cody Signed-off-by: Paolo Bonzini Reviewed-by: Max Reitz Message-id: 20170515100059.15795-3-pbonzini@redhat.com Signed-off-by: Jeff Cody --- block/curl.c | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/block/curl.c b/block/curl.c index 562340f..18b82bc 100644 --- a/block/curl.c +++ b/block/curl.c @@ -148,6 +148,7 @@ static void curl_multi_do(void *arg); static void curl_multi_read(void *arg); =20 #ifdef NEED_CURL_TIMER_CALLBACK +/* Called from curl_multi_do_locked, with s->mutex held. */ static int curl_timer_cb(CURLM *multi, long timeout_ms, void *opaque) { BDRVCURLState *s =3D opaque; @@ -164,6 +165,7 @@ static int curl_timer_cb(CURLM *multi, long timeout_ms,= void *opaque) } #endif =20 +/* Called from curl_multi_do_locked, with s->mutex held. */ static int curl_sock_cb(CURL *curl, curl_socket_t fd, int action, void *userp, void *sp) { @@ -213,6 +215,7 @@ static int curl_sock_cb(CURL *curl, curl_socket_t fd, i= nt action, return 0; } =20 +/* Called from curl_multi_do_locked, with s->mutex held. */ static size_t curl_header_cb(void *ptr, size_t size, size_t nmemb, void *o= paque) { BDRVCURLState *s =3D opaque; @@ -227,6 +230,7 @@ static size_t curl_header_cb(void *ptr, size_t size, si= ze_t nmemb, void *opaque) return realsize; } =20 +/* Called from curl_multi_do_locked, with s->mutex held. */ static size_t curl_read_cb(void *ptr, size_t size, size_t nmemb, void *opa= que) { CURLState *s =3D ((CURLState*)opaque); @@ -265,7 +269,9 @@ static size_t curl_read_cb(void *ptr, size_t size, size= _t nmemb, void *opaque) request_length - offset); } =20 + qemu_mutex_unlock(&s->s->mutex); acb->common.cb(acb->common.opaque, 0); + qemu_mutex_lock(&s->s->mutex); qemu_aio_unref(acb); s->acb[i] =3D NULL; } @@ -306,8 +312,6 @@ static int curl_find_buf(BDRVCURLState *s, size_t start= , size_t len, if (clamped_len < len) { qemu_iovec_memset(acb->qiov, clamped_len, 0, len - clamped= _len); } - acb->common.cb(acb->common.opaque, 0); - return FIND_RET_OK; } =20 @@ -854,8 +858,8 @@ static void curl_readv_bh_cb(void *p) // we can just call the callback and be done. switch (curl_find_buf(s, start, acb->nb_sectors * BDRV_SECTOR_SIZE, ac= b)) { case FIND_RET_OK: - qemu_aio_unref(acb); - // fall through + ret =3D 0; + goto out; case FIND_RET_WAIT: goto out; default: --=20 2.9.3 From nobody Fri May 3 18:25:12 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zoho.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1494950434714161.33951691030882; Tue, 16 May 2017 09:00:34 -0700 (PDT) Received: from localhost ([::1]:42928 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dAetl-00082r-7j for importer@patchew.org; Tue, 16 May 2017 12:00:33 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:46796) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dAeoC-0003UX-Tg for qemu-devel@nongnu.org; Tue, 16 May 2017 11:54:50 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dAeoB-0001dt-Tl for qemu-devel@nongnu.org; Tue, 16 May 2017 11:54:48 -0400 Received: from mx1.redhat.com ([209.132.183.28]:54506) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1dAeo7-0001a0-Va; Tue, 16 May 2017 11:54:44 -0400 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 8F57BC04B920; Tue, 16 May 2017 15:54:32 +0000 (UTC) Received: from localhost (ovpn-116-10.phx2.redhat.com [10.3.116.10]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 44570A92A8; Tue, 16 May 2017 15:54:28 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com 8F57BC04B920 Authentication-Results: ext-mx07.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=redhat.com Authentication-Results: ext-mx07.extmail.prod.ext.phx2.redhat.com; spf=pass smtp.mailfrom=jcody@redhat.com DKIM-Filter: OpenDKIM Filter v2.11.0 mx1.redhat.com 8F57BC04B920 From: Jeff Cody To: qemu-block@nongnu.org Date: Tue, 16 May 2017 11:54:16 -0400 Message-Id: <20170516155420.10106-5-jcody@redhat.com> In-Reply-To: <20170516155420.10106-1-jcody@redhat.com> References: <20170516155420.10106-1-jcody@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.31]); Tue, 16 May 2017 15:54:42 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PULL 4/8] curl: avoid recursive locking of BDRVCURLState mutex X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org, jcody@redhat.com, qemu-stable@nongnu.org, qemu-devel@nongnu.org, Paolo Bonzini Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: Paolo Bonzini The curl driver has a ugly hack where, if it cannot find an empty CURLState, it just uses aio_poll to wait for one to be empty. This is probably buggy when used together with dataplane, and the simplest way to fix it is to use coroutines instead. A more immediate effect of the bug however is that it can cause a recursive call to curl_readv_bh_cb and recursively taking the BDRVCURLState mutex. This causes a deadlock. The fix is to unlock the mutex around aio_poll, but for cleanliness we should also take the mutex around all calls to curl_init_state, even if reaching the unlock/lock pair is impossible. The same is true for curl_clean_state. Reported-by: Kun Wei Tested-by: Richard W.M. Jones Reviewed-by: Max Reitz Reviewed-by: Jeff Cody Signed-off-by: Paolo Bonzini Message-id: 20170515100059.15795-4-pbonzini@redhat.com Cc: qemu-stable@nongnu.org Cc: Jeff Cody Signed-off-by: Paolo Bonzini Signed-off-by: Jeff Cody --- block/curl.c | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/block/curl.c b/block/curl.c index 18b82bc..c160810 100644 --- a/block/curl.c +++ b/block/curl.c @@ -282,6 +282,7 @@ read_end: return size * nmemb; } =20 +/* Called with s->mutex held. */ static int curl_find_buf(BDRVCURLState *s, size_t start, size_t len, CURLAIOCB *acb) { @@ -454,6 +455,7 @@ static void curl_multi_timeout_do(void *arg) #endif } =20 +/* Called with s->mutex held. */ static CURLState *curl_init_state(BlockDriverState *bs, BDRVCURLState *s) { CURLState *state =3D NULL; @@ -472,7 +474,9 @@ static CURLState *curl_init_state(BlockDriverState *bs,= BDRVCURLState *s) break; } if (!state) { + qemu_mutex_unlock(&s->mutex); aio_poll(bdrv_get_aio_context(bs), true); + qemu_mutex_lock(&s->mutex); } } while(!state); =20 @@ -535,6 +539,7 @@ static CURLState *curl_init_state(BlockDriverState *bs,= BDRVCURLState *s) return state; } =20 +/* Called with s->mutex held. */ static void curl_clean_state(CURLState *s) { int j; @@ -566,6 +571,7 @@ static void curl_detach_aio_context(BlockDriverState *b= s) BDRVCURLState *s =3D bs->opaque; int i; =20 + qemu_mutex_lock(&s->mutex); for (i =3D 0; i < CURL_NUM_STATES; i++) { if (s->states[i].in_use) { curl_clean_state(&s->states[i]); @@ -581,6 +587,7 @@ static void curl_detach_aio_context(BlockDriverState *b= s) curl_multi_cleanup(s->multi); s->multi =3D NULL; } + qemu_mutex_unlock(&s->mutex); =20 timer_del(&s->timer); } @@ -684,6 +691,7 @@ static int curl_open(BlockDriverState *bs, QDict *optio= ns, int flags, return -EROFS; } =20 + qemu_mutex_init(&s->mutex); opts =3D qemu_opts_create(&runtime_opts, NULL, 0, &error_abort); qemu_opts_absorb_qdict(opts, options, &local_err); if (local_err) { @@ -769,7 +777,9 @@ static int curl_open(BlockDriverState *bs, QDict *optio= ns, int flags, DPRINTF("CURL: Opening %s\n", file); s->aio_context =3D bdrv_get_aio_context(bs); s->url =3D g_strdup(file); + qemu_mutex_lock(&s->mutex); state =3D curl_init_state(bs, s); + qemu_mutex_unlock(&s->mutex); if (!state) goto out_noclean; =20 @@ -813,11 +823,12 @@ static int curl_open(BlockDriverState *bs, QDict *opt= ions, int flags, } DPRINTF("CURL: Size =3D %zd\n", s->len); =20 + qemu_mutex_lock(&s->mutex); curl_clean_state(state); + qemu_mutex_unlock(&s->mutex); curl_easy_cleanup(state->curl); state->curl =3D NULL; =20 - qemu_mutex_init(&s->mutex); curl_attach_aio_context(bs, bdrv_get_aio_context(bs)); =20 qemu_opts_del(opts); @@ -828,6 +839,7 @@ out: curl_easy_cleanup(state->curl); state->curl =3D NULL; out_noclean: + qemu_mutex_destroy(&s->mutex); g_free(s->cookie); g_free(s->url); qemu_opts_del(opts); --=20 2.9.3 From nobody Fri May 3 18:25:12 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zoho.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1494950208934679.6277215000162; Tue, 16 May 2017 08:56:48 -0700 (PDT) Received: from localhost ([::1]:42890 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dAeq5-0004ps-Ev for importer@patchew.org; Tue, 16 May 2017 11:56:45 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:46798) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dAeoC-0003Ul-VG for qemu-devel@nongnu.org; Tue, 16 May 2017 11:54:50 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dAeoB-0001dy-Ub for qemu-devel@nongnu.org; Tue, 16 May 2017 11:54:49 -0400 Received: from mx1.redhat.com ([209.132.183.28]:52618) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1dAeo8-0001aR-St; Tue, 16 May 2017 11:54:45 -0400 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id A3DAB4E4C4; Tue, 16 May 2017 15:54:33 +0000 (UTC) Received: from localhost (ovpn-116-10.phx2.redhat.com [10.3.116.10]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 30934C09C1; Tue, 16 May 2017 15:54:33 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com A3DAB4E4C4 Authentication-Results: ext-mx09.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=redhat.com Authentication-Results: ext-mx09.extmail.prod.ext.phx2.redhat.com; spf=pass smtp.mailfrom=jcody@redhat.com DKIM-Filter: OpenDKIM Filter v2.11.0 mx1.redhat.com A3DAB4E4C4 From: Jeff Cody To: qemu-block@nongnu.org Date: Tue, 16 May 2017 11:54:17 -0400 Message-Id: <20170516155420.10106-6-jcody@redhat.com> In-Reply-To: <20170516155420.10106-1-jcody@redhat.com> References: <20170516155420.10106-1-jcody@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.38]); Tue, 16 May 2017 15:54:38 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PULL 5/8] curl: split curl_find_state/curl_init_state X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org, jcody@redhat.com, qemu-devel@nongnu.org, Paolo Bonzini Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: Paolo Bonzini If curl_easy_init fails, a CURLState is left with s->in_use =3D 1. Split curl_init_state in two, so that we can distinguish the two failures and call curl_clean_state if needed. While at it, simplify curl_find_state, removing a dummy loop. The aio_poll loop is moved to the sole caller that needs it. Reviewed-by: Jeff Cody Signed-off-by: Paolo Bonzini Reviewed-by: Max Reitz Message-id: 20170515100059.15795-5-pbonzini@redhat.com Signed-off-by: Jeff Cody --- block/curl.c | 52 ++++++++++++++++++++++++++++++---------------------- 1 file changed, 30 insertions(+), 22 deletions(-) diff --git a/block/curl.c b/block/curl.c index c160810..a522381 100644 --- a/block/curl.c +++ b/block/curl.c @@ -456,34 +456,27 @@ static void curl_multi_timeout_do(void *arg) } =20 /* Called with s->mutex held. */ -static CURLState *curl_init_state(BlockDriverState *bs, BDRVCURLState *s) +static CURLState *curl_find_state(BDRVCURLState *s) { CURLState *state =3D NULL; - int i, j; - - do { - for (i=3D0; istates[i].acb[j]) - continue; - if (s->states[i].in_use) - continue; + int i; =20 + for (i =3D 0; i < CURL_NUM_STATES; i++) { + if (!s->states[i].in_use) { state =3D &s->states[i]; state->in_use =3D 1; break; } - if (!state) { - qemu_mutex_unlock(&s->mutex); - aio_poll(bdrv_get_aio_context(bs), true); - qemu_mutex_lock(&s->mutex); - } - } while(!state); + } + return state; +} =20 +static int curl_init_state(BDRVCURLState *s, CURLState *state) +{ if (!state->curl) { state->curl =3D curl_easy_init(); if (!state->curl) { - return NULL; + return -EIO; } curl_easy_setopt(state->curl, CURLOPT_URL, s->url); curl_easy_setopt(state->curl, CURLOPT_SSL_VERIFYPEER, @@ -536,7 +529,7 @@ static CURLState *curl_init_state(BlockDriverState *bs,= BDRVCURLState *s) QLIST_INIT(&state->sockets); state->s =3D s; =20 - return state; + return 0; } =20 /* Called with s->mutex held. */ @@ -778,13 +771,18 @@ static int curl_open(BlockDriverState *bs, QDict *opt= ions, int flags, s->aio_context =3D bdrv_get_aio_context(bs); s->url =3D g_strdup(file); qemu_mutex_lock(&s->mutex); - state =3D curl_init_state(bs, s); + state =3D curl_find_state(s); qemu_mutex_unlock(&s->mutex); - if (!state) + if (!state) { goto out_noclean; + } =20 // Get file size =20 + if (curl_init_state(s, state) < 0) { + goto out; + } + s->accept_range =3D false; curl_easy_setopt(state->curl, CURLOPT_NOBODY, 1); curl_easy_setopt(state->curl, CURLOPT_HEADERFUNCTION, @@ -879,8 +877,18 @@ static void curl_readv_bh_cb(void *p) } =20 // No cache found, so let's start a new request - state =3D curl_init_state(acb->common.bs, s); - if (!state) { + for (;;) { + state =3D curl_find_state(s); + if (state) { + break; + } + qemu_mutex_unlock(&s->mutex); + aio_poll(bdrv_get_aio_context(bs), true); + qemu_mutex_lock(&s->mutex); + } + + if (curl_init_state(s, state) < 0) { + curl_clean_state(state); ret =3D -EIO; goto out; } --=20 2.9.3 From nobody Fri May 3 18:25:13 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zoho.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1494950418639187.6342016539403; Tue, 16 May 2017 09:00:18 -0700 (PDT) Received: from localhost ([::1]:42912 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dAetV-0007sF-1D for importer@patchew.org; Tue, 16 May 2017 12:00:17 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:46908) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dAeoK-0003dE-I4 for qemu-devel@nongnu.org; Tue, 16 May 2017 11:54:57 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dAeoJ-0001jA-HW for qemu-devel@nongnu.org; Tue, 16 May 2017 11:54:56 -0400 Received: from mx1.redhat.com ([209.132.183.28]:52602) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1dAeoE-0001fO-P2; Tue, 16 May 2017 11:54:50 -0400 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 8B56C3DBCC; Tue, 16 May 2017 15:54:39 +0000 (UTC) Received: from localhost (ovpn-116-10.phx2.redhat.com [10.3.116.10]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 33C56A92B7; Tue, 16 May 2017 15:54:34 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com 8B56C3DBCC Authentication-Results: ext-mx06.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=redhat.com Authentication-Results: ext-mx06.extmail.prod.ext.phx2.redhat.com; spf=pass smtp.mailfrom=jcody@redhat.com DKIM-Filter: OpenDKIM Filter v2.11.0 mx1.redhat.com 8B56C3DBCC From: Jeff Cody To: qemu-block@nongnu.org Date: Tue, 16 May 2017 11:54:18 -0400 Message-Id: <20170516155420.10106-7-jcody@redhat.com> In-Reply-To: <20170516155420.10106-1-jcody@redhat.com> References: <20170516155420.10106-1-jcody@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.30]); Tue, 16 May 2017 15:54:49 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PULL 6/8] curl: convert CURLAIOCB to byte values X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org, jcody@redhat.com, qemu-devel@nongnu.org, Paolo Bonzini Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: Paolo Bonzini This is in preparation for the conversion from bdrv_aio_readv to bdrv_co_preadv, and it also requires changing some of the size_t values to uint64_t. This was broken before for disks > 2TB, but now it would break at 4GB. Reviewed-by: Jeff Cody Signed-off-by: Paolo Bonzini Reviewed-by: Max Reitz Message-id: 20170515100059.15795-6-pbonzini@redhat.com Signed-off-by: Jeff Cody --- block/curl.c | 44 ++++++++++++++++++++++---------------------- 1 file changed, 22 insertions(+), 22 deletions(-) diff --git a/block/curl.c b/block/curl.c index a522381..a71a428 100644 --- a/block/curl.c +++ b/block/curl.c @@ -97,8 +97,8 @@ typedef struct CURLAIOCB { BlockAIOCB common; QEMUIOVector *qiov; =20 - int64_t sector_num; - int nb_sectors; + uint64_t offset; + uint64_t bytes; =20 size_t start; size_t end; @@ -116,7 +116,7 @@ typedef struct CURLState CURL *curl; QLIST_HEAD(, CURLSocket) sockets; char *orig_buf; - size_t buf_start; + uint64_t buf_start; size_t buf_off; size_t buf_len; char range[128]; @@ -127,7 +127,7 @@ typedef struct CURLState typedef struct BDRVCURLState { CURLM *multi; QEMUTimer timer; - size_t len; + uint64_t len; CURLState states[CURL_NUM_STATES]; char *url; size_t readahead_size; @@ -258,7 +258,7 @@ static size_t curl_read_cb(void *ptr, size_t size, size= _t nmemb, void *opaque) continue; =20 if ((s->buf_off >=3D acb->end)) { - size_t request_length =3D acb->nb_sectors * BDRV_SECTOR_SIZE; + size_t request_length =3D acb->bytes; =20 qemu_iovec_from_buf(acb->qiov, 0, s->orig_buf + acb->start, acb->end - acb->start); @@ -283,18 +283,18 @@ read_end: } =20 /* Called with s->mutex held. */ -static int curl_find_buf(BDRVCURLState *s, size_t start, size_t len, +static int curl_find_buf(BDRVCURLState *s, uint64_t start, uint64_t len, CURLAIOCB *acb) { int i; - size_t end =3D start + len; - size_t clamped_end =3D MIN(end, s->len); - size_t clamped_len =3D clamped_end - start; + uint64_t end =3D start + len; + uint64_t clamped_end =3D MIN(end, s->len); + uint64_t clamped_len =3D clamped_end - start; =20 for (i=3D0; istates[i]; - size_t buf_end =3D (state->buf_start + state->buf_off); - size_t buf_fend =3D (state->buf_start + state->buf_len); + uint64_t buf_end =3D (state->buf_start + state->buf_off); + uint64_t buf_fend =3D (state->buf_start + state->buf_len); =20 if (!state->orig_buf) continue; @@ -810,7 +810,7 @@ static int curl_open(BlockDriverState *bs, QDict *optio= ns, int flags, } #endif =20 - s->len =3D (size_t)d; + s->len =3D d; =20 if ((!strncasecmp(s->url, "http://", strlen("http://")) || !strncasecmp(s->url, "https://", strlen("https://"))) @@ -819,7 +819,7 @@ static int curl_open(BlockDriverState *bs, QDict *optio= ns, int flags, "Server does not support 'range' (byte ranges)."); goto out; } - DPRINTF("CURL: Size =3D %zd\n", s->len); + DPRINTF("CURL: Size =3D %" PRIu64 "\n", s->len); =20 qemu_mutex_lock(&s->mutex); curl_clean_state(state); @@ -859,14 +859,14 @@ static void curl_readv_bh_cb(void *p) BlockDriverState *bs =3D acb->common.bs; BDRVCURLState *s =3D bs->opaque; =20 - size_t start =3D acb->sector_num * BDRV_SECTOR_SIZE; - size_t end; + uint64_t start =3D acb->offset; + uint64_t end; =20 qemu_mutex_lock(&s->mutex); =20 // In case we have the requested data already (e.g. read-ahead), // we can just call the callback and be done. - switch (curl_find_buf(s, start, acb->nb_sectors * BDRV_SECTOR_SIZE, ac= b)) { + switch (curl_find_buf(s, start, acb->bytes, acb)) { case FIND_RET_OK: ret =3D 0; goto out; @@ -894,7 +894,7 @@ static void curl_readv_bh_cb(void *p) } =20 acb->start =3D 0; - acb->end =3D MIN(acb->nb_sectors * BDRV_SECTOR_SIZE, s->len - start); + acb->end =3D MIN(acb->bytes, s->len - start); =20 state->buf_off =3D 0; g_free(state->orig_buf); @@ -909,9 +909,9 @@ static void curl_readv_bh_cb(void *p) } state->acb[0] =3D acb; =20 - snprintf(state->range, 127, "%zd-%zd", start, end); - DPRINTF("CURL (AIO): Reading %llu at %zd (%s)\n", - (acb->nb_sectors * BDRV_SECTOR_SIZE), start, state->range); + snprintf(state->range, 127, "%" PRIu64 "-%" PRIu64, start, end); + DPRINTF("CURL (AIO): Reading %" PRIu64 " at %" PRIu64 " (%s)\n", + acb->bytes, start, state->range); curl_easy_setopt(state->curl, CURLOPT_RANGE, state->range); =20 curl_multi_add_handle(s->multi, state->curl); @@ -936,8 +936,8 @@ static BlockAIOCB *curl_aio_readv(BlockDriverState *bs, acb =3D qemu_aio_get(&curl_aiocb_info, bs, cb, opaque); =20 acb->qiov =3D qiov; - acb->sector_num =3D sector_num; - acb->nb_sectors =3D nb_sectors; + acb->offset =3D sector_num * BDRV_SECTOR_SIZE; + acb->bytes =3D nb_sectors * BDRV_SECTOR_SIZE; =20 aio_bh_schedule_oneshot(bdrv_get_aio_context(bs), curl_readv_bh_cb, ac= b); return &acb->common; --=20 2.9.3 From nobody Fri May 3 18:25:13 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zoho.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1494950318474687.7478335763312; Tue, 16 May 2017 08:58:38 -0700 (PDT) Received: from localhost ([::1]:42905 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dAers-0006gH-2z for importer@patchew.org; Tue, 16 May 2017 11:58:36 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:46801) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dAeoD-0003Us-0c for qemu-devel@nongnu.org; Tue, 16 May 2017 11:54:50 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dAeoB-0001df-L6 for qemu-devel@nongnu.org; Tue, 16 May 2017 11:54:49 -0400 Received: from mx1.redhat.com ([209.132.183.28]:54454) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1dAeo6-0001ZA-Q5; Tue, 16 May 2017 11:54:42 -0400 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id D5703C04B927; Tue, 16 May 2017 15:54:35 +0000 (UTC) Received: from localhost (ovpn-116-10.phx2.redhat.com [10.3.116.10]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 2A60AC09C0; Tue, 16 May 2017 15:54:35 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com D5703C04B927 Authentication-Results: ext-mx07.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=redhat.com Authentication-Results: ext-mx07.extmail.prod.ext.phx2.redhat.com; spf=pass smtp.mailfrom=jcody@redhat.com DKIM-Filter: OpenDKIM Filter v2.11.0 mx1.redhat.com D5703C04B927 From: Jeff Cody To: qemu-block@nongnu.org Date: Tue, 16 May 2017 11:54:19 -0400 Message-Id: <20170516155420.10106-8-jcody@redhat.com> In-Reply-To: <20170516155420.10106-1-jcody@redhat.com> References: <20170516155420.10106-1-jcody@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.31]); Tue, 16 May 2017 15:54:41 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PULL 7/8] curl: convert readv to coroutines X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org, jcody@redhat.com, qemu-devel@nongnu.org, Paolo Bonzini Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: Paolo Bonzini This is pretty simple. The bottom half goes away because, unlike bdrv_aio_readv, coroutine-based read can return immediately without yielding. However, for simplicity I kept the former bottom half handler in a separate function. Reviewed-by: Jeff Cody Signed-off-by: Paolo Bonzini Reviewed-by: Max Reitz Message-id: 20170515100059.15795-7-pbonzini@redhat.com Signed-off-by: Jeff Cody --- block/curl.c | 92 ++++++++++++++++++++++++--------------------------------= ---- 1 file changed, 37 insertions(+), 55 deletions(-) diff --git a/block/curl.c b/block/curl.c index a71a428..1c04903 100644 --- a/block/curl.c +++ b/block/curl.c @@ -76,10 +76,6 @@ static CURLMcode __curl_multi_socket_action(CURLM *multi= _handle, #define CURL_TIMEOUT_DEFAULT 5 #define CURL_TIMEOUT_MAX 10000 =20 -#define FIND_RET_NONE 0 -#define FIND_RET_OK 1 -#define FIND_RET_WAIT 2 - #define CURL_BLOCK_OPT_URL "url" #define CURL_BLOCK_OPT_READAHEAD "readahead" #define CURL_BLOCK_OPT_SSLVERIFY "sslverify" @@ -94,11 +90,12 @@ static CURLMcode __curl_multi_socket_action(CURLM *mult= i_handle, struct BDRVCURLState; =20 typedef struct CURLAIOCB { - BlockAIOCB common; + Coroutine *co; QEMUIOVector *qiov; =20 uint64_t offset; uint64_t bytes; + int ret; =20 size_t start; size_t end; @@ -269,11 +266,11 @@ static size_t curl_read_cb(void *ptr, size_t size, si= ze_t nmemb, void *opaque) request_length - offset); } =20 + acb->ret =3D 0; + s->acb[i] =3D NULL; qemu_mutex_unlock(&s->s->mutex); - acb->common.cb(acb->common.opaque, 0); + aio_co_wake(acb->co); qemu_mutex_lock(&s->s->mutex); - qemu_aio_unref(acb); - s->acb[i] =3D NULL; } } =20 @@ -283,8 +280,8 @@ read_end: } =20 /* Called with s->mutex held. */ -static int curl_find_buf(BDRVCURLState *s, uint64_t start, uint64_t len, - CURLAIOCB *acb) +static bool curl_find_buf(BDRVCURLState *s, uint64_t start, uint64_t len, + CURLAIOCB *acb) { int i; uint64_t end =3D start + len; @@ -313,7 +310,8 @@ static int curl_find_buf(BDRVCURLState *s, uint64_t sta= rt, uint64_t len, if (clamped_len < len) { qemu_iovec_memset(acb->qiov, clamped_len, 0, len - clamped= _len); } - return FIND_RET_OK; + acb->ret =3D 0; + return true; } =20 // Wait for unfinished chunks @@ -331,13 +329,13 @@ static int curl_find_buf(BDRVCURLState *s, uint64_t s= tart, uint64_t len, for (j=3D0; jacb[j]) { state->acb[j] =3D acb; - return FIND_RET_WAIT; + return true; } } } } =20 - return FIND_RET_NONE; + return false; } =20 /* Called with s->mutex held. */ @@ -382,11 +380,11 @@ static void curl_multi_check_completion(BDRVCURLState= *s) continue; } =20 + acb->ret =3D -EIO; + state->acb[i] =3D NULL; qemu_mutex_unlock(&s->mutex); - acb->common.cb(acb->common.opaque, -EIO); + aio_co_wake(acb->co); qemu_mutex_lock(&s->mutex); - qemu_aio_unref(acb); - state->acb[i] =3D NULL; } } =20 @@ -844,19 +842,11 @@ out_noclean: return -EINVAL; } =20 -static const AIOCBInfo curl_aiocb_info =3D { - .aiocb_size =3D sizeof(CURLAIOCB), -}; - - -static void curl_readv_bh_cb(void *p) +static void curl_setup_preadv(BlockDriverState *bs, CURLAIOCB *acb) { CURLState *state; int running; - int ret =3D -EINPROGRESS; =20 - CURLAIOCB *acb =3D p; - BlockDriverState *bs =3D acb->common.bs; BDRVCURLState *s =3D bs->opaque; =20 uint64_t start =3D acb->offset; @@ -866,14 +856,8 @@ static void curl_readv_bh_cb(void *p) =20 // In case we have the requested data already (e.g. read-ahead), // we can just call the callback and be done. - switch (curl_find_buf(s, start, acb->bytes, acb)) { - case FIND_RET_OK: - ret =3D 0; - goto out; - case FIND_RET_WAIT: - goto out; - default: - break; + if (curl_find_buf(s, start, acb->bytes, acb)) { + goto out; } =20 // No cache found, so let's start a new request @@ -889,7 +873,7 @@ static void curl_readv_bh_cb(void *p) =20 if (curl_init_state(s, state) < 0) { curl_clean_state(state); - ret =3D -EIO; + acb->ret =3D -EIO; goto out; } =20 @@ -904,7 +888,7 @@ static void curl_readv_bh_cb(void *p) state->orig_buf =3D g_try_malloc(state->buf_len); if (state->buf_len && state->orig_buf =3D=3D NULL) { curl_clean_state(state); - ret =3D -ENOMEM; + acb->ret =3D -ENOMEM; goto out; } state->acb[0] =3D acb; @@ -921,26 +905,24 @@ static void curl_readv_bh_cb(void *p) =20 out: qemu_mutex_unlock(&s->mutex); - if (ret !=3D -EINPROGRESS) { - acb->common.cb(acb->common.opaque, ret); - qemu_aio_unref(acb); - } } =20 -static BlockAIOCB *curl_aio_readv(BlockDriverState *bs, - int64_t sector_num, QEMUIOVector *qiov, int nb_sectors, - BlockCompletionFunc *cb, void *opaque) +static int coroutine_fn curl_co_preadv(BlockDriverState *bs, + uint64_t offset, uint64_t bytes, QEMUIOVector *qiov, int flags) { - CURLAIOCB *acb; + CURLAIOCB acb =3D { + .co =3D qemu_coroutine_self(), + .ret =3D -EINPROGRESS, + .qiov =3D qiov, + .offset =3D offset, + .bytes =3D bytes + }; =20 - acb =3D qemu_aio_get(&curl_aiocb_info, bs, cb, opaque); - - acb->qiov =3D qiov; - acb->offset =3D sector_num * BDRV_SECTOR_SIZE; - acb->bytes =3D nb_sectors * BDRV_SECTOR_SIZE; - - aio_bh_schedule_oneshot(bdrv_get_aio_context(bs), curl_readv_bh_cb, ac= b); - return &acb->common; + curl_setup_preadv(bs, &acb); + while (acb.ret =3D=3D -EINPROGRESS) { + qemu_coroutine_yield(); + } + return acb.ret; } =20 static void curl_close(BlockDriverState *bs) @@ -971,7 +953,7 @@ static BlockDriver bdrv_http =3D { .bdrv_close =3D curl_close, .bdrv_getlength =3D curl_getlength, =20 - .bdrv_aio_readv =3D curl_aio_readv, + .bdrv_co_preadv =3D curl_co_preadv, =20 .bdrv_detach_aio_context =3D curl_detach_aio_context, .bdrv_attach_aio_context =3D curl_attach_aio_context, @@ -987,7 +969,7 @@ static BlockDriver bdrv_https =3D { .bdrv_close =3D curl_close, .bdrv_getlength =3D curl_getlength, =20 - .bdrv_aio_readv =3D curl_aio_readv, + .bdrv_co_preadv =3D curl_co_preadv, =20 .bdrv_detach_aio_context =3D curl_detach_aio_context, .bdrv_attach_aio_context =3D curl_attach_aio_context, @@ -1003,7 +985,7 @@ static BlockDriver bdrv_ftp =3D { .bdrv_close =3D curl_close, .bdrv_getlength =3D curl_getlength, =20 - .bdrv_aio_readv =3D curl_aio_readv, + .bdrv_co_preadv =3D curl_co_preadv, =20 .bdrv_detach_aio_context =3D curl_detach_aio_context, .bdrv_attach_aio_context =3D curl_attach_aio_context, @@ -1019,7 +1001,7 @@ static BlockDriver bdrv_ftps =3D { .bdrv_close =3D curl_close, .bdrv_getlength =3D curl_getlength, =20 - .bdrv_aio_readv =3D curl_aio_readv, + .bdrv_co_preadv =3D curl_co_preadv, =20 .bdrv_detach_aio_context =3D curl_detach_aio_context, .bdrv_attach_aio_context =3D curl_attach_aio_context, --=20 2.9.3 From nobody Fri May 3 18:25:13 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zoho.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1494950620031918.4916456967048; Tue, 16 May 2017 09:03:40 -0700 (PDT) Received: from localhost ([::1]:43015 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dAewj-00028l-08 for importer@patchew.org; Tue, 16 May 2017 12:03:37 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:47012) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dAeoV-0003na-ID for qemu-devel@nongnu.org; Tue, 16 May 2017 11:55:08 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dAeoP-0001mA-Q9 for qemu-devel@nongnu.org; Tue, 16 May 2017 11:55:07 -0400 Received: from mx1.redhat.com ([209.132.183.28]:53336) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1dAeoN-0001ki-CD; Tue, 16 May 2017 11:54:59 -0400 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id F16EB23E6CD; Tue, 16 May 2017 15:54:42 +0000 (UTC) Received: from localhost (ovpn-116-10.phx2.redhat.com [10.3.116.10]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 7CCC11725D; Tue, 16 May 2017 15:54:36 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com F16EB23E6CD Authentication-Results: ext-mx05.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=redhat.com Authentication-Results: ext-mx05.extmail.prod.ext.phx2.redhat.com; spf=pass smtp.mailfrom=jcody@redhat.com DKIM-Filter: OpenDKIM Filter v2.11.0 mx1.redhat.com F16EB23E6CD From: Jeff Cody To: qemu-block@nongnu.org Date: Tue, 16 May 2017 11:54:20 -0400 Message-Id: <20170516155420.10106-9-jcody@redhat.com> In-Reply-To: <20170516155420.10106-1-jcody@redhat.com> References: <20170516155420.10106-1-jcody@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.29]); Tue, 16 May 2017 15:54:58 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PULL 8/8] curl: do not do aio_poll when waiting for a free CURLState X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org, jcody@redhat.com, qemu-devel@nongnu.org, Paolo Bonzini Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: Paolo Bonzini Instead, put the CURLAIOCB on a wait list and yield; curl_clean_state will wake the corresponding coroutine. Because of CURL's callback-based structure, we cannot easily convert everything to CoMutex/CoQueue; keeping the QemuMutex is simpler. However, CoQueue is a simple wrapper around a linked list, so we can easily use QSIMPLEQ and open-code a CoQueue, protected by the BDRVCURLState QemuMutex instead of a CoMutex. Reviewed-by: Jeff Cody Signed-off-by: Paolo Bonzini Reviewed-by: Max Reitz Message-id: 20170515100059.15795-8-pbonzini@redhat.com Signed-off-by: Jeff Cody --- block/curl.c | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-) diff --git a/block/curl.c b/block/curl.c index 1c04903..2a244e2 100644 --- a/block/curl.c +++ b/block/curl.c @@ -99,6 +99,8 @@ typedef struct CURLAIOCB { =20 size_t start; size_t end; + + QSIMPLEQ_ENTRY(CURLAIOCB) next; } CURLAIOCB; =20 typedef struct CURLSocket { @@ -134,6 +136,7 @@ typedef struct BDRVCURLState { bool accept_range; AioContext *aio_context; QemuMutex mutex; + QSIMPLEQ_HEAD(, CURLAIOCB) free_state_waitq; char *username; char *password; char *proxyusername; @@ -533,6 +536,7 @@ static int curl_init_state(BDRVCURLState *s, CURLState = *state) /* Called with s->mutex held. */ static void curl_clean_state(CURLState *s) { + CURLAIOCB *next; int j; for (j =3D 0; j < CURL_NUM_ACB; j++) { assert(!s->acb[j]); @@ -549,6 +553,14 @@ static void curl_clean_state(CURLState *s) } =20 s->in_use =3D 0; + + next =3D QSIMPLEQ_FIRST(&s->s->free_state_waitq); + if (next) { + QSIMPLEQ_REMOVE_HEAD(&s->s->free_state_waitq, next); + qemu_mutex_unlock(&s->s->mutex); + aio_co_wake(next->co); + qemu_mutex_lock(&s->s->mutex); + } } =20 static void curl_parse_filename(const char *filename, QDict *options, @@ -766,6 +778,7 @@ static int curl_open(BlockDriverState *bs, QDict *optio= ns, int flags, } =20 DPRINTF("CURL: Opening %s\n", file); + QSIMPLEQ_INIT(&s->free_state_waitq); s->aio_context =3D bdrv_get_aio_context(bs); s->url =3D g_strdup(file); qemu_mutex_lock(&s->mutex); @@ -866,8 +879,9 @@ static void curl_setup_preadv(BlockDriverState *bs, CUR= LAIOCB *acb) if (state) { break; } + QSIMPLEQ_INSERT_TAIL(&s->free_state_waitq, acb, next); qemu_mutex_unlock(&s->mutex); - aio_poll(bdrv_get_aio_context(bs), true); + qemu_coroutine_yield(); qemu_mutex_lock(&s->mutex); } =20 --=20 2.9.3