From nobody Wed Apr 8 04:41:05 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=quarantine dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1773160194; cv=none; d=zohomail.com; s=zohoarc; b=muTaI9WicS0rAxDOX6GBM9GPCzx/PrWmC8EKSWnPRSJsKS+qpY6BPrGnC9nqQJGypgW8rNm9y8/NBdYZGMHH0EUfVHsTNgx1TWqafzyuC76w1YVAdLRCgxA2XDFs/826pWH/Kz7pIVlijIx1LAs5/9E9WqgM5zMLb71SQfZJNS8= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1773160194; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=3NYvw0OxBs5HdokP5YldS9J+T2s8QfdGmfB9yCRG9hQ=; b=Blv1Wh8kcowP+vlUhjtif4uEK76gF8Jmadx3n7fgEWX/J1Kyk7TYX8yF6kSu+y24ZEZ5BU4CVkDHQmnzSogKFwkxOxEKRv3hyfAl+laPSwTPUKPfZRGB6IqJmojuTErRdBVrOtCNtg7ErdQBZZz3K+67a3Zkv0QNxOlf3RrNZHY= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=quarantine dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1773160194417621.3689955078261; Tue, 10 Mar 2026 09:29:54 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1vzzvZ-0003tO-9c; Tue, 10 Mar 2026 12:27:25 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1vzzvW-0003h4-Kx for qemu-devel@nongnu.org; Tue, 10 Mar 2026 12:27:22 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1vzzvT-00010E-Rq for qemu-devel@nongnu.org; Tue, 10 Mar 2026 12:27:22 -0400 Received: from mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-416-Ime2abjDPvCAHVFPOXL4UA-1; Tue, 10 Mar 2026 12:27:17 -0400 Received: from mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.12]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 7A6CF19560B3; Tue, 10 Mar 2026 16:27:16 +0000 (UTC) Received: from merkur.fritz.box (unknown [10.45.224.112]) by mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 10E9819560A6; Tue, 10 Mar 2026 16:27:14 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1773160038; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=3NYvw0OxBs5HdokP5YldS9J+T2s8QfdGmfB9yCRG9hQ=; b=ahiULIqrdyrPRwfx+vXYZV8GtaeGtggzzJRdWkSg6mKnK5jmvtFqUjyUBoSQQ4KQ5BvI+8 gcwwPB2lAVwNh9OHG/vLlaDtrRWee7QQOnGzTsE4o6QrKNdQxKeDdnSziedD3CsB/XD0j7 3VEVNdLSYXhwff/pFggayDOExRLOVm4= X-MC-Unique: Ime2abjDPvCAHVFPOXL4UA-1 X-Mimecast-MFC-AGG-ID: Ime2abjDPvCAHVFPOXL4UA_1773160036 From: Kevin Wolf To: qemu-block@nongnu.org Cc: kwolf@redhat.com, qemu-devel@nongnu.org Subject: [PULL 28/28] block/curl: add support for S3 presigned URLs Date: Tue, 10 Mar 2026 17:26:22 +0100 Message-ID: <20260310162622.333137-29-kwolf@redhat.com> In-Reply-To: <20260310162622.333137-1-kwolf@redhat.com> References: <20260310162622.333137-1-kwolf@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.0 on 10.30.177.12 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.133.124; envelope-from=kwolf@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -3 X-Spam_score: -0.4 X-Spam_bar: / X-Spam_report: (-0.4 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.819, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.903, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: qemu development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1773160195475158500 Content-Type: text/plain; charset="utf-8" From: Antoine Damhet S3 presigned URLs are signed for a specific HTTP method (typically GET for our use cases). The curl block driver currently issues a HEAD request to discover the web server features and the file size, which fails with 'HTTP 403' (forbidden). Add a 'force-range' option that skips the HEAD request and instead issues a minimal GET request (querying 1 byte from the server) to extract the file size from the 'Content-Range' response header. To achieve this the 'curl_header_cb' is redesigned to generically parse HTTP headers. $ $QEMU -drive driver=3Dhttps,\ 'url=3Dhttps://s3.example.com/some.img?X-Amz-Security-Token=3D= XXX', force-range=3Dtrue Enabling the 'force-range' option without the web server specified with @url supporting it might cause the server to respond successfully with 'HTTP 200' and attempt to send the whole file body. With the 'CURLOPT_NOBODY' option set the libcurl will skip reading after the headers and close the connection. QEMU still gracefully detects the missing feature. This might waste a small number of TCP packets but is otherwise transparent to the user. Acked-by: Markus Armbruster Signed-off-by: Antoine Damhet Message-ID: <20260227-fix-curl-v3-v3-3-eb8a4d88feef@scaleway.com> Reviewed-by: Kevin Wolf Signed-off-by: Kevin Wolf --- qapi/block-core.json | 8 +- docs/system/device-url-syntax.rst.inc | 6 ++ block/curl.c | 104 ++++++++++++++++++-------- block/trace-events | 1 + 4 files changed, 85 insertions(+), 34 deletions(-) diff --git a/qapi/block-core.json b/qapi/block-core.json index 8ba1fdc49d6..f8d446b3d6e 100644 --- a/qapi/block-core.json +++ b/qapi/block-core.json @@ -4587,12 +4587,18 @@ # @cookie-secret: ID of a QCryptoSecret object providing the cookie # data in a secure way. See @cookie for the format. (since 2.10) # +# @force-range: Don't issue a HEAD HTTP request to discover if the +# http server supports range requests and rely only on GET +# requests. This is especially useful for S3 presigned URLs where +# HEAD requests are unauthorized. (default: false; since 11.0) +# # Since: 2.9 ## { 'struct': 'BlockdevOptionsCurlHttp', 'base': 'BlockdevOptionsCurlBase', 'data': { '*cookie': 'str', - '*cookie-secret': 'str'} } + '*cookie-secret': 'str', + '*force-range': 'bool'} } =20 ## # @BlockdevOptionsCurlHttps: diff --git a/docs/system/device-url-syntax.rst.inc b/docs/system/device-url= -syntax.rst.inc index aae65d138c0..996ce5418ff 100644 --- a/docs/system/device-url-syntax.rst.inc +++ b/docs/system/device-url-syntax.rst.inc @@ -179,6 +179,12 @@ These are specified using a special URL syntax. get the size of the image to be downloaded. If not set, the default timeout of 5 seconds is used. =20 + ``force-range`` + Don't issue a HEAD HTTP request to discover if the http server + server supports range requests and rely only on GET requests. This + is especially useful for S3 presigned URLs where HEAD requests + are unauthorized. It defaults to 'false'. + Note that when passing options to qemu explicitly, ``driver`` is the value of . =20 diff --git a/block/curl.c b/block/curl.c index 6dccf002564..66aecfb20ec 100644 --- a/block/curl.c +++ b/block/curl.c @@ -62,10 +62,12 @@ #define CURL_BLOCK_OPT_PASSWORD_SECRET "password-secret" #define CURL_BLOCK_OPT_PROXY_USERNAME "proxy-username" #define CURL_BLOCK_OPT_PROXY_PASSWORD_SECRET "proxy-password-secret" +#define CURL_BLOCK_OPT_FORCE_RANGE "force-range" =20 #define CURL_BLOCK_OPT_READAHEAD_DEFAULT (256 * 1024) #define CURL_BLOCK_OPT_SSLVERIFY_DEFAULT true #define CURL_BLOCK_OPT_TIMEOUT_DEFAULT 5 +#define CURL_BLOCK_OPT_FORCE_RANGE_DEFAULT false =20 struct BDRVCURLState; struct CURLState; @@ -206,27 +208,33 @@ static size_t curl_header_cb(void *ptr, size_t size, = size_t nmemb, void *opaque) { BDRVCURLState *s =3D opaque; size_t realsize =3D size * nmemb; - const char *p =3D ptr; - const char *end =3D p + realsize; - const char *t =3D "accept-ranges : bytes "; /* A lowercase template */ + g_autofree char *header =3D g_strstrip(g_strndup(ptr, realsize)); + char *val =3D strchr(header, ':'); =20 - /* check if header matches the "t" template */ - for (;;) { - if (*t =3D=3D ' ') { /* space in t matches any amount of isspace i= n p */ - if (p < end && g_ascii_isspace(*p)) { - ++p; - } else { - ++t; - } - } else if (*t && p < end && *t =3D=3D g_ascii_tolower(*p)) { - ++p, ++t; - } else { - break; - } + if (!val) { + return realsize; } =20 - if (!*t && p =3D=3D end) { /* if we managed to reach ends of both stri= ngs */ - s->accept_range =3D true; + *val++ =3D '\0'; + g_strchomp(header); + while (g_ascii_isspace(*val)) { + ++val; + } + + trace_curl_header_cb(header, val); + + if (!g_ascii_strcasecmp(header, "accept-ranges")) { + if (!g_ascii_strcasecmp(val, "bytes")) { + s->accept_range =3D true; + } + } else if (!g_ascii_strcasecmp(header, "Content-Range")) { + /* Content-Range fmt is `bytes begin-end/full_size` */ + val =3D strchr(val, '/'); + if (val) { + if (qemu_strtou64(val + 1, NULL, 10, &s->len) < 0) { + s->len =3D UINT64_MAX; + } + } } =20 return realsize; @@ -668,6 +676,11 @@ static QemuOptsList runtime_opts =3D { .type =3D QEMU_OPT_STRING, .help =3D "ID of secret used as password for HTTP proxy auth", }, + { + .name =3D CURL_BLOCK_OPT_FORCE_RANGE, + .type =3D QEMU_OPT_BOOL, + .help =3D "Assume HTTP range requests are supported", + }, { /* end of list */ } }, }; @@ -690,6 +703,7 @@ static int curl_open(BlockDriverState *bs, QDict *optio= ns, int flags, #endif const char *secretid; const char *protocol_delimiter; + bool force_range; int ret; =20 bdrv_graph_rdlock_main_loop(); @@ -807,35 +821,56 @@ static int curl_open(BlockDriverState *bs, QDict *opt= ions, int flags, } =20 s->accept_range =3D false; + s->len =3D UINT64_MAX; + force_range =3D qemu_opt_get_bool(opts, CURL_BLOCK_OPT_FORCE_RANGE, + CURL_BLOCK_OPT_FORCE_RANGE_DEFAULT); + /* + * When minimal CURL will be bumped to `7.83`, the header callback + m= anual + * parsing can be replaced by `curl_easy_header` calls + */ if (curl_easy_setopt(state->curl, CURLOPT_NOBODY, 1L) || curl_easy_setopt(state->curl, CURLOPT_HEADERFUNCTION, curl_header_= cb) || curl_easy_setopt(state->curl, CURLOPT_HEADERDATA, s)) { - pstrcpy(state->errmsg, CURL_ERROR_SIZE, - "curl library initialization failed."); - goto out; + goto out_init; + } + if (force_range) { + if (curl_easy_setopt(state->curl, CURLOPT_CUSTOMREQUEST, "GET") || + curl_easy_setopt(state->curl, CURLOPT_RANGE, "0-0")) { + goto out_init; + } } + if (curl_easy_perform(state->curl)) goto out; - /* CURL 7.55.0 deprecates CURLINFO_CONTENT_LENGTH_DOWNLOAD in favour of - * the *_T version which returns a more sensible type for content leng= th. - */ + + if (!force_range) { + /* + * CURL 7.55.0 deprecates CURLINFO_CONTENT_LENGTH_DOWNLOAD in favo= ur of + * the *_T version which returns a more sensible type for content + * length. + */ #if LIBCURL_VERSION_NUM >=3D 0x073700 - if (curl_easy_getinfo(state->curl, CURLINFO_CONTENT_LENGTH_DOWNLOAD_T,= &cl)) { - goto out; - } + if (curl_easy_getinfo(state->curl, CURLINFO_CONTENT_LENGTH_DOWNLOA= D_T, + &cl)) { + goto out; + } #else - if (curl_easy_getinfo(state->curl, CURLINFO_CONTENT_LENGTH_DOWNLOAD, &= cl)) { - goto out; - } + if (curl_easy_getinfo(state->curl, CURLINFO_CONTENT_LENGTH_DOWNLOA= D, + &cl)) { + goto out; + } #endif - if (cl < 0) { + if (cl >=3D 0) { + s->len =3D cl; + } + } + + if (s->len =3D=3D UINT64_MAX) { pstrcpy(state->errmsg, CURL_ERROR_SIZE, "Server didn't report file size."); goto out; } =20 - s->len =3D cl; - if ((!strncasecmp(s->url, "http://", strlen("http://")) || !strncasecmp(s->url, "https://", strlen("https://"))) && !s->accept_range) { @@ -856,6 +891,9 @@ static int curl_open(BlockDriverState *bs, QDict *optio= ns, int flags, qemu_opts_del(opts); return 0; =20 +out_init: + pstrcpy(state->errmsg, CURL_ERROR_SIZE, + "curl library initialization failed."); out: error_setg(errp, "CURL: Error opening file: %s", state->errmsg); curl_easy_cleanup(state->curl); diff --git a/block/trace-events b/block/trace-events index c9b4736ff88..d170fc96f15 100644 --- a/block/trace-events +++ b/block/trace-events @@ -191,6 +191,7 @@ ssh_server_status(int status) "server status=3D%d" curl_timer_cb(long timeout_ms) "timer callback timeout_ms %ld" curl_sock_cb(int action, int fd) "sock action %d on fd %d" curl_read_cb(size_t realsize) "just reading %zu bytes" +curl_header_cb(const char *key, const char *val) "looking at %s: %s" curl_open(const char *file) "opening %s" curl_open_size(uint64_t size) "size =3D %" PRIu64 curl_setup_preadv(uint64_t bytes, uint64_t start, const char *range) "read= ing %" PRIu64 " at %" PRIu64 " (%s)" --=20 2.53.0